i'dWc@sdZddlZddlZddlZddlmZddlmZddljZ ddl jj Z ej jZeZdZdZdZdZd ed ed ed Zed ed ZejdededZejdeddededdedZejdedZejdedZejdeddeddededZejdedZdefdYZ dS(s? mim.py: Manifest Input Module. Provides functionality to load, modify (via set, add, overlay), retrieve info, validate and write out an XML manifest file. Nodes are specified as a path from the root. Throughout this module, a "branch" is a part of a path between two unquoted or unbracketed slashes. Nodepath provided can have branches of the following form: /a # element defined by tag /a=5 # element defined by tag and value (non-leaf) or # assign a value to element defined by tag (leaf) /a[b=6] # element defined by tag, and tag and value of a child /a[b/c=7] # element defined by tag, and tag and value of # non-direct decendent /a[2] # element defined by tag and ID in tree /a@attr # (leaf) attribute defined by name and the element # it is a part of /a@attr=8 # (leaf) element defined by tag, and the name and value # of an attribute it has. /a[@attr=8] # element defined by tag, and the name and value of an # attribute it has. iN(tStringIO(tetrees [A-Za-z]\w*s("[^ "]+")s('[^ ']+')s([^ '"/\[\]=@]+)t(t|t)s(/s)*t^t=t$s)?s\[s(@s\]$s((s@)?s\]s \[.*=.*\]$t ManifestInputcBsIeZdZdedZdZedZedZ dZ dZ ddZ e edZd Zd Zd Zed Zd ZddZdZdZdZdZdZedZedZedZedZdZdZedZ ee dZ!RS(sz Class which implements the Manifest Input Module proper. Provides the functionality to manipulate XML files. cCsl|dkr!tjtjn||_d|_d|_d|_d|_d|_ d|_ t |_ t jdtdtd||_ytjj|j}Wn1tk r}|jtjkrd}qnX|r|j|j|j|_n||_ |jrL|jjrL|jjj|_|dkrL|jjj}qLn|dk rh|j|ndS(s Instantiate a new object. Initialization includes: - querying the environment for AIM_MANIFEST to get the manifest filename. If named file is present, initialize data tree from it. If not set or file is not available, display a warning and continue. - Opening and storing the schema data. Err out if not accessible or incorrect. - Call process_dtd module to create tables from DTD for ordering child nodes in the data tree. These tables will be used for doing overlays. Args: evolving_file: Pathname to the file providing initial data, and storing the result. schema_file: Pathname to the DTD file to use. Overrides any DTD specified in the manifest, if specified. Optional. attribute_defaults: boolean, setting passed to instantiation of etree.XMLParser. Note that the defaults are taken from the embedded DTD in the manifest, even if a different DTD is passed in via "schema_file". Optional. Raises: IOError - Could not access DTD file IOError - Could not digest DTD file MimDTDError - SchemaData error processing DTD data from file MimDTDInvalid - Error parsing DTD MimEtreeParseError - Error parsing XML manifest file MimInvalidError - No output file name provided MimInvalidError - No schema name provided OSError from not being able to access the manifest tremove_blank_textt remove_pistattribute_defaultsiN(tNonetmilibtMimInvalidErrort ERR_NO_EVFILEt evfile_namettreetschemat schema_datatschema_file_from_mfesttschema_file_from_inittschema_file_from_overlaytFalset commit_neededRt XMLParsertTruetparsertostpathtgetsizetOSErrorterrnotENOENTtparse_xml_filetdocinfot system_urlt load_schema(tselft evolving_filet schema_fileR t mfest_sizeterr((smim.pyt__init__ws6&              cCsa|dkr!tjtjnytj||_Wntk rz}t|jdtj i|j d6|d6nJtj k r}tj g|j jtD]}|j^qnXytj||_Wntk r}t|jdtji|j d6|d6n@tjk r\}tjtji|d6t|d6nXdS(s # Open schema for validator, and build table of children order. Args: schema_file: DTD Returns: initializes self.schema and self.schema_data Raises: IOError - Could not access DTD file. IOError - Could not digest DTD file. MimDTDInvalid - Error parsing DTD MimDTDError - SchemaData error processing DTD data from file. itmserrtmfiletmerrN(R R Rt ERR_NO_SCHEMARtDTDRtIOErrortargstIOERR_DTD_ACCESStstrerrort DTDParseErrort MimDTDInvalidt error_logtfilter_from_leveltGET_ALLt__repr__tpdtdt SchemaDataRtIOERR_DTD_DIGESTtMimErrort MimDTDErrortERR_SCHDATA_PROCtstr(R&R(R*tmsg((smim.pyR%s& / cCseytj||}WnHtjk r`tjg|jjtD]}|j^qBnX|S(s[ Call etree.parse with proper exception handling. Args: manifest_name: XML file to parse parser: parser to use Returns: tree: XML data tree as set up by etree.parse Raises: MimEtreeParseError: Error parsing XML manifest. IOError: Error reading manifest ( RtparsetXMLSyntaxErrorR tMimEtreeParseErrorR7R8R9R:(t manifest_nameRRRB((smim.pyR"s/cCs(|dkr!tjtjn| s2|j r|j||j|_|jr$|jjr$|jjj|_ |jjjdk r|j r|j |jjjqq$nu|j||j}|r|jr|j r|j |jj|jj|_ n|j |jj|jddS(s) Args: overlay_filename: pathname or URL of XML file containing replacement data (if incremental=False) or data to overlay (incremental=True) incremental: True: overlay data on top of existing. False: replace existing data. Raises: IOError - Error reading overlay_filename MimInvalidError - Argument is missing or invalid MimEtreeParseError - IO errors or parser errors while parsing. MimDTDInvalid - Error reading DTD N(R R RtERR_ARG_INVALIDRR"RR#R$RRR%Rt_overlay_recursetgetroot(R&toverlay_filenamet incrementalt overlay_tree((smim.pytload s  cCs|stjtjn|js9tjtjntj|\}}}|j||}t |dkrtj tj n||fS(si Retrieve a list of elements that match path. Args: path: Xpath-like expression of element or attribute to retrieve. (see header of this file for syntax) Returns: orig_list: List of matching elements. attr: attribute as stripped off of path. Will be None if path refers to an element and not an attribute. Raises: MimInvalidError: Argument is missing or invalid MimEmptyTreeError: No XML data present. MimMatchError: Path matches no elements. Errors raised by etree.getpath() i( R RRGRtMimEmptyTreeErrortERR_EMPTY_TREERt_path_preprocesst _xpath_searchtlent MimMatchErrortERR_NO_ELEM_MATCH(R&Rtxpatht final_valtattrt orig_list((smim.pyt __getlist:s cCsp|jj|}|jdjd}d}x9|D]1}|d|7}|ddkr7|d7}q7q7W|S(s Get path to current element, in a form to return to the user. If Xpath doesn't append an identifier to a node since there is only one of that node, append a [1] to it. Do this because when a second like-tagged node is added, Xpath will automatically append the [1] to the current node and we want the path to be consistent. Args: element: The element to return the path to. Returns: path to the given element. t/tit]s[1](Rtgetpathtlstriptsplit(R&telementtrpathtbranchestbranch((smim.pyR]Zs cCs|js!tjtjgn|js?tjtjn|dkrW|j}n[tj dt dt }t tj |j}|j ||}|j|}|d}y|jj|WnKtjk rtjg|jjjtD]}|j^qnXdS(sJ Perform XML validation against the DTD. Args: idxpath: indexed path to top of subtree if a subtree, rather than entire tree, is to be validated Raises: MimEmptyTreeError - No XML data present Various lxml.etree (StandardError subclass) exceptions R R iN(RR R6R/RRNROR RRRRttostringR"RUt assertValidtDocumentInvalidR7R8R9R:(R&tidxpatht tree_to_uset tmpparsertdatattmptreetmyelemRB((smim.pytvalidatess$      cCs|jstjtjn|r1|jntj|jdtjt}|r~d|jj j |j f|d itwNR,tmdest(RR RNRORmRRdRt splitlinesRIttagRRtinsertRtopenRt writelinesR1tIOERR_DTD_DESTR4RR(R&Rmtuse_mfest_schematxml_datatoutfileR*((smim.pytcommits2  !#  #     "cCs|dkr!tjtjn|j|\}}g}xL|D]D}|dkra||_n|j|||j|j|qCWt |_ ||dk fS(s Change a value or add/change an attribute in one or more elements Args: path: Xpath-like expression of element to change or attribute to change/set. (see header of this file for syntax) value: Value to set. "" is allowed. Returns: list of rpaths: An Xpath-like expression for each retrieved element. Each rpath branch refers to a single element by referencing a numeric element ID (for example "/auto_install[1]") is_attr: boolean which is True if path is to an attribute Raises: MimInvalidError - Argument is missing or invalid Errors raised by __getlist() Errors raised by etree.getpath() N( R R RRGt_ManifestInput__getlistttexttsettappendR]RR(R&Rtvaluet elem_listRWtret_listtelem((smim.pyR}s     cCs|j|\}}g}x||D]t}|dk rY||jkr"|j|}qbq"n |j}|j|dk r|jnd|j|fq"Wt|stj tj n|S(s Retrieve a value or attribute from elements matching "path". Args: path: Xpath-like expression of element or attribute to retrieve. (see header of this file for syntax) Returns: one or more tuples consisting of: rval: value requested, stripped of any enveloping white space rpath: Xpath-like expression of retrieved element. Each rpath branch refers to a single element by referencing a numeric element ID (for example "/auto_install[1]") Raises: MimMatchError - Path matches no attributes Errors raised by __getlist() N( R{R tattribR|R~tstripR]RRR RStERR_NO_ATTR_MATCH(R&RRRWtrvalsRtrval((smim.pytgets   ! cCs#|j|\}}g}|dkrQx'|D]}|j|j|q.Wnx|D]}|dkr|j}|dk r|j|qtjtjqX||j j krX|j |=|j|j|qXqXWt |dkr tj tj nt|_||dk fS(s Delete elements or an attribute within matching elements. The root of the tree cannot be deleted. Args: path: Xpath-like expression of element or attribute to retrieve. (see header of this file for syntax) Returns: list of rpaths: An Xpath-like expression for each retrieved element. Each rpath branch refers to a single element by referencing a numeric element ID (for example "/auto_install[1]") is_attr: boolean which is True if path is to an attribute Raises: MimInvalidError - Cannot delete tree root node MimMatchError - Path matches no attributes Errors raised by __getlist() iN(R{R R~R]t getparenttremoveR Rt ERR_DEL_ROOTRtkeysRRRSRRR(R&RRRWRRtparent((smim.pytdelete"s$        cCs=x6|D](}tj||}|dkrPqqWd}|S(s Return index of where to insert an element in curr_elem's children. Find which element in list_insert_before exists as a child of curr_elem, and return its index. (The child to insert can be inserted at that position, before the found element.) Args: list_insert_before: list of elements curr_elem can be inserted before It is assumed that list_insert_before was generated based on the element to insert. It can be empty but not None. curr_elem: parent element under which the child will be inserted. Returns: Index of where to insert the child. i(Rtsearch_children_for_tag_match(tlist_insert_beforet curr_elemtchildt insert_at_idx((smim.pytfind_insertion_index\s   cCsw| s|dkr(tjtjntj|t \}}}|jd}tj|}t j |dstjtj n|j st j|d}t j||_ n|st|dkst j |drp|d|j jjkrptjtjqpnVt|dkrpt j |drp|d|j jjkrptjtjng} x3|rt j |dr| jd|jqyW|rvd} tj |ds|j|\}} nddj|} |r d| } n|j| | } t| dkrBtjtjn't| dkritjtjn| d} nn|r|j j} | }|jj| j|djd} | j |dn|j j} | d} xC| D];}|jr|jj!| j|\}}n g}t"}|dkr]tjtj#i|d6| jd6nt| s~t j$| |} qtj%| |}|dkrrtj&|| }t j$| |}|dkr| j||n|jrC|jj'| |}|dk r!tj&|g| }n|dkrC| j||qCn|j sY|dkri| j(|n|} q|rt j$| |}|}xKt)|dt| D]0}| |j| |jkr|}Pn|}qW| j|||} q|| dkrtjtj*n| |} qW|dk rI| j+||n|dk ra|| _,nt"|_-|j.| S( sN Add an element. Parent portion of the given path must match only one element. Element must be given a value, or alternatively, the path may go to an attribute of the new element. (Note: use set to change or add an attribute to an existing element.) General algorithm: - If path has non-simple* branches, the portion of the path from the beginning until the last non-simple branch inclusive, must lead to a unique node. - From that unique node and beyond, or if the entire path is simple, from the beginning of the path, the path is checked branch by branch, and if duplicates are allowed (or, of course, if the node does not exist), a new node is created. Otherwise, an existing one is followed. * A simple branch is simply an identifier. /a/b are two simple branches. A non-simple branch is any other kind of branch, such as /a=5 or /b[c/d@e=123]. Non-simple branches always specify a value and may specify a subpath. The goal here is to honor subpaths which may be specified to narrow down where to add new items, but to still create a second node of a given tag where duplicates are allowed, where appropriate. Args: path: Xpath-like expression of element or attribute to retrieve. (see header of this file for syntax) value: Value to set. "" is allowed. Returns: rpath: Xpath-like expression of retrieved element. Each rpath branch refers to a single element by referencing a numeric element ID (for example "/auto_install[1]") Raises: MimInvalidError - Argument is missing or invalid MimInvalidError - Final path branch has a value or is invalid MimInvalidError - Cannot add a second tree root node MimMatchError - Ambiguity error: Parent path matches more than one element MimMatchError - No path found to element Errors raised by etree.getpath() s//iiiRZtmnodetmparentN(/R R RRGRRPtSTRIP_FINAL_UNBKT_VALUEt startswitht branch_splitt IDENT_ONLY_REtmatchtERR_FINAL_BRANCH_VAL_INVALIDRRtElementt ElementTreeRRRIRrt ERR_2ND_ROOTRstpoptBKT_NODE_VALUE_REt_strip_final_valuetjoinRQRStERR_AMBIG_PARENT_PATHtERR_NO_PARENT_PATHRtfind_parent_pathRtextendtfind_element_infoRtERR_NODE_PLACEMENTt SubElementRRtfind_sparse_locnR~tranget ERR_LEAF_DUPSR}R|RR](R&RRRURVRWtmidstarttleft_sett new_elementt right_settfinal_branch_valuet left_pathtnonsimple_targetsRt old_right_setRct insert_beforetmults_oktnode_w_same_tagt insertion_idxtnew_elemt name_aftert insert_aftert list_elem_idx((smim.pytadds3        !                       cCsy|jj|}Wn:tjk rR}tjtji|jdd6nXt|dkr|dk rg}|j d}xX|D]P}|j |kr|j |q|j dkr|dkr|j |qqW|}n|S(s^ Perform an xpath search Args: xpath: xpath of the elements to retrieve. final_val: value of path's final element, or None if none Returns: A list of elements which match the xpath. Can be an empty list. Raises: MimInvalidError - Error parsing path : iR.is'"R[N( RRURtXPathEvalErrorR RtERR_ETREE_PARSE_XPATHR2RRR RR|R~(R&RURVRR*tnew_rvalR((smim.pyRQns   cCsqd|kr$|jd\}}n|jdd\}}}|jj||\}}|sddS|j|fS(sy Get the quantity defined for an element and its siblings (defined as or'd elements) Args: path: xpath of the element Returns: tuple (qty_for_path_element, sib_dict) or (None, None) e.g., for: '/auto_install/ai_instance/software/source/publisher' with DTD definition: returns tuple info from find_element_data(): ('*', {'sibs': {'publisher': '*', 'file': '*', 'dir': '1'}, 'qty': '1'}) Raises: nothing t@RZiN(NN(R_trsplitRtfind_element_dataR tqty(R&Rt _attr_namet _parent_patht parent_tagt element_tagt elem_datatsib_dict((smim.pytget_element_qtys  cCsd|krd S|jd\}}|jdd\}}}|jj||}|d krtjtji|d6|d6n||krtjtji|d6|d6n||d||d fS( sA Get information for an attribute Args: path: xpath of the attribute Returns: tuple of attribute information from the dtd: (type, value) or (None, None) e.g., for: '/auto_install/ai_instance@auto_reboot' returns: ('(true|false)', '"false"') Raises: MimInvalidError - Element in path is not a child of parent in path e.g., ai_instance is not child of auto_install Element does not have attribute RRZitelem1telem2RRWiiN(NN( R R_RRtfind_attr_infoR RtERR_ELEM_NOT_CHILDtERR_ATTR_NOT_IN_ELEM(R&Rt attr_nameRRRt attr_dict((smim.pyt_get_attribute_infos       cCs}|j|\}}|dkr%dS|jdr^|jdr^|jdjd}n|}|jd}||fS(s Get the dtd values of an attribute Args: path: xpath of the attribute Returns tuple: The type and value for the attribute or None, None Examples: For path='/auto_install/ai_instance@auto_reboot', returns: (['true', 'false'], 'false') For path='/auto_install/ai_instance/software@name', returns: ('CDATA', '#IMPLIED') Raises: nothing RRs()Rs'"N(NN(RR RtendswithRR_(R&RtatypeRtval_list((smim.pytget_attribute_valuess cCsF|j|\}}xE||gD]7}|tt|kr"tjtj|q"q"W||}||}|dj} x0t| D]"\} } | |kr| } PqqW||kr| d7} n| j| t j dx/t| D]!\} } | |kr| } qqW| | }| | | | to Caller is expected to ensure element is located at path Args: path - indexed path of element, except for base element being moved which is non-indexed e.g., if moving software: /auto_install[1]/ai_instance[1]/software from_position - current position of element(0 based) to_position - desired position of element (0 based) Returns: string - new indexed path of element Raises: MimError if to/from_position out of range iittmpelem(R{RRRR R>tERR_UNABLE_TO_MOVRt enumerateRsRRRRR](R&Rt from_positiont to_positionRt_attrtpost from_elemtto_elemRtindexRtto_indext from_indextmoving((smim.pyt move_elements*         cCs|j|\}}t|dkr@tjtj|n|d}|dk rZtS|jrgtS|js|j rtSt S(sg Determine if element specified by path has any attributes, subelements or text defined. Args: path: Xpath-like expression of element Returns: boolean True, if no content exists, False otherwise Raises: MimError - Path defined has more than one match Errors raised by __getlist() iiN( R{RRR R>tERR_MORE_THAN_ONE_MATCHR RR|titemst getchildrenR(R&RRRWR((smim.pytelement_has_no_contents    cCst|jtkS(s Checks to see if node is a comment. Args: node: node to check Returns: True: node is a comment; False: node is an element. (ttypeRrRA(tnode((smim.pyt _is_comment3s cCsXtj|r$tj|j}n0tj|j|j}|j|_|j|_|S(s Clone an element, for inclusion into the main tree. Clone its tag, text and attributes. Do not clone or attach its children. Handles comment and regular elements. Does not currently handle other types of elements, but it could be enhanced to do so if needed. Args: orig_element: Element to clone Returns: A copy of the orig_element, but without its children. Raises: Whatever is raised by etree.Comment or etree.Element. ( RRRtCommentR|RRrRttail(t orig_elementR((smim.pyt_clone_element@s   cCs:tj|}x$|D]}|jtj|qW|S(s Recursively clone a tree of elements. Args: orig_element: Root of the tree to clone. Returns: the root of the clone. Raises: Whatever is raised by etree.Comment or etree.Element. (RRR~t_clone_subtree(RRR((smim.pyRYs  cCs4x-t|D]\}}|j|kr |Sq WdS(s! Search children of "parent" element for the first with a matching tag. Args: parent: Parent element containing children to search. tag: tag name to search for. Returns: Child element with matching tag. Raises: N/A i(RRr(RRrtidxR((smim.pyRkscCs|jj|j|j\}}|dkrZtjtji|jd6|jd6nd}t|s|dk r|j|n|j |j |dS|dk r|j nt j ||j}|dkr]|rd}||j}xct|dt|D]<} t|| jtr||| jkrK| }PqKqqWt|}|j||j ||dk rx!|D]} |j|| qWndSt|s|j |||<|dk rx!|D]} |j|| qWndS||} | jj|jj|j| _|dk rx!|D]} |j|| q=Wqn@t|s|dk rx|D]} |j | q|Wn|j |j |dSx|D](} t j || }|dkrPqqW|js+|jj||j} | dk r+t j| g|}q+n|j sA|dkrPt|}n|j||j ||dk rx!|D]} |j|| q|WndS| S(s Determine how to add a node (overlay_element) to its potential future parent (main_parent), and add it. If overlay_element has a tag that matches an existing child of main_parent, additional handling is needed. - If two elements with the same tag are allowed (mults_ok) then just add the new node. - If two elements with the same tag are not allowed, then see if the overlay_element has children. - If the overlay_element has children, keep the like-tagged element already in the tree. It will be used to get to the next node in the path, to travel to the spot where something new can be added. Add value and any attributes of the new element to the original one. - If the overlay_element is a leaf node, then replace the node in the main tree with the overlay_element. Assumes self.schema_data is initialized. Args: main_parent: Parent of where new element would be added. overlay_element: New element to add. May have attributes. comments_to_install: comments to add to main tree just before the element being added. Returns: A reference to the affected element in the tree. This could be an added element, or an original element. Raises: Various exceptions from schema module's find_element_info() method MimInvalidError - Node cannot be placed as child of RRiiiN(RRRrR R RRRRRR~RtreverseRRRt isinstanceRARsRtupdatet iteritemsR|RR(R&t main_parenttoverlay_elementtcomments_to_installRRRRtinsert_after_tagRtcommentt ret_elementRR((smim.pyt_overlay_processs*                                   c Csfd}|jj|krt|sQ|j|}tjd||_qbxtt|D]i}||}t j |r|dkrg}n|j tj |j qd|j|||d}qdWn|j|||}|dk rbxj|D]_}t j |rB|dkr&g}n|j tj |j q|j|||d}qWndS(s( Recurse through the tree in pre-order fashion, calling _overlay_process on encountered nodes. Orchestrates overlay of full subtree starting with overlay_element. Args: main_parent: node in main tree where overlay_element will be attached. overlay_element: New node to overlay into main tree. comments_to_install: List of comments to install before the overlay_element under the main_parent. Raises: Exceptions raised by _overlay_process(). R`N(R t getroottreeRIRRRRRRRRRR~RR|RHR( R&RRRtamassed_commentsRtiRtnew_main_parent((smim.pyRH-s4                cCsZ|djd\}}}|rJ|s=tjtjn||d2s0    11