NgAddlmZddlZddlmZddlmZddlmZm Z ddl m Z m Z ddl mZmZddlmZmZmZmZd Z d2d3dZd4dZd5dZd6dZd6d Zd7d#Zd8d%Zd9d'Zd9d(Z d:d;d.Zd) annotationsN) OrderedDict)chain)SequenceType) BeautifulSoupTag)elementsontology)CSS_CLASS_TO_ELEMENT_TYPE_MAP)HTML_TAG_AND_CSS_NAME_TO_ELEMENT_TYPE_MAP$HTML_TAG_TO_DEFAULT_ELEMENT_TYPE_MAP+ONTOLOGY_CLASS_TO_UNSTRUCTURED_ELEMENT_TYPE2Tontology_elementontology.OntologyElement parent_idstr page_numberintdepthfilename str | Noneadd_img_alt_textboolreturnlist[elements.Element]c g}|jtjjkr|tkr|!t |tjr|j}t |tjsL|tj d|j dtj || d|||gz }g}|jD]@}t||j |t |tjrdn|d z|| }||z }At!|} || z }njt"|j} | } || } | | |j dtj || |||} | g}|S) a5 Converts an OntologyElement object to a list of unstructured Element objects. To preserve the structure of the ontology, the function is recursive and the tree structure is represented in flatten list by the parent_id attribute in the metadata of each Element object. To preserve all the attributes of the ontology element, the HTML code is injected to unstructured Element in ElementMetadata.text_as_html attribute. For Layout elements, the function creates an empty Text Element (with the HTML code injected the same way). TODO (Pluto): Better way would be to have special Element type in Unstructured Args: ontology_element (OntologyElement): The ontology element to be converted. parent_id (str, optional): The ID of the parent element. Defaults to None. page_number (int, optional): The page number of the element. Defaults to None. depth (int, optional): The depth of the element in the hierarchy. Defaults to 0. filename (str, optional): The name of the file the element comes from. Defaults to None. add_img_alt_text (bool): Whether to include the alternative text of images in the output. Defaults to True. Returns: list[Element]: A list of unstructured Element objects. Nvlm_partitionerF) add_children)r text_as_htmlrcategory_depthr)text element_iddetection_originmetadatar)rrrrr)r) elementTyper ElementTypeEnumlayoutRECURSION_LIMIT isinstancePagerDocumentr TextidElementMetadatato_htmlchildren!ontology_to_unstructured_elementscombine_inline_elementsr __class__to_text)rrrrrrelements_to_returnr4childcombined_children element_classhtml_code_of_ontology_element element_textunstructured_elements g/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/partition/html/transformations.pyr5r5sB#x'?'FFF5TcKcKc  :.> #N#N *6K*H,=>>   /2%6%5"+%5%=%=5%=%Q%Q$/',!)     #  %.  E5*-'%&68IJJYaaPUXYPY!!1 E  HH3H==//CDTD^_ (8(@(@(B(B%'//AQ/RR ,}'*.-#:'$!  33 r cg}d}|D]i}||}t||r;|xjd|jzz c_|jxjd|jjzz c_R|||}j||||S)a Combines consecutive inline elements into a single element. Inline elements can be also combined with text elements. Combined elements contains multiple HTML tags together eg. { 'text': "Text from element 1 Text from element 2", 'metadata': { 'text_as_html': "

Text from element 1

Text from element 2" } } Args: elements (list[Element]): A list of elements to be combined. Returns: list[Element]: A list of combined elements. N )#can_unstructured_elements_be_mergedr$r'r"append)r result_elementscurrent_element next_elements r@r6r6os&OO + +  "*O  . M M +  C,*;$; ;  $ 1 1S<;P;]5] ] 1 1 1  " "? 3 3 3*OO"/// rArGelements.ElementrHc|jj|jjkrdSt|jjdd}t|jjdd}dt ||D}|D]-}|jrdSt|st|sdS.dS)z Elements can be merged when: - They are on the same level in the HTML tree - Neither of them has children - All elements are inline elements or text element F html.parser recursivec,g|]}t|S)parse_html_to_ontology_element).0html_tags r@ z7can_unstructured_elements_be_merged..s0  'x00rAT) r'r#rr"find_allrr4is_inline_elementis_text_element)rGrHcurrent_html_tagsnext_html_tagsontology_elementsrs r@rDrDs.,2G2VVVu% -}hh#<#8#E}UU^^_N/@@ .  $ 55!"233 GW7X7X 55 4rAcTtjtjtjtjtjtjtjtjg}tj j g}tfd|DrdStfd|DrdSdS)z>Categories or classes that we want to combine with inline textc38K|]}t|VdSNr-rQclass_rs r@ z"is_text_element..s. K KF:& / / K K K K K KrATc3.K|]}j|kVdSr\r)rQcategoryrs r@r`z"is_text_element..s, T T  '8 3 T T T T T TrAF) r NarrativeTextQuote ParagraphFootnoteFootnoteReferenceCitation BibliographyGlossaryr*r'any)r text_classestext_categoriess` r@rVrVs " L /89O K K K Kl K K KKKt T T T TO T T TTTt 5rActjg}tjjtjjg}t fd|DrdSt fd|DrdSdS)z@Categories or classes that we want to combine with text elementsc38K|]}t|VdSr\r]r^s r@r`z$is_inline_element..s. M MF:& / / M M M M M MrATc3.K|]}j|kVdSr\rbrcs r@r`z$is_inline_element..s, V V  '8 3 V V V V V VrAF)r Hyperlinkr*specialized_text annotationrm)rinline_classesinline_categoriess` r@rUrUs()N 1 +  M M M Mn M M MMMt V V V VDU V V VVVt 5rAunstructured_elementsSequence[elements.Element]c:t}|djj}|0tj}||dj_tjd|i||<|D]}t|jjd d}|D]_}t|}|||j <|jjr8|jj|vr*||jjj |`|d\}}|S) a9 Converts a sequence of unstructured Element objects to an OntologyElement object. The function caches the elements in a dictionary and each element is assigned to its parent. At the end the root element is popped from the dictionary and returned. Such approach comes with limitations: - The parent element has to be in the list before the child element Args: unstructured_elements (Sequence[Element]): The sequence of unstructured Element objects. Returns: OntologyElement: The converted OntologyElement object. rNr1)additional_attributesrKFrL)last)rr'rr OntologyElementgenerate_unique_idr/rr"rTrPr1r4rEpopitem) rxid_to_element_mappingdocument_element_idelement html_as_tags html_as_tagrroot_id root_elements r@!unstructured_elements_to_ontologyrsS$(MM/2;E"&6IIKK6Ia )3191B#%89222-.)dd$W%5%BMRR[[\  ( d dK=kJJ :J !"2"5 6) dg.>.HLa.a.a%g&6&@AJQQRbccc d299u9EEG\ rA html_codect|}t|}t|d}|dd}|s|dd}|st dt |}|S)ag Parses the given HTML code and converts it into an Element object. Args: html_code (str): The HTML code to be parsed. Parsing HTML will start from
. Returns: OntologyElement: The parsed Element object. Raises: ValueError: If no element is found in the HTML. rKbodyr/)r_divr.zKNo or
element found in the HTML.)#remove_empty_divs_from_html_content#remove_empty_tags_from_html_contentrfind ValueErrorrP)rsoupdocumentdocument_elements r@parse_html_to_ontologyrs4I>>I3I>>I M 2 2Dyy y33H 399U6922   Y   6h?? rA html_contentct|d}|d}t|D]}|js|t |S)NrKr)rrTreversedattrsunwrapr)rrdivsrs r@rr3s[ } 5 5D ==  D~~y  JJLLL t99rAcht|d}dfd}||t|S)NrKc|jdvrdS|rdS|jrdS|dsdSdS)N) pspanrh1h2h3h4h5h6FT)strip)namerrget_text)tags r@is_emptyz5remove_empty_tags_from_html_content..is_empty?sY 8S S S5 88:: 5 9 5||$|'' 4urAct|D]!}|r|"dSr\)rT decompose)rrrs r@remove_empty_tagsz>remove_empty_tags_from_html_content..remove_empty_tagsOsB==??  Cx}}    rA)rr)rrrrs @r@rr<sY } 5 5D      d t99rAr(rr recursion_depthontology.OntologyElement | Nonect|\}}t|}|jdkrtjddd|S|tjkrt d|jDp!|jtj j k}|o tk}|rd}fd|j D}n8d d|jD}g}||||| } | S) a Converts a BeautifulSoup Tag object into an OntologyElement object. This function is recursive. First tries to recognize a class from Unstructured Ontology, then if class is matched tries to go deeper inside HTML tree. The recursive parsing is ended if the class is not recognized or there are no HTML Tags inside HTML - just text. Then it is parsed to Paragraph or UncategorizedText object. Args: soup (Tag): The BeautifulSoup Tag object to be converted. recursion_depth (int): Flag to control limit of recursion depth. Returns: OntologyElement: The converted OntologyElement object. brrN)r$css_class_name html_tag_namer{c3@K|]}t|tVdSr\)r-r rQcontents r@r`z1parse_html_to_ontology_element..vs,FFW 7C((FFFFFFrAc g|]}t|#t|trt |dzn3t jt|S)r()r)r$)rrr-r rPr rg)rQr:rs r@rSz2parse_html_to_ontology_element..}s   5zz!! eS))A.uoXYFYZZZZ'SZZ-=-=-?-?@@@    rA cPg|]#}t|$SrO)rrrs r@rSz2parse_html_to_ontology_element..s*LLL7#g,,,,..LLLrA)r$r4rr{)'extract_tag_and_ontology_class_from_tagget_escaped_attributesrr rgUncategorizedTextrmcontentsr)r*r+r,r4joinr) rrontology_html_tagontology_class escaped_attrs has_childrenshould_unwrap_htmlr$r4output_elements ` r@rPrPYsY )PPT(U(U%~*400M yD!"/      85 5 G FF FFF F F K >   '8+C+J J &L/_*L         yyLLdmLLLMMSSUU#^ '+ N rA*tuple[str, Type[ontology.OntologyElement]]cd\}}|jdr3|jtj|j|jddf}}|sn|jdrT|jddtvr:t j|jdd}|jd}|s'|jt vr|jt |j}}|sd}tj}||fS)a Extracts the HTML tag and corresponding ontology class from a BeautifulSoup Tag object. The CSS class is prioritized over the HTML tag. If not recognized soup.name and UnstructuredText is returned. Args: soup (Tag): The BeautifulSoup Tag object to extract information from. Returns: tuple: A tuple containing the HTML tag (str) and the ontology class (Type[OntologyElement]). )NNclassrr) rgetrr r allowed_tagsrr r)rrRr<s r@rrs)Hm z~~g "&)-V-Z Y 7+A. /. . - 3 JNN7 # #3 Jw  "&C C C59$*W:Ma:PQQ  =??/2 ]TY*NNN"&)-QRVR[-\- 3 2 ] ""rAci}|jD]X\}}tj|}d}|r6t |t r d|D}ntj|}|||<Y|S)z Escapes the attributes of a BeautifulSoup Tag object. Args: soup (Tag): The BeautifulSoup Tag object whose attributes need to be escaped. Returns: dict: A dictionary with escaped attribute names and values. Nc6g|]}tj|SrO)htmlescape)rQvs r@rSz*get_escaped_attributes..s ? ? ?AQ ? ? ?rA)ritemsrrr-list)rrkeyvalue escaped_key escaped_values r@rrsMj&&((33 Uk#&&   3%&& 3 ? ? ? ? ? $ E 2 2 %2 k"" rA)NNrNT)rrrrrrrrrrrrrr)r rrr)rGrIrHrIrr)rrrr)rxryrr)rrrr)rrrr)r()rr rrrr)rr rr)rr )" __future__rr collectionsr itertoolsrtypingrrbs4rr unstructured.documentsr r unstructured.documents.mappingsr r rrr,r5r6rDrVrUrrrrrPrrrOrAr@rs"""""" ######!!!!!!!!""""""""55555555 ! WWWWWt%%%%PD0$0000f:<'(88888v-#-#-#-#`rA