NgUdZddlmZddlmZmZddlmZddlm Z m Z m Z m Z m Z mZmZddlmZddlmZddlmZdd lmZdd lmZmZmZmZmZmZmZm Z m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)e e*e fZ+d e,d< ddZ-ddZ.Gdde Z/ee/Z0d e,d< GddZ1GddZ2Gdde2Z3Gdd ej4Z5Gd!d"e5Z6Gd#d$e5Z7Gd%d&e5Z8Gd'd(e5Z9Gd)d*e6Z:Gd+d,e5Z;Gd-d.e5Z<Gd/d0ej4Z=Gd1d2e=Z>Gd3d4e=Z?Gd5d6e=Z@Gd7d8e=ZAGd9d:e=ZBGd;dZDejEd?@ZFejGeCAZHejIeHZJeFKeJeJLdBMidCe5dDe5dEe5dFe5dGe5dHe5dIe5dJe5dKe5dLe5dMe5dNe5dOe7dPe7dQe7dRe7dSe7idTe7dUe6dVe:dWe8dXe8dYe9dZe;d[e>d\e?d]e@d^e@d_e?d`e=dae=dbe=dce=ddeAidee=dfe=dge=dhe=die=dje=dke=dle=dme=dne=doe=dpe=dqe=dre=dse=dte=due=idve=dwe=dxe=dyeBdzeBd{e<d|e<d}e<d~e<de<de<de<de<de<de<de<dBS)aProvides the HTML parser used by `partition_html()`. The names "flow" and "phrasing" derive from the language of the HTML Standard. PRINCIPLES - _Elements are paragraphs._ Each paragraph in the HTML document should become a distinct element. In particular, a paragraph should not be split into two elements and an element should not contain more than one paragraph. - _An empty paragraph is not an Element._ A paragraph which contains no text or contains only whitespace does not give rise to an Element (is skipped). - _The browser rendering is the document._ The HTML "source-code" is not the document. The document is the way that HTML is rendered by a browser (Chrome for a first authority). This foundational principle gives rise to a few that are more specific. - _Whitespace is normalized._ Whitespace used for formatting the HTML source is _normalized_ to a single space between text segments. More specifically: - Any leading or trailing space on a paragraph is removed. - All other runs of whitespace in the paragraph are reduced to a single space (" "). - Whitespace is never added where none existed in the HTML source. - Whitespace within a `
` element is the exception and is not normalized. Its
    whitespace is preserved excepting a leading and/or trailing newline ("
").

- _Block-items are paragraphs._ Visible content in HTML can be divided into _block-items_ and
  _phrasing content_ (aka. _inline content_).
  - As an example, a `
` element is a block item and a `` element is phrasing.
  - A block item starts a new paragraph and so represents an Element boundary.
  - A phrasing item affects the appearance of a run of text within a paragraph, like making it
    bold or making it into a link.
  - Some elements can take either role, depending upon their ancestors and descendants.
  - The final authority for whether a particular element is displayed as a block or as inline
    "formatting" is the CSS. We do not attempt to interpret the CSS and assume the default role
    for each element.

Other background

- The parser's design is _recursive_, consistent with the recursive (tree) structure of HTML. The
  nodes of the tree are _HTML elements_. Unfortunately this naming sometimes conflicts with
  Unstructured _document-elements_. In the parser code the term "document-element" is used when
  there may be ambiguity.

- The parser is primarily composed of `lxml` Custom Element Classes. The gist is you write a class
  like `Anchor` and then tell the `lxml` parser that all `` elements should be instantiated
  using the `Anchor` class. We also provide a default class for any elements that we haven't
  called out explicitly.

- _Anatomy of an HTML element._ Some basic terms are important to know to understand the domain
  language of the parser code. Consider this example:
  ```html
  

    
Text bold child tail of child

    tail of p
  

  ```
  - An element can have _text_.
    - All visible content within an HTML document is the text (or tail) of some element.
    - The text of the `
` element (`p.text`) is "Text ".
    - Note the formatting whitespace is included.
  - An element can have _child elements_.
    - The `
` element (`p`) is a child of `div`.
    - `b` is a child of `p`.
  - An element can have a _tail_.
    - Whatever text follows an element, before the next element starts, is the tail of
      that element.
    - `b.tail` is `" tail of child"`. Note the included whitespace.
    - `p.tail` is `"
    tail of p
"`.
    - Tail text is _accessed_ via the element that precedes it but that element does not
      _influence_ its tail text. For example, "tail of child" does not appear in a bold
      typeface even though it is the tail of `b`.
)annotations)defaultdictdeque)MappingProxyType)AnyIterableIteratorMapping
NamedTupleSequencecast)etree)	TypeAlias)
clean_bullets)htmlify_matrix_of_cell_texts)	AddressElementElementMetadataEmailAddressListItem
NarrativeTextTableTextTitle)is_bulleted_textis_email_addressis_possible_narrative_textis_possible_titleis_us_city_state_zip)lazypropertyr
AnnotationrIterable[Annotation]returnctttttftt}|D]}|D]o\}}t|tr:||ttt|T|||ptt|S)zpCombine individual text-segment annotations into an element-level annotation.

    Sequence is significant.
    )r
rstrlistitems
isinstanceextendrappendrdict)rcombined_annotationsakvs     ^/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/partition/html/parser.py_consolidate_annotationsr1xs
 CcN ;[=N=NOO
22GGII	2	2DAq!T""
2$Q'..tDIq/A/ABBBB$Q'..q1111		2D!566777textr%ctd|S)a`text` with normalized whitespace.

    - leading and trailing whitespace are removed
    - all whitespace segments within text (spacing between words) are reduced to a single space
      each.

    Produces the empty string when `text` contains only whitespace.
     )joinstripsplitr3s r0_normalize_textr:s*88DJJLL&&(()))r2c(eZdZUdZded<ded<dS)TextSegmentaAn annotated string from a Phrasing element.

    Annotations are for emphasis and for links. The text includes any leading, trailing, and
    inter-word whitespace, just as it occurred in the HTML. The text-segments for a paragraph are
    consolidated once the paragraph is fully parsed and whitespace it normalized at that time. It
    cannot be normalized prior to that without distoring or losing inter-word spacing.

    However, text within annotations, like the text of a link, is normalized since its full extents
    are known.
    r%r3r!
annotationN)__name__
__module____qualname____doc____annotations__r2r0r<r<s3		IIIr2r<Phrasec(eZdZdZdZddZdd	Zd
S)
_PhraseAccumulatoraAccumulates sequential `TextSegment`s making them available as iterable on flush().

    - The accumulator starts empty.
    - `.flush()` is a Phrase iterator and generates zero or one Phrase.
    - `.flush()` generates zero items when no text-segments have been accumulated
    - `flush()` resets the accumulator to its initial empty state.

    So far, phrases are used only by the Anchor class.
    cg|_dSN)_text_segmentsselfs r0__init__z_PhraseAccumulator.__init__s13r2text_segmentr<r#Nonec:|j|dS)z&Add `text_segment` to this collection.NrIr*rKrMs  r0addz_PhraseAccumulator.add""<00000r2Iterator[Phrase]c#K|jdd}|j|sdSt|VdS)zMGenerate each of the stored `TextSegment` objects and clears the accumulator.N)rIcleartuple)rK
text_segmentss  r0flushz_PhraseAccumulator.flushsU+AAA.
!!###	FM"""""""r2NrMr<r#rN)r#rT)r>r?r@rArLrRrYrCr2r0rFrFsU4441111	#	#	#	#	#	#r2rFcJeZdZdZddZdd	Zdd
ZddZeddZ	dS)_ElementAccumulatora"Accumulates sequential `TextSegment`s and forms them into an element on flush().

    The text segments come from element text or tails and any contiguous phrasing elements that
    follow that text or tail.

    - The accumulator starts empty.
    - `.flush()` is an element iterator and generates zero or one Element.
    - `.flush()` generates zero elements when no text-segments have been accumulated or the ones
      that have been accumulated contain only whitespace.
    - `flush()` resets the accumulator to its initial empty state.
    elementetree.ElementBasec"||_g|_dSrH)_elementrI)rKr]s  r0rLz_ElementAccumulator.__init__s
13r2rMr<r#rNc:|j|dS)z6Add `text_segment` to this Element-under-construction.NrPrQs  r0rRz_ElementAccumulator.addrSr2
ElementClstype[Element] | NoneIterator[Element]c#hK|j}|jdd}|j|r|sdS|/t|}|dS|turt|}|sdS||}||tditd|Dd|iVdS)zIGenerate zero-or-one document-`Element` object and clear the accumulator.Nc3$K|]}|jVdSrHr=.0tss  r0	z,_ElementAccumulator.flush..
s$*Q*QR2=*Q*Q*Q*Q*Q*Qr2category_depthmetadatarC)	_normalized_textrIrVderive_element_type_from_textrr_category_depthrr1)rKrbnormalized_textrXrls     r0rYz_ElementAccumulator.flushs/+AAA.
!!###	O	F6GGJ!X%%"/"@"@&F--j99j$**Q*Q=*Q*Q*QQQ-


	
	
	
	
	
r2
type[Element]
int | Nonec|tur@|jjdvr0td|jDndS|t
ur2|jjdvr"t
|jjddz
ndSdS)z8Not clear on concept. Something to do with hierarchy ...)liddc$g|]
}|jdv|S))dlolul)tag)ries  r0
z7_ElementAccumulator._category_depth..s%]]]1J\A\A\QA\A\A\r2r)h1h2h3h4h5h6N)rr`r|len
iterancestorsrint)rKrbs  r0rqz#_ElementAccumulator._category_depths!!=$44]]
 ; ; = =]]]^^^
=$(LLLDM%a())A--
tr2r%cddd|jDS)aVConsolidate text-segment text values into a single whitespace-normalized string.

        This normalization is suitable for text inside a block element including any segments from
        phrasing elements immediately following that text. The spec is:

        - All text segments are concatenated (without adding or removing whitespace)
        - Leading and trailing whitespace are removed.
        - Each run of whitespace in the string is reduced to a single space.

        For example:
          "  
   foo  bar
baz bada 	 bing
  "
        becomes:
          "foo bar baz bada bing"
        r5c3$K|]}|jVdSrHr9rhs  r0rkz7_ElementAccumulator._normalized_text..4s$FFBFFFFFFr2)r6rIr8rJs r0roz$_ElementAccumulator._normalized_text$s@ xxFF$2EFFFFFLLNNOOOr2N)r]r^rZ)rbrcr#rd)rbrsr#rtr#r%)
r>r?r@rArLrRrYrqpropertyrorCr2r0r\r\s

44441111!
!
!
!
F$PPPXPPPr2r\c*eZdZdZeddZdS)_PreElementAccumulatorzXAccumulator specific to `
` element, preserves (most) whitespace in normalized text.r#r%cdd|jD}|drdnd}|drdnt	|}|||S)zConsolidate `texts` into a single whitespace-normalized string.

        This normalization is specific to the `
` element. Only a leading and or trailing
        newline is removed. All other whitespace is preserved.
        rc3$K|]}|jVdSrHr9rhs  r0rkz:_PreElementAccumulator._normalized_text..As$==2rw======r2
rr)r6rI
startswithendswithr)rKr3startends    r0roz'_PreElementAccumulator._normalized_text:soww==)<=====__T**1MM$''6bbSYYE#Ir2Nr)r>r?r@rArrorCr2r0rr7s8bb
Xr2rcbeZdZdZdZeddZddZedd	Z		dddZ
ddZdS)Flowz~Base and default class for elements that act like a div.

    These can contain other flow elements or phrasing elements.
    Nr#boolcdS)NFrCrJs r0is_phrasingzFlow.is_phrasingZsur2rdc#lKt|}||jpd||jEd{V|rw|djrJtt|}|Ed{V||j	pd|Ed{V|udSdS)5Generate paragraph string for each block item within.rNr)
r_element_from_text_or_tailr3_ElementClsrr
rpopleft
iter_elementstail)rKq
block_items   r0rzFlow.iter_elements^s%*$KK2249?AtGWXXXXXXXXX	Qt''''dAIIKK00J!//11111111166z7L"aPPPPPPPPP		Q	Q	Q	Q	Qr2r\c t|Sz9Text-segment accumulator suitable for this block-element.)r\rJs r0_element_accumzFlow._element_accumks#4(((r2r3r%rdeque[Flow | Phrasing]rbrcc#K|j}|||D]L}t|tr||-||Ed{V|VM||Ed{VdS)zGenerate zero-or-one paragraph formed from text and leading phrasing elements.

        Note this mutates `q` by popping phrasing elements off as they are processed.
        N)r_iter_text_segmentsr(r<rRrY)rKr3rrb
element_accumnodes      r0rzFlow._element_from_text_or_tailps+
,,T155		D$,,
!!$'''')..z:::::::::



 &&z22222222222r2Iterator[TextSegment | Element]c#Kt|iV|r_|djrTtt|}|Ed{V|r|djPdSdSdSdS)aGenerate zero-or-more `TextSegment`s or `Element`s from text and leading phrasing.

        Note that while this method is named "._iter_text_segments()", it can also generate
        `Element` objects when a block item is nested within a phrasing element. This is not
        technically valid HTML, but folks write some wacky HTML and the browser is pretty forgiving
        so we try to do the right thing (what the browser does) when that happens, generally
        interpret each nested block as its own paragraph and generate a separate `Element` object
        for each.

        This method is used to process the text or tail of a block element, including any phrasing
        elements immediately following the text or tail.

        For example, this 
:

            

               For a moment, nothing happened.
               
Then, after a second or so, nothing continued to happen.

               The dolphins had always believed that they were far more intelligent.
            


        Should generate three distinct elements:
        - One for the div's text "For a " and the  phrasing element after it,
        - one for the 
 element, and
        - one for the tail of the 
 and the phrasing  element that follows it.

        This method is invoked to process the first line beginning "For a" and the third line
        beginning "The dolphins", in two separate calls.

        Note this method mutates `q` by popping phrasing elements off as they are processed.
        rN)r<rr
Phrasingriter_text_segments)rKr3rr}s    r0rzFlow._iter_text_segmentssB$#####	.AaD$	.Xqyy{{++A++---------	.AaD$	.	.	.	.	.	.	.	.	.r2r#rr#rdr#r\rH)r3r%rrrbrcr#rd)r3r%rrr#r)r>r?r@rArrrrr rrrrCr2r0rrQsK
XQQQQ)))\)
X\33333&$.$.$.$.$.$.r2rceZdZdZdS)	BlockItemzCustom element-class for `
` element, `
`, and others like it.

    These can appear in a flow container like a div but can only contain phrasing content.
    Nr>r?r@rArCr2r0rrsr2rceZdZdZeZdS)HeadingzcAn `
..
` element.

    These are distinguished because they generate a `Title` element.
    N)r>r?r@rArrrCr2r0rrs
KKKr2rceZdZdZdS)	ListBlockaxEither a `
    ` or `
      ` element, maybe a `
      ` element at some point.

    The primary reason for distinguishing these is because they increment the hierarchy depth for
    lists that are nested inside them.

    Can only contain `
    1. ` elements (ignoring `