NglhUddlmZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddlm Z ddl mZddlmZmZmZmZmZmZmZmZmZmZmZddlZddlmZmZddlm Z erddl!m"Z"m#Z#ee$e$e$e$fZ%d e&d <ee$e$fZ'd e&d <ee'd fZ(d e&d <dZ)edZ*edZ+dcdZ,dddZ-Gddee*Z.ded$Z/dfd%Z0 dgdhd+Z1did-Z2djd1Z3dkd2Z4dld5Z5d6Z6dmd<Z7dnd@Z8dodBZ9dpdqdHZ: drdsdNZ; dtdudWZ dwdyd`Z?GdadbZ@dS)z) annotationsN)wraps) combinations) TYPE_CHECKINGAnyCallableGenericIterableIteratorListOptionalTupleTypeVarcast) ParamSpec TypeAlias) __version__)ElementTextrBoxPoint.Points)z%Y-%m-%dz%Y-%m-%dT%H:%M:%Sz%Y-%m-%d+%H:%M:%Sz%Y-%m-%dT%H:%M:%S%z_T_PfuncCallable[_P, List[Element]]args_P.argskwargs _P.kwargsreturndict[str, Any]c tj|}tditt|j||}|jD](}|j|vr|j|jur|j||j<)|S)zMMap both explicit and default arguments of decorated func call by param name.) inspect signaturedictzip parametersvaluesnamedefaultempty)rrrsig call_argsargs N/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/utils.pyget_call_args_applying_defaultsr20s  D ! !C $ Q QtC,E,E'F'F Q& Q QI~$$&&.. 89 $ $CI)E)E"%+Ich   file_pathstrboolcN|tjS)zTrue when file_path is in the Python-defined tempdir. The Python-defined temp directory is platform dependent (macOS != Linux != Windows) and can also be determined by an environment variable (TMPDIR, TEMP, or TMP). ) startswithtempfile gettempdir)r4s r1is_temp_file_pathr;>s!    3 5 5 6 66r3c,eZdZdZddZddd ZddZdS) lazypropertya$Decorator like @property, but evaluated only on first access. Like @property, this can only be used to decorate methods having only a `self` parameter, and is accessed like an attribute on an instance, i.e. trailing parentheses are not used. Unlike @property, the decorated method is only evaluated on first access; the resulting value is cached and that same value returned on second and later access without re-evaluation of the method. Like @property, this class produces a *data descriptor* object, which is stored in the __dict__ of the *class* under the name of the decorated method ('fget' nominally). The cached value is stored in the __dict__ of the *instance* under that same name. Because it is a data descriptor (as opposed to a *non-data descriptor*), its `__get__()` method is executed on each access of the decorated attribute; the __dict__ item of the same name is "shadowed" by the descriptor. While this may represent a performance improvement over a property, its greater benefit may be its other characteristics. One common use is to construct collaborator objects, removing that "real work" from the constructor, while still only executing once. It also de-couples client code from any sequencing considerations; if it's accessed from more than one location, it's assured it will be ready whenever needed. Loosely based on: https://stackoverflow.com/a/6849299/1902513. A lazyproperty is read-only. There is no counterpart to the optional "setter" (or deleter) behavior of an @property. This is critically important to maintaining its immutability and idempotence guarantees. Attempting to assign to a lazyproperty raises AttributeError unconditionally. The parameter names in the methods below correspond to this usage example:: class Obj(object) @lazyproperty def fget(self): return 'some result' obj = Obj() Not suitable for wrapping a function (as opposed to a method) because it is not callable. fgetCallable[..., _T]r!NonecV||_|j|_tj||dS)aY*fget* is the decorated method (a "getter" function). A lazyproperty is read-only, so there is only an *fget* function (a regular @property can also have an fset and fdel function). This name was chosen for consistency with Python's `property` class which uses this name for the corresponding parameter. N)_fget__name___name functoolsupdate_wrapper)selfr>s r1__init__zlazyproperty.__init__rs- ]  t,,,,,r3Nobjrtyperc||S|j|j}|$||}||j|j<t t |S)aCalled on each access of 'fget' attribute on class or instance. *self* is this instance of a lazyproperty descriptor "wrapping" the property method it decorates (`fget`, nominally). *obj* is the "host" object instance when the attribute is accessed from an object instance, e.g. `obj = Obj(); obj.fget`. *obj* is None when accessed on the class, e.g. `Obj.fget`. *type* is the class hosting the decorated getter method (`fget`) on both class and instance attribute access. )__dict__getrDrBrr)rGrIrJvalues r1__get__zlazyproperty.__get__sY ;K   ,, =JJsOOE',CL $Br3rNc td)aRaises unconditionally, to preserve read-only behavior. This decorator is intended to implement immutable (and idempotent) object attributes. For that reason, assignment to this property must be explicitly prevented. If this __set__ method was not present, this descriptor would become a *non-data descriptor*. That would be nice because the cached value would be accessed directly once set (__dict__ attrs have precedence over non-data descriptors on instance attribute lookup). The problem is, there would be nothing to stop assignment to the cached value, which would overwrite the result of `fget()` and break both the immutability and idempotence guarantees of this decorator. The performance with this __set__() method in place was roughly 0.4 usec per access when measured on a 2.8GHz development machine; so quite snappy and probably not a rich target for optimization efforts. zcan't set attribute)AttributeError)rGrIrNs r1__set__zlazyproperty.__set__s&2333r3)r>r?r!r@N)rIrrJrr!r)rIrrNrr!r@)rC __module__ __qualname____doc__rHrOrRr$r3r1r=r=Gs`((T - - - -:444444r3r=datalist[dict[str, Any]]filenamer@ct|d5}|d|DddddS#1swxYwYdS)Nzw+c3DK|]}tj|dzVdS) N)jsondumps).0datums r1 z save_as_jsonl..s1JJEtz%0047JJJJJJr3)open writelines)rWrY output_files r1 save_as_jsonlres h  KJJTJJJJJJKKKKKKKKKKKKKKKKKKs >AAclt|5}d|DcdddS#1swxYwYdS)Nc6g|]}tj|Sr$)r]loads)r_lines r1 z#read_from_jsonl..s"888T 4  888r3)rb)rY input_files r1read_from_jsonlrls h9:88Z888999999999999999999s )-- dependenciesstr | list[str]extras Optional[str].Callable[[Callable[_P, _T]], Callable[_P, _T]]cHttrgdfd }|S)NrCallable[_P, _T]r!cfdtdfd }tdfd }tjr|S|S) Ncg}D]&}t|s||'t|dkrHtdd|drddndd|d zdS) Nrz$Following dependencies are missing: z, . z5Please install them using `pip install "unstructured[z]"`.z'Please install them using `pip install  z`.)dependency_existsappendlen ImportErrorjoin) missing_depsdeprmros r1 run_checkz;requires_dependencies..decorator..run_checks&(L# - -(--- '',,,<  1$$!V499\;R;RVVV"b`RX````asxxP\G]G]aaa %$r3rrrr c(|i|SrSr$rrrrs r1wrapperz9requires_dependencies..decorator..wrappers# IKKK4((( (r3c8K|i|d{VSrSr$rs r1 wrapper_asyncz?requires_dependencies..decorator..wrapper_asyncs9 IKKKt.v........ .r3)rrrr )rasyncioiscoroutinefunction)rrrrrmros` @r1 decoratorz(requires_dependencies..decorators       t ) ) ) ) ) )  ) t / / / / / /  /  &t , , ! r3)rrsr!rs) isinstancer5)rmrors`` r1requires_dependenciesrsJ,$$&$~ < r3 dependencyc tj|n.#t$r!}|t|vrYd}~dSYd}~nd}~wwxYwdS)NFT) importlib import_moduler{repr)res r1rxrxsp ++++  a 55555 !  4s A=Ait Iterable[_T]Tuple[_T, Iterator[_T]]ct|} t|}n#t$rtdwxYw||fS)NzYExpected at least 1 element in iterable from which to retrieve first, got empty iterable.)iternext StopIteration ValueError)riteratorouts r1_first_and_remaining_iteratorrs]BxxH 8nn         =s!;c*t|\}}|S)zRReturns the first item from an iterable. Raises an error if the iterable is empty.)r)rr_s r1firstrs *2 . .FC Jr3 Iterable[Any]rczt|\}}td|Drtd|S)zqReturns the only element from a singleton iterable. Raises an error if the iterable is not a singleton. c3K|]}dVdS)TNr$r_rs r1razonly..s" " "A4 " " " " " "r3zRExpected only 1 element in passed argument, instead there are at least 2 elements.)ranyr)rrrs r1onlyrsP 2"55MC " " " " """  `    Jr3c tjdd}n#t$rd}YnwxYwdt jddd} tjddkrtjddkrd tvrhtj d tzd zt j zd z|zd zt j zdzt|zdzdStj d tzd zt j zd z|zd zt j zdzt|zdzdSdSdS#t$rYdSwxYw)Nz nvidia-smiTF.SCARF_NO_ANALYTICStrue DO_NOT_TRACKdevz:https://packages.unstructured.io/python-telemetry?version=z &platform=z&pythonz&arch=z&gpu=z &dev=truez &dev=false) subprocess check_output Exceptionr|platformpython_versionsplitosgetenvrrequestsrMsystemmachiner5) gpu_presentrs r1scarf_analyticsrs ---  XXh577==cBB2A2FGGN 9) * *f 4 4>9R9RV\9\9\ ## P!""#o''(  % %  &(()+&& '" "      P!""#o''(  % %  &(()+&& '# #     ! 5 49\9\:      s& ((-BE:A&E:: FFs list[str]nintlist[tuple[str, ...]]cg}tt||z dzD]V}g}t|D] }||||z!|t|W|S)zSGenerate n-grams from a list of strings where `n` (int) is the size of each n-gram.)rangerzrytuple)rr ngrams_listingramjs r1ngramsr:s*,K 3q66A:> " "))q # #A LL1q5 " " " "5<<(((( r3 first_string second_string"tuple[float, set[tuple[str, ...]]]c\|sdtfSt||}t||}|sdtfSt|t|z}t|t|z dz}||fS)zTCalculate the percentage of common_ngrams between string A and B with reference to Ard)setrrrz)rrrfirst_string_ngramssecond_string_ngrams common_ngrams percentages r1!calculate_shared_ngram_percentagerFs #%%x !3!3!5!5q99!-"5"5"7"7;; #%%x+,,s3G/H/HHMm$$s+>'?'??3FJ } $$r3'tuple[float, set[tuple[str, ...]], str]ct}t|t|kr%t|dz }n(t|dz }||}}d}|s"t|||\}}|dkrn|dz}|"t |d|t |dzfS)aGFrom two strings, calculate the shared ngram percentage. Returns a tuple containing... - The largest n-gram percentage shared between the two strings. - A set containing the shared n-grams found during the calculation. - A string representation of the size of the largest shared n-grams found. rrr)rrzrrroundr5)rr shared_ngramsrngram_percentages r1"calculate_largest_ngram_percentagerYs+.%%M <      3}':':'<'<#=#===  ""$$ % % )  ##%% & & *&3\m  *K   + + '- 66  FA  !1 % %}c!a%jj @@r3 parent_target child_targetaddfloatcdt|dkrdSgd}|rbt|dkrOt|}|dxx|zcc<|dxx|zcc<|dxx|z cc<|dxx|z cc<t|dkrJ|d|dkr8|d|dkr&|d|dkr|d|dkrdSt|dkoC|d|dcxko |dknco!|d|dcxko |dkncS) zTrue if the child_target bounding box is nested in the parent_target. Box format: [x_bottom_left, y_bottom_left, x_top_right, y_top_right]. The parameter 'add' is the pixel error tolerance for extra pixels outside the parent region FrrrrrrrT)rzlist)rrrparent_targetss r1 is_parent_boxrxs  =Qu!\\N !s=!!Q&&m,,qS qS qS qS  LQ !_q 1 1 1l1oXYIZ6Z6Z !_q 1 1 1l1oXYIZ6Z6Zt |   ! q\!_AAAAq0AAAAA F 1 a E E E EN14E E E E Er3totalbox1box2intersection_ratio_method!tuple[float, float, float, float]cT|d\}}|d\}}|d\}}|d\} } ||z ||z z} | |z | |z z} t||} t||}t|| }t|| }td|| z td||z z}t| | }t| | }| | z}|dkr|dkrdS||z dz}n0|dkr|dkrdS||z dz}n| | zdkrdS|| | z|z z dz}t|d|||fS)a7Calculate the percentage of overlapped region. Calculate the percentage with reference to the biggest element-region (intersection_ratio_method="parent"), the smallest element-region (intersection_ratio_method="partial"), or the disjunctive union region (intersection_ratio_method="total") rrparentrrpartial)maxminr)rrrx1y1x2y2x3y3x4y4 area_box1 area_box2x_intersection1y_intersection1x_intersection2y_intersection2intersection_areamax_areamin_area total_areaoverlap_percentages r1calculate_overlap_percentagers!WFB !WFB !WFB !WFBbR"W%IbR"W%I"bkkO"bkkO"bkkO"bkkOA@AAC /)EE9i((H9i((HY&J H,, q==:/(:cA "i / / q==:/(:cA  !a ' ':/9y3HK\3\]add #Q ' '8Z GGr3$@box_pair$list[Points] | tuple[Points, Points] label_pairlist[str] | tuple[str, str] text_pairix_pairsm_overlap_thresholdcd\}}}}|\} } |\} } |\} }|\}}t| | d\}}}}||kr| d|d| d|dg}d}n| s| d|d| d|dg}d| }n|s| d|d| d|dg}d| }n| |vs|| vr| d|d| d|dg}d}nt| |\}}}t|d }|s| d|d| d|dg}d }nl| d|d| d|dg}t| t|kr| n| }d |zd |d z}d|d|}|||||||fS)aClassifies the overlapping case for an element_pair input. There are 5 cases of overlapping: 'Small partial overlap' 'Partial overlap with empty content' 'Partial overlap with duplicate text (sharing 100% of the text)' 'Partial overlap without sharing text' 'Partial overlap sharing {calculate_largest_ngram_percentage(...)}% of the text' Returns: overlapping_elements: List[str] - List of element types with their `ix` value. Ex: ['Title(ix=0)'] overlapping_case: str - See list of cases above overlap_percentage: float largest_ngram_percentage: float max_area: float min_area: float total_area: float )NNNNrr(ix=)zSmall partial overlapz&partial overlap with empty content in z#partial overlap with duplicate textrz$partial overlap without sharing textzof the text from(z-gram)zpartial overlap sharing z% )rrrrzr)rrrrroverlapping_elementsoverlapping_caserlargest_ngram_percentagerrtype1type2text1text2 ix_element1 ix_element2rrrr largest_nref_types r1identify_overlapping_casers4\X*,>@X JD$LE5LE5&K;W  "+<<<88Z 000 ( (+ ( ( ( ( (+ ( ( ( 3( e,,k,,,,,k,,,$  POO  ! e,,k,,,,,k,,,$  POO   e^^u~~,,k,,,,,k,,,$  E  6Xu66 2 $a(--Eq'I'I $+ e00+00000+000($$J  00+00000+000($%( $6$6U[[]]9K9K$K$K55QV-8;Py;P;P;PP#d>V#d#dZb#d#d   r3 coordinatesc:|d\}}|d\}}||||fS)zAccepts a set of Points and returns the lower-left and upper-right coordinates. Expects four coordinates representing the corners of a rectangle, listed in this order: bottom-left, top-left, top-right, bottom-right. rrr$)rx_bottom_left_1y_bottom_left_1 x_top_right_1 y_top_right_1s r1_convert_coordinates_to_boxr *s/ (31~$O_#.q> M= O]M IIr3nested_error_tolerance_pxc J|\}}|\}}dd|D} dd|D} |dd}|dd}t|} t|} | \} }}}| \}}}}| |ko||k}||ko||k}d\}}}}}}d\}}}|r|rt||d \}} } } t||d \}}}}t | | | r#|d | d |d | d g}d|d|}d}|d | d }nSt | | | r#|d | d |d | d g}d|d|}d}|d | d }nt |||| | f|\}}}}}}}||||pd|pd|pd|pd|pd|pdf S)aIdentify if overlapping or nesting elements exist and, if so, the type of overlapping case. Returns: overlapping_elements: List[str] - List of element types & their `ix` value. Ex: ['Title(ix=0)'] overlapping_case: str - See list of cases above overlap_percentage: float overlap_percentage_total: float largest_ngram_percentage: float max_area: float min_area: float total_area: float c:g|]}||Sr$ isnumericr_chs r1rjz8identify_overlapping_or_nesting_case..K%@@@"@2@@@r3c:g|]}||Sr$r&r(s r1rjz8identify_overlapping_or_nesting_case..Lr*r3rN)NNNNNN)NNNrr r)rr r znested z in r)rr)r|stripr rrr)!rrrr"rrrrrrr box1_corners box2_cornersrrrrx_bottom_left_2y_bottom_left_2 x_top_right_2 y_top_right_2horizontal_overlapvertical_overlaprparent_elementrroverlap_percentage_totalrrrrrs! r1$identify_overlapping_or_nesting_caser76s&JD$LE5''@@@@@AAK''@@@@@AAK !""IOO  E !""IOO  E.t44L.t44LEQBO_m]EQBO_m](=8\]_=\&6Z=?;Z   &6"Hh -.-,H  &-- - - ) !Q >Z  &.> > > :Hh |9R S S S! ,,k,,,,,k,,,$  <;;E;; !$  %99;999NN <;T U U U ,,k,,,,,k,,,$  <;;E;; !$  %99;999NN*k*%9  $ "( a %A %A A Aa  r3elements list['Text']!tuple[bool, list[dict[str, Any]]]c|djjpd}dt|D}dt|D}dt|D}t|D]\}}|jjpd} | dz } |jjrRt t |jjd} || | || |d|j || |j d } g} tt|||d D]\} \}}}tt|d }tt|d }tt|d }t|||D]\}}}t|||||\ }}}}}}}}}|r\| ||||d ||d t|d d t|d d t|d d ddd} | | fS)zLCatch overlapping and nested bounding boxes cases across a list of elements.rcg|]}gSr$r$rs r1rjz7catch_overlapping_and_nested_bboxes..s*H*H*H!2*H*H*Hr3cg|]}gSr$r$rs r1rjz7catch_overlapping_and_nested_bboxes..s#A#A#A1B#A#A#Ar3cg|]}gSr$r$rs r1rjz7catch_overlapping_and_nested_bboxes..s$B$B$BAR$B$B$Br3rpointsrvF)startr%upxˆ2)rr6rrr)rr5rrmetadataT)rC page_numberr enumeraterrrto_dictrycategorytextr(rrr7r) r8r"r num_pagespages_of_bboxes text_labels text_contentixelementrD n_page_to_ixboxdocument_with_overlapping_flagoverlapping_cases page_bboxes page_labels page_textpage_bboxes_combinationspage_labels_combinationstext_content_combinationsrrrrr5rrr6rrrrs r1#catch_overlapping_and_nested_bboxesrYs %16QI*H*HuY7G7G*H*H*HO#A#Ai0@0@#A#A#AK$B$By1A1A$B$B$BL **88 G&27a "Q   ' 6vw/;CCEEhOPPC L ) 0 0 5 5 5L!((B)D)D'2B)D)DEEE\"))',7777%*".0>G O[,77???/6/6: :k; $( [!(D(D#E#E #' [!(D(D#E#E $(i)C)C$D$D!/2 $ $ %0 0 ' 6' 6 +Hj)5)$  $ "((  6!((0D*8,<1C.F.F.F8P;S8V8V8V+01+=+=(D(D(D+01+=+=(D(D(D-2:q-A-A*H*H*H %%  26.O' 6R *+< < AA AA AAArWr!r@c|j5t|jd5}||dddn #1swxYwYddddS#1swxYwYdS)Nw)r_rbr4write)rGrWrbs r1 write_filezFileHandler.write_files Y ! !dnc** !d 4    ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !s3AA AA AA AAAc|j5tj|jrtj|jddddS#1swxYwYdSrS)r_rpathexistsr4remove)rGs r1 cleanup_filezFileHandler.cleanup_files Y * *w~~dn-- * $.))) * * * * * * * * * * * * * * * * * *s>AAAN)r4r5)rWr5r!r@)rCrTrUrHrcrgrlr$r3r1r[r[sZ%%%% !!!! *****r3r[)rrrrrr r!r")r4r5r!r6)rWrXrYr5r!r@)rYr5r!rXrS)rmrnrorpr!rq)rr5)rrr!r)rrr!r)rrr!r)rrrrr!r)rr5rr5rrr!r)rr5rr5r!r)r)rrrrrrr!r6)r)rrrrrr5r!r)r) rrrrrrrrrr)rr)r!r) rrrrrrr"rrr)r8r9r"rrrr!r:)A __future__rrrErr%r]rrrr9r]r itertoolsrtypingrrrr r r r r rrrrtyping_extensionsrrunstructured.__version__runstructured.documents.elementsrrrr__annotations__rr DATE_FORMATSrrr2r;r=rerlrrxrrrrrrrrrrr r7rYr[r$r3r1rus2""""""" """"""                          22222222000000>========ueUE122222&&&&&%*%%%%%\  WT]]Yt__    7777j4j4j4j4j472;j4j4j4ZKKKK 9999!%%%%%P        ( ( ( V    %%%%&AAAA>>&-.H.H.H.H.Hl#' bbbbbJJJJJ &'"& iiiii\&'"&I=I=I=I=I=X**********r3