Ng,dZddlmZddlZddlZddlZddlZddlZddl m Z m Z m Z m Z ddlmZddlmZmZddlmZddlmZdd lmZed ZddZgdgddZefd dZ d!d"dZd#dZd#dZdS)$z>Helpers used across multiple partitioners to compute metadata.) annotationsN)AnyCallableIteratorSequence) ParamSpec)ElementElementMetadata)FileType)apply_lang_metadata)get_call_args_applying_defaults_Pfilenamestrreturn str | Nonectj|sdStjtj|}|dS)a Modification time of file at path `filename`, if it exists. Returns `None` when `filename` is not a path to a file on the local filesystem. Otherwise returns date and time in ISO 8601 string format (YYYY-MM-DDTHH:MM:SS) like "2024-03-05T17:02:53". Nz%Y-%m-%dT%H:%M:%S%z)ospathisfiledtdatetime fromtimestampgetmtimestrftime)r modify_dates b/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/partition/common/metadata.pyget_last_modified_datersY 7>>( # #t+++BG,<,>r file_typeFileType | NoneDCallable[[Callable[_P, list[Element]]], Callable[_P, list[Element]]]cdfd }|S)aPost-process element-metadata for this document. This decorator adds a post-processing step to a partitioner, primarily to apply metadata that is common to all partitioners. It assumes the following responsibilities: - Hash element-ids. Computes and applies SHA1 hash element.id when `unique_element_ids` argument is False. - Element Hierarchy. Computes and applies `parent_id` metadata based on `category_depth` etc. added by partitioner. - Language metadata. Computes and applies `language` metadata based on a language detection model. - Apply `filetype` (MIME-type) metadata. There are three cases; first one in this order that applies is used: - `metadata_file_type` argument is present in call, use that. - `file_type` decorator argument is populated, use that. - `file_type` decorator argument is omitted or None, don't apply `.metadata.filetype` (assume the partitioner will do that for itself, like `partition_image()`. - Replace `filename` with `metadata_filename` when present. - Replace `last_modified` with `metadata_last_modified` when present. - Apply `url` metadata when present. funcCallable[_P, list[Element]]rcLtjdfd }|S) aThe decorator function itself. This function is returned by the `apply_metadata()` function and is the actual decorator. Think of `apply_metadata()` as a factory function that configures this decorator, in particular by setting its `file_type` value. args_P.argskwargs _P.kwargsrr.c|i|}tg|Ri|}t|}|d}|dd}tt |||}i}|dp }| |j|d<|dp|d}|r||d<|d } | r| |d <|d } | r| |d <|D]3} | jjr| jtd i|4|d d} | durt|}t|}|S)N languagesdetect_language_per_elementF)r*rOrPmetadata_file_typefiletypemetadata_filenamermetadata_last_modified last_modifiedurlunique_element_ids) r _uniqueify_elements_and_metadatar6r:r mime_typer3attached_to_filenameupdater _assign_hash_idsrB)rJrLr* call_argsrOrPmetadata_kwargsrQrrTrVr<rWrCrGs rwrapperz2apply_metadata..decorator..wrapperstT,V,,H7NtNNNvNNI8AAH" k22I*3--8UW\*]*] '#%'0KH/1O"+/C!D!D!Q !-.@.J +!}}%899VY]]:=V=VH 7.6 +&/]]3K%L%L "% J3I0--&&C -),&$ L L#8 ''(J(J/(J(JKKKK(1}}5I5'Q'Q !U**+H55-X66HOr)rJrKrLrMrr.) functoolswraps)rGr`rCs` r decoratorz!apply_metadata..decoratorsM   O O O O O O   O br)rGrHrrHrX)rCrcs` rapply_metadatardxs1@ZZZZZZx rcd|D}dtj|D}t||D]\}}|||S)a.Converts `.id` of each element from UUID to hash. The hash is based on the `.text` of the element, but also on its page-number and sequence number on that page. This provides for deterministic results even when the document is split into one or more fragments for parallel processing. c&g|]}|jjSrX)r3 page_number).0es r z$_assign_hash_ids..s===qAJ*===rcBg|]\}}t|D]\}}|SrX) enumerate)rh_group seq_on_pages rrjz$_assign_hash_ids..sO Au'.. K r) itertoolsgroupbyzip id_to_hash)r* page_numberspage_seq_numbersr<seq_on_page_counters rr]r]s>=H===L!),77 ),H6F(G(G00$$.//// Orc:dd}t||S)zEnsure each of `elements` and their metadata are unique instances. This prevents hard-to-diagnose bugs downstream when mutating one element unexpectedly also mutates others because they are the same instance. r*r.rIterator[Element]c3Kt}t}|D]}t||vrtj|}t|j|vrtj|j|_|t||t|j|VdS)zLSubstitute deep-copies of any non-unique elements or metadata in `elements`.N)setr7copydeepcopyr3add)r* seen_elements seen_metadatar<s riter_unique_elementsz>_uniqueify_elements_and_metadata..iter_unique_elementss"%%% "%%%   G'{{m++-00'"##}44#'=1A#B#B   bkk * * *   b!122 3 3 3MMMM  r)r*r.rrx)r:)r*rs rrYrY s2     $$X.. / //r)rrrr)r*r+r,r-rr.)N)rCrDrrE)r*r.rr.) __doc__ __future__rr{rrrarprtypingrrrrtyping_extensionsrunstructured.documents.elementsr r unstructured.file_utils.modelr "unstructured.partition.common.langr unstructured.utilsr rrHIERARCHY_RULE_SETrBrdr]rYrXrrrsDD""""""  444444444444''''''DDDDDDDD222222BBBBBB>>>>>>Yt__ 7 7 7 7      8BT00000r"&|||||~*000000r