NgX|UdZddlmZddlZddlZddlmZmZmZm Z m Z ddl Z ddl m Z ddlmZmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddl m!Z!ede"efZ#de$d< dHdddddej%ddigdddddddddddddddddddId?Z&diddfdJdBZ'dKdEZ(GdFdGZ)dS)Lz9Provides partitioning with automatic file-type detection.) annotationsN)IOAnyCallableLiteralOptional) TypeAlias)DataSourceMetadataElement)detect_filetypeis_json_processable)FileType)logger)UnsupportedFileFormatError) exactly_one)check_language_args)PartitionStrategy)dependency_exists.r PartitionerF)pdfjpgpngheicT) content_typefile file_filenameurlinclude_page_breaksstrategyencodingparagraph_grouperheadersskip_infer_table_types ssl_verify ocr_languages languagesdetect_language_per_elementpdf_infer_table_structureextract_images_in_pdfextract_image_block_typesextract_image_block_output_dirextract_image_block_to_payload xml_keep_tagsdata_source_metadatametadata_filenamerequest_timeouthi_res_model_name model_namestarting_page_numberfilename Optional[str]rrOptional[IO[bytes]]rrrboolr strr!r"/Optional[Callable[[str], str]] | Literal[False]r#dict[str, str]r$ list[str]r%r&r'Optional[list[str]]r(r)r*r+r,r-r.r/Optional[DataSourceMetadata]r0r1 Optional[int]r2r3r4intkwargsrc H t||||r|rtd||}tjd|d||rtjdt | pg| } |t ||| | |\}}n.| ikrtjdt||||| }||d t|| |}t}|tj kr&| |}|d||||| |d |} n|tjkr'| |}!|!d|||| |||d |} na|tjkr'| |}"|"d|||| |||d |} n*|tjkr%| |}#|#d|||| |d |} n|tjkr&| |}$|$d||||| |d|} n|tjkr&| |}%|%d||||| |d|} n|jdkr/| |}&|&d||d|||| |p||||||d |} nO|tjkrBt+||std| |}'|'d||d|} n|tjkr&| |}(|(d||||| |d|} n|tjkr$| |})|)d||| |d|} n|tjkr'| |}*|*d|||| |||d |} n\|tjkr%| |}+|+d|||| |d|} n'|tjkr/| |},|,d||d|||| |p||||||d |} n|tjkr'| |}-|-d||||| ||d|} n|tjkr(| |}.|.d||||| |||d|} ny|tjkr&| |}/|/d||||| |d|} nC|tjkr&| |}0|0d||||| |d|} n |tjkr#| |}1|1d||| |d|} n|tj kr%| |}2|2d||||| |d|} n|tj!tj"fvr%| |}3|3d|||| ||d|} nf|tj#kr%| |}4|4d||||| |d|} n1|tj$krg} n|sdnd|}5tK|5d|d| D]W}6||6j&_'||6j&_(|*tj)|}7|7|7j*nd|6j&_+F|j*|6j&_+X| S) a5Partitions a document into its constituent elements. Uses libmagic to determine the file's type and route it to the appropriate partitioning function. Applies the default parameters for each partitioning function. Use the document-type specific partitioning functions if you need access to additional kwarg options. Parameters ---------- filename A string defining the target filename path. content_type A string defining the file content in MIME type file A file-like object using "rb" mode --> open(filename, "rb"). metadata_filename When file is not None, the filename (string) to store in element metadata. E.g. "foo.txt" url The url for a remote document. Pass in content_type if you want partition to treat the document as a specific content_type. include_page_breaks If True, the output will include page breaks if the filetype supports it strategy The strategy to use for partitioning PDF/image. Uses a layout detection model if set to 'hi_res', otherwise partition simply extracts the text from the document and processes it. encoding The encoding method used to decode the text input. If None, utf-8 will be used. headers The headers to be used in conjunction with the HTTP request if URL is set. skip_infer_table_types The document types that you want to skip table extraction with. ssl_verify If the URL parameter is set, determines whether or not partition uses SSL verification in the HTTP request. languages The languages present in the document, for use in partitioning and/or OCR. For partitioning image or pdf documents with Tesseract, you'll first need to install the appropriate Tesseract language pack. For other partitions, language is detected using naive Bayesian filter via `langdetect`. Multiple languages indicates text could be in either language. Additional Parameters: detect_language_per_element Detect language per element instead of at the document level. pdf_infer_table_structure Deprecated! Use `skip_infer_table_types` to opt out of table extraction for any document type. If True and strategy=hi_res, any Table Elements extracted from a PDF will include an additional metadata field, "text_as_html," where the value (string) is a just a transformation of the data into an HTML . The "text" field for a partitioned Table Element is always present, whether True or False. extract_images_in_pdf Only applicable if `strategy=hi_res`. If True, any detected images will be saved in the path specified by 'extract_image_block_output_dir' or stored as base64 encoded data within metadata fields. Deprecation Note: This parameter is marked for deprecation. Future versions will use 'extract_image_block_types' for broader extraction capabilities. extract_image_block_types Only applicable if `strategy=hi_res`. Images of the element type(s) specified in this list (e.g., ["Image", "Table"]) will be saved in the path specified by 'extract_image_block_output_dir' or stored as base64 encoded data within metadata fields. extract_image_block_to_payload Only applicable if `strategy=hi_res`. If True, images of the element type(s) defined in 'extract_image_block_types' will be encoded as base64 data and stored in two metadata fields: 'image_base64' and 'image_mime_type'. This parameter facilitates the inclusion of element data directly within the payload, especially for web-based applications or APIs. extract_image_block_output_dir Only applicable if `strategy=hi_res` and `extract_image_block_to_payload=False`. The filesystem path for saving images of the element type(s) specified in 'extract_image_block_types'. xml_keep_tags If True, will retain the XML tags in the output. Otherwise it will simply extract the text from within the tags. Only applies to partition_xml. request_timeout The timeout for the HTTP request if URL is set. Defaults to None meaning no timeout and requests will block indefinitely. hi_res_model_name The layout detection model used when partitioning strategy is set to `hi_res`. model_name The layout detection model used when partitioning strategy is set to `hi_res`. To be deprecated in favor of `hi_res_model_name`. starting_page_number Indicates what page number should be assigned to the first page in the document. This information will be reflected in elements' metadata and can be be especially useful when partitioning a document that is part of a larger document. )rr5rzOnly one of metadata_filename and file_filename is specified. metadata_filename is preferred. file_filename is marked for deprecation.NzuThe file_filename kwarg will be deprecated in a future version of unstructured. Please use metadata_filename instead.r0z]The pdf_infer_table_structure kwarg is deprecated. Please use skip_infer_table_types instead.)rrr#r%r1zUThe headers kwarg is set but the url kwarg is not. The headers kwarg will be ignored.) file_pathrr!rmetadata_file_pathr)r5rr!infer_table_structurer'r()r5rrEr'r(r4r )r5rr!r'r()r5rrrEr'r()r5rrr!r'r(image) r5rrrrEr r'r2r*r+r,r-r4)r5rzDetected a JSON file that does not conform to the Unstructured schema. partition_json currently only processes serialized Unstructured output.)r5rr'r()r5rrr'r()r5rrrEr'r(r )r5rrrEr'r(r4r )r5rr!r"r'r()r5rrEr'r(r4)r5rr!r.r'r(z Invalid filez Invalid file z. The z) file type is not supported in partition.),r ValueErrorrwarning setdefaultrfile_and_type_from_urlr seekdecide_table_extraction_PartitionerLoaderrCSVgetDOCDOCXEMLEPUBHTMLpartitioner_shortnameJSONr MDMSGODTORGPDFPPTPPTXRSTRTFTSVTXTXLSXLSXXMLEMPTYrmetadatar data_sourcefrom_mime_type mime_typefiletype)8r5rrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4rA file_typerEpartitioner_loader partition_csvelements partition_docpartition_docxpartition_emailpartition_epubpartition_htmlpartition_imagepartition_json partition_md partition_msg partition_odt partition_org partition_pdf partition_pptpartition_pptx partition_rst partition_rtf partition_tsvpartition_textpartition_xlsx partition_xmlmsgelement out_filetypes8 W/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/partition/auto.py partitionrs lTH#6666 ]  W    ) 4    )+<===      $IO]CCI 0%!+    ii b== N5   $%0      ! 3! ,--HL  *..y99  = "7(C      hl " "*..y99  =  "7(C!5       hm # #+// ::!>  "7(C!5       hl " ",00;;"? (C       hm # #+// ::!>  3"7(C      hm # #+// ::!>  3(C       (G 3 3,00;;"?  3"7/=:"7&?+I+I!5     hm # #"H4@@@ Z ,// ::!>I8$II&II hk ! !)--i88 <  3"7(C      hl " "*..y99  = (C       hl " "*..y99  =  "7(C!5       hl " "*..y99  =  3(C       hl " "*..y99  =  3"7/=:"7&?+I+I!5     hl " "*..y99  =   3"7(C       hm # #+// ::!>   3"7(C!5       hl " "*..y99  =  3"7(C      hl " "*..y99  =  3"7(C      hl " "*..y99  = (C       hl " "+// ::!> /(C      x|X]3 3 3+// ::!> "7(C!5      hl " "*..y99  = '(C      hn $ $$,Lnn2L(2L2L( N N) N N N   <<"';$  ##2<@@LBNBZ (>(>`dG  % %(1(;G  % % Oreturntuple[io.BytesIO, FileType]cZtj||||}tj|j}|p|jddx}r?|dd}|j }t|||}||fS)N)r#verifytimeoutz Content-Type;r)rr!r) requestsrPioBytesIOcontentr#splitstriplowerr!r ) rrr#r%r1responserr!rks rrKrKs|C_]]]H :h& ' 'D#Qx'7';';ND'Q'QQ|B#))#..q17799??AA  HD8,WWWH >rrkOptional[FileType]c\|r|jnd}|dkr|p||vS||vS)Nr)namer)rkr$r)doc_types rrMrMsK )1:x}""$$$dH5 )RHr0r6r1r?r2r6r3r6r4r@rAr) rr9rr6r#r;r%r8r1r?rr)rkrr$r<r)r8rr8)*r __future__rrrtypingrrrrrrtyping_extensionsr unstructured.documents.elementsr r unstructured.file_utils.filetyper r unstructured.file_utils.modelrunstructured.loggerrunstructured.partition.commonr$unstructured.partition.common.commonr"unstructured.partition.common.langr&unstructured.partition.utils.constantsrunstructured.utilsrlistrrAUTOrrKrMrNrGrrrsG???"""""" 77777777777777''''''GGGGGGGGQQQQQQQQ222222&&&&&&DDDDDD<<<<<<BBBBBBDDDDDD000000!#tG}"45 5555#r#' $#' %%*"IM (E(E(E#'%)(-&+"'5948+09='+%)'+ $ !9rrrrrrn #' %) (2222"-P-P-P-P-P-P-P-P-P-Pr