NgQddlZddlZddlZddlmZmZddlmZmZm Z m Z m Z ddl m Z ddlmZGddeeZGdd eZGd d eZdS) N)ABCabstractmethod)DictIteratorOptionalTupleUnion)Document) BaseLoaderc!eZdZdZdddddddd d dddd d d d ed ededeeefdededededededeeefdeeefdeeefdeedeedd f dZ de e fdZ e defdZdedefd Zd!ed"ede e fd#Z d(d!ed ed$eede e fd%Zd&edeeeffd'Zd S))DedocBaseLoadera Base Loader that uses `dedoc` (https://dedoc.readthedocs.io). Loader enables extracting text, tables and attached files from the given file: * `Text` can be split by pages, `dedoc` tree nodes, textual lines (according to the `split` parameter). * `Attached files` (when with_attachments=True) are split according to the `split` parameter. For attachments, langchain Document object has an additional metadata field `type`="attachment". * `Tables` (when with_tables=True) are not split - each table corresponds to one langchain Document object. For tables, Document object has additional metadata fields `type`="table" and `text_as_html` with table HTML representation. documentTF auto_tabbyrus+eng:autoN)split with_tableswith_attachmentsrecursion_deep_attachmentspdf_with_text_layerlanguagepagesis_one_column_documentdocument_orientationneed_header_footer_analysisneed_binarizationneed_pdf_table_analysis delimiterencoding file_pathrrrrrrrrrrrrr r!returnc.dtD|_hd|_||jvrt d|d|jd||_||_||_|jdkrdnd}||jd <||jd <d S) a Initialize with file path and parsing parameters. Args: file_path: path to the file for processing split: type of document splitting into parts (each part is returned separately), default value "document" "document": document text is returned as a single langchain Document object (don't split) "page": split document text into pages (works for PDF, DJVU, PPTX, PPT, ODP) "node": split document text into tree nodes (title nodes, list item nodes, raw text nodes) "line": split document text into lines with_tables: add tables to the result - each table is returned as a single langchain Document object Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing PDF documents, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer and images, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing PDF documents is_one_column_document: detect number of columns for PDF without a textual layer and images, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer and images, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result for parsing PDF and images need_binarization: clean pages background (binarize) for PDF without a textual layer and images need_pdf_table_analysis: parse tables for PDF without a textual layer and images delimiter: column separator for CSV, TSV files encoding: encoding of TXT, CSV, TSV c"i|] \}}|dv || S)>selfrr"r).0keyvalues f/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/dedoc.py z,DedocBaseLoader.__init__..ds4# # # UGGG GGG>linenodepagerGot $ for `split`, but should be one of ``r/treelinearstructure_typeneed_content_analysisN)localsitemsparsing_parametersvalid_split_values ValueErrorrrr")r&r"rrrrrrrrrrrrr r!r6s r+__init__zDedocBaseLoader.__init__#sB# # $hhnn..# # #  #G"F"F / / //u//+///  &"#':#7#7X4B 01;K 7888r-c#Kddl} ddlm}n#t$rtdwxYw||}d|jd_|5}||j i|j d|i }dddn #1swxYwY| | |j Ed{VdS) Lazily load documents.rN) DedocManagerzE`dedoc` package not found, please install it with `pip install dedoc`)manager_configTloggerattachments_dir)r" parameters document_treer)tempfilededocr@ ImportError _make_configconfigdisabledTemporaryDirectoryparser"r:_split_document to_api_schemadictr)r&rGr@ dedoc_managertmpdirrFs r+ lazy_loadzDedocBaseLoader.lazy_loadws  * * * * * * *   W  % D4E4E4G4GHHH 26 X&/  ( ( * * f)//.Qd5Q7H&QQ0M                '''5577<<>>dj(           s)0(B$$B(+B(cdS)zu Make configuration for DedocManager according to the file extension and parsing parameters. Nr'r&s r+rJzDedocBaseLoader._make_configs r- paragraphcdfd|dD}|r |dd|n|d}|S)z1Get text (recursively) of the document tree node. c:g|]}|Sr') _json2txt)r( subparagraphr&s r+ z-DedocBaseLoader._json2txt..s5    |,,   r- subparagraphstext)join)r&rWsubparagraphs_textr_s` r+r[zDedocBaseLoader._json2txtsy!YY    $-o$>     " #y 8 8$6 8 8 86"  r-rFdocument_metadatac#Kt|ddkr*|dD]}|||Ed{V dSt|di||dVdS)z4Parse recursively document tree obtained by `dedoc`.r^rrFrbNr_metadata page_contentre)len_parse_subparagraphsr )r&rFrbr\s r+riz$DedocBaseLoader._parse_subparagraphss }_- . . 2 2 -o >   44".BS5   *62K-Kz1JK     r-additional_metadatac#FK|d}|ri||}|dkr7||dd}t||VnQ|dkr|ddd}|d dd }d }|D]i} | dd |kr||| z }-t|i|d |iV| dd }|| }jt|i|d |iVn|d krJ|dddD]4} | d} t|| i|| V5nK|d kr*||dd|Ed{Vntd|d|jd|jrI|ddD]:} || \} } t| i| dd| dV;|dD]'}|||jddiEd{V(dS)z=Split document into parts according to the `split` parameter.rercontent structure)rWrfr0r^rpage_idr.r/rdNr1r2r3tablestable)type text_as_html attachmentsrr attachment)rFrrj) r[r rir<r;r _get_tablerOr)r&rFrrjrbr_nodesrn page_textr/ line_metadatarq table_text table_htmlrus r+rOzDedocBaseLoader._split_documents**5  M L#4 L8K L  J  >>M),D[,Q>RRD7HIII I I I I f__!),[9/JEAhz*95GI 5 5 #I.'99!5!55II"%.!J$5!Jy'!J!J#:.y9G $t 4 4II&B-By'BB     f__%i0=oN   $Z 0 !%!5!5C 1C]C f__00+I6{C"31         /u//+///    &y1(;  )-)?)?& J!+ + '(2( 6  J++(j%+\$:,          r-rqc d}|dD]9}|D]/}|dd|dDz }|dz }0|dz }:d}|dD]w}|d z }|D]h}dd |dD}tj|}|d z }|d r|d z }|d|dd|dd|dz }i|dz }x|dz }||fS)z.Get text and HTML representation of the table.rocells c3&K|] }|dV dSr_Nr'r(r.s r+ z-DedocBaseLoader._get_table..s&&N&NtF|&N&N&N&N&N&Nr-lines rYzK z c3&K|] }|dV dSrr'rs r+rz-DedocBaseLoader._get_table..s&%M%Mtd6l%M%M%M%M%M%Mr-z z z
z
)r`htmlescape)r&rqrzrowcellr{ cell_texts r+rvzDedocBaseLoader._get_tables_ >  C # #chh&N&NW &N&N&NNNN d" $ JJ  > $ $C ( "J   II%M%MtG}%M%M%MMM  K 22 e#  $<";;J>i>>Y>>+4>>>  ) #JJ** :%%r-N)__name__ __module__ __qualname____doc__strboolr intrr=rr rTrrQrJr[rirOrrvr'r-r+r r sz(  -2*,#/!&,$*8=.348#'"&#RLRLRLRL RL  RL  T * RL%(RL!RLRLRL!$RL"RL&+39%5RL!d+RL"'sDy!1RL C=!RL"3-#RL$ %RLRLRLRLh 8H-    , d   ^  4 C     ! 6:  (     &/3 IIII&d^ I (  IIIIV&&sCx&&&&&&r-r ceZdZdZdefdZdS)DedocFileLoaderaw DedocFileLoader document loader integration to load files using `dedoc`. The file loader automatically detects the file type (with the correct extension). The list of supported file types is gives at https://dedoc.readthedocs.io/en/latest/index.html#id1. Please see the documentation of DedocBaseLoader to get more details. Setup: Install ``dedoc`` package. .. code-block:: bash pip install -U dedoc Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocFileLoader loader = DedocFileLoader( file_path="example.pdf", # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } r#cHddlm}||j|j|jS)Nr)make_manager_config)r"parsing_paramsr)dedoc.utils.langchainrr"r:r)r&rs r+rJzDedocFileLoader._make_config`s>======""n2*    r-N)rrrrrQrJr'r-r+rrs<@@D d      r-rc#@eZdZdZdddddddd d d dddd d d d ededededeeefdededededededeeefdeeefdeeefdeedeedd f"fdZ de e fdZ de fd Zded ed!e deeeee efffd"ZxZS)#DedocAPIFileLoaderaU Load files using `dedoc` API. The file loader automatically detects the file type (even with the wrong extension). By default, the loader makes a call to the locally hosted `dedoc` API. More information about `dedoc` API can be found in `dedoc` documentation: https://dedoc.readthedocs.io/en/latest/dedoc_api_usage/api.html Please see the documentation of DedocBaseLoader to get more details. Setup: You don't need to install `dedoc` library for using this loader. Instead, the `dedoc` API needs to be run. You may use Docker container for this purpose. Please see `dedoc` documentation for more details: https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker .. code-block:: bash docker pull dedocproject/dedoc docker run -p 1231:1231 Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocAPIFileLoader loader = DedocAPIFileLoader( file_path="example.pdf", # url=..., # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } zhttp://0.0.0.0:1231rTFrrrrrN)urlrrrrrrrrrrrrr r!r"rrrrrrrrrrrrrr r!r#ct|||||||| | | | | |||||_d|jd<dS)a Initialize with file path, API url and parsing parameters. Args: file_path: path to the file for processing url: URL to call `dedoc` API split: type of document splitting into parts (each part is returned separately), default value "document" "document": document is returned as a single langchain Document object (don't split) "page": split document into pages (works for PDF, DJVU, PPTX, PPT, ODP) "node": split document into tree nodes (title nodes, list item nodes, raw text nodes) "line": split document into lines with_tables: add tables to the result - each table is returned as a single langchain Document object Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing PDF documents, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer and images, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing PDF documents is_one_column_document: detect number of columns for PDF without a textual layer and images, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer and images, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result for parsing PDF and images need_binarization: clean pages background (binarize) for PDF without a textual layer and images need_pdf_table_analysis: parse tables for PDF without a textual layer and images delimiter: column separator for CSV, TSV files encoding: encoding of TXT, CSV, TSV )r"rrrrrrrrrrrrr r!json return_formatN)superr=rr:)r&r"rrrrrrrrrrrrrr r! __class__s r+r=zDedocAPIFileLoader.__init__spB #-'A 3#9!5(C/$;    "39000r-c#K||j|j|j}|||jEd{VdS)r?)rr"rDrEN) _send_filerr"r:rOr)r&doc_trees r+rTzDedocAPIFileLoader.lazy_load sa??DNt?V#  ''hdj'QQQQQQQQQQQr-ciSrr'rVs r+rJzDedocAPIFileLoader._make_configs r-rDcddl}tj|}t |d5}d||fi}||d||}dddn #1swxYwY|jdkr)td|j tj |j } | S) z7Send POST-request to `dedoc` API and return the resultsrNrbfilez/upload)filesdatazError during file handling: ) requestsospathbasenameopenpost status_coder<rldecoderloads) r&rr"rDr file_namerrrresults r+rzDedocAPIFileLoader._send_files G$$Y// )T " " Mdi./E oooU LLA M M M M M M M M M M M M M M M =C  PAIrsJ ########.-----@@@@@@H&H&H&H&H&j#H&H&H&VJ J J J J oJ J J Zxxxxxxxxxxr-