Ng`<<dZddlZddlZddlZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZer dd lmZdd lmZejeZeGd dZe dddGddeZ dS)zModule contains a PDF parser based on Document AI from Google Cloud. You need to install two libraries to use this parser: pip install google-cloud-documentai pip install google-cloud-documentai-toolbox N) dataclass) TYPE_CHECKINGIteratorListOptionalSequence) deprecated)Document) batch_iterate)BaseBlobParser)Blob)get_client_info) OperationDocumentProcessorServiceClientc(eZdZUdZeed<eed<dS)DocAIParsingResultsz/Dataclass to store Document AI parsing results. source_path parsed_pathN)__name__ __module__ __qualname____doc__str__annotations__n/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/parsers/docai.pyrrs099rrz0.0.32z1.0z&langchain_google_community.DocAIParser)sinceremovalalternative_importc eZdZdZddddddeddeedeedeefd Zd ed ee fd Z d$d ede deedee e d ee f dZ d%deedeede de d ee f dZde ed ee fdZde ed e dfdZde dd e fdZdddd dd deedeedeed!e de deed e dfd"Zde dd e efd#ZdS)& DocAIParserz`Google Cloud Document AI` parser. For a detailed explanation of Document AI, refer to the product documentation. https://cloud.google.com/document-ai/docs/overview N)clientlocationgcs_output_pathprocessor_namer$rr%r&r'ct|t|krtdd}|r(tj||std|d||_||_|r ||_d S ddlm}ddl m }n"#t$r}td|d }~wwxYw||d  } || td |_d S)aInitializes the parser. Args: client: a DocumentProcessorServiceClient to use location: a Google Cloud location where a Document AI processor is located gcs_output_path: a path on Google Cloud Storage to store parsing results processor_name: full resource name of a Document AI processor or processor version You should provide either a client or location (and then a client would be instantiated). zGYou must specify either a client or a location to instantiate a client.z?projects\/[0-9]+\/locations\/[a-z\-0-9]+\/processors\/[a-z0-9]+zProcessor name z has the wrong format. If your prediction endpoint looks like https://us-documentai.googleapis.com/v1/projects/PROJECT_ID/locations/us/processors/PROCESSOR_ID:process, use only projects/PROJECT_ID/locations/us/processors/PROCESSOR_ID part.r) ClientOptionsrZdocumentai package not found, please install it with `pip install google-cloud-documentai`Nz-documentai.googleapis.com) api_endpointz document-ai)module)client_options client_info) bool ValueErrorre fullmatch_gcs_output_path_processor_name_clientgoogle.api_core.client_optionsr)google.cloud.documentair ImportErrorr) selfr$r%r&r'patternr)rexcoptionss r__init__zDocAIParser.__init__2s]* <<4>> ) )  U  ",w"G"G . !0-  !DLLL HHHHHHRRRRRRR   !=  $m (DDDG:9&+=AAADLLLs6 B B" BB"blobreturnc#PK||g|jEd{VdS)zParses a blob lazily. Args: blobs: a Blob to parse This is a long-running operation. A recommended way is to batch documents together and use the `batch_parse()` method. r&N) batch_parser3)r9r>s r lazy_parsezDocAIParser.lazy_parsels>##TFDresponses r z-DocAIParser.online_process..su    ..t{HrDrErFrIrJrKrLr;rVrWrNrhs ` @@ronline_processzDocAIParser.online_processws$  / / / / / /               9     W W W W W W W   A  ) II0I J J J J  9C L " " 4 4 4 4 !<00  % %)'33 I"m@/@4!/)-E!!!#'% &           !)/    s' 727A A! AA!<blobs timeout_seccheck_in_interval_secc# K|p|j}|std|||}d|D}td|d}||rat j|||z }||krtd|dtd||a| | } | | Ed {Vd S) aParses a list of blobs lazily. Args: blobs: a list of blobs to parse. gcs_output_path: a path on Google Cloud Storage to store parsing results. timeout_sec: a timeout to wait for Document AI to complete, in seconds. check_in_interval_sec: an interval to wait until next check whether parsing operations have been completed, in seconds This is a long-running operation. A recommended way is to decouple parsing from creating LangChain Documents: >>> operations = parser.docai_parse(blobs, gcs_path) >>> parser.is_running(operations) You can get operations names and save them: >>> names = [op.operation.name for op in operations] And when all operations are finished, you can use their results: >>> operations = parser.operations_from_names(operation_names) >>> results = parser.get_results(operations) >>> docs = parser.parse_from_results(results) :An output path on Google Cloud Storage should be provided.rAc&g|]}|jjSr) operationrXrgops r z+DocAIParser.batch_parse..sBBB2<,BBBrz9Started parsing with Document AI, submitted operations %srz#Timeout exceeded! Check operations z later!.) operationsN) r3r0 docai_parseloggerdebug is_runningtimesleep TimeoutError get_resultsparse_from_results) r9rtr&rurv output_pathroperation_names time_elapsedresultss rrBzDocAIParser.batch_parsesJ4&>)>  L %%e[%II BBzBBB G    ooj))  J, - - - 1 1Lk))"U_UUU LL   ooj)) ""j"99**733333333333rrc# K ddlm}ddlm}ddlmn"#t $r}t d|d}~wwxYw|D]6 | j\}}|||} fd|DEd{V7dS)Nr) split_gcs_uri) _get_shardsrMrOc3K|]@}|jD]6}t|j|j|jjdV7AdSr])rQr rbrdrer)rgshardr^rNresults rriz1DocAIParser.parse_from_results..s !K   !2!24; !K!K&*&6&BTUUr)7google.cloud.documentai_toolbox.utilities.gcs_utilitiesr1google.cloud.documentai_toolbox.wrappers.documentrrlrNr8r) r9rrrr;gcs_bucket_name gcs_prefixshardsrNrs @@rrzDocAIParser.parse_from_resultss)        V U U U U U W W W W W W W   A     F*7-8J*K*K 'OZ [*==F $           s 838rrcx ddlmn"#t$r}td|d}~wwxYwfd|DS)z5Initializes Long-Running Operations from their names.r)GetOperationRequestzhlong running operations package not found, please install it with `pip install gapic-google-longrunning`NcZg|]'}j|(S))rX)request)r5 get_operation)rgrXrr9s rr}z5DocAIParser.operations_from_names..sK    L & &/B/B/M/M/M & N N   r)!google.longrunning.operations_pb2rr8)r9rr;rs` @roperations_from_namesz!DocAIParser.operations_from_names s           :        '    s *%*rc4td|DS)Nc3@K|]}| VdS)N)doner{s rriz)DocAIParser.is_running..s,66Rrwwyy=666666r)any)r9rs rrzDocAIParser.is_runnings66:666666ri)r&r' batch_sizerDrErc  ddlmddlm}m}n"#t $r} t d| d} ~ wwxYw|p|j} | td|p|j}|tdg} t||D]}   fd | D  }  j | |  }|r|||nd}| |j|| ||d| S)a3Runs Google Document AI PDF Batch Processing on a list of blobs. Args: blobs: a list of blobs to be parsed gcs_output_path: a path (folder) on GCS to store results processor_name: name of a Document AI processor. batch_size: amount of documents per batch enable_native_pdf_parsing: a config option for the parser field_mask: a comma-separated list of which fields to include in the Document AI response. suggested: "text,pages.pageNumber,pages.layout" Document AI has a 1000 file limit per batch, so batches larger than that need to be split into multiple requests. Batch processing is an async long-running operation and results are stored in a output GCS bucket. rrH)rKrLr*Nrxz0A Document AI processor name should be provided.)sizeiterablecVg|]%}|j|jpd&S)rRrS)rorfrp)rgr>rIs rr}z+DocAIParser.docai_parse..OsP ! #..$(I&*m&H7H/r) documents) gcs_documents)rTrE)gcs_output_configrP)rVT)rXinput_documentsdocument_output_configrZr[)rjrIrkrKrLr8r3r0r4r BatchDocumentsInputConfig GcsDocumentsDocumentOutputConfigGcsOutputConfigappendr5batch_process_documentsBatchProcessRequest)r9rtr&r'rrDrErKrLr;rrbatch input_config output_configrZrIs @rrzDocAIParser.docai_parses6  / / / / / / R R R R R R R R R   9   &>)>  L (?4+?  !OPP P " UCCC& & E%??(55 %* 6@  L';;","A"Q"Q'J#R##<M- (y2K         4422+(4/<(7*. 3    s 1,1ct ddlmn"#t$r}td|d}~wwxYwfd|DS)Nr)BatchProcessMetadatar*cg|]e}t|jr |jjn#|jjjD]}t |j|jfS))rr) isinstanceraindividual_process_statuses deserializevaluerinput_gcs_sourceoutput_gcs_destination)rgr|statusrs rr}z+DocAIParser.get_results..~s    bk+?@@. 77)55K%-    "3"9        r)google.cloud.documentai_v1rr8)r9rr;rs @rrzDocAIParser.get_resultsus  G G G G G G G   9       !    s )$))TNN)Nrrrs)rrrrrrr=r rr rCr/rintrqrrBrrrrrrrrrr#r#&s >B"&)-(, 8889:83- 8 "# 8 ! 8888t St S(: S S S S+/$(*. F F F $(F SM F T#Y' F (  F F F F V*.%' /4/4~/4"#/4 /4 # /4 (  /4/4/4/4b/0 ( 4 T#Y 4 CT    "7T+%6747777*.(,*.$(UUU~U"# U ! U  U$(USMU k UUUUn d;&7 DATrs !!!!!!DDDDDDDDDDDDDD666666------333333DDDDDDBBBBBBBBBBBBG333333FFFFFF  8 $ $     ? ` ` ` ` ` .` `  ` ` ` r