NgQ,dZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZejeZ Gdd eZ!Gd d eZ"dS) z:Pebblo's safe dataloader is a wrapper for document loadersN)version)AnyDictIterableIteratorListOptional)Document) BaseLoader) BATCH_SIZE_BYTESPLUGIN_VERSIONApp FrameworkIndexedDocumentPebbloLoaderAPIWrappergenerate_size_based_batches get_full_pathget_loader_full_pathget_loader_type get_runtimeget_source_sizecFeZdZUdZdZeed< d dddded ed ed ed e ed ede ededefdZ de e fdZ d!dZdee fdZed!dZdefdZde efdZdede e fdZde e fdZde dede fdZdeddfdZdS)"PebbloSafeLoaderzkPebblo Safe Loader class is a wrapper around document loaders enabling the data to be scrutinized. F_discover_sentNlocal)classifier_locationanonymize_snippetslangchain_loadernameowner descriptionapi_key load_semanticclassifier_urlrrc|rt|tstd||_tt j|_||_tj dp||_ ||_ ||_ t|j|_g|_g|_tt%|jdddd} t)| |_t-|j|_t0|_| |j|jd|jdkrdt|jini|_||_t;|||| |_|j|jdS) NzMust specify a valid name.PEBBLO_LOAD_SEMANTIC.'r)loader source_path source_typesource_path_size)r#rr%r) isinstancestr NameErrorapp_nameuuiduuid4load_idr+osenvirongetr$r!r"rr,docs docs_with_idtypesplitrr-rr.r batch_sizeloader_details_get_app_detailsappr pb_clientsend_loader_discover) selfrr r!r"r#r$r%rr loader_names g/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/pebblo.py__init__zPebbloSafeLoader.__init__%s ::dC00 :899 9 4:<<(( & Z^^,BCCT} &/ <<$& 35$t{++,,22377;AA#FFqI *;77 /0@ A A*!++   (1,,$S)>%?%?@@  ((**/ 3)1     ++DH55555returnct|j|_||jS)zxLoad Documents. Returns: list: Documents fetched from load method of the wrapped `loader`. )r+loadr9classify_in_batches)rCs rErJzPebbloSafeLoader.loadVs3 K$$&&    """yrGct|j|j}g}t|}t |D]\}}||dz k}||_||_|j|j|j |j |}| ||j r| |}n|}||||_dS)z Classify documents in batches. This is to avoid API timeouts when sending large number of documents. Batches are generated based on the page_content size. ) loading_endN)rr9r=len enumerate _index_docsr:rAclassify_documentsr@r>_add_pebblo_specific_metadatar$_add_semantic_to_docs _unindex_docsextend) rCbatchesprocessed_docs total_batchesibatch is_last_batchclassified_docsbatch_processed_docss rErKz$PebbloSafeLoader.classify_in_batchesas )D It) ) *,G !'** 8 8HAu"#}q'8"8MDI $ 0 0 2 2D "n??!#) @O  . . ? ? ?! <'+'A'A/'R'R$$'+'9'9';';$  ! !"6 7 7 7 7" rGc#K |j}nP#t$rC}|jjjd}t |t||d}~wwxYw t|}n#t$r g|_ YdSwxYwt|f|_ | |_ |j |j |j|j}|||jr|||_ n||_ |j dV)zLoad documents in lazy fashion. Raises: NotImplementedError: raised when lazy_load id not implemented within wrapped loader. Yields: list: Documents from loader's lazy loading. z does not implement lazy_load()NTr)r+ lazy_loadNotImplementedError __class____name__loggererrornext StopIterationr9listrQr:rArRr@r>rSr$rTrU)rC doc_iteratorexcerr_strdocclassified_docs rEr`zPebbloSafeLoader.lazy_loadsf 8;0022LL" 8 8 8.7XXXG LL ! ! !%g..C 7 8  <((      cV DI $ 0 0 2 2D !^>>!48T-@N  . .~ > > >! 1 66~FF  ..00 )A,   ! s& A+>A&&A+0BBBcd|_dS)NT)r)clss rEset_discover_sentz"PebbloSafeLoader.set_discover_sents!rGct\}}t|j|j|j|j||t tdtd}|S)z\Fetch app details. Internal method. Returns: App: App details. langchain_community)r r)r r!r"r5runtime frameworkplugin_versionclient_version) rrr2r!r"r5r rr)rCrtrsr@s rEr?z!PebbloSafeLoader._get_app_detailssl )]] 7*(L)$* 566     rGcBdt|jD}|S)z Indexes the documents and returns a list of IndexedDocument objects. Returns: List[IndexedDocument]: A list of IndexedDocument objects with unique IDs. c ng|]2\}}tddt|i|3S)pb_id)rr0dict.0rZrls rE z0PebbloSafeLoader._index_docs..sM   3  7 7#a&& 7CHHJJ 7 7   rG)rPr9)rCr:s rErQzPebbloSafeLoader._index_docss2  #DI..   rGr]cd|jD}|D]7}|d}||vr||||8d|D}|S)aF Adds semantic metadata to the given list of documents. Args: classified_docs (Dict): A dictionary of dictionaries containing the classified documents with pb_id as key. Returns: List[Document]: A list of Document objects with added semantic metadata. cPi|]#}|jt|j|j$S) page_contentmetadata)ryr rrr}rls rE z:PebbloSafeLoader._add_semantic_to_docs..s>    IxS-= UUU   rGrycg|]}|Srzrzrs rEr~z:PebbloSafeLoader._add_semantic_to_docs..s!G!G!G##!G!G!GrG)r:valuesr8_add_semantic_to_doc)rCr] indexed_docsrmdoc_idsemantic_metadata_docss rErTz&PebbloSafeLoader._add_semantic_to_docss  (   .4466 P PN#''00F%%)),v*>OOO!G!G1D1D1F1F!G!G!G%%rGcBdt|jD}|S)z Converts a list of IndexedDocument objects to a list of Document objects. Returns: List[Document]: A list of Document objects. cJg|] \}}t|j|j!Sr)r rrr|s rEr~z2PebbloSafeLoader._unindex_docs..s>   3 #"2S\ J J J   rG)rPr:)rCr9s rErUzPebbloSafeLoader._unindex_docss2  #D$566    rGrlrmct|di|jd<t|di|jd<|S)a4 Adds semantic metadata to the given document in-place. Args: doc (Document): A Document object. classified_doc (dict): A dictionary containing the classified document. Returns: Document: The Document object with added semantic metadata. entitiespebblo_semantic_entitiestopicspebblo_semantic_topics)rhr8keysr)rCrlrms rErz%PebbloSafeLoader._add_semantic_to_docsy48   z2 . . 3 3 5 54 4  /026   x , , 1 1 3 32 2  -. rGc |jD]}|j}|jjjdkr,t |d|j|d<n?t |d|d|j|d<||jidd|d<dS)z*Add Pebblo specific metadata to documents.SharePointLoadersource full_path pb_checksumN) r:rr+rbrcrr8r,ry)rCr]rl doc_metadatas rErSz.PebbloSafeLoader._add_pebblo_specific_metadatas$  CO"9q 1222x "ADH --hqkcxHHH H H H H I IrGcbg}|D]}|||S)z` Load text data into Documents. Returns: List of Documents )r`append)rC documentsrls rErJzPebbloTextLoader.loadIs? >>## " "C   S ! ! ! !rG)rcrrrrr0r rrrrFrr r`rJrzrGrErrs!%#'-148###}# # d3i # 4S>* #Dc3h01# ####6I8H-IIII d8n      rGr)#rloggingr6r3importlib.metadatartypingrrrrrr langchain_core.documentsr )langchain_community.document_loaders.baser $langchain_community.utilities.pebblor r rrrrrrrrrr getLoggerrcrdrrrzrGrErs@@ &&&&&&@@@@@@@@@@@@@@@@------@@@@@@                              8 $ $uuuuuzuuup=====z=====rG