Ngc4ddlmZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z mZmZmZddlmZddlmZddlmZdd lmZdd lmZmZdd lmZdd lmZej e!Z"d Z#dZ$dZ%dZ&gdZ'gdZ(dgZ)gdZ*e'e(e)e*dZ+Gdde,eZ-GddeZ.GddeZ/GddeZ0GddeZ1Gd d!eZ2d;d%Z3dd+Z6d?d,Z7 d@dAd2Z8dBd4Z9dCd6Z:dDd8Z;Gd9d:eZr>QsD, III# III)B4MMM* GGGOOOMMM(G44r5r>c*eZdZUdZded< ded<dS) FrameworkzPebblo Framework instance.r9nameversionNr;r4r5r6rMrMjs-$$ III LLL##r5rMcreZdZUdZded< ded< ded< ded< ded < d ed < ded < d ed <dS)AppzPebblo AI application.r9rNownerrD descriptionload_idr>rKrM frameworkplugin_versionclient_versionNr;r4r5r6rQrQss   IIIJJJ!LLL-%'***r5rQceZdZUdZded< ded< ded< ded< ded< d ed < d ed < ded < ded< d ed<dS)DoczPebblo document.r9rNrRlistdocsrVrTdictloader_detailsbool loading_end source_ownerclassifier_locationanonymize_snippetsNr;r4r5r6rYrYs III0JJJJJJ.LLL-+7,%UUr5rYrBr9returnc|rd|vsd|dks|dvr|Stj|}|r|}t |S)zReturn an absolute local path for a local file/directory, for a network related path, return as is. Args: path (str): Relative path to be resolved. Returns: str: Resolved absolute path. z:///r)unknown-r+)pathlibPathexistsresolver9)rB full_paths r6 get_full_pathrmss  TMM 47NN 1 1 1  T""I(%%'' y>>r5loadercTtD] \}}||vr|cSdS)zReturn loader type among, file, dir or in-memory. Args: loader (str): Name of the loader, whose type is to be resolved. Returns: str: One of the loader type among, file/dir/in-memory. unsupported)LOADER_TYPE_MAPPINGitems)rn loader_typeloaderss r6get_loader_typerusE!4 9 9 ; ; W W       =r5rcddlm}m}m}m}d}t |t std|S|j } d|vrHt ||rd|j d|j }nt ||rd|j d|j }nd |vr!|d }|rd |vr|d }|r|d|}n_d |vr |d }nQd |vr |d }nCd |vr<|d }|r0t |trt|dkr|d}nt ||rd}nt ||r d|j}n|jjdkr|dr|d} d| }n|dr6|dg} dd| D}nJ|dr5|dg} dd| D}n#t($rYnwxYwt+t-|S)zReturn an absolute source path of source of loader based on the keys present in Document. Args: loader (BaseLoader): Langchain document loader, derived from Baseloader. r)r%rr&rrgzGloader is not derived from BaseLoader, source location will be unknown!bucketzgc://rezs3://sourcechannelrB file_path web_pathsr+z notiondb://r' folder_idz+https://drive.google.com/drive/u/2/folders/file_idsz, cg|]}d|d S)z https://drive.google.com/file/d/z/viewr4).0file_ids r6 z(get_loader_full_path..s4#J7IIIr5 document_idscg|]}d|d S)z#https://docs.google.com/document/d/z/editr4)rdoc_ids r6rz(get_loader_full_path..s4"LfKKKr5)$langchain_community.document_loadersr%rr&r isinstancerloggererror__dict__rwblobkeyrZlen database_id __class__r.getjoin Exceptionrmr9) rnr%rr&rlocation loader_dictryr{r|r}rs r6get_loader_full_pathrsH fj ) ) U   /K/ { " "&-00 @@6=@@6;@@FL11 @?6=??6:??  $ $"8,H 7I44%i07"*66W66H { " ""6*HH K ' '";/HH K ' '#K0I (Z 488 (S^^a=O=O$Q<  0 0 "HH  / / 9V%799HH   &*= = ={++ 'OOK88 TTT,, &??:r::99'/ 00 *~rBB 99&2      X ' ''sGH H&%H&Tuple[Framework, Runtime]c 4t}td|dd}tj}t |jtjd|dd|j |j t|dd|d d }d |jvrd |_ d |_ td|td|||fS)zFetch the current Framework and Runtime details. Returns: Tuple[Framework, Runtime]: Framework and Runtime for the current app instance. langchainlibrary_versionN)rNrOPWDrFrfrKruntime_version)rArBrFrGrHrErIrJDarwindesktopzMac OSXz framework zruntime )r rMrrFunamer>noderGenvironsystemrOget_ipr@rKrdebug) runtime_envrUrrKs r6 get_runtimers  *++K +//2CT"J"JI N  E Z Z Y77 <= 88I66$):IFF   G7:  # LL)i))*** LL%G%%&&& g r5cddl}|} ||}n%#t$r|d}YnwxYw|S)zJFetch local runtime ip address. Returns: str: IP address rN localhost)socket gethostname gethostbynamer)rrA public_ips r6rr.sp MMM     D6((.. 666((55 6 s0AAr[List[Document]max_batch_sizeintList[List[Document]]cJg}g}d}|D]}t|jd}||kr||gF||z|kr||g}d}||||z }|r|||S)a Generate batches of documents based on page_content size. Args: docs: List of documents to be batched. max_batch_size: Maximum size of each batch in bytes. Defaults to 100*1024(100KB) Returns: List[List[Document]]: List of batches of documents rutf-8)r page_contentencodeappend)r[rbatches current_batchcurrent_batch_sizedocdoc_sizes r6generate_size_based_batchesr>s%'G$&M++C,33G<<== n $ $ NNC5 ! ! ! !!H,~==}--- " %&"   % % % ( *  &}%%% Nr5rzc ddl}tj|j}||j}n#t $rd}YnwxYw|S)zFetch owner of local file path. Args: file_path (str): Local file path. Returns: str: Name of owner. rNrf)pwdrGstatst_uidgetpwuidpw_namer)rzrfile_owner_uidfile_owner_names r6get_file_owner_from_pathrfsb$ ++2,,~66> $$$#$ s7: A A  source_pathc|sdSd}tj|r tj|}ntj|rd}tj|D]l\}}}|D]c}tj||}tj|s"|tj|z }dm|}|S)zFetch size of source path. Source can be a directory or a file. Args: source_path (str): Local path of data source. Returns: int: Source size in bytes. r)rGrBisfilegetsizeisdirwalkrislink)rsize total_sizedirpath_ filenamesffps r6get_source_sizerys q D w~~k"" w{++ { # # %'W[%9%9 6 6 !GQ  6 6W\\'1--w~~b))6"'//""5"55J 6 Kr5datacN|d}t|}|S)zCalculate the content size in bytes: - Encode the string to bytes using a specific encoding (e.g., UTF-8) - Get the length of the encoded bytes. Args: data (str): Data string. Returns: int: Size of string in bytes. r)rr)rencoded_contentrs r6calculate_content_sizers'kk'**O   D Kr5ceZdZUdZded< dZded< ded< ded< d Zd ed < d2fd Zd3dZ d4d5dZ d6dZ d4d7dZ d8d#Z e d9d:d,Ze d;d/Ze d|jr|d|jintd|S)z Generate headers for the request. args: cloud_request (bool): flag indicating whether the request is for Pebblo cloud. returns: dict: Headers for the request. zapplication/json)Acceptz Content-Typez x-api-keyz,API key is missing for Pebblo cloud request.)rrrr)rrrs r6rz$PebbloLoaderAPIWrapper._make_headers(s`).    O| O T\:;;;;MNNNr5r[ List[dict]r`rrc |j|j|t|j|d||j|jd }|durd|d<d|vr ||dd<t d i|d}|S) a Build the payload for document classification. Args: app (App): App instance. docs (List[dict]): List of documents to be classified. loader_details (dict): Loader details. source_owner (str): Owner of the source. source_aggregate_size (int): Aggregate size of the source. loading_end (bool): Boolean indicating the halt of data loading by loader. Returns: dict: Payload for document classification. false) rNrRr[rVrTr]r_r`rarbTtruer_r]rrr4)rNrRrrTrarbrYr\)rrr[r]r`rr_rs r6rz3PebbloLoaderAPIWrapper.build_classification_payload?s0HY,{,"(#'#;"&"9 # #  $  %+GM "7**)()*AB....%%D%99r5NmethodurlrOptional[dict]timeoutOptional[Response]c   t|||||}td||jjt t |jjr |jjngt |j|jtj kr#t d|jno|jtj kr#t d|j n7|jtj kr"t d|j|S#t$rt d|Yn1t$r%}t d|Yd}~nd}~wwxYwdS) a Make a request to the Pebblo API Args: method (str): HTTP method (GET, POST, PUT, DELETE, etc.). url (str): URL for the request. headers (dict): Headers for the request. payload (Optional[dict]): Payload for the request (for POST, PUT, etc.). timeout (int): Timeout for the request in seconds. Returns: Optional[Response]: Response object if the request is successful. )rrrrrz5Request: method %s, url %s, len %s response status %szPebblo Server: Error z$Pebblo received an invalid payload: z-Pebblo returned an unexpected response code: zUnable to reach server %sz'An Exception caught in make_request: %sN)rrrrr9rbody status_coderINTERNAL_SERVER_ERRORr BAD_REQUESTrOKrr)rrrrrresponser s r6rz#PebbloLoaderAPIWrapper.make_requestks* I3gwH LLG $C1A1FN(--BOOPPH())    #z'GGGMx7KMMNNNN%)???UhmUUVVVV%66.+.. O = = = NN6 < < < < < I I I NNDa H H H H H H H H ItsD(D++%F FE;;FrTuple[List[dict], int]c g}d}d|D}d}|D]}|di}|dg} |ddkr*t|d|d } n7t|d |d|} |d t| } |d t| } t |d } t | }||z }|ddpd}|| | ||did| d| rd| ini| d| ini|ddkr|s|d|d <d}||fS)a Prepare documents for classification. Args: docs_with_id (List[IndexedDocument]): List of documents to be classified. source_path (str): Source path of the documents. loader_details (dict): Contains loader info. Returns: Tuple[List[dict], int]: Documents and the aggregate size of the source. rc6g|]}|Sr4)r\)rrs r6rzJPebbloLoaderAPIWrapper.prepare_docs_for_classification..s :::csxxzz:::r5Fmetadataauthorized_identitiesrnr(rxrrlrRrrr:N last_modified)rrr:r& file_ownersource_path_sizesource_full_urlT)rrmrrr9rr)rrr]r[r doc_contentsource_path_updater doc_metadatadoc_authorized_identitiesdoc_source_pathdoc_source_ownerdoc_source_sizerpage_content_sizers r6rz6PebbloLoaderAPIWrapper.prepare_docs_for_classifications5$ !::\::: ". *. *C77:r22L(4(8(89PRT(U(U %h'+==="/ $$X~m/LMM###0 $$#$((;??## ,//1/BB   +..v7W7WXXOsww~6677L 6| D D  !%6 6 !WWWd++0qF KK'#2#%(WWZ%<%<%@%@%Q%Q"2 5 02KLL+6,_==   (x(,>>>*?1=0@0@AR0S0S}-%)"***r5rc B|D]}||di}||d|d|di|did|ddS) z Update the document data with classified information. Args: docs (List[dict]): List of document data to be updated. classified_docs (dict): The dictionary containing classified documents. r: pb_checksumloader_source_pathentitiestopics)r3r4r5r6rN)rrr)r[rdoc_dataclassified_datas r6rz&PebbloLoaderAPIWrapper.update_doc_datas H-11(72CRHHO OO#2#6#6}#E#E*9*=*=>R*S*S / 3 3J C C-11(B??     LL     r5)rr)rrQrcr)F) rrrrQr]r\r_r^rcr\)rr\rcr)rr^rcr\)rrQr[rr]r\r`r9rrr_r^rcr\)Nr) rr9rr9rr\rrrrrcr)rrrr9r]r\rcr!)r[rrr\rcr)r.r/r0r1r<rarbrrr rrr staticmethodrrr __classcell__)rs@r6rrs(("&&&&&I!!!!&!$$$$$U # # # # # #NNNNH" <<<<<| U U U U.****X #' ////\/bD+D+D+\D+L   \     r5r)rBr9rcr9)rnr9rcr9)rnrrcr9)rcr)rcr9)r)r[rrrrcr)rzr9rcr9)rr9rcr)rr9rcr)= __future__rrloggingrGrhrFenumrhttprtypingrrrr r langchain_core.documentsr langchain_core.envr langchain_core.utilsr pydanticrrequestsrrrequests.exceptionsr)langchain_community.document_loaders.baser getLoggerr.rrrrBATCH_SIZE_BYTES file_loader dir_loader in_memory cloud_folderrqr9r-r8r>rMrQrYrmrurrrrrrrrr4r5r6rMs""""""  33333333333333------666666555555&&&&&&&&000000@@@@@@  8 $ $11           -----S$---%%%%%h%%%55555i5552$$$$$ $$$+++++)+++*VVVVV)VVV2.    E(E(E(E(P>    "1;%%%%%P&2     W W W W W YW W W W W r5