Ng $ddlmZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZmZmZdd lmZmZGd d eZdS) ) annotationsN)Path)AnyDictListOptionalTupleUnion)CallbackManagerForRetrieverRun)Document) BaseRetriever)convert_to_secret_strget_from_dict_or_envpre_init) ConfigDict SecretStrceZdZUdZded< dZded< edZed4d5d Z e d4d6dZ e d4d7dZ e d8dZ d9d:dZd;dZdd*Zd?d-Zd@d1ZdAd3ZdS)BNeuralDBRetrieverz0Document retriever that uses ThirdAI's NeuralDB.r thirdai_keyNrdbforbid)extra Optional[str]returnNonec ddlm}tjd||pt jddS#t$rtdwxYw)Nr) licensingzthirdai.neural_db THIRDAI_KEYz{Could not import thirdai python package and neuraldb dependencies. Please install it with `pip install thirdai[neural_db]`.) thirdair importlibutil find_specactivateosgetenv ImportError)rrs k/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/retrievers/thirdai_neuraldb.py_verify_thirdai_libraryz)NeuralDBRetriever._verify_thirdai_librarys  ) ) ) ) ) ) N $ $%8 9 9 9   {Fbi .F.F G G G G G   K  s AAA, model_kwargsdictc rt|ddlm}|||jdi|S)a Create a NeuralDBRetriever from scratch. To use, set the ``THIRDAI_KEY`` environment variable with your ThirdAI API key, or pass ``thirdai_key`` as a named parameter. Example: .. code-block:: python from langchain_community.retrievers import NeuralDBRetriever retriever = NeuralDBRetriever.from_scratch( thirdai_key="your-thirdai-key", ) retriever.insert([ "/path/to/doc.pdf", "/path/to/doc.docx", "/path/to/doc.csv", ]) documents = retriever.invoke("AI-driven music therapy") r neural_dbrr)rr(rr-NeuralDB)clsrr)ndbs r' from_scratchzNeuralDBRetriever.from_scratch*sQ: 11+>>>,,,,,,s{|s|/K/Kl/K/KLLLL checkpointUnion[str, Path]ct|ddlm}|||j|S)a! Create a NeuralDBRetriever with a base model from a saved checkpoint To use, set the ``THIRDAI_KEY`` environment variable with your ThirdAI API key, or pass ``thirdai_key`` as a named parameter. Example: .. code-block:: python from langchain_community.retrievers import NeuralDBRetriever retriever = NeuralDBRetriever.from_checkpoint( checkpoint="/path/to/checkpoint.ndb", thirdai_key="your-thirdai-key", ) retriever.insert([ "/path/to/doc.pdf", "/path/to/doc.docx", "/path/to/doc.csv", ]) documents = retriever.invoke("AI-driven music therapy") rr,r.)rr(rr-r0from_checkpoint)r1r5rr2s r'r8z!NeuralDBRetriever.from_checkpointLsO< 11+>>>,,,,,,s{s|/K/KJ/W/WXXXXr4valuesrcHtt|dd|d<|S)z'Validate ThirdAI environment variables.rr)rr)r1r9s r'validate_environmentsz'NeuralDBRetriever.validate_environmentsos5!6   ! ! } r4Tsources List[Any]trainbool fast_modekwargsc \||}|jjd|||d|dS)asInserts files / document sources into the retriever. Args: train: When True this means that the underlying model in the NeuralDB will undergo unsupervised pretraining on the inserted files. Defaults to True. fast_mode: Much faster insertion with a slight drop in performance. Defaults to True. )r<r>fast_approximationNr/)_preprocess_sourcesrinsert)selfr<r>r@rAs r'rEzNeuralDBRetriever.insert{sV **733 (        r4listcddlm}|s|Sg}|D]0}t|ts||.|dr)|||~|dr)||||dr*|| |td|d|S)zChecks if the provided sources are string paths. If they are, convert to NeuralDB document objects. Args: sources: list of either string paths to PDF, DOCX or CSV files, or NeuralDB document objects. rr,z.pdfz.docxz.csvzCould not automatically load z. Only files with .pdf, .docx, or .csv extensions can be loaded automatically. For other formats, please use the appropriate document object from the ThirdAI library.) rr- isinstancestrappendlowerendswithPDFDOCXCSV RuntimeError)rFr<r2preprocessed_sourcesdocs r'rDz%NeuralDBRetriever._preprocess_sourcessR -,,,,, N!  Cc3'' $++C000099;;''// (// ====YY[[))'22 (// >>>>YY[[))&11(// ====&PPPP $#r4queryrJ document_idintc<|j||dS)a!The retriever upweights the score of a document for a specific query. This is useful for fine-tuning the retriever to user behavior. Args: query: text to associate with `document_id` document_id: id of the document to associate query with. N)rtext_to_result)rFrTrUs r'upvotezNeuralDBRetriever.upvotes" uk22222r4query_id_pairsList[Tuple[str, int]]c:|j|dS)aGiven a batch of (query, document id) pairs, the retriever upweights the scores of the document for the corresponding queries. This is useful for fine-tuning the retriever to user behavior. Args: query_id_pairs: list of (query, document id) pairs. For each pair in this list, the model will upweight the document id for the query. N)rtext_to_result_batch)rFrZs r' upvote_batchzNeuralDBRetriever.upvote_batchs  $$^44444r4sourcetargetc<|j||dS)a=The retriever associates a source phrase with a target phrase. When the retriever sees the source phrase, it will also consider results that are relevant to the target phrase. Args: source: text to associate to `target`. target: text to associate `source` to. N)r associate)rFr_r`s r'rbzNeuralDBRetriever.associates" &&)))))r4 text_pairsList[Tuple[str, str]]c:|j|dS)a.Given a batch of (source, target) pairs, the retriever associates each source phrase with the corresponding target phrase. Args: text_pairs: list of (source, target) text pairs. For each pair in this list, the source will be associated with the target. N)rassociate_batch)rFrcs r'rfz!NeuralDBRetriever.associate_batchs   +++++r4 run_managerr List[Document]c  d|vrd|d<|jjdd|i|}d|DS#t$r}td||d}~wwxYw)zRetrieve {top_k} contexts with your retriever for a given query Args: query: Query to submit to the model top_k: The max number of context results to retrieve. Defaults to 10. top_k rTc g|]K}t|j|j|j|j|j|j|ddLS))id upvote_idsr_metadatascorecontext) page_contentrp)r textrnror_rprqrr).0refs r' z=NeuralDBRetriever._get_relevant_documents..sn   !$!f&)n"%*$'L!$#&;;q>>       r4z"Error while retrieving documents: Nr/)rsearch Exception ValueError)rFrTrgrA referenceses r'_get_relevant_documentsz)NeuralDBRetriever._get_relevant_documentss Nf$$"$w'>>e>v>>J  &     N N NE!EEFFA M Ns(+ A AA pathc:|j|dS)zSaves a NeuralDB instance to disk. Can be loaded into memory by calling NeuralDB.from_checkpoint(path) Args: path: path on disk to save the NeuralDB instance to. N)rsave)rFr~s r'rzNeuralDBRetriever.saves  Tr4)N)rrrr)rrr)r*rr)r5r6rrrr)r9rrr)TT) r<r=r>r?r@r?rAr*rr)r<rGrrG)rTrJrUrVrr)rZr[rr)r_rJr`rJrr)rcrdrr)rTrJrgr rArrrh)r~rJrr)__name__ __module__ __qualname____doc____annotations__rr model_config staticmethodr( classmethodr3r8rr;rErDrYr^rbrfr}rr/r4r'rrs::BNNNN:L    \ &*MMMM[MB&* Y Y Y Y[ YD   X       0$$$$@3333 5 5 5 5 * * * *,,,,NNNN<r4r) __future__rr r$pathlibrtypingrrrrr r langchain_core.callbacksr langchain_core.documentsr langchain_core.retrieversr langchain_core.utilsrrrpydanticrrrr/r4r'rs"""""" ::::::::::::::::CCCCCC------333333VVVVVVVVVV********sssss sssssr4