Ng FddlmZddlZddlZddlZddlmZddlmZddl m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lm Z e!gd Z"dZ#ddZ$GddeZ%dS)) annotationsN) ConfigParser)Path)AnyCallableDictIterableListOptionalTuple)Document) Embeddings guard_import) VectorStore)Docstore)InMemoryDocstore)maximal_marginal_relevance)angular euclidean manhattanhammingdotrreturnrc tdS)z1Import annoy if available, otherwise raise error.annoyrb/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/vectorstores/annoy.pydependable_annoy_importr s   rc<eZdZdZdHd ZedIdZ dJdKdZdLdZ dMdNd$Z dMdOd&Z dMdPd(Z dMdQd*Z dMdRd+Z dMdSd,Z dTdUd2Z dTdVd3Zeded4dfdWd:Zeded4dfdXd;Zeded4dfdYd>ZdZd[dDZed?dEd\dGZdS)]Annoya`Annoy` vector store. To use, you should have the ``annoy`` python package installed. Example: .. code-block:: python from langchain_community.vectorstores import Annoy db = Annoy(embedding_function, index, docstore, index_to_docstore_id) embedding_functionrindexrmetricstrdocstorerindex_to_docstore_idDict[int, str]cL||_||_||_||_||_dS)z%Initialize with necessary components.N)r#r$r%r'r()selfr#r$r%r'r(s r__init__zAnnoy.__init__*s/#5    $8!!!rrOptional[Embeddings]cdSNr)r+s r embeddingszAnnoy.embeddings9s trNtexts Iterable[str] metadatasOptional[List[dict]]kwargs List[str]c  td)Nz=Annoy does not allow to add new data once the index is build.)NotImplementedError)r+r1r3r5s r add_textszAnnoy.add_texts>s " K   ridxs List[int]dists List[float]List[Tuple[Document, float]]cg}t||D]m\}}|j|}|j|}t |t st d|d||||fn|S)aTurns annoy results into a list of documents and scores. Args: idxs: List of indices of the documents in the index. dists: List of distances of the documents in the index. Returns: List of Documents and scores. Could not find document for id , got )zipr(r'search isinstancer ValueErrorappend)r+r:r<docsidxdist_iddocs rprocess_index_resultszAnnoy.process_index_resultsHsT5)) % %IC+C0C-&&s++Cc8,, U !S3!S!Sc!S!STTT KKd $ $ $ $ r embeddingkintsearch_kcp|j|||d\}}|||Sa}Return docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the query and score for each TrRinclude_distances)r$get_nns_by_vectorrL)r+rOrPrRr:r<s r&similarity_search_with_score_by_vectorz,Annoy.similarity_search_with_score_by_vector\sEj22 q8t3  e))$666rdocstore_indexcp|j|||d\}}|||SrT)r$get_nns_by_itemrL)r+rYrPrRr:r<s r%similarity_search_with_score_by_indexz+Annoy.similarity_search_with_score_by_indexnsEj00 AD1  e))$666rqueryc^||}||||}|S)a~Return docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the query and score for each )r#rX)r+r]rPrRrOrGs rsimilarity_search_with_scorez"Annoy.similarity_search_with_scores3++E22 ::9aRR rList[Document]c H||||}d|DS)aReturn docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the embedding. cg|]\}}|Srr.0rK_s r z5Annoy.similarity_search_by_vector..222Q222r)rX)r+rOrPrRr5docs_and_scoress rsimilarity_search_by_vectorz!Annoy.similarity_search_by_vectors6EE q(  32/2222rc H||||}d|DS)azReturn docs most similar to docstore_index. Args: docstore_index: Index of document in docstore k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the embedding. cg|]\}}|Srrrcs rrfz4Annoy.similarity_search_by_index..rgr)r\)r+rYrPrRr5rhs rsimilarity_search_by_indexz Annoy.similarity_search_by_indexs6DD Ax  32/2222rc H||||}d|DS)alReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. search_k: inspect up to search_k nodes which defaults to n_trees * n if not provided Returns: List of Documents most similar to the query. cg|]\}}|Srrrcs rrfz+Annoy.similarity_search..rgr)r_)r+r]rPrRr5rhs rsimilarity_searchzAnnoy.similarity_searchs/;;E1hOO22/2222r?fetch_k lambda_multfloatc  j||dd fd D}ttj|gtj|||} fd|D}g} |D]h} j| } j| } t| tstd| d | | | i| S) aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. fetch_k: Number of Documents to fetch to pass to MMR algorithm. k: Number of Documents to return. Defaults to 4. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. rNFrUcDg|]}j|Sr)r$get_item_vector)rdir+s rrfzAAnnoy.max_marginal_relevance_search_by_vector..s)BBBdj0033BBBr)dtype)rPrsc,g|]}|dk|S)rNr)rdrxr:s rrfzAAnnoy.max_marginal_relevance_search_by_vector..s"EEEQ"WWDGWWWrr@rA) r$rWrnparrayfloat32r(r'rCrDr rErF)r+rOrPrrrsr5r0 mmr_selectedselected_indicesrGrxrJrKr:s` @r'max_marginal_relevance_search_by_vectorz-Annoy.max_marginal_relevance_search_by_vectors2z++ wu,  CBBBTBBB 1 Hi[ 3 3 3 #    FEEE\EEE!  A+A.C-&&s++Cc8,, U !S3!S!Sc!S!STTT KK     rc b||}|||||}|S)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. )rs)r#r)r+r]rPrrrsr5rOrGs rmax_marginal_relevance_searchz#Annoy.max_marginal_relevance_searchsA0++E22 ;; q'{<   rdr0List[List[float]]rtreesn_jobsc |tvr'td|dtttd} |stdt |d} | | |} t |D]\} } | | | | ||g}t |D]5\} }|r|| ni}| t||6d tt |Dtfd t |D}||j | ||S) NzUnsupported distance metric: z. Expected one of rz/embeddings must be provided to build AnnoyIndexrr%)r) page_contentmetadatacPi|]#}|ttj$Sr)r&uuiduuid4)rdrxs r z Annoy.__from..6s(KKKq#djll++KKKrc(i|]\}}||Srr)rdrxrK index_to_ids rrz Annoy.__from..8s# D D DVQ[^S D D Dr) INDEX_METRICSrElistrlen AnnoyIndex enumerateadd_itembuildrFr ranger embed_query)clsr1r0rOr3r%rrr5rfr$rxemb documentstextrr'rs @r__fromz Annoy.__froms  & &=F=='+M':':==  W%% PNOO O  1    6 22 ++ # #FAs NN1c " " " " E& )))  '' M MGAt'08y||bH   X4(KKK L L L LKKU3y>>5J5JKKK # D D D Dy/C/C D D D  s9(%;OOOrc T||}|j|||||||fi|S)aConstruct Annoy wrapper from raw documents. Args: texts: List of documents to index. embedding: Embedding function to use. metadatas: List of metadata dictionaries to associate with documents. metric: Metric to use for indexing. Defaults to "angular". trees: Number of trees to use for indexing. Defaults to 100. n_jobs: Number of jobs to use for indexing. Defaults to -1. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the Annoy database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import Annoy from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() index = Annoy.from_texts(texts, embeddings) )embed_documents _Annoy__from) rr1rOr3r%rrr5r0s r from_textszAnnoy.from_texts<sJH..u55 sz :y)VUF  NT   rtext_embeddingsList[Tuple[str, List[float]]]c Zd|D}d|D} |j|| |||||fi|S)aConstruct Annoy wrapper from embeddings. Args: text_embeddings: List of tuples of (text, embedding) embedding: Embedding function to use. metadatas: List of metadata dictionaries to associate with documents. metric: Metric to use for indexing. Defaults to "angular". trees: Number of trees to use for indexing. Defaults to 100. n_jobs: Number of jobs to use for indexing. Defaults to -1 This is a user friendly interface that: 1. Creates an in memory docstore with provided embeddings 2. Initializes the Annoy database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import Annoy from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() text_embeddings = embeddings.embed_documents(texts) text_embedding_pairs = list(zip(texts, text_embeddings)) db = Annoy.from_embeddings(text_embedding_pairs, embeddings) cg|] }|d S)rrrdts rrfz)Annoy.from_embeddings..s///!1///rcg|] }|d S)rrs rrfz)Annoy.from_embeddings..s444qad444r)r) rrrOr3r%rrr5r1r0s rfrom_embeddingszAnnoy.from_embeddingses]J0////44O444 sz :y)VUF  NT   rF folder_pathprefaultboolNonect|}tj|dt}|jj|jd|d<|jt|dz |t|dz d5}tj |j |j |f|d d d d S#1swxYwYd S) aSave Annoy index, docstore, and index_to_docstore_id to disk. Args: folder_path: folder path to save index, docstore, and index_to_docstore_id to. prefault: Whether to pre-load the index into memory. T)exist_ok)rr%ANNOY index.annoy)r index.pklwbN)rosmakedirsrr$rr%saver&openpickledumpr'r()r+rrpath config_objectfiles r save_localzAnnoy.save_localsK   D4(((($ k" " g D=011HEEE $$d + + Yt K(A=QSW X X X Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Ys #B::B>B>)allow_dangerous_deserializationrc|stdt|}td}t|dz d5}t j|\}}} dddn #1swxYwYt | dd} | dd} || | } | t|d z ||j | | ||S) aRLoad Annoy index, docstore, and index_to_docstore_id to disk. Args: folder_path: folder path to load index, docstore, and index_to_docstore_id from. embeddings: Embeddings to use when generating queries. allow_dangerous_deserialization: whether to allow deserialization of the data which involves loading a pickle file. Pickle files can be modified by malicious actors to deliver a malicious payload that results in execution of arbitrary code on your machine. aBThe de-serialization relies loading a pickle file. Pickle files can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine.You will need to set `allow_dangerous_deserialization` to `True` to enable deserialization. If you do this, make sure that you trust the source of the data. For example, if you are loading a file that you created, and know that no one else has modified the file, then this is safe to do. Do not set this to `True` if you are loading a file from an untrusted source (e.g., some random site on the internet.).rrrbNrrr%rr) rErrrrloadrQrr&r) rrr0rrrrr'r(rrr%r$s r load_localzAnnoy.load_localsJ(/  "   K  W%% $$d + + t    $                  g&s+ , ,w'1  6 22 3tm+,,---s  "E68=Q   sA((A,/A,) r#rr$rr%r&r'rr(r))rr-r/)r1r2r3r4r5rrr6)r:r;r<r=rr>)rMrN)rOr=rPrQrRrQrr>)rYrQrPrQrRrQrr>)r]r&rPrQrRrQrr>) rOr=rPrQrRrQr5rrr`) rYrQrPrQrRrQr5rrr`) r]r&rPrQrRrQr5rrr`)rMrprq) rOr=rPrQrrrQrsrtr5rrr`) r]r&rPrQrrrQrsrtr5rrr`)r1r6r0rrOrr3r4r%r&rrQrrQr5rrr")r1r6rOrr3r4r%r&rrQrrQr5rrr")rrrOrr3r4r%r&rrQrrQr5rrr")F)rr&rrrr)rr&r0rrrrr")__name__ __module__ __qualname____doc__r,propertyr0r9rLrXr\r_rirlrorr classmethodDEFAULT_METRICrrrrrrrrr"r"sw   9 9 9 9X+/     *CE77777&@B77777&79&CE33333(@B33333(7933333( -----d < +/$#P#P#P#P[#PJ +/$& & & & [& P +/$) ) ) ) [) VYYYYY( 16 6 6 6 6 6 [6 6 6 rr")rr)& __future__rrrr configparserrpathlibrtypingrrrr r r r numpyr{langchain_core.documentsr langchain_core.embeddingsrlangchain_core.utilsrlangchain_core.vectorstoresr!langchain_community.docstore.baser&langchain_community.docstore.in_memoryr&langchain_community.vectorstores.utilsr frozensetrrr r"rrrrs"""""" %%%%%%GGGGGGGGGGGGGGGGGG------000000------333333666666CCCCCCMMMMMM QQQRR !!!!      K     r