Ng|,ddlmZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z mZmZmZmZmZmZddlZddlmZddlmZddlmZddlmZdd lmZm Z dd l!m"Z"dd l#m$Z$m%Z%ej&e'Z(dddZ)ddZ*GddeZ+dS)) annotationsN)Path) AnyCallableDictIterableListOptionalSizedTupleUnionDocument) Embeddings)run_in_executor) VectorStore) AddableMixinDocstore)InMemoryDocstore)DistanceStrategymaximal_marginal_relevanceno_avx2Optional[bool]returnrc|/dtjvr!ttjd} |rddlm}nddl}n#t $rt dwxYw|S)aM Import faiss if available, otherwise raise error. If FAISS_NO_AVX2 environment variable is set, it will be considered to load FAISS with no AVX2 optimization. Args: no_avx2: Load FAISS strictly with no AVX2 optimization so that the vectorstore is portable and compatible with other devices. N FAISS_NO_AVX2r) swigfaisszCould not import faiss python package. Please install it with `pip install faiss-gpu` (for CUDA supported GPU) or `pip install faiss-cpu` (depending on Python version).)osenvironboolgetenvfaissr ImportError)rr"s b/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/vectorstores/faiss.pydependable_faiss_importr%&s?bj88ry1122    0 0 0 0 0 0 0 LLL     H    Ls AAxyx_namestry_nameNonect|trot|trZt|t|kr:t|d|d|dt|d|dt| dS)Nz and z% expected to be equal length but len(z)=z and len() isinstancer len ValueError)r&r'r(r*s r$_len_check_if_sizedr0As!U 1e 4 4 Q3q669I9I A AF A A A A VV A A.4 A A8;A A A    FceZdZdZddejfdidZedjdZdkdZ dkdZ dldZ dldZ dmdnd'Z dmdod*Z dmdod+Z dmdpd.Z dqdrd8Z dqdrd9Z dqdsd;Z dqdsd<Z dqdtd?Z dqdud@Z dqdvdAZ dqdvdBZd/d0dCddDdwdGZd/d0dCddDdwdHZ dxdydIZ dxdydJZ dxdzdKZ dxdzdLZd{d|dNZd}dQZ e!dddejfd~dSZ"e! dmddTZ#e! dmddVZ$e! dmddWZ%e! dmddXZ&ddd[Z'e! ddd\dd^Z(dd`Z)e!dd\ddbZ*dddZ+ dqdsdeZ, dqdsdfZ-e.ddhZ/dS)FAISSuFAISS vector store integration. See [The FAISS Library](https://arxiv.org/pdf/2401.08281) paper. Setup: Install ``langchain_community`` and ``faiss-cpu`` python packages. .. code-block:: bash pip install -qU langchain_community faiss-cpu Key init args — indexing params: embedding_function: Embeddings Embedding function to use. Key init args — client params: index: Any FAISS index to use. docstore: Docstore Docstore to use. index_to_docstore_id: Dict[int, str] Mapping of index to docstore id. Instantiate: .. code-block:: python import faiss from langchain_community.vectorstores import FAISS from langchain_community.docstore.in_memory import InMemoryDocstore from langchain_openai import OpenAIEmbeddings index = faiss.IndexFlatL2(len(OpenAIEmbeddings().embed_query("hello world"))) vector_store = FAISS( embedding_function=OpenAIEmbeddings(), index=index, docstore= InMemoryDocstore(), index_to_docstore_id={} ) Add Documents: .. code-block:: python from langchain_core.documents import Document document_1 = Document(page_content="foo", metadata={"baz": "bar"}) document_2 = Document(page_content="thud", metadata={"bar": "baz"}) document_3 = Document(page_content="i will be deleted :(") documents = [document_1, document_2, document_3] ids = ["1", "2", "3"] vector_store.add_documents(documents=documents, ids=ids) Delete Documents: .. code-block:: python vector_store.delete(ids=["3"]) Search: .. code-block:: python results = vector_store.similarity_search(query="thud",k=1) for doc in results: print(f"* {doc.page_content} [{doc.metadata}]") .. code-block:: python * thud [{'bar': 'baz'}] Search with filter: .. code-block:: python results = vector_store.similarity_search(query="thud",k=1,filter={"bar": "baz"}) for doc in results: print(f"* {doc.page_content} [{doc.metadata}]") .. code-block:: python * thud [{'bar': 'baz'}] Search with score: .. code-block:: python results = vector_store.similarity_search_with_score(query="qux",k=1) for doc, score in results: print(f"* [SIM={score:3f}] {doc.page_content} [{doc.metadata}]") .. code-block:: python * [SIM=0.335304] foo [{'baz': 'bar'}] Async: .. code-block:: python # add documents # await vector_store.aadd_documents(documents=documents, ids=ids) # delete documents # await vector_store.adelete(ids=["3"]) # search # results = vector_store.asimilarity_search(query="thud",k=1) # search with score results = await vector_store.asimilarity_search_with_score(query="qux",k=1) for doc,score in results: print(f"* [SIM={score:3f}] {doc.page_content} [{doc.metadata}]") .. code-block:: python * [SIM=0.335304] foo [{'baz': 'bar'}] Use as Retriever: .. code-block:: python retriever = vector_store.as_retriever( search_type="mmr", search_kwargs={"k": 1, "fetch_k": 2, "lambda_mult": 0.5}, ) retriever.invoke("thud") .. code-block:: python [Document(metadata={'bar': 'baz'}, page_content='thud')] NFembedding_function/Union[Callable[[str], List[float]], Embeddings]indexrdocstorerindex_to_docstore_idDict[int, str]relevance_score_fn"Optional[Callable[[float], float]] normalize_L2r distance_strategyrc>t|tstd||_||_||_||_||_||_ ||_ |jtj kr%|j r tjd|jdSdSdS)z%Initialize with necessary components.t`embedding_function` is expected to be an Embeddings object, support for passing in a function will soon be removed.z2Normalizing L2 is not applicable for metric type: N)r-rloggerwarningr4r6r7r8r=override_relevance_score_fn _normalize_L2rEUCLIDEAN_DISTANCEwarningswarn)selfr4r6r7r8r:r<r=s r$__init__zFAISS.__init__s,j99  NNB   #5   $8!!2+=()  "&6&I I I" J M9 $ 699      J I I Ir1rOptional[Embeddings]cHt|jtr|jndSN)r-r4rrGs r$ embeddingszFAISS.embeddingss)$1:>> D # # r1texts List[str]List[List[float]]ctjtrj|Sfd|DS)Nc:g|]}|S)r4).0textrGs r$ z*FAISS._embed_documents..s'DDDdD++D11DDDr1)r-r4rembed_documentsrGrNs` r$_embed_documentszFAISS._embed_documentssJ d-z : : E*::5AA ADDDDeDDD Dr1cKt|jtr |j|d{VSt dNr?)r-r4raembed_documents ExceptionrXs r$_aembed_documentszFAISS._aembed_documentss^ d-z : : 0AA%HHHHHHHH H B r1rUr) List[float]ct|jtr|j|S||SrK)r-r4r embed_queryrGrUs r$ _embed_queryzFAISS._embed_querysB d-z : : 1*66t<< <**400 0r1cKt|jtr |j|d{VSt dr[)r-r4r aembed_queryr]rbs r$ _aembed_queryzFAISS._aembed_query s^ d-z : : 0==dCCCCCCCC CB r1 Iterable[str]rMIterable[List[float]] metadatasOptional[Iterable[dict]]idsOptional[List[str]]c~ t}t|jtst d|jdt ||dd|p d|D}dt ||D}t ||ddt ||dd |rzFAISS.__add..'s""5"5!2"5"5"5"5"5"5r1c6g|]\}}t||S)) page_contentmetadatar)rTtms r$rVzFAISS.__add..(s6   59QH!a 0 0 0   r1 documentsrMrkz$Duplicate ids found in the ids list.dtypecNg|]"}ttj#SrS)r)uuiduuid4ros r$rVzFAISS.__add..9s&777Ac$*,,''777r1ci|]\}}|| SrSrS)rTid_docs r$ zFAISS.__add..:sHHHS3HHHr1c"i|] \}}|z| SrSrS)rTjr~ starting_lens r$rzFAISS.__add..<s$JJJC|a'JJJr1)r%r-r7rr/r0zipr.setnparrayfloat32rCr<r6addr8 enumerateupdate) rGrNrMrirkr" _metadatasrwvector index_to_idrs @r$__addz FAISS.__adds())$-66 @'+}@@@  E9g{CCC5"5"5u"5"5"5   =@ =S=S    Iz; MMMIsK???  E3s88s3s88}},,CDD D*BJ777   '   v & & & v777777 HHCY4G4GHHHIII4455 JJJJ9S>>JJJ  !((555 r1Optional[List[dict]]kwargsc |t|}||}|||||S)alRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of unique IDs. Returns: List of ids from adding the texts into the vectorstore. rirk)listrY _FAISS__addrGrNrirkrrMs r$ add_textszFAISS.add_texts@s<"U **511 zz%yczJJJr1cKt|}||d{V}|||||S)aRun more texts through the embeddings and add to the vectorstore asynchronously. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of unique IDs. Returns: List of ids from adding the texts into the vectorstore. Nr)rr^rrs r$ aadd_textszFAISS.aadd_textsUsR$U 11%88888888 zz%yczJJJr1text_embeddings!Iterable[Tuple[str, List[float]]]c Lt|\}}|||||S)aAdd the given texts and embeddings to the vectorstore. Args: text_embeddings: Iterable pairs of string and embedding to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of unique IDs. Returns: List of ids from adding the texts into the vectorstore. r)rr)rGrrirkrrNrMs r$add_embeddingszFAISS.add_embeddingsks-& 1zzz%yczJJJr1 embeddingkintfilter)Optional[Union[Callable, Dict[str, Any]]]fetch_kList[Tuple[Document, float]]c xt}tj|gtj}|jr|||j|||n|\}} g} |||} t| dD]\} } | dkr |j | }|j |}t|tstd|d||4| |jr#| ||d| f| ||d| f|dF|jt&jt&jfvr t,jn t,jfd| D} | d|S) aReturn docs most similar to query. Args: embedding: Embedding vector to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Union[Callable, Dict[str, Any]]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. **kwargs: kwargs to be passed to similarity search. Can include: score_threshold: Optional, a floating point value between 0 to 1 to filter the resulting set of retrieved docs Returns: List of documents most similar to the query text and L2 distance in float for each. Lower score represents more similarity. rxNrCould not find document for id , got score_thresholdc6g|]\}}|||fSrSrS)rTr similaritycmprs r$rVz@FAISS.similarity_search_with_score_by_vector..sD#C3z?33j!r1)r%rrrrCr<r6search_create_filter_funcrr8r7r-rr/rtappendgetr=rMAX_INNER_PRODUCTJACCARDoperatorgele)rGrrrrrr"rscoresindicesdocs filter_funcri_idrrrs @@r$&similarity_search_with_score_by_vectorz,FAISS.similarity_search_with_score_by_vectors4())9+RZ888   '   v & & &*++FAAWUU  226::Kgaj)) 1 1DAqBww+A.C-&&s++Cc8,, U !S3!S!Sc!S!STTT!;s|,,5KKfQil 3444 S&)A,/0000 **%677  &)$68H8PQRR [  '+D BQBxr1cBKtd|j|f|||d|d{VS)aReturn docs most similar to query asynchronously. Args: embedding: Embedding vector to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, Any]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. **kwargs: kwargs to be passed to similarity search. Can include: score_threshold: Optional, a floating point value between 0 to 1 to filter the resulting set of retrieved docs Returns: List of documents most similar to the query text and L2 distance in float for each. Lower score represents more similarity. Nrrr)rr)rGrrrrrs r$'asimilarity_search_with_score_by_vectorz-FAISS.asimilarity_search_with_score_by_vectorsc:%   7              r1queryc T||}|j||f||d|}|S)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of documents most similar to the query text with L2 distance in float. Lower score represents more similarity. rr)rcrrGrrrrrrrs r$similarity_search_with_scorez"FAISS.similarity_search_with_scoresT0%%e,, :t:          r1cpK||d{V}|j||f||d|d{V}|S)aReturn docs most similar to query asynchronously. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of documents most similar to the query text with L2 distance in float. Lower score represents more similarity. Nr)rfrrs r$asimilarity_search_with_scorez#FAISS.asimilarity_search_with_score s0,,U33333333 ATA                r1Optional[Dict[str, Any]]List[Document]c >|j||f||d|}d|DS)aYReturn docs most similar to embedding vector. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the embedding. rcg|]\}}|SrSrSrTrrps r$rVz5FAISS.similarity_search_by_vector..K222Q222r1)rrGrrrrrdocs_and_scoress r$similarity_search_by_vectorz!FAISS.similarity_search_by_vector-sQ.F$E         32/2222r1cNK|j||f||d|d{V}d|DS)ahReturn docs most similar to embedding vector asynchronously. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. If a callable, it must take as input the metadata dict of Document and return a bool. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the embedding. rNcg|]\}}|SrSrSrs r$rVz6FAISS.asimilarity_search_by_vector..krr1)rrs r$asimilarity_search_by_vectorz"FAISS.asimilarity_search_by_vectorMss.!M L  !  ! !  ! !       32/2222r1c >|j||f||d|}d|DS)aReturn docs most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the query. rcg|]\}}|SrSrSrs r$rVz+FAISS.similarity_search..rr1)rrGrrrrrrs r$similarity_searchzFAISS.similarity_searchmsJ(<$; 1 #W  8>  32/2222r1cNK|j||f||d|d{V}d|DS)aReturn docs most similar to query asynchronously. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: (Optional[Dict[str, str]]): Filter by metadata. Defaults to None. fetch_k: (Optional[int]) Number of Documents to fetch before filtering. Defaults to 20. Returns: List of Documents most similar to the query. rNcg|]\}}|SrSrSrs r$rVz,FAISS.asimilarity_search..rr1)rrs r$asimilarity_searchzFAISS.asimilarity_searchsm(!C B 1! #W! ! 8>! !       32/2222r1?rr lambda_multrrfloatcjtj|gtj||n|dz\}}||}g} |dD]} | dkr j| } j| } t| tstd| d| || j r| | tj| g}fd|dD} ttj|gtj| || }g}|D]} |d| dkrj|d| } j| } t| tstd| d| | | |d| f|S) azReturn docs and their similarity scores selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents and similarity scores selected by maximal marginal relevance and score for each. rxNrrrrcjg|]/}|dkjt|0S)r)r6 reconstructr)rTrrGs r$rVzLFAISS.max_marginal_relevance_search_with_score_by_vector..s5TTTAQSGGdj,,SVV44GGGr1)rr)r6rrrrrr8r7r-rr/rtrr)rGrrrrrrrrfiltered_indicesrrrrM mmr_selectedrs` r$2max_marginal_relevance_search_with_score_by_vectorz8FAISS.max_marginal_relevance_search_with_score_by_vectors8*++ Hi[ 3 3 3~GG7Q;    226::K! QZ / /77/2m**3//!#x00Y$%Ws%W%WRU%W%WXXX;s|,,/$++A...h 0122GTTTTgajTTT 1 Hi[ 3 3 3 #     8 8Aqz!}""+GAJqM:C-&&s++Cc8,, U !S3!S!Sc!S!STTT  " "C1#6 7 7 7 7r1c HKtd|j|||||d{VS)aReturn docs and their similarity scores selected using the maximal marginal relevance asynchronously. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents and similarity scores selected by maximal marginal relevance and score for each. Nr)rr)rGrrrrrs r$3amax_marginal_relevance_search_with_score_by_vectorz9FAISS.amax_marginal_relevance_search_with_score_by_vectorsU:%   C #          r1c N||||||}d|DS)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. rcg|]\}}|SrSrSrs r$rVzAFAISS.max_marginal_relevance_search_by_vector..'rr1)rrGrrrrrrrs r$'max_marginal_relevance_search_by_vectorz-FAISS.max_marginal_relevance_search_by_vector s@4QQ GVR  32/2222r1c^K||||||d{V}d|DS)a(Return docs selected using the maximal marginal relevance asynchronously. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. rNcg|]\}}|SrSrSrs r$rVzBFAISS.amax_marginal_relevance_search_by_vector..Hrr1)rrs r$(amax_marginal_relevance_search_by_vectorz.FAISS.amax_marginal_relevance_search_by_vector)sf6JJQ[QWK        32/2222r1c V||}|j|f||||d|}|S)aReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering (if needed) to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. r)rcr rGrrrrrrrrs r$max_marginal_relevance_searchz#FAISS.max_marginal_relevance_searchJsU4%%e,, ;t;  #       r1crK||d{V}|j|f||||d|d{V}|S)a+Return docs selected using the maximal marginal relevance asynchronously. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch before filtering (if needed) to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. Returns: List of Documents selected by maximal marginal relevance. Nr)rfrrs r$amax_marginal_relevance_searchz$FAISS.amax_marginal_relevance_searchos4,,U33333333 BTB  #             r1rc l|tdt||j}|rtd|d|jDfd|D|jtj tj |j |fdt|jD}dt|D|_d S) zDelete by ID. These are the IDs in the vectorstore. Args: ids: List of ids to delete. Returns: Optional[bool]: True if deletion is successful, False otherwise, None if not implemented. NzNo ids provided to delete.zESome specified ids do not exist in the current store. Ids not found: ci|]\}}|| SrSrS)rTidxr~s r$rz FAISS.delete..sUUUxsC#sUUUr1c h|] }| SrSrS)rTr~reversed_indexs r$ zFAISS.delete..s>>>3>#.>>>r1rxc"g|] \}}|v | SrSrS)rTrr~index_to_deletes r$rVz FAISS.delete..s2   3'' '''r1ci|]\}}|| SrSrS)rTrr~s r$rz FAISS.delete..s$S$S$S3Q$S$S$Sr1T)r/r differencer8valuesitemsr6 remove_idsrfromiterint64r7deletesortedr)rGrkr missing_ids remaining_idsrrs @@r$rz FAISS.deletesN ;9:: :#hh))$*C*J*J*L*LMM  !!!  VU43L3R3R3T3TUUU>>>>#>>> bk/JJJKKK S!!!     !:!@!@!B!BCC   %T$S)M:R:R$S$S$S!tr1targetr+cFt|jtstdt |j}|j|jg}|jD]^\}}|j |}t|tstd| ||z||f_|j d|Dd|D}|j |dS)zMerge another FAISS object with the current one. Add the target FAISS to the current one. Args: target: FAISS object you wish to merge into the current one Returns: None. z'Cannot merge with this type of docstorezDocument should be returnedci|] \}}}|| SrSrS)rTrprrs r$rz$FAISS.merge_from..s AAA 33AAAr1ci|] \}}}|| SrSrS)rTr6rrps r$rz$FAISS.merge_from..s AAAmeS!ucAAAr1N)r-r7rr/r.r8r6 merge_fromrrrrrr)rGrr full_infor target_idrrs r$r zFAISS.merge_froms.$-66 HFGG G4455  fl+++ "7==?? A ALAy/((33Cc8,, @ !>???   lQ. 3? @ @ @ @ AAyAAABBBAAyAAA  !((55555r1rc t} |tjkr)| t |d} n(| t |d} |dt} |di} ||| | | f||d|} | ||||| S)Nrr7r8)r<r=r) r%rr IndexFlatIPr. IndexFlatL2poprr)clsrNrMrrirkr<r=rr"r6r7r8vecstores r$__fromz FAISS.__froms())  0 B B B%%c*Q-&8&899EE%%c*Q-&8&899E::j*:*<*<==%zz*@"EE3     &/      ujI3GGGr1c R||}|j|||f||d|S)aOConstruct FAISS wrapper from raw documents. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the FAISS database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import FAISS from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() faiss = FAISS.from_texts(texts, embeddings) r)rW _FAISS__fromrrNrrirkrrMs r$ from_textszFAISS.from_textssR6..u55 sz            r1 list[str]cbK||d{V}|j|||f||d|S)aeConstruct FAISS wrapper from raw documents asynchronously. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the FAISS database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import FAISS from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() faiss = await FAISS.afrom_texts(texts, embeddings) Nr)r\rrs r$ afrom_textszFAISS.afrom_textssh6%55e<<<<<<<< sz            r1c tt|\}}|jt|t||f||d|S)aConstruct FAISS wrapper from raw documents. This is a user friendly interface that: 1. Embeds documents. 2. Creates an in memory docstore 3. Initializes the FAISS database This is intended to be a quick way to get started. Example: .. code-block:: python from langchain_community.vectorstores import FAISS from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() text_embeddings = embeddings.embed_documents(texts) text_embedding_pairs = zip(texts, text_embeddings) faiss = FAISS.from_embeddings(text_embedding_pairs, embeddings) r)rrr)rrrrirkrrNrMs r$from_embeddingszFAISS.from_embeddings@s\: 1zsz KK             r1c*K|j||f||d|S)z:Construct FAISS wrapper from raw documents asynchronously.r)r)rrrrirkrs r$afrom_embeddingszFAISS.afrom_embeddingsgsA#s"           r1 folder_path index_namecjt|}|ddt}||jt ||dz t ||dz d5}tj|j |j f|ddddS#1swxYwYdS)aSave FAISS index, docstore, and index_to_docstore_id to disk. Args: folder_path: folder path to save index, docstore, and index_to_docstore_id to. index_name: for saving with a specific index file name T)exist_okparents.faiss.pklwbN) rmkdirr% write_indexr6r)openpickledumpr7r8)rGr!r"pathr"fs r$ save_localzFAISS.save_localysK   D$ ///()) $*c$J1F1F1F*F&G&GHHH$J,,,,d 3 3 Gq K(ABA F F F G G G G G G G G G G G G G G G G G Gs9"B((B,/B,)allow_dangerous_deserializationr1c P|stdt|}t}|t ||dz }t ||dz d5} t j| \} } dddn #1swxYwY|||| | fi|S)aLoad FAISS index, docstore, and index_to_docstore_id from disk. Args: folder_path: folder path to load index, docstore, and index_to_docstore_id from. embeddings: Embeddings to use when generating queries index_name: for saving with a specific index file name allow_dangerous_deserialization: whether to allow deserialization of the data which involves loading a pickle file. Pickle files can be modified by malicious actors to deliver a malicious payload that results in execution of arbitrary code on your machine. BThe de-serialization relies loading a pickle file. Pickle files can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine.You will need to set `allow_dangerous_deserialization` to `True` to enable deserialization. If you do this, make sure that you trust the source of the data. For example, if you are loading a file that you created, and know that no one else has modified the file, then this is safe to do. Do not set this to `True` if you are loading a file from an untrusted source (e.g., some random site on the internet.).r&r'rbN)r/rr% read_indexr)r+r,load) rr!rMr"r1rr.r"r6r/r7r8s r$ load_localzFAISS.load_locals./  "   K  '))  Tz,A,A,A%A!B!BCC$J,,,,d 3 3 q  $               s:uh0DOOOOOs-BBBbytescNtj|j|j|jfS)zCSerialize FAISS index, docstore, and index_to_docstore_id to bytes.)r,dumpsr6r7r8rLs r$serialize_to_byteszFAISS.serialize_to_bytess |TZ8QRSSSr1 serializedc l|stdtj|\}}}|||||fi|S)zGDeserialize FAISS index, docstore, and index_to_docstore_id from bytes.r3)r/r,loads)rr<rMr1rr6r7r8s r$deserialize_from_byteszFAISS.deserialize_from_bytessf/  "    L      s:uh0DOOOOOr1Callable[[float], float]c|j|jS|jtjkr|jS|jtjkr|jS|jtjkr|jStd)a8 The 'correct' relevance function may differ depending on a few things, including: - the distance / similarity metric used by the VectorStore - the scale of your embeddings (OpenAI's are unit normed. Many others are not!) - embedding dimensionality - etc. NzJUnknown distance strategy, must be cosine, max_inner_product, or euclidean) rBr=rr%_max_inner_product_relevance_score_fnrD_euclidean_relevance_score_fnCOSINE_cosine_relevance_score_fnr/rLs r$_select_relevance_score_fnz FAISS._select_relevance_score_fns  + 73 3  !%5%G G G= =  #'7'J J J5 5  #'7'> > >2 2  r1c |td|j|f|||d|}fd|D}|S)?Return docs and their similarity scores on a scale from 0 to 1.NLrelevance_score_fn must be provided to FAISS constructor to normalize scoresrc0g|]\}}||fSrSrSrTrscorer:s r$rVzBFAISS._similarity_search_with_relevance_scores..;   1;eS$$U++ ,   r1)rFr/r rGrrrrrrdocs_and_rel_scoresr:s @r$(_similarity_search_with_relevance_scoresz.FAISS._similarity_search_with_relevance_scoress"<<>>  %9 <$;            ?N   #"r1cK|td|j|f|||d|d{V}fd|D}|S)rHNrIrc0g|]\}}||fSrSrSrKs r$rVzCFAISS._asimilarity_search_with_relevance_scores..4rMr1)rFr/rrNs @r$)_asimilarity_search_with_relevance_scoresz/FAISS._asimilarity_search_with_relevance_scoress"<<>>  %9 !C B !  ! !  ! !           ?N   #"r1 Callable[[Dict[str, Any]], bool]ctrSttstdt dfd }|S)a Create a filter function based on the provided filter. Args: filter: A callable or a dictionary representing the filter conditions for documents. Returns: Callable[[Dict[str, Any]], bool]: A function that takes Document's metadata and returns True if it satisfies the filter conditions, otherwise False. z5filter must be a dict of metadata or a callable, not rtDict[str, Any]rr c`tfdDS)Nc3K|]L\}}t|tr||vn||kVMdSrK)r-rr)rTkeyvaluerts r$rqzAFAISS._create_filter_func..filter_func..QsvCeT**0 S!!U**\\#&&%/r1)allr)rtrs`r$rz.FAISS._create_filter_func..filter_funcPsF#),,..  r1)rtrVrr )callabler-dictr/type)rrs` r$rzFAISS._create_filter_func9su F   M&$'' VV VV       r1)r4r5r6rr7rr8r9r:r;r<r r=r)rrI)rNrOrrP)rUr)rr_)NN) rNrgrMrhrirjrkrlrrO) rNrgrirrkrlrrrrO) rrrirrkrlrrrrO)rNr) rr_rrrrrrrrrr) rr)rrrrrrrrrr) rr_rrrrrrrrrr) rr_rrrrrrrrrr) rr)rrrrrrrrrr) rr_rrrrrrrrrr)rrrN)rr_rrrrrrrrrrrr)rr)rrrrrrrrrrrrrK)rkrlrrrr)rr3rr+)rNrgrMrPrrrirjrkrlr<r r=rrrrr3) rNrOrrrirrkrlrrrr3) rNrrrrirrkrlrrrr3) rrrrrirjrkrlrrrr3)r6)r!r)r"r)rr+) r!r)rMrr"r)r1r rrrr3)rr8) r<r8rMrr1r rrrr3)rr@)rrrrT)0__name__ __module__ __qualname____doc__rrDrHpropertyrMrYr^rcrfrrrrrrrrrrrrrrrrrrrr  classmethodrrrrr r0r7r;r?rFrPrS staticmethodrrSr1r$r3r3Js}}PBF".>.Q!!!!!F   X EEEE    1111 /3#' '''''X+/#' KKKKK0+/#' KKKKK2+/#' KKKKK2<@ ?????H<@ % % % % % T<@     J<@     J+/ 33333F<@ 33333F<@ 333338<@ 33333: <@BBBBBBP <@% % % % % % T <@ 33333D <@ 33333H <@ #####P <@ #####J     D6666@ /3#'".>.Q[> +/#' " " " " [" H +/#' " " " " [" H /3#' $ $ $ $ [$ L /3#'     [ "GGGGG& " 1P 16 1P1P1P1P1P[1PfTTTT 16 PPPPP[P><<@ #####@<@ #####:\r1r3rK)rrrr) r&rr'rr(r)r*r)rr+), __future__rloggingrrr,r{rEpathlibrtypingrrrrr r r r r numpyrlangchain_core.documentsrlangchain_core.embeddingsrlangchain_core.runnables.configrlangchain_core.vectorstoresr!langchain_community.docstore.baserr&langchain_community.docstore.in_memoryr&langchain_community.vectorstores.utilsrr getLoggerr_r@r%r0r3rSr1r$rss""""""                       ------000000;;;;;;333333DDDDDDDDCCCCCC  8 $ $6    NNNNNKNNNNNr1