NgddlmZddlZddlZddlmZmZmZmZm Z m Z m Z m Z ddl ZddlmZddlmZddlmZddlmZddlmZerdd lmZmZdd lmZejeZ Gd d eZ!dS) ) annotationsN) TYPE_CHECKINGAnyIterableListOptionalTupleUnioncast)Document) Embeddings) batch_iterate) VectorStore)maximal_marginal_relevance AsyncIndexIndex) InfoResultceZdZdZ dUdddVdZedWdZdXdZdYdZ dZddd[d'Z dZddd\d)Z d]ddd^d,Z d]ddd^d-Z d_ddd`d3Z d_ddd`d4Zdad7Z d_dddbd8Z d_dddbd9Z d_dddcd:Z d_dddcd;Z d_ddddd<Z d_ddddd=Z d_ddd`d>Z d_ddd`d?Z dedddfdEZ dedddfdFZ dedddgdGZ dedddgdHZe dhdddidJZe dhdddidKZ djdddkdPZ djdddkdQZ dldSZ!dldTZ"dS)mUpstashVectorStorea/Upstash Vector vector store To use, the ``upstash-vector`` python package must be installed. Also an Upstash Vector index is required. First create a new Upstash Vector index and copy the `index_url` and `index_token` variables. Then either pass them through the constructor or set the environment variables `UPSTASH_VECTOR_REST_URL` and `UPSTASH_VECTOR_REST_TOKEN`. Example: .. code-block:: python from langchain_openai import OpenAIEmbeddings from langchain_community.vectorstores import UpstashVectorStore embeddings = OpenAIEmbeddings(model="text-embedding-3-large") vectorstore = UpstashVectorStore( embedding=embeddings, index_url="...", index_token="..." ) # or import os os.environ["UPSTASH_VECTOR_REST_URL"] = "..." os.environ["UPSTASH_VECTOR_REST_TOKEN"] = "..." vectorstore = UpstashVectorStore( embedding=embeddings ) textN namespacetext_keystrindexOptional[Index] async_indexOptional[AsyncIndex] index_url Optional[str] index_token embedding!Optional[Union[Embeddings, bool]]rc ddlm}m} n#t$rtdwxYw|rPt || st dt |||_t d|rPt ||st dt |||_ t d|rA|r?| |||_||||_ t d nF|sD|sB| j |_|j |_ t d ||_ ||_ ||_d S) a Constructor for UpstashVectorStore. If index or index_url and index_token are not provided, the constructor will attempt to create an index using the environment variables `UPSTASH_VECTOR_REST_URL`and `UPSTASH_VECTOR_REST_TOKEN`. Args: text_key: Key to store the text in metadata. index: UpstashVector Index object. async_index: UpstashVector AsyncIndex object, provide only if async functions are needed index_url: URL of the UpstashVector index. index_token: Token of the UpstashVector index. embedding: Embeddings object or a boolean. When false, no embedding is applied. If true, Upstash embeddings are used. When Upstash embeddings are used, text is sent directly to Upstash and embedding is applied there instead of embedding in Langchain. namespace: Namespace to use from the index. Example: .. code-block:: python from langchain_community.vectorstores.upstash import UpstashVectorStore from langchain_community.embeddings.openai import OpenAIEmbeddings embeddings = OpenAIEmbeddings() vectorstore = UpstashVectorStore( embedding=embeddings, index_url="...", index_token="...", namespace="..." ) # With an existing index from upstash_vector import Index index = Index(url="...", token="...") vectorstore = UpstashVectorStore( embedding=embeddings, index=index, namespace="..." ) rrzdCould not import upstash_vector python package. Please install it with `pip install upstash_vector`.zGPassed index object should be an instance of upstash_vector.Index, got z#Using the index passed as parameterzLPassed index object should be an instance of upstash_vector.AsyncIndex, got z)Using the async index passed as parameter)urltokenz;Created index from the index_url and index_token parametersz)Created index using environment variablesN)upstash_vectorrr ImportError isinstance ValueErrortype_indexloggerinfo _async_indexfrom_env _embeddings _text_key _namespace) selfrrrr!r#r$rrrs d/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/vectorstores/upstash.py__init__zUpstashVectorStore.__init__;sp  8 8 8 8 8 8 8 8 8   G    ?eU++  );;))  DK KK= > > >  Ek:66  / ,,// !,D  KKC D D D  E E%I[AAADK * y L L LD  KKU V V V V E{ E(%.**DK 3 3 5 5D  KKC D D D$!#s %returnc|jS)z/Access the query embedding object if available.)r3r6s r7 embeddingszUpstashVectorStore.embeddingss texts Iterable[str]#Union[List[List[float]], List[str]]c|jstdt|jtr'|jt |St |S)z)Embed strings using the embeddings objectLNo embeddings object provided. Pass an embeddings object to the constructor.)r3r,r+r embed_documentslist)r6r>s r7_embed_documentsz#UpstashVectorStore._embed_documentssj @  d& 3 3 A#33DKK@@ @E{{r=Union[List[float], str]c|jstdt|jtr|j|S|S)z-Embed query text using the embeddings object.rB)r3r,r+r embed_query)r6rs r7 _embed_queryzUpstashVectorStore._embed_querysZ @  d& 3 3 6#//55 5 r=  documentsList[Document]idsOptional[List[str]] batch_sizeintembedding_chunk_sizekwargsr List[str]c Zd|D}d|D}|j|f|||||d|S) Get the embeddings for the documents and add them to the vectorstore. Documents are sent to the embeddings object in batches of size `embedding_chunk_size`. The embeddings are then upserted into the vectorstore in batches of size `batch_size`. Args: documents: Iterable of Documents to add to the vectorstore. batch_size: Batch size to use when upserting the embeddings. Upstash supports at max 1000 vectors per request. embedding_batch_size: Chunk size to use when embedding the texts. namespace: Namespace to use from the index. Returns: List of ids from adding the texts into the vectorstore. cg|] }|j S page_content.0docs r7 z4UpstashVectorStore.add_documents..777c!777r=cg|] }|j SrXmetadatar[s r7r^z4UpstashVectorStore.add_documents..777cS\777r=) metadatasrPrNrRr add_texts r6rLrNrPrRrrSr>rds r7 add_documentsz UpstashVectorStore.add_documentssf:87Y77777Y777 t~  !!5      r=Iterable[Document]c jKd|D}d|D}|j|f|||||d|d{VS)rVcg|] }|j SrXrYr[s r7r^z5UpstashVectorStore.aadd_documents..r_r=cg|] }|j SrXrar[s r7r^z5UpstashVectorStore.aadd_documents..rcr=rdrNrPrRrN aadd_textsrgs r7aadd_documentsz!UpstashVectorStore.aadd_documentss:87Y77777Y777 $T_  !!5            r=rdOptional[List[dict]]c ||j}t|}|p d|D}|r d|D}n d|D}t||D]\}} | ||j<t dt ||D]} || | |z} || | |z} || | |z} || }t|t| || D]*}|jj d|tt|d|+|S)c Get the embeddings for the texts and add them to the vectorstore. Texts are sent to the embeddings object in batches of size `embedding_chunk_size`. The embeddings are then upserted into the vectorstore in batches of size `batch_size`. Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. ids: Optional list of ids to associate with the texts. batch_size: Batch size to use when upserting the embeddings. Upstash supports at max 1000 vectors per request. embedding_batch_size: Chunk size to use when embedding the texts. namespace: Namespace to use from the index. Returns: List of ids from adding the texts into the vectorstore. NcNg|]"}ttj#SrXruuiduuid4r\_s r7r^z0UpstashVectorStore.add_texts..6&777Ac$*,,''777r=c6g|]}|SrXcopyr\ms r7r^z0UpstashVectorStore.add_texts..: 555a555r=cg|]}iSrXrXrxs r7r^z0UpstashVectorStore.add_texts..<++++++r=rvectorsrrX) r5rDzipr4rangelenrErr.upsertr rr6r>rdrNrPrRrrSrbri chunk_texts chunk_idschunk_metadatasr<batchs r7rfzUpstashVectorStore.add_textss@  IU 777777  ,559555II++U+++I")U33 , ,NHd'+HT^ $ $q#e**&:;;  AA(<$< <=KA$8 889I'A0D,D(DEO..{;;J&C :GG  # "!T#y-A-AEK  r=c &K||j}t|}|p d|D}|r d|D}n d|D}t||D]\}} | ||j<t dt ||D]} || | |z} || | |z} || | |z} || }t|t| || D]0}|jj d|tt|d|d{V1|S)rsNcNg|]"}ttj#SrXrurxs r7r^z1UpstashVectorStore.aadd_texts..urzr=c6g|]}|SrXr|r~s r7r^z1UpstashVectorStore.aadd_texts..yrr=cg|]}iSrXrXrxs r7r^z1UpstashVectorStore.aadd_texts..{rr=rrrX) r5rDrr4rrrErr1rr rrs r7rozUpstashVectorStore.aadd_textsQs@  IU 777777  ,559555II++U+++I")U33 , ,NHd'+HT^ $ $q#e**&:;;  AA(<$< <=KA$8 889I'A0D,D(DEO..{;;J&C :GG  /d'.!T#y-A-AEK  r=querykfilterList[Tuple[Document, float]]c L|j||f|||d|S)Retrieve texts most similar to query and convert the result to `Document` objects. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: Optional metadata filter in str format namespace: Namespace to use from the index. Returns: List of Documents most similar to the query and score for each rrr)&similarity_search_by_vector_with_scorerIr6rrrrrSs r7similarity_search_with_scorez/UpstashVectorStore.similarity_search_with_scoresE*;t:   e $ $ ()&I  QW   r=c\K|j||f|||d|d{VS)rrN)'asimilarity_search_by_vector_with_scorerIrs r7asimilarity_search_with_scorez0UpstashVectorStore.asimilarity_search_with_scoresh*BTA   e $ $ ()&I  QW         r=resultsrcg}|D]}|j}|rQ|j|vrH||j}t||}|||jf\t d|jd|S)NrZrbzFound document with no `z` key. Skipping.)rbr4popr appendscorer/warning)r6rdocsresrbrr]s r7_process_resultsz#UpstashVectorStore._process_resultss  C|H DNh66||DN33D8DDD S#),----Ot~OOO r=c |pd}||j}t|tr|jjd||d||d|}n|jjd||d||d|}||Sz>Return texts whose embedding is closest to the given embeddingrNT)datatop_kinclude_metadatarr)vectorrrrrrX)r5r+rr.rrr6r$rrrrSrs r7rz9UpstashVectorStore.similarity_search_by_vector_with_scores2  I i % % 'dk'!%#   GG(dk' !%#   G$$W---r=c K|pd}||j}t|tr|jjd||d||d|d{V}n|jjd||d||d|d{V}||Sr)r5r+rr1rrrs r7rz:UpstashVectorStore.asimilarity_search_by_vector_with_scores2  I i % % 3D-3!%#   GG4D-3 !%#   G$$W---r=c >|j|f|||d|}d|DS)aReturn documents most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: Optional metadata filter in str format namespace: Namespace to use from the index. Returns: List of Documents most similar to the query and score for each rcg|]\}}|SrXrXr\r]rys r7r^z8UpstashVectorStore.similarity_search..1222Q222r=rr6rrrrrSdocs_and_scoress r7similarity_searchz$UpstashVectorStore.similarity_searchsJ(<$;  v  >D  32/2222r=cNK|j|f|||d|d{V}d|DS)arReturn documents most similar to query. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: Optional metadata filter in str format namespace: Namespace to use from the index. Returns: List of Documents most similar to the query rNcg|]\}}|SrXrXrs r7r^z9UpstashVectorStore.asimilarity_search..Jrr=rrs r7asimilarity_searchz%UpstashVectorStore.asimilarity_search3sm(!C B ! v! ! >D! !       32/2222r=c >|j|f|||d|}d|DS)Return documents closest to the given embedding. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. filter: Optional metadata filter in str format namespace: Namespace to use from the index. Returns: List of Documents most similar to the query rcg|]\}}|SrXrXrs r7r^zBUpstashVectorStore.similarity_search_by_vector..crr=)rr6r$rrrrSrs r7similarity_search_by_vectorz.UpstashVectorStore.similarity_search_by_vectorLsK(F$E  6Y  BH  32/2222r=cNK|j|f|||d|d{V}d|DS)rrNcg|]\}}|SrXrXrs r7r^zCUpstashVectorStore.asimilarity_search_by_vector..|rr=)rrs r7asimilarity_search_by_vectorz/UpstashVectorStore.asimilarity_search_by_vectoresm(!M L ! 6Y! ! BH! !       32/2222r=c &|j|f|||d|S)` Since Upstash always returns relevance scores, default implementation is used. rrrs r7(_similarity_search_with_relevance_scoresz;UpstashVectorStore._similarity_search_with_relevance_scores~s71t0  v  >D   r=c6K|j|f|||d|d{VS)rrNrrs r7)_asimilarity_search_with_relevance_scoreszD         r=?fetch_k lambda_multfloatc  |j}tjtsJt|trjjd ||dd|pd|d| njjd ||dd|pd|d| ttj |gtj d D||} fd |D} fd | DS) gReturn docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: embedding: Embedding to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. filter: Optional metadata filter in str format namespace: Namespace to use from the index. Returns: List of Documents selected by maximal marginal relevance. NTrrrinclude_vectorsrrrrrrrrrdtypecg|] }|j SrXrr\items r7r^zNUpstashVectorStore.max_marginal_relevance_search_by_vector.. - - -TT[ - - -r=rrc*g|]}|jSrXrar\rrs r7r^zNUpstashVectorStore.max_marginal_relevance_search_by_vector.. >>>AGAJ'>>>r=cbg|]+}t|j|,Srr rr4r\rbr6s r7r^zNUpstashVectorStore.max_marginal_relevance_search_by_vector..D    (,,"@"@8 T T T   r=rX) r5r+r<r rr.rrnparrayfloat32 r6r$rrrrrrS mmr_selectedselectedrs ` @r7'max_marginal_relevance_search_by_vectorz:UpstashVectorStore.max_marginal_relevance_search_by_vectorsJ<  I$/:66666 i % % 'dk' $!%|# GG(dk'  $!%|# G2 Hi[ 3 3 3 - -W - - -#    ?>>>>>>    $    r=c  K|j}tjtsJt|tr"jjd ||dd|pd|d|d{V n!jjd ||dd|pd|d|d{V ttj |gtj d D||} fd |D} fd | DS) rNTrrrrcg|] }|j SrXrrs r7r^zOUpstashVectorStore.amax_marginal_relevance_search_by_vector..rr=rc*g|]}|jSrXrars r7r^zOUpstashVectorStore.amax_marginal_relevance_search_by_vector..rr=cbg|]+}t|j|,Srrrs r7r^zOUpstashVectorStore.amax_marginal_relevance_search_by_vector.. rr=rX) r5r+r<r rr1rrrrrrs ` @r7(amax_marginal_relevance_search_by_vectorz;UpstashVectorStore.amax_marginal_relevance_search_by_vectors>  I$/:66666 i % % 3D-3 $!%|# GG4D-3  $!%|# G2 Hi[ 3 3 3 - -W - - -#    ?>>>>>>    $    r=c T||}|jd||||||d|S)^Return docs selected using the maximal marginal relevance. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents. Args: query: Text to look up documents similar to. k: Number of Documents to return. Defaults to 4. fetch_k: Number of Documents to fetch to pass to MMR algorithm. lambda_mult: Number between 0 and 1 that determines the degree of diversity among the results with 0 corresponding to maximum diversity and 1 to minimum diversity. Defaults to 0.5. filter: Optional metadata filter in str format namespace: Namespace to use from the index. Returns: List of Documents selected by maximal marginal relevance. r$rrrrrrX)rIr r6rrrrrrrSr$s r7max_marginal_relevance_searchz0UpstashVectorStore.max_marginal_relevance_search%sT<%%e,, ;t; #      r=c dK||}|jd||||||d|d{VS)rrNrX)rIrrs r7amax_marginal_relevance_searchz1UpstashVectorStore.amax_marginal_relevance_searchNsv<%%e,, BTB #            r=r c  \|d|||| | | | d| }||||||| |S)Create a new UpstashVectorStore from a list of texts. Example: .. code-block:: python from langchain_community.vectorstores.upstash import UpstashVectorStore from langchain_community.embeddings import OpenAIEmbeddings embeddings = OpenAIEmbeddings() vector_store = UpstashVectorStore.from_texts( texts, embeddings, ) )r$rrrr!r#rrmrXreclsr>r$rdrNrRrPrrrr!r#rrS vector_stores r7 from_textszUpstashVectorStore.from_textswsw>s  ##        !!5    r=c  lK|d|||| | | | d| }|||||| |d{V|S)r)r$rrrrr!r#)rdrNrPrrRNrXrnrs r7 afrom_textszUpstashVectorStore.afrom_textss>s  ##      %% !!5 &         r= delete_allOptional[bool] Optional[int]Nonec ||j}|r|j|nA|0t||D]}|j||nt ddSaMDelete by vector IDs Args: ids: List of ids to delete. delete_all: Delete all vectors in the index. batch_size: Batch size to use when deleting the embeddings. namespace: Namespace to use from the index. Upstash supports at max 1000 deletions per request. Nr)rNrz+Either ids or delete_all should be provided)r5r.resetrdeleter,r6rNrrPrrSrs r7rzUpstashVectorStore.deletes$  I  L K    2 2 2 2 _&z377 C C ""u "BBBB CJKK Ktr=cK||j}|r"|j|d{VnG|6t||D]$}|j||d{V%nt ddSr)r5r1rrrr,r s r7adeletezUpstashVectorStore.adeletes$  I  L#))I)>> > > > > > > > > _&z377 O O'..5I.NNNNNNNNNN OJKK Ktr=rc4|jS)8Get statistics about the index. Returns: - total number of vectors - total number of vectors waiting to be indexed - total size of the index on disk in bytes - dimension count for the index - similarity function selected for the index )r.r0r;s r7r0zUpstashVectorStore.infos{!!!r=cDK|jd{VS)r N)r1r0r;s r7ainfozUpstashVectorStore.ainfo)s/&++---------r=)rNNNNN)rrrrrr r!r"r#r"r$r%rr)r9r%)r>r?r9r@)rrr9rF)NrJrK)rLrMrNrOrPrQrRrQrr"rSrr9rT)rLrirNrOrPrQrRrQrr"rSrr9rT)NNrJrK)r>r?rdrqrNrOrPrQrRrQrr"rSrr9rT)rN) rrrrQrr"rr"rSrr9r)rrr9r) r$rFrrQrr"rr"rSrr9r) rrrrQrr"rr"rSrr9rM) r$rFrrQrr"rr"rSrr9rM)rrrN)r$rFrrQrrQrrrr"rr"rSrr9rM)rrrrQrrQrrrr"rr"rSrr9rM) NNrKrJrNNNN)r>rTr$r rdrqrNrOrRrQrPrQrrrrrr r!r"r#r"rrrSrr9r)NNrK) rNrOrrrPrrr"rSrr9r)r9r)#__name__ __module__ __qualname____doc__r8propertyr<rErIrhrprfrorrrrrrrrrrrrrrr classmethodrrrr r0rrXr=r7rrs  H!%,0#'%)7;^$^$^$^$^$^$^$@   X     "$($( ( $(( ( ( ( ( ( Z$($( ( $(( ( ( ( ( ( Z+/#'$( =$(======D+/#'$( =$(======D $  $(       8 $  $(       2    " $ #. $( #.#.#.#.#.#.P $ #. $( #.#.#.#.#.#.P $ 3 $( 3333338 $ 3 $( 3333338 $ 3 $( 3333338 $ 3 $( 3333338 $  $(       & $  $(       & $ A $(A A A A A A L $ B $(B B B B B B N $ ' $(' ' ' ' ' ' X $ ' $(' ' ' ' ' ' R +/#'$(!%,0#'%)111111[1f +/#'$(!%,0#'%)111111[1j$(%)$(  $( B$(%)$(  $( > " " " " . . . . . .r=r)" __future__rloggingrvtypingrrrrrr r r numpyrlangchain_core.documentsr langchain_core.embeddingsr langchain_core.utils.iterrlangchain_core.vectorstoresr&langchain_community.vectorstores.utilsrr)rrupstash_vector.typesr getLoggerrr/rrXr=r7r!s["""""" SSSSSSSSSSSSSSSSSSSS------000000333333333333000000000//////  8 $ $[.[.[.[.[.[.[.[.[.[.r=