NghddlmZddlZddlmZmZmZmZmZm Z m Z ddl m Z ddl mZddlmZerddlmZdd d d d ZejeZGd d eZdS)) annotationsN) TYPE_CHECKINGAnyDictIterableListOptionalTuple)Document) Embeddings) VectorStore HippoClient localhost7788admin)hostportusernamepasswordceZdZdZ dBdCdZdDdZ dEdFdZ dGdHd!ZdId"ZdJd#Z dId$Z dKdLd.Z dMdNd6Z dMdOd8Z dMdPd;ZedddedddfdQdAZdS)RHippoa`Hippo` vector store. You need to install `hippo-api` and run Hippo. Please visit our official website for how to run a Hippo instance: https://www.transwarp.cn/starwarp Args: embedding_function (Embeddings): Function used to embed the text. table_name (str): Which Hippo table to use. Defaults to "test". database_name (str): Which Hippo database to use. Defaults to "default". number_of_shards (int): The number of shards for the Hippo table.Defaults to 1. number_of_replicas (int): The number of replicas for the Hippo table.Defaults to 1. connection_args (Optional[dict[str, any]]): The connection args used for this class comes in the form of a dict. index_params (Optional[dict]): Which index params to use. Defaults to IVF_FLAT. drop_old (Optional[bool]): Whether to drop the current collection. Defaults to False. primary_field (str): Name of the primary key field. Defaults to "pk". text_field (str): Name of the text field. Defaults to "text". vector_field (str): Name of the vector field. Defaults to "vector". The connection args used for this class comes in the form of a dict, here are a few of the options: host (str): The host of Hippo instance. Default at "localhost". port (str/int): The port of Hippo instance. Default at 7788. user (str): Use which user to connect to Hippo instance. If user and password are provided, we will add related header in every RPC call. password (str): Required when user is provided. The password corresponding to the user. Example: .. code-block:: python from langchain_community.vectorstores import Hippo from langchain_community.embeddings import OpenAIEmbeddings embedding = OpenAIEmbeddings() # Connect to a hippo instance on localhost vector_store = Hippo.from_documents( docs, embedding=embeddings, table_name="langchain_test", connection_args=HIPPO_CONNECTION ) Raises: ValueError: If the hippo-api python package is not installed. testdefaultNFembedding_functionr table_namestr database_namenumber_of_shardsintnumber_of_replicasconnection_argsOptional[Dict[str, Any]] index_paramsOptional[dict]drop_oldOptional[bool]c ||_||_||_||_||_||_d|_d|_d|_g|_ |t}| ||_ d|_ |j |j|jr'|r%|j |j|jn2#t $r%} t#jd|jd| d} ~ wwxYw |j |j|jr*|j |j|j|_ n2#t $r%} t#jd|jd| d} ~ wwxYw|dS)Npktextvectorz+An error occurred while deleting the table z: z*An error occurred while getting the table )r r"embedding_funcrrr%_primary_field _text_field _vector_fieldfieldsDEFAULT_HIPPO_CONNECTION_create_connection_aliashccolcheck_table_exists delete_table Exceptionloggingerror get_table_get_env) selfrrrr r"r#r%r'es b/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/vectorstores/hippo.py__init__zHippo.__init__Ps!1"40$*(#!%!#  "6O//@@ **4?DPQQ    MWWWTUWW       s22A B?? C. C))C.2AE E1 E,,E1dictreturnrc ddlm}n"#t$r}td|d}~wwxYw|dd}|dd |dd}|d d}|` ^d |vr7|d }d  fd |D}n2t |d zt z}ntd  t d|d||g||S#t$r!}t d|d}~wwxYw)z*Create the connection to the Hippo server.rrzQUnable to import transwarp_hipp_api, please install with `pip install hippo-api`.Nrrrshivar,cg|] }|d S):).0hrs r? z2Hippo._create_connection_alias..s#)G)G)GAQ----)G)G)GrGz/Missing standard address type for reuse attemptzcreate HippoClient[])rpwdzFailed to create new connection) transwarp_hippo_api.hippo_clientr ImportErrorgetsplitjoinr ValueErrorloggerinfor8r:) r=r#rr>rrrhosts given_addressrs @r?r3zHippo._create_connection_aliass  D D D D D D D   +   $''55#''55'++J@@'++J@@   0d{{ 3 #)G)G)G)G)G)G)G H H #D C#d)) ; NOO O  KK>m>>> ? ? ?; xPPP P    LL: ; ; ;G s( )$)6,D## E-E  E embeddingsOptional[list] metadatasOptional[List[dict]]Nonectd|0td|||||dS)Nzinit ...zcreate collection)rUrV_create_collection_extract_fields _create_index)r=rYr[s r?r<zHippo._get_envso  J  ! KK+ , , ,  # #J : : :  rLlistc ddlm}ddlm}t |d}t d|g}|||jd|j |||j d|j |||j d|j d|i|r|d D]x\}}t|tr9t |} |||d|j d| iS|||d|j yt d ||j|jd||j|j|j |j|j|j|_t d |jd |jd dS)Nr) HippoField) HippoTypez[_create_collection] dim: TF dimension) type_paramsz[_create_collection] fields: )nameauto_idr1rr r"z$[_create_collection] : create table z in z successfully)rOrdtranswarp_hippo_api.hippo_typerelenrUdebugappendr.STRINGr/r0 FLOAT_VECTORitems isinstancerbr4 create_tablerrr r"r;r5rV) r=rYr[rdredimr1keyvalue value_dims r?r_zHippo._create_collectionsp @?????<<<<<<*Q-   7#77888  jj!4dI>> ,!2#6    7$$T_d6HII  S O S S151C S S S     rLcddlm}t|j|rq|jj}t d||D]!}|j|j "t d|jdSdS)z,Grab the existing fields from the Collectionr HippoTablez[_extract_fields] schema:z04 [_extract_fields] fields:N) rOryrqr5schemarUrlr1rmrh)r=ryrzxs r?r`zHippo._extract_fieldss?????? dh + + GX_F LL=V== > > > + + ""16**** LLE EE F F F F F  G GrLcddlm}t|j|r|j|j|j|ji}|dd}|dS|j|j|j|jdD]4}t d||d|j kr|cS5dS)z0Return the vector index information if it existsrrxembedding_indexesNz[_get_index] embedding_indexes column) rOryrqr5r4get_table_inforrrQrUrlr0)r=ry table_infor}r{s r? _get_indexzHippo._get_indexs?????? dh + + !//!3c$/2&& !+/BD I I  (t//ASTTO%'!!ALL!VCT!V!VWWW{d&888 9trLc h ddlm}ddlm}m}t |j|r |||jd|j |j dd|_|j |j |jd|jd |jd |jd  t|j|jdtd dS|j |j|j|j|jd}|j|j|j |j d}||jd |jd <|jd dkr||jd |jd <|j |j |jd|jd |jd t|j|jddS|jd dks|jd dkr||jd |jd <|j |j |jd|jd |jd |jd d|jddt|j|jddS|jd dkr||jd |jd <|j |j |jd|jd |jd |jd d|jdd|jdd|jdt|j|jddS|jd dkr||jd |jd <|j |j |jd|jd |jd |jd|jd|jdt|j|jddSt/ddSdSdS) z Create a index on the collectionrrx) IndexType MetricTypeNlangchain_auto_create ) index_name metric_type index_typenlistrrrr)rzcreate index successfully)IVF_FLATFLATIVF_SQIVF_PQHNSW)ipIPl2L2rrrnprobe)rrrnbitsm)rrrrrMef_construction ef_search)rrrzeIndex name does not match, please enter the correct index name. (FLAT, IVF_FLAT, IVF_PQ,IVF_SQ, HNSW))rOryrjrrrqr5rr%rr create_indexr0rUrlactivate_indexrVrrrrrrQrT)r=ryrr index_dict metric_dicts r?razHippo._create_indexsr??????HHHHHHHH dh + +m 0A0A0I  ($,&='1}&/&8!# ))D%H))*),7),7)-8"/8 *LL//0A,0OPPKK ;<<<<<%.$6 )"+"2"+"2 ) ""J)m(m(m(m ##K 8C)-88D%m4(6&@@:D -l;;),7-- . -l; -l; -m<    H33D4El4STT),7:EE,\:hFF:D -l;;),7-- . -l; -l; -m<"&"3"7"7"D"D#'#4#8#82#F#F .  H33D4El4STT*<8HDD:D -l;;),7-- . -l; -l; -m<"&"3"7"7"D"D#'#4#8#82#F#F"&"3"7"7"C"C"/33C88.     H33D4El4STT*<8FBB:D -l;;),7-- . -l; -l; -m<"/33C88,0,=,A,ABS,T,T&*&7&;&;K&H&H.  H33D4El4STT)DSm m 0I0I((rLtexts Iterable[str]timeout Optional[int] batch_sizekwargsr List[str]c ddlm}|rtd|DrtdgSt |}td| j|}n#t$rfd|D}YnwxYwt|dkrtdgStdt|tj |s ||j |j|itd|td j|Q|D]N}|D]7\} } | jvr)| g| 8Otj j} t| } d jvrjd td | t)d| |D]t+|z| fd jD} j | }td|c#t0$r#}td| |d }~wwxYwdgS)a Add text to the collection. Args: texts: An iterable that contains the text to be added. metadatas: An optional list of dictionaries, each dictionary contains the metadata associated with a text. timeout: Optional timeout, in seconds. batch_size: The number of texts inserted in each batch, defaults to 1000. **kwargs: Other optional parameters. Returns: A list of strings, containing the unique identifiers of the inserted texts. Note: If the collection has not yet been created, this method will create a new collection. rrxc3"K|] }|dkV dS)NrH)rIts r? z"Hippo.add_texts..s&33AG333333rLzNothing to insert, skipping.z[add_texts] texts: cDg|]}j|SrH)r- embed_query)rIr{r=s r?rKz#Hippo.add_texts..s*LLL$-99!<<LLLrLzdd_texts] len_embeddings:z[add_texts] metadatas:z[add_texts] fields:Nr*z[add_texts] total_count:c0g|]}|SrHrH)rIr{endi insert_dicts r?rKz#Hippo.add_texts..s&FFFQ;q>!C%0FFFrLz05 [add_texts] insert z0Failed to insert batch starting at entity: %s/%sr)rOryallrUrlrbr-embed_documentsNotImplementedErrorrkrqr5r<r/r0r1rp setdefaultrmremoverangemin insert_rowsrVr8r:)r=rr[rrrryrYdrtruvectors total_count insert_listresr>rrrs` @@@r? add_textszHippo.add_textssU4 @????? 33U33333  LL7 8 8 8IU  2522333 M,<>uEEEF  [!12333$D$67'll 4;   K  t $ $ $ = ==>>>q+z22  Aa*nk22CFFFFFF$+FFFK h**;77 :S::;;;;    F;   t s*/B B%$B% 7K K0 K++K0querykparamexpr Optional[str]List[Document]c |jtdgS|jd|||||d|}d|DS)a Perform a similarity search on the query string. Args: query (str): The text to search for. k (int, optional): The number of results to return. Default is 4. param (dict, optional): Specifies the search parameters for the index. Defaults to None. expr (str, optional): Filtering expression. Defaults to None. timeout (int, optional): Time to wait before a timeout error. Defaults to None. kwargs: Keyword arguments for Collection.search(). Returns: List[Document]: The document results of the search. N!No existing collection to search.)rrrrrcg|]\}}|SrHrH)rIdoc_s r?rKz+Hippo.similarity_search..s&&&Q&&&rLrH)r5rUrlsimilarity_search_with_score)r=rrrrrrrs r?similarity_searchzHippo.similarity_searchsk4 8  LL< = = =I/d/ 1Eg  IO  '&#&&&&rLList[Tuple[Document, float]]c |jtdgS|j|}|jd|||||d|}|S)a Performs a search on the query string and returns results with scores. Args: query (str): The text being searched. k (int, optional): The number of results to return. Default is 4. param (dict): Specifies the search parameters for the index. Default is None. expr (str, optional): Filtering expression. Default is None. timeout (int, optional): The waiting time before a timeout error. Default is None. kwargs: Keyword arguments for Collection.search(). Returns: List[float], List[Tuple[Document, any, any]]: Nr) embeddingrrrrrH)r5rUrlr-r&similarity_search_with_score_by_vector) r=rrrrrrrrets r?rz"Hippo.similarity_search_with_scorest6 8  LL< = = =I'33E:: 9d9 1Eg  QW   rLr List[float]c |jtdgS|jdd}||jtd|jtd|gtd|td|td||j|j|g|||td |jd z}g} d } tfd |DD]} d t|| D} t| |j| } tdd |d || }| dz } | | |f| S)a Performs a search on the query string and returns results with scores. Args: embedding (List[float]): The embedding vector being searched. k (int, optional): The number of results to return. Default is 4. param (dict): Specifies the search parameters for the index. Default is None. expr (str, optional): Filtering expression. Default is None. timeout (int, optional): The waiting time before a timeout error. Default is None. kwargs: Keyword arguments for Collection.search(). Returns: List[Tuple[Document, float]]: Resulting documents and scores. Nrz search_field:zvectors:zoutput_fields:ztopk:zdsl:) search_fieldr output_fieldstopkdslz-[similarity_search_with_score_by_vector] res:z%scoresrc,g|]}d|S)rrH)rIfieldrs r?rKz@Hippo.similarity_search_with_score_by_vector..fs!DDDU3q6%=DDDrLci|]\}}|| SrHrH)rIrrus r? z@Hippo.similarity_search_with_score_by_vector..gsOOO\UEE5OOOrL) page_contentmetadataz;[similarity_search_with_score_by_vector] res[0][score_col]:r) r5rUrlr1rr0rr/zipr poprm)r=rrrrrrr score_colrcountrpmetarscorers @r?rz,Hippo.similarity_search_with_score_by_vector.s 4 8  LL< = = =I  AAA T/000  9T%799::: - --... 5m55666 [Q[[!!! ]D]]###hnn+K'     JSJJKKK$y0 DDDDmDDDE % %EOOS5N5NOOOD1A(B(BTRRRC LL9%(VI%699   F9%e,E QJE JJU| $ $ $ $ rLDict[str, Any]Optional[Dict[Any, Any]] search_paramsbool'Hippo'c |i}td|d|||||| d| } td|td|| ||| S)a Creates an instance of the VST class from the given texts. Args: texts (List[str]): List of texts to be added. embedding (Embeddings): Embedding model for the texts. metadatas (List[dict], optional): List of metadata dictionaries for each text.Defaults to None. table_name (str): Name of the table. Defaults to "test". database_name (str): Name of the database. Defaults to "default". connection_args (dict[str, Any]): Connection parameters. Defaults to DEFAULT_HIPPO_CONNECTION. index_params (dict): Indexing parameters. Defaults to None. search_params (dict): Search parameters. Defaults to an empty dictionary. drop_old (bool): Whether to drop the old collection. Defaults to False. kwargs: Other arguments. Returns: Hippo: An instance of the VST class. Nz'00 [from_texts] init the class of Hippo)rrrr#r%r'z[from_texts] texts:z[from_texts] metadatas:)rr[rH)rUrVrlr) clsrrr[rrr#r%rr'r vector_dbs r? from_textszHippo.from_textsssF  M =>>>C (!'+%       2522333 :y::;;;%9===rL)rrrrNNF)rr rrrrr r!r"r!r#r$r%r&r'r()r#rArBr)NN)rYrZr[r\rBr])N)rYrbr[r\rBr])rBr])rBr$)NNr) rrr[r\rrrr!rrrBr)rNNN)rrrr!rr&rrrrrrrBr)rrrr!rr&rrrrrrrBr)rrrr!rr&rrrrrrrBr)rrrr r[r\rrrrr#rr%rrr$r'rrrrBr)__name__ __module__ __qualname____doc__r@r3r<r_r`rrarrrr classmethodr2rrHrLr?rrs55t!& !"#48'+#(77777rDTXCG? ? ? ? ? B G G G G*rrrrn+/!% VVVVVv $"!% ' ' ' ' 'J $"!% %%%%%T $"!% CCCCCJ +/ &*B15261111[111rLr) __future__rr9typingrrrrrr r langchain_core.documentsr langchain_core.embeddingsr langchain_core.vectorstoresr rOrr2 getLoggerrrUrrHrLr?rs""""""LLLLLLLLLLLLLLLLLL------000000333333=<<<<<<     8 $ $M M M M M KM M M M M rL