NgYddlmZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlmZddlmZddlmZdd lmZmZejZddZGddeZGddeZGddeZdS)) annotationsN)sha1)Thread)AnyDictIterableListOptionalTuple)Document) Embeddings) VectorStore) BaseSettingsSettingsConfigDictsstrargsrreturnboolc|D] }||vrdS dS)z Check if a string contains multiple substrings. Args: s: string to check. *args: substrings to check. Returns: True if all substrings are in the string, False otherwise. FT)rras d/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/vectorstores/myscale.pyhas_mul_sub_strrs- A::55  4ceZdZUdZdZded<dZded<dZd ed <dZd ed <d Z ded <dZ ded<dddddZ ded<dZ ded<dZ ded<dZded<d&d Zed!d"d#d$%ZdS)'MyScaleSettingsaMyScale client configuration. Attribute: myscale_host (str) : An URL to connect to MyScale backend. Defaults to 'localhost'. myscale_port (int) : URL port to connect with HTTP. Defaults to 8443. username (str) : Username to login. Defaults to None. password (str) : Password to login. Defaults to None. index_type (str): index type string. index_param (dict): index build parameter. database (str) : Database name to find the table. Defaults to 'default'. table (str) : Table name to operate on. Defaults to 'vector_table'. metric (str) : Metric to compute distance, supported are ('L2', 'Cosine', 'IP'). Defaults to 'Cosine'. column_map (Dict) : Column type map to project column name onto langchain semantics. Must have keys: `text`, `id`, `vector`, must be same size to number of columns. For example: .. code-block:: python { 'id': 'text_id', 'vector': 'text_embedding', 'text': 'text_plain', 'metadata': 'metadata_dictionary_in_json', } Defaults to identity map. localhostrhosti intportN Optional[str]usernamepasswordMSTG index_typezOptional[Dict[str, str]] index_paramidtextvectormetadata)r(r)r*r+zDict[str, str] column_mapdefaultdatabase langchaintableCosinemetricitemrrc"t||SN)getattr)selfr3s r __getitem__zMyScaleSettings.__getitem__UstT"""rz.envutf-8myscale_ignore)env_fileenv_file_encoding env_prefixextra)r3rrr)__name__ __module__ __qualname____doc__r__annotations__r!r#r$r&r'r,r.r0r2r8r model_configrrrrr!s>DD"H"""""H""""J,0K0000 ""JHEF####&%! LLLrrceZdZdZ d7d8fd Zed9d Zd:dZd;dZddZ e d?d@d"Z dAd#Z d7dBd)Z dCdDd.Z dCdEd/Z dCdFd1ZdGd2Z dHdId5ZedAd6ZxZS)JMyScalea`MyScale` vector store. You need a `clickhouse-connect` python package, and a valid account to connect to MyScale. MyScale can not only search with simple vector indexes. It also supports a complex query with multiple conditions, constraints and even sub-queries. For more information, please visit [myscale official site](https://docs.myscale.com/en/overview/) N embeddingr configOptional[MyScaleSettings]kwargsrrNonec  ddlm}n#t$rtdwxYw ddlm}||_n#t$r d|_YnwxYwt |||_nt|_|jsJ|jj r |jj sJ|jj r$|jj r|jj r |jjsJdD]}||jj vsJ|jjdvsJ|jjd vrt d t%|d }|jjr>d d d|jjDznd}d|jj d|jj d|jj dd|jj dd|jj dd|jj dd|jj dd|d|jj dd|jjd|jjd|d|jj dd } ||_d!|_d"|_||_|jjd#vrd$nd%|_|d+|jj |jj |jj|jjd&||_ |j d'n?#tB$r2} t "d(|jj#d)Yd} ~ nd} ~ wwxYw|j d*|j | dS),zMyScale Wrapper to LangChain embedding (Embeddings): config (MyScaleSettings): Configuration to MyScale Client Other keyword arguments will pass into [clickhouse-connect](https://docs.myscale.com/) r) get_clientzlCould not import clickhouse connect python package. Please install it with `pip install clickhouse-connect`.)tqdmc|Sr5r)xs rz"MyScale.__init__..s1rN)r(r*r)r+)IPCOSINEL2)ipcosinel2z_Lower case metric types will be deprecated the future. Please use one of ('IP', 'Cosine', 'L2')z try this out, ,c&g|]\}}d|d|dS)'=r).0kvs r z$MyScale.__init__..s*VVVda\\\Q\\\VVVrz( CREATE TABLE IF NOT EXISTS .z( r(z String, r)r*z! Array(Float32), r+zP JSON, CONSTRAINT cons_vec_len CHECK length( z) = z$, VECTOR INDEX vidx z TYPE z&( 'metric_type=r\z,) ) ENGINE = MergeTree ORDER BY z \)rdr\)rTrUASCDESC)rr!r#r$z"SET allow_experimental_json_type=1zClickhouse version=z6 - There is no allow_experimental_json_type parameter.z$SET allow_experimental_object_type=1r)$clickhouse_connectrN ImportErrorrOpgbarsuper__init__rIrrr!r,r.r0r2upperloggerwarninglen embed_queryr'joinitemsr&dimBS must_escape _embeddings dist_orderr#r$clientcommand Exceptiondebugserver_version) r7rHrIrKrNrOr_rs index_paramsschema__ __class__s rrkzMyScale.__init__ns  5 5 5 5 5 5 5   K    % ! ! ! ! ! !DJJ % % %$DJJJ %    DKK)++DK{{4DK$4444 K "  $  !  "    6 / /A ......{!''))-CCCCC ; !7 7 7 NNG    )''7788{& D388VVdk6M6S6S6U6UVVVWW W W   (, (<   ?C{?P   '-   '/   '1   ' 3   [+H5   z%MyScale.escape_str..s>VV1!t/?*?*?$'1QVVVVVVr)rq)r7rs` r escape_strzMyScale.escape_strs+wwVVVVPUVVVVVVrtransacr column_names Iterable[str]c  d|}g}|D]<}dfd|D}|d|d=djjdjjd|dd|d }|S) NrZc\g|](}dt|d)Sr\)rr)r^_nr7s rraz'MyScale._build_istr..s7FFFb9dooc"gg66999FFFr()z8 INSERT INTO TABLE rcz)) VALUES z )rqappendrIr.r0)r7rrks_datani_strs` r _build_istrzMyScale._build_istrs XXl # # # #AFFFFAFFFGGA LLQ " " " "[),0K,=@B%   rcf|||}|j|dSr5)rrxry)r7rr_i_strs r_insertzMyScale._inserts3!!'<88 F#####r texts metadatasOptional[List[dict]] batch_sizer idsOptional[Iterable[str]] List[str]c |p d|D}|jj}g}|d||d||dt|jj|i}|p d|D}tt j|||d<tt|t|z dksJt| \} } d} | t| d t| D]} t| | |jjd|j ksJ|| t||krD| r| t!|j|| g } | g}t|dkr,| r| ||| d |DS#t&$rG} t(d t-| dt/| dgcYd} ~ Sd} ~ wwxYw)aRun more texts through the embeddings and add to the vectorstore. Args: texts: Iterable of strings to add to the vectorstore. ids: Optional list of ids to associate with the texts. batch_size: Batch size of insertion metadata: Optional column data to be inserted Returns: List of ids from adding the texts into the vectorstore. cvg|]6}t|d7S)r9)rencode hexdigest)r^ts rraz%MyScale.add_texts..s8IIIad188G,,--7799IIIrr(r)r*cg|]}iSrr)r^rs rraz%MyScale.add_texts..s!4!4!4"!4!4!4rr+rNzInserting data...)desctotal)targetrcg|]}|Srr)r^is rraz%MyScale.add_texts..s###!A###r   )rIr,maprvrpjsondumpsrosetziprrriindexrsrrqrrstartrzrmerrortyper)r7rrrrrKcolmap_rrkeysvaluesrr`es r add_textszMyScale.add_textssb*III5III+( DM3 FOU H s4#3#?GG 4!4!4e!4!4!4 ,/ I,F,F WZ()3w<<#l"3"33449999L..001 f AZZV #6c)nn  ! !1TZZ (>x(HIIJKKtxWWWWq!!!w<<:--!dl'4IIIAGGIII G7||aFFHHH Wd+++##s### #    LLS477SSCFFSSS T T TIIIIII sD*G<< I sP$+"6OO9JOOO DK$DDt{'7DDDD UT[%9UUUU  "" >DK( > >4;+< > >  -//  A TAfITTT1V9TTTT EE   rq_emb List[float]topk where_strr"c8dtt|}|rd|}nd}d|jjdd|jjdd|jjd |jjd |d |jjd d |d|jd|d }|S)NrZ PREWHERE rb SELECT r)z, r+z, dist FROM rc  ORDER BY distance(r*, []) AS dist  LIMIT )rqrrrIr,r.r0rwr7rrr q_emb_strq_strs r _build_qstrzMyScale._build_qstrRsHHSe__--  /I//IIIK*62' 3+&*.):  ${5h?  EN    rrr_List[Document]c R|j|j|||fi|S)aPerform a similarity search with MyScale Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of Documents )similarity_search_by_vectorrvrp)r7rr_rrKs rsimilarity_searchzMyScale.similarity_searchfs@&0t/   ( ( / /I  AG   rc L|||} fdj|DS#t$rG}t dt|dt|dgcYd}~Sd}~wwxYw)Perform a similarity search with MyScale by vectors Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of (Document, similarity) cg|]?}t|jjd|jjd@S)r)r+ page_contentr+r rIr,r^rr7s rraz7MyScale.similarity_search_by_vector..s^  !"4;#9&#A!Bt{5jABrrrrN rrxrrrzrmrrrr7rHr_rrKrrs` rrz#MyScale.similarity_search_by_vector}.  Ay99  **511??AA      LLS477SSCFFSSS T T TIIIIII 7A B#.sl    %&t{'=f'E%F!"4;#9*#E!FfI    rrrrN rrvrprxrrrzrmrrrr7rr_rrKrrs` r'similarity_search_with_relevance_scoresz/MyScale.similarity_search_with_relevance_scores*  !1!=!=e!D!DaSS     **511??AA        LLS477SSCFFSSS T T TIIIIII 7A** B;4.s 9 9 9rR 9 9 9rr(z IN (rz AND z DELETE FROM rcz WHERE TF) rorqrrIr,r.r0rxryrzrmrr) r7rrrKcondsid_listwhere_str_finalqstrrs rdeletezMyScale.deletesc" KI-- V.--  K3s88a<<ii 9 9S 9 9 9::G LLDK248IIwIII J J J  $ LL # # #5zzA~~~~!,,u-- '4;/ ' '$+2C ' '$ ' '   K   % % %4    LLQ 55555 sC-- D)7'D$$D)c&|jjdS)Nr+)rIr,rs rmetadata_columnzMyScale.metadata_columns{%j11rr5)rHr rIrJrKrrrL)rr )rrrr)rrrrrr)rrrrrrL)NrN) rrrrrr rrrKrrr)NNNr)rrrHr rrrIrJrrrr rKrrrGrrrrrr rr"rrrN) rrr_r rr"rKrrr rHrr_r rr"rKrrr rrr_r rr"rKrrr)rrL)NN)rrrr"rKrrr)r@rArBrCrkpropertyrrrrr classmethodrrrrrrrrr __classcell__rs@rrGrG`s   -1^%^%^%^%^%^%^%@   X WWWW    $$$$+/'+ 66666p 59,0,0[<*IM*BF     4#' """""JBF#####J    $(#'$$$$$L222X22222rrGcheZdZdZdgfdfd Z dd dZ d!d"dZ d!d#dZed$dZ xZ S)%MyScaleWithoutJSONzsMyScale vector store without metadata column This is super handy if you are working to a SQL-native table NrHr rIrJmust_have_colsrrKrrrLc Ltj||fi|||_dS)agBuilding a myscale vector store without metadata column embedding (Embeddings): embedding model config (MyScaleSettings): Configuration to MyScale Client must_have_cols (List[str]): column names to be included in query Other keyword arguments will pass into [clickhouse-connect](https://docs.myscale.com/) N)rjrkr)r7rHrIrrKrs rrkzMyScaleWithoutJSON.__init__s4 F55f555)7rrrrr rr"rcHdtt|}|rd|}nd}d|jjddd|jd|jjd|jjd |d |jjd d |d |jd|d }|S)NrZrrbrr)z, dist, z FROM rcrrr*rrr) rqrrrIr,rr.r0rwrs rrzMyScaleWithoutJSON._build_qstrsHHSe__--  /I//IIIK*62$-..+&*.):  ${5h?  EN    rrr_rc L|||} fdj|DS#t$rG}t dt|dt|dgcYd}~Sd}~wwxYw)rcg|]:tjjdfdjD;S)r)c"i|] }|| Srrr^r_rs r zMMyScaleWithoutJSON.similarity_search_by_vector...AsCCC!a1CCCrrr rIr,rrs @rrazBMyScaleWithoutJSON.similarity_search_by_vector..>sg  !"4;#9&#A!BCCCCt/BCCCrrrrNrrs` rrz.MyScaleWithoutJSON.similarity_search_by_vector%rrrrc |j|||} fdj|DS#t $rG}tdt|dt|dgcYd}~Sd}~wwxYw)rcg|]BtjjdfdjDdfCS)r)c"i|] }|| Srrrs rrzYMyScaleWithoutJSON.similarity_search_with_relevance_scores...ds!G!G!Ga!QqT!G!G!Grrrrrs @rrazNMyScaleWithoutJSON.similarity_search_with_relevance_scores..`su    %&t{'=f'E%F!G!G!G!G43F!G!G!GfI    rrrrNrrs` rrz:MyScaleWithoutJSON.similarity_search_with_relevance_scoresIrrcdS)Nrbrrs rrz"MyScaleWithoutJSON.metadata_columnnsrr) rHr rIrJrrrKrrrLr5rrr r r) r@rArBrCrkrrrr rr rs@rrrs-1$& 8888888&IM.#' """""JBF#####JXrr)rrrrrr) __future__rrlogginghashlibr threadingrtypingrrrr r r langchain_core.documentsr langchain_core.embeddingsr langchain_core.vectorstoresrpydantic_settingsrr getLoggerrmrrrGrrrrr(s"""""" ================------000000333333>>>>>>>>         <<<<<l<<<~V2V2V2V2V2kV2V2V2r wwwwwwwwwwr