NgdddlmZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZddlmZddlmZddlmZdd lmZmZejZddZGddeZGddeZdS)) annotationsN)sha1)Thread)AnyDictIterableListOptionalTupleUnion)Document) Embeddings) VectorStore) BaseSettingsSettingsConfigDictsstrargsrreturnboolc|D] }||vrdS dS)z Check if a string contains multiple substrings. Args: s: string to check. *args: substrings to check. Returns: True if all substrings are in the string, False otherwise. FT)rras g/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/vectorstores/clickhouse.pyhas_mul_sub_strrs- A::55  4ceZdZUdZdZded<dZded<dZd ed <dZd ed <d Z d ed<dZ d ed<ddgZ ded<iZ ded<ddddddZ ded<dZded<d Zded!<d"Zded#<d-d'Zed(d)d*d+,ZdS).ClickhouseSettingsa`ClickHouse` client configuration. Attribute: host (str) : An URL to connect to MyScale backend. Defaults to 'localhost'. port (int) : URL port to connect with HTTP. Defaults to 8443. username (str) : Username to login. Defaults to None. password (str) : Password to login. Defaults to None. secure (bool) : Connect to server over secure connection. Defaults to False. index_type (str): index type string. index_param (list): index build parameter. index_query_params(dict): index query parameters. database (str) : Database name to find the table. Defaults to 'default'. table (str) : Table name to operate on. Defaults to 'vector_table'. metric (str) : Metric to compute distance, supported are ('angular', 'euclidean', 'manhattan', 'hamming', 'dot'). Defaults to 'angular'. https://github.com/spotify/annoy/blob/main/src/annoymodule.cc#L149-L169 column_map (Dict) : Column type map to project column name onto langchain semantics. Must have keys: `text`, `id`, `vector`, must be same size to number of columns. For example: .. code-block:: python { 'id': 'text_id', 'uuid': 'global_unique_id' 'embedding': 'text_embedding', 'document': 'text_plain', 'metadata': 'metadata_dictionary_in_json', } Defaults to identity map. localhostrhostiintportN Optional[str]usernamepasswordFrsecureannoy index_typez 'L2Distance'dzOptional[Union[List, Dict]] index_paramzDict[str, str]index_query_paramsiduuiddocument embeddingmetadata)r,r-r.r/r0 column_mapdefaultdatabase langchaintableangularmetricitemrrc"t||SN)getattr)selfr8s r __getitem__zClickhouseSettings.__getitem___stT"""rz.envutf-8 clickhouse_ignore)env_fileenv_file_encoding env_prefixextra)r8rrr)__name__ __module__ __qualname____doc__r __annotations__r"r$r%r&r(r*r+r1r3r5r7r=r model_configrrrrr!sJ""HDD"H"""""H""""F 'J''''0>/DKDDDD)+++++  ""JHEF####&%!  LLLrrceZdZdZ d8d9fd Zd:d;dZeddZ d?dZ d@dAd$Z e dBdCd'Z dDd(Z d8dEd-Z dFdGd2Z dFdHd3Z dFdId5ZdJd6ZedDd7ZxZS)K Clickhouseu ClickHouse vector store integration. Setup: Install ``langchain_community`` and ``clickhouse-connect``: .. code-block:: bash pip install -qU langchain_community clickhouse-connect Key init args — indexing params: embedding: Embeddings Embedding function to use. Key init args — client params: config: Optional[ClickhouseSettings] ClickHouse client configuration. Instantiate: .. code-block:: python from langchain_community.vectorstores import Clickhouse, ClickhouseSettings from langchain_openai import OpenAIEmbeddings settings = ClickhouseSettings(table="clickhouse_example") vector_store = Clickhouse(embedding=OpenAIEmbeddings(), config=settings) Add Documents: .. code-block:: python from langchain_core.documents import Document document_1 = Document(page_content="foo", metadata={"baz": "bar"}) document_2 = Document(page_content="thud", metadata={"bar": "baz"}) document_3 = Document(page_content="i will be deleted :(") documents = [document_1, document_2, document_3] ids = ["1", "2", "3"] vector_store.add_documents(documents=documents, ids=ids) Delete Documents: .. code-block:: python vector_store.delete(ids=["3"]) # TODO: Fill out example output. Search: .. code-block:: python results = vector_store.similarity_search(query="thud",k=1) for doc in results: print(f"* {doc.page_content} [{doc.metadata}]") .. code-block:: python # TODO: Example output # TODO: Fill out with relevant variables and example output. Search with filter: .. code-block:: python # TODO: Edit filter if needed results = vector_store.similarity_search(query="thud",k=1,filter="metadata.baz='bar'") for doc in results: print(f"* {doc.page_content} [{doc.metadata}]") .. code-block:: python # TODO: Example output # TODO: Fill out with example output. Search with score: .. code-block:: python results = vector_store.similarity_search_with_score(query="qux",k=1) for doc, score in results: print(f"* [SIM={score:3f}] {doc.page_content} [{doc.metadata}]") .. code-block:: python # TODO: Example output # TODO: Fill out with example output. Async: .. code-block:: python # add documents # await vector_store.aadd_documents(documents=documents, ids=ids) # delete documents # await vector_store.adelete(ids=["3"]) # search # results = vector_store.asimilarity_search(query="thud",k=1) # search with score results = await vector_store.asimilarity_search_with_score(query="qux",k=1) for doc,score in results: print(f"* [SIM={score:3f}] {doc.page_content} [{doc.metadata}]") .. code-block:: python # TODO: Example output # TODO: Fill out with example output. Use as Retriever: .. code-block:: python retriever = vector_store.as_retriever( search_type="mmr", search_kwargs={"k": 1, "fetch_k": 2, "lambda_mult": 0.5}, ) retriever.invoke("thud") .. code-block:: python # TODO: Example output Nr/rconfigOptional[ClickhouseSettings]kwargsrrNonec ` ddlm}n#t$rtdwxYw ddlm}||_n#t$r d|_YnwxYwt |||_nt|_|jsJ|jj r |jj sJ|jj r$|jj r|jj r |jjsJdD]}||jj vsJ|jjdvsJt|d }t#|jjt&rI|jjr;d d |jjDnUd nSt#|jjt,r)d d |jjDn |jj}||||_||_d|_d|_||_d|_|d|jj |jj |jj|jj|jj d||_! |j!"dn?#tF$r2} tH%d|j!j&dYd} ~ nd} ~ wwxYw|j!"d|jj'r(|j!"d|jj'd|j!"|jdS)aNClickHouse Wrapper to LangChain Args: embedding_function (Embeddings): embedding function to use config (ClickHouseSettings): Configuration to ClickHouse Client kwargs (any): Other keyword arguments will pass into [clickhouse-connect](https://docs.clickhouse.com/) r) get_clientzlCould not import clickhouse connect python package. Please install it with `pip install clickhouse-connect`.)tqdmc|Sr:r)xrOs rz%Clickhouse.__init__..sQrN)r,r/r.r0r-)r6 euclidean manhattanhammingdottest,c&g|]\}}d|d|dS)'=r).0kvs r z'Clickhouse.__init__..s*SSS41a,a,,!,,,SSSrc,g|]}t|Sr)r)r`ps rrcz'Clickhouse.__init__.. sBBBQ#a&&BBBr\)rgr^ASC)r r"r$r%r&z"SET allow_experimental_json_type=1zClickhouse version=z6 - There is no allow_experimental_json_type parameter.z$SET allow_experimental_object_type=1zSET allow_experimental_z_index=1r)(clickhouse_connectrR ImportErrorrSpgbarsuper__init__rMrr r"r1r3r5r7len embed_query isinstancer*rjoinitemsr _schemaschemadimBS must_escapeembedding_function dist_orderr$r%r&clientcommand Exceptionloggerdebugserver_versionr() r<r/rMrOrRrSraru index_params_ __class__s rrmzClickhouse.__init__s  5 5 5 5 5 5 5   K    / ! ! ! ! ! !DJJ / / /..DJJJ /    DKK,..DK{{4DK$4444 K "  $  !  "    E / /A ......{!&     )''//00$+1488 ;*SS4;3J3P3P3R3RSSSTTTR dk5t<<-BB$+*ABBBCCC[, ll3 55 &"+!j !![)[);%         K   D E E E E    LLFdk&@FFF          BCCC ; !  K  J$+*@JJJ    DK(((((s/ $ 6A  A 'J J> (J99J>rdrur!rr#rc|jjrd|jjd|jjd|jjdd|jjdd|jjdd|jjd d |jjd d |jjdd |d|jjdd|jjd|dSd|jjd|jjd|jjdd|jjdd|jjdd|jjd d|jjd d|jjdd |dS)aCreate table schema :param dim: dimension of embeddings :param index_params: parameters used for index This function returns a `CREATE TABLE` statement based on the value of `self.config.index_type`. If an index type is specified that index will be created, otherwise no index will be created. In the case of there being no index, a linear scan will be performed when the embedding field is queried. z# CREATE TABLE IF NOT EXISTS .z( r,z Nullable(String), r.r/z Array(Float32), r0z JSON, r-zb UUID DEFAULT generateUUIDv4(), CONSTRAINT cons_vec_len CHECK length( z) = z, INDEX vec_idx z TYPE (zg) GRANULARITY 1000 ) ENGINE = MergeTree ORDER BY uuid SETTINGS index_granularity = 8192 z+ CREATE TABLE IF NOT EXISTS z( z' Nullable(String), z% Array(Float32), z JSON, zY UUID DEFAULT generateUUIDv4(), CONSTRAINT cons_vec_len CHECK length(zD ) ENGINE = MergeTree ORDER BY uuid )rMr(r3r5r1)r<rurs rrszClickhouse._schemaHs ; !   $(K$8   ;?;;L   [ #D )   [ #J /   [ #K 0   [ #J /   [ #F +   ' 4   ;>    ;1+>        #/     ,0K,@  CG;CT  [+D1  [+J7  [+K8   [+J7   [+F3   .{;  BE    rc|jS)aProvides access to the embedding mechanism used by the Clickhouse instance. This property allows direct access to the embedding function or model being used by the Clickhouse instance to convert text documents into embedding vectors for vector similarity search. Returns: The `Embeddings` instance associated with this Clickhouse instance. )rxr<s r embeddingszClickhouse.embeddingsps &&rvaluecFdfd|DS)aEscape special characters in a string for Clickhouse SQL queries. This method is used internally to prepare strings for safe insertion into SQL queries by escaping special characters that might otherwise interfere with the query syntax. Args: value: The string to be escaped. Returns: The escaped string, safe for insertion into SQL queries. rdc3DK|]}|jvr j|n|VdSr:)rwrv)r`cr<s r z(Clickhouse.escape_str..s>VV1!t/?*?*?$'1QVVVVVVr)rq)r<rs` r escape_strzClickhouse.escape_str}s-wwVVVVPUVVVVVVrtransacr column_names Iterable[str]c  d|}g}|D]<}dfd|D}|d|d=djjdjjd|dd|d }|S) aConstruct an SQL query for inserting data into the Clickhouse database. This method formats and constructs an SQL `INSERT` query string using the provided transaction data and column names. It is utilized internally during the process of batch insertion of documents and their embeddings into the database. Args: transac: iterable of tuples, representing a row of data to be inserted. column_names: iterable of strings representing the names of the columns into which data will be inserted. Returns: A string containing the constructed SQL `INSERT` query. r\c\g|](}dt|d)S)r^)rr)r`_nr<s rrcz0Clickhouse._build_insert_sql..s7FFFb9dooc"gg66999FFFrr)z8 INSERT INTO TABLE rz)) VALUES z )rqappendrMr3r5)r<rrks_datani_strs` r_build_insert_sqlzClickhouse._build_insert_sqls XXl # # # #AFFFFAFFFGGA LLQ " " " "[),0K,=@B%   rcf|||}|j|dS)a/Execute an SQL query to insert data into the Clickhouse database. This method performs the actual insertion of data into the database by executing the SQL query constructed by `_build_insert_sql`. It's a critical step in adding new documents and their associated data into the vector store. Args: transac:iterable of tuples, representing a row of data to be inserted. column_names: An iterable of strings representing the names of the columns into which data will be inserted. N)rrzr{)r<rr _insert_querys r_insertzClickhouse._inserts5..w EE  M*****r texts metadatasOptional[List[dict]] batch_sizeidsOptional[Iterable[str]] List[str]c |p d|D}|jj}g}|d||d||d|jt |i}|p d|D}t t j|||d<tt|t|z dksJt| \} } d} | t| d t| D]} t| | |jjd|jksJ|| t||krD| r| t#|j|| g } | g}t|dkr,| r| ||| d |DS#t($rG} t*d t/| dt1| dgcYd} ~ Sd} ~ wwxYw)aInsert more texts through the embeddings and add to the VectorStore. Args: texts: Iterable of strings to add to the VectorStore. ids: Optional list of ids to associate with the texts. batch_size: Batch size of insertion metadata: Optional column data to be inserted Returns: List of ids from adding the texts into the VectorStore. cvg|]6}t|d7S)r>)rencode hexdigest)r`ts rrcz(Clickhouse.add_texts..s8IIIad188G,,--7799IIIrr,r.r/cg|]}iSrr)r`rs rrcz(Clickhouse.add_texts..s!4!4!4"!4!4!4rr0rNzInserting data...)desctotal)targetrcg|]}|Srr)r`is rrcz(Clickhouse.add_texts..s###!A###r   )rMr1rxembed_documentslistmapjsondumpsrnsetziprrrkindexrurrqrrstartr|r}errortyper)r<rrrrrOcolmap_rrkeysvaluesrrbes r add_textszClickhouse.add_textssj*III5III+( DM3 J  K $"9"I"I$u++"V"V 4!4!4e!4!4!4 ,/ I,F,F WZ()3w<<#l"3"33449999L..001 f AZZV #6c)nn  ! !$**T[%;K%HIIJKKtxWWWWq!!!w<<:--!dl'4IIIAGGIII G7||aFFHHH Wd+++##s### #    LLS477SSCFFSSS T T TIIIIII sD*H IDK( > >4;+< > >  -//  A TAfITTT1V9TTTT EE   rq_emb List[float]topk where_strcdtt|}|rd|}nd}g}|jjr:|jjD]-}|d|d|jj|.d|jjdd|jjd d |jjd |jjd |d |jjdd|d|j d|dd|d }|S)aConstruct an SQL query for performing a similarity search. This internal method generates an SQL query for finding the top-k most similar vectors in the database to a given query vector.It allows for optional filtering conditions to be applied via a WHERE clause. Args: q_emb: The query vector as a list of floats. topk: The number of top similar items to retrieve. where_str: opt str representing additional WHERE conditions for the query Defaults to None. Returns: A string containing the SQL query for the similarity search. r\z PREWHERE rdzSETTING r_z SELECT r.z, r0z, dist FROM rz z! ORDER BY L2Distance(r/z, [z]) AS dist z LIMIT  ) rqrrrMr+rr1r3r5ry)r<rrr q_emb_str settings_strsraq_strs r_build_query_sqlzClickhouse._build_query_sql$sr$HHSe__--  /I//III ; ) Y[3 Y Y$$%W%W%WDK4RST4U%W%WXXXXK*:6' 3+&*.):  "&!7 !D  JS   ((=11 rrraList[Document]c R|j|j|||fi|S)aPerform a similarity search with ClickHouse Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of Documents )similarity_search_by_vectorrxro)r<rrarrOs rsimilarity_searchzClickhouse.similarity_searchKs@&0t/  # / / 6 69  HN   rc L|||} fdj|DS#t$rG}t dt|dt|dgcYd}~Sd}~wwxYw)aPerform a similarity search with ClickHouse by vectors Args: query (str): query string k (int, optional): Top K neighbors to retrieve. Defaults to 4. where_str (Optional[str], optional): where condition string. Defaults to None. NOTE: Please do not let end-user to fill this and always be aware of SQL injection. When dealing with metadatas, remember to use `{self.metadata_column}.attribute` instead of `attribute` alone. The default name for it is `metadata`. Returns: List[Document]: List of documents cg|]?}t|jjd|jjd@S)r.r0 page_contentr0r rMr1r`rr<s rrcz:Clickhouse.similarity_search_by_vector..{s^  !"4;#9*#E!Ft{5jABrrrrN) rrzrrr|r}rrr)r<r/rarrOrrs` rrz&Clickhouse.similarity_search_by_vectorbs.%%iI>>  **511??AA      LLS477SSCFFSSS T T TIIIIII s7A B#.sl    %&t{'=j'I%J!"4;#9*#E!FfI    rrrrN) rrxrorzrrr|r}rrr)r<rrarrOrrs` r'similarity_search_with_relevance_scoresz2Clickhouse.similarity_search_with_relevance_scoress&%%  # / / 6 69       **511??AA        LLS477SSCFFSSS T T TIIIIII s7A** B;4rsf"""""" DDDDDDDDDDDDDDDDDD------000000333333>>>>>>>>         FFFFFFFFRK 2K 2K 2K 2K 2K 2K 2K 2K 2K 2r