NgK,ddlmZddlZddlZddlZddlZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZmZejeZdZdZd Zd Zd ZGd d eZdS)) annotationsN)AnyIterableListOptionalType)Document) Embeddings)VST VectorStore embeddingidtext embeddingssimilarity_scorec|eZdZdZdeeeedd$d Ze d%dZ d&d'dZ d(d)dZ e d&d*d!Zd+d#ZdS),DuckDBa; `DuckDB` vector store. This class provides a vector store interface for adding texts and performing similarity searches using DuckDB. For more information about DuckDB, see: https://duckdb.org/ This integration requires the `duckdb` Python package. You can install it with `pip install duckdb`. *Security Notice*: The default DuckDB configuration is not secure. By **default**, DuckDB can interact with files across the entire file system, which includes abilities to read, write, and list files and directories. It can also access some python variables present in the global namespace. When using this DuckDB vectorstore, we suggest that you initialize the DuckDB connection with a secure configuration. For example, you can set `enable_external_access` to `false` in the connection configuration to disable external access to the DuckDB connection. You can view the DuckDB configuration options here: https://duckdb.org/docs/configuration/overview.html Please review other relevant security considerations in the DuckDB documentation. (e.g., "autoinstall_known_extensions": "false", "autoload_known_extensions": "false") See https://python.langchain.com/docs/security for more information. Args: connection: Optional DuckDB connection embedding: The embedding function or model to use for generating embeddings. vector_key: The column name for storing vectors. Defaults to `embedding`. id_key: The column name for storing unique identifiers. Defaults to `id`. text_key: The column name for storing text. Defaults to `text`. table_name: The name of the table to use for storing embeddings. Defaults to `embeddings`. Example: .. code-block:: python import duckdb conn = duckdb.connect(database=':memory:', config={ # Sample configuration to restrict some DuckDB capabilities # List is not exhaustive. Please review DuckDB documentation. "enable_external_access": "false", "autoinstall_known_extensions": "false", "autoload_known_extensions": "false" } ) embedding_function = ... # Define or import your embedding function here vector_store = DuckDB(conn, embedding_function) vector_store.add_texts(['text1', 'text2']) result = vector_store.similarity_search('text1') N) connection vector_keyid_keytext_key table_namer Optional[Any]r r rstrrrrc ddl}n#t$rtdwxYw||_||_||_||_||_||_|jtd|tj d|p|j dddi |_ | |j |j|_dS) z?Initialize with DuckDB connection and setup for vector storage.rNzMCould not import duckdb package. Please install it with `pip install duckdb`.z0An embedding function or model must be provided.a?No DuckDB connection provided. A new connection will be created.This connection is running in memory and no data will be persisted.To persist data, specify `connection=duckdb.connect(...)` when using the API. Please review the documentation of the vectorstore for security recommendations on configuring the connection.z:memory:enable_external_accessfalse)databaseconfig)duckdb ImportError _embedding _vector_key_id_key _text_key _table_name ValueErrorwarningswarnconnect _connection _ensure_tabletable_table)selfrr rrrrr s c/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/vectorstores/duckdb.py__init__zDuckDB.__init__Ts  MMMM   ?    #% !% ? "OPP P   MJ   & )<)<)A7(K*=* *  &,,T-=>> s!returnOptional[Embeddings]c|jS)z6Returns the embedding object used by the vector store.)r")r/s r0rzDuckDB.embeddingss texts Iterable[str] metadatasOptional[List[dict]]kwargsr List[str]c 2d} ddl}d}n*#t$rtdYnwxYw|dd|D}|p d|D}|jt|}g}t|D]\} } || } |r-| t|krtj || nd} |r3| |j || |j| |j| d | is|jd |jd || | | | g|rX|j|} |jd | |jd |jd |S)aTurn texts into embedding and add it to the database using Pandas DataFrame Args: texts: Iterable of strings to add to the vectorstore. metadatas: Optional list of metadatas associated with the texts. kwargs: Additional parameters including optional 'ids' to associate with the texts. Returns: List of ids of the added texts. FrNTzgUnable to import pandas. Install it with `pip install -U pandas` to improve performance of add_texts().idscNg|]"}ttj#Sruuiduuid4.0_s r0 z$DuckDB.add_texts..s& B B BqTZ\\!2!2 B B Br5cNg|]"}ttj#Sr?r@rCs r0rFz$DuckDB.add_texts..s&777Ac$*,,''777r5metadataz INSERT INTO z VALUES (?,?,?,?)dfz SELECT * FROM df)pandasr!loggerinfopopr"embed_documentslist enumeratelenjsondumpsappendr$r%r#r+executer& DataFrame from_dictregister)r/r6r8r: have_pandaspdr=rdataidxrr rHrIs r0 add_textszDuckDB.add_textss"      KK    KK9      jj B BE B B BCC777777_44T%[[AA "5))  IC"3I!$s9~~!5!5 9S>***     c#h()"H  ((F4#3FFFXtY9  ''--B   % %dB / / /   $ $Bt/BBB    s  $22querykintList[Document]c 2j|}jdjjj|}jjj g| tg td |fdtt!DS)aPerforms a similarity search for a given query string. Args: query: The query string to search for. k: The number of similar texts to return. Returns: A list of Documents most similar to the query. list_cosine_similarity)excludez descc g|]q}tj|d|r?itjd|dtt|inirS)rHrE) page_contentrH)r r%rRloadsSIMILARITY_ALIAS)rDr\docsr/s r0rFz,DuckDB.similarity_search..s     !$.1#6  #C( jj!1#!677+(**D1A,B3,G        r5)r" embed_queryr FunctionExpressionColumnExpressionr#ConstantExpressionr.selectStarExpressionaliasriorderlimitfetchdfrangerQ)r/r_r`r:r rdrjs` @r0similarity_searchzDuckDB.similarity_searchsO//66 !%!?!? $ K ( ()9 : : K * *9 5 5" "  DK K..r.::*001ABB  U&--- . . U1XX WYY       SYY''    r5cls Type[VST]c T|dd}|dt}|dt}|dt}|dt} t |||||| } | j|fd|i|| S) aCreates an instance of DuckDB and populates it with texts and their embeddings. Args: texts: List of strings to add to the vector store. embedding: The embedding function or model to use for generating embeddings. metadatas: Optional list of metadata dictionaries associated with the texts. kwargs: Additional keyword arguments including: - connection: DuckDB connection. If not provided, a new connection will be created. - vector_key: The column name for storing vectors. Default "vector". - id_key: The column name for storing unique identifiers. Default "id". - text_key: The column name for storing text. Defaults to "text". - table_name: The name of the table to use for storing embeddings. Defaults to "embeddings". Returns: An instance of DuckDB with the provided texts and their embeddings added. rNrrrr)rr rrrrr8)getDEFAULT_VECTOR_KEYDEFAULT_ID_KEYDEFAULT_TEXT_KEYDEFAULT_TABLE_NAMErr]) rwr6r r8r:rrrrrinstances r0 from_textszDuckDB.from_textss:ZZ d33 ZZ .@AA Hn55::j*:;;ZZ .@AA !!!     5@@I@@@@r5Nonec d|jd|jd|jd|jd }|j|dS)z0Ensures the table for storing embeddings exists.z$ CREATE TABLE IF NOT EXISTS z ( z" VARCHAR PRIMARY KEY, z VARCHAR, z9 FLOAT[], metadata VARCHAR ) N)r&r$r%r#r+rU)r/create_table_sqls r0r,zDuckDB._ensure_table$sq $($4  \  ^          !122222r5) rrr r rrrrrrrr)r2r3)N)r6r7r8r9r:rr2r;)r^)r_rr`rar:rr2rb) rwrxr6r;r r r8r9r:rr2r)r2r)__name__ __module__ __qualname____doc__r{r|r}r~r1propertyrr]rv classmethodrr,r?r5r0rrs::~%),$(,)?)?)?)?)?)?VX+/BBBBBJ$%) ) ) ) ) V +/ ....[.` 3 3 3 3 3 3r5r) __future__rrRloggingrAr(typingrrrrrlangchain_core.documentsr langchain_core.embeddingsr langchain_core.vectorstoresr r getLoggerrrKr{r|r}r~rirr?r5r0rs""""""  66666666666666------00000088888888  8 $ $ !%W3W3W3W3W3[W3W3W3W3W3r5