Ng?CddlmZddlZddlZddlmZmZmZmZm Z m Z m Z m Z m Z mZddlmZddlmZddlmZerddlZGddeZdS) ) annotationsN) TYPE_CHECKINGAnyCallableDictIterableListOptionalTupleTypeUnion)Document) Embeddings) VectorStoreceZdZdZ dCdDd ZedEdZ dFdGdZ dHdIdZ dHdJd!Z dHdKd%Z dHdLd'Z dMd+Z dNd,Z dHdOd-Z dHdPd/Ze dFdQd5Ze dRdSd?ZdTdAZdUdBZdS)VMarqoa`Marqo` vector store. Marqo indexes have their own models associated with them to generate your embeddings. This means that you can selected from a range of different models and also use CLIP models to create multimodal indexes with images and text together. Marqo also supports more advanced queries with multiple weighted terms, see See https://docs.marqo.ai/latest/#searching-using-weights-in-queries. This class can flexibly take strings or dictionaries for weighted queries in its similarity search methods. To use, you should have the `marqo` python package installed, you can do this with `pip install marqo`. Example: .. code-block:: python import marqo from langchain_community.vectorstores import Marqo client = marqo.Client(url=os.environ["MARQO_URL"], ...) vectorstore = Marqo(client, index_name) Nclient marqo.Client index_namestradd_documents_settingsOptional[Dict[str, Any]]searchable_attributesOptional[List[str]]page_content_builder)Optional[Callable[[Dict[str, Any]], str]]c  ddl}n#t$rtdwxYwt||jst dt |||_||_|in||_||_ ||_ dg|_ d|_ dS)zInitialize with Marqo client.rNRCould not import marqo python package. Please install it with `pip install marqo`.z2client should be an instance of marqo.Client, got texti) marqo ImportError isinstanceClient ValueErrortype_client _index_name_add_documents_settings_searchable_attributesr tensor_fields_document_batch_size)selfrrrrrr s b/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/vectorstores/marqo.py__init__zMarqo.__init__4s  LLLL   >   &%,// ST&\\SS  %(0BB6L $'<#$8!$X$(!!!s!returnOptional[Embeddings]cdSNr,s r- embeddingszMarqo.embeddingsTsttexts Iterable[str] metadatasOptional[List[dict]]kwargsr List[str]c |j|j}d|vr|dds|drt dg}d}t |D]R\}}||rtj||ntjid} | | |dz }Sg} td||j D]}|j|jj ||||j zfd|j i|j} | dr d |d ||j zd } t| | d | d Dz } | S)aUpload texts with metadata (properties) to Marqo. You can either have marqo generate ids for each document or you can provide your own by including a "_id" field in the metadata objects. Args: texts (Iterable[str]): am iterator of texts - assumed to preserve an order that matches the metadatas. metadatas (Optional[List[dict]], optional): a list of metadatas. Raises: ValueError: if metadatas is provided and the number of metadatas differs from the number of texts. Returns: List[str]: The list of ids that were added. index_defaults!treat_urls_and_pointers_as_imageszMarqo.add_texts is disabled for multimodal indexes. To add documents with a multimodal index use the Python client for Marqo directly.r)rmetadatar*errorsz.Error in upload for documents in index range [,z], check Marqo logs.cg|] }|d S)_idr3).0items r- z#Marqo.add_texts..s>>>DDK>>>r6items)r&indexr' get_settingsgetr$ enumeratejsondumpsappendranger+ add_documentsr*r( RuntimeError) r,r7r9r;settings documentsnum_docsirdocidsresponseerr_msgs r- add_textszMarqo.add_textsXs0<%%d&677DDFF  ( ()*+NO )||?@@ )T +-  ''  GAt8AUDJy|444tzRT~~C   S ! ! ! MHHq(D$=>> ? ?AIt|))$*:;;I!a$";;;<"0.H ! ,)Q))444))) #7+++ >>HW,=>>> >CC r6queryUnion[str, Dict[str, float]]kintList[Document]c ^|||}||}|S)a{Search the marqo index for the most similar documents. Args: query (Union[str, Dict[str, float]]): The query for the search, either as a string or a weighted query. k (int, optional): The number of documents to return. Defaults to 4. Returns: List[Document]: k documents ordered from best to worst match. r^r`)marqo_similarity_search/_construct_documents_from_results_without_score)r,r^r`r;resultsrUs r-similarity_searchzMarqo.similarity_searchs5 ..Ua.@@HHQQ r6List[Tuple[Document, float]]c^|||}||}|S)aReturn documents from Marqo that are similar to the query as well as their scores. Args: query (str): The query to search with, either as a string or a weighted query. k (int, optional): The number of documents to return. Defaults to 4. Returns: List[Tuple[Document, float]]: The matching documents and their scores, ordered by descending score. rd)re,_construct_documents_from_results_with_score)r,r^r`rgscored_documentss r-similarity_search_with_scorez"Marqo.similarity_search_with_scores6"..Ua.@@LLWUUr6queries&Iterable[Union[str, Dict[str, float]]]List[List[Document]]c |||}g}|dD],}||}||-|S)aSearch the marqo index for the most similar documents in bulk with multiple queries. Args: queries (Iterable[Union[str, Dict[str, float]]]): An iterable of queries to execute in bulk, queries in the list can be strings or dictionaries of weighted queries. k (int, optional): The number of documents to return for each query. Defaults to 4. Returns: List[List[Document]]: A list of results for each query. rnr`result)marqo_bulk_similarity_searchrfrPr,rnr`r; bulk_resultsbulk_documentsrgrUs r-bulk_similarity_searchzMarqo.bulk_similarity_searchse&88A8NN /1#H- - -GLLWUUI  ! !) , , , ,r6"List[List[Tuple[Document, float]]]c |||}g}|dD],}||}||-|S)a5Return documents from Marqo that are similar to the query as well as their scores using a batch of queries. Args: query (Iterable[Union[str, Dict[str, float]]]): An iterable of queries to execute in bulk, queries in the list can be strings or dictionaries of weighted queries. k (int, optional): The number of documents to return. Defaults to 4. Returns: List[Tuple[Document, float]]: A list of lists of the matching documents and their scores for each query rrrs)rtrkrPrus r-!bulk_similarity_search_with_scorez'Marqo.bulk_similarity_search_with_scorese&88A8NN =?#H- - -GII'RRI  ! !) , , , ,r6rgDict[str, List[Dict[str, str]]]List[Tuple[Document, Any]]cg}|dD]{}|j |d}n||}tj|dd}|t |||df||S)Helper to convert Marqo results into documents. Args: results (List[dict]): A marqo results object with the 'hits'. include_scores (bool, optional): Include scores alongside documents. Defaults to False. Returns: Union[List[Document], List[Tuple[Document, float]]]: The documents or document score pairs if `include_scores` is true. hitsNrr@{} page_contentr@_scorerrNloadsrLrPrr,rgrUresrr@s r-rkz2Marqo._construct_documents_from_results_with_scores13 6?  C(06{0055z#''*d";";<  r60Dict[str, List[Dict[str, List[Dict[str, str]]]]]c,dfd|Di}|S)aReturn documents from Marqo using a bulk search, exposes Marqo's output directly Args: queries (Iterable[Union[str, Dict[str, float]]]): A list of queries. k (int, optional): The number of documents to return for each query. Defaults to 4. Returns: Dict[str, Dict[List[Dict[str, Dict[str, Any]]]]]: A bulk search results object rscg|]<}jj|j=S)rr)rFr^r`r,s r-rHz6Marqo.marqo_bulk_similarity_search..Ps^ ""4#344;;43NVW<r6r3)r,rnr`rvs` ` r-rtz"Marqo.marqo_bulk_similarity_search@sE %  r6cls Type[Marqo]rU embeddingUnion[Embeddings, None]c Rd|D}d|D}|j|fd|i|S)aReturn VectorStore initialized from documents. Note that Marqo does not need embeddings, we retain the parameter to adhere to the Liskov substitution principle. Args: documents (List[Document]): Input documents embedding (Any, optional): Embeddings (not required). Defaults to None. Returns: VectorStore: A Marqo vectorstore cg|] }|j Sr3)rrFds r-rHz(Marqo.from_documents..ms333A333r6cg|] }|j Sr3)r@rs r-rHz(Marqo.from_documents..ns333AQZ333r6r9) from_texts)rrUrr;r7r9s r-from_documentszMarqo.from_documentsZsI&4333333333 s~eCCyCFCCCr6http://localhost:8882Turlapi_key)Optional[Callable[[Dict[str, str]], str]]index_settingsverboseboolc   ddl} n#t$rtdwxYw|s ttj}| j||} ||| pi| rtd|dn%#t$r| rtd|d YnwxYw|||||pi| }| |||S) aReturn Marqo initialized from texts. Note that Marqo does not need embeddings, we retain the parameter to adhere to the Liskov substitution principle. This is a quick way to get started with marqo - simply provide your texts and metadatas and this will create an instance of the data store and index the provided data. To know the ids of your documents with this approach you will need to include them in under the key "_id" in your metadatas for each text Example: .. code-block:: python from langchain_community.vectorstores import Marqo datastore = Marqo(texts=['text'], index_name='my-first-index', url='http://localhost:8882') Args: texts (List[str]): A list of texts to index into marqo upon creation. embedding (Any, optional): Embeddings (not required). Defaults to None. index_name (str, optional): The name of the index to use, if none is provided then one will be created with a UUID. Defaults to None. url (str, optional): The URL for Marqo. Defaults to "http://localhost:8882". api_key (str, optional): The API key for Marqo. Defaults to "". metadatas (Optional[List[dict]], optional): A list of metadatas, to accompany the texts. Defaults to None. this is only used when a new index is being created. Defaults to "cpu". Can be "cpu" or "cuda". add_documents_settings (Optional[Dict[str, Any]], optional): Settings for adding documents, see https://docs.marqo.ai/0.0.16/API-Reference/documents/#query-parameters. Defaults to {}. index_settings (Optional[Dict[str, Any]], optional): Index settings if the index doesn't exist, see https://docs.marqo.ai/0.0.16/API-Reference/indexes/#index-defaults-object. Defaults to {}. Returns: Marqo: An instance of the Marqo vector store rNr)rr) settings_dictzCreated z successfully.zIndex z exists.)rrr) r r!ruuiduuid4r# create_indexprint Exceptionr\)rr7rr9rrrrrrrrr;r rinstances r-rzMarqo.from_textsqsFt  LLLL   >    +TZ\\**J#w777 5    .:NB  O O O =;;;;<<< 5 5 5 53z333444 5#  "7#9#?R!5     5),,,s!.BB*)B*List[Dict[str, str]]c@|jdS)zHelper to see your available indexes in marqo, useful if the from_texts method was used without an index name specified Returns: List[Dict[str, str]]: The list of indexes rg)r& get_indexesr4s r-rzMarqo.get_indexess|'')))44r6cp|j|jdS)zvHelper to see the number of documents in the index Returns: int: The number of documents numberOfDocuments)r&rJr' get_statsr4s r-get_number_of_documentszMarqo.get_number_of_documentss/ |!!$"233==??@STTr6)NNN) rrrrrrrrrr)r/r0r2)r7r8r9r:r;rr/r<)r])r^r_r`rar;rr/rb)r^r_r`rar/ri)rnror`rar;rr/rp)rnror`rar;rr/ry)rgr|r/r})rgr|r/rb)r^r_r`rar/r|)rnror`rar/r) rrrUrbrrr;rr/r) NNrrrNNNNT)r7r<rrr9r:rrrrrrrrrrrrrrrrr;rr/r)r/r)r/ra)__name__ __module__ __qualname____doc__r.propertyr5r\rhrmrxr{rkrfrert classmethodrrrrr3r6r-rrs :<@59JN )))))@X +/>>>>>F0     2<668(IJ4.2DDDD[D,*.*;?59JN37VVVV[Vp5555UUUUUUr6r) __future__rrNrtypingrrrrrr r r r r langchain_core.documentsrlangchain_core.embeddingsrlangchain_core.vectorstoresrr rr3r6r-rs2""""""                         .-----000000333333LLLUUUUUKUUUUUr6