NgfddlmZddlZddlmZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZGd d eZdS) ) annotationsN)Path)AnyDictIterableListOptional)CallbackManagerForRetrieverRunDocument) BaseRetriever) ConfigDictceZdZUdZdZded< ded< dZded< dZd ed < ed Z e d)d*dZ e ddd+dZ d,dZ d-d.d#Ze d$dd%d/d(ZdS)0TFIDFRetrieverz`TF-IDF` retriever. Largely based on https://github.com/asvskartheek/Text-Retrieval/blob/master/TF-IDF%20Search%20Engine%20(SKLEARN).ipynb Nr vectorizerList[Document]docs tfidf_arrayintkT)arbitrary_types_allowedtexts Iterable[str] metadatasOptional[Iterable[dict]] tfidf_paramsOptional[Dict[str, Any]]kwargsreturnc  ddlm}n#t$rtdwxYw|pi}|di|}||}|p d|D}dt ||D}|d|||d|S)Nr)TfidfVectorizerzNCould not import scikit-learn, please install with `pip install scikit-learn`.c3K|]}iVdSN).0_s `/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/retrievers/tfidf.py z,TFIDFRetriever.from_texts..4s"!4!4"!4!4!4!4!4!4c6g|]\}}t||S) page_contentmetadatar )r&tms r( z-TFIDFRetriever.from_texts..5s)WWWAa!444WWWr*rrrr%)sklearn.feature_extraction.textr" ImportError fit_transformzip) clsrrrrr"rrrs r( from_textszTFIDFRetriever.from_texts!s  G G G G G G G   !   $)r $_44|44  ..u55 4!4!4e!4!4!4 WWUIAVAVWWWsWjtWWPVWWWs #)r documentsIterable[Document]c Ptd|D\}}|jd|||d|S)Nc32K|]}|j|jfVdSr$r,)r&ds r(r)z0TFIDFRetriever.from_documents..@s+ Q Q!!.!*!= Q Q Q Q Q Qr*)rrrr%)r6r8)r7r9rrrrs r(from_documentszTFIDFRetriever.from_documents8sQ Q Qy Q Q QRys~ li  KQ   r*querystr run_managerr cddlm}j|g}|j|d}fd|j ddddD}|S)Nr)cosine_similarity)c*g|]}j|Sr%)r)r&iselfs r(r1z:TFIDFRetriever._get_relevant_documents..PsPPPty|PPPr*rD)sklearn.metrics.pairwiserCr transformrreshapeargsortr)rGr?rArC query_vecresults return_docss` r(_get_relevant_documentsz&TFIDFRetriever._get_relevant_documentsEs ?>>>>>O-- G  $#D$4i@@HH   QPPPW__->->wyy-I$$B$-OPPP r*tfidf_vectorizer folder_path file_nameNonecz ddl}n#t$rtdwxYwt|}|dd||j||dz t ||dz d5}tj|j|j f|ddddS#1swxYwYdS)NrBCould not import joblib, please install with `pip install joblib`.T)exist_okparents.joblib.pklwb) joblibr4rmkdirdumpropenpicklerr)rGrQrRr[pathfs r( save_localzTFIDFRetriever.save_localSs*   MMMM   T   K   D$ ///  DOTy,A,A,A%ABBB$I++++T 2 2 :a KD$45q 9 9 9 : : : : : : : : : : : : : : : : : :s!"B00B47B4F)allow_dangerous_deserializationrRrcboolcd ddl}n#t$rtdwxYw|stdt|}|||dz }t ||dz d5}t j|\}} dddn #1swxYwY|||| S) aLoad the retriever from local storage. Args: folder_path: Folder path to load from. allow_dangerous_deserialization: Whether to allow dangerous deserialization. Defaults to False. The deserialization relies on .joblib and .pkl files, which can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine. You will need to set this to `True` to use deserialization. If you do this, make sure you trust the source of the file. file_name: File name to load from. Defaults to "tfidf_vectorizer". Returns: TFIDFRetriever: Loaded retriever. rNrUaThe de-serialization of this retriever is based on .joblib and .pkl files.Such files can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine.You will need to set `allow_dangerous_deserialization` to `True` to load this retriever. If you do this, make sure you trust the source of the file, and you are responsible for validating the file came from a trusted source.rXrYrbr2)r[r4 ValueErrorrloadr^r_) r7rQrcrRr[r`rrarrs r( load_localzTFIDFRetriever.load_localis20  MMMM   T   / .   K  [[9(=(=(=!=>> $I++++T 2 2 /a!' A D+ / / / / / / / / / / / / / / / sjtMMMMs!5BB B)NN) rrrrrrrrr r)r9r:rrrrr r)r?r@rAr r r)rP)rQr@rRr@r rS)rQr@rcrdrRr@r r)__name__ __module__ __qualname____doc__r__annotations__rrr model_config classmethodr8r>rOrbrir%r*r(rr sO JK AJJJJ(: $L/315 XXXX[X, 26      [      ",:::::, 16+ 5N5N5N5N5N[5N5N5Nr*r) __future__rr_pathlibrtypingrrrrr langchain_core.callbacksr langchain_core.documentsr langchain_core.retrieversr pydanticrrr%r*r(rxs"""""" 66666666666666CCCCCC------333333RNRNRNRNRN]RNRNRNRNRNr*