Nga)PdZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZddlmZmZddlmZdd lmZmZdd lmZdd lmZmZdd lm Z m!Z!dd l"m#Z#edZ$edeZ%de e$de e$ge%fde e$fdZ&GddeZ'dS)zo Ensemble retriever that ensemble the results of multiple retrievers by using weighted Reciprocal Rank Fusion N) defaultdict)Hashable)chain) AnyCallableDictIterableIteratorListOptionalTypeVarcast)#AsyncCallbackManagerForRetrieverRunCallbackManagerForRetrieverRun)Document) BaseRetriever RetrieverLike)RunnableConfig) ensure_config patch_config)ConfigurableFieldSpecget_unique_config_specs)model_validatorTH)bounditerablekeyreturnc#Kt}|D]*}||x}|vr|||V+dS)a Yield unique elements of an iterable based on a key function. Args: iterable: The iterable to filter. key: A function that returns a hashable key for each element. Yields: Unique elements of the iterable based on the key function. N)setadd)rrseeneks Y/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain/retrievers/ensemble.py unique_by_keyr'(sZ 55D QKA $ $ HHQKKKGGGc ,eZdZUdZeeed<eeed<dZe ed<dZ e e ed<e deefd Zed ed ee efdefd Z dde de ededeefdZ dde de ededeefdZde dedeefdZde dedeefdZddde dede edeefdZddde dede edeefdZdeeedeefdZdS)EnsembleRetrieveraeRetriever that ensembles the multiple retrievers. It uses a rank fusion. Args: retrievers: A list of retrievers to ensemble. weights: A list of weights corresponding to the retrievers. Defaults to equal weighting for all retrievers. c: A constant added to the rank, controlling the balance between the importance of high-ranked items and the consideration given to lower-ranked items. Default is 60. id_key: The key in the document's metadata used to determine unique documents. If not specified, page_content is used. retrieversweights<cNid_keyrc>td|jDS)z+List configurable fields for this runnable.c3.K|]}|jD]}|VdSN) config_specs).0 retrieverspecs r& z1EnsembleRetriever.config_specs..QsJ' ' i>T' ' 6:D' ' ' ' ' ' ' r()rr+)selfs r&r3zEnsembleRetriever.config_specsNs4'' ' "&/' ' '    r(before)modevaluescr|ds!t|d}d|z g|z|d<|S)Nr,r+)getlen)clsr; n_retrieverss r& set_weightszEnsembleRetriever.set_weightsUsGzz)$$ Bvl344L!"\!1 2\ AF9  r(inputconfigkwargsc *ddlm}t|}||dd|dd|dg|j|di|j}|jd|fd |d p|i|} | ||| }|j |fi||S#t$r}| ||d}~wwxYw) Nr)CallbackManager callbacksverboseFtagsmetadatarIinheritable_tags local_tagsinheritable_metadatalocal_metadatanamerun_name run_managerrD) langchain_core.callbacksrGr configurer>rJrKon_retriever_startget_name rank_fusionon_retriever_end Exceptionon_retriever_error) r8rCrDrErGcallback_managerrTresultr$s r&invokezEnsembleRetriever.invoke]se =<<<<<v&&*44 JJ{ # # JJy%00#ZZ33y!'J!;!;=5  :&9    J'':4==??    %%eV%TTF )K (     M     * *1 - - -G sC,, D6D  Dc ^Kddlm}t|}||dd|dd|dg|j|di|j}|jd|fd |d p|i|d{V} | ||| d{V}|j |fi|d{V|S#t$r"}| |d{V|d}~wwxYw) Nr)AsyncCallbackManagerrHrIFrJrKrLrQrRrS) rUrarrVr>rJrKrWrX arank_fusionrZr[r\) r8rCrDrErar]rTr^r$s r&ainvokezEnsembleRetriever.ainvoke~s BAAAAAv&&/99 JJ{ # # JJy%00#ZZ33y!'J!;!;=:  @,?    J'':4==??          ,,;v-F/+.       M   0033 3 3 3 3 3 3 3G s D D, D''D,queryrTc2|||}|S)z Get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. )rYr8rdrTfused_documentss r&_get_relevant_documentsz)EnsembleRetriever._get_relevant_documentss"**5+>>r(cBK|||d{V}|S)z Asynchronously get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. N)rbrfs r&_aget_relevant_documentsz*EnsembleRetriever._aget_relevant_documentss5"!% 1 1% E EEEEEEEr()rDcfdt|jD}tt|D]}d||D||<||}|S)z Retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. c g|]D\}}|td|dzES retriever_r=)tag)rH)r_r get_childr4ir5rDrdrTs r& z1EnsembleRetriever.rank_fusion..sw   9   k&;&;@RQqS@R@R&;&S&S     r(cg|]<}t|tr#ttt|n|=S) page_content) isinstancestrrrr4docs r&rsz1EnsembleRetriever.rank_fusion..sQ!!!:DC9M9MVd3nn5555SV!!!r() enumerater+ranger?weighted_reciprocal_rankr8rdrTrDretriever_docsrrrgs ``` r&rYzEnsembleRetriever.rank_fusions&      !*$/ : :   s>**++  A!!)!,!!!N1   77GGr(c Ktjfdt|jDd{V}t t |D]}d||D||<||}|S)z Asynchronously retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. c g|]D\}}|td|dzESrm)rcrrprqs r&rsz2EnsembleRetriever.arank_fusion..sz!Ay !! +*?*?DVQRSTQTDVDV*?*W*Wr(Nc\g|])}t|tst|n|*Sru)rwrrys r&rsz2EnsembleRetriever.arank_fusion..sH!!!3=S(2K2KTc****QT!!!r()asynciogatherr{r+r|r?r}r~s ``` r&rbzEnsembleRetriever.arank_fusions& '~%.do$>$>       s>**++  A!!)!,!!!N1   77GGr( doc_listsct|tjkrtdttt |jD]U\}}t |dD]>\}}j|jn|j jxx||j zz z cc<?Vtj |}tt|fddfd}|S) a Perform weighted Reciprocal Rank Fusion on multiple rank lists. You can find more details about RRF here: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf Args: doc_lists: A list of rank lists, where each rank list contains unique items. Returns: list: The final aggregated list of items sorted by their weighted RRF scores in descending order. zz.Es'{*$$dk2r(TcPj|jn|jjSr2r)rz rrf_scorer8s r&rz.Ls'I$(K$7  S\$+=Vr()reverser)r?r, ValueErrorrfloatzipr{r/rvrKr.r from_iterablesortedr') r8rdoc_listweightrankrzall_docs sorted_docsrs ` @r&r}z*EnsembleRetriever.weighted_reciprocal_ranksV y>>S.. . .N  '2%&8&8 #It| < < . . Hf&xq999 . . c ;.(( \$+6  tdf}- . .&y11       r(r2) __name__ __module__ __qualname____doc__r r__annotations__rr.intr/r rxpropertyrr3r classmethodrrrBrrr_rcrrhrrjrYrbr}r(r&r*r*9s  ]#### %[ AsKKK FHSM     d#89   X _(###c3hC[$#>B"*>":MP hD>B!!!"*>":!MP! h!!!!F4  h *9  h 4,0 ''''4' ( ' h ''''\,0 ))))9) ( ) h ))))V1d8n-1 h111111r(r*)(rr collectionsrcollections.abcr itertoolsrtypingrrrr r r r r rrUrrlangchain_core.documentsrlangchain_core.retrieversrrlangchain_core.runnablesrlangchain_core.runnables.configrrlangchain_core.runnables.utilsrrpydanticrrrr'r*rr(r&rs ######$$$$$$                      .-----BBBBBBBB333333GGGGGGGG%$$$$$ GCLL GCx   HQKhsAv.>8A;"WWWWW WWWWWr(