gtdZddlZddlZddlZddlmZmZmZmZddl Z ddl m Z ddl mZddlmZmZmZmZmZmZdd lmZdd lmZer dd lmZmZmZerddlZeje Z!d Z"Gd dZ#Gdde#Z$Gdde#Z%Gdde%Z&Gdde%Z'GddZ(dS)z#RAG Retriever model implementation.N)IterableListOptionalTuple)PreTrainedTokenizer) BatchEncoding) cached_fileis_datasets_availableis_faiss_availableloggingrequires_backends strtobool) RagConfig) RagTokenizer)Dataset load_datasetload_from_diskzAhttps://storage.googleapis.com/huggingface-nlp/datasets/wiki_dpr/ceZdZdZdejdeefdZd dejde ejejffdZ dZ d Z d S) IndexzL A base class for the Indices encapsulated by the [`RagRetriever`]. doc_idsreturnct)z Returns a list of dictionaries, containing titles and text of the retrieved documents. Args: doc_ids (`np.ndarray` of shape `(batch_size, n_docs)`): A tensor of document indices. NotImplementedErrorselfrs a/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/rag/retrieval_rag.py get_doc_dictszIndex.get_doc_dicts1s "!question_hidden_statesct)a$ For each query in the batch, retrieves `n_docs` documents. Args: question_hidden_states (`np.ndarray` of shape `(batch_size, vector_size)`): An array of query vectors. n_docs (`int`): The number of docs retrieved per query. Returns: `np.ndarray` of shape `(batch_size, n_docs)`: A tensor of indices of retrieved documents. `np.ndarray` of shape `(batch_size, vector_size)`: A tensor of vector representations of retrieved documents. r)rr#n_docss r get_top_docszIndex.get_top_docs;s "!r!ct)zA Returns `True` if index is already initialized. rrs ris_initializedzIndex.is_initializedKs "!r!ct)a  A function responsible for loading the index into memory. Should be called only once per training run of a RAG model. E.g. if the model is trained on multiple GPUs in a distributed setup, only one of the workers will load the index. rr(s r init_indexzIndex.init_indexQs "!r!Nr") __name__ __module__ __qualname____doc__npndarrayrdictr rr&r)r+r!rrr,s"RZ"DJ""""""2:"ERTR\^`^hRhLi"""" """ """""r!rceZdZdZdZdZdZdZdZdZ dZ d Z d e j fd Zdd e jdee je jffdZdS) LegacyIndexa An index which can be deserialized from the files built using https://github.com/facebookresearch/DPR. We use default faiss index parameters as specified in that repository. Args: vector_size (`int`): The dimension of indexed vectors. index_path (`str`): A path to a *directory* containing index files compatible with [`~models.rag.retrieval_rag.LegacyIndex`] z,hf_bert_base.hnswSQ8_correct_phi_128.c_indexzpsgs_w100.tsv.pklc~g|_||_||_||_d|_d|_dS)NF)index_id_to_db_id index_path_load_passagespassages vector_sizeindex_index_initialized)rr<r9s r__init__zLegacyIndex.__init__isA!#$++-- & "'r!c Ftj|} t||}n/#t$r"d|d|d|d|d|d }t |wxYw|rt d|n t d|d||S) Nz Can't load 'z'. Make sure that: - 'zB' is a correct remote path to a directory containing a file named z - or 'z=' is the correct path to a directory containing a file named z. z loading file z from cache at )ospathisdirr EnvironmentErrorloggerinfo)rr9filenameis_localresolved_archive_filemsgs r _resolve_pathzLegacyIndex._resolve_pathqs7==,, ($/ H$E$E ! ! ( ( (rxrr rrdlrr#rrbjrrr  #3'' '  (  Z KK?(=?? @ @ @ @ KKXXXAVXX Y Y Y$$s 2,Acztd|j||j|j}t t jddstdt|d5}tj |}dddn #1swxYwY|S)NLoading passages from TRUST_REMOTE_CODEFalsezThis part uses `pickle.load` which is insecure and will execute arbitrary code that is potentially malicious. It's recommended to never unpickle data that could have come from an untrusted source, or that could have been tampered with. If you already verified the pickle data and decided to use it, you can set the environment variable `TRUST_REMOTE_CODE` to `True` to allow it.rb) rErFr9rKPASSAGE_FILENAMErrAenvironget ValueErroropenpickleload)r passages_path passages_filer;s rr:zLegacyIndex._load_passagess >T_>>???**4?Dr(s rr)zLegacyIndex.is_initialized &&r!ctj|jdzd}d|j_d|j_||_|d|_dS)NriT) r^ IndexHNSWFlatr<hnswefSearchefConstructionr=rer>)rr=s rr+zLegacyIndex.init_indexsW#D$4q$8#>>! $' !  !!!"&r!rcg}|D]1}d|D}fd|D}||2g}|D]7}i}d|D|d<d|D|d<||8|S)NcFg|]}tt|Sr4)strint).0doc_ids r z-LegacyIndex.get_doc_dicts..s&<<<3s6{{##<<.s <<.s 8 8 8CQ 8 8 8r!titlecg|] }|d Srr4r{s rrwz-LegacyIndex.get_doc_dicts..s7773A777r!text)append)rrdoc_list doc_ids_iidsdocs doc_dictsdoc_dicts` rr zLegacyIndex.get_doc_dictss  " "I<<)<<..s8SSSDJ**3v;;77<SSSr!r4rurrs rrwz,LegacyIndex.get_top_docs..s/lllX_SSSS7SSSlllr!c,g|]}fd|DS)cDg|]}tj|Sr4)rtr8rys rrwz7LegacyIndex.get_top_docs...s)JJJD*6233JJJr!r4rs rrwz,LegacyIndex.get_top_docs..s.cccwJJJJ'JJJcccr!)r1zerosr`reshapehstackr=searcharray) rr#r%aux_dimquery_nhsw_vectors_docs_idsvectorsrs ` rr&zLegacyIndex.get_top_docss(3566iHHHPPQSUVWWY(>'HIIj''(:FCC 8llllcklllccccZbcccx}}bhw////r!Nr,)r-r.r/r0r]rRr?rKr:rer)r+r1rr r2rr&r4r!rr6r6Zs  DN*(((%%%$   JJJ$'''''' RX    002:0ERTR\^`^hRhLi000000r!r6ceZdZddZdefdZdZdZdej de e fd Z dd ej de ej ej ffd Zd S) HFIndexBaseFc||_||_||_|||ddgdddS)N) with_indexnumpy embeddingsTr)columnsoutput_all_columnsr)r<datasetr>_check_dataset_format set_format)rr<rindex_initializeds rr?zHFIndexBase.__init__sW& "3 "".?"@@@7\Nt[deeeeer!rctt|jts$tdt |jt hdt |jjz dkrtd|jj|r*d|j vrtddSdS)Nz5Dataset should be a datasets.Dataset object, but got >rr}rrzDataset should be a dataset with the following columns: title (str), text (str) and embeddings (arrays of dimension vector_size), but got columns rzMissing faiss index in the dataset. Make sure you called `dataset.add_faiss_index` to compute it or `dataset.load_faiss_index` to load one from the disk.) isinstancerr TypeErrortyper`set column_namesrU list_indexes)rrs rrz!HFIndexBase._check_dataset_formats$,00 jhTXY]YeTfTfhhii i ...T\5N1O1OO P PST T T?#'<#<??   ,dl.G.G.I.IIIK   IIr!ctrgrr(s rr+zHFIndexBase.init_indexs!###r!c|jSrgrhr(s rr)zHFIndexBase.is_initializedrir!rrcTfdtjdDS)NcZg|]'}j|(Sr4)rtolist)ruirrs rrwz-HFIndexBase.get_doc_dicts..s0SSSa WQZ..001SSSr!r)rangeshapers``rr zHFIndexBase.get_doc_dictss0SSSSS5qAQ;R;RSSSSr!r"r#c jd||\}}fd|D}d|D}tt|D]i}t|||krNt j||t j|t||z jfg||<jt j|t j|fS)Nrc>g|]}jd|DS)cg|] }|dk| Srr4)rurs rrwz7HFIndexBase.get_top_docs...s;;;AAFFaFFFr!)r)ruindicesrs rrwz,HFIndexBase.get_top_docs..s/PPP ;;;;;<PPPr!cg|] }|d S)rr4r{s rrwz,HFIndexBase.get_top_docs..s5553|$555r!) r search_batchrr`r1vstackrr<r)rr#r%rrrrrs` rr&zHFIndexBase.get_top_docss**<9OQWXX3PPPPCPPP55555s7||$$ m mA71:''Y BHfs7ST:>VX\Xh=i4j4j'kll x}}bhw////r!N)Fr,)r-r.r/r?boolrr+r)r1r2rr3r rr&r4r!rrrsffff     $$$'''TRZTDJTTTT002:0ERTR\^`^hRhLi000000r!rc deZdZdZ d dededed eed eef fd Zd ZxZ S)CanonicalHFIndexa A wrapper around an instance of [`~datasets.Datasets`]. If `index_path` is set to `None`, we load the pre-computed index available with the [`~datasets.arrow_dataset.Dataset`], otherwise, we load the index from the indicated path on disk. Args: vector_size (`int`): the dimension of the passages embeddings used by the index dataset_name (`str`, optional, defaults to `wiki_dpr`): A dataset identifier of the indexed dataset on HuggingFace AWS bucket (list all available datasets and ids with `datasets.list_datasets()`). dataset_split (`str`, optional, defaults to `train`) Which split of the `dataset` to load. index_name (`str`, optional, defaults to `train`) The index_name of the index associated with the `dataset`. The index loaded from `index_path` will be saved under this name. index_path (`str`, optional, defaults to `None`) The path to the serialized faiss index on disk. use_dummy_dataset (`bool`, optional, defaults to `False`): If True, use the dummy configuration of the dataset for tests. wiki_dprtrainNFr< dataset_name dataset_split index_namer9ct|dut|duzdkrtd||_||_||_||_||_||_t d|jt|jd|j|j|}t ||ddS)Nrz,Please provide `index_name` or `index_path`.rMF)rsplitdummyrevisionr) rtrUrrrr9use_dummy_datasetdataset_revisionrErFrsuperr?) rr<rrrr9rrr __class__s rr?zCanonicalHFIndex.__init__s zT! " "St);%<%< < A AKLL L(*$$!2 0 @T->@@AAA  $(%     gGGGGGr!c |jDtd|j|jd|jn|td|jd|jt|jdd|j|j|j |j |_|j ddgdd|_ dS) Nr\rfilez with index name T)with_embeddingsrrrrrr)rr) r9rErFrload_faiss_indexrrrrrrrr>r(s rr+zCanonicalHFIndex.init_index!s ? & KK?do?? @ @ @ L ) ),T_ ) M M M M KKcd.?ccRVRacc d d d'! $(?,.DL L # #Gl^X\ # ] ] ]"&r!)rrNNFN) r-r.r/r0rtrsrr?r+ __classcell__rs@rrrs0'$$($(HHHH H SM H SM HHHHHH8'''''''r!rcFeZdZdZddeffd ZedZdZxZ S) CustomHFIndexa A wrapper around an instance of [`~datasets.Datasets`]. The dataset and the index are both loaded from the indicated paths on disk. Args: vector_size (`int`): the dimension of the passages embeddings used by the index dataset_path (`str`): The path to the serialized dataset on disk. The dataset should have 3 columns: title (str), text (str) and embeddings (arrays of dimension vector_size) index_path (`str`) The path to the serialized faiss index on disk. Nr<cbt|||du||_dS)Nr)rr?r9)rr<rr9rs rr?zCustomHFIndex.__init__Bs2 gtASTTT$r!ctd|||tdt|}||||S)NrMzPlease provide `dataset_path` and `index_path` after calling `dataset.save_to_disk(dataset_path)` and `dataset.get_index('embeddings').save(index_path)`.)r<rr9)rErFrUr)clsr< dataset_pathr9rs rrzCustomHFIndex.load_from_diskFsf ;\;;<<<  :#5J !..s{G SSSSr!c|sLtd|j|jd|jd|_dSdS)Nr\rrT)r)rErFr9rrr>r(s rr+zCustomHFIndex.init_indexQsh""$$ + KK?do?? @ @ @ L ) ),T_ ) M M M&*D # # # + +r!rg) r-r.r/r0rtr? classmethodrr+rrs@rrr4s  %%C%%%%%%TT[T+++++++r!rc VeZdZdZdfd ZedZeddZdZ dZ dd Z d e d e d ee fd Zdejde d eejejffdZdejde d eejeeffdZdefdZ ddeee dejd efdZxZS) RagRetrievera Retriever used to get documents from vector queries. It retrieves the documents embeddings as well as the documents contents, and it formats them to be used with a RagModel. Args: config ([`RagConfig`]): The configuration of the RAG model this Retriever is used with. Contains parameters indicating which `Index` to build. You can load your own custom dataset with `config.index_name="custom"` or use a canonical one (default) from the datasets library with `config.index_name="wiki_dpr"` for example. question_encoder_tokenizer ([`PreTrainedTokenizer`]): The tokenizer that was used to tokenize the question. It is used to decode the question and then use the generator_tokenizer. generator_tokenizer ([`PreTrainedTokenizer`]): The tokenizer used for the generator part of the RagModel. index ([`~models.rag.retrieval_rag.Index`], optional, defaults to the one defined by the configuration): If specified, use this index instead of the one built using the configuration Examples: ```python >>> # To load the default "wiki_dpr" dataset with 21M passages from wikipedia (index name is 'compressed' or 'exact') >>> from transformers import RagRetriever >>> retriever = RagRetriever.from_pretrained( ... "facebook/dpr-ctx_encoder-single-nq-base", dataset="wiki_dpr", index_name="compressed" ... ) >>> # To load your own indexed dataset built with the datasets library. More info on how to build the indexed dataset in examples/rag/use_own_knowledge_dataset.py >>> from transformers import RagRetriever >>> dataset = ( ... ... ... ) # dataset must be a datasets.Datasets object with columns "title", "text" and "embeddings", and it must have a faiss index >>> retriever = RagRetriever.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base", indexed_dataset=dataset) >>> # To load your own indexed dataset built with the datasets library that was saved on disk. More info in examples/rag/use_own_knowledge_dataset.py >>> from transformers import RagRetriever >>> dataset_path = "path/to/my/dataset" # dataset saved via *dataset.save_to_disk(...)* >>> index_path = "path/to/my/index.faiss" # faiss index saved via *dataset.get_index("embeddings").save(...)* >>> retriever = RagRetriever.from_pretrained( ... "facebook/dpr-ctx_encoder-single-nq-base", ... index_name="custom", ... passages_path=dataset_path, ... index_path=index_path, ... ) >>> # To load the legacy index built originally for Rag's paper >>> from transformers import RagRetriever >>> retriever = RagRetriever.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base", index_name="legacy") ```NTc^||_t|ddgt|p|||_||_||_|j|_|j |_ ||_ |jr| d|_ d|_dS)Ndatasetsr^F)_init_retrievalrrr? _build_indexr=generator_tokenizerquestion_encoder_tokenizerr%retrieval_batch_size batch_sizeconfiginit_retrievalctx_encoder_tokenizerreturn_tokenized_docs)rrrrr=rrs rr?zRagRetriever.__init__s-$W 5666 7d//77 #6 *D'm  5   "    ! ! !%)"%*"""r!c :|jdkr!t|j|jptS|jdkr,t |j|j|jSt|j|j |j |j|j|j |j S)Nlegacycustom)r<rr9)r<rrrr9rr) rr6retrieval_vector_sizer9LEGACY_INDEX_PATHrrrYrrrrrrs rrzRagRetriever._build_indexs   ( (,!6%6  ( * * //"8#1!,0  $"8#^$2!,!,"(":!'!8 r!c Dt|ddg|ddptj|fi|}t j||}|j}|j}|d|_t|j |}n| |}|||||S)Nrr^rrr)rrr=) rpoprfrom_pretrainedrquestion_encoder generatorrrrr) rretriever_name_or_pathindexed_datasetkwargsr rag_tokenizerrrr=s rrzRagRetriever.from_pretraineds# G4555Hd++jy/HI_/j/jci/j/j$45KTZ[[[ %2%C"+5  & (F !&">PPEE$$V,,Es 'A 3     r!ct|jtr|jj^t j|d}|jj d |||j_|jj t j|d}|jjj d}|jj|||jjj d<||j_ |j|t!|j|j}||dS)Nzhf_dataset_index.faissr hf_dataset)rr)rr=rrr9rArBjoinr get_indexsaverY_indexesr save_to_disksave_pretrainedrrr)rsave_directoryr9rY faiss_indexrs rrzRagRetriever.save_pretraineds* dj- 0 0 :{%-W\\.:RSS  ",,\::?? KKK)3 &{(0 " ^\ J J "j09==lKK  "// >>>.cat_input_and_docs##C(( *%abbM !!#&& +%crcN ~I% (==H4;K^^ammvvcCJr!c g|]L}tD]:}|d||d||;MS)r}r)r)rurjrr input_stringsr%rs rrwz1RagRetriever.postprocess_docs..s    6]]     Q #Q"a         r! max_lengthT)rreturn_tensorspadding truncation input_idsattention_mask)rr`rbatch_encode_plusrmax_combined_length) rrrrr%rrag_input_stringscontextualized_inputsrs ````` @rpostprocess_docszRagRetriever.postprocess_docss"              3t99%%    !% 8 J J {6)  !K! ! %[13HIY3ZZZr!t chunk_sizerc\fdtdtDS)Nc*g|]}||zSr4r4)rurr%r$s rrwz.RagRetriever._chunk_tensor..s'LLL!!a*n$%LLLr!r)rr`)rr$r%s ``r _chunk_tensorzRagRetriever._chunk_tensors3LLLLLuQA /K/KLLLLr!r#r%c|||j}g}g}|D]}tj}|j||\}}t dtj|z d|j||||tj |tj |fS)Nzindex search time: z sec, batch size ) r(rtimer=r&rEdebugrextendr1r) rr#r%question_hidden_states_batched ids_batchedvectors_batched start_timerrs r_main_retrievezRagRetriever._main_retrieves)-););ooQgQmoo      s # # #  " "7 + + + + H[ ! ! H_ % %  r!cn|||\}}|||j|fS)a% Retrieves documents for specified `question_hidden_states`. Args: question_hidden_states (`np.ndarray` of shape `(batch_size, vector_size)`): A batch of query vectors to retrieve with. n_docs (`int`): The number of docs retrieved per query. Return: `Tuple[np.ndarray, np.ndarray, List[dict]]`: A tuple with the following objects: - **retrieved_doc_embeds** (`np.ndarray` of shape `(batch_size, n_docs, dim)`) -- The retrieval embeddings of the retrieved docs per query. - **doc_ids** (`np.ndarray` of shape `(batch_size, n_docs)`) -- The ids of the documents in the index - **doc_dicts** (`List[dict]`): The `retrieved_doc_embeds` examples per query. )r1r=r )rr#r%rretrieved_doc_embedss rretrievezRagRetriever.retrieve/s>&)-(;(;rPsY*) 222222222222555555444444rrrrrrrrrrrrrrrr((((((******?>>>>>>>>>>LLL  H % %X+"+"+"+"+"+"+"+"\h0h0h0h0h0%h0h0h0V'0'0'0'0'0%'0'0'0TB'B'B'B'B'{B'B'B'J!+!+!+!+!+K!+!+!+HJJJJJJJJJJr!