g jddlZddlmZmZmZmZmZddlZddl Z ddl m Z ddl m Z ddlmZmZerddlmZdd lmZdd lmZGd d ZGd deZGddeZGddeZdZdeeefdededeeeffdZdeeefdedeeeffdZ dS)N) TYPE_CHECKINGAnyDictOptionalTuple) DynamicCache)isin_mps_friendly)LogitsProcessorListMinLengthLogitsProcessor)PreTrainedModel)PreTrainedTokenizerBase)GenerationConfigceZdZdZdejdeejeejffdZ dejdejde fdZ dS) CandidateGeneratorz`Abstract base class for all candidate generators that can be applied during assisted generation. input_idsreturnc0t|jd)ag Fetches the candidates to be tried for the current input. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) Return: `torch.LongTensor` of shape `(batch_size, candidate_length)` containing the candidate sequences to be assessed by the model and, optionally, a `torch.FloatTensor` of shape `(batch_size, candidate_length, vocabulary_size)` containing the logits associated to each candidate. zT is an abstract class. Only classes inheriting this class can call `get_candidates`.NotImplementedError __class__)selfrs g/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/generation/candidate_generator.pyget_candidatesz!CandidateGenerator.get_candidates$s$"~ s s s   scores num_matchesc0t|jd) Updates the candidate generation strategy based on the outcomes. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) scores (`torch.FloatTensor` of shape `(batch_size, candidate_length, config.vocab_size)`): Prediction scores of a language modeling head. These can be logits for each vocabulary when not using beam search or log softmax for each vocabulary token when using beam search num_matches (`int`): The number of matches between the candidate sequences and the model predictions. z_ is an abstract class. Only classes inheriting this class can call `update_candidate_strategy`.rrrrrs rupdate_candidate_strategyz,CandidateGenerator.update_candidate_strategy5s*"~ + + +   rN) __name__ __module__ __qualname____doc__torch LongTensorrr FloatTensorrintr"rrrr!sjj (8 U5CSU]^c^oUpCp=q    " 53C UM^ mp      rrc eZdZdZ ddejddddded eejd d f d Z dejd e ejeej ffdZ dejdej de fdZdS)AssistedCandidateGeneratorar `CandidateGenerator` class to be used for assisted generation and speculative decoding. This class generates candidates through the use of a smaller model. Read the following blog post for more information: https://huggingface.co/blog/assisted-generation Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) assistant_model (`PreTrainedModel`): The model to be used for generating candidates. This model should be smaller than the main model. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. model_kwargs (`Dict`): The keyword arguments that will be passed to the main model, and are used as base inputs for the assistant model as well. inputs_tensor (`torch.Tensor`, *optional*): The model input tensor. In encoder-decoder models, this is the encoder input. Nrassistant_modelrgeneration_configr model_kwargs inputs_tensorlogits_processorr c|j}||}|||}||_|jj|_|jj|_|j|jj_i}|D]a\} } | dvrXt| tj r'| |ntj | || <bd|vr|s|d=d|vr |d|d<n^|jjrC|||jj|\}} }|||| |j}nd|vr |d|d<||_|jjrd|_ngd|vr\d|_|jdtj|jdd f|jtj |jd <nd|_||n t7|_tj ||_d |j_d |j_|j|j_d |j_|jj |_!d|j_ d|j_"|jD]&} t| tFrtId 'd|j_%dS)N)encoder_outputsassistant_encoder_outputspast_key_valuesnum_logits_to_keepr5r4decoder_input_idsrdecoder_attention_maskrr )devicedtypeattention_maskTzPassing `MinLengthLogitsProcessor` when using `assisted_generation is disabled. Please pass in `min_length` into `.generate()` instead)&r:tor.r/num_assistant_tokensassistant_confidence_threshold eos_token_iditems isinstancer'Tensordetachcopydeepcopykeys_supports_num_logits_to_keepconfigis_encoder_decoder_prepare_model_inputs bos_token_id._prepare_encoder_decoder_kwargs_for_generationassistant_kwargs input_ids_keygetonesshapelongr r2return_dict_in_generate output_scores is_assistant min_lengthmain_model_min_lengthmin_new_tokensr ValueErrorcache_implementation) rrr.r/r0r1r2r:rNkeyvaluemodel_input_name processors r__init__z#AssistedCandidateGenerator.__init___s,!'LL((  $),,V44M /$3$E$Z!.=.O.n+?P>\.;&,,..  JC]]]1;E5<1P1PjELLNN%%f---VZVcdiVjVj!% #3#8#8#:#: : :?CoCoCqCq : !56 &, 6 62>?Z2[ . / /  # 6 R@O@e@e@MO_AA =M+-= /]]/1A?Cd    , . .2>?P2Q . / 0  ! 4 -!4D   "2 2 2!,D 6:6K6O6O( IOA.29;KSXS]^^^77D !"2 3 3 "-D 5E4P 0 0ViVkVk!%/@!A!A9=6/3,@D@c=.2+&*%;%F",-)04-.  I)%=>>  M 7;333rrc ||jj}|jd}t t |j|jj|z dz }tt ||j |z d}|dkr|dfS|j dddu}|ru|dz }t|j|j d|dz |j d<t|j ||jjj|_ t#|j ||_ |j|d|d|d|jd |ji}|jjd i||j }|j|j d<t-j|jd } |j} | | fS) Z Fetches the candidates to be tried for the current input. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) Return: `torch.LongTensor` of shape `(batch_size, candidate_length)` containing the candidate sequences to be assessed by the model and a `torch.FloatTensor` of shape `(batch_size, candidate_length, vocabulary_size)` containing the logits associated to each candidate. r rNr6rYmax_new_tokensr/r2dimr+)r=r.r:rRminr*r>r/ max_lengthmaxrXrNrP_crop_past_key_values_prepare_attention_maskrIrJ_prepare_token_type_idsrOr2generater6r'stackr sequences) rr new_cur_lenrdrYhas_past_key_valuesnew_cache_sizeassistant_generation_kwargsassistant_outputcandidate_logits candidate_idss rrz)AssistedCandidateGenerator.get_candidatessLL!5!<==  ob) S!:;;T=S=^al=lop=pqqS1Kk1YZZ\]^^ Q  d? "#3778I4PPX\\  `(1_N7L$d&;AFFF(2 ...rrrc|jjjdvrI|t|jkr|xjdz c_dSt d|jdz |_dSdS)r > heuristicheuristic_transientg@g?N)r.r/num_assistant_tokens_scheduler*r>rir!s rr"z4AssistedCandidateGenerator.update_candidate_strategys}   1 OT   c$";<<<<))S0)))),/T5NQT5T,U,U)))  rNN)r#r$r%r&r'r(rrrCr`rr)rr*r"r+rrr-r-Hs81526V;V;#V;+V;. V;  V;   - V;0V;V;V;V;p5/(85/U5CSU]^c^oUpCp=q5/5/5/5/nV53CVUM^VmpVVVVVVrr-ceZdZdZ ddejddddddd d d ed eejd dffd Z e dZ e dZ e dZ dZdejdeejeejffdZxZS)-AssistedCandidateGeneratorDifferentTokenizersa `CandidateGenerator` class to be used for Universal Assisted Generation (UAD): assisted generation with different tokenizers for the assistant and main models. This class generates candidates through the use of a smaller model. The main model input tokens are re-encoded into assistant model tokens, then candidate tokens are generated in the assistant encoding, which are in turn re-encoded into main model candidate tokens. Validation then proceeds as explained above. The re-encoding steps involve decoding token ids into text and then encoding the text using a different tokenizer. Since re-encoding the tokens may result in tokenization discrepancies, UAD finds the longest common subsequence between the source and target encodings, to ensure the new tokens include the correct prompt suffix. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) assistant_model (`PreTrainedModel`): The model to be used for generating candidates. This model should be smaller than the main model. target_tokenizer (`PreTrainedTokenizerBase`): The tokenizer used for the target model. assistant_tokenizer (`PreTrainedTokenizerBase`): The tokenizer used for the assistant model. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. logits_processor (`LogitsProcessorList`): An instance of [`LogitsProcessorList`]. List of instances of class derived from [`LogitsProcessor`] used to modify the prediction scores of the language modeling head applied at each generation step. model_kwargs (`Dict`): The keyword arguments that will be passed to the main model, and are used as base inputs for the assistant model as well. inputs_tensor (`torch.Tensor`, *optional*): The model input tensor. In encoder-decoder models, this is the encoder input. Nrr.rtarget_tokenizerrassistant_tokenizerr/rr0r1r2r c t||||||||_||_d|_d|_d|_d|_dS)N )superr`r~r prev_tokensprev_assistant_idstarget_lookbehindassistant_lookbehind) rrr.r~rr/r0r1r2rs rr`z6AssistedCandidateGeneratorDifferentTokenizers.__init__)s` O5F Vceuvvv 0#6 "&!#$&!!!rcZt}i}|D]}tj|}t|}||vr=||d}|dz }|d|jdkr|d|jdkrt|}||||d|dfdkr |dz }|dz }nn.|d|jdkr|d|jdk|||<|S)a Calculates the length of the longest diagonal sequence in a given matrix. Args: input_matrix (torch.Tensor): The input matrix. nonzero_idx (torch.Tensor): The indices of the non-zero elements in the matrix. Returns: dict: A dictionary where the keys are the indices of the non-zero elements and the values are the lengths of the longest diagonal sequences starting from those indices. r r)setr'clonetupletolistaddrR) input_matrix nonzero_idxvisiteddiagsidx start_idxtuple_start_idx cur_diag_lens r_get_longest_diag_dictzDAssistedCandidateGeneratorDifferentTokenizers._get_longest_diag_dict=sG%% & &C C((I#I$4$4$6$677O')) KK ( ( (L NIA,!3A!6669Q<,J\]^J_;_;_"' (8(8(:(:";"; O,,, ! il :;q@@ A%LNIIA,!3A!6669Q<,J\]^J_;_;_&E#JJ rc0t||}t|}t|}t j|}||}||}||fS)a Returns the start index and length of the longest diagonal in the given input. Args: input_matrix (numpy.ndarray): The input matrix. Returns: tuple: A tuple containing the start index and length of the longest diagonal. )r}rnonzerolistvaluesrGnpargmax)rr diags_values diags_keys best_diagdiag_start_indexdiag_start_lengths r_get_longest_diag_indexzEAssistedCandidateGeneratorDifferentTokenizers._get_longest_diag_indexas>TT ,..00  ELLNN++ %**,,'' Il++ %i0(3!222rc|j|k}tj|stj|}|t }|sdSt |\}}|d|z}|d|z}|j d|z }|dd||zdf} |dd|||zf} || | fS)a Input: prompt: 2D array of shape (batch_size, prompt_length), represents the original prompt tokens prompt_plus_new_tokens: 2D array of shape (batch_size, prompt_length), represents the suffix of the original prompt, with additional new tokens. Output: discrepancy_length: int, represents the number of tokens that need to be replaced from prompt new_tokens_only: 2D array of shape (batch_size, new_token_length), represents the new tokens that are not in prompt discrepancy_only: 2D array of shape (batch_size, discrepancy_length), represents the new tokens that are in prompt but not in prompt_plus_new_tokens )NNNrr N) Tr' is_tensortensorr=r*anyitemr}rrR) promptprompt_plus_new_tokens compare_matcompare_mat_intlongest_locationlongest_diag_lengthnew_token_start_indexdiscrepancy_with_olddiscrepancy_lengthnew_tokens_onlydiscrepancy_onlys r_get_tokens_diagz>AssistedCandidateGeneratorDifferentTokenizers._get_tokens_diagus-.&8 {++ 4,{33K%..--""$$))++ $##0]0u0u 1 1 --!1 36I I/25HH$l1o0DDJJLL04IL^4^4`4`1`a1 AA$'|jj d|j z }|j |dd|dffi|}|j d}|j dd| df}t||\} } } |j } | | dkrx| j ddkrg| | j dkr | | dd| df<nG| | j dkr6| | j dz } | ddd| f} | | dd| j d df<| }| j ddkrtj| | gd} n.tj| |gd} n|j |fi|} ||_| |_ | j d}t#t%||j|z d}|jdddu}|rx|dz |z }t-|j|jd|dz |jd<t/|j||jjj|_t5|j||_|j| d|d |d |jd |ji}|jd d|jjdi||j}|j j d}||j z }| |j!dd|df|j|j}|j d}|dd| df}t||\}}}|}|*|j ddkrtj||gd}ntj||gd}||_tE|jd r|ddd|jj#f}|j$|jd<|j!|_|j d|j dkr|dfS|dfS)rbrN)rrr rcrer6rYrdr/r2r<rhr+)%r*r>r=r.r:r~rrprev_target_idsrRrrrr}rr'catrirgrXrNrPrjrkrIrJrlrOr/r2poprmrrohasattrrhr6)rrrdconvert_kwargsremove_from_pkvstart_index_in_target_windownew_assistant_idsprompt_use_length prompt_userrrassistant_input_idsdiscrepancy_length_diffrprYrqrrrsrtnum_prev_assistantstart_assistant_look_indexnew_target_ids_from_windowtarget_prompt_use_lengthtarget_prompt_use_target_new_tokens_onlynew_target_idss rrzNNz[lmm B 1A#'"9 *%)).>.DQ.G!.K.K)-=-CA-FFFGW+AAA0B/B/C/C,CDD+.>.DQ.GGG2DGWG]^_G`2`/.A!!!E^G^F^E^B^._+O_+AAA0@0Fq0I/I/K/K,KL&8O"(+a//*/)5H/4Z`b*c*c*c'',i1DFW0X^`&a&a&a###N$"Mi"j"j[i"j"j #,D "5)/3 S1Kk1YZZ\]^^#3778I4PPX\\  `(1_>N7L$d&;#CA#F %aaa*B)B)C)C&CD'T'e'e 9( ( $ !1# ! -%+A.22!&N'>_`$$(  '1 , ,0F!0K0KSTT T1L0KrrrcL|d}|j|dzkr|dfSd}d}tt|j|dz ddD]}|d|d}|d| df}||kd}|d d} | D]} | |z} | |jz} t| ||j} | | kru|d| | f}d }t||j } tj| }| dkr$|d }|d|}n|rn |t|dkr|dfS|d}tj||fd}|dfS) a Fetches the candidates to be tried for the current input. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids) Return: `torch.LongTensor` of shape `(num_candidates, candidate_length)`: The candidate sequences to be tried. r NFrrc) dimensionsizesteprreT)as_tuple)rrhrangergrunfoldallrrr r@r'numelrlen unsqueezer)rr input_length chosen_ids match_found ngram_sizewindows ngram_tensormatches match_indicesrrend_idxmaskmatch_indices_eosfirst_eos_indexcandidate_input_idss rrz-PromptLookupCandidateGenerator.get_candidatesPs!~~a((  ?lQ. . .d? "  D$@,QRBR S SUVXZ[[! ! J&&!&LLG%Q _5L,.333::G$OOTO::1=M%  *, #d&<<g|T_EEw&&!*1i.?+?!@J"&K -Z9JKKD(- d(;(;%(..00144*;A*>*C*C*E*E%/0@0@%A E'    ZA!5!5d? " ))!,, #iJ(?QGGG"D((rrrcdS)r Nr+r!s rr"z8PromptLookupCandidateGenerator.update_candidate_strategys  r)NrNr)r#r$r%r&r'rCr*r`r(rrr)rr"r+rrrr1s  "&*!#'+ U Ul U U"% U  U U U U>)(8>)U5CSU]^c^oUpCp=q>)>)>)>)@53CUM^mprrc g}|jjrtt|D]o}|||dddddd|ddf||dddddd|ddf||d||dfpt |}nd|jjvs2|jj d|jj dvr|jj r;tt|D]}||ddd|ddf||<n)tt|D]}||ddddd|ddf||< nt|tr| |n|tt|D]}||ggfkrT|||dddddd|ddf||dddddd|ddffd|||d||dft |}|S)z9Crops the past key values up to a certain maximum length.rNr r gptbigcode)rIrJrrappendrrr#lower architectures multi_queryrBr crop)modelr6rhnew_pastrs rrjrjsH |&"*_--..  C OO#C(+AAAqqq+:+qqq,@A#C(+AAAqqq+:+qqq,@A#C(+#C(+      // 17799 9 9 ".<5S''CSH|##  !D!DJqM1!!%aaa):*:):&:!; X A  !&D$--AP`@a2b2b+cik!l!l!l X--!"89 a  3=aaaARBRAR>R3SL/ 0 0  ! !!!!!RSS!!!QQQ,/66q:JAqQQH389j(=SYZ3[3[3[L/ 0 < / /!"89 a  3=aaaARBRAR>R3SL/ 0 0  ! !!!!!RSS!!!),33A7GKKH389j(=SYZ3[3[3[L/ 0 rc4d|vs|d|S|d}|dddfd}||jdz }|dkr|ddd|f}n=|dkr7|d|}tj|d|gd|d<|S)zXExpands or crops the model's token_type_ids for decoding purposes, to the defined lengthtoken_type_idsNrcr rre)rrRrr'r)r0rr final_token_typetype_length_difftoken_type_copiess rrlrls|++|rs5  <<<<<<<<<<<<<< &&&&&&------IIIIIIII6000000AAAAAA555555$ $ $ $ $ $ $ $ N}V}V}V}V}V!3}V}V}V@f$f$f$f$f$4Nf$f$f$R mmmmm%7mmm`&&&R!$sCx.!c!_c!hlmprumuhv!!!!H$sCx.cdSVX[S[nr