g"dZddlmZmZmZddlZddlZddlmZddlm Z ddl m Z ddl m Z mZmZdd lmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZddlmZej e!Z"dZ#dZ$de%de%dej&fdZ'dej&dej&fdZ(dej&dej&dej&dej&fdZ)Gddej*Z+Gddej*Z,Gd d!ej*Z-Gd"d#eZ.d$Z/d%Z0ed&e/Gd'd(e.Z1ed)e/Gd*d+e.eZ2dS),zPyTorch CodeGen model.)OptionalTupleUnionN)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCache StaticCache)GenerationMixin)AttentionMaskConverter)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) CodeGenConfigzSalesforce/codegen-2B-monornum_posdimreturncddtjd|dtj|z zz }tjdtj|tj|}tjtj|tj|fdS) N?i'rdtypez i , j -> i jrr)torcharangeint64einsumfloatcatsincos)rrinv_freq sinusoid_inps h/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/codegen/modeling_codegen.pycreate_sinusoidal_positionsr,)se QQek J J JS PQRH< WEK0X0X0X0^0^0`0`bjkkqqssL 9ei --uy/F/FGQ O O OOxc|dddddddddf}|dddddddddf}tj| |fd}|dS)Nrrr )r!stackflatten)r.x1x2s r+rotate_every_twor60ss 111aaaCCaC<B 111aaaADqD= B bS"I2&&&A 99R==r-tensorr'r(ctj|dddddddfdd}tj|dddddddfdd}||zt||zzS)Nrr)r!repeat_interleaver6)r7r'r(s r+apply_rotary_pos_embr:8sy  !#aaaD!!!m"4a ; ;C  !#aaaD!!!m"4a ; ;C SL-f55; <"##;N$NOOr-ct|jdkr,|ddddd}ngt|jdkr+|dddd}n$t dt|j|dd ||zfz}||S) zM Merges attn_head_size dim and num_attn_heads dim into n_ctx rrrrz3Input tensor rank should be one of [4, 5], but is: Nr1)lenrapermute contiguousrPsizeview)r[r7rNattn_head_size new_shapes r+ _merge_headszCodeGenAttention._merge_headsbs v|   ! !^^Aq!Q22==??FF   ! # #^^Aq!Q//::<> MD$<<F !K 9"EEEuc!!%)A4=Y_!``T%=t}U[\\!!%)A4=Y_!`` aAq)).  !\%8 8 8-001DEEO#2D  .;vv|B'71'<"EEES ? &111aaa!24?!223EAAAqqq$/"3"334F!!!QQQ#4T_#445E111aaaDO$5$556F(S99E(S99E)UFO444CIufo2666EE&sC55C(S99Ekk!Q1%% aAq))  !)-"0 L $**366-2E+F+Ft~_kllJC%)JJuc5.R[$\$\! \'' T5Mt}]] mmK00 ((55  +  '   &Gr-N)NNNNNNFFN)rK __module__ __qualname__r@rfrqr~rr! FloatTensorr LongTensorboolrrTensorr __classcell__rJs@r+r<r<>sXXXXXX< & & &$ ))))D'+6:3715$),159LL 12LUOL!!23 L u/0 L E-. LD>L$D>L!!12L  elE%,//0u|U5<%8% c@Q:RRST V LLLLLLLLr-r<cNeZdZfdZdeejdejfdZxZS) CodeGenMLPc(t|j}tj|||_tj|||_t|j|_ tj |j |_ dSr) r?r@n_embdrrVfc_infc_outr activation_functionactrBrEdropout)r[intermediate_sizer\rMrJs r+r@zCodeGenMLP.__init__so M Yy*;<< i 19== &45z&"455 r-rrc||}||}||}||}|Sr)rrrr)r[rs r+rzCodeGenMLP.forwardsL =11 //  M22  ]33 r-) rKrrr@rr!rrrrs@r+rrse66666Xe.?%@UEVr-rcTeZdZdfd Z ddeejdeedeejdeejdeejd ee d ee d eejd e e ej ee ej e ejd ffffdZ xZS) CodeGenBlockNct|j|jn d|jz}t j|j|j|_t|||_ t|||_ dS)Nrieps) r?r@n_innerrr LayerNormlayer_norm_epsilonln_1r<attnrmlp)r[r\rG inner_dimrJs r+r@zCodeGenBlock.__init__ss &,n&@FNNa&-FW LF4MNNN $VY77 i00r-Frrryrrzrrrr.c |} ||}|||||||||} | d} | dd} ||} | | z| z}|r|f| z} n|f| ddz} | S)Nrrryrrzrrrrr)rrr)r[rrryrrzrrrresidual attn_outputsr}rfeed_forward_hidden_statess r+rzCodeGenBlock.forwards! -00 yy'!)%/)!   #1o qrr"%)XXm%<%<"#&@@8K  5$&0GG$&4Gr-rr)rKrrr@rr!rr rrrrrrrrs@r+rrs)111111'+6:3715$),159"" 12"UO"!!23 " u/0 " E-. "D>"$D>"!!12" uU\"HU5<uGXZ]G]A^3^-_$`` a""""""""r-rcJeZdZdZeZdZdZdgZdZ dZ dZ dZ fdZ dZxZS)CodeGenPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformerTrpast_key_valuesc:tj|i|dSr)r?r@)r[inputskwargsrJs r+r@zCodeGenPreTrainedModel.__init__/s%&+F+++++r-ct|tjfrT|jjd|jj|j |jj dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|jj |jj ddSdS)zInitialize the weights.)meanstdNr) isinstancerrVweightdatanormal_r\initializer_ranger>zero_ Embedding padding_idxrfill_)r[modules r+ _init_weightsz$CodeGenPreTrainedModel._init_weights2s+ fryl + + * M  & &CT[5R & S S S{&  &&((((('&  - - * M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r-)rKrr__doc__r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_cache_class_supports_quantized_cache_supports_static_cacher@rrrs@r+rr s !L%&*#'("3  $!,,,,,*******r-raJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`CodeGenConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoProcenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_attention_heads,)` or `(n_layer, num_attention_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_dim)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*): Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Two formats are allowed: - a [`~cache_utils.Cache`] instance, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache); - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy cache format. The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned. If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. zaThe bare CodeGen Model transformer outputting raw hidden-states without any specific head on top.ceZdZfdZdZdZeede e e e  dde ejde eeeeejfd e ejd e ejd e ejd e ejd e ejde ede ede ede ede ejdeee ffdZd ejdejdejdedef dZed ejdededejdejdejdefdZxZS) CodeGenModelc<tj|_j|_t jj|j|_t jj |_ t j fdtj D|_t j|jj|_t%jjjz|_d|_|dS)Nc2g|]}t|S))rG)r).0ir\s r+ z)CodeGenModel.__init__..s&aaaa Vq A A Aaaar-rF)r?r@rrM vocab_sizerrwterB embd_pdropdrop ModuleListrangen_layerhrrln_fminrYn_ctxrNgradient_checkpointing post_initr[r\rJs `r+r@zCodeGenModel.__init__s     +< 14>BBJv011 aaaa5QWQ_K`K`aaabbLV5NOOO f/A[1[\\&+# r-c|jSrrr[s r+get_input_embeddingsz!CodeGenModel.get_input_embeddingss xr-c||_dSrrr[new_embeddingss r+set_input_embeddingsz!CodeGenModel.set_input_embeddingss !r-batch_size, sequence_length checkpoint output_typerN input_idsrrytoken_type_idsrrz inputs_embedsrroutput_hidden_states return_dictrrc &| | n |jj} | | n |jj} ||n |jj}| | n |jj} |du|duz rt d|jr%|jr|rt dd}|| |}d} |rVt|tsAd} |t}n.tj|}t d|jd}| 7||nd}t#j|||z|j} || d}|||| || }|||jj}|}|0|d |}| |}||z}||}d ||d f}d}| rd nd}| rd nd}t7|jD]\}}| r||fz}|jr0|jr)||j|d|||||| | }n||||||||| | }|d}|dur|d}| r|||rd ndfz}||}||}| r||fz}|r|nd}| r| }| stCd ||||fDStE||||S)Nz:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FTzWe detected that you are passing `past_key_values` as a tuple of tuples. This is deprecated and will be removed in v4.47. Please convert your cache or use an appropriate `Cache` class (https://huggingface.co/docs/transformers/kv_cache#legacy-cache-format)rrrr0rrc3K|]}||V dSrr)rvs r+ z'CodeGenModel.forward..5s1bcbobobobobor-)last_hidden_staterr attentions)#r\rrruse_return_dictrPrtrainingrHrIrrr r from_legacy_cacheraget_seq_lengthr!r"r unsqueeze_update_causal_mask get_head_maskrrnrrm enumerater_gradient_checkpointing_func__call__rto_legacy_cachetupler)r[rrryrrrzrrrrrrreturn_legacy_cache seq_lengthpast_seen_tokensr|rtoken_type_embeds output_shapenext_decoder_cacheall_self_attentionsall_hidden_statesrblockr next_caches r+rzCodeGenModel.forwards;*2C1N--TXT_Tq$8$D $+Jj "+!6IIDK?L]  &&y$+2EFF %  %+00Z@@N $ 8 8 ),==M -00 J (:(:2(>(>? !$5?bb4"6@BBD!$&))! ^! ^HAu# I$58H$H!* t} ;;N! aL%"   %"/.#.!-'l'&7#1   $AJMD  %,QZ"  ^&9W)EZQQYZ=[<]&]# -00 %**<88  E 1]4D D +4>''$  6#3355J ):7HJ]^ '+&+*     r- input_tensorc |jjdkr |d|vr|SdS||nd}t|t}|jjdkr#|s!|st j||||jrdS|j|j } }|j d} |r| } n/t|tj r |j dn|| zdz} ||| | || ||j d} |jjdkrB|@|j jd kr0|s.tj|j} t j| | } | S) Nflash_attention_2rrsdpa)rpast_key_values_length is_trainingrr0)sequence_length target_lengthrrr batch_sizecuda)r\_attn_implementationr rr r_ignore_causal_mask_sdpar rrraget_max_cache_shaper!r5_prepare_4d_causal_attention_mask_with_cache_positiontypefinfor_unmask_unattended)r[ryrrrrrusing_static_cacherrr$r%r| min_dtypes r+r z CodeGenModel._update_causal_maskAs ; +/B B B)c^.C.C%%4 @O?Z?99;;;`a'EE ; +v 5 5>P 5Yj 5%>*'7 M   t$*L,?v&,Q/  +??AAMMnel;;<$R((%7!; PP +')#)!,Q   K , 6 6*%*f44%5  E**.I0CKQZ[[Kr-r$r%rrr&c ||dkr|}n+tj|j} tj||f| ||}|dkrtj|d}|tj|||ddkz}|ddddddf|ddd}|| }|j d} |ddddddd| f|ddddddfz} | dk} |ddddddd| f | | |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. device (`torch.device`): The device to plcae the 4D attention mask on. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nri) fill_valuerrr)diagonalrr0r) rr!r-rfulltriur"r`expandclonera masked_fill) ryr$r%rrrr&rr|r0 mask_length padding_masks r+r+zBCodeGenModel._prepare_4d_causal_attention_mask_with_cache_positionsD  %.*<*<*>*>!*C*C(KK E**.I* -0Ye\bK!###jqAAA 5< fEEEH^H^_acdHeHee eK%dD!!!QQQ&67>>z1bRTUUK))//11 ,226 *111aaaL[L+@ANSTSTSTVZ\`bcbcbcScDdd +q0 5@AAAqqq,;,AV5W5c5c )66 AAAqqq!!!\k\12r-) NNNNNNNNNNNN)rKrrr@rrrCODEGEN_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOCrr!rrr rrrrrr  staticmethodintrrr+rrs@r+rrs  """+*+C+J+JKh+i+ijj&+$15NR6:59371559$(,0/3&*59G G E,-G "%uU5<5H/I(I"JKG !!23 G !!12 G u/0 G E-.G   12G D>G $D>G 'tnG d^G !!12G  u-- .G G G kj G T? ?l? ?  ?  ????B5 555{ 5  5  5555\55555r-rzM The CodeGen Model transformer with a language modeling head on top. c!eZdZdgZfdZdZdZee de e e e  ddeejd eeeeeejfd eejd eejd eejd eejdeejdeejdeedeedeedeedeejdeee ffdZed eeejdejdeeejfdZxZS)CodeGenForCausalLMzlm_head.weightct|t||_t j|j|j|_| dSr) r?r@rrrrVrrlm_headrrs r+r@zCodeGenForCausalLM.__init__s[    '//y0ABB  r-c|jSrrDrs r+get_output_embeddingsz(CodeGenForCausalLM.get_output_embeddingss |r-c||_dSrrFrs r+set_output_embeddingsz(CodeGenForCausalLM.set_output_embeddingss % r-rrNrrryrrrzrlabelsrrrrrrc| | n |jj} ||||||||| | | | |  }|d}||t j}d}|||j}|dddddf}|dddf}t}|| d| d| d}||j }| s|f|ddz}||f|zn|St|||j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N) rryrrrzrrrrrrr.r0r)losslogitsrrr)r\rrrDrSr!rRrrlrrnrmrrrrr)r[rrryrrrzrrJrrrrrtransformer_outputsr lm_logitsrL shift_logits shift_labelsloss_fctoutputs r+rzCodeGenForCausalLM.forwards8&1%>L!#qrr'?5577L'))H8L--b,2C2CB2G2GHH,J[J[\^J_J_``D77=.//D F\$7$;;F)-)9TGf$$v E%/?-;*5     r-beam_idxc:tfd|DS)a This function is used to re-order the `past_key_values` cache if [`~PretrainedModel.beam_search`] or [`~PretrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct beam_idx at every generation step. c3NK|]}tfd|DV dS)c3tK|]2}|d|jV3dS)rN) index_selectrSr)r past_staterTs r+rz>CodeGenForCausalLM._reorder_cache...)sCjjQ[*))!X[[9J-K-KLLjjjjjjr-Nr)rrrTs r+rz4CodeGenForCausalLM._reorder_cache..(sU   jjjj_ijjj j j      r-rZ)rrTs `r+_reorder_cachez!CodeGenForCausalLM._reorder_caches=    -      r-) NNNNNNNNNNNNN)rKrr_tied_weights_keysr@rGrIrr;r<rr=rr>rr!rrr rrrrrr?r[rrs@r+rBrBsZ++&&&+*+C+J+JKh+i+ijj&*$15NR6:59371559-1$(,0/3&*59D D E,-D "%uU5<5H/I(I"JKD !!23 D !!12 D u/0 D E-.D   12D )*D D>D $D>D 'tnD d^D !!12D  u,, -D D D kj D L  uU\23  ?D|  uU\" #    \      r-rB)3rtypingrrrr!torch.utils.checkpointrtorch.nnr activationsr cache_utilsr r r generationr modeling_attn_mask_utilsrmodeling_outputsrrmodeling_utilsrutilsrrrrconfiguration_codegenr get_loggerrKrHr=r>r@rr,r6r:Moduler<rrrCODEGEN_START_DOCSTRINGr;rrBrr-r+rksQ)))))))))) %%%%%%!!!!!!;;;;;;;;;;))))))>>>>>>OOOOOOOO------uuuuuuuuuuuu000000  H % %2!PP3P5<PPPP ==EL=u|=X]Xd==== \\\\\ry\\\@(+++++29+++\ * * * * *_ * * *F EPg_____)__ _D   i i i i i /i i  i i i r-