g"dZddlmZmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZdd lmZmZdd lmZmZmZmZdd lmZdd lmZmZmZmZm Z m!Z!m"Z"d dl#m$Z$erddl%m&Z&e!j'e(Z)dZ*dZ+gdZ,dZ-dZ.dZ/Gddej0Z1Gddej2Z3Gdde3Z4Gdde3Z5e3e4e5dZ6Gdd ej2Z7d!Z8ed"e8Gd#d$eZ9d%Z:Gd&d'e9Z;ed"e8Gd(d)e9Z<Gd*d+e9eZ=ed,e8Gd-d.e9Z>ed/e8Gd0d1e9Z?dS)2zPyTorch OPT model.)ListOptionalTupleUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GenerationMixin)!_prepare_4d_causal_attention_mask*_prepare_4d_causal_attention_mask_for_sdpa)BaseModelOutputWithPastCausalLMOutputWithPastQuestionAnsweringModelOutput SequenceClassifierOutputWithPast)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardis_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10loggingreplace_return_docstrings) OPTConfig)_flash_attention_forwardzfacebook/opt-350mr)rizArthurZ/opt-350m-dummy-scg\(\?z 'LABEL_0'cjeZdZdZdedeffd Z d dejded eejffd Z xZ S) OPTLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcjd|_t||jz|dSN)offsetsuper__init__)selfr"r# __class__s `/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/opt/modeling_opt.pyr)z&OPTLearnedPositionalEmbedding.__init__Hs3  $+5}EEEEErNattention_maskpast_key_values_length position_idsc|>tj|d}||zdz }|dd|df}t||jzS)z3`input_ids_shape` is expected to be [bsz x seqlen].Nrdim)torchcumsumlongr(forwardr')r*r.r/r0r+s r,r7z%OPTLearnedPositionalEmbedding.forwardNsq   <A>>>L(>9A=CCEEL'+A+B+B(BCLww|dk9:::r-rN) __name__ __module__ __qualname____doc__intr)r4 LongTensorrr7 __classcell__r+s@r,r!r!CsFsF3FFFFFF'(37 ;;(;!$;u/0 ;;;;;;;;;;r-r!cjeZdZdZ ddedeffd Zdejde de d ejfd Z dd ejd e ejde e ejde ejde ejdede ejd e eje eje e ejffdZ xZS) OPTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFconfig is_decoderc t||_|j|_|j|_|j|_|j |_ |j|jz|_ d|_ |j |jz|jkr td|jd|jd|j dz|_ ||_tj|j|j|j |_tj|j|j|j |_tj|j|j|j |_tj|j|j|j |_dS)NTz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩bias)r(r)rC hidden_size embed_dimnum_attention_heads num_headsattention_dropoutdropout enable_biashead_dim is_causal ValueErrorscalingrDrLineark_projv_projq_projout_proj)r*rCrDkwargsr+s r,r)zOPTAttention.__init__bsH  +3/ !-$.8  MDN *t~ = =8dn88%)^888 }d* $iTEUVVV iTEUVVV iTEUVVV  $.$.tGWXXX r-tensorseq_lenbszreturnc||||j|jddS)Nrr&)viewrKrO transpose contiguous)r*rYrZr[s r,_shapezOPTAttention._shapes<{{3GGQQRSUVWWbbdddr-N hidden_stateskey_value_statespast_key_valuer.layer_head_maskoutput_attentionsr0cd |du}|\} } } |||jz} |r||d} |d}n>|rU|||d| } |||d| }n||||d| } |||d| }t j|d| gd} t j|d|gd}nT|||d| } |||d| }|jr| |f}| |j zd|j f}|| | | j |} | j |} |j |}| d}t j | | dd}|| |j z| |fkr2td| |j z| |fd|||| d| |fkr+td | d| |fd|| | |j | ||z}t j|t jt j|jj|j }| | |j z| |}|jt jkrJt,j|dt j t j}n!t,j|d}|||j fkr-td |j fd|| dddd| | |j | |z}| | |j z| |}|r=| | |j | |}| | |j z| |}nd}t,j||j|j }t j ||}|| |j z| |j fkr5td| |j | |j fd|| | |j | |j }| dd}|| | |j}||}|||fS)#Input shape: Batch x Time x ChannelNrrr&r2z$Attention weights should be of size z , but is z!Attention mask should be of size device)r3dtypez/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size ) sizerVrRrarTrUr4catrDrKrOr^bmmr_rQmaxrYfinforlminrkfloat16r functionalsoftmaxfloat32torMroreshaperIrW)r*rbrcrdr.rerfr0is_cross_attentionr[tgt_len_ query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r,r7zOPTAttention.forwards.T9',,..Wa{{=11DL@  L."<'*J)!,LL  LT[[1A%B%BBLLJ;;t{{3C'D'Db#NNLL  'T[[%?%?SIIJ;;t{{='A'A2sKKLN1$5z#BJJJJ 9nQ&7%FANNNLLT[[%?%?SIIJ;;t{{='A'A2sKKL ? 8),7NDN*B > Ct{{<#>>CZP $Z_j1 (|(*5 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**   %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL 9el5;|7I+J+J+NWcWjkkkL(,,S4>-A7GTTL   . .=002U]0[[^^_d_lmmLL=0020FFL  &##%%$.)::: 1t~FW11',,..11+//2q!<<|?P?PQTVZVdfmov?w?wwL',,S4>-A7GTTL  ) %1$5$5c4>7T[$\$\ !055cDN6JGU\]]LL$( !]**<4AAr-)FNNNNFN)r9r:r;r<rboolr)r4Tensorr=rarrr7r?r@s@r,rBrB_sGG !YYYYYYYYY:eU\eCeceeleeee 488<1526"'/3xBxB|xB#5<0xB!u|!45 xB !. xB "%,/ xB xBu|,xB u|Xel3XeEL>Q5RR SxBxBxBxBxBxBxBxBr-rBc0eZdZdZfdZ ddejdeejdeeejdeejd eejd e d eejd eejeejeeejffd Z xZ S)OptFlashAttention2aB OPT flash attention module. This module inherits from `OPTAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cbtj|i|t |_dSN)r(r)r_flash_attn_uses_top_left_mask)r*argsrXr+s r,r)zOptFlashAttention2.__init__s9$)&))) 3V2W2W.W+++r-NFrbrcrdr.rerfr0r\c |du}|\} } } ||} |r||d} |d} n>|rU|||d| } |||d| } n||||d| } |||d| } t j|d| gd} t j|d| gd} nT|||d| } |||d| } |jr| | f}| jd}| jd}| | ||j |j } | dd | ||j |j } | dd | ||j |j } |j r|jnd}| j}|t jkrt jrt j}n3t'|jd r |jj}n|jjj}t.d |d | |} | |} | |} t5| | | |||||j|j }|| ||j |j z}||}|sd}|||fS) rhNrrrir&r2_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)r0rMrPuse_top_left_mask)rprVrarTrUr4rqrDshaper^rKrOr_rorMrlryis_autocast_enabledget_autocast_gpu_dtypehasattrrCrweightlogger warning_oncerzrrPrr{rW)r*rbrcrdr.rerfr0r|r[r~rrr query_lengthr} attn_dropout input_dtype target_dtyperrs r,r7zOptFlashAttention2.forward s.T9!&&(( Q{{=11  L."<'*J)!,LL  LT[[1A%B%BBLLJ;;t{{3C'D'Db#NNLL  'T[[%?%?SIIJ;;t{{='A'A2sKKLN1$5z#BJJJJ 9nQ&7%FANNNLLT[[%?%?SIIJ;;t{{='A'A2sKKL ? 8),7N#)!, "2&$((lDNDMZZ ))!Q//44S'4>SWS`aa #--a3388gt~W[Wdee '+}=t||# #( %- ' '(** 8$;== &?@@ 8#{B #{17   $ $$$    (??<88L#|44J'??<88L.     % n"A    !, 3 3Ct~X\XeGe f fmm$9::   )$( !1>AAr-r) r9r:r;r<r)r4rrrrr7r?r@s@r,rrs/XXXXX488<1526"'/3fBfB|fB#5<0fB!u|!45 fB !. fB "%,/ fB fBu|,fB u|Xel3XeEL>Q5RR SfBfBfBfBfBfBfBfBr-rc*eZdZdZ d dejdeejdeeejdeejdeejd ed eejd eejeejeeejfffd Z xZ S)OPTSdpaAttentiona@ OPT sdpa attention module. This module inherits from `OPTAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of sdpa attention and deal with padding tokens in case the input contains any of them. NFrbrcrdr.rerfr0r\c n|s|Atdt||||||S|du}|\} } } |||jz} || d| } |r||d} |d}n>|rU|||d| } || |d| }n||||d| } || |d| }tj |d| gd} tj |d|gd}nT|||d| } || |d| }|j r| |f}|}||ddddddd| j df}|| dkrd nd }tjj| | |||jr|jnd |d }|dd}|| | d}||}|d|fS)NzOPTModel is using SDPA attention, which currently does not support output_attentions=True.failing back to eager attention. remove warning using attn_implementation="eager".)rbr.rerdrfrcrirrr&r2rTFrg?) attn_mask dropout_prPscale)rrr(r7rprVrRrarTrUr4rqrDrrrwscaled_dot_product_attentionrorMr_r`r^rW)r*rbrcrdr.rerfr0r|r[q_lenr~rrr causal_maskrPrr+s r,r7zOPTSdpaAttention.forward}s   ;   e    77??+- /-"3!1 # .T9%**,, UA{{=11DL@ {{<S99   L."<'*J)!,LL  LT[[1A%B%BBLLJ;;t{{3C'D'Db#NNLL  'T[[%?%?SIIJ;;t{{='A'A2sKKLN1$5z#BJJJJ 9nQ&7%FANNNLLT[[%?%?SIIJ;;t{{='A'A2sKKL ? 8),7N%  %%aaaAAA/E1A"1E/E&EFK(/EAIIDD5 h)FF   !&*m> !&&sE266 mmK00 D.00r-r) r9r:r;r<r4rrrrr7r?r@s@r,rrvs 488<1526"'/3X1X1|X1#5<0X1!u|!45 X1 !. X1 "%,/ X1 X1u|,X1 u|Xel3XeEL>Q5RR SX1X1X1X1X1X1X1X1X1X1r-r)eagerflash_attention_2sdpac*eZdZdeffd Z ddejdeejdeejdeeejd ee d ee d eej d eej eeej ej fffd Z xZ S)OPTDecoderLayerrCc@t|j|_t |j|d|_|j|_|j|_t|j |_ tj |j|j|_tj|j|j|j|_tj|j|j|j|_tj |j|j|_dS)NT)rCrDelementwise_affinerF)r(r)rHrIOPT_ATTENTION_CLASSES_attn_implementation self_attndo_layer_norm_beforerMr activation_function activation_fnr LayerNormlayer_norm_elementwise_affineself_attn_layer_normrSffn_dimrNfc1fc2final_layer_normr*rCr+s r,r)zOPTDecoderLayer.__init__s +.v/JKSYfjkkk$*$?!~ #F$>?$&L Nv/S% % % !9T^V^&BTUUU9V^T^&BTUUU " T^PVPt u u ur-NFrbr.rerdrf use_cacher0r\c|}|jr||}|||||||\}} } tj||j|j}||z}|js||}|j} |d| d}|}|jr| |}| |}| |}| |}tj||j|j}||z| }|js| |}|f} |r| | fz } |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`, *optional*): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states )rbrdr0r.rerfrmri)rrrrrwrMrorr{rprrrrr^) r*rbr.rerdrfrr0residualself_attn_weightspresent_key_valuehidden_states_shapeoutputss r,r7zOPTDecoderLayer.forwards4!  $ E 55mDDM?Cnn')%)+/ ?M? ? ; (*; --mt|VZVc-dd  =0 ( E 55mDDM,1%--b-2D2DR2H2HII    $ A 11-@@M// **=99 //  --mt|VZVc-dd !M1778KLL ( A 11-@@M "  , )+ +G  , )+ +Gr-)NNNFFN)r9r:r;rr)r4rrrrr> FloatTensorr7r?r@s@r,rrs%vyvvvvvv(26268<,1$)37MM|M!.M"%,/ M !u|!45 M $D> MD>Mu/0M u (51BEDU1U+V"WW XMMMMMMMMr-raH This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`OPTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zQThe bare OPT Model outputting raw hidden-states without any specific head on top.c.eZdZeZdZdZdgZdZdZ dZ dS)OPTPreTrainedModelmodelTrc|jj}t|tjrJ|jjd||j |jj dSdSt|tj rS|jjd||j -|jj|j  dSdSdS)Nr)meanstd) rCinit_std isinstancerrSrdatanormal_rGzero_ Embedding padding_idx)r*modulers r, _init_weightsz OPTPreTrainedModel._init_weights^sk" fbi ( ( ? M  & &CS & 9 9 9{&  &&((((('&  - - ? M  & &CS & 9 9 9!- "6#56<<>>>>> ? ?--r-N) r9r:r;r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_2_supports_sdparr-r,rrRsJ L&*#*+!N ? ? ? ? ?r-raF Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. for padding use -1. [What are position IDs?](../glossary#position-ids) ceZdZdZdeffd ZdZdZ ddej de e e fd e d e ej d e ej d e e f d Z ddejd e ej d e ej de eejde ejde e d e e de e de e de ejdee effdZxZS) OPTDecoderz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`OPTDecoderLayer`] Args: config: OPTConfig rCctj|_j|_j|_j|_j|_tj jj |j|_ tjj|_j jkr'tjjj d|_nd|_j jkr'tjj jd|_nd|_jr-js&tjjj|_nd|_tjfdt3jD|_jdk|_jdk|_d|_| dS)NFrFrc.g|]}tSr)r).0r~rCs r, z'OPTDecoder.__init__..s!$f$f$f_V%<%<$f$f$fr-rr)!r(r)rM layerdrop pad_token_idrmax_position_embeddingsmax_target_positions vocab_sizerrword_embed_proj_dim embed_tokensr!rHembed_positionsrS project_out project_inr_remove_final_layer_normrrr ModuleListrangenum_hidden_layerslayersr_use_flash_attention_2 _use_sdpagradient_checkpointing post_initrs `r,r)zOPTDecoder.__init__s    ~ )!.$*$B! +L):F&+# r-c|jSrrr*s r,get_input_embeddingszOPTDecoder.get_input_embeddingss   r-c||_dSrrr*values r,set_input_embeddingszOPTDecoder.set_input_embeddingss!r-N inputs_embeds input_shaper/r. head_maskrfc|\}}||z} |jr.|d|vr|nd} |tj|| |jn|}| |fS|tj|| |j}n2|jd| kr!t d|jdd| d|jr|s|t||||} nt||||} | |fS)z: Updates the causal mask for the decoder. Nrrjrz'The provided attention mask has length z, but its length should be z0 (sum of the lengths of current and past inputs)) rr4onesrkrrQrrr) r*rrr/r.rrf batch_size seq_lengthmask_seq_lengthcausal_attention_masks r,_update_causal_maskzOPTDecoder._update_causal_masksO"- J0:=  & 97E7QVW[iViViNNpt !") :}?STTTT#  ).8 8  !"Z OML`aaaNN  !! $ 7 7U.:Nq:QUU"UUU  > "3  8I$N ]tj |d} | |zdz } | dd| df} ||| | }|j||}||z}|jr%|jr|rt$d d }|rd nd}|rd nd}|rd nd}t)|gd gD]z\}}|s|dt+|jkrCt d|dt+|jd|dd{t/|jD]\}}|r||fz }|jr tjg}||jkr4|||nd}|jr3|jr,||j|| |||ndd||| }n||| | |||nd|||}|d}|r|||rdndfz }|r ||dfz }|j||}|j||}|r||fz }|r|nd}| st=d||||fDSt?||||S)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(num_hidden_layers, num_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. for padding use -1. [What are position IDs?](../glossary#position-ids) NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timerizEYou have to specify either decoder_input_ids or decoder_inputs_embedsrr&rr2)r0zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FrrzThe `z` should be specified for z layers, but it is for r)r.r0rerdrfrc3K|]}||V dSrr)rvs r, z%OPTDecoder.forward..s(ttqfgfsfsfsfsfsttr-last_hidden_stater rb attentions) rCrfr ruse_return_dictrQrpr^rrr r4r5r6rrrrorrziplenr enumeraterandr_gradient_checkpointing_func__call__rrtupler)r*r r.rr rrrfr r r0rr/r pos_embedsrball_hidden_statesall_self_attnsnext_decoder_cacher mask_nameidx decoder_layerdropout_probabilityrd layer_outputs next_caches r,r7zOPTDecoder.forward sB2C1N--TXT_Tq$8$D $+Jj "+!6IIDKstt t  "#..**K!r;r?;;II  &',,..ss3KKdee e   --i88MCRC^!3A!6!PY[l1 1 -~   <A>>>L(>9A=CCEEL'+A+B+B(BCL)).:P_k)ll ? & OOM::M% 2  & "4= " "##p" #7@BBD0:d#,6RR$%( k]$C$C   Iy$>>##A&3t{+;+;<<$3 33SEUEU33%NN,,Q/333 #,DK"8"8( 6( 6 C# 6!m%55!} &+jnn#&775D5P_S11VZN* t}  $ A A!*!)&/&;IcNN%  ! ! !. !#8!-7@7LYs^^RV#1&7'!!! *!,M V"}:K5RQQQR'S&UU"  6=#3"55  , 11-@@M   ' ,,];;M  2 -!1 1 +4>''$  utt]J@QSa$bttttt t&+&+%     r-)NNN NNNNNNNNNN)r9r:r;r<rr)rrr4rrr=rrr r>rrrrr7r?r@s@r,rrs%y%%%%%%N!!!"""26,0,0)5)5|)538_)5!$ )5 !. )5 EL) )5$D>)5)5)5)5Z'+15,0=A59$(,0/3&*37| | #| !.| EL) | "$u'8"9: |   12 | D>| $D>| 'tn| d^| u/0|  u-- .| | | | | | | | r-rceZdZdeffd ZdZdZdZee e e e e e ddejd eejd eejd eeejd eejd eedeedeedeedeejdeee ffdZxZS)OPTModelrCct|t||_|dSr)r(r)rdecoderrrs r,r)zOPTModel.__init__s@    !&))  r-c|jjSrr+rrs r,rzOPTModel.get_input_embeddingss |((r-c||j_dSrr-rs r,rzOPTModel.set_input_embeddingss$) !!!r-c|jSr)r+rs r, get_decoderzOPTModel.get_decoder |r-) checkpoint output_typerexpected_outputNr r.rr rrrfr r r0r\c ||n |jj}||n |jj}||n |jj}| | n |jj} |||| |||||||  } | s| St | j| j| j | j S)N r r.r0rr rrrfr r r) rCrfr rrr+rrr rbr) r*r r.rr rrrfr r r0decoder_outputss r,r7zOPTModel.forwards(2C1N--TXT_Tq$8$D $+Jj "+!6IIDKrrrrrrrr7r?r@s@r,r)r)s y )))***+*+?@@&+$. '+15,0=A59$(,0/3&*37* * #* !.* EL) * "$u'8"9: *   12 * D>* $D>* 'tn* d^* u/0*  u-- .* * * A@* * * * * r-r)ceZdZdgZfdZdZdZdZdZdZ dZ e e e  dd ejd eejd eejdeeejdeejdeejdeedeedeedeedeejdeee ffdZedZxZS)OPTForCausalLMzlm_head.weightct|t||_t j|j|jd|_| dSNFrF) r(r)r)rrrSrrlm_headrrs r,r)zOPTForCausalLM.__init__sc    f%% y!;V=NUZ[[[  r-c$|jjjSrrr+rrs r,rz#OPTForCausalLM.get_input_embeddings!z!..r-c(||jj_dSrrBrs r,rz#OPTForCausalLM.set_input_embeddings$*/ '''r-c|jSrr@rs r,get_output_embeddingsz$OPTForCausalLM.get_output_embeddings'r1r-c||_dSrrG)r*new_embeddingss r,set_output_embeddingsz$OPTForCausalLM.set_output_embeddings*s % r-c||j_dSrrr+)r*r+s r, set_decoderzOPTForCausalLM.set_decoder-s$ r-c|jjSrrMrs r,r0zOPTForCausalLM.get_decoder0s z!!r-r3rNr r.rr rlabelsrrfr r r0r\c ||n |jj}| | n |jj} | | n |jj} |j||| |||||| |  } || d} d}||| j }| dddddf}|dddf}t}|| d|jj | d}| s| f| ddz}||f|zn|St|| | j| j| jS)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(num_hidden_layers, num_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. The two additional tensors are only required when the model is used as a decoder in a Sequence to Sequence model. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. for padding use -1. [What are position IDs?](../glossary#position-ids) Returns: Example: ```python >>> from transformers import AutoTokenizer, OPTForCausalLM >>> model = OPTForCausalLM.from_pretrained("facebook/opt-350m") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious. I'm just a little bit of a weirdo." ```Nr6r.rirlosslogitsr rbr)rCrfr rrr+r@r`rzrkr r^rrr rbr)r*r r.rr rrQrrfr r r0rrUrT shift_logits shift_labelsloss_fctoutputs r,r7zOPTForCausalLM.forward3s|2C1N--TXT_Tq$8$D $+Jj &1%.sCnnU_j--aZ=N1O1OPPnnnnnnr-)r)r r^reordered_past layer_pasts ` r,_reorder_cachezOPTForCausalLM._reorder_cachesQ)  J nnnncmnnnnn NNr- NNNNNNNNNNN)r9r:r;_tied_weights_keysr)rrrHrKrNr0rrr:r4r>rrrrrrrr7 staticmethodrar?r@s@r,r=r=s*+///000&&&%%%"""+AP_```'+15,0=A59-1$(,0/3&*37H H #H !.H EL) H "$u'8"9: H   12 H )*H D>H $D>H 'tnH d^H u/0H  u,, -H H H a`H T\r-r=a The OPT Model transformer with a sequence classification head on top (linear layer). [`OPTForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZdeffd Zeeeee e e e  dde ejde ejde ejde eeejd e ejd e ejd e ed e ed e ede ede ejdeee ffdZdZdZxZS)OPTForSequenceClassificationrCct||j|_t||_t j|j|jd|_| dSr?) r(r) num_labelsr)rrrSrscorerrs r,r)z%OPTForSequenceClassification.__init__sj     +f%% Yv94?QVWWW  r-)r2r3rr4 expected_lossNr r.rr rrQrrfr r r0r\c | | n |jj} ||||| ||||| |  } | d} || }||jdd\}}n|jdd\}}|jjd}n|rt j||jj ddz }||jdz}| |j }n)d}t |jjd|t j||j |f}d}|Z|jjf|jdkr d |j_nN|jdkr7|jt jks|jt jkr d |j_n d |j_|jjd krWt+}|jdkr1|||}n|||}n|jjd krGt/}||d|j|d}n*|jjd krt3}|||}| s|f| ddz}||f|zn|St5||| j| j| j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N r r.r0rrrrfr r rr&rirz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rj regressionsingle_label_classificationmulti_label_classificationrS)rCrrrirrr4eqr=argmaxrzrkrrr+r9arange problem_typerhrlr6r squeezer r^rrr rbr)r*r r.rr rrQrrfr r r0transformer_outputsrbrUrsequence_lengthsequence_lengths pooled_logitsrTrXrYs r,r7z$OPTForSequenceClassification.forwards)8&1%rrrrrr7rrr?r@s@r,rfrfs y+*+?@@:4$2. 156:15@D59-1$(,0/3&*37[ [ E,-[ !!23[ E-. [ "%el(;"<= [   12 [ )*[ D>[ $D>[ 'tn[ d^[ u/0[  u66 7[ [ [ A@[ z///0000000r-rfz The OPT Model transformer with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZdeffd Zeeeee  dde e j de e j de e j de eee jd e e j d e e j d e e j d e ed e ede ede ede e j deeeffdZdZdZxZS)OPTForQuestionAnsweringrCct|t||_t j|jd|_|dSr%) r(r)r)rrrSr qa_outputsrrs r,r)z OPTForQuestionAnswering.__init__UsX    f%% )F$>BB r-rPNr r.rr rstart_positions end_positionsrrfr r r0r\c | | n |jj} ||||| |||| | |  } | d}||}|dd\}}|d}|d}d}||t|dkr|d}t|dkr|d}|d}| d| |j }| d| |j }t|}|||}|||}||zdz }| s||f| ddz}||f|zn|St|||| j| j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Returns: Example: ```python >>> from transformers import AutoTokenizer, OPTForQuestionAnswering >>> import torch >>> torch.manual_seed(4) # doctest: +IGNORE_RESULT >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> # note: we are loading a OPTForQuestionAnswering from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> model = OPTForQuestionAnswering.from_pretrained("facebook/opt-350m") >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" >>> inputs = tokenizer(question, text, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> answer_offset = len(tokenizer(question)[0]) >>> predict_answer_tokens = inputs.input_ids[ ... 0, answer_offset + answer_start_index : answer_offset + answer_end_index + 1 ... ] >>> predicted = tokenizer.decode(predict_answer_tokens) >>> predicted ' a nice puppet' ```Nrlrrrir2) ignore_indexr&)rT start_logits end_logitsrbr)rCrrrsplitrtr`rrpclamprzrkr rrbr)r*r r.rr rrrrrfr r r0rurbrUrr total_loss ignored_indexrX start_lossend_lossrYs r,r7zOPTForQuestionAnswering.forward]s;x&1%rrrrrr7rrr?r@s@r,rrMsy+*+?@@+GVefff156:15@D596:48$(,0/3&*37j j E,-j !!23j E-. j "%el(;"<= j   12 j "%"23j   01j D>j $D>j 'tnj d^j u/0j  u22 3j j j gfA@j X///0000000r-r)@r<typingrrrrr4torch.utils.checkpointrtorch.nnrr r activationsr generationr modeling_attn_mask_utilsrrmodeling_outputsrrrrmodeling_utilsrutilsrrrrrrrconfiguration_optrmodeling_flash_attention_utilsr get_loggerr9rr9r:r;r{r}r|rr!ModulerBrrrrOPT_START_DOCSTRINGrr8rr)r=rfrrr-r,rs//////////// AAAAAAAAAA!!!!!!)))))) .-----)(((((KJJJJJJ  H % %)&+F'(;;;;;BL;;;8[B[B[B[B[B29[B[B[B|vBvBvBvBvBvBvBvBr_1_1_1_1_1|_1_1_1F+ _____bi___D"W??????? ?(@F\ \ \ \ \ #\ \ \ ~WA A A A A !A A A Hqqqqq'qqqh s0s0s0s0s0#5s0s0s0l {0{0{0{0{00{0{0{0{0{0r-