gjdZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZdd lmZmZdd lmZmZmZmZdd lmZdd lmZmZmZmZd dl m!Z!ej"e#Z$dZ%dZ&Gddej'Z(Gddej'Z)Gddej*Z+Gdde+Z,e+e,dZ-Gddej*Z.GddeZ/dZ0dZ1ed e0Gd!d"e/Z2ed#e0Gd$d%e/eZ3ed&e0Gd'd(e/Z4ed)e0Gd*d+e/Z5dS),zPyTorch BioGPT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GenerationMixin)!_prepare_4d_causal_attention_mask*_prepare_4d_causal_attention_mask_for_sdpa))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) BioGptConfigzmicrosoft/biogptrcLeZdZdZdedeffd Zd dejdeffd ZxZ S) BioGptLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcjd|_t||jz|dS)N)offsetsuper__init__)selfrr __class__s f/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/biogpt/modeling_biogpt.pyr"z)BioGptLearnedPositionalEmbedding.__init__9s3  $+5}EEEEErattention_maskpast_key_values_lengthc|}tj|d||zdz }|dd|df}t ||jzS)z3`input_ids_shape` is expected to be [bsz x seqlen].rdimN)longtorchcumsumtype_asr!forwardr )r#r'r( positionsr$s r%r0z(BioGptLearnedPositionalEmbedding.forward?s',,..\.a888@@PPSaaggiilmm aaa!7!8!889 wwy4;6777r&)r) __name__ __module__ __qualname____doc__intr"r- LongTensorr0 __classcell__r$s@r%rr4sFsF3FFFFFF 8 8e&6 8PS 8 8 8 8 8 8 8 8 8 8r&rc \eZdZdZd dedededeeffd Zdej ffd Z xZ S) BioGptScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?rr padding_idx embed_scalec\t|||||_dSN)r!r"r>)r#rrr=r>r$s r%r"z"BioGptScaledWordEmbedding.__init__Rs-  DDD&r& input_idscVt||jzSr@)r!r0r>)r#rAr$s r%r0z!BioGptScaledWordEmbedding.forwardVs!wwy))D,<<|rU|||d|} |||d|} n||||d|} |||d|} tj|d| gd} tj|d| gd} nT|||d|} |||d|} |j r| | f}||j zd|j f}|| | |j |} | j |} | j |} | d}tj| | dd}|||j z| |fkr2t!d||j z| |fd||||d| |fkr+t!d |d| |fd|| ||j | ||z}| ||j z| |}t"j|d}|||j fkr-t!d |j fd|| dddd| ||j | |z}| ||j z| |}|r=| ||j | |}| ||j z| |}nd}t"j||j|j }tj|| }|||j z| |j fkr7t!d ||j z| |j fd|| ||j | |j }|dd}| || |j}||}|||fS) #Input shape: Batch x Time x ChannelNrrrr*z$Attention weights should be of size , but is z!Attention mask should be of size z/Head mask for a single layer should be of size ptraining `attn_output` should be of size )sizerWrSshaper`rUrVr-catrKrIrQr]reshapebmmr^rRr functionalsoftmaxrJrmrHrX)r#rarbrcr'rdreis_cross_attentionr[tgt_len_ query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r%r0zBioGptAttention.forwards.T9',,..Wa{{=11DL@  L*q!'*.>.DQ.GGG(*J)!,LL  LT[[1A%B%BBLLJ;;t{{3C'D'Db#NNLL  'T[[%?%?SIIJ;;t{{='A'A2sKKLN1$5z#BJJJJ 9nQ&7%FANNNLLT[[%?%?SIIJ;;t{{='A'A2sKKL ? 8),7NDN*B > Ct{{<#>>CZP 'Z'4 +|+Z8 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**   %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL',,S4>-A7GTTL},,\r,BB  &##%%$.)::: 1t~FW11',,..11+//2q!<<|?P?PQTVZVdfmov?w?wwL',,S4>-A7GTTL  ) %1$5$5c4>7T[$\$\ !055cDN6JGU\]]LL$( !]**<4AAr&)rGFTFNNNNNF)r2r3r4r5r6rCboolrrr"r-rDr`rr0r8r9s@r%rFrF[sGG  )-CCCC C  C  CC&CCCCCC>eU\eCeceeee 488<1526"'vBvB|vB#5<0vB!u|!45 vB !. vB "%,/ vB vB u|Xel3XeEL>Q5RR SvBvBvBvBvBvBvBvBr&rFc eZdZ d dejdeejdeeejdeejdeejded eejeejeeejfffd ZxZ S) BioGptSdpaAttentionNFrarbrcr'rdrerfc|s|Atdt||||||S|du}|\}} } ||} |r6|4|djd|jdkr|d} |d} n>|rU|||d|} || |d|} n||||d|} || |d|} tj |d| gd} tj |d| gd} nT|||d|} || |d|} |j r| | f}|| | |} |j r || dkrd nd }tjj| | | ||jr|jnd | }|||j| |jfkr5t+d ||j| |jfd||dd}||| |j}||}|d|fS)rhNaBioGptModel is using BioGptSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `layer_head_mask` not None. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rbrcr'rdrerrrrir*TFrG) attn_mask dropout_prMrnrj)logger warning_oncer!r0rorWrpr`rUrVr-rqrKrMrrtscaled_dot_product_attentionrmrJrIrQrRr^rrrHrX)r#rarbrcr'rdrervr[rwrxryrzr{rMrr$s r%r0zBioGptSdpaAttention.forwardsK   ;   l   77??!1-- /"3 # .T9',,..Wa{{=11  L*q!'*.>.DQ.GGG(*J)!,LL  LT[[1A%B%BBLLJ;;t{{3C'D'Db#NNLL  'T[[%?%?SIIJ;;t{{='A'A2sKKLN1$5z#BJJJJ 9nQ&7%FANNNLLT[[%?%?SIIJ;;t{{='A'A2sKKL ? 8),7N{{<#>> !N`~/E'TU++DD[` h)FF   $&*mQ5RR Sf1f1f1f1f1f1f1f1f1f1r&r)eagersdpaceZdZdeffd Z ddejdeejdeejd eeejd ee d ee d eej eeej ej fffd Z xZ S)BioGptDecoderLayerrNc,t|j|_t |j|j|j|jdd|_|j |_ t|j |_ |j|_tj|j|_tj|j|j|_tj|j|j|_tj|j|_dS)NT)rHrIrJrKrM)r!r" hidden_sizerHBIOGPT_ATTENTION_CLASSES_attn_implementationnum_attention_headsattention_probs_dropout_prob self_attnhidden_dropout_probrJr hidden_act activation_fnactivation_dropoutr LayerNormself_attn_layer_normrTintermediate_sizefc1fc2final_layer_normr#rNr$s r%r"zBioGptDecoderLayer.__init__ks +1&2MNn07    1 #F$56"(";$&L$@$@!9T^V-EFF9V5t~FF " T^ < <r&NFTrar'rdrcre use_cacherfcz|}||}| |ddnd}||||||\}} } tj||j|j}||z}|}||}||}||}tj||j |j}| |}tj||j|j}||z}|f} |r| | fz } |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Nr)rarcr'rdrerk) rrrrtrJrmrrrrr) r#rar'rdrcrerresidualself_attn_past_key_valueself_attn_weightspresent_key_valueoutputss r%r0zBioGptDecoderLayer.forwardso0!11-@@ :H9S>"1"#5#5Y] >Bnn'3)+/ ?M? ? ; (*; --mt|VZVc-dd  =0 !--m<< // **=99  --mt?Vaean-oo //  --mt|VZVc-dd  =0  "  , )+ +G  , )+ +Gr&)NNNFT) r2r3r4rr"r-rDrrr FloatTensorr0r8r9s@r%rrjs=|======026268<,1$(<<|<!.<"%,/ < !u|!45 < $D> <D>< u (51BEDU1U+V"WW X<<<<<<<<r&rc(eZdZdZeZdZdZdZdZ dS)BioGptPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. biogptTct|tjrT|jjd|jj|j |jj dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|jj |jj ddSdS)zInitialize the weightsrG)meanstdNr<) isinstancerrTweightdatanormal_rNinitializer_rangerLzero_ Embeddingr=rfill_)r#modules r% _init_weightsz#BioGptPreTrainedModel._init_weightss) fbi ( ( * M  & &CT[5R & S S S{&  &&((((('&  - - * M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r&N) r2r3r4r5r config_classbase_model_prefixsupports_gradient_checkpointing_supports_sdparr&r%rrsE  L &*#N*****r&raJ This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`~BioGptConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare BioGPT Model transformer outputting raw hidden-states without any specific head on top.ceZdZdeffd ZdZdZee de e e e  ddeejd eejd eejd eejd eeeejd eedeedeedeedeee ffdZxZS) BioGptModelrNct|_j|_j|_j|_j|_ j rtj jnd}tj|j|j ||_t!j|j|_t'jfdt+jD|_t'j|j|_d|_jdk|_|dS)Nr<)r>c.g|]}tSr)r).0rxrNs r% z(BioGptModel.__init__..0s"$i$i$iA%7%?%?$i$i$ir&Fr)r!r"rN layerdroprrJrrH pad_token_idr=scale_embeddingmathsqrtr; vocab_size embed_tokensrmax_position_embeddingsembed_positionsr ModuleListrangenum_hidden_layerslayersr layer_normgradient_checkpointingr _use_sdpa post_init)r#rNr>r$s ` r%r"zBioGptModel.__init__"s!     )1 +!.7=7MVdi 2333SV 5  t~t/?[    @@^`d`noom$i$i$i$ivOgIhIh$i$i$ijj ,t~66&+#4> r&c|jSr@rr#s r%get_input_embeddingsz BioGptModel.get_input_embeddings8s   r&c||_dSr@rr#values r%set_input_embeddingsz BioGptModel.set_input_embeddings;s!r&batch_size, sequence_length checkpoint output_typerNrAr' head_mask inputs_embedspast_key_valuesrreoutput_hidden_states return_dictrfc ||n |jj}||n |jj}||n |jj}| | n |jj} ||t d||} | } n=|,|dd} |dddddf} nt d||ddjdnd} ||| }|Btj |jd|jd| zftj |j }nD|jd| | dzkr*t d|jdd | | dzd | || } |jr|s|t|| || }nt!|| || }|| z}t"j||j|j }|jr%|jr|rt,d d }|rdnd}|rdnd}d}|rdnd}t1|jD]\}}|r||fz }|jr tjg}||jkr4|||nd}|jr2|jr+||j|||||ndd||}n||||||nd|||}|d}|r|||rdndfz }|r ||dfz }|r||fz }||}|r|nd}| st?d|||||fDStA|||||S)NzDYou cannot specify both input_ids and inputs_embeds at the same timeriz5You have to specify either input_ids or inputs_embedsrrr)dtypedevicez'The provided attention mask has length z, but its length should be z0 (sum of the lengths of current and past inputs)rkzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr)r'rdrcrerc3K|]}||V dSr@r)rvs r% z&BioGptModel.forward..s0= ===r&)last_hidden_staterra attentionscross_attentions)!rNrerruse_return_dictrRrorprr-onesrrrrrr rrtrJrmrrr enumeraterrandr_gradient_checkpointing_func__call__rtupler)r#rAr'rrrrrerrinput input_shaper(r1raall_hidden_statesall_self_attnsall_cross_attentionsnext_decoder_cacheidx decoder_layerdropout_probabilityrc layer_outputs next_caches r%r0zBioGptModel.forward>s$2C1N--TXT_Tq$8$D $+Jj "+!6IIDKcdd d  "E**,,KK  &',,..ss3K!!!!QQQ(+EETUU UDSC^!3A!6!Q(O O Om.:Nq:Qmm)KN:mmm  ((9OPP > "3  8IH ]''$  '5FXlm  9+&+%1     r&) NNNNNNNNN)r2r3r4rr"rrrBIOGPT_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOCrr-r7rrrDrrr0r8r9s@r%rrs |,!!!"""+*+B+I+IJg+h+hii&=$156:1559@D$(,0/3&*J J E,-J !!23J E-. J   12 J "%el(;"<= J D>J $D>J 'tnJ d^J  u?? @J J J ji J J J J J r&rzHBioGPT Model with a `language modeling` head on top for CLM fine-tuning.ceZdZdgZfdZdZdZee de e e e  ddeejd eejd eejd eejd eeeejd eejdeedeedeedeedeee ffdZedZxZS)BioGptForCausalLMzoutput_projection.weightct|t||_t j|j|jd|_| dSNFrP) r!r"rrrrTrroutput_projectionrrs r%r"zBioGptForCausalLM.__init__sb    !&)) !#6+=v?PW\!]!]!] r&c|jSr@r rs r%get_output_embeddingsz'BioGptForCausalLM.get_output_embeddingss %%r&c||_dSr@r )r#new_embeddingss r%set_output_embeddingsz'BioGptForCausalLM.set_output_embeddingss!/r&rrNrAr'rrrlabelsrrerrrfc B| | n |jj} ||||||||| |  } | d} || } d}|| ddddddf}|ddddf}t }||d|jj|d}| s| f| ddz}||f|zn|St|| | j | j | j | j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)r'rrrrrerrrrir)losslogitsrrarr) rNrrr r_rr]rrrrarr)r#rAr'rrrrrrerrrsequence_outputprediction_scoreslm_lossshifted_prediction_scoresloss_fctoutputs r%r0zBioGptForCausalLM.forwardsj2&1%.*sCnnU_j--aZ=N1O1OPPnnnnnnr&)r)rr reordered_past layer_pasts ` r%_reorder_cachez BioGptForCausalLM._reorder_cache%sQ)  J nnnncmnnnnn NNr& NNNNNNNNNN)r2r3r4_tied_weights_keysr"r rrrrrrrrrr-r7rrrDrrr0 staticmethodr#r8r9s@r%rrs55&&&000+*+B+I+IJg+h+hii&5$156:1559@D-1$(,0/3&*7 7 E,-7 !!237 E-. 7   12 7 "%el(;"<= 7 )*7 D>7 $D>7 'tn7 d^7  u77 87 7 7 ji 7 r\r&rz BioGPT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. ceZdZfdZeeeeee  dde e j de e j de e j de e j de eee jd e e j d e e j d e ed e ed e ede edeeeffdZxZS)BioGptForTokenClassificationcxt||j|_t||_t |dr|j|j}n|j}tj ||_ tj |j |j|_ |dS)Nclassifier_dropout)r!r" num_labelsrrhasattrr*rrDropoutrJrTr classifierr)r#rNr*r$s r%r"z%BioGptForTokenClassification.__init__7s     +!&)) 6/ 0 0 1` a classification loss is computed (Cross-Entropy). Nrr'rrrrerrrrirr)rrrar)rNrrrJr.rr]r+r-whererY ignore_indexr/rrar)r#rAr/r'rrrrrrerrtransformer_outputsrarrr active_loss active_logits active_labelsrs r%r0z$BioGptForTokenClassification.forwardEs4&1%= $D>= 'tn= d^=  u++ ,= = = DC = = = = = r&r(a The BioGpt Model transformer with a sequence classification head on top (linear layer). [`BioGptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it is required to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZdeffd Zeeeee e  dde e j de e jde e jde eee jd e e jd e e j d e ed e ed e ede edeee ffdZdZdZxZS)BioGptForSequenceClassificationrNct||j|_t||_t j|j|jd|_| dSr ) r!r"r+rrrrTrscorerrs r%r"z(BioGptForSequenceClassification.__init__si     +!&)) Yv14?OOO  r&rNrAr'rrrrrrerrrfc t| | n |jj} ||||||||| |  } | d} || } ||jdd\}}n|jdd\}}|jjd}ny|Nt j||jjddz | j }n)d}t |j jd| t j|| j |f}d}|Z|jjf|jdkr d |j_nN|jdkr7|jt jks|jt jkr d |j_n d |j_|jjd krWt+}|jdkr1|||}n|||}n|jjd krGt/}||d|j|d}n*|jjd krt3}|||}| s|f| ddz}||f|zn|St5||| j| j| j S) r1Nr2rrrirz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)rrrrar)rNrrr<rprr-nesumrrrrr$r2arange problem_typer+rr,r6r squeezerr]rrrrar)r#rAr'rrrrrrerrr5rar batch_sizesequence_length pooled_logitsrrrs r%r0z'BioGptForSequenceClassification.forwards2&1%V $D>V 'tnV d^V  u66 7V V V DC V p((()))))))r&r:)6r5rtypingrrrr-torch.utils.checkpointrtorch.nnrrr activationsr generationr modeling_attn_mask_utilsr rmodeling_outputsrrrrmodeling_utilsrutilsrrrrconfiguration_biogptr get_loggerr2rrrrrr;ModulerFrrrrBIOGPT_START_DOCSTRINGrrrr(r:rr&r%rYs. )))))))))) AAAAAAAAAA!!!!!!))))))uuuuuuuu .----- /.....  H % %(  88888r|8882 = = = = = = = =[B[B[B[B[Bbi[B[B[B~g1g1g1g1g1/g1g1g1V  RRRRRRRRj*****O***8 4nfm m m m m 'm m m `RTjXXXXX-XXXv R R R R R #8R R R j l)l)l)l)l)&;l)l)l)l)l)r&