g*0dZddlZddlmZmZmZmZmZmZddl Z ddl Z ddl m Z ddl mZmZmZddlmZddlmZdd lmZdd lmZdd lmZmZmZmZmZmZdd l m!Z!dd l"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.ddl/m0Z0e+rddl1m2Z2e-j3e4Z5dZ6dZ7de8de8de j9fdZ:de8de8de j9fdZ;Gdde j<Z=Gdde j<Z>Gdde>Z?Gdd e>Z@Gd!d"e j<ZAe>e?e@d#ZBGd$d%e j<ZCGd&d'e j<ZDGd(d)e!ZEd*ZFd+ZGe)d,eFGd-d.eEZHe)d/eFGd0d1eEZIe)d2eFGd3d4eEZJe)d5eFGd6d7eEZKe)d8eFGd9d:eEZLe)d;eFGd<d=eEZMdS)>z PyTorch DistilBERT model adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) and in part from HuggingFace PyTorch version of Google AI Bert model (https://github.com/google-research/bert) N)DictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)get_activation)PretrainedConfig)is_deepspeed_zero3_enabled)#_prepare_4d_attention_mask_for_sdpa)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indices"is_torch_greater_or_equal_than_2_2prune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardis_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10loggingreplace_return_docstrings)DistilBertConfig)_flash_attention_forwardzdistilbert-base-uncasedr%n_posdimoutc$troddl}|j|d5tjdkrt|||ddddS#1swxYwYdSt|||dS)Nr) modifier_rankr'r(r))r deepspeedzeroGatheredParameterstorch distributedget_rank_create_sinusoidal_embeddings)r'r(r)r-s n/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/distilbert/modeling_distilbert.pycreate_sinusoidal_embeddingsr5Hs!##E ^ . .s! . D D M M ))++q00-EsLLLL M M M M M M M M M M M M M M M M M M &EsDDDDDDs5A11A58A5c tjfdt|D}d|_t jtj|dddddf|dddddf<t jtj|dddddf|dddddf<|dS)NcJg|]fdtDS)c Rg|]#}tjdd|dzzz z $S)i')nppower).0jr(poss r4 z<_create_sinusoidal_embeddings...Ts7\\\AcBHUAaL34F$G$GG\\\)range)r<r>r(s @r4r?z1_create_sinusoidal_embeddings..Ts=uuuad\\\\\QVWZQ[Q[\\\uuur@Frr9r$) r:arrayrA requires_gradr0 FloatTensorsincosdetach_)r'r(r) position_encs ` r4r3r3Ss8uuuuhmnshthtuuuvvLC$RVLADqD,A%B%BCCC14a4L$RVLADqD,A%B%BCCC14a4LKKMMMMMr@cdeZdZdeffd ZddejdeejdejfdZxZ S) Embeddingsconfigcttj|j|j|j|_tj|j|j|_ tj |jd|_ tj |j |_ | dtj|jdddS)N) padding_idx-q=eps position_ids)r$F) persistent)super__init__r Embedding vocab_sizer( pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormDropoutdropoutregister_bufferr0arangeexpandselfrK __class__s r4rUzEmbeddings.__init__\s !|F,=vzW]Wjkkk#%<0NPVPZ#[#[ fje<<<z&.11   EL)GHHOOPWXXej      r@N input_ids input_embedsreturnc|||}|d}t|dr|jddd|f}nNt j|tj|j}|d |}| |}||z}| |}| |}|S)a Parameters: input_ids (torch.Tensor): torch.tensor(bs, max_seq_length) The token ids to embed. input_embeds (*optional*, torch.Tensor): The pre-computed word embeddings. Can only be passed if the input ids are `None`. Returns: torch.tensor(bs, max_seq_length, dim) The embedded tokens (plus position embeddings, no token_type embeddings) Nr$rQ)dtypedevicer) rYsizehasattrrQr0r`longrj unsqueeze expand_asr[r\r^)rcrerf seq_lengthrQr[ embeddingss r4forwardzEmbeddings.forwardgs  // ::L!&&q)) 4 ( ( J,QQQ  ^>yIIL"66|DD!$77 ^^J// \\*-- r@N) __name__ __module__ __qualname__rrUr0Tensorrrr __classcell__rds@r4rJrJ[s  /       Xel=S_d_kr@rJceZdZdeffd ZdeefdZ ddej dej d ej d ej d e ej d e d e ej dffdZ xZS)MultiHeadSelfAttentionrKct||_|j|_|j|_t j|j|_d|_ |j|jzdkr td|jd|jdt j |j|j|_ t j |j|j|_ t j |j|j|_t j |j|j|_t!|_|j|jz|_dS)NpFrzself.n_heads: z must divide self.dim:  evenly in_features out_features)rTrUrKn_headsr(r r]attention_dropoutr^ is_causal ValueErrorLinearq_link_linv_linout_linset pruned_headsattention_head_sizerbs r4rUzMultiHeadSelfAttention.__init__s  ~ :zF$<===  8dl "a ' 'ddlddSWS[dddee eY6:FJOOO Y6:FJOOO Y6:FJOOO yVZfjQQQ &)ee#'8t|#;   r@headsct|dkrdSt||j|j|j\}}t |j||_t |j||_t |j||_t |j |d|_ |jt|z |_|j|jz|_ |j ||_dS)Nrr$r() lenrrrrrrrrrr(union)rcrindexs r4 prune_headsz"MultiHeadSelfAttention.prune_headss u::?? F7 4<!94;L  u( E:: ' E:: ' E:: )$,1EEE |c%jj0 +dl: -33E::r@NFquerykeyvaluemask head_maskoutput_attentionsrg.c|\}}|d} jjzdd| f} dtjdtjffd } dtjdtjffd } | |} | |}| |}| tj z } tj | | dd}|dk |  |}||tjtj|jj}t&j|d }|}|||z}tj ||}| |}|}|r||fS|fS)  Parameters: query: torch.tensor(bs, seq_length, dim) key: torch.tensor(bs, seq_length, dim) value: torch.tensor(bs, seq_length, dim) mask: torch.tensor(bs, seq_length) Returns: weights: torch.tensor(bs, n_heads, seq_length, seq_length) Attention weights context: torch.tensor(bs, seq_length, dim) Contextualized layer. Optional: only if `output_attentions=True` r$xrgcf|djddSseparate headsrRr$r9viewr transposerbs dim_per_headrcs r4shapez-MultiHeadSelfAttention.forward..shapes-66"b$, ==GG1MM Mr@c|dddjzSz group headsr$r9rRr contiguousrrrs r4unshapez/MultiHeadSelfAttention.forward..unshapes<;;q!$$//1166r2t|l?Z[[ [r@r9r rrRr)rkr(rr0rwrrrmathsqrtmatmulrrro masked_filltensorfinforiminr functionalsoftmaxr^r)rcrrrrrrq_lengthr(k_length mask_reshprrqkvscoresweightscontextrrs` @@r4rrzMultiHeadSelfAttention.forwards7("JJLLHc88A;;x4</ !Q)  NU\ Nel N N N N N N N N \u| \  \ \ \ \ \ \ \ \ E$**U## $ $ E$**S// " " E$**U## $ $  ,'' 'aQ!2!233  ++55f==## %,u{6<88<==  -''B'77,,w''   )G,w**''"",,w''  W% %: r@NF)rtrurvrrUrintrr0rwrboolrrrrxrys@r4r{r{s</<<<<<<,;c;;;;,-1"'>>|>\>| > l > EL) > > u|S !>>>>>>>>r@r{ceZdZdZfdZ ddejdejdejdejd eejd ed e ejd ffd Z xZ S)DistilBertFlashAttention2aS DistilBert flash attention module. This module inherits from `MultiHeadSelfAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cbtj|i|t |_dSrs)rTrUr!_flash_attn_uses_top_left_mask)rcargskwargsrds r4rUz"DistilBertFlashAttention2.__init__s9$)&))) 3V2W2W.W+++r@NFrrrrrrrg.c |\}}jjzdtjdtjffd } | |} | |} | |} jr j j nd} | j tj krtj rtj}n3tj dr j j}njjj }t$d|d| |} | |} | |} t+| | | ||| jj}||jz}|}|r||fS|fS) rrrgc>|djS)rrR)rrr batch_sizerrcs r4reshapez2DistilBertFlashAttention2.forward..reshapes66*b$, EE Er@_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)r^use_top_left_maskr)rkr(rr0rwrrrtrainingrKrrifloat32is_autocast_enabledget_autocast_gpu_dtyperlrweightlogger warning_oncetor&rrrr)rcrrrrrrrr(r query_states key_states value_states attn_dropout target_dtype attn_weightsattn_weights_reshaped attn_outputrrs` @@r4rrz!DistilBertFlashAttention2.forwards(%*JJLL! Hcx4</  Fu| F  F F F F F F F F wtzz%0011 WTZZ__-- wtzz%0011 8< Nt{443    . .(** 7$;== &?@@ 7#{B #z06   $ $$$    (??<88L#|44J'??<88L/      "An    !- 4 4Z4 !r@r) rtrurv__doc__rUr0rwrrrrrrxrys@r4rrsXXXXX-1"'N"N"|N"\N"| N" l N" EL) N" N" u|S !N"N"N"N"N"N"N"N"r@rceZdZdeffd Z ddejdejdejdejd eejd ed e ejd fffd Z xZ S)DistilBertSdpaAttentionrKc~t||j|_t |_dS)N)rK)rTrUr dropout_probrrequire_contiguous_qkvrbs r4rUz DistilBertSdpaAttention.__init__Ss: '''"4*L&L###r@NFrrrrrrrg.ct|s|@tdt||||||S|\}}jjzdtjdtjffd }dtjdtjffd } | |} | |} | |} j rN| j jdkr>|<| } | } | } tjj| | | |jrjndd } | | } | } | fS) rNaDistilBertSdpaAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rrgcf|djddSrrrs r4rz.DistilBertSdpaAttention.forward..shapes/66*b$, EEOOPQSTUU Ur@c|dddjzSrrrs r4rz0DistilBertSdpaAttention.forward..unshapes=;;q!$$//1166z2t|VbGbcc cr@cudarF) attn_mask dropout_pr)rrrTrrrkr(rr0rwrrrrrjtyperr rscaled_dot_product_attentionrrr)rcrrrrrr_rrrrrrrrrds` @@r4rrzDistilBertSdpaAttention.forwardXs(   5   d    77??!  !::<< Aqx4</  VU\ Vel V V V V V V V V du| d  d d d d d d d d E$**U## $ $ E$**S// " " E$**U## $ $  & 18=F+B+BtGW A A Ah)FF +/=Ad''c G  gk** ll;// ~r@r rtrurvrrUr0rwrrrrrrxrys@r4rrRsM/MMMMMM-1"'GG|G\G| G l G EL) G G u|S !GGGGGGGGGGr@rcleZdZdeffd ZdejdejfdZdejdejfdZxZ S)FFNrKcrttj|j|_|j|_d|_tj|j|j |_ tj|j |j|_ t|j |_ dS)Nr}r$r)rTrUr r]r^chunk_size_feed_forward seq_len_dimrr( hidden_dimlin1lin2r activationrbs r4rUz FFN.__init__s zFN333 '-'E$I&*6CTUUU I&*;&*UUU ():;;r@inputrgcDt|j|j|j|Srs)rff_chunkrr)rcrs r4rrz FFN.forwards (8TVZVfhmnnnr@c||}||}||}||}|Srs)rrrr^)rcrrs r4rz FFN.ff_chunksG IIe   OOA   IIaLL LLOOr@) rtrurvrrUr0rwrrrrxrys@r4rrs</<<<<<<oU\oelooooelu|r@r)eagerflash_attention_2sdpac eZdZdeffd Z d dejdeejdeejded e ejd ff d Z xZ S) TransformerBlockrKct|j|jzdkr t d|jd|jdt |j||_tj |jd|_ t||_ tj |jd|_ dS)Nrzconfig.n_heads z must divide config.dim rrN)normalized_shaperP)rTrUr(rrDISTILBERT_ATTENTION_CLASSES_attn_implementation attentionr r\ sa_layer_normrffnoutput_layer_normrbs r4rUzTransformerBlock.__init__s  : &! + +jv~jjW]Wajjjkk k5f6QRSYZZ\6:5QQQv;;!#vzu!U!U!Ur@NFrrrrrg.ch|||||||}|r|\}}n>t|tur tdt|d|d}|||z}||}|||z}|f}|r|f|z}|S)ae Parameters: x: torch.tensor(bs, seq_length, dim) attn_mask: torch.tensor(bs, seq_length) Returns: sa_weights: torch.tensor(bs, n_heads, seq_length, seq_length) The attention weights ffn_output: torch.tensor(bs, seq_length, dim) The output of the transformer block contextualization. )rrrrrrz$sa_output must be a tuple but it is z typer)rrtuple TypeErrorr r r ) rcrrrr sa_output sa_weights ffn_outputoutputs r4rrzTransformerBlock.forwards"NN/ #    %$- !IzzIe++ ]tI ] ] ]^^^!! I&&y1}55 XXi(( #'#9#9*y:P#Q#Q   , ]V+F r@)NNFrrys@r4rrs V/ V V V V V V -1,0"' )) <)EL))EL) )  ) u|S ! ))))))))r@rceZdZdeffd Z ddejdeejdeejded ed eed e e e ejd fffd Z xZ S) TransformerrKctj|_tjfdt jD|_d|_dS)Nc.g|]}tS)r)r<rrKs r4r?z(Transformer.__init__..s"#]#]#]$4V$<$<#]#]#]r@F)rTrUn_layersr ModuleListrAlayergradient_checkpointingrbs `r4rUzTransformer.__init__sa  ]#]#]#]#]eFOF\F\#]#]#]^^ &+###r@NFrrrroutput_hidden_states return_dictrg.cZ|rdnd}|rdnd}|} t|jD]\} } |r|| fz}|jr,|jr%|| j| ||| |} n| | ||| |} | d} |rAt | dkrtdt | | d} || fz}t | dkrtdt | |r|| fz}|std | ||fDSt| || S) a Parameters: x: torch.tensor(bs, seq_length, dim) Input sequence embedded. attn_mask: torch.tensor(bs, seq_length) Attention mask on the sequence. Returns: hidden_state: torch.tensor(bs, seq_length, dim) Sequence of hidden states in the last (top) layer all_hidden_states: Tuple[torch.tensor(bs, seq_length, dim)] Tuple of length n_layers with the hidden states from each layer. Optional: only if output_hidden_states=True all_attentions: Tuple[torch.tensor(bs, n_heads, seq_length, seq_length)] Tuple of length n_layers with the attention weights from each layer Optional: only if output_attentions=True rNrRr9z7The length of the layer_outputs should be 2, but it is rr$z7The length of the layer_outputs should be 1, but it is c3K|]}||V dSrsr)r<rs r4 z&Transformer.forward..?s(ggqYZYfYfYfYfYfggr@)last_hidden_state hidden_states attentions) enumeraterrr_gradient_checkpointing_func__call__rrr r)rcrrrrrrall_hidden_statesall_attentions hidden_statei layer_module layer_outputsr#s r4rrzTransformer.forwards.#7@BBD0:d (44 u uOA|# H$5$G!* t}  $ A A ) aL% !! !-  aL% !! ),L  u}%%**$%s_bcp_q_q%s%sttt*1- !/:-!?}%%**$%s_bcp_q_q%s%sttt+  D 1\O C  hgg\3Dn$Uggggg g*:KXf    r@)NNFFN)rtrurvrrUr0rwrrrrrrrrxrys@r4rrs,/,,,,,,-1,0"'%*&*C C <C EL)C EL) C  C # C d^C  elC&7 88 9C C C C C C C C r@rc@eZdZdZeZdZdZdZdZ dZ de j fdZ dS)DistilBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. N distilbertTmodulec.t|tjrT|jjd|jj|j |jj dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|jj |jj ddSt|tr=|jjr3t!|jj|jj|jjdSdSdS)zInitialize the weights.r)meanstdNg?) isinstancer rrdatanormal_rKinitializer_rangebiaszero_rVrMr\fill_rJsinusoidal_pos_embdsr5rZr(r[)rcr0s r4 _init_weightsz'DistilBertPreTrainedModel._init_weightsSs fbi ( (  M  & &CT[5R & S S S{&  &&((((('&  - -  M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - -  K  " " $ $ $ M  $ $S ) ) ) ) )  + +  0P  ( 3T[_fF`Fg         r@)rtrurvrr% config_classload_tf_weightsbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_2_supports_sdpar Moduler<rr@r4r.r.Fs^ $LO$&*#!NBIr@r.aD This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`DistilBertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zfThe bare DistilBERT encoder/transformer outputting raw hidden-states without any specific head on top.ceZdZdeffd ZdejfdZdefdZ dejfdZ dejfd Z d e ee e effd Zeed eeee  ddeejdeejdeejdeejdeedeedeedeeeejdfffdZxZS)DistilBertModelrKct|t||_t ||_|jdk|_|jdk|_| dS)Nrr) rTrUrJrqr transformerr_use_flash_attention_2 _use_sdpa post_initrbs r4rUzDistilBertModel.__init__sr    $V,,&v..&,&AEX&X#4> r@rgc|jjSz1 Returns the position embeddings )rqr[rcs r4get_position_embeddingsz'DistilBertModel.get_position_embeddingss22r@new_num_position_embeddingsc||jjz }|dkrdStd|d||j_|jjj}tj |jj|jj |j_|jj r1t|jj|jj |jjntj5|dkr*tj||jjjd| <n+tj|d||jj_dddn #1swxYwY|jj|jdS) Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`. Arguments: new_num_position_embeddings (`int`): The number of new position embedding matrix. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. rNz(Setting `config.max_position_embeddings=z`...r,)rKrZrinforqr[rcloner rVr(r;r5r0no_grad Parameterrrj)rcrOnum_position_embeds_diffold_position_embeddings_weights r4resize_position_embeddingsz*DistilBertModel.resize_position_embeddingss$?Ad#d  $q ( ( F `?Z```aaa.I +)-)L)S)Y)Y)[)[&.0l4;;^`d`k`o.p.p+ ; +  (k9t{TXTlTs       +a//]_]i6^^DO7>?YAY@Y?YZZBD67P8P7PQBBDO7>                 +..t{;;;;;s(AEEEc|jjSrsrqrYrMs r4get_input_embeddingsz$DistilBertModel.get_input_embeddingss ..r@new_embeddingsc||j_dSrsrZrcr\s r4set_input_embeddingsz$DistilBertModel.set_input_embeddingss*8'''r@heads_to_prunec|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrGrrr)rcr`rrs r4 _prune_headszDistilBertModel._prune_headssV +0022 G GLE5   "5 ) 3 ? ? F F F F G Gr@batch_size, num_choices checkpoint output_typer=Nreattention_maskr inputs_embedsrrr.c||n |jj}||n |jj}||n |jj}||t d|+||||}n.||dd}nt d||jn|j} |du} |||jj }| ||} |j r |d|vr|nd}n@|tj || }|jr!| r|st|| j|d}|| |||||S) NzDYou cannot specify both input_ids and inputs_embeds at the same timerRz5You have to specify either input_ids or inputs_embedsr)rjr$)tgt_len)rrrrrr)rKrruse_return_dictr%warn_if_padding_and_no_attention_maskrkrj get_head_masknum_hidden_layersrqrHr0onesrIrrirG) rcrerhrrirrr input_shaperjhead_mask_is_nonerqs r4rrzDistilBertModel.forwards 2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U%.%:!! @T%-&&y$+2OPP __Y >>  & 0>0JqTbObOb^^imNN%!&K!G!G!G~ "3  4 'tn4 d^4  elC&7 88 94 4 4 ji 4 4 4 4 4 r@rEz?DistilBert Model with a `masked language modeling` head on top.ceZdZdgZdeffd ZdejfdZde fdZ dej fdZ d ej fd Z eed eeee  ddeejdeejdeejdeejdeejdeedeedeedeeeejdfffdZxZS)DistilBertForMaskedLMzvocab_projector.weightrKct|t|j|_t ||_t j|j|j|_ t j |jd|_ t j|j|j |_ |t j|_dS)NrNrO)rTrUrrrEr/r rr(vocab_transformr\vocab_layer_normrWvocab_projectorrJr mlm_loss_fctrbs r4rUzDistilBertForMaskedLM.__init__/s    ():;;)&11!yVZ@@ " VZU C C C!yV5FGG /11r@rgc4|jSrLr/rNrMs r4rNz-DistilBertForMaskedLM.get_position_embeddings>66888r@rOc:|j|dSrQNr/rXrcrOs r4rXz0DistilBertForMaskedLM.resize_position_embeddingsD! 223NOOOOOr@c|jSrsr|rMs r4get_output_embeddingsz+DistilBertForMaskedLM.get_output_embeddingsRs ##r@r\c||_dSrsrr^s r4set_output_embeddingsz+DistilBertForMaskedLM.set_output_embeddingsUs-r@rdreNrerhrrilabelsrrr.c (||n |jj}||||||||} | d} || } || } || } || } d} |P|| d| d|d} |s| f| ddz} | | f| zn| St| | | j | j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. NrerhrrirrrrrRr$losslogitsr"r#) rKrlr/rzrr{r|r}rrkrr"r#)rcrerhrrirrrr dlbrt_outputr"prediction_logitsmlm_lossrs r4rrzDistilBertForMaskedLM.forwardXsP.&1%> 112CDD 001BCC  (():)?)?DUDZDZ[]D^D^)_)_agalalmoapapqqH N')L,</ 'tn/ d^/  ~uU\3%677 8/ / / ji / / / / / r@rxz DistilBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZdeffd ZdejfdZdefdZ e e de eee dd eejd eejd eejd eejdeejdeedeedeedeeeejdfffdZxZS)#DistilBertForSequenceClassificationrKct||j|_||_t ||_t j|j|j|_ t j|j|j|_ t j |j |_ |dSrs)rTrU num_labelsrKrEr/r rr(pre_classifier classifierr]seq_classif_dropoutr^rJrbs r4rUz,DistilBertForSequenceClassification.__init__s     + )&11 i FJ??)FJ0ABBz&"<==  r@rgc4|jSrLrrMs r4rNz;DistilBertForSequenceClassification.get_position_embeddingsrr@rOc:|j|dSrrrs r4rXz>DistilBertForSequenceClassification.resize_position_embeddingsrr@zbatch_size, sequence_lengthreNrerhrrirrrr.c d||n |jj}||||||||} | d} | dddf} || } t j| } || } || } d} |Z|jjf|j dkr d|j_nN|j dkr7|j tj ks|j tj kr d|j_n d|j_|jjdkrWt}|j dkr1|| |} n|| |} n|jjdkrGt!}|| d|j |d} n*|jjdkrt%}|| |} |s| f| ddz}| | f|zn|St'| | | j| j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr$ regressionsingle_label_classificationmulti_label_classificationrRr)rKrlr/rr ReLUr^r problem_typerrir0rmrr squeezer rr rr"r#)rcrerhrrirrrrdistilbert_outputr) pooled_outputrrloss_fctrs r4rrz+DistilBertForSequenceClassification.forwardsT.&1%C 'tnC d^C  'u|S/@)AA BC C C nm C C C C C r@rz DistilBert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZdeffd ZdejfdZdefdZ e e de eee dd eejd eejd eejd eejdeejdeejdeedeedeedeeeejdfffdZxZS)DistilBertForQuestionAnsweringrKcbt|t||_t j|j|j|_|jdkrtd|jt j |j |_ | dS)Nr9z)config.num_labels should be 2, but it is )rTrUrEr/r rr(r qa_outputsrr] qa_dropoutr^rJrbs r4rUz'DistilBertForQuestionAnswering.__init__ s    )&11)FJ0ABB   ! !\IZ\\]] ]z&"344  r@rgc4|jSrLrrMs r4rNz6DistilBertForQuestionAnswering.get_position_embeddingsrr@rOc:|j|dSrrrs r4rXz9DistilBertForQuestionAnswering.resize_position_embeddings rr@rdreNrerhrristart_positions end_positionsrrr.c | | n |jj} |||||||| } | d} || } || } | dd\} }| d} |d}d}||t| dkr|d}t| dkr|d}| d}| d|}| d|}tj |}|| |}|||}||zdz }| s| |f| ddz}||f|zn|St|| || j| j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrr$rRr) ignore_indexr9)r start_logits end_logitsr"r#)rKrlr/r^rsplitrrrrkclampr r rr"r#)rcrerhrrirrrrrrr"rrr total_loss ignored_indexr start_lossend_lossrs r4rrz&DistilBertForQuestionAnswering.forward.s%8&1%D 'tnD d^D  +U5<3D-EE FD D D ji D D D D D r@rz DistilBert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. ceZdZdeffd ZdejfdZdefdZ e e e e ee dd eejd eejd eejd eejd eejdeedeedeedeeeejdfffdZxZS) DistilBertForTokenClassificationrKc6t||j|_t||_t j|j|_t j|j |j|_ | dSrs) rTrUrrEr/r r]r^r hidden_sizerrJrbs r4rUz)DistilBertForTokenClassification.__init__sx     +)&11z&.11 )F$68IJJ r@rgc4|jSrLrrMs r4rNz8DistilBertForTokenClassification.get_position_embeddingsrr@rOc:|j|dSrrrs r4rXz;DistilBertForTokenClassification.resize_position_embeddingsrr@reNrerhrrirrrr.c ||n |jj}||||||||} | d} || } || } d} |Ft } | | d|j|d} |s| f| ddz}| | f|zn|St| | | j | j S)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. NrhrrirrrrrRr$r) rKrlr/r^rr rrrr"r#)rcrerhrrirrrroutputssequence_outputrrrrs r4rrz(DistilBertForTokenClassification.forwards*&1%. 'tn. d^.  $eEL#,=&>> ?. . . HG . . . . . r@rz DistilBert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. ceZdZdeffd ZdejfdZdefdZ e e de ee dd eejd eejd eejd eejdeejdeedeedeedeeeejdfffdZxZS)DistilBertForMultipleChoicerKc\t|t||_t j|j|j|_t j|jd|_t j |j |_ | dS)Nr$) rTrUrEr/r rr(rrr]rr^rJrbs r4rUz$DistilBertForMultipleChoice.__init__s    )&11 i FJ??)FJ22z&"<==  r@rgc4|jSrLrrMs r4rNz3DistilBertForMultipleChoice.get_position_embeddingsrr@rOc:|j|dS)a Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`. Arguments: new_num_position_embeddings (`int`) The number of new position embeddings. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. Nrrs r4rXz6DistilBertForMultipleChoice.resize_position_embeddingsrr@z(batch_size, num_choices, sequence_length)rgr=Nrerhrrirrrr.c |||n |jj}| |jdn |jd} |)|d|dnd}|)|d|dnd}|=|d|d|dnd}||||||||} | d} | dddf} || } tj| } | | } | | } | d| }d}|t}|||}|s|f| ddz}||f|zn|St||| j | jS)aW labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Returns: Examples: ```python >>> from transformers import AutoTokenizer, DistilBertForMultipleChoice >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased") >>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> choice0 = "It is eaten with a fork and a knife." >>> choice1 = "It is eaten while held in the hand." >>> labels = torch.tensor(0).unsqueeze(0) # choice0 is correct (according to Wikipedia ;)), batch size 1 >>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True) >>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels) # batch size is 1 >>> # the linear classifier still needs to be trained >>> loss = outputs.loss >>> logits = outputs.logits ```Nr$rRrrr)rKrlrrrkr/rr rr^rr rr"r#)rcrerhrrirrrr num_choicesrr)rrreshaped_logitsrrrs r4rrz#DistilBertForMultipleChoice.forwardsX&1%G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImq(   r=#5#5b#9#9=;M;Mb;Q;Q R R R  // )'/!5#"  qz $QQQT* ++M:: ! -00  ]33 // ++b+66  '))H8OV44D F%''!""+5F)-)9TGf$$v E("!/)     r@r)rtrurvrrUr rVrNrrXrrsrtr#rrvrr0rwrrrrrrrxrys@r4rrs /      99999 Pc P P P P+*#**+UVV+DSbccc-115,004-1,0/3&*T T EL)T !.T EL) T   - T )* T $D>T 'tnT d^T  (% c0A*BB CT T T dcT T T T T r@r)Nrrtypingrrrrrrnumpyr:r0r torch.nnr r r activationsrconfiguration_utilsrintegrations.deepspeedrmodeling_attn_mask_utilsrmodeling_outputsrrrrrrmodeling_utilsr pytorch_utilsrrrrutilsrrrr r!r"r#configuration_distilbertr%modeling_flash_attention_utilsr& get_loggerrtrrurvrrwr5r3rCrJr{rrrrrrr.DISTILBERT_START_DOCSTRINGrsrErxrrrrrr@r4rsR   :::::::::::::::: AAAAAAAAAA))))))333333@@@@@@KKKKKK.----- 766666KJJJJJJ  H % %/$ EE#EELEEEE3U\++++++++\eeeeeRYeeeP^"^"^"^"^" 6^"^"^"BMMMMM4MMM`"),$2 #  77777ry777tJ J J J J ")J J J \D "JlD D D D D /D D D NIa a a a a 5a a a H k k k k k *Ck k k \ l l l l l %>l l l ^ T T T T T '@T T T n x x x x x ";x x x x x r@