g3~dZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZmZmZdd lmZmZmZmZd dlm Z ej!e"Z#dZ$dZ%Gdde j&Z'Gdde j&Z(Gdde j&Z)de(iZ*Gdde j&Z+Gdde j&Z,Gdde j&Z-Gdde j&Z.Gd d!e j&Z/Gd"d#eZ0d$Z1d%Z2ed&e1Gd'd(e0Z3Gd)d*e j&Z4Gd+d,e j&Z5ed-e1Gd.d/e0Z6eGd0d1eZ7ed2e1Gd3d4e0Z8dS)5zPyTorch Splinter model.N) dataclass)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN))BaseModelOutputWithPastAndCrossAttentions ModelOutputQuestionAnsweringModelOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging)SplinterConfigztau/splinter-baserc eZdZdZfdZ d deejdeejdeejdeejd ee d e f d Z xZ S) SplinterEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cdttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|_|dt%j|jddt+|dd|_dS) N) padding_idxeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandgetattrr"selfconfig __class__s j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/splinter/modeling_splinter.pyr%zSplinterEmbeddings.__init__+s !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2H&J\%]%]"f&8f>STTTz&"<==   EL)GHHOOPWXXej    (/v7PR\']']$$$Nr input_idstoken_type_idsr inputs_embedspast_key_values_lengthreturnc||}n|dd}|d}||jdd|||zf}|+tj|tj|jj}|||}||}||z} |jdkr| |} | | z } | | } | | } | S)Nr rdtypedevicer#) sizerr5zeroslongrGr*r.r"r,r/r3) r:r?r@rrArB input_shape seq_lengthr. embeddingsr,s r=forwardzSplinterEmbeddings.forward<s  #..**KK',,..ss3K ^  ,QQQ0FVlIl0l-lmL  !"[EJtO`OghhhN   00;;M $ : :> J J"%::  ': 5 5"&":":<"H"H  - -J^^J// \\*-- r>)NNNNr) __name__ __module__ __qualname____doc__r%rr5 LongTensor FloatTensorintrrN __classcell__r<s@r=rr(sQQ^^^^^&1559375901 E,-!!12u/0    12  !)  r>rc,eZdZdfd ZdejdejfdZ ddejdeejd eejd eejd eejd ee e ejd ee de ejfdZ xZ S)SplinterSelfAttentionNcDt|j|jzdkr0t |ds t d|jd|jd|j|_t |j|jz |_|j|jz|_tj |j|j|_ tj |j|j|_ tj |j|j|_ tj|j|_|pt#|dd|_|jdks |jd kr6|j|_tjd |jzd z |j|_|j|_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r"r# relative_keyrelative_key_queryr)r$r%r(num_attention_headshasattr ValueErrorrUattention_head_size all_head_sizerLinearquerykeyvaluer1attention_probs_dropout_probr3r8r"r+r&distance_embedding is_decoderr:r;r"r<s r=r%zSplinterSelfAttention.__init__`s    : :a ? ?PVXhHiHi ?8F$688 4888  $*#= #&v'9F( ' -zC C $  '> 9 9T=Y]q=q=q+1+ID (&(l1v7U3UXY3Y[_[s&t&tD # +r>xrCc|dd|j|jfz}||}|ddddS)Nr rr_rr )rHr`rcviewpermute)r:rm new_x_shapes r=transpose_for_scoresz*SplinterSelfAttention.transpose_for_scoreszsPffhhssmt'?AY&ZZ FF;  yyAq!$$$r>F hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsc^||}|du} | r||d} |d} |}n4| rS|||} |||} |}n||||} |||} t j|d| gd} t j|d| gd} nP|||} |||} ||} |du} |jr| | f}t j| | dd}|j dks |j dkrt| j d| j d}}| r>t j |dz tj |j dd}n:t j|tj |j dd}t j|tj |j dd}||z }|||jzdz }|| j }|j dkrt jd | |}||z}n?|j dkr4t jd | |}t jd | |}||z|z}|t+j|jz }|||z}t0j|d}||}|||z}t j|| }|dddd }|dd|jfz}||}|r||fn|f}|jr||fz}|S)Nrrr_dimr r]r^rE)rFzbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rfrrrgrhr5catrkmatmul transposer"shapetensorrJrGror6rjr+torFeinsummathsqrtrcr functionalsoftmaxr3rp contiguousrHrd)r:rsrtrurvrwrxrymixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss r=rNzSplinterSelfAttention.forwardsZ!JJ}55 3$>  O."<&q)I(+K3NN  O11$((;P2Q2QRRI33DJJ?T4U4UVVK3NN  '11$((=2I2IJJI33DJJ}4M4MNNK >!#4i"@aHHHI)^A%6 $D!LLLKK11$((=2I2IJJI33DJJ}4M4MNNK//0ABB "$. ? 6(5N!< Y5H5HR5P5PQQ  '> 9 9T=Y]q=q=q'2'8';Y_Q=O*L w!&j1nEJWdWk!l!l!l!q!q"""'l%*UbUi!j!j!j!o!oprtu!v!v"\*EJ}OcdddiijkmoppN%6H#'#:#:8dFb;bef;f#g#g #7#:#:AR#:#S#S +~==+0<8H+Wk+l+l(#36N#N  -1EEE16>NP[]q1r1r./4|S"S%**+BCC 6G]=/22mM] ? 2 11Gr>NNNNNNF) rOrPrQr%r5TensorrrrrTrboolrNrVrWs@r=rYrY_s.,,,,,,4%el%u|%%%%7;15=A>BDH,1cc|c!!23cE-. c ((9: c !)): ; c!uU->'?!@Ac$D>c u| ccccccccr>rYcPeZdZfdZdejdejdejfdZxZS)SplinterSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNr) r$r%rrer(denser/r0r1r2r3r9s r=r%zSplinterSelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== r>rs input_tensorrCc||}||}|||z}|Srrr3r/r:rsrs r=rNzSplinterSelfOutput.forward@ =11  ]33 }|'CDD r>rOrPrQr%r5rrNrVrWs@r=rri>>>>> U\RWR^r>reagerceZdZdfd ZdZ ddejdeejdeejdeejd eejd ee e ejd ee d e ejfd Z xZ S)SplinterAttentionNctt|j|||_t ||_t|_dS)Nr") r$r%SPLINTER_SELF_ATTENTION_CLASSES_attn_implementationr:routputset pruned_headsrls r=r%zSplinterAttention.__init__s` 3F4OP ,C    )00 EEr>ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrrr{)lenrr:r`rcrrrfrgrhrrrdunion)r:headsindexs r= prune_headszSplinterAttention.prune_headss u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::r>FrsrtrurvrwrxryrCc ||||||||}||d|} | f|ddz} | S)Nrr)r:r) r:rsrtrurvrwrxry self_outputsattention_outputrs r=rNzSplinterAttention.forwardsayy    ! "     ;;|A FF#% QRR(88r>rr) rOrPrQr%rr5rrrTrrrNrVrWs@r=rrs"""""";;;*7;15=A>BDH,1|!!23E-.  ((9:  !)): ; !uU->'?!@A$D> u| r>rcBeZdZfdZdejdejfdZxZS)SplinterIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r$r%rrer(intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr9s r=r%zSplinterIntermediate.__init__/sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r>rsrCcZ||}||}|Sr)rr)r:rss r=rNzSplinterIntermediate.forward7s, =11 00?? r>rrWs@r=rr.s^99999U\elr>rcPeZdZfdZdejdejdejfdZxZS)SplinterOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r$r%rrerr(rr/r0r1r2r3r9s r=r%zSplinterOutput.__init__?sf Yv79KLL f&8f>STTTz&"<== r>rsrrCc||}||}|||z}|Srrrs r=rNzSplinterOutput.forwardErr>rrWs@r=rr>rr>rc eZdZfdZ ddejdeejdeejdeejdeejd eeeejd ee d eejfd Z d Z xZ S) SplinterLayerct|j|_d|_t ||_|j|_|j|_|jr/|jst|dt |d|_ t||_ t||_ dS)Nrz> should be used as a decoder model if cross attention is addedr#r)r$r%chunk_size_feed_forward seq_len_dimr attentionrkadd_cross_attentionrbcrossattentionr intermediaterrr9s r=r%zSplinterLayer.__init__Ns '-'E$*622 +#)#=  # `? j D!h!h!hiii"3FT^"_"_"_D 088$V,, r>NFrsrtrurvrwrxryrCc | |ddnd}||||||} | d} |jr| dd} | d} n | dd} d} |jrp|nt|dstd|d| |d dnd}|| ||||||}|d} | |ddz} |d} | | z} t |j|j|j| }|f| z} |jr| | fz} | S) Nr_)ryrxrrr rz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r}) rrkrarbrrfeed_forward_chunkrr)r:rsrtrurvrwrxryself_attn_past_key_valueself_attention_outputsrrpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputs r=rNzSplinterLayer.forward\s:H9S>"1"#5#5Y] !%   /3 "0" " 2!4 ? 1,QrT2G 6r :  ,QRR0G'+$ ? Q4@4!122  DdDDD @N?Yrss(;(;_c %&*&9&9 %&)!'' # 7q9  7" ==G,C2+F ( 14P P 0  #T%A4CSUe   /G+ ? 5!2 44Gr>c\||}|||}|Sr)rr)r:rintermediate_outputrs r=rz SplinterLayer.feed_forward_chunks2"//0@AA{{#68HII r>r) rOrPrQr%r5rrrTrrrNrrVrWs@r=rrMs - - - - -"7;15=A>BDH,1??|?!!23?E-. ? ((9: ? !)): ; ?!uU->'?!@A?$D>? u| ????Br>rcLeZdZfdZ ddejdeejdeejdeejd eejd eeeejd ee d ee d ee dee de eeje ffdZ xZ S)SplinterEncoderct|_tjfdt jD|_d|_dS)Nc.g|]}tS)r).0_r;s r= z,SplinterEncoder.__init__..s!#c#c#caM&$9$9#c#c#cr>F) r$r%r;r ModuleListrangenum_hidden_layerslayergradient_checkpointingr9s `r=r%zSplinterEncoder.__init__s`  ]#c#c#c#c5IaCbCb#c#c#cdd &+###r>NFTrsrtrurvrwpast_key_valuesrryoutput_hidden_states return_dictrCc | rdnd} |rdnd} |r|jjrdnd} |jr%|jr|rtdd}|rdnd}t |jD]\}}| r| |fz} |||nd}|||nd}|jr)|jr"||j |||||||}n||||||||}|d}|r ||dfz }|r$| |dfz} |jjr | |dfz} | r| |fz} | std||| | | fDSt||| | | S) NrzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frr rr_c3K|]}||V dSrr)rvs r= z*SplinterEncoder.forward..s4  =!===  r>last_hidden_staterrs attentionscross_attentions) r;rrtraininglogger warning_once enumerater_gradient_checkpointing_func__call__tupler )r:rsrtrurvrwrrryrrall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cachei layer_modulelayer_head_maskrx layer_outputss r=rNzSplinterEncoder.forwardsB#7@BBD$5?bb4%6d4;;Zdrr`d  & "4= " "##p" #,6RR$(44# V# VOA|# I$58H$H!.7.CillO3B3N_Q//TXN* t}  $ A A )!"#)*"% ! ! !- !"#)*"%!! *!,M ;"}R'8&::"  V&9]1=M) NNNNNNFFT)rOrPrQr%r5rrrTrrrr rNrVrWs@r=rrsD,,,,,7;15=A>BEI$(,1/4&*S S |S !!23S E-. S ((9: S !)): ; S "%e.?(@"ABS D>S $D>S 'tnS d^S  uU\"$MM NS S S S S S S S r>rc$eZdZdZeZdZdZdZdS)SplinterPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. splinterTct|tjrT|jjd|jj|j |jj dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|jj |jj ddSdS)zInitialize the weightsg)meanstdNg?)rrreweightdatanormal_r;initializer_rangebiaszero_r&rr/fill_)r:modules r= _init_weightsz%SplinterPreTrainedModel._init_weights s) fbi ( ( * M  & &CT[5R & S S S{&  &&((((('&  - - * M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r>N) rOrPrQrRr config_classbase_model_prefixsupports_gradient_checkpointingrrr>r=r r s@ "L"&*#*****r>r aK This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SplinterConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a/ Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `{0}`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `{0}`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `{0}`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare Splinter Model transformer outputting raw hidden-states without any specific head on top.c!eZdZdZfdZdZdZdZee de e e e dd eejd eejd eejd eejd eejdeejdeejdeejdeeejdeedeedeedeedeee ffdZxZS) SplinterModela* The model is an encoder (with only self-attention) following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. ct|||_t||_t ||_|dSr)r$r%r;rrMrencoder post_initr9s r=r%zSplinterModel.__init__esX     ,V44&v..  r>c|jjSrrMr*)r:s r=get_input_embeddingsz"SplinterModel.get_input_embeddingsos ..r>c||j_dSrr!)r:rhs r=set_input_embeddingsz"SplinterModel.set_input_embeddingsrs*/'''r>c|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r:heads_to_prunerrs r= _prune_headszSplinterModel._prune_headsusU +0022 C CLE5 L u % / ; ;E B B B B C Cr>batch_size, sequence_length checkpoint output_typerNr?rtr@rrurArvrwrrryrrrCc r| | n |jj} | | n |jj} | | n |jj} |jjr| | n |jj} nd} ||t d|+||||}n.||dd}nt d|\}}||j n|j }| | ddj dnd}|tj |||zf|}|!tj |tj| }|||}|jjrL|J|\}}}||f}|tj ||}||}nd}|||jj}|||||| }||||||| | | | | }|d}| s|f|d dzSt+||j|j|j|j S)a  encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timer z5You have to specify either input_ids or inputs_embedsrr_)rGrE)r?rr@rArB) rtrurvrwrrryrrrr)r;ryruse_return_dictrkrrb%warn_if_padding_and_no_attention_maskrHrGrr5onesrIrJget_extended_attention_maskinvert_attention_mask get_head_maskrrMrr rrsrr)r:r?rtr@rrurArvrwrrryrrrK batch_sizerLrGrBextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputssequence_outputs r=rNzSplinterModel.forward}sR2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U!, J%.%:!! @TDSC^!3A!6!) NNNNNNNNNNNNN)rOrPrQrRr%r"r$r(rSPLINTER_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrr5rrrTrrrrNrVrWs@r=rrZs  ///000CCC+*+D+K+KLi+j+jkk&=$-11515/3,0048<9==A$(,0/3&*w w EL)w !.w !. w u|, w EL) w   -w  ( 5w !) 6w "$u'8"9:w D>w $D>w 'tnw d^w  u?? @w w w lk w w w w w r>rcDeZdZdfd ZdejdejfdZxZS)SplinterFullyConnectedLayergeluc t||_||_t j|j|j|_t||_t j |j|_ dSr) r$r% input_dim output_dimrrerr act_fnr/)r:rErFrr<s r=r%z$SplinterFullyConnectedLayer.__init__sa "$Yt~t?? Z( do66r>inputsrCc||}||}||}|Sr)rrGr/)r:rHrss r=rNz#SplinterFullyConnectedLayer.forwards; 6**  M22 }55 r>)rCrrWs@r=rBrBsc777777elu|r>rBc(eZdZdZfdZdZxZS)QuestionAwareSpanSelectionHeadzf Implementation of Question-Aware Span Selection (QASS) head, described in Splinter's paper: ctt|j|j|_t|j|j|_t|j|j|_t|j|j|_tj |j|jd|_ tj |j|jd|_ dS)NF)r) r$r%rBr(query_start_transformquery_end_transformstart_transform end_transformrrestart_classifierend_classifierr9s r=r%z'QuestionAwareSpanSelectionHead.__init__s %@ASU[Ug%h%h"#>v?QSYSe#f#f :6;MvOabb89KVM_`` " &*PW\ ] ] ] i(:Fch|\}}}|ddd|}tj|d|}||}||}||} ||} | |} | ddd} tj | | } | |} | ddd} tj | | } | | fS)Nr r)r|rrr_) rH unsqueezerepeatr5gatherrMrNrOrPrQrprrR)r:rH positionsrr|r gathered_repsquery_start_repsquery_end_reps start_repsend_repsrs start_logits end_logitss r=rNz&QuestionAwareSpanSelectionHead.forward sKKMM 1c##B''..q!S99 V%@@@ 55mDD11-@@))&11 %%f----.>?? ''1a00 |M:>> ++N;; ##Aq!,,\-:: Z''r>)rOrPrQrRr%rNrVrWs@r=rKrKsV \ \ \ \ \(((((((r>rKz Splinter Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZfdZeedeee e  dde e j de e j de e j de e j d e e j d e e j d e e jd e e jd e ede ede ede e jdeee ffdZxZS)SplinterForQuestionAnsweringct|t||_t ||_|j|_|dSrr$r%rr rK splinter_qassquestion_token_idrr9s r=r%z%SplinterForQuestionAnswering.__init__=]    %f-- ;FCC!'!9 r>r)r*Nr?rtr@rrurAstart_positions end_positionsryrrquestion_positionsrCc | | n |jj} d} | |Atjtj||jd}n?tj|dtj |j |j }| d} d} | ||||||| | |  }|d}||| \}}| r*|d |d }}|N|d |z tj|jjzz}|d |z tj|jjzz}d}||t'|d kr|d}t'|d kr|d}|d }|d||d|t+| }|||}|||}||zd z }| s||f|d dz}||f|zn|St-||||j|j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. NFr r{r)rFlayoutrGTrtr@rrurAryrrr ignore_indexr_lossr]r^rsr)r;r.r5argmaxeqrdrUrIrHrJrjrGrTr rcsqueezefinforFminrclamp_r rrsr)r:r?rtr@rrurArfrgryrrrhquestion_positions_were_none"question_position_for_each_examplerr<r]r^ total_loss ignored_indexloss_fct start_lossend_lossrs r=rNz$SplinterForQuestionAnswering.forwardGsH&1%'> @R@RST@U@U*L  %'1~+=\M_A`A`Ad*ddL#q>'9U[IY=Z=Z=^&^^J  &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M  " "1m 4 4 4  M 2 2 2']CCCH!,@@Jx M::H$x/14J R"J/'!""+=F/9/EZMF**6 Q+%!!/)     r> NNNNNNNNNNNN)rOrPrQr%rr=r>rr?rr@rr5rrSrrrrNrVrWs@r=r`r`5s+*+D+K+KLi+j+jkk&0$-11515/3,0046:48,0/3&*9=^ ^ EL)^ !.^ !. ^ u|, ^ EL) ^   -^ "%"23^   01^ $D>^ 'tn^ d^^ %U%56^  u22 3^ ^ ^ lk ^ ^ ^ ^ ^ r>r`ceZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ee ejed<dZ ee ejed<dS)SplinterForPreTrainingOutputa Class for outputs of Splinter as a span selection model. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when start and end positions are provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-start scores (before SoftMax). end_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-end scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nror]r^rsr)rOrPrQrRrorr5rT__annotations__r]r^rsrrrr>r=rrs.)-D(5$ %,,,&*L%#***$(J!(((8rz Splinter Model for the recurring span selection task as done during the pretraining. The difference to the QA task is that we do not have a question, but multiple question tokens that replace the occurrences of recurring spans instead. ceZdZfdZeed ddeej deej deej deej deej d eej d eej d eej d ee d ee dee deej de e effdZdej dej fdZxZS)SplinterForPreTrainingct|t||_t ||_|j|_|dSrrbr9s r=r%zSplinterForPreTraining.__init__rer>z*batch_size, num_questions, sequence_lengthNr?rtr@rrurArfrgryrrrhrCc | | n |jj} | ||td| |td| ||} |||||||| | |  } | d}|\}}}||| \}}| d}|x|d|||}|d|z tj |j j zz}|d|z tj |j j zz}d}||| dtd|dz | dtd|dz t|jj}||||z||||z}||||z||||z}||zdz }| s||f| ddz}||f|zn|St%|||| j| j S) a start_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. NzCquestion_positions must be specified in order to calculate the lossz>question_positions must be specified when input_embeds is usedrkrrrlr_rn)r;r. TypeError_prepare_question_positionsr rHrcrTr7r5rsrFrtrumaxr r)rorrsr)r:r?rtr@rrurArfrgryrrrhrr<r4sequence_lengthr|r]r^ num_questions attention_mask_for_each_questionrxrzr{r|rs r=rNzSplinterForPreTraining.forwardsB&1%$?$? @ @ @ (T[5MNNNH!!!*}">H%x/14J R"J/'!""+=F/9/EZMF**6 Q+%!!/)     r>crtj||jjk\}}tj|}tj|d|f|jjtj |j }tj d|D}||||f<|S)NrrEc6g|]}tj|Sr)r5r6)rns r=rzFSplinterForPreTraining._prepare_question_positions..Qs AAAa%,q//AAAr>) r5wherer;rdbincountfullrHrr)rJrGr~)r:r?rowsflat_positionsrrWcolss r=rz2SplinterForPreTraining._prepare_question_positionsHs${9 8U+UVVnt,, J ^^A   1 1 3 3 4 K $*#    yAA=AAABB . $*r>r})rOrPrQr%rr=r>rr5rrSrrrrrNrrVrWs@r=rrs+*!(()UVV -11515/3,0046:48,0/3&*9=b b EL)b !.b !. b u|, b EL) b   -b "%"23b   01b $D>b 'tnb d^b %U%56b  u22 3b b b b H U\ el        r>r)9rRr dataclassesrtypingrrrrr5torch.utils.checkpointrtorch.nnr activationsr modeling_outputsr r rmodeling_utilsr pytorch_utilsrrrutilsrrrrconfiguration_splinterr get_loggerrOrr?r@ModulerrYrrrrrrrr SPLINTER_START_DOCSTRINGr=rrBrKr`rrrr>r=rs !!!!!!//////////// %%%%%%!!!!!!tttttttttt------lllllllllluuuuuuuuuuuu222222  H % %)"33333333nCCCCCBICCCN         "# 00000 000h     29        RY   SSSSSBISSSnZ Z Z Z Z biZ Z Z z*****o***8 /dh\ \ \ \ \ +\ \ \ ~")$#(#(#(#(#(RY#(#(#(L o o o o o #:o o o d :::::;:: :>  }}}}}4}}}}}r>