g!dZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZmZmZddlmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZdd lm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(e%j)e*Z+dZ,dZ-de j.de/de/fdZ0dGde j.de j1dee/fdZ2Gdde j3Z4Gdde j5Z6Gdde j5Z7Gd d!e j5Z8Gd"d#e j5Z9Gd$d%e j5Z:Gd&d'e j5Z;Gd(d)eZeGd.d/e Z?eGd0d1e Z@eGd2d3e ZAd4ZBd5ZCd6ZDGd7d8e<ZEGd9d:e<ZFe#d;eBGd<d=e<ZGe#d>eBGd?d@eHs99;;LC ,gg'GD$)*11#q'7KKNNuUUM-'M+77 8J8J8L8LekZ_N`N`Ndee6 E ""r,cLeZdZdZdedeffd Zd dejdeffd ZxZ S) LEDLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcLt||dSN)super__init__)selfrArB __class__s r*rFz&LEDLearnedPositionalEmbedding.__init___s# 77777r,rinput_ids_shapepast_key_values_lengthc|dd\}}tj|||ztj|jj}t |S)z3`input_ids_shape` is expected to be [bsz x seqlen].N)r.device)r6arangelongweightrMrEforward)rGrIrJr9seq_len positionsrHs r*rQz%LEDLearnedPositionalEmbedding.forwardbs[&rr* WL "$:W$DEJ_c_j_q   wwy)))r,)r) __name__ __module__ __qualname____doc__intrFr6SizerQ __classcell__rHs@r*r@r@Zs8s83888888**uz*3**********r,r@ceZdZfdZ ddZedZedZeddefdZ ed e j fd Z d e j d e j d e fdZde j de j d e fdZedZdZdZdZxZS)LEDEncoderSelfAttentionct|j|jzdkr t d|jd|jd|j|_t |j|jz |_|j|_tj |j|j|_ tj |j|j|_ tj |j|j|_ tj |j|j|_tj |j|j|_tj |j|j|_|j|_||_|j|j}|dzdksJd|jd||dksJd|jd||dz|_||_dS) NrzThe hidden size (z6) is not a multiple of the number of attention heads ()rLz`attention_window` for layer z has to be an even value. Given z has to be positive. Given )rErF hidden_sizenum_attention_headsr' num_headsrXhead_dim embed_dimrLinearquerykeyvalue query_global key_global value_globalattention_probs_dropout_probdropoutlayer_idattention_windowone_sided_attn_window_sizeconfig)rGrqrnrorHs r*rFz LEDEncoderSelfAttention.__init__ms    : :a ? ?8F$688 4888  3F.1KKLL +Yv14>BB 9V/@@Yv14>BB If&8$.II)F$6GGIf&8$.II:   !24=A q A % % % l4= l lZj l l & % % q g4= g gUe g g ! +;a*?' r,NFc  |dd}||}||} ||} |\} } } | |jksJd|jd| |t j|jz}| | | |j |jdd}| | | |j |jdd} | || |j }|dkddddddf}| ||tj|jj}| ||||j }||z }t)|| | |j |j dzdzgks;Jd| d | d |j d |j dzdzd | |rN||\}}}}||| |||| }tj||fd }~t0j|d tj}|a||j fks(Jd|j fd|| ddd d|z}tj||ddddddfd}| |}~t0j||j|j}| | | |j |jdd} |r|| ||||}n||| |j }|| | |j |jfks Jd|dd | | | !}|rr|"|||||||\}}||ddd|df}| tG|dd ||ddd <d||<|ddf}|r||fz }|r|r||fzn|S)a [`LEDEncoderSelfAttention`] expects *len(hidden_states)* to be multiple of *attention_window*. Padding to *attention_window* happens in [`LEDEncoderModel.forward`] to avoid redoing the padding on each layer. The *attention_mask* is changed in [`LEDEncoderModel.forward`] from 0, 1, 2 to: - -10000: no attention - 0: local attention - +10000: global attention rrz&hidden_states should have embed_dim = z , but has N)r1rLz$local_attn_probs should be of size (z, z), but is of size ) query_vectors key_vectorsmax_num_global_attn_indicesis_index_global_attn_nonzero"is_local_index_global_attn_nonzero%is_local_index_no_global_attn_nonzeror#dimrzr./Head mask for a single layer should be of size , but is ptraining) value_vectors attn_probsrurvrwzUnexpected size) hidden_statesrulayer_head_maskrwrvrxis_index_masked)$ transposerfrgrhr1rdmathsqrtrcviewrb _sliding_chunks_query_key_matmulrptype_asr4r6r7r.r8new_oneslist_get_global_attn_indices"_concat_with_global_key_attn_probscatr functionalsoftmaxfloat32rmr(_compute_attn_output_with_global_indices'_sliding_chunks_matmul_attn_probs_valuereshape contiguous'_compute_global_attn_output_from_hiddenlen)rGrattention_maskrris_index_global_attnis_global_attnoutput_attentionsrsrtrrR batch_sizerd attn_scores#remove_from_windowed_attention_mask float_mask diagonal_maskrurvrwrxglobal_key_attn_scoresr attn_outputglobal_attn_outputglobal_attn_probsnonzero_global_attn_outputoutputss r*rQzLEDEncoderSelfAttention.forwards(&//155  =11 hh}--  =11 )6););)=)=&Y  ' ' ' YDN Y Yi Y Y ( ' ' 4=111 %**7JPTP]^^hhijlmnn !&&w DNDMZZddefhijj ;; ;(G  0>/BAAAqqq$PTDT.U+9@@OO[[ /]=P1Q1Q1U  ==   Z__%6%6  7 7TEd  }$ K$$&&''   N  +a /! 3 ,     `: ` ` ` `DN ` `/!3a7 ` `KVK[K[K]K] ` `      '--.BCC  +,25&*%L%L+',G-I3U6[ &M&& " )%;[$IrRRRK']** Ru}+    &"'')).u$.ARuu]l]q]q]s]suu)--aB::ZGJ&z?111aaatCS3TVYZZ '' 44  ]**:PTP]*^^ %**7JPTP]^^hhijlmnn   GG+%,G-I3U HKKFFM4+JK!!j'4>4=%YYYY[lYYY!++Aq1199':yYYddff   9484`4`+,G /3U-I6[ /5a55 1  1*<215qqq:\]^:__* & ?Y>]>]6q9::B??K4TTrT: ; 89J3 4((A..0  %  } $G2@dEVdw+---]ddr,ctj||}|jg|dd|d|dR}|S)z)pads rows and then flips rows and columnsNr#)rrpadrr1)hidden_states_paddedpaddings r* _pad_and_transpose_last_two_dimsz8LEDEncoderSelfAttention._pad_and_transpose_last_two_dims8s "}00 '   938 ! & & ( (" - /C/H/H/L/L NbNgNghjNkNk   $#r,c2|\}}}}tj|d|dzf}|||d}|ddddd| f}||||||z}|ddddddddf}|S)aY shift every row 1 step right, converting columns into diagonals. Example: ```python chunked_hidden_states: [ 0.4983, 2.6918, -0.0071, 1.0492, -1.8348, 0.7672, 0.2986, 0.0285, -0.7584, 0.4206, -0.0405, 0.1599, 2.0514, -1.1600, 0.5372, 0.2629, ] window_overlap = num_rows = 4 ``` (pad & diagonalize) => [ 0.4983, 2.6918, -0.0071, 1.0492, 0.0000, 0.0000, 0.0000 0.0000, -1.8348, 0.7672, 0.2986, 0.0285, 0.0000, 0.0000 0.0000, 0.0000, -0.7584, 0.4206, -0.0405, 0.1599, 0.0000 0.0000, 0.0000, 0.0000, 2.0514, -1.1600, 0.5372, 0.2629 ] rrr#N)r1rrrr)chunked_hidden_statestotal_num_heads num_chunkswindow_overlap hidden_dims r*_pad_and_diagonalizez,LEDEncoderSelfAttention._pad_and_diagonalizeCsBCXB\B\B^B^?^Z " 1 1 !A~'9#:! ! !6 : : Z! ! !6 AAqqq"N?" "! !6 : : Z*9T! ! !6aaaAAAssl C$$r, onnx_exportc@|s||dtj|d|dzd|dz|d}t |}|ddzdz |d<t |}|ddz|d<|||S|dtj|d|ddz |dz|dg}tj||j}t|dD],}|dd||z||zd|zzddf|dd|ddddf<-|S) zBconvert into overlapping chunks. Chunk size = 2w, overlap size = wrrrLtrunc rounding_moder1striderMN) rr1r6divrr as_stridedemptyrMrange)rrr chunk_size chunk_strideoverlapping_chunkschunks r*_chunkzLEDEncoderSelfAttention._chunkts R)..""1%% -,,Q//.12DU\]]]"""1%% Mm002233J&qMA-1JqM 4 4 6 677L*1o2LO ++L+QQ Q   q ! ! Im((++^7 S S SVW W Q    q ! !  #[M5>)EN,BQEW,WWYZYZYZZ2 qqq%AAA~ . ."!r,returnc>|||dzdg}|ddddddf}|d}|ddd|ddd|dzf}||}t j|td | ||ddd|ddd|dzf<|dd| ddd|dz df}||}t j|td | ||dd| ddd|dz df<dS)Nrr)dims)rr inf) rtrilflipr2r1r6 full_likefloatwherer5) input_tensoraffected_seq_lenbeginning_mask_2dbeginning_mask ending_maskbeginning_input ending_inputs r*_mask_invalid_locationsz/LEDEncoderSelfAttention._mask_invalid_locationss(112BDTWXDXYY^^``eelmkneoo*4D!!!+;<$))v)66 &qqq*;+;*;QQQ@VBRUVBV@V'VW'../C/C/E/EFFHM eEll]I I %##%% 7 7 QQQ)))111.D0@10D.DDE$AAA(8'8'9'9111@PST@T>U>W>W$WX !(():):)<)<== LQO 5<<-M M %   ""L 1 1 QQQ))**AAA1AA1E/F/H/HHIIIr,rfrgrc |\}}}}||dzzdksJd|dzd|||ksJtj||ddz }|dd||z||}|dd||z||}|||t |jdd }|||t |jdd }tjd ||f} | | d } | ||z|dz||dzdzf} | d d d d d |d |dzf| d d d dd d |d f<| d d d|d d |dzf| d d dd d |d f<| d d d d |dz d|dzd f| d d dd d d d |f<| d d dd |dz d|z d f| d d dd|d|f<| |||d|zdzdd} | | || S)a Matrix multiplication of query and key tensors using with a sliding window attention pattern. This implementation splits the input into overlapping chunks of size 2w (e.g. 512 for pretrained LEDEncoder) with an overlap of size window_overlap rLrz&Sequence length should be multiple of z. Given rrrrFzbcxd,bcyd->bcxy)rrrr)rNr#) r1r6rrrrgetattrrqeinsumrr$rr) rGrfrgrrrRrbrc chunks_count!diagonal_chunked_attention_scoresdiagonal_attention_scoress r*rz8LEDEncoderSelfAttention._sliding_chunks_query_key_matmulsH 49::<<0 GY ~) *a / / / YNQ4F Y YPW Y Y 0 / /zz||sxxzz))))y.PPPSTT 1%%--j9.DgxXXmmAq!!))*y*@'8TT E>74; W\3]3]^^kk#~wt{MSX/Y/YZZ -2L9JUTWL,Y,Y)-1,Q,Q -|-R- - )%F$O$O ) #\A%5~~XYGY\]G] ^% % ! Ab AAqqq/>/#7^a%7#7 7A !!!!SbS!!!^__"<=@a AAr>??$8nq&8$8 8@ !!!!RNOO";<@a AAqqqNQ&'",nq.@.B.B B@ !!!!QRRO^O";<Op AAq&NQ&&N(:(<(< <O !!!!Q.(8!N:J"JK %>$B$B  7A,>,B% % )Aq// " $$%>OOO((r,rrhc|\}}}}||dzzdksJ|dd|ddksJ|dd|zdzksJtj||ddz }|dd||ztj||d|d|zdz} |dd||z||}t j|dd||fd } ||z|dzd|z|f} | } | d|| dz| d| df} | | | } | | } tj d | | f}| ||||ddS) z Same as _sliding_chunks_query_key_matmul but for attn_probs and value tensors. Returned tensor will be of the same shape as `attn_probs` rLrNr rrrr#rhrzbcwd,bcdh->bcwh) r1r6rrrrrrrrrrr)rGrrhrrrRrbrcrchunked_attn_probs padded_valuechunked_value_sizechunked_value_stride chunked_valuecontexts r*rz?LEDEncoderSelfAttention._sliding_chunks_matmul_attn_probs_values)49::<<0 GY.1,-2222  !$ RaR(88888q!!Q%7!%;;;;;y.PPPSTT (11!Q77??  " Ig~W E E E    "   1%%--j9.DgxXX}((A~~0V^`(aa )94lQ6FNHZ\de+2244  # 1!4 4  #  #   %//5GPd/ee !667IJJ,03E}2UVV||J 7HEEOOPQSTUUUr,cx|d}|}|d}t j||j|dk}|d}|dkd}||||fS)z&A&A&C&C#(<'C'CT'C'R'R$&+\ '0D0K& & & # - -" - 5 5&6" .H-O-OY]-O-^-^*2Lq1P0Y0Ycg0Y0h0h- ' ( . 1   r,cj|jd}||||j|j}||||<t jd||f} | dd} t j| jj | |d|dddddf<| dd} | S)Nrzblhd,bshd->blhsrr ) r%r$rbrcr6rrr7r.r8) rGrtrsrurvrwrxrkey_vectors_only_globalattn_probs_from_global_keys r*rz:LEDEncoderSelfAttention._concat_with_global_key_attn_probs8s!&q) #."7"7 3T^T]# # GRRnFo BC&+\2CmUlEm%n%n"&@%I%I!Q%O%O" K28 9 9 = # 1! 46[\]6^`a`a`acdcdcd d &@%I%I!Q%O%O"))r,cN|jd}|dd|}||||j|j}||||<t j|dd|dddd} |d|| d|z  } | | ||j } | | zS)Nrr#rrL) r%narrowr$rbrcr6matmulrr&r1rrrp) rGrrrurvrwrattn_probs_only_globalvalue_vectors_only_globalattn_output_only_globalattn_probs_without_globalattn_output_without_globals r*rz@LEDEncoderSelfAttention._compute_attn_output_with_global_indicesVs6 %a( ",!2!22q:U!V!V$1$;$; 3T^T]% % !IVVrHs!"DE #(, " , ,Q 2 2 8 8 : :&B&BT^+-H''' #M11 # + +,> ? ?4Q5RR Sr,rcPeZdZdZ ddedededed ef fd Zd ej d ed efdZ ddej de ej de e ej de ej de ej dede ej e ej e e ej ffdZ xZS)LEDDecoderAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr~FTrdrbrm is_decoderbiasct||_||_||_||z|_|j|z|jkrt d|jd|d|jdz|_||_tj ||||_ tj ||||_ tj ||||_ tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩r)rErFrdrbrmrcr'scalingrrrek_projv_projq_projout_proj)rGrdrbrmrrrHs r*rFzLEDDecoderAttention.__init__ s "" !Y. =9 $ 6 6"dn""""" }d* $i 94@@@ i 94@@@ i 94@@@  )YTBBB r,tensorrRr9c||||j|jddS)NrrL)rrbrcrr)rGrrRr9s r*_shapezLEDDecoderAttention._shape&s<{{3GGQQRSUVWWbbdddr,Nrkey_value_statespast_key_valuerrrrc |du}|\}} } |||jz} |r||d} |d} n>|rU|||d|} |||d|} n||||d|} |||d|} t j|d| gd} t j|d| gd} nT|||d|} |||d|} |jr| | f}||j zd|j f}|| | |j |} | j |} | j |} | d}t j | | dd}|||j z| |fkr2td||j z| |fd||||d| |fkr+td |d| |fd|| ||j | ||z}| ||j z| |}tj|d}|||j fkr-td |j fd|| dddd| ||j | |z}| ||j z| |}|r=| ||j | |}| ||j z| |}nd}tj||j|j }t j || }|||j z| |j fkr5td ||j | |j fd|| ||j | |j  dd|| | }||}|||fS) rNrrr#rLryz$Attention weights should be of size r}z!Attention mask should be of size r|rz `attn_output` should be of size )r1rrrrrr6rrrbrcrrrr'rrrrmrrr)rGrrr rrris_cross_attentionr9r/rd query_states key_states value_states proj_shaper: attn_weightsattn_weights_reshapedrrs r*rQzLEDDecoderAttention.forward)sU.T9"/"4"4"6"6Wi{{=11DL@  L."<'*J)!,LL  LT[[1A%B%BBLLJ;;t{{3C'D'Db#NNLL  'T[[%?%?SIIJ;;t{{='A'A2sKKLN1$5z#BJJJJ 9nQ&7%FANNNLLT[[%?%?SIIJ;;t{{='A'A2sKKL ? 8),7NDN*B > Ct{{<#>>CZP $Z_j1 (|(*5 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**   %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL',,S4>-A7GTTL},,\r,BB  &##%%$.)::: 1t~FW11',,..11+//2q!<<|?P?PQTVZVdfmov?w?wwL',,S4>-A7GTTL  ) %1$5$5c4>7T[$\$\ !055cDN6JGU\]]LL$( !]**<4AAr,)r~FT)NNNNF)rTrUrVrWrXrr5rFr6rrrrrQrZr[s@r*rr sGG   CCCC C  C  CCCCCC4eU\eCeceeee 488<1526"'lBlB|lB#5<0lB!u|!45 lB !. lB "%,/ lB lB u|Xel3XeEL>Q5RR SlBlBlBlBlBlBlBlBr,rcdeZdZdedeffd Z d dejdejdejfd ZxZ S) LEDEncoderLayerrqrnct|j|_t |||_t j|j|_|j |_ t|j |_ |j |_ t j|j|j|_t j|j|j|_t j|j|_dSrD)rErFr rdr self_attnr LayerNormself_attn_layer_normrmr activation_function activation_fnactivation_dropoutreencoder_ffn_dimfc1fc2final_layer_normr s r*rFzLEDEncoderLayer.__init__s ,VX>>$&L$@$@!~ #F$>?"(";9T^V-CDD9V3T^DD " T^ < <r,NFrrrc |}||||||||} | d}tj||j|j}||z}||}|}|||}tj||j|j}| |}tj||j|j}||z}| |}|j tj krtj|s&tj|r9tj|j jdz } tj|| | }|f| ddzS)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape *(batch, seq_len, embed_dim)* attention_mask (`torch.FloatTensor`): attention mask of size *(batch, 1, tgt_len, src_len)* where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size *(encoder_attention_heads,)*. rrri)r8rrN)r,rrrmrr.r0r3r1r4r5r.r6float16isinfanyisnanr7rclamp) rGrrrrrrrresidual attn_outputs clamp_values r*rQzLEDEncoderLayer.forwards$!~~')++!5)/&  %Q  --mt|VZVc-dd  =0 11-@@  **488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd  =0 --m<<  %- / / K & & * * , , 005 M0J0J0N0N0P0P 0 +m&9::>EK!K K<[YYYM,qrr"222r,)NNNF) rTrUrVrrXrFr6rrQrZr[s@r*r*r*s =y =C = = = = = ="!.3.3|.3 .3 .3.3.3.3.3.3.3.3r,r*ceZdZdeffd Z ddejdeejdeejd eejd eejd eejd eeejd ee dee fdZ xZ S)LEDDecoderLayerrqct|j|_t |j|j|jd|_|j|_t|j |_ |j |_ tj|j|_t |j|j|jd|_tj|j|_tj|j|j|_tj|j|j|_tj|j|_dS)NT)rdrbrmr)rmr)rErFr rdrdecoder_attention_headsattention_dropoutr,rmr r/r0r1rr-r. encoder_attnencoder_attn_layer_normredecoder_ffn_dimr3r4r5rGrqrHs r*rFzLEDDecoderLayer.__init__s ,n4,     ~ #F$>?"(";$&L$@$@!/ N  *,     (*|DN'C'C$9T^V-CDD9V3T^DD " T^ < <r,NFTrrencoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskr r use_cachec x|} | |ddnd} ||| |||\}} } tj||j|j}| |z}||}d}d}|z|} | |ddnd}|||||||\}}}tj||j|j}| |z}||}| |z} |} || |}tj||j |j}| |}tj||j|j}| |z}| |}|f}|r|| |fz }| r|| fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape *(batch, seq_len, embed_dim)* attention_mask (`torch.FloatTensor`): attention mask of size *(batch, 1, tgt_len, src_len)* where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape *(batch, seq_len, embed_dim)* encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size *(batch, 1, tgt_len, src_len)* where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size *(decoder_attention_heads,)*. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for encoder attention heads in a given layer of size *(decoder_attention_heads,)*. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`): Whether the base model outputs attentions. This requires the attentions tensor to be reshaped in this function. NrL)rr rrrrr)rrrrr r) r,rrrmrr.rDrEr0r3r1r4r5)rGrrrHrIrrJr rrKr<self_attn_past_key_valueself_attn_weightspresent_key_valuecross_attn_present_key_valuecross_attn_weightscross_attn_past_key_valuers r*rQzLEDDecoderLayer.forwards,:!:H9S>"1"#5#5Y] >Bnn'3)+/ ?M? ? ; (*; --mt|VZVc-dd  =0 11-@@ (,$! ,$H@N?Yrss(;(;_c %NRN_N_+!65 :8"3 O`OO KM-/KM11-4 >G  , )+ +Gr,)NNNNNNFT) rTrUrVrrFr6rrrr5rQrZr[s@r*r@r@s=y======:268<9=26=A8<,1$(VV|V!.V ( 5 V !) 6 V "%,/ V%-U\$:V!u|!45V$D>VD>VVVVVVVVr,r@cJeZdZdZdedededeffd ZdejfdZ xZ S) LEDClassificationHeadz-Head for sentence-level classification tasks. input_dim inner_dim num_classespooler_dropoutcttj|||_tj||_tj|||_dS)N)r)rErFrredenseDropoutrmr)rGrUrVrWrXrHs r*rFzLEDClassificationHead.__init__MsY Yy)44 zN333  )[99 r,rc||}||}tj|}||}||}|SrD)rmrZr6tanhr)rGrs r*rQzLEDClassificationHead.forwardYs[ ]33  =11  =11  ]33  m44 r,) rTrUrVrWrXrrFr6rrQrZr[s@r*rTrTJs77 : : : :  : : : : : :U\r,rTc6eZdZeZdZdZdZedZ dS)LEDPreTrainedModelledTc|jj}t|tjrJ|jjd||j |jj dSdSt|tj rS|jjd||j -|jj|j  dSdSdS)Nr~)meanstd) rqinit_std isinstancerrerPdatanormal_rzero_ Embedding padding_idx)rGmodulercs r* _init_weightsz LEDPreTrainedModel._init_weightsgsk" fbi ( ( ? M  & &CS & 9 9 9{&  &&((((('&  - - ? M  & &CS & 9 9 9!- "6#56<<>>>>> ? ?--r,c|jj}tjgddddd|gg|j}|||d}|S)N)r rLr rLr)rr)rqr r6rrMne)rG pad_tokenr dummy_inputss r*ruzLEDPreTrainedModel.dummy_inputsrsaK, L"2"2"2Q2q)4L!MVZVabbb 'll955"  r,N) rTrUrVr config_classbase_model_prefixsupports_gradient_checkpointingrlpropertyrur,r*r_r_bsOL&*# ? ? ?Xr,r_ceZdZUdZejed<dZee ejdfed<dZ ee ejdfed<dZ ee ejdfed<dS)LEDEncoderBaseModelOutputa Base class for LEDEncoder's outputs, with potential hidden states, local and global attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, x + attention_window + 1)`, where `x` is the number of tokens with global attention mask. Local attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Those are the attention weights from every token in the sequence to every token with global attention (first `x` values) and to every token in the attention window (remaining `attention_window + 1` values). Note that the first `x` values refer to tokens with fixed positions in the text, but the remaining `attention_window + 1` values refer to tokens with relative positions: the attention weight of a token to itself is located at index `x + attention_window / 2` and the `attention_window / 2` preceding (succeeding) values are the attention weights to the `attention_window / 2` preceding (succeeding) tokens. If the attention window contains a token with global attention, the attention weight at the corresponding index is set to 0; the value should be accessed from the first `x` attention weights. If a token has global attention, the attention weights to all other tokens in `attentions` is set to 0, the values should be accessed from `global_attentions`. global_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, x)`, where `x` is the number of tokens with global attention mask. Global attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Those are the attention weights from every token with global attention to every token in the sequence. last_hidden_stateN.r attentionsglobal_attentions) rTrUrVrWr6 FloatTensor__annotations__rrrr~rrzr,r*r|r|}s!!F((((=AM8E%"3S"89:AAA:>Ju0#567>>>AExe&7&< =>EEEEEr,r|ceZdZUdZdZejed<dZe e ejed<dZ e e ejdfed<dZ e e ejdfed<dZe e ejdfed<dZe ejed <dZe e ejdfed <dZe e ejdfed <dZe e ejdfed <dS) LEDSeq2SeqModelOutputaC Base class for model encoder's outputs that also contains : pre-computed hidden states that can speed up sequential decoding. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the decoder of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. past_key_values (`List[torch.FloatTensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `torch.FloatTensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_heads, sequence_length, embed_size_per_head)`). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the decoder at the output of each layer plus the initial embedding outputs. decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads. encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder of the model. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the encoder at the output of each layer plus the initial embedding outputs. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. encoder_global_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, x)`, where `x` is the number of tokens with global attention mask. Global attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Those are the attention weights from every token with global attention to every token in the sequence. Nr}past_key_values.decoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_staterHencoder_attentionsencoder_global_attentions)rTrUrVrWr}r6rrrrrrrrrrrHrrrzr,r*rrs;55n,0u(///9=OXd5#456===EI8E%*;S*@$ABIIIBFu'8#'=!>?FFF@DhuU%6%;<=DDD=Ax(9:AAAEI8E%*;S*@$ABIIIBFu'8#'=!>?FFFIMxe.?.D(EFMMMMMr,rceZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejdfed<dZee ejdfed<dZee ejdfed <dZeejed <dZee ejdfed <dZee ejdfed <dZee ejdfed <dS)LEDSeq2SeqLMOutputa Base class for sequence-to-sequence language models outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`List[torch.FloatTensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `torch.FloatTensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_heads, sequence_length, embed_size_per_head)`). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the decoder at the output of each layer plus the initial embedding outputs. decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads. encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder of the model. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the encoder at the output of each layer plus the initial embedding outputs. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. encoder_global_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, x)`, where `x` is the number of tokens with global attention mask. Global attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Those are the attention weights from every token with global attention to every token in the sequence. Nlosslogitsr.rrrrrHrrrTrUrVrWrrr6rrrrrrrrrrrHrrrzr,r*rrR33j)-D(5$ %,,, $FE $$$9=OXd5#456===EI8E%*;S*@$ABIIIBFu'8#'=!>?FFF@DhuU%6%;<=DDD=Ax(9:AAAEI8E%*;S*@$ABIIIBFu'8#'=!>?FFFIMxe.?.D(EFMMMMMr,rceZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejdfed<dZee ejdfed<dZee ejdfed <dZeejed <dZee ejdfed <dZee ejdfed <dZee ejdfed <dS)"LEDSeq2SeqSequenceClassifierOutputa  Base class for outputs of sequence-to-sequence sentence classification models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `label` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). past_key_values (`List[torch.FloatTensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `torch.FloatTensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_heads, sequence_length, embed_size_per_head)`). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the decoder at the output of each layer plus the initial embedding outputs. decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads. encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder of the model. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the encoder at the output of each layer plus the initial embedding outputs. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. encoder_global_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, x)`, where `x` is the number of tokens with global attention mask. Global attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Those are the attention weights from every token with global attention to every token in the sequence. Nrrr.rrrrrHrrrrzr,r*rr0rr,rceZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ee ejed<dZ eeejdfed<dZeeejdfed <dZeeejdfed <dZeejed <dZeeejdfed <dZeeejdfed <dZeeejdfed<dS)&LEDSeq2SeqQuestionAnsweringModelOutputa| Base class for outputs of sequence-to-sequence question answering models. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Span-start scores (before SoftMax). end_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Span-end scores (before SoftMax). past_key_values (`List[torch.FloatTensor]`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): List of `torch.FloatTensor` of length `config.n_layers`, with each tensor of shape `(2, batch_size, num_heads, sequence_length, embed_size_per_head)`). Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see `past_key_values` input) to speed up sequential decoding. decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the decoder at the output of each layer plus the initial embedding outputs. decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads. cross_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the decoder's cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads. encoder_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder of the model. encoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the encoder at the output of each layer plus the initial embedding outputs. encoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the self-attention heads. encoder_global_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, x)`, where `x` is the number of tokens with global attention mask. Global attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Those are the attention weights from every token with global attention to every token in the sequence. Nr start_logits end_logitsr.rrrrrHrr)rTrUrVrWrrr6rrrrrrrrrrrrHrrrzr,r*rrsse55n)-D(5$ %,,,&*L%#***$(J!(((9=OXd5#456===EI8E%*;S*@$ABIIIBFu'8#'=!>?FFF@DhuU%6%;<=DDD=Ax(9:AAAEI8E%*;S*@$ABIIIBFu'8#'=!>?FFFIMxe.?.D(EFMMMMMr,ra) This model inherits from [`PreTrainedModel`]. See the superclass documentation for the generic methods the library implements for all its models (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for general usage and behavior. Parameters: config ([`LEDConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Summarization example: ```python >>> import torch >>> from transformers import AutoTokenizer, LEDForConditionalGeneration >>> model = LEDForConditionalGeneration.from_pretrained("allenai/led-large-16384-arxiv") >>> tokenizer = AutoTokenizer.from_pretrained("allenai/led-large-16384-arxiv") >>> ARTICLE_TO_SUMMARIZE = '''Transformers (Vaswani et al., 2017) have achieved state-of-the-art ... results in a wide range of natural language tasks including generative language modeling ... (Dai et al., 2019; Radford et al., 2019) and discriminative ... language understanding (Devlin et al., 2019). ... This success is partly due to the self-attention component which enables the network to capture contextual ... information from the entire sequence. While powerful, the memory and computational requirements of ... self-attention grow quadratically with sequence length, making it infeasible (or very expensive) to ... process long sequences. To address this limitation, we present Longformer, a modified Transformer ... architecture with a self-attention operation that scales linearly with the sequence length, making it ... versatile for processing long documents (Fig 1). This is an advantage for natural language tasks such as ... long document classification, question answering (QA), and coreference resolution, where existing approaches ... partition or shorten the long context into smaller sequences that fall within the typical 512 token limit ... of BERT-style pretrained models. Such partitioning could potentially result in loss of important ... cross-partition information, and to mitigate this problem, existing methods often rely on complex ... architectures to address such interactions. On the other hand, our proposed Longformer is able to build ... contextual representations of the entire context using multiple layers of attention, reducing the need for ... task-specific architectures.''' >>> inputs = tokenizer.encode(ARTICLE_TO_SUMMARIZE, return_tensors="pt") >>> # Global attention on the first token (cf. Beltagy et al. 2020) >>> global_attention_mask = torch.zeros_like(inputs) >>> global_attention_mask[:, 0] = 1 >>> # Generate Summary >>> summary_ids = model.generate(inputs, global_attention_mask=global_attention_mask, num_beams=3, max_length=32) >>> print(tokenizer.decode(summary_ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=True)) ``` a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`LedTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) LED uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_led._prepare_decoder_inputs`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. global_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to decide the attention given on each token, local attention or global attention for the encoder. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the [Longformer paper](https://arxiv.org/abs/2004.05150) for more details. Mask values selected in `[0, 1]`: - 0 for local attention (a sliding window attention), - 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them). head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, *optional*: `hidden_states`, *optional*: `attentions`) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. ceZdZdZddedeejffd Zde j de j fdZ d e j de j d e j d e fd Z dd ZxZS) LEDEncoderz Transformer encoder consisting of *config.encoder_layers* self-attention layers. Each layer is a [`LEDEncoderLayer`]. Args: config: LEDConfig embed_tokens (nn.Embedding): output embedding Nrq embed_tokensctj|_j|_j}j|_j|_ tj trMj dzdkrtdj dkrtdj gjz_ nItj jkr,tdjdtj |||_n%t#jj||j|_t)|j ||_t#jfdt/jD|_t#j||_d|_|dS) NrLrz1`config.attention_window` has to be an even valuez,`config.attention_window` has to be positivezQ`len(config.attention_window)` should equal `config.num_hidden_layers`. Expected z, given c0g|]}t|Srz)r*).0irqs r* z'LEDEncoder.__init__..~s#$f$f$fA_VQ%?%?$f$f$fr,F)rErFrmencoder_layerdrop layerdropr r rjmax_encoder_position_embeddingsmax_source_positionsrerorXr'num_hidden_layersrrrri vocab_sizer@embed_positions ModuleListrencoder_layerslayersr-layernorm_embeddinggradient_checkpointing post_init)rGrqrrdrHs ` r*rFzLEDEncoder.__init__^s    ~ 1N !.$*$J! f-s 3 3 &*a// !TUUU&!++ !OPPP'-'>&?&BZ&ZF # #6*++v/GGG a & 8aaBEfF]B^B^aa  # ,D   " V-> 4K[ \ \D <  %   m$f$f$f$fvOdIeIe$f$f$fgg #%< #:#: &+# r,rglobal_attention_maskc&| ||dzz}n|dz}|S)Nrrz)rGrrs r*_merge_to_attention_maskz#LEDEncoder._merge_to_attention_masks.  %+/Dq/HINN3Q6Nr,r inputs_embedsr ct|jjtr |jjnt |jj}|dzdkrt d|||jn|j}|dd\}}|||zz |z} | dkrtd|d|| zd||$tj |d| f|}|[| || f|jj tj } || } tj|| gd }tj |d| fd }| |||fS) zbA helper function to pad tokens and mask to work with implementation of Longformer self-attention.rLrz2`attention_window` should be an even value. Given Nz(Input ids are automatically padded from z to z0 to be a multiple of `config.attention_window`: r)r.rryF)rerqrorXrr'r%logger warning_oncerrrnew_fullr r6rOrr) rGrrrr ro input_shaperrR padding_leninput_ids_paddinginputs_embeds_paddings r*_pad_to_window_sizezLEDEncoder._pad_to_window_sizes$+6<< 3DK ( (T[122  a 1 $ $dRbddee e)2)>iooMDW )"1"o G''4D*DDHXX ??   A7AAR]H]AA.>AA   $M--i![9IQ]-^^ ($1$:$:-K,*%;%%! )-(9(9:K(L(L% % =:O*PVX Y Y Y ]..K 0/NI~}DDr,c ||n |jj}||n |jj}||n |jj}||t d||t d|||}|@t j|dd|j t j }|| ||}| ||||jj \}}}|1|} |d| d}n||dd} |#t||jddddddf}|dk} |dk} | } || } || z}||}t,j||j|j}|rd nd}|rd nd}|r| rd nd}|p|dt5|jkr@t d t5|jd |dd t9|jD]\}}|r||fz}t jg}|jr||jkrd }n^|jr3|jr,| |j!|||||nd| | | |}n||||||nd| | | |}|d}|rB||d"ddfz}| r ||d"ddfz}|r||fz}dkrI|ddd f}|rtGfd|D}|rtGfd|D}|stGd||||fDStI||||S)aO Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) global_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to decide the attention given on each token, local attention or global attention for the encoder. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the [Longformer paper](https://arxiv.org/abs/2004.05150) for more details. Mask values selected in `[0, 1]`: - 0 for local attention (a sliding window attention), - 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them). head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsr#)rMr.)rrrr rrrzz&The head_mask should be specified for  layers, but it is for r)NNN)rrrrrrrrLr c.g|]}|ddd fSrDrzrstaters r*rz&LEDEncoder.forward..es-'\'\'\Eaaa;,.>(?'\'\'\r,c:g|]}|ddddd ddfSrDrzrs r*rz&LEDEncoder.forward..hs9'b'b'b%aaaMk\M111.D(E'b'b'br,c3K|]}||V dSrDrzrvs r* z%LEDEncoder.forward..ks1eferererererr,r}rr~r)%rqroutput_hidden_statesuse_return_dictr'rr6onesr1rMrOrrr rr>r.flattenr9itemrrrrrmrrr enumeraterandrr_gradient_checkpointing_func__call__rtupler|)rGrrr head_maskrrr return_dictrrrr embed_posrencoder_statesall_attentionsall_global_attentionsidx encoder_layerdropout_probability layer_outputsrs @r*rQzLEDEncoder.forwardsDp2C1N--TXT_Tq$8$D $+Jj &1%cdd d  =#8TUU U   --i88M  !"Z (:(:(<(K`aaNAE@X@X)'1 AYA A = Y   #..**K!r;r?;;II  &',,..ss3K  %@Q^QdeefgfgfgijlmopopopfpqN)1,-1-5577;;==BBDD((55 % 1 00??  --mt|VZVc-dd 3=0:d'8 V^ VRV  ~~"c$+&6&666 /S=M=M//!((+///#,DK"8"8& h& h C# C!/=2B!B"'*R.. } 1"5"F"F 2 .4=$($E$E%.%&*3*? #T',&) % %MM%2M%'5;D;P3VZ(7-A'5*;%%%M!.a 0   h!/=3C3M3MaQR3S3S2U!U!h,A]STEUE_E_`acdEeEeDg,g)  ?+}.>>N ??)!!!]{l]*:;M# ^!&'\'\'\'\^'\'\'\!]!]  d!&'b'b'b'bSa'b'b'b!c!c )>>K`a )+(%3     r,rD)NNNNNNNN)rTrUrVrWrrrrirFr6rrrXrrQrZr[s@r*rrTs %%y%8N%%%%%%N u| \a\h    )E<)E )E| )E  )E)E)E)EZ"!w w w w w w w w r,rcfeZdZdZddedeejffd Z ddZ xZ S) LEDDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`LEDDecoderLayer`] Args: config: LEDConfig embed_tokens (nn.Embedding): output embedding Nrqrc2tj|_j|_j|_j|_|||_ n*tj j j |j|_ t|jj |_tjfdt#jD|_tjj |_d|_|dS)Nc.g|]}tSrz)r@)r_rqs r*rz'LEDDecoder.__init__..s!$c$c$c_V%<%<$c$c$cr,F)rErFrmdecoder_layerdroprr rjmax_decoder_position_embeddingsmax_target_positionsrrrirr r@rrrdecoder_layersrr-rrr)rGrqrrHs ` r*rFzLEDDecoder.__init__s    ~ 1!.$*$J!  # ,D   " V->PTP` a aD <  % N  m$c$c$c$ceFLaFbFb$c$c$cdd #%<#?#? &+# r,c | | n |jj} | | n |jj} | | n |jj} | | n |jj} || t d|1|}|d|d}n.| | dd}nt d||ddjdnd}| | |} d}|ddkrt|| j | j |}|"| |t|| j |d z}||t|| j |d }|||}| |z}||}t j||j|j }|jr%|jr| rt*d d } | rd nd}| rd nd}| rd nd}| rd nd}t/||gddgD]z\}}|s|dt1|jkrCt d|dt1|jd|dd{t5|jD]\}}| r||fz }|jr t7jg}||jkr4|||nd}|jr?|jr8||j|||||||nd|||ndd| | }n(||||||||nd|||nd|| |  }|d}| r||| rdndfz }| r||dfz }||dfz }| r||fz }| r|nd}| stAd|||||fDStC|||||S)az Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) global_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to decide the attention given on each token, local attention or global attention. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the [Longformer paper](https://arxiv.org/abs/2004.05150) for more details. Mask values selected in `[0, 1]`: - 0 for local attention (a sliding window attention), - 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them). encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer#zEYou have to specify either decoder_input_ids or decoder_inputs_embedsrrLr)rJ)r/rzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frzrcross_attn_head_maskzThe `z` should be specified for rr)rrHrIrrJr rrKr c3K|]}||V dSrDrzrs r*rz%LEDDecoder.forward..ps0= ===r,)r}rrr~r)"rqrrrKrr'r1rr%rrr.rMr>rrrrrmrrrrziprrrr6rrrrrr)rGrrrrHrIrrrrrKrrrrrJcombined_attention_maskrSrall_hidden_statesall_self_attnsall_cross_attentionsnext_decoder_cache attn_mask mask_namer decoder_layerrr r next_caches r*rQzLEDDecoder.forwardsit2C1N--TXT_Tq$8$D $+Jj "+!6IIDKstt t  "#..**K!r;r?;;II  &',,..ss3KKdee eDSC^!3A!6!>##A&#dk*:*:::$3 33SEUEU33%NN,,Q/333#,DK"8"8. <. < C# 6!m%55!} &+jnn#&775D5P_S11VZN* t}  $ A A!*!+)*&/&;IcNN1E1Q(--W[% ! ! !. !#:*?+A7@7LYs^^RV5I5U,S11[_#1&7' ! ! ! *!,M V"}:K5RQQQR'S&UU"  <=#3"55$q)9(;;$  2 -!1 1 +4>''$  '5FXlm  9+&+%1     r,rD) NNNNNNNNNNNNN) rTrUrVrWrrrrirFrQrZr[s@r*rrvsy8N2""#!!e e e e e e e e r,rzQThe bare LED Model outputting raw hidden-states without any specific head on top.c'eZdZddgZdeffd ZdZdZdZdZ e e e e ee  dd eejd eejd eejdeejdeejdeejdeejdeeeejdeejdeeeejdeejdeejdeedeedeedeedeeejeff"dZxZS)LEDModeldecoder.embed_tokens.weightencoder.embed_tokens.weightrqc6t||j|j}}t j||j||_t||j|_ t||j|_ | dSrD) rErFr rrrir sharedrencoderrdecoderr)rGrqrjrrHs r*rFzLEDModel.__init__s    "("5v7HZ l:v~{KK !&$+66 !&$+66  r,c|jSrD)rrGs r*get_input_embeddingszLEDModel.get_input_embeddingss {r,cX||_|j|j_|j|j_dSrD)rrrr)rGrhs r*set_input_embeddingszLEDModel.set_input_embeddingss' $(K !$(K !!!r,c|jSrD)rrs r* get_encoderzLEDModel.get_encoder |r,c|jSrD)rrs r* get_decoderzLEDModel.get_decoderrr, checkpoint output_typervNrrdecoder_input_idsdecoder_attention_maskrdecoder_head_maskrencoder_outputsrrrdecoder_inputs_embedsrKrrrrc||n |jj}||n |jj}| | n |jj} ||n |jj}|'| %t ||jj|jj}||||| || |||}n|rt|tsjt|dt|dkr|dndt|dkr|dndt|dkr|dnd}| |||d|||| | | ||| }|s||zSt|j|j|j|j|j|j|j|j|j S) N)rrrrrrrrrrrLr r) rrrHrIrrrrrKrrr) r}rrrrrrHrr)rqrrrKrr+r r!rrer|rrrr}rrr~rr)rGrrrrrrrrrrrrrKrrrdecoder_outputss r*rQzLEDModel.forwards22C1N--TXT_Tq$8$D $+Jj "+!6IIDK)F 24;3T[5W!!   ""ll#-&;#+"3%9'+  OO O=V!W!W 7"1!"4474H4H14L4Loa00RV14_1E1E1I1I?1--t8;O8L8Lq8P8P/!"4"4VZ O,,'1"1!"4#1'!5+//!5#'    5"_4 4$-?+;"1"?.9,=&5&G"1"?.9&5&G    r,NNNNNNNNNNNNNNNN)rTrUrV_tied_weights_keysrrFrrrrrLED_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOCrr6 LongTensorrrrr5rrrQrZr[s@r*rr~so 89VW y      000 +*+?@@&&$15158<=A,0487;EI=AEI59=A$(,0/3&*#S S E,-S !.S $E$45 S !))9 : S EL) S $EL1S 'u|4S "%e.?(@"ABS  ((9:S "%e.?(@"ABS   12S  ((9:S D>S $D>S 'tn!S "d^#S $ uU\"$99 :%S S S A@ S S S S S r,rzKThe LED Model with a language modeling head. Can be used for summarization.c*<eZdZdZdgZgdZdeffd ZdZdZ d%d e d e e d e j ffd Zd e d dfd ZdZdZeeeeeee d&de ejde ejde ejde ejde ejde ejde ejde eeejde ejde eeejde ejde ejde ejde ede ed e ed!e ed eeeje ff$d"Z!dejfd#Z"e#d$Z$xZ%S)'LEDForConditionalGenerationr`final_logits_bias)rrzlm_head.weightrqclt|t||_|dt jd|jjjftj |j |jjjd|_ | dS)NrrFr)rErFrr`register_bufferr6zerosrrArrer lm_headrrGs r*rFz$LEDForConditionalGeneration.__init__ s    F## 0%+q$(/B`>a2b2bcccy1OV[\\\  r,c4|jSrD)r`rrs r*rz'LEDForConditionalGeneration.get_encoder x##%%%r,c4|jSrD)r`rrs r*rz'LEDForConditionalGeneration.get_decoder rr,Nnew_num_tokenspad_to_multiple_ofrct||}||jjd|S)Nr)rEresize_token_embeddings_resize_final_logits_biasrPr%)rGrrnew_embeddingsrHs r*rz3LEDForConditionalGeneration.resize_token_embeddings sB88I[\\ &&~'<'B1'EFFFr,c|jjd}||kr|jddd|f}nBtjd||z f|jj}tj|j|gd}|d|dS)Nr#rrryr)rr%r6rrMrr)rGrold_num_tokensnew_bias extra_biass r*rz5LEDForConditionalGeneration._resize_final_logits_bias s/5b9 ^ + +-aaa..@AHHa.)H%IRVRhRopppJy$"8*!E1MMMH 0(;;;;;r,c|jSrDrrs r*get_output_embeddingsz1LEDForConditionalGeneration.get_output_embeddings rr,c||_dSrDr%)rGrs r*set_output_embeddingsz1LEDForConditionalGeneration.set_output_embeddings# s % r,)rrvrrrrrrrrrrrrlabelsrKrrrc||n |jj}| G|rtdd}|'| %t | |jj|jj}||||||| |||| | | ||||}||d|j z}d}| Kt}|| d|jj | d}|s|f|ddz}||f|zn|St|||j|j|j|j|j|j|j|j S) a  labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Returns: Conditional generation example: ```python >>> from transformers import AutoTokenizer, LEDForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("allenai/led-base-16384") >>> TXT = "My friends are but they eat too many carbs." >>> model = LEDForConditionalGeneration.from_pretrained("allenai/led-base-16384") >>> input_ids = tokenizer([TXT], return_tensors="pt")["input_ids"] >>> prediction = model.generate(input_ids)[0] >>> print(tokenizer.decode(prediction, skip_special_tokens=True)) ```NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rrrrrrrrrrrrKrrrrr#r rrrrrrrrHrr)rqrrwarningr+r r!r`rrr rrrrrrrrrHrr)rGrrrrrrrrrrrrr)rKrrrr lm_logitsmasked_lm_lossloss_fctr s r*rQz#LEDForConditionalGeneration.forward& sZ&1%. sCrrU_j--aZ=N1O1OPPrrrrrrr,rL)r)rr6reordered_past layer_pasts ` r*_reorder_cachez*LEDForConditionalGeneration._reorder_cache sm)  J rrrrcmnpopnpcqrrrrrQRR.! NNr,rDNNNNNNNNNNNNNNNNN)&rTrUrVrw_keys_to_ignore_on_load_missingr rrFrrrXrrrirrr&r(rr rrrrLED_GENERATION_EXAMPLEr6rrrrr5rrrQr1rr9rZr[s@r*rrs<':&;#iiiy&&&&&&cxX[}hjht <<<<<<&&&+*+?@@?YYY.//15158<=A,0487;EI=AEI59=A-1$(,0/3&*%] ] E,-] !.] $E$45 ] !))9 : ] EL) ] $EL1] 'u|4] "%e.?(@"AB]  ((9:] "%e.?(@"AB]   12]  ((9:] )*] D>] $D>!] "'tn#] $d^%] & uU\"$66 7'] ] ] 0/ZYA@] ~hELhhhh\r,rz LED model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c'ZeZdZddgZdeffd Zeeee e e  dde e jde e jd e e jd e e jd e e jd e e jd e e jde eee jde e jde e jde e jde e jde ede ede ede edeee jeff"dZxZS)LEDForSequenceClassificationrrrqc tjdttj|fi|t ||_t|j|j|j |j |_ | dS)NzThe `transformers.LEDForSequenceClassification` class is deprecated and will be removed in version 5 of Transformers. No actual method were provided in the original paper on how to perfom sequence classification.) warningswarn FutureWarningrErFrr`rTr  num_labelsclassifier_dropoutclassification_headr)rGrqkwargsrHs r*rFz%LEDForSequenceClassification.__init__ s  (     **6***F###8 N N    % $ $   r,rNrrrrrrrrrrrr)rKrrrrc,||n |jj}| d} || td|jj|||||| ||||| | | |||}|d}||jj|j }ttj | ddkrtd||ddf|dd|ddddddf}||}d}| n|jjp|jjdkr d |j_nS|jjdkr7| jtjks| jtjkr d |j_n d |j_|jjd kr\t/}|jjdkr1||| }n||| }n|jjd krLt3}||d|jj| d}n*|jjd krt5}||| }|s|f|ddz}||f|zn|St7|||j|j|j|j|j |j!|j"|j# S) a3 labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for rrrrrrrrrrrKrrrrrz7All examples must have the same number of tokens.r# regressionsingle_label_classificationmulti_label_classificationr+)$rqrNotImplementedErrorrHrTr`eq eos_token_idr3rMrr6unique_consecutiverr'rr1rE problem_typerCr.rOrXr squeezer r rrrrrrrHrr)rGrrrrrrrrrrrr)rKrrrrreos_masksentence_representationrrr/r s r*rQz$LEDForSequenceClassification.forward sH<&1%Kbdd (( )/#9"7/!5+'"7/!5#  "  << 899<<]=QRR u' Q88 9 9A = =VWW W"/!!! "<"A"A-BTBTUVBWBWY[]j]o]opr]s]s"t"t AAr111H# ))*ABB  {'/;)Q../;DK,,[+a//V\UZ5O5OSYS_chclSlSl/LDK,,/KDK,{'<77"99;)Q..#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B 0F G GUWYY)-III,..x// FY,F)-)9TGf$$v E1#3")"?&9$5&-&G")"?&9&-&G    r,r )rTrUrVr rrFrr rr rrrr6rrrrr5rrrQrZr[s@r*r>r> s)89VWy&+*+?@@&3$15158<=A,0487;EI=A59=A-1$(,0/3&*#b b E,-b !.b $E$45 b !))9 : b EL) b $EL1b 'u|4b "%e.?(@"ABb  ((9:b   12b  ((9:b )*b D>b $D>b 'tn!b "d^#b $ uU\"$FF G%b b b A@ b b b b b r,r>z LED Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layer on top of the hidden-states output to compute `span start logits` and `span end logits`). c)peZdZddgZfdZeeeee e  dde e j de e jde e j d e e j d e e jd e e jd e e jd e eee jde e jde e j de e j de e jde e jde ede ede ede edeee jeff$dZxZS)LEDForQuestionAnsweringrrct|d|_|j|_t||_t j|j|j|_| dS)NrL) rErFrCrr`rrer` qa_outputsrrGs r*rFz LEDForQuestionAnswering.__init__) sm     +F##)F$68IJJ r,rNrrrrrrrrrstart_positions end_positionsrrrKrrrrc||n |jj}| | d}|||||| ||||| | ||||}|d}||}|dd\}}|d}|d}d}| | t| dkr| d} t| dkr| d} |d}| d|} | d|} t|}||| }||| }||zd z }|s||f|ddz}||f|zn|St||||j |j |j|j|j|j|j|j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (*sequence_length*). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (*sequence_length*). Position outside of the sequence are not taken into account for computing the loss. NFrHrrr#ry) ignore_indexrL) rrrrrrrrrHrr)rqrr`rWsplitrQrrr1r;r rrrrrrrHrr)rGrrrrrrrrrrXrYrrrKrrrrsequence_outputrrr total_loss ignored_indexr/ start_lossend_lossr s r*rQzLEDForQuestionAnswering.forward5 sdH&1%_ $D>!_ "'tn#_ $d^%_ & uU\"$JJ K'_ _ _ A@ _ _ _ _ _ r,rUrD)KrWrr@ dataclassesrtypingrrrrr6torch.utils.checkpointrtorch.nnr r r activationsr generationrmodeling_attn_mask_utilsrmodeling_outputsrrrrrmodeling_utilsrutilsrrrrrrrconfiguration_ledr get_loggerrTrr rrrXr+r.r>rir@Moduler]rrr*r@rTr_r|rrrrLED_START_DOCSTRINGr<r rrrrr>rUrzr,r*rps !!!!!!//////////// AAAAAAAAAA!!!!!!))))))HHHHHH.-----)(((((  H % %. %, c [^     ##el#5;#YabeYf####$*****BL***$x 5x 5x 5x 5x 5bix 5x 5x 5v")DLBLBLBLBLB")LBLBLB^;3;3;3;3;3bi;3;3;3|qqqqqbiqqqhBI06 'F'F'F'F'F 'F'F 'FT @N@N@N@N@NK@N@N @NF ?N?N?N?N?N?N?N ?ND ?N?N?N?N?N?N?N ?ND BNBNBNBNBN[BNBN BNJ $LdN_ _ _ _ _ #_ _ _ D E E E E E #E E E PWv v v v v !v v v rQSfVVVVV"4oVVVr ~ ~ ~ ~ ~ #5~ ~ ~ B t t t t t 0t t t t t r,