g vdZddlZddlZddlmZddlmZmZmZddl Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZmZmZmZmZmZmZdd lmZdd l m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/dZ0dZ1e+rddl2m3Z4e(rddl5m6Z6e,j7e8Z9dZ:dZ;dZdZ?dZ@dZAdZBdZCddgZDdZEd ZFeGd!d"e#ZG dud#eeHeHfd$eId%eHd&ee jJd'eHd(e jKf d)ZL dvd*ed+eHd,ee jKfd-ZMGd.d/e jNZOGd0d1e jNZPGd2d3e jNZQGd4d5e jNZRGd6d7e jNZSGd8d9e jNZTGd:d;eTZUGd<d=e jNZVGd>d?e jNZWGd@dAeWZXGdBdCeWZYeWeYeXdDZZGdEdFe jNZ[GdGdHe jNZ\GdIdJe jNZ]GdKdLe jNZ^GdMdNe jNZ_GdOdPe jNZ`GdQdRe jNZaGdSdTe jNZbGdUdVe jNZcGdWdXeZddYZedZZfe%d[eeGd\d]edZge%d^eeGd_d`edZhe%daeeGdbdcedZie%ddeedeGdfdgedZje%dheeGdidjedZke%dkeeGdldmedZlGdndoe jNZmGdpdqe jNZne%dreeGdsdtedZodS)wzPyTorch Wav2Vec2 model.N) dataclass)OptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)BaseModelOutputCausalLMOutputMaskedLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)PreTrainedModel)#is_torch_greater_or_equal_than_1_13) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forward cached_fileis_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10is_peft_availableis_safetensors_availableloggingreplace_return_docstrings)Wav2Vec2Configzadapter.{}.binzadapter.{}.safetensors) load_file)_flash_attention_forwardr"zfacebook/wav2vec2-base-960h)r!i$iz['MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL'g= ףpJ@zsuperb/wav2vec2-base-superb-ksz '_unknown_'g)\(@zanton-l/wav2vec2-base-superb-sdzanton-l/wav2vec2-base-superb-svg\(\?c(eZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ee ejed<dZee ejed<dZeejed <dZeejed <dS) Wav2Vec2ForPreTrainingOutputa1 Output type of [`Wav2Vec2ForPreTraining`], with potential hidden states and attentions. Args: loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) . (classification) loss. projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. contrastive_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The contrastive loss (L_m) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) . diversity_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The diversity loss (L_d) as stated in the [official paper](https://arxiv.org/pdf/2006.11477.pdf) . Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentionscontrastive_lossdiversity_loss)__name__ __module__ __qualname____doc__r(rtorch FloatTensor__annotations__r)r*r+r,rr-r.r/j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.pyr'r'bs<)-D(5$ %,,,*.e'...48 1888/35,3338sequence_lengths r9compute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spans~i,6DwNOOoy99 [ (? : :-z)_compute_mask_indices..s 9 9 9!o 9 9 9r8dtyperF)replace) ValueErrornprandomranditemsumdetachtolistrangezerosboolchoicearangelen concatenateonesint32appendarray broadcast_toreshaperDput_along_axis)r:r;r<r=r> batch_sizerI input_lengths spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrErFspec_aug_mask_idxdummy_mask_idxoffsetsrGrHs `` ` @@r9_compute_mask_indicesrpsP0#(JQABBB_$$ :^i : :'6 : : :   innQ$$&&G         $  % 2%%''..000 9 9 9 9uZ'8'8 9 9 9Hj/:$GGGM11/BBa%55 11,??I,, IlkAo6 7 7RW-    ! !Q & &-q0NN.q1NN (;o(MUWU] ^ ^ ^ao o p   !!"34444"455111aaa:&5H+(V,33J@SVa@abbi $$T4]3Gog 4G'UVV^^'+5G,g5/A"555GVYZGZ-!0CCDm%7B??? r8features_shape num_negativesmask_time_indicesc|\}}tj|}tj|||ftj}||t ntj|t }t|D]}||dz }|||} tj tj|dzdddf|dz|f} tj d||dz|f} | | | kxxdz cc<| | ||||<||xx||zz cc<|S)z> Sample `num_negatives` vectors from feature vectors. )r:rPNrOr!r)size) rSr^r[rbastyper\rarZrWrerTrandint) rqrrrsrhrHsequence_length_rangesampled_negative_indices batch_idxhighmapped_masked_indicesfeature_indicessampled_indicess r9_sample_negative_indicesrs #1JIo66 "xz?M.Zbdbjkkk+<*G  &&&RWUckoMpMpMp:&& K K  +//11A5 56G 6R S/")D1H*=*=aaag*FPQS`Habb)++At4!8]:S+TT?:;;;q@;;;MbbqLr +,=i,HI !+++y?/JJ++++ ##r8c&eZdZdfd ZdZxZS)Wav2Vec2NoLayerNormConvLayerrcZt|dkr|j|dz nd|_|j||_t j|j|j|j||j||j |_ t|j |_ dS)Nrr! kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__s r9rz%Wav2Vec2NoLayerNormConvLayer.__init__(s !#C#Caar8cJ|jdkr|ddddd|j f}|SNr)rrs r9rzWav2Vec2SamePadLayer.forwards;   " ")!!!QQQ0F43F2F0F*FGMr8rrs@r9rrsLKKKKKr8rc.eZdZdZfdZdZdZxZS)Wav2Vec2FeatureEncoderz.Construct the features from raw audio waveformctjdkr7tdgfdt jdz Dz}nDjdkr!fdt jD}nt djd tj||_ d |_ d |_ dS) Ngrouprrc8g|]}t|dzS)r!r)rrLirs r9rNz3Wav2Vec2FeatureEncoder.__init__..s>NNNIJ,Va!eDDDNNNr8r!layerc2g|]}t|S)r)rrs r9rNz3Wav2Vec2FeatureEncoder.__init__..s4CD*6A>>>r8z`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rrfeat_extract_normrrZnum_feat_extract_layersrRr ModuleList conv_layersgradient_checkpointing_requires_grad)rrrrs ` r9rzWav2Vec2FeatureEncoder.__init__s   #w . .5fqIIIJNNNNNSTZTruvTvNwNwNNNKK % 0 0HMfNlHmHmKKt1Ittt =55&+#"r8cP|D] }d|_ d|_dSNF) parameters requires_gradrrparams r9_freeze_parametersz)Wav2Vec2FeatureEncoder._freeze_parameterss4__&& ( (E"'E  #r8c|dddf}|jr|jrd|_|jD]>}|jr*|jr#|jr||j|}3||}?|S)NT)rtrainingrrr_gradient_checkpointing_func__call__)r input_valuesr, conv_layers r9rzWav2Vec2FeatureEncoder.forwards$QQQW-    /4= /*.M '* : :J" :t'B :t} : $ A A'!!! !+ = 9 9 r8)r0r1r2r3rrrrrs@r9rrs\88#####&$$$ r8rceZdZfdZxZS)Wav2Vec2FeatureExtractorct|tjd|jjd|jjdjdtdS)Nz The class `zD` has been depreciated and will be removed in Transformers v5. Use `rz ` instead.)rrwarningswarnrr0 __bases__ FutureWarningrrrs r9rz!Wav2Vec2FeatureExtractor.__init__sy      E$.1 E EN,Q/8 E E E       r8)r0r1r2rrrs@r9rrs8         r8rc$eZdZfdZdZxZS)Wav2Vec2FeatureProjectionc.ttj|jd|j|_tj|jd|j|_ tj |j |_ dS)NrJeps) rrrrrlayer_norm_epsrLinearr projectionDropoutfeat_proj_dropoutdropoutrs r9rz"Wav2Vec2FeatureProjection.__init__sn ,vr':@UVVV)FOB$79KLLz&":;; r8c||}||}||}||fSr)rrr)rr,norm_hidden_statess r9rz!Wav2Vec2FeatureProjection.forwardsC!__];;(:;;  ]33 000r8rrs@r9rrsG<<<<< 1111111r8rcheZdZdZ ddededed ed ed ed eeffd Z de j dedefdZ dde j dee j dee e j dee j dee j dede e j ee j ee e j ffdZxZS)Wav2Vec2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsr is_decoderr is_causalrc t||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ tj ||||_ tj ||||_tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g)r)rrrrrhead_dimrrRscalingrr rrk_projv_projq_projout_proj) rrrrrrr rrs r9rzWav2Vec2Attention.__init__s "" !Y.  MI %$. 8 83dn33%.333 }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r8tensorseq_lenbszc||||j|jddSr)viewrr r contiguousrrrrs r9_shapezWav2Vec2Attention._shapes<{{3GGQQRSUVWWbbdddr8r,key_value_statespast_key_valuer=layer_head_maskoutput_attentionsr?cZ |du}|\}} } |||jz} |r6|4|djd|jdkr|d} |d} n>|rU|||d|} |||d|} n||||d|} |||d|} tj|d| gd} tj|d| gd} nT|||d|} |||d|} |j r| | f}||j zd|j f}|| | |j |} | j |} | j |} | d}tj| | dd}|||j z| |fkr2t!d||j z| |fd||||d| |fkr+t!d |d| |fd|| ||j | ||z}| ||j z| |}t"j|d}|||j fkr-t!d |j fd|| dddd| ||j | |z}| ||j z| |}|r=| ||j | |}| ||j z| |}nd}t"j||j|j }tj|| }|||j z| |j fkr7t!d ||j z| |j fd|| ||j | |j }|dd}| || |j}||}|||fS) #Input shape: Batch x Time x ChannelNrr%r!rJrz$Attention weights should be of size , but is z!Attention mask should be of size z/Head mask for a single layer should be of size )pr `attn_output` should be of size )rurr r:rr rr4catrrr rrfbmmrrRr functionalsoftmaxrrrr)rr,rrr=rris_cross_attentionrtgt_lenrM query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r9rzWav2Vec2Attention.forwards.T9',,..Wa{{=11DL@  L*q!'*.>.DQ.GGG(*J)!,LL  LT[[1A%B%BBLLJ;;t{{3C'D'Db#NNLL  'T[[%?%?SIIJ;;t{{='A'A2sKKLN1$5z#BJJJJ 9nQ&7%FANNNLLT[[%?%?SIIJ;;t{{='A'A2sKKL ? 8),7NDN*B > Ct{{<#>>CZP 'Z'4 +|+Z8 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**   %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL',,S4>-A7GTTL},,\r,BB  &##%%$.)::: 1t~FW11',,..11+//2q!<<|?P?PQTVZVdfmov?w?wwL',,S4>-A7GTTL  ) %1$5$5c4>7T[$\$\ !055cDN6JGU\]]LL$( !]**<4AAr8)rFTFNNNNNF)r0r1r2r3rCfloatr\rr"rr4Tensorrrrrrs@r9rrsGG  +/CCCC C  C  CC(CCCCCC>eU\eCeceeee 488<1526"'vBvB|vB#5<0vB!u|!45 vB !. vB "%,/ vB vB u|Xel3XeEL>Q5RR SvBvBvBvBvBvBvBvBr8rc2eZdZdZfdZdejdedefdZ dd ejd e ejd e e ejd e ejd e ejde de eje eje e ejffdZ xZ S)Wav2Vec2FlashAttention2aL Wav2Vec2 flash attention module. This module inherits from `Wav2Vec2Attention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cbtj|i|t |_dSr)rrr_flash_attn_uses_top_left_mask)rargskwargsrs r9rz Wav2Vec2FlashAttention2.__init__s9$)&))) 3V2W2W.W+++r8rrrcF||||j|jSr)rrr rs r9_reshapez Wav2Vec2FlashAttention2._reshapes{{3GGGr8NFr,rrr=rrr?c R|rtd|du}|\}} } |||d|} |r^|\|djd|jdkr:|ddd} |ddd} ng|rV|||d|} |||d|} n||||d|} |||d|} tj |ddd| gd} tj |ddd| gd} nT|||d|} |||d|} |j r,| dd| ddf}| jd}|||djdz }| j }|tj krtj rtj}n3t|jdr |jj}n|jjj }t&d |d | |} | |} | |} t-| | | || |jr|jnd |j|j }||| d}||}|sd}|||fS) NzDWav2Vec2FlashAttention2 attention does not support output_attentionsrJrr%r!rr_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .r)rr use_top_left_mask)rRrur<rr:rr rr4r#rrPfloat32is_autocast_enabledget_autocast_gpu_dtyperrr>rlogger warning_oncetor$rrr r8rfr)rr,rrr=rrr'rq_lenrMr)r*r+ kv_seq_len input_dtype target_dtyper1r.s r9rzWav2Vec2FlashAttention2.forwards  ecdd d.T9%**,, UA}}T[[%?%?SII  N*q!'*.>.DQ.GGG(*44Q::J)!,66q!<Q5RR Si9i9i9i9i9i9i9i9r8r6c eZdZ d dejdeejdeeejdeejdeejded eejeejeeejfffd ZxZ S) Wav2Vec2SdpaAttentionNFr,rrr=rrr?c|s|Atdt||||||S|du}|\}} } ||} |r6|4|djd|jdkr|d} |d} n>|rU|||d|} || |d|} n||||d|} || |d|} tj |d| gd} tj |d| gd} nT|||d|} || |d|} |j r| | f}|| | |} |j r || dkrd nd }tjj| | | ||jr|jnd | }|||j| |jfkr5t+d ||j| |jfd||dd}||| |j}||}|d|fS)rNaWav2Vec2Model is using Wav2Vec2SdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `layer_head_mask` not None. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrr=rrrr%r!rJrTFr) attn_mask dropout_pr r"r )rDrErrrurr:rr rr4r#rr rr%scaled_dot_product_attentionrrrr rRrrfrr)rr,rrr=rrr'rr(rMr)r*r+r r1rs r9rzWav2Vec2SdpaAttention.forwardsK   ;   l   77??!1-- /"3 # .T9',,..Wa{{=11  L*q!'*.>.DQ.GGG(*J)!,LL  LT[[1A%B%BBLLJ;;t{{3C'D'Db#NNLL  'T[[%?%?SIIJ;;t{{='A'A2sKKLN1$5z#BJJJJ 9nQ&7%FANNNLLT[[%?%?SIIJ;;t{{='A'A2sKKL ? 8),7N{{<#>> !N`~/E'TU++DD[` h)FF   $&*mQ5RR Sf1f1f1f1f1f1f1f1f1f1r8rL)eagersdpaflash_attention_2c$eZdZfdZdZxZS)Wav2Vec2FeedForwardcttj|j|_tj|j|j|_ t|j trt|j |_n |j |_tj|j|j|_tj|j|_dSr)rrrractivation_dropoutintermediate_dropoutrrintermediate_sizeintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrs r9rzWav2Vec2FeedForward.__init__s $&Jv/H$I$I!"$)F,>@X"Y"Y f' - - 9'-f.?'@D $ $'-'8D $If&>@RSS j)>??r8c||}||}||}||}||}|Sr)rZr^rXr_rars r9rzWav2Vec2FeedForward.forwardsg// >> 00?? 11-@@ ))-88 ++M:: r8rrs@r9rUrUsL @ @ @ @ @r8rUc&eZdZfdZddZxZS)Wav2Vec2EncoderLayerctt|j|j|j|jd|_tj |j |_ tj |j|j |_t||_tj |j|j |_dS)NFrrrrr)rrWAV2VEC2_ATTENTION_CLASSES_attn_implementationrnum_attention_headsattention_dropout attentionrrr`rrrrrU feed_forwardfinal_layer_normrs r9rzWav2Vec2EncoderLayer.__init__s 3F4OP(0,    z&"788 ,v'9v?TUUU/77 " V-?VEZ [ [ [r8NFc|}||||\}}}||}||z}||}|||z}||}|f}|r||fz }|SNr=r)rkrrrlrmrr,r=r attn_residualr.rMoutputss r9rzWav2Vec2EncoderLayer.forwards% )- .L]*8* * & |Q ]33 % 5  66 %(9(9-(H(HH --m<<  "  '   &Gr8rrrs@r9rdrdsQ \ \ \ \ \r8rdcXeZdZfdZ ddejdeejdefdZxZ S) #Wav2Vec2EncoderLayerStableLayerNormctt|j|j|j|jd|_tj |j |_ tj |j|j |_t||_tj |j|j |_t%|ddt'||_dSd|_dS)NFrfradapter_attn_dim)rrrgrhrrirjrkrrr`rrrrrUrlrmgetattrWav2Vec2AttnAdapterLayer adapter_layerrs r9rz,Wav2Vec2EncoderLayerStableLayerNorm.__init__s 3F4OP(0,     z&"788 ,v'9v?TUUU/77 " V-?VEZ [ [ [ 6-t 4 4 @!9&!A!AD   !%D   r8NFr,r=rcJ|}||}||||\}}}||}||z}||||z}|j|||z}|f}|r||fz }|Sro)rrkrrlrmrzrqs r9rz+Wav2Vec2EncoderLayerStableLayerNorm.forwards &  66 )- .L]*8* * & |Q ]33 % 5 %(9(9$:O:OP]:^:^(_(__   ))D,>,>},M,MMM "  '   &Gr8r) r0r1r2rr4r4rr\rrrs@r9rurus~&&&&&*26"' |!. r8ruc deZdZfdZ d dejdeejdeded ef d Z xZ S) Wav2Vec2Encoderct|_t|_t jjj|_ t j j |_ t j fdtjD|_d|_jdk|_dS)Nrc.g|]}tSr7)rdrLrMrs r9rNz,Wav2Vec2Encoder.__init__..s"$k$k$ka%9&%A%A$k$k$kr8FrSrrrrpos_conv_embedrrrrrrr`rrrZnum_hidden_layerslayersrrh_use_flash_attention_2rs `r9rzWav2Vec2Encoder.__init__s  =fEE,v'9v?TUUUz&"788 m$k$k$k$k5QWQiKjKj$k$k$kll &+#&,&AEX&X###r8NFTr,r=routput_hidden_states return_dictc(|rdnd}|rdnd}||ddd|jd}d||<|jr |d|vr|nd}nd|ddddddf|jz }|t j|jjz}| |jdd|jd|jd}| |} || z}| |}| |}tpt|} |jD]} |r||fz}t jg} |jr| |jjkrdnd } | r| rB|jr%|jr|| j|||}n| ||| }|d}| rd }|r ||dfz}|r||fz}|st/d |||fDSt1||| S)Nr7rJr!r%r?rOTFrpNNc3K|]}||V dSrr7rLvs r9 z*Wav2Vec2Encoder.forward..5(mmq_`_l_l_l_l_lmmr8last_hidden_stater,r-) unsqueezerepeatr:rrFrPr4finfominexpandrrrr r rrUrr layerdroprrrtupler rr,r=rrrall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusrdropout_probabilityskip_the_layer layer_outputss r9rzWav2Vec2Encoder.forwards#7@BBD$5?bb4  %$2$<$Q2R2R2V!V!/!6!6"(+Q0DR0H.J^_aJb""#11-@@%(;;  66  ]33 022R6LT6R6R [ P PE# I$58H$H!#(*R.. %)]o8KdkNc8c8cTTjoN! 1[ 1. 4= $($E$E%&) %%MM%*E%nXi%%%M!.a 0  - ,   P&9]1=M.Es" b b bQ 0 8 8 b b br8FrSrrs `r9rz'Wav2Vec2EncoderStableLayerNorm.__init__>s  =fEE,v'9v?TUUUz&"788 m b b b b%H`BaBa b b b   ',#&,&AEX&X###r8NFTcX|rdnd}|rdnd}||ddd|jd}|||jz}|jr |d|vr|nd}nd|ddddddf|jz }|t j|jjz}| |jdd|jd|jd}| |} || z}| |}tpt|} |jD]} |r||fz}t jg} |jr| |jjkrdnd } | r| rB|jr%|jr|| j|||}n| ||| }|d}| rd }|r ||dfz}||}|r||fz}|st/d |||fDSt1||| S)Nr7rJr!r%rOrrTFrprc3K|]}||V dSrr7rs r9rz9Wav2Vec2EncoderStableLayerNorm.forward..rr8r)rrr:rFrPrr4rrrrrr r rrUrrrrrrrrr rs r9rz&Wav2Vec2EncoderStableLayerNorm.forwardJs#7@BBD$5?bb4  %$2$<$Q2R2R2V!V!/!6!6"(+Q0DR0H.J^_aJb""#11-@@%(;;  ]33 022R6LT6R6R [ P PE# I$58H$H!#(*R.. %)]o8KdkNc8c8cTTjoN! 1[ 1. 4= $($E$E%&) %%MM%*E%nXi%%%M!.a 0  - ,   P&9]1=MrJ) flattenrr:r4where zeros_likerWmeanexplog)probsmask mask_extendedmarginal_probs perplexitys r9_compute_perplexityz1Wav2Vec2GumbelVectorQuantizer._compute_perplexitys   LLNN111dD=9@@MMMK ue6Fu6M6MNNE"YY1Y-- :NN"ZZAZ..NY .59^VZEZ;[;[*[ac d d ddeeiikk r8c|j\}}}||}|||z|jzd}|jrt j||j d |}tj |||z|jdd}| ||}n|d} ||jd| ddd}|||z|jd}| ||}|||zd}|d|jz} | ||z|j|jd} | d||d} | |fS)NrJT)tauhardrr!rr)r:rrrrrr%gumbel_softmaxr3rtype_asr4r&rargmax new_zerosscatter_rrrrW) rr,rsrhrHrcodevector_probscodevector_soft_distrcodevector_idxcodevectors_per_grouprs r9rz%Wav2Vec2GumbelVectorQuantizer.forwards3@3F0 O[((77 %**:+G$/+Y[]^^ = W!};;##%%4+;$ <  gm$$  $)="":#?RTUU[[]]ce$$$ 112FHYZZJJ+11b199N,66}7JKKTTN''A..    044Z/5QSWSbdfgg 112BDUVVJ+00o1MrRR 0 : :2 > >AQ Q+00o1Mt`d`moqrr !oob))..z?BOO J&&r8r) r0r1r2r3r staticmethodrrrrs@r9rrsv (   \ #'#'#'#'#'#'#'#'r8rc$eZdZfdZdZxZS)Wav2Vec2AdapterctjjkrCt jjj|_t jj|_ndx|_|_t j fdtj D|_ j |_ dS)Nc36K|]}tVdSr)Wav2Vec2AdapterLayerrs r9rz+Wav2Vec2Adapter.__init__..s,#k#kQ$8$@$@#k#k#k#k#k#kr8)rroutput_hidden_sizerrrprojrproj_layer_normrrZnum_adapter_layersrrrs `r9rzWav2Vec2Adapter.__init__s   $(: : : &"4f6OPPDI#%<0I#J#JD /3 3DI,m#k#k#k#k%PVPiJjJj#k#k#kkk )r8cX|j1|j*||}||}|dd}|jD]=}tj}|jr ||jkr ||}>|dd}|Sr)rrrrrSrTrr)rr,rlayerdrop_probs r9rzWav2Vec2Adapter.forwards 9 T%9%E IIm44M 00??M%//155 [ 5 5EY--//N= 5^dn%D%D %m 4 4 %//155 r8rrs@r9rrsG * * * * *r8rc$eZdZfdZdZxZS)rcttj|jd|jz|j|jd|_dS)Nr%r!)rr)rrrrradapter_kernel_sizeadapter_striderrs r9rzWav2Vec2AdapterLayer.__init__sU I  % ) )  &(     r8cr||}tj|d}|S)Nr!r)rrr%glurs r9rzWav2Vec2AdapterLayer.forward s3 -00  ))-Q)?? r8rrs@r9rrsG     r8rc4eZdZfdZdejfdZxZS)ryctt|j|_|j|_t j|j|_t j |j|j|_ t j |_ t j |j|j|_ dS)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)rrrw input_dimr hidden_dimrrnormrlinear_1ReLUact_fnlinear_2rs r9rz!Wav2Vec2AttnAdapterLayer.__init__s 0 ,L11  $/4>BB gii  $.$/BB r8r,c||}||}||}||}|Sr)rrrrrs r9rz Wav2Vec2AttnAdapterLayer.forward"sL -00  m44  M22  m44 r8)r0r1r2rr4r5rrrs@r9ryrys[ C C C C CU%6r8ryceZdZdZeZdZdZdZdZ dZ dZ dde e jefdeefd Z dd ed e jfd Zd ZdZddefdZdS)Wav2Vec2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. wav2vec2rTc t|trL|j|jd|j_d|j_dSt|t ro|jjj dd|jj j tj|jdSt|t"rtj |jjddt'jd|jjd|jjzz ztj|jj ddSt|t0r}t'jd|jjz }tj|jj| |tj|jj | |dSt|tjrT|jj d|jj|j |j j dSdSt|tjtjfr?|j j |jj d dSt|tj!rtj"|j|j [t'j|j#|j|jdzz }tj|j | |dSdSdS) zInitialize the weightsTrr!)rstdrr%)abNr)$r[Wav2Vec2ForPreTraining project_hidreset_parameters project_q_is_hf_initializedrrrdatanormal_rzero_rinituniform_rrrmathsqrtr in_channels constant_rr in_featuresrrinitializer_rangerrfill_rkaiming_normal_r)rmoduleks r9 _init_weightsz%Wav2Vec2PreTrainedModel._init_weights9s f4 5 5" 9   / / 1 1 1   - - / / /48F  126F  / / /  = > > 9   % * 2 2 2 C C C   # ( . . 0 0 0 G  V/ 0 0 0 0 0  ? @ @ 9 GOO " !v{'>q'AFKD['["\]]]     G  fk. 2 2 2 2 2  9 : : 9 !f/;;<._conv_out_lengthis&9\K7wWWWZ[[ [r8r!)rrziprrrZrr)rrirr rrrMs r9 _get_feat_extract_output_lengthsz8Wav2Vec2PreTrainedModel._get_feat_extract_output_lengths`s2=1Ddk--+  \ \ \ $'t{'> @W#X#X Q Q K,,]KPPMM  _4;9:: _ _ 0 04;C] ^ ^ r8feature_vector_lengthr=c |ddddf}|||}|tj}|jd}tj||f|j|j}d|tj |jd|j|dz f<| dgd dg }|S)NrJrrr)rPdevicer!)r) cumsumrrFr4longr:r[rPrr^flipr\)rrr=rnon_padded_lengthsoutput_lengthsrhs r9"_get_feature_vector_attention_maskz:Wav2Vec2PreTrainedModel._get_feature_vector_attention_maskws ,22r2::111b5A>>?Q_j>kk'**5:66#)!,  . /~7KTbTi   uv ^%9!%<^EZ[[[]kno]opq',,bT2299"==BBB4HHMMOOr8c|jjt|jdi}|D]N\}}t |t r4|D]\}}||d||g< Ot |tr9|j D]\}}||dd|g< |S)NzF has no adapter layers. Make sure to define `config.adapter_attn_dim`.r?lm_head) rrwrRr named_modulesr[rynamed_parametersjoinWav2Vec2ForCTCr)radapter_weightsrr param_namers r9 _get_adaptersz%Wav2Vec2PreTrainedModel._get_adapterss ; ' /vvvww w ..00 J JLD&&":;; J)/)@)@)B)BJJ%JDIOCHHdJ-?$@$@AA dN + + E#|<<>> E E e?D)T): ; ;<<r8c|D],}t|tr||-t|tr||jdSdS)zc (Re-)initialize attention adapter layers and lm head for adapter-only fine-tuning N)modulesr[ryrrr)rrs r9init_adapter_layersz+Wav2Vec2PreTrainedModel.init_adapter_layerss} llnn + +F&":;; +""6*** dN + + -   t| , , , , , - -r8 target_langc 6|jjtd|d||jkr"|s td|ddS|dd}|dd}|d d}|d d}|d d}|d d} |d d} |dd} |dtrdnd} | -tj dt| td| } |jj } d}| dur|t |} t| |||||| | | }t|}n9#t $r| rYn*t"$r| rt!d| d| d|dYnwxYw|t$ |} t| |||||| | | }t&rddini}t)j|fddi|}n1#t $rt"$rt!d| d| d|dwxYw|t/|t/z }t/t/|z }t3|dkr)td|dd|dt3|dkr)td|dd|d|djd}||jjkr=t;j|jj||j |j! |_"||j_fd!|#D}|$|d"||_dS)#a Load a language adapter model from a pre-trained adapter model. Parameters: target_lang (`str`): Has to be a language id of an existing adapter weight. Adapter weights are stored in the format adapter..safetensors or adapter..bin force_load (`bool`, defaults to `True`): Whether the weights shall be loaded even if `target_lang` matches `self.target_lang`. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (i.e., do not try to download the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. mirror (`str`, *optional*): Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. Activate the special ["offline-mode"](https://huggingface.co/transformers/installation.html#offline-mode) to use this method in a firewalled environment. Examples: ```python >>> from transformers import Wav2Vec2ForCTC, AutoProcessor >>> ckpt = "facebook/mms-1b-all" >>> processor = AutoProcessor.from_pretrained(ckpt) >>> model = Wav2Vec2ForCTC.from_pretrained(ckpt, target_lang="eng") >>> # set specific language >>> processor.tokenizer.set_target_lang("spa") >>> model.load_adapter("spa") ``` NzCannot load_adapter for - if `config.adapter_attn_dim` is not defined.z#Adapter weights are already set to r? cache_dirforce_downloadFresume_downloadproxieslocal_files_onlytokenuse_auth_tokenrevisionuse_safetensorszrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.zV`token` and `use_auth_token` are both specified. Please set only the argument `token`.)filenamer)r*r+r,r-r/r(zCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named weights_onlyT map_locationcpurzThe adapter weights z has unexpected keys: z, z has missing keys: zlm_head.weightrrPcNi|]!\}}|||"Sr7)rF)rLrrrs r9 z8Wav2Vec2PreTrainedModel.load_adapter..as0QQQdaaoa011QQQr8)strict)%rrwrRr%rDwarningpoprrrr _name_or_pathWAV2VEC2_ADAPTER_SAFE_FILEformatrsafe_load_fileEnvironmentError ExceptionWAV2VEC2_ADAPTER_PT_FILErr4loadr!setkeysr_rr: vocab_sizerrrrrPritemsload_state_dict)rr% force_loadr:r(r)r*r+r,r-r.r/r0model_path_or_id state_dictfilepath weight_pathweights_only_kwargunexpected_keys missing_keystarget_vocab_sizers @r9 load_adapterz$Wav2Vec2PreTrainedModel.load_adaptersJ~ ; ' /r rrrss s $* * *: * NNOOOO P P P FJJ{D11 $4e<< **%6==**Y--!::&8%@@ 7D))$4d;;::j$// **%6@X@Z@Z8e`eff  % ME     l#E;4  % ' '188EEH )$%#1$3#%5%'    ,K88 #   "    "*J5EJJ=MJJ?GJJJ   /66{CCH )$%#1$3#%5%'    @c%jnd%;%;hj""Z!&) $      &F1AFF9IFF;CFFF ,,..joo//0037K7K7M7M3N3NN?//1122S9J9J5K5KK   ! # #tKttW[W`W`apWqWqtttuu u    " "nKnnTXT]T]^jTkTknnnoo o''78>qA  6 6 69 .0A$+]a]gDL&7DK "RQQQj>N>N>P>PQQQ  Z666's$'F(( G7$GG>9H88.I&r)T)r0r1r2r3r" config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_2_supports_sdparrr4 LongTensorrCrr\rrr!r$r]rQr7r8r9rr,s  "L"$O&*#!N%9%9%9PZ^"5#3S#89HPQU0Y]%(:?:J( - - -}'}'}'}'}'}'}'}'r8ra Wav2Vec2 was proposed in [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://arxiv.org/abs/2006.11477) by Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving etc.). This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Wav2Vec2Config`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`Wav2Vec2Processor.__call__`] for details. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) `attention_mask` should only be passed if the corresponding processor has `config.return_attention_mask == True`. For all models whose processor has `config.return_attention_mask == False`, such as [wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base-960h), `attention_mask` should **not** be passed to avoid degraded performance when doing batched inference. For such models `input_values` should simply be padded with 0 and passed without `attention_mask`. Be aware that these models also yield slightly different results depending on whether `input_values` is padded or not. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare Wav2Vec2 Model transformer outputting raw hidden-states without any specific head on top.cteZdZdeffd ZdZdZ ddejde ejde ej fd Z e e eeeed e  dd e ejde ejde ejd e ede ede edeeeffdZxZS) Wav2Vec2Modelrct|||_t||_t ||_|jdks |jdkrBtj tj |j |_|jrt#||_nt'||_|jrt+|nd|_|dS)Nr)rrrrfeature_extractorrfeature_projectionmask_time_probmask_feature_probrrr4r4rrmasked_spec_embeddo_stable_layer_normrencoderr}rradapter post_initrs r9rzWav2Vec2Model.__init__s     !7!?!?";F"C"C  3 & &&*BS*H*H%'\%,v?Q2R2R2[2[2]2]%^%^D "  & 39&AADLL*622DL282DNv...$  r8cbtjdt|dSz Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nrrrfreeze_feature_encoderrs r9freeze_feature_extractorz&Wav2Vec2Model.freeze_feature_extractor;   Q    ##%%%%%r8c8|jdS Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)r\rrjs r9riz$Wav2Vec2Model.freeze_feature_encoders 1133333r8Nr,rsr=ct|jdds|S|\}}}|#|j|j||<n|jjdkr|jr|t||f|jj|jj ||jj }tj ||j tj}|j|j||<|jjdkr|jrt||f|jj|jj|jj}tj ||j tj}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779). apply_spec_augmentTNr)r;r<r=r>r5)r;r<r>rJ)rxrrur`rFrPr^rrpmask_time_lengthmask_time_min_masksr4rrr\r_mask_feature_lengthmask_feature_min_masksr)rr,rsr=rhrHrmask_feature_indicess r9_mask_hidden_statesz!Wav2Vec2Model._mask_hidden_statesst{$8$?? ! 4A3E3E3G3G0 O[  (/3/E/H/HI\/]/]M+ , , [ '! + + + 5_-+4 K8-+9 !!! !& ->}G[chcm n n n /3/E/H/HI\/]/]M+ , ; (1 , , ,#8[)+7 K;+< $$$ $)<0D]Mainis#t#t#t #74#@#G#GO]_#`#` 23M. /r8audio checkpoint output_typerRmodalityexpected_outputrrrrr?c:||n |jj}||n |jj}||n |jj}||}|dd}|#||jd|d}||\}}| |||}| |||||} | d}|j | |}|s||f| ddzSt||| j | jS) Nr!r%Fr)rsr=r=rrrr)rextract_featuresr,r-)rrruse_return_dictr\rrr:r]rwrbrcrr,r-) rrr=rsrrrrr,encoder_outputss r9rzWav2Vec2Model.forwards|"2C1N--TXT_Tq$8$D $+Jj &1%uEN+/*A*ABR*S*S' '00 ->~1  ,, )/!5# '  (* < # LL77M K!#34qrr7JJ J&+-)7&1     r8rNNNNN)r0r1r2r"rrkrir4r5rrXrwrWAV_2_VEC_2_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPEr4r\rrrrrs@r9rZrZs ~( & & &444:>59 ,,(,$E$56,!!12 ,,,,\+*+GHH&+$. 269=,0/3&*2 2 u|,2 !.2 $E$56 2 $D> 2 'tn 2 d^2  u-- .2 2 2 IH2 2 2 2 2 r8rZz5Wav2Vec2 Model with a quantizer and `VQ` head on top.ceZdZdeffd ZdefdZdZdZe dde j d e j d e j defd Z e eeee  ddee jdee jdee jdee jdeedeedeedeeeffdZxZS)rrct|t||_t j|j|_t||_ t j |j |j |_ t j |j|j |_|dSr)rrrZrrrfeat_quantizer_dropoutdropout_featuresr quantizerrrproj_codevector_dimrrrrdrs r9rzWav2Vec2ForPreTraining.__init__:s    %f-- " 6+H I I6v>>9V%79STT6#8&:TUU r8rc||j_dS)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rr)rrs r9set_gumbel_temperaturez-Wav2Vec2ForPreTraining.set_gumbel_temperatureGs&1"""r8cbtjdt|dSrfrhrjs r9rkz/Wav2Vec2ForPreTraining.freeze_feature_extractorMrlr8cB|jjdSrnrr\rrjs r9riz-Wav2Vec2ForPreTraining.freeze_feature_encoderY! '::<<<<>> import torch >>> from transformers import AutoFeatureExtractor, Wav2Vec2ForPreTraining >>> from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices >>> from datasets import load_dataset >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base") >>> model = Wav2Vec2ForPreTraining.from_pretrained("facebook/wav2vec2-base") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values # Batch size 1 >>> # compute masked indices >>> batch_size, raw_sequence_length = input_values.shape >>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item() >>> mask_time_indices = _compute_mask_indices( ... shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2 ... ) >>> sampled_negative_indices = _sample_negative_indices( ... features_shape=(batch_size, sequence_length), ... num_negatives=model.config.num_negatives, ... mask_time_indices=mask_time_indices, ... ) >>> mask_time_indices = torch.tensor(data=mask_time_indices, device=input_values.device, dtype=torch.long) >>> sampled_negative_indices = torch.tensor( ... data=sampled_negative_indices, device=input_values.device, dtype=torch.long ... ) >>> with torch.no_grad(): ... outputs = model(input_values, mask_time_indices=mask_time_indices) >>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states) >>> cosine_sim = torch.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, dim=-1) >>> # show that cosine similarity is much higher than random >>> cosine_sim[mask_time_indices.to(torch.bool)].mean() > 0.5 tensor(True) >>> # for contrastive loss training model should be put into train mode >>> model = model.train() >>> loss = model( ... input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices ... ).loss ```N)r=rrrsrrr!Fr)rsrJr%r z-infirW) reduction)r(r)r*r+r,r-r.r/)$rrrFr4r\rrrrr:rrrrPrrpermutercontrastive_logits_temperatureallanyr3rrfrurrr% cross_entropyrrrWdiversity_loss_weightr'r,r-)rrr=rsryrrrrstransformer_featuresrquantized_featuresr+r(r.r/rhrHrnegative_quantized_featuresr neg_is_postargetnum_codevectorss r9rzWav2Vec2ForPreTraining.forwardusJ&1%uEN59NN 0A5C5 5 110224>3H3NOO!^^,>??3777. # /7I7O 4J +=*A*A"k*R*R(--//44R88+ '+F*J*JOR++gaAq!! ( 44"47++$ : F-0KKPPQSTTJ~~ 7).vqrr :&%%a++33B AGGF,11333t;FFq!LLTTVVF!}::6<<>>6]b:cc "kCdkFggO.1FF/Y]n]r]r]t]ttN$dk&G.&XXD c24FH]^ahijikikalll(*<>STW^_`_a_aWbb b+1'9"7!/)-)    r8)r)NNNNNN)r0r1r2r"rrCrrkrirr4r5rrrr r'rrr4 BoolTensorr\rrrrrs@r9rr8s ~      1#1111 & & &===  * ,"- \(+*+GHH+GVefff268 ^ 'tn^ d^^  u22 3^ ^ ^ gfIH^ ^ ^ ^ ^ r8rz6Wav2Vec2 Model with a `language modeling` head on top.ceZdZfdZee d dejdeej dee dee dee deej d e e effd ZxZS) Wav2Vec2ForMaskedLMcRt|tjdtt ||_tj|j |_ tj |j |j |_|dS)NzSThe class `Wav2Vec2ForMaskedLM` is deprecated. Please use `Wav2Vec2ForCTC` instead.)rrrrrrZrrr final_dropoutrrrrErrdrs r9rzWav2Vec2ForMaskedLM.__init__s     acp   &f-- z&"677 y!3V5FGG  r8Nrr=rrrlabelsr?c||n |jj}|||||}|d}||}||} |s| f|ddz} | St | |j|jS)N)rrrrr%)rr,r-)rrrrrrr,r-) rrr=rrrrrsr,routputs r9rzWav2Vec2ForMaskedLM.forward(s&1% q 'tn q d^ q&q un$ %qqqIHqqqqqr8rzfWav2Vec2 Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).a. target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`Wav2Vec2ForCTC`] with adapters. Uses 'eng' by default. c>eZdZddeeffd ZdZdZdZdZ e e e e eeee dd eejd eejd eed eed eedeejdeeeffdZxZS)rNr%ct|t||_t j|j|_||_|j td|j dt|dr|j r|jn|j}t j||j |_|dS)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.r)rrrZrrrrrr%rErRrrrrrrrrd)rrr%rrs r9rzWav2Vec2ForCTC.__init__Qs    %f-- z&"677 &   $HHHH *1)G)G vFL^ vF % %djdv y!3V5FGG  r8c|j}|)t|jddtd|d|2t|jddtddS|||ddSdS)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. NrwzCannot pass `target_lang`: r'z)By default `target_lang` is set to 'eng'.T)rH)r%rxrrRrDinforQ)rr%s r9 tie_weightszWav2Vec2ForCTC.tie_weightshs&  "wt{||jjkrt d|jj||||||}|d}||}||} d} |Z||ntj |tj }| | d tj } |dk} | d} || }tj| dtjdd}tjjd 5tj||| | |jj|jj|jj } dddn #1swxYwY|s| f|t6dz}| | f|zn|St9| | |j|j S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rrrOrJ)rrPr!F)enabled)blankr zero_infinityr(rr,r-)rrrDrErRrrrr4 ones_likerrrWrF masked_selectrr% log_softmaxrArbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONrr,r-)rrr=rrrrrsr,rr(ri labels_masktarget_lengthsflattened_targets log_probsrs r9rzWav2Vec2ForCTC.forwards0&1% D 'tn D d^ D &D  un$ %D D D IHD D D D D r8rz Wav2Vec2 Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. c&eZdZfdZdZdZdZeee e e e de e ddeejd eejd eed eed eed eejdeee ffdZxZS)!Wav2Vec2ForSequenceClassificationct|t|dr|jrt dt ||_|jdz}|jr.tj tj ||z |_ tj|j|j|_tj|j|j|_|dS)Nrz_Sequence classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r!)rrrrrRrZrruse_weighted_layer_sumrrr4ra layer_weightsrrclassifier_proj_size projector num_labels classifierrdrr num_layersrs r9rz*Wav2Vec2ForSequenceClassification.__init__s    6= ) ) f.@ q &f-- -1  ( S!#ej.D.Dz.Q!R!RD 6#5v7RSS)F$?ARSS r8cbtjdt|dSrfrhrjs r9rkz:Wav2Vec2ForSequenceClassification.freeze_feature_extractor rlr8cB|jjdSrnrrjs r9riz8Wav2Vec2ForSequenceClassification.freeze_feature_encoder rr8cL|jD] }d|_ dSrrrs r9rz3Wav2Vec2ForSequenceClassification.freeze_base_model rr8rx)rzr{rRr|r}rNrr=rrrrr?c||n |jj}|jjrdn|}||||||}|jjrx|t}t j|d}tj |j d} || dddz d}n|d}| |}||d} nh||jd|} d|| <| d| d ddz } || } d} |Kt%}|| d|jj| d} |s| f|tdz}| | f|zn|St)| | |j|j S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrr!rrJrrr)rrrrrr4stackrr%r&rrrWrrrr:rrrrr,r-)rrr=rrrrrsr, norm_weights pooled_output padding_maskrr(loss_fctrs r9rz)Wav2Vec2ForSequenceClassification.forward s&2&1%->r1a-H-HHMMRSMTTMM#AJM}55  !)..1.55MMBB=CVWXCY[ijjL+.M<- ()--!-44|7G7GA7G7N7N7S7STVXY7Z7ZZM//  '))H8FKKDK,BCCV[[QS__UUD FY)F)G)G!HHF)-)9TGf$$v E'!/)     r8r)r0r1r2rrkrirrrr_SEQ_CLASS_CHECKPOINTrr_SEQ_CLASS_EXPECTED_OUTPUT_SEQ_CLASS_EXPECTED_LOSSrr4r4r\rrrrrs@r9rrsJ" & & &===(((+*+GHH(,$2. 26,0/3&*)-; ; u|,; !.; $D> ; 'tn ; d^ ; &;  u.. /; ; ; IH; ; ; ; ; r8rzd Wav2Vec2 Model with a frame classification head on top for tasks like Speaker Diarization. c$eZdZfdZdZdZdZeee e e e de  ddeejd eejd eejd eed eed eedeee ffdZxZS)#Wav2Vec2ForAudioFrameClassificationct|t|dr|jrt dt ||_|jdz}|jr.tj tj ||z |_ tj|j|j|_|j|_|dS)NrzbAudio frame classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r!)rrrrrRrZrrrrrr4rarrrrr init_weightsrs r9rz,Wav2Vec2ForAudioFrameClassification.__init__i s    6= ) ) f.@ t &f-- -1  ( S!#ej.D.Dz.Q!R!RD )F$68IJJ + r8cbtjdt|dSrrhrjs r9rkz->r1a-H-HHMMRSMTTMM#AJM//  '))H8FKKDO<t>t>tuuD Y)F)G)G!HHFM$!/)     r8r)r0r1r2rrkrirrrr_FRAME_CLASS_CHECKPOINTrr_FRAME_EXPECTED_OUTPUTrr4r4r\rrrrrs@r9rrb sG & & &===(((+*+GHH*)$. 26)-,0/3&*3 3 u|,3 !.3 & 3 $D> 3 'tn 3 d^3  u++ ,3 3 3 IH3 3 3 3 3 r8rc&eZdZdfd ZdZxZS) AMSoftmaxLoss>@皙?c tt|||_||_||_t jtj ||d|_ t j |_ dS)NT)r) rrrscalemarginrrrr4randnrrr()rrrrrrs r9rzAMSoftmaxLoss.__init__ sk mT""++---  $l5;y*#E#EUYZZZ ')) r8c|}tj|jd}tj|d}t j||}||jz }tj||j }|j t j | ||z}| ||}|S)Nrrr!)rrr% normalizerr4mmrone_hotrrrr\r() rr,rr cos_thetapsionehotrr(s r9rzAMSoftmaxLoss.forward s!!((!(<< // 1/EE H]F33 $+%&&vt??ek&++--iHHHyy(( r8)rrrrs@r9rr sL******       r8rcDeZdZdfd ZdejdejfdZxZS) TDNNLayerrct|dkr|j|dz n |j||_|j||_|j||_|j||_tj |j|jz|j|_ tj |_ dS)Nrr!)rrtdnn_dimrr tdnn_kernelr tdnn_dilationdilationrrkernelrrrs r9rzTDNNLayer.__init__ s . s#QQQy++QQQr8rJr%)rrrZrrrrrr4rarrrr rrZr_rtdnnxvector_output_dimr\rrr objectiver)rrr tdnn_layersrs ` r9rzWav2Vec2ForXVector.__init__ s)    %f-- -1  ( S!#ej.D.Dz.Q!R!RD 6#5vq7IJJQQQQU3v;O;O5P5PQQQ M+.. !#6?2+>+BFD]!^!^)F$=v?XYY&v'@&BSTT r8cbtjdt|dSrrhrjs r9rkz+Wav2Vec2ForXVector.freeze_feature_extractor" rlr8cB|jjdSrnrrjs r9riz)Wav2Vec2ForXVector.freeze_feature_encoder. rr8cL|jD] }d|_ dSrrrs r9rz$Wav2Vec2ForXVector.freeze_base_model5 rr8ricDd}|jjD]}|||d}|S)z? Computes the output length of the TDNN layers c||z |zdzS)Nr!r7r s r9r zEWav2Vec2ForXVector._get_tdnn_output_lengths.._conv_out_lengthB s!;.69A= =r8r!)rr )rrir rs r9_get_tdnn_output_lengthsz+Wav2Vec2ForXVector._get_tdnn_output_lengths= sE  > > >  ;2 L LK,,]KKKMMr8rxryNrr=rrrrr?c>||n |jj}|jjrdn|}||||||}|jjrx|t}t j|d}tj |j d} || dddz d}n|d}| |}|jD] } | |}|-|d} |d} n|| d} || }g} g} t'|D]k\}}| ||d|fd| ||d|fdlt j| } t j| } t j| | gd}||}||}d}||||}|s||f|tdz}||f|zn|St3||||j|jS) rNTrr!rrJr)r(r embeddingsr,r-)rrrrrr4rrr%r&rrrWrrrrrr! enumeratercr#r\rrrr,r-)rrr=rrrrrsr,r tdnn_layer mean_features std_featuresfeat_extract_output_lengthstdnn_output_lengthsrlengthstatistic_poolingoutput_embeddingsrr(rs r9rzWav2Vec2ForXVector.forwardL s0&1%->r1a-H-HHMMRSMTTMM#AJM}55 ) 6 6J&J}55MM  !)..1.55M(,,,33LL*.*O*OP^PbPbghPbPiPi*j*j '"&"?"?@["\"\ ML&':;; J J 6$$]1gvg:%>%C%C%C%J%JKKK##M!WfW*$=$A$Aa$A$H$HIIII!K 66M ;|44L!I}l&CLLL 223DEE!233  >>&&11D F/07;X;Y;Y3ZZF)-)9TGf$$v E(!/)     r8r)r0r1r2rrkrirrr4rXrCr!rrr_XVECTOR_CHECKPOINTrr_XVECTOR_EXPECTED_OUTPUTrr4r\rrrrs@r9rr s~& & & &===((( eE I 'tn I d^ I &I  um# $I I I IHI I I I I r8rrr)pr3rr dataclassesrtypingrrrnumpyrSr4torch.utils.checkpointrtorch.nnr activationsr integrations.deepspeedr integrations.fsdpr modeling_outputsr rrrrrrmodeling_utilsr pytorch_utilsrrrrrrrrrrrrr configuration_wav2vec2r"rAr<safetensors.torchr#r>modeling_flash_attention_utilsr$ get_loggerr0rDrrrrrrrrrrrr-r.r'rCr3rXndarrayrprModulerrrrrrrrrr6rLrgrUrdrur}rrrrryrWAV_2_VEC_2_START_DOCSTRINGrrZrrrrrrr rr7r8r9rAsj  !!!!!!)))))))))) %%%%%%!!!!!!@@@@@@777777.-----@@@@@@                          322222,5>======KJJJJJJ  H % %!"#4&u9*<Q8 &7&7&7&7&7;&7&7 &7Z26 tt c?tttU-. t  t Z ttttpZ^!$!$!$*-!$BJ2:BV!$!$!$!$H29*60*****bi***Z29+++++RY+++\     5    1 1 1 1 1 1 1 1 [B[B[B[B[B [B[B[B~|9|9|9|9|9/|9|9|9~h1h1h1h1h1-h1h1h1X !0")0     29   F*****")***ZR R R R R biR R R jV V V V V RYV V V rI'I'I'I'I'BII'I'I'Xbi>29$ry2y'y'y'y'y'oy'y'y'x &# LhP P P P P +P P P fQSnoo\ \ \ \ \ 4\ \ po\ ~RTopp*q*q*q*q*q1*q*qqp*qZp  T T T T T ,T T   T n q q q q q (?q q q h  g g g g g *Ag g  g TBI. >  O O O O O 0O O  O O O r8