g4 dZddlZddlZddlmZmZmZddlZddl Z ddl m cm Z ddlZ ddl m Z ddlmZddlmZddlmZdd lmZdd lmZmZmZmZmZmZdd lmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%d dl&m'Z'e%j(e)Z*dZ+dZ,dZ-gdZ.dZ/dZ0dZ1ddgZ2dZ3dZ4 dXdee5e5fde6de5dee j7de5dej8f dZ9Gdd e j:Z;Gd!d"e j:Z<Gd#d$e j:Z=Gd%d&e j:Z>Gd'd(e j:Z?Gd)d*e j:Z@Gd+d,e@ZAGd-d.e j:ZBGd/d0e j:ZCGd1d2e j:ZDGd3d4e j:ZEGd5d6e j:ZFGd7d8e j:ZGGd9d:e j:ZHGd;de j:ZJGd?d@e j:ZKGdAdBeZLdCZMdDZNe"dEeMGdFdGeLZOe"dHeMGdIdJeLZPe"dKeMGdLdMeLZQe"dNeMGdOdPeLZRGdQdRe j:ZSGdSdTe j:ZTe"dUeMGdVdWeLZUdS)YzPyTorch WavLM model.N)OptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)BaseModelOutputCausalLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardis_peft_availablelogging) WavLMConfigrz1patrickvonplaten/wavlm-libri-clean-100h-base-plus)ri$izZ'mister quilter is the aposle of the middle classes and we are glad to welcome his gospel'gQ)@zmicrosoft/wavlm-base-plus-sdzmicrosoft/wavlm-base-plus-svg ףp= ?shape mask_prob mask_lengthattention_mask min_masksreturnc@|\}dkrtdkrtdddtjdfd}|9|dnfd t|D}tj |ft }g} |} | d kr|S|D]} || } tj tj | dz z | d } t| d krdz }n| d }tj| tj| | z tj |zg} | | tj| } tj| dddddf|| f} | || z} tj ddddf}tj||| f|| z}| |z} | dz krdz | | dz k<tj|| dd|S)af Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://arxiv.org/abs/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonrrrsequence_lengths d/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/wavlm/modeling_wavlm.pycompute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spanrs~i,6DwNOOoy99 [ (? : :-z)_compute_mask_indices..s 9 9 9!o 9 9 9r,dtyperF)replace) ValueErrornprandomranditemsumdetachtolistrangezerosboolchoicearangelen concatenateonesint32appendarray broadcast_toreshaper%put_along_axis)rrrrr batch_sizer+ input_lengths spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanr&r'spec_aug_mask_idxdummy_mask_idxoffsetsr(r)s `` ` @@r*_compute_mask_indicesrTLsP0#(JQABBB_$$ :^i : :'6 : : :   innQ$$&&G         $  % 2%%''..000 9 9 9 9uZ'8'8 9 9 9Hj/:$GGGM11/BBa%55 11,??I,, IlkAo6 7 7RW-    ! !Q & &-q0NN.q1NN (;o(MUWU] ^ ^ ^ao o p   !!"34444"455111aaa:&5H+(V,33J@SVa@abbi $$T4]3Gog 4G'UVV^^'+5G,g5/A"555GVYZGZ-!0CCDm%7B??? r,c&eZdZdfd ZdZxZS)WavLMNoLayerNormConvLayerrcZt|dkr|j|dz nd|_|j||_t j|j|j|j||j||j |_ t|j |_ dS)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__s r*r]z"WavLMNoLayerNormConvLayer.__init__s s: #:Q#>!#C#Caar,cJ|jdkr|ddddd|j f}|SNr)rros r*rqzWavLMSamePadLayer.forwardBs;   " ")!!!QQQ0F43F2F0F*FGMr,rsrxs@r*rr=sLKKKKKr,rc.eZdZdZfdZdZdZxZS)WavLMFeatureEncoderz.Construct the features from raw audio waveformctjdkr7tdgfdt jdz Dz}nDjdkr!fdt jD}nt djd tj||_ d |_ d |_ dS) Ngrouprrkc8g|]}t|dzS)rr)rVr0irjs r*r2z0WavLMFeatureEncoder.__init__..Ps>KKKFG)&1q5AAAKKKr,rlayerc2g|]}t|S)r)rzrs r*r2z0WavLMFeatureEncoder.__init__..Ts'vvv126AFFFvvvr,z`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) r\r]feat_extract_normrr>num_feat_extract_layersr6r ModuleList conv_layersgradient_checkpointing_requires_grad)rirjrrls ` r*r]zWavLMFeatureEncoder.__init__Ls   #w . .26AFFFGKKKKKPQWQorsQsKtKtKKKKK % 0 0vvvvPUV\VtPuPuvvvKKt1Ittt =55&+#"r,cP|D] }d|_ d|_dS)NF) parameters requires_gradrriparams r*_freeze_parametersz&WavLMFeatureEncoder._freeze_parameters]s4__&& ( (E"'E  #r,c|dddf}|jr|jrd|_|jD]>}|jr*|jr#|jr||j|}3||}?|S)NT)rtrainingrrr_gradient_checkpointing_func__call__)ri input_valuesrp conv_layers r*rqzWavLMFeatureEncoder.forwardbs$QQQW-    /4= /*.M '* : :J" :t'B :t} : $ A A'!!! !+ = 9 9 r,)rtrurv__doc__r]rrqrwrxs@r*rrIs\88#####"$$$ r,rceZdZfdZxZS)WavLMFeatureExtractorct|tjd|jjd|jjdjdtdS)Nz The class `zD` has been depreciated and will be removed in Transformers v5. Use `rz ` instead.)r\r]warningswarnrlrt __bases__ FutureWarningrirjrls r*r]zWavLMFeatureExtractor.__init__vsy      E$.1 E EN,Q/8 E E E       r,)rtrurvr]rwrxs@r*rrus8         r,rc$eZdZfdZdZxZS)WavLMFeatureProjectionc.ttj|jd|j|_tj|jd|j|_ tj |j |_ dS)Nr-eps) r\r]rr}r^layer_norm_epsr~Linearr projectionDropoutfeat_proj_dropoutdropoutrs r*r]zWavLMFeatureProjection.__init__sn ,vr':@UVVV)FOB$79KLLz&":;; r,c||}||}||}||fSrn)r~rr)rirpnorm_hidden_statess r*rqzWavLMFeatureProjection.forwardsC!__];;(:;;  ]33 000r,rsrxs@r*rrsG<<<<< 1111111r,rceZdZdZ ddededed ed ed ef fd Z ddej de ej de ej dede ej e ej e e ej ff dZ dej deejejfdej dedej ej ff dZdededej fdZdej dej fdZxZS)WavLMAttentionz=Multi-headed attention from 'Attention Is All You Need' paper@ T embed_dim num_headsr num_buckets max_distancehas_relative_position_biasct||_||_||_||z|_|j|z|jkrt d|jd|d|jdz|_tj |||_ tj |||_ tj |||_ tj |||_ ||_||_tjt#jd|jdd|_tj |jd|_|r&tj|j|j|_dSdS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gr)r\r]rrrhead_dimr6scalingrrk_projv_projq_projout_projrr ParametertorchrEgru_rel_pos_constgru_rel_pos_linear Embeddingrel_attn_embed)rirrrrrrrls r*r]zWavLMAttention.__init__sa "" !Y. MI %$. 8 83dn33%.333 }d* i 955 i 955 i 955  )Y77 &(!#ejDNAq.Q.Q!R!R"$)DM1"="= % Q"$,t/?"P"PD    Q Qr,NFrrpr position_biasoutput_attentionsr cF|\}}}|^|||}|d|ddd||jz||}||jdd|jdfz} | dddd} || } | | jdddz d} tj |  dd\} } | | |j zd z zd z} | ||jzdd|z}|d||f}|||||\}}|||fS) z'Attention layer with relative attentionNrrr-rr)rr?g@)size compute_bias unsqueezerepeatviewrrpermuterr;rsigmoidchunkrtorch_multi_head_self_attention)rirprrrindexbsztgt_lenr1gated_hidden_statesrelative_position_projgate_agate_b gate_outputgated_position_bias attn_output attn_weightss r*rqzWavLMAttention.forwards(,,..Wa   --gw??M''**11#q!Q??DDS4>EY[bdkll  ,001DSbS1IT^]_L`1`aa199!Q1EE"&!8!89L!M!M!7!>DDQBDOO)? ?# EFL *..sT^/CRKKm[166GW7MNN$($H$H >+>@Q% % ! \L-77r,r c|ddx}x}}||dnd}dx} } d} tj||||j|jt jdgt j|j j |j j |j j f| | | |j |jj|jj |j|||d|j j|j j|j j\} } | dd} | E| dddf| jdd|jfz| jddz} | | fS)zCsimple wrapper around torch's multi_head_attention_forward functionrrNFT)use_separate_proj_weight q_proj_weight k_proj_weight v_proj_weight)rneFmulti_head_attention_forwardrrremptycatrr[rrrrrrrIr)rirprr rquerykeyvaluekey_padding_maskbias_kbias_v add_zero_attnr rs r*rz.WavLMAttention.torch_multi_head_self_attentionsx,55a;;;;e3A3M>,,Q///SW %&$B    N N K   It{')94;;KL M M    L M M  M   %)+,+,+,+% % % ! \2"++Aq11  #(40=="2A2&$.)::\=OPQPRPR=SSLL((r, query_length key_lengthctj|tjdddf}tj|tjdddf}||z }||}||jjj}||}|gd}|S)Nr3)rrr) rrBlong_relative_positions_buckettorrdevicer)rir r!context_positionmemory_positionrelative_positionrelative_position_bucketvaluess r*rzWavLMAttention.compute_biass < EJGGG4P,zDDDT111WM+.>>#'#B#BCT#U#U #;#>#>t?R?Y?`#a#a $$%=>> ** r,relative_positionsc|jdz}|dktj|z}tj|}|dz}||k}tj||z }|tj|j|z z }|||z z}||ztj}tj |tj ||dz }|tj |||z }|Sr) rr%rr#abslogfloatmathrmin full_likewhere)rir,rrelative_buckets max_exactis_smallrelative_positions_if_largerelative_position_if_larges r*r$z)WavLMAttention._relative_positions_bucket!s&!+ .266uzBB[P"Y'9::1$ % 1&+i0B0H0H0J0JY0V&W&W#&ADHTM^ajMjDkDk&k#&A[S\E\&]#&/2M&M%Q%QRWR\%]%]"%*Y &8RT_bcTc(d(d& & " EK2DF`aaar,)rrrTNNFr)rtrurvrr$r0r@r]rTensorrrrq FloatTensorr LongTensor BoolTensorrrr$rwrxs@r*rrsGG +/"Q"Q"Q"Q "Q  "Q  "Q%)"Q"Q"Q"Q"Q"QN2604"' '8'8|'8!.'8  - '8  '8 u|Xel3XeEL>Q5RR S'8'8'8'8R5)(5)e.0@@A5)#. 5)  5)  U. / 5)5)5)5)n#%BS U=N SXSd        r,rc$eZdZfdZdZxZS)WavLMFeedForwardcttj|j|_tj|j|j|_ t|j trt|j |_n |j |_tj|j|j|_tj|j|_dSrn)r\r]rractivation_dropoutintermediate_dropoutrrintermediate_sizeintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrs r*r]zWavLMFeedForward.__init__8s $&Jv/H$I$I!"$)F,>@X"Y"Y f' - - 9'-f.?'@D $ $'-'8D $If&>@RSS j)>??r,c||}||}||}||}||}|Srn)rErIrCrJrLros r*rqzWavLMFeedForward.forwardEsg// >> 00?? 11-@@ ))-88 ++M:: r,rsrxs@r*r@r@7sL @ @ @ @ @r,r@c2eZdZd dedeffd Zd dZxZS) WavLMEncoderLayerTrjrctt|j|j|j|j|j||_tj |j |_ tj |j|j|_t!||_tj |j|j|_dSN)rrrrrrrr\r]rrnum_attention_headsattention_dropoutrmax_bucket_distance attentionrrrKrr}rr~r@ feed_forwardfinal_layer_normrirjrrls r*r]zWavLMEncoderLayer.__init__P '(0,*3'A    z&"788 ,v'9v?TUUU,V44 " V-?VEZ [ [ [r,NFrc|}||||||\}}}||}||z}||}|||z}||}||f}|r||fz }|S)Nrrrr)rVrr~rWrX) rirprrrr attn_residualroutputss r*rqzWavLMEncoderLayer.forward_s% 59^^ )'/ 6D6 6 2 |] ]33 % 5  66 %(9(9-(H(HH --m<<  -0  '   &Gr,Tr:rtrurvrr@r]rqrwrxs@r*rOrOOsm \ \{ \ \ \ \ \ \ \r,rOc2eZdZddedeffd Zd dZxZS) WavLMEncoderLayerStableLayerNormTrjrctt|j|j|j|j|j||_tj |j |_ tj |j|j|_t!||_tj |j|j|_dSrQrRrYs r*r]z)WavLMEncoderLayerStableLayerNorm.__init__yrZr,NFc|}||}|||||\}}}||}||z}||||z}||f}|r||fz }|S)N)rrr)r~rVrrWrX)rirprrrr]rr^s r*rqz(WavLMEncoderLayerStableLayerNorm.forwards%  66 59^^ )'/ 6D6 6 2 |]  ]33 % 5 %(9(9$:O:OP]:^:^(_(__  -0  '   &Gr,r_)NNFr`rxs@r*rbrbxsm \ \{ \ \ \ \ \ \ \r,rbc.eZdZfdZ ddZxZS) WavLMEncoderct|_t|_t jjj|_ t j j |_ t j fdtjD|_d|_dS)Nrc:g|]}t|dkSr)r)rOrs r*r2z)WavLMEncoder.__init__..s, u u uPQ v16 K K K u u ur,Fr\r]rjrpos_conv_embedrr}rrr~rrKrrr>num_hidden_layerslayersrrs `r*r]zWavLMEncoder.__init__s  :6BB,v'9v?TUUUz&"788 m u u u uUZ[a[sUtUt u u u   ',###r,NFTc|rdnd}|rdnd}|d||<||}||z}||}||}tpt |} d} t |jD]\} } |r||fz}tjg} |j o| dko| |j j k}|r| rJ|j r&|j r| | j||| |}n| ||| || }|dd\}} |rd}|r ||dfz}|r||fz}|std|||fDSt!|||S) Nr/rrr\rNNNc3K|]}||V dSrnr/r0vs r* z'WavLMEncoder.forward..(mmq_`_l_l_l_l_lmmr,last_hidden_staterp attentions)rkr~rr r enumeratermrr9rrj layerdroprrrtupler rirprroutput_hidden_states return_dictall_hidden_statesall_self_attentionsposition_embeddings synced_gpusrrrdropout_probabilityskip_the_layer layer_outputss r*rqzWavLMEncoder.forwards#7@BBD$5?bb4  %-0M>/ *"11-@@%(;;  66  ]33 022R6LT6R6R  !$+..! P! PHAu# I$58H$H!#(*R.. !]fq1uf:MPTP[Pe:eN! A[ A.4=$($E$E%&%) %%MM%*E%'5&3*; %%%M0=RaR/@, } 3 2   P&9]1=M.s=   1UVZ[U[]]]   r,Frjrs `r*r]z$WavLMEncoderStableLayerNorm.__init__s  :6BB,v'9v?TUUUz&"788 m    v788      ',###r,NFTc|rdnd}|rdnd}|d||<||}||z}||}tpt|} d} t |jD]\} } |r||fz}t jg} |jo| dko| |j j k}|r| rI|j r&|jr| | j ||| |}n| |||| }|dd\}} |rd}|r ||dfz}||}|r||fz}|std|||fDSt!|||S)Nr/r)rrrrroc3K|]}||V dSrnr/rqs r*rsz6WavLMEncoderStableLayerNorm.forward..<rtr,ru)rkrr r rxrmrr9rrjryrrrr~rzr r{s r*rqz#WavLMEncoderStableLayerNorm.forwards#7@BBD$5?bb4  %-.M>/ *"11-@@%(;;  ]33 022R6LT6R6R  !$+.. P PHAu# I$58H$H!#(*R.. !]fq1uf:MPTP[Pe:eN! A[ A.4=$($E$E%&%) %%MM%*E%'5*;&3 %%%M 0=RaR/@, } 3 2   P&9]1=MeZdZdZfdZedZdZxZS)WavLMGumbelVectorQuantizerz Vector quantization using gumbel softmax. See [CATEGORICAL REPARAMETERIZATION WITH GUMBEL-SOFTMAX](https://arxiv.org/pdf/1611.01144.pdf) for more information. ct|j|_|j|_|j|jzdkr td|jd|jdtj tj d|j|jz|j|jz|_ tj |jd|j|jz|_d|_dS)Nrz`config.codevector_dim z5 must be divisible by `config.num_codevector_groups` z for concatenation.rr-r)r\r]num_codevector_groupsrnum_codevectors_per_groupnum_varscodevector_dimr6rrrr< codevectorsrr^ weight_proj temperaturers r*r]z#WavLMGumbelVectorQuantizer.__init__Hs  68  4? 2a 7 7%&*?%%6:o%%% <  a4=!@&BW[_[jBj k k  9V_R%8$/DM:YZZr,c |d}tjtj|tj|dzzd }|S)NrrgHz>r-)meanrexpr;r/)probsmarginal_probs perplexitys r*_compute_perplexityz.WavLMGumbelVectorQuantizer._compute_perplexity]s^**Y .59^VZEZ;[;[*[ac d d ddeeiikk r,c|j\}}}||}|||z|jzd}|jrt j||j d}| |}tj |||z|jdd}| |}n|d}|j|jd|ddd}|||z|jd}| |}|||zd}|d|jz} | ||z|j|jd} | d||d} | |fS)Nr-T)tauhardrrrr)rrrrrr functionalgumbel_softmaxr0rtype_asrsoftmaxrargmax new_zerosscatter_rrrr;) rirprLr)rcodevector_probscodevector_soft_distrcodevector_idxcodevectors_per_grouprs r*rqz"WavLMGumbelVectorQuantizer.forwardcs3@3F0 O[((77 %**:+G$/+Y[]^^ = D!};;M >AQ Q+00o1Mt`d`moqrr !oob))..z?BOO J&&r,) rtrurvrr] staticmethodrrqrwrxs@r*rrBsl *\ "'"'"'"'"'"'"'r,rc$eZdZfdZdZxZS) WavLMAdapterctjjkrCt jjj|_t jj|_ndx|_|_t j fdtj D|_ j |_ dS)Nc36K|]}tVdSrn)WavLMAdapterLayer)r0r1rjs r*rsz(WavLMAdapter.__init__..s,#h#h!$5f$=$=#h#h#h#h#h#hr,)r\r]output_hidden_sizerrrprojr}proj_layer_normrr>num_adapter_layersrmryrs `r*r]zWavLMAdapter.__init__s   $(: : : &"4f6OPPDI#%<0I#J#JD /3 3DI,m#h#h#h#huVMfGgGg#h#h#hhh )r,cX|j1|j*||}||}|dd}|jD]=}tj}|jr ||jkr ||}>|dd}|Sr)rrrrmr7r8rry)rirprlayerdrop_probs r*rqzWavLMAdapter.forwards 9 T%9%E IIm44M 00??M%//155 [ 5 5EY--//N= 5^dn%D%D %m 4 4 %//155 r,rsrxs@r*rrsG * * * * *r,rc$eZdZfdZdZxZS)rcttj|jd|jz|j|jd|_dS)Nrr)rZr)r\r]rraradapter_kernel_sizeadapter_striderers r*r]zWavLMAdapterLayer.__init__sU I  % ) )  &(     r,cr||}tj|d}|S)Nrr)rerrgluros r*rqzWavLMAdapterLayer.forwards3 -00  ))-Q)?? r,rsrxs@r*rrsG     r,rceZdZdZeZdZdZdZdZ d de e j e fdeefd Z d d e d e j fd ZdS)WavLMPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. wavlmrTc \t|tro|jjjdd|jjjtj |j dSt|trtj |j jddtjd|j jd|j jzz ztj |j jddSt|t&r}tjd|jjz }tj |jj| |tj |jj| |dSt|tjrT|jjd|jj|j |jjdSdSt|tjtjfr?|jj|jjddSt|tjrtj |j|j[tj|j|j|jdzz }tj |j| |dSdSdS) zInitialize the weightsrr)rstdrr)abNr)rFrrrdatanormal_r[zero_rinituniform_rrrer1sqrtrY in_channels constant_rr in_featuresrrjinitializer_ranger}rfill_rakaiming_normal_r)rimoduleks r* _init_weightsz"WavLMPreTrainedModel._init_weightss f8 9 9 9   % * 2 2 2 C C C   # ( . . 0 0 0 G  V/ 0 0 0 0 0  < = = 9 GOO " !v{'>q'AFKD['["\]]]     G  fk. 2 2 2 2 2  6 7 7 9 !f/;;<._conv_out_lengths&9\K7wWWWZ[[ [r,r)rjrziprbrcr>rr)rirMrrrYrZr1s r* _get_feat_extract_output_lengthsz5WavLMPreTrainedModel._get_feat_extract_output_lengthss2=1Ddk--+  \ \ \ $'t{'> @W#X#X Q Q K,,]KPPMM  _4;9:: _ _ 0 04;C] ^ ^ r,feature_vector_lengthrc |ddddf}|||}|tj}|jd}tj||f|j|j}d|tj |jd|j|dz f<| dgd dg }|S)Nr-rrr)r4r&r)r&) cumsumrr%rr#rr?r4r&rBflipr@)rirrrnon_padded_lengthsoutput_lengthsrLs r*"_get_feature_vector_attention_maskz7WavLMPreTrainedModel._get_feature_vector_attention_masks ,22r2::111b5A>>?Q_j>kk'**5:66#)!,  . /~7KTbTi   uv ^%9!%<^EZ[[[]kno]opq',,bT2299"==BBB4HHMMOOr,rn)rtrurvrr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingrrrr=r$rr@rrr/r,r*rrs L$O&*#999DZ^"5#3S#89HPQU0Y]%(:?:Jr,ra WavLM was proposed in [WavLM: Unified Speech Representation Learning with Labeled and Unlabeled Data](https://arxiv.org/abs/2110.13900) by Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving etc.). This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`WavLMConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aI Args: input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`Wav2Vec2Processor.__call__`] for details. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) `attention_mask` should only be passed if the corresponding processor has `config.return_attention_mask == True`. For all models whose processor has `config.return_attention_mask == False`, `attention_mask` should **not** be passed to avoid degraded performance when doing batched inference. For such models `input_values` should simply be padded with 0 and passed without `attention_mask`. Be aware that these models also yield slightly different results depending on whether `input_values` is padded or not. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z_The bare WavLM Model transformer outputting raw hidden-states without any specific head on top.cteZdZdeffd ZdZdZ ddejde ejde ej fd Z e e eeeed e  dd e ejde ejde ejd e ede ede edeeeffdZxZS) WavLMModelrjct|||_t||_t ||_|jdks |jdkrBtj tj |j |_|jrt#||_nt'||_|jrt+|nd|_|dS)Nr)r\r]rjrfeature_extractorrfeature_projectionmask_time_probmask_feature_probrrrr;rrmasked_spec_embeddo_stable_layer_normrencoderrfrradapter post_initrs r*r]zWavLMModel.__init__Rs     !4V!>DLL'//DL/5/AK|F+++t  r,cbtjdt|dSz Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nrrrfreeze_feature_encoderris r*freeze_feature_extractorz#WavLMModel.freeze_feature_extractorf;   Q    ##%%%%%r,c8|jdS Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rrrs r*rz!WavLMModel.freeze_feature_encoderrs 1133333r,Nrpmask_time_indicesrct|jdds|S|\}}}|#|j|j||<n|jjdkr|jr|t||f|jj|jj ||jj }tj ||j tj}|j|j||<|jjdkr|jrt||f|jj|jj|jj}tj ||j tj}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://arxiv.org/abs/1904.08779). apply_spec_augmentTNr)rrrr)r&r4)rrrr-)getattrrjrrr%r4rrrTmask_time_lengthmask_time_min_masksrtensorr&r@rmask_feature_lengthmask_feature_min_masksexpand)rirprrrLr)rmask_feature_indicess r*_mask_hidden_stateszWavLMModel._mask_hidden_statesyst{$8$?? ! 4A3E3E3G3G0 O[  (/3/E/H/HI\/]/]M+ , , [ '! + + + 5_-+4 K8-+9 !!! !& ->}G[chcm n n n /3/E/H/HI\/]/]M+ , ; (1 , , ,#8[)+7 K;+< $$$ $)<0D]Mainis#t#t#t #74#@#G#GO]_#`#` 23M. /r,audio checkpoint output_typermodalityexpected_outputrrr|r}r c:||n |jj}||n |jj}||n |jj}||}|dd}|#||jd|d}||\}}| |||}| |||||} | d}|j | |}|s||f| ddzSt||| j | jS) NrrFr)rrrrr|r}r)rvextract_featuresrprw)rjrr|use_return_dictrrrrrr rrrrprw) rirrrrr|r}rrpencoder_outputss r*rqzWavLMModel.forwards|"2C1N--TXT_Tq$8$D $+Jj &1%uEN+/*A*ABR*S*S' '00 ->~1  ,, )/!5# '  (* < # LL77M K!#34qrr7JJ J&+-)7&1     r,)NNNNNNN)rtrurvrr]rrrr<rr=r rWAVLM_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPEr;r@rrrqrwrxs@r*rrLs {( & & &444:>59 ,,(,$E$56,!!12 ,,,,\+*+ABB&+$. 269=,0/3&*2 2 u|,2 !.2 $E$56 2 $D> 2 'tn 2 d^2  u-- .2 2 2 CB2 2 2 2 2 r,rzcWavLM Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).c>eZdZddeeffd ZdZdZdZdZ e e e e eeee dd eejd eejd eed eed eedeejdeeeffdZxZS) WavLMForCTCN target_langct|t||_t j|j|_||_|j td|j dt|dr|j r|jn|j}t j||j |_|dS)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `WavLMForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.r)r\r]rrrr final_dropoutrr vocab_sizer6rlrrrrrlm_headr)rirjrrrls r*r]zWavLMForCTC.__init__s    '' z&"677 &   $HHHH *1)G)G vFL^ vF % %djdv y!3V5FGG  r,c|j}|)t|jddtd|d|2t|jddtddS|||ddSdS)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. Nadapter_attn_dimzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load)rrrjr6loggerinfo load_adapter)rirs r* tie_weightszWavLMForCTC.tie_weightss&  "wt{||jjkrt d|jj||||||}|d}||}||} d} |Z||ntj |tj }| | d tj } |dk} | d} || }tj| dtjdd}tjjd 5tj||| | |jj|jj|jj } dddn #1swxYwY|s| f|t6dz}| | f|zn|St9| | |j|j S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rrr3r-)rr4rF)enabled)blank reduction zero_infinitylosslogitsrprw)rjrr%r!r6rrr"r ones_liker#rr;r% masked_selectrr log_softmaxfloat32rbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONr rprw)rirrrr|r}r5r^rpr=r<rM labels_masktarget_lengthsflattened_targets log_probsoutputs r*rqzWavLMForCTC.forward1s0&1% D 'tn D d^ D &D  un$ %D D D CBD D D D D r,rz WavLM Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. c"eZdZfdZdZdZdZeee e e e d dde ejd e ejd e ed e ed e ed e ejdeee ffdZxZS)WavLMForSequenceClassificationct|t|dr|jrt dt ||_|jdz}|jr.tj tj ||z |_ tj|j|j|_tj|j|j|_|dS)Nrz\Sequence classification does not support the use of WavLM adapters (config.add_adapter=True)r)r\r]rrr6rrrluse_weighted_layer_sumrrrrE layer_weightsrrclassifier_proj_size projector num_labels classifierrrirj num_layersrls r*r]z'WavLMForSequenceClassification.__init__s    6= ) ) f.@ n  '' -1  ( S!#ej.D.Dz.Q!R!RD 6#5v7RSS)F$?ARSS r,cbtjdt|dSrrrs r*rz7WavLMForSequenceClassification.freeze_feature_extractorrr,cB|jjdSrr-rs r*rz5WavLMForSequenceClassification.freeze_feature_encoderr.r,cL|jD] }d|_ dSr0r1rs r*r2z0WavLMForSequenceClassification.freeze_base_modelr3r,r )rrrrNrrrr|r}r5r c||n |jj}|jjrdn|}||||||}|jjrx|t}t j|d}tj |j d} || dddz d}n|d}| |}||d} nh||jd|} d|| <| d| d ddz } || } d} |Kt%}|| d|jj| d} |s| f|tdz}| | f|zn|St)| | |j|j S)  labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrrrr-rrr;)rjrrTrrIrstackrrrrUrr;rWrrrrYrrXrrprw)rirrrr|r}r5r^rp norm_weights pooled_output padding_maskr=r<loss_fctrNs r*rqz&WavLMForSequenceClassification.forwards&0&1%->r1a-H-HHMMRSMTTMM#AJM}55  !)..1.55MMBB=CVWXCY[ijjL+.M<- ()--!-44|7G7GA7G7N7N7S7STVXY7Z7ZZM//  '))H8FKKDK,BCCV[[QS__UUD FY)F)G)G!HHF)-)9TGf$$v E'!/)     r,r)rtrurvr]rrr2rrrrrrrrr;r@rrrqrwrxs@r*rRrRsD$ & & &:::(((+*+ABB&,$ 26,0/3&*)-; ; u|,; !.; $D> ; 'tn ; d^ ; &;  u.. /; ; ; CB; ; ; ; ; r,rRza WavLM Model with a frame classification head on top for tasks like Speaker Diarization. c$eZdZfdZdZdZdZeee e e e de  ddeejd eejd eejd eed eed eedeee ffdZxZS) WavLMForAudioFrameClassificationct|t|dr|jrt dt ||_|jdz}|jr.tj tj ||z |_ tj|j|j|_|j|_|dS)Nrz_Audio frame classification does not support the use of WavLM adapters (config.add_adapter=True)r)r\r]rrr6rrrlrTrrrrErUrrrXrY init_weightsrZs r*r]z)WavLMForAudioFrameClassification.__init__s    6= ) ) f.@ q  '' -1  ( S!#ej.D.Dz.Q!R!RD )F$68IJJ + r,cbtjdt|dSr+rrs r*rz9WavLMForAudioFrameClassification.freeze_feature_extractorrr,cB|jjdSrr-rs r*rz7WavLMForAudioFrameClassification.freeze_feature_encoder!r.r,cL|jD] }d|_ dSr0r1rs r*r2z2WavLMForAudioFrameClassification.freeze_base_model(r3r,r r Nrrr5rr|r}r c ||n |jj}|jjrdn|}||||||}|jjrx|t}t j|d}tj |j d} || dddz d}n|d}| |} d} |`t} | | d|jt j| d|jd} |s| f|tdz} | St#| | |j|j S) r`NTrrrr-r)axisr;)rjrrTrrIrrarrrrUrr;rYrrXrrrprw)rirrr5rr|r}r^rprbr=r<rerNs r*rqz(WavLMForAudioFrameClassification.forward0s0&1%->r1a-H-HHMMRSMTTMM#AJM//  '))H8FKKDO<t>t>tuuD Y)F)G)G!HHFM$!/)     r,r)rtrurvr]rrr2rrr_FRAME_CLASS_CHECKPOINTrr_FRAME_EXPECTED_OUTPUTrrr;r@rrrqrwrxs@r*rgrgsG & & &:::(((+*+ABB*)$. 26)-,0/3&*3 3 u|,3 !.3 & 3 $D> 3 'tn 3 d^3  u++ ,3 3 3 CB3 3 3 3 3 r,rgc&eZdZdfd ZdZxZS) AMSoftmaxLoss>@皙?c tt|||_||_||_t jtj ||d|_ t j |_ dS)NT)r) r\rrr]scalemarginrXrrrrandnrrr<)ri input_dimrXrvrwrls r*r]zAMSoftmaxLoss.__init__psk mT""++---  $l5;y*#E#EUYZZZ ')) r,c|}tj|jd}tj|d}t j||}||jz }tj||j }|j t j | ||z}| ||}|S)Nrrr)flattenrr normalizerrmmrwone_hotrXrvr4r@r<) rirpr5r cos_thetapsionehotr=r<s r*rqzAMSoftmaxLoss.forwardxs!!((!(<< // 1/EE H]F33 $+%&&vt??ek&++--iHHHyy(( r,)rsrtrsrxs@r*rrrrosL******       r,rrcDeZdZdfd ZdejdejfdZxZS) TDNNLayerrct|dkr|j|dz n |j||_|j||_|j||_|j||_tj |j|jz|j|_ tj |_ dS)Nrr)r\r]tdnn_dimr_r` tdnn_kernelrY tdnn_dilationdilationrrkernelReLUrgrhs r*r]zTDNNLayer.__init__s .s#QQQy++QQQr,r-r)r\r]rrrlrTrrrrErUrrrrWr>rCrtdnnxvector_output_dimrrYrrrX objectiveri)rirjr[ tdnn_layersrls ` r*r]zWavLMForXVector.__init__s)    '' -1  ( S!#ej.D.Dz.Q!R!RD 6#5vq7IJJQQQQU3v;O;O5P5PQQQ M+.. !#6?2+>+BFD]!^!^)F$=v?XYY&v'@&BSTT r,cbtjdt|dSr+rrs r*rz(WavLMForXVector.freeze_feature_extractorrr,cB|jjdSrr-rs r*rz&WavLMForXVector.freeze_feature_encoderr.r,cL|jD] }d|_ dSr0r1rs r*r2z!WavLMForXVector.freeze_base_modelr3r,rMcDd}|jjD]}|||d}|S)z? Computes the output length of the TDNN layers c||z |zdzS)Nrr/rs r*rzBWavLMForXVector._get_tdnn_output_lengths.._conv_out_lengths!;.69A= =r,r)rjr)rirMrrYs r*_get_tdnn_output_lengthsz(WavLMForXVector._get_tdnn_output_lengthssE  > > >  ;2 L LK,,]KKKMMr,r r Nrrrr|r}r5r c>||n |jj}|jjrdn|}||||||}|jjrx|t}t j|d}tj |j d} || dddz d}n|d}| |}|jD] } | |}|-|d} |d} n|| d} || }g} g} t'|D]k\}}| ||d|fd| ||d|fdlt j| } t j| } t j| | gd}||}||}d}||||}|s||f|tdz}||f|zn|St3||||j|jS) r`NTrrrr-r)r<r= embeddingsrprw)rjrrTrrIrrarrrrUrr;rWrrrrrrxrGrrrYrrrprw)rirrrr|r}r5r^rprb tdnn_layer mean_features std_featuresfeat_extract_output_lengthstdnn_output_lengthsrlengthstatistic_poolingoutput_embeddingsr=r<rNs r*rqzWavLMForXVector.forwards0&1%->r1a-H-HHMMRSMTTMM#AJM}55 ) 6 6J&J}55MM  !)..1.55M(,,,33LL*.*O*OP^PbPbghPbPiPi*j*j '"&"?"?@["\"\ ML&':;; J J 6$$]1gvg:%>%C%C%C%J%JKKK##M!WfW*$=$A$Aa$A$H$HIIII!K 66M ;|44L!I}l&CLLL 223DEE!233  >>&&11D F/07;X;Y;Y3ZZF)-)9TGf$$v E(!/)     r,r)rtrurvr]rrr2rrr=r$rrrr_XVECTOR_CHECKPOINTrr_XVECTOR_EXPECTED_OUTPUTrr;r@rrqrwrxs@r*rrs~& & & &:::((( eE I 'tn I d^ I &I  um# $I I I CBI I I I I r,rr)Vrr1rtypingrrrnumpyr7rtorch.nn.functionalrrrtorch.utils.checkpointtorch.nnr activationsr integrations.deepspeedr integrations.fsdpr modeling_outputsr r rrrrmodeling_utilsrrrrrrrconfiguration_wavlmr get_loggerrtr&rIrrrrOrProrprrr$r0r=ndarrayrTModulerVrzrrrrrrrr@rOrbrfrrrrrWAVLM_START_DOCSTRINGrrrrRrgrrrrr/r,r*rs )))))))))) %%%%%%!!!!!!@@@@@@777777.------,,,,,  H % %!" J&t9Q526 tt c?tttU-. t  t Z ttttp ,bi8bi2*****29***\ )))))"))))X     /    1 1 1 1 1RY 1 1 1c c c c c RYc c c Nry0&&&&& &&&R"""""ry"""JO O O O O 29O O O dP P P P P ")P P P fC'C'C'C'C'C'C'C'N29@ $UUUUU?UUUp("Je P P P P P %P P  P fm T T T T T &T T  T n s s s s s %9s s s l g g g g g ';g g g VBI0 > O O O O O *O O O O O r,