g|||fdkr|dz}n*|||fdkr|dz}ntd|d|d |dk|dktj| ddd } tj| ddd } | | fS) z Measures similarity between two temporal sequences: the input audio and the output tokens. Used to generate token-level timestamps. rdtyper)rrrNz9Internal error in dynamic time warping. Unexpected trace[, z]. Please file a bug report.r) r"nponesfloat32infrangeappend RuntimeErrorarray)r, output_length input_lengthcosttracejic0c1c2ct text_indices time_indicess r)_dynamic_time_warpingrFBs #),M< 7MA%|a'78 K K Kbf TD Wma')9:"* M M M MEDJ 1lQ& ' 'q-!+,,  Aa!eQUl#Ba!eQhBaQhBBww27711bR"WW1111Aq1u -1DAJE!Q$KK   AA AAE!QQQ$KE!!!Q$KLL a%%1q55AE"""AE""" A;!   FA FAA 1a4[A   FAA 1a4[A   FAApAppQRppp  a%%1q558L))$$B$/L8L))$$B$/L  %%r+cj|/tfd|Dd}|rt||dSdS)Nc3<K|]}t||VdSN) isinstance).0clslogit_processor_classs r) z2_get_attr_from_logit_processors..zs3jj:cShCiCijjjjjjjr+)nextgetattr)logits_processorrMattribute_namelogit_processors ` r)_get_attr_from_logit_processorsrTxsO#jjjj/?jjjlpqq  B?NDAA A 4r+rightlongestcjd}g}|dvrtd||dvrtd||dkr|td|D]} | td| Ddkr~tjd | Dd } | | | d} |tj|| g} || t |t|d }||||tjg| |dkr|d zn|}tt|D]H} |t|| z } |dkrd| fn| df} tj || | ||| <Itj |d }|S)Nr)rUleftz5`padding_side` must be either 'right' or 'left', not )rV max_lengthz8`padding` must be either 'longest' or 'max_length', not rYz>`cut_off_length` must be specified when `padding='max_length'`cg|] }|d StokensrKds r) z&_pad_to_max_length..s4_4_4_QQx[4_4_4_r+cg|] }|d Sr[r]r^s r)r`z&_pad_to_max_length..s!L!L!L!!H+!L!L!Lr+rdimdevicerrU)r$value) r!lentorchcatr6maxtensorr5Fr$stack)current_segments pad_token_idre padding_sidepaddingbos_token_tensorcut_off_lengthmax_total_length sequencescurrent_segment_listsequencer> pad_lengthr$s r)_pad_to_max_lengthrys'I,,,_Q]__```///]T[]]^^^ L ^%;YZZZ 0>>  +4_4_J^4_4_4_0`0`cd0d0dy!L!L7K!L!L!LRTUUUH)#^O$4$45+ 9&6%ABB   X & & &"#3S25G5GHH    )   - . . . .   U\"V<<< = = = =-4 -D-D~))JZ 3'(( ) )HH%IaL(9(99 !-!8!8q*ooz1ouYq\s,GGG !  I1---I r+c0deZdZ d=dZ d>deejdeedeedee d ee e ejge e fd e d ee d eed eeee efdee deejdeedee deeeeedffdeedeedeedee deejdedee de dee f.dZfdZedZd Zd!Zd"Zd#Zed$Zed%Zed&Zed'Zd(Zed)Zd*Z d?deej!d,eeej!e"fdeede d-ejf d.Z#ed/Z$ed0Z%ed1Z&ed2Z'ed3Z(ed4Z)d5Z*ed6Z+ed7Z,ed8Z-d9Z.ed:Z/ed;Z0ed<Z1xZ2S)@WhisperGenerationMixin{Gz?Nc@ gt|jjD]<t jfd|jDd=t jfd|Dgdd}d|vr|j dk d }||n||z}t j |j ddd|f|j ddd||z fdd|df<ddddd|f dkd t jfd tjd Dd|pd jd}|jjd }t j||d zftj|jj } |Kt'|t(rd d|dzfn%t'|t*t,t.jfr;t3t/j|d krd d|d dzfnt'|tjr;t3t j|d krd d|d dzfnnt'|t(r|n|t3|z} t'|tjr|n|}t/j|| }|t'|t(rgt jddd} t jdd} | z | z tA|jj!d t|D]m} |t'|t,t*t.jtjfr~| d d|| dzf}t j|ddd} t j|dd} || z | z }tA||jj!}|d }n| }tE|#$ \}}t/j%t/j&|dd 'tP}|||z}t j)|| | d df<o| S)a Calculates token-level timestamps using the encoder-decoder cross-attentions and dynamic time-warping (DTW) to map each output token to a position in the input audio. If `num_frames` is specified, the encoder-decoder cross-attentions will be cropped before applying DTW. Returns: tensor containing the timestamps in seconds for each predicted token c g|] }| Sr]r])rKxr>s r)r`zDWhisperGenerationMixin._extract_token_timestamps..s._._._qt._._._r+rrbc:g|]\}}|dd|fSrIr])rKlhcross_attentionss r)r`zDWhisperGenerationMixin._extract_token_timestamps..s0UUUTQ/2111a48UUUr+)rrrr N beam_indicesrrc rg|]3}tjdddd|ddfddd|f4S)Nr)rcindex)rh index_select)rKr>rweightss r)r`zDWhisperGenerationMixin._extract_token_timestamps..sf&wqqq!!!Qz':VWVWVWYZVZI[\\\r+rr/re.TF)rckeepdimunbiased)rcr)rr)constant_values)*r5configdecoder_layersr6rhrirrmpermutersumrj zeros_like masked_fillr"ruzerosr3rerJintlisttupler1ndarrayrguniqueTensorcpurepeatstdmeanr*median_filter_widthrFdoublenumpyr$diffastypeboolrk)selfgenerate_outputsalignment_headstime_precision num_frames num_input_ids weight_lengthr: batch_size timestamps repeat_timerr batch_idxr,rDrEjumps jump_timesrrr>rs @@@@r)_extract_token_timestampsz0WhisperGenerationMixin._extract_token_timestampssft{122 i iA  # #EI._._._._=M=^._._._ef$g$g$g h h h h+UUUU_UUUVV//,,,// - - -.:b@EEbIIMMOOM-:-BMM XeHeM!+,<,I!!!^m^J[,\]]L.>.KAAAOnQ^anQnOnLn.oLMNN* +aaaN]N23G(33LB4FJJLk"<#5a#899 G%D(8(;(A!(D %/5a8 [ ) *%-HXHbHi     ! *c** @!#'8q'8"89Jubj(ABB @s29U_K`K`GaGaefGfGf!#';A!);';";<J77 @C Z@X@X @S@S@U@U@[@[@]@]?])^)^ &L,F27<00&!LLLSSTXYYE%e,~=J(- Z(@(@Jy!""} % %r+Finput_featuresgeneration_configrQstopping_criteriaprefix_allowed_tokens_fn synced_gpusreturn_timestampstasklanguageis_multilingual prompt_idsprompt_condition_typecondition_on_prev_tokens temperature.compression_ratio_thresholdlogprob_thresholdno_speech_thresholdnum_segment_framesattention_maskrreturn_token_timestampsreturn_segmentsreturn_dict_in_generatec  ;<d|vr/|d}tjdt|j|fi|\}}|jjjjd|jjj jdz}||j j z}| |||\}}||k}| ||||}||||}|| || ||||||||||| ||| |||||j || }|| d |vr|d djn|j}|jd} |||| |dd|}|| |t5|t6t8fs|gn|}!|!d}|||||\}"}#|j<|||#|"||| |\}$}%}}#}"}}&| | |%|}'|#|"k!r|"||#|"|%|$\}}%}$|#|z|z }(|"|#z #|})|$||#|)||%|$}*tK|tLd}+|'|%||'|$|&| ||j |j|+| \},}|(|j |,||5|D]2}-tS|-dr |-*|,jd3|+|*|,|%|$|#||"|!|||||||&||||\}.}/}0}&}1tY|.D]\}2}3|$|2}4|0|2r|#|4xx|)|4z cc<,|-|3|/|(||)|||4|2| \}5}6|'|4xx|5z cc<|r|#|4xx|"|2z cc<w|#|4xx|6z cc<|#|"k!| |j.d kr d!|'Dn|'}7t_|7|j0|jd"#}8|r|8|7d$S|rg|j1F|j2?tgj4|8jddf|j5}9tgj6|8|9gd%}8|r+i}:|8|:d&<tgj7d'|/Dd%|:d(<n|8}:|r|j8r|9|/|1|8j|; Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the model's default generation configuration. You can override any `generation_config` by passing the corresponding parameters to generate(), e.g. `.generate(inputs, num_beams=4, do_sample=True)`. For an overview of generation strategies and code examples, check out the [following guide](./generation_strategies). Parameters: input_features (`torch.Tensor` of shape `(batch_size, feature_size, sequence_length)`, *optional*): Float values of log-mel features extracted from the raw speech waveform. The raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `List[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] for details. generation_config (`~generation.GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default will be used, which had the following loading priority: 1) from the `generation_config.json` model file, if it exists; 2) from the model configuration. Please note that unspecified parameters will inherit [`~generation.GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. stopping_criteria (`StoppingCriteriaList`, *optional*): Custom stopping criteria that complement the default stopping criteria built from arguments and a generation config. If a stopping criteria is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. prefix_allowed_tokens_fn (`Callable[[int, torch.Tensor], List[int]]`, *optional*): If provided, this function constraints the beam search to allowed tokens only at each step. If not provided no constraint is applied. This function takes 2 arguments: the batch ID `batch_id` and `input_ids`. It has to return a list with the allowed tokens for the next generation step conditioned on the batch ID `batch_id` and the previously generated tokens `inputs_ids`. This argument is useful for constrained generation conditioned on the prefix, as described in [Autoregressive Entity Retrieval](https://arxiv.org/abs/2010.00904). synced_gpus (`bool`, *optional*, defaults to `False`): Whether to continue running the while loop until max_length (needed to avoid deadlocking with `FullyShardedDataParallel` and DeepSpeed ZeRO Stage 3). return_timestamps (`bool`, *optional*): Whether to return the timestamps with the text. This enables the `WhisperTimestampsLogitsProcessor`. task (`str`, *optional*): Task to use for generation, either "translate" or "transcribe". The `model.config.forced_decoder_ids` will be updated accordingly. language (`str` or list of `str`, *optional*): Language token to use for generation, can be either in the form of `<|en|>`, `en` or `english`. For batched generation, a list of language tokens can be passed. You can find all the possible language tokens in the `model.generation_config.lang_to_id` dictionary. is_multilingual (`bool`, *optional*): Whether or not the model is multilingual. prompt_ids (`torch.Tensor`, *optional*): Rank-1 tensor of token IDs created by passing text to [`~WhisperProcessor.get_prompt_ids`] that is provided as a prompt to each chunk. This can be used to provide or "prompt-engineer" a context for transcription, e.g. custom vocabularies or proper nouns to make it more likely to predict those words correctly. It cannot be used in conjunction with `decoder_start_token_id` as it overwrites this value. prompt_condition_type (`str`, *optional*): Only relevant for long-form transcription. Condition type of `prompt_ids`. 'first-segment' means only the first segment is conditioned on `prompt_ids`. 'all-segments' means each segment is conditioned on `prompt_ids`. Make sure to enable `condition_on_prev_tokens` for 'all-segments'. Defaults to 'first-segment'. For short-term transcription only 'first-segment' is possible. condition_on_prev_tokens (`bool`, *optional*): Only relevant for long-form transcription. Whether to condition each segment on the previous segment. As shown in the [the Whisper paper](https://cdn.openai.com/papers/whisper.pdf), this can help to improve performance. temperature (`float` or list of `float`, *optional*): The temperature to be used for generation. Passing a single `float` value and `do_sample=True` activates generation using sampling. For long-form transcription, temperature fallback can be activated by passing a list of float values such as (0.0, 0.2, 0.4, 0.6, 0.8, 1.0). As shown in the [the Whisper paper](https://cdn.openai.com/papers/whisper.pdf), this can help to improve performance. compression_ratio_threshold (`float`, *optional*): Only relevant for long-form transcription. If defined, the zlib compression rate of each segment will be computed. If the compression rate of a segment is higher than `compression_ratio_threshold`, temperature fallback is activated: the generated segment is discarded and the generation is repeated using a higher temperature. The intuition behind this feature is that segments with very high compression rates suffer from a lot of repetition. The unwanted repetition can be reduced by injecting more randomness by increasing the temperature. If `compression_ratio_threshold` is defined make sure that `temperature` is a list of values. A common value for `compression_ratio_threshold` is 1.35. As shown in the [the Whisper paper](https://cdn.openai.com/papers/whisper.pdf), this can help to improve performance. logprob_threshold (`float`, *optional*): Only relevant for long-form transcription. If defined, the average log-probability of each segment will be computed. If the log-probability of a given segment is lower than `logprob_threshold`, temperature fallback is activated: the generated segment is discarded and the generation is repeated using a higher temperature. The intuition behind this feature is that segments of low log-probability can be improved by injecting more randomness by increasing the temperature. If `logprob_threshold` is defined make sure that `temperature` is a list of values. A common value for `logprob_threshold` is -1.0. As shown in the [the Whisper paper](https://cdn.openai.com/papers/whisper.pdf), this can help to improve performance. no_speech_threshold (`float`, *optional*): Only relevant for long-form transcription. If defined, the "no-speech" token combined with the `logprob_threshold` is used to determine whether a segment contains only silence. In this case, the transcription for this segment is skipped. As shown in the [the Whisper paper](https://cdn.openai.com/papers/whisper.pdf), this can help to improve performance. num_segment_frames (`int`, *optional*): The number of frames a single segment is made of. If not defined, `num_segment_frames` defaults to the model's stride times the maximum input length. attention_mask (`torch.Tensor`, *optional*): `attention_mask` needs to be passed when doing long-form transcription using a batch size > 1. time_precision (`int`, *optional*, defaults to 0.02): The duration of output token in seconds. *E.g.* 0.02 means that a generated token on average accounts for 20 ms. return_token_timestamps (`bool`, *optional*): Whether to return token-level timestamps with the text. This can be used with or without the `return_timestamps` option. To get word-level timestamps, use the tokenizer to group the tokens into words. return_segments (`bool`, *optional*, defaults to `False`): Whether to additionally return a list of all segments. Note that this option can only be enabled when doing long-form transcription. return_dict_in_generate (`bool`, *optional*, defaults to `False`): Whether or not to return a [`~utils.ModelOutput`] instead of just returning the generated tokens. Note that when doing long-form transcription, `return_dict_in_generate` can only be enabled when `return_segments` is set True. In this case the generation outputs of each segment is added to each segment. kwargs (`Dict[str, Any]`, *optional*): Ad hoc parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with *decoder_*. Return: [`~utils.ModelOutput`] or `torch.LongTensor` or `Dict[str, Any]`: A [`~utils.ModelOutput`] (if `return_dict_in_generate=True` or when `config.return_dict_in_generate=True`) or a `torch.FloatTensor` or a dict of segments when `return_segments=True`. If the passed input is > 30 seconds / > 3000 mel input features and `return_segments=True` then a dictionary of generated sequence ids, called `sequences` and a list of each generated segment is returned. else if the passed input is <= 30 seconds / >= 3000 mel input features, the possible [`~utils.ModelOutput`] types are: - [`~generation.GenerateEncoderDecoderOutput`], - [`~generation.GenerateBeamEncoderDecoderOutput`] else only the generated output sequence ids are returned. Example: - *Longform transcription*: To transcribe or translate audios longer than 30 seconds, process the audio files without truncation and pass all mel features at once to generate. ```python >>> import torch >>> from transformers import AutoProcessor, WhisperForConditionalGeneration >>> from datasets import load_dataset, Audio >>> processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en") >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en") >>> model.cuda() # doctest: +IGNORE_RESULT >>> # load audios > 30 seconds >>> ds = load_dataset("distil-whisper/meanwhile", "default")["test"] >>> # resample to 16kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=16000)) >>> # take first 8 audios and retrieve array >>> audio = ds[:8]["audio"] >>> audio = [x["array"] for x in audio] >>> # make sure to NOT truncate the input audio, to return the `attention_mask` and to pad to the longest audio >>> inputs = processor(audio, return_tensors="pt", truncation=False, padding="longest", return_attention_mask=True, sampling_rate=16_000) >>> inputs = inputs.to("cuda", torch.float32) >>> # transcribe audio to ids >>> generated_ids = model.generate(**inputs) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True) >>> transcription[0] " Folks, if you watch the show, you know, I spent a lot of time right over there. Patiently and astutely scrutinizing the boxwood and mahogany chest set of the day's biggest stories developing the central headline pawns, definitely maneuvering an oso topical night to F6, fainting a classic Sicilian, nade door variation on the news, all the while seeing eight moves deep and patiently marshalling the latest press releases into a fisher's shows in Lip Nitsky attack that culminates in the elegant lethal slow-played, all-passant checkmate that is my nightly monologue. But sometimes, sometimes, folks, I. CHEERING AND APPLAUSE Sometimes I startle away, cubside down in the monkey bars of a condemned playground on a super fun site. Get all hept up on goofballs. Rummage that were discarded tag bag of defective toys. Yank out a fist bowl of disembodied doll limbs, toss them on a stained kid's place mat from a defunct dennies. set up a table inside a rusty cargo container down by the Wharf and challenged toothless drifters to the godless bughouse blitz of tournament that is my segment. Meanwhile." ``` - *Shortform transcription*: If passed mel input features are < 30 seconds, the whole audio will be transcribed with a single call to generate. ```python >>> import torch >>> from transformers import AutoProcessor, WhisperForConditionalGeneration >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en") >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = processor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_features = inputs.input_features >>> generated_ids = model.generate(inputs=input_features) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> transcription ' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.' ``` rzXThe input name `inputs` is deprecated. Please make sure to use `input_features` instead.r)r input_stridekwargsrrrr)r is_shortformrrrrrrrrrrrrr)rr)rrrrr)rencoder_outputsr num_beams)rrQ begin_indexrrerr)rrtotal_input_framesr)rseek max_frames init_tokensrrr)rrr)rrrcur_bsz batch_idx_map)rj)rrseek_num_framesrrrsuppress_tokens) rrrnrdo_condition_on_prev_tokensrrrrerr)rdecoder_input_idsrNset_begin_indexr) segment_inputrrrrrr temperaturesrrQrrrrrrrrr) seek_sequence seek_outputs time_offsettimestamp_beginrrrprev_idxidxr first-segmentc"g|] }|dd S)rNr])rKrs r)r`z3WhisperGenerationMixin.generate..s - - -qQqrrU - - -r+rU)rerp)rusegmentsrbrucg|] }|d S)token_timestampsr]r^s r)r`z3WhisperGenerationMixin.generate..s:g:g:gUV1=O;P:g:g:gr+rencoder_attentionsc3DK|]}j|ddVdSrI)rrKr> dict_outputsnum_return_sequencess r)rNz2WhisperGenerationMixin.generate..sV@@ !);A>?U?UAU?UV@@@@@@r+encoder_hidden_statesc3DK|]}j|ddVdSrI)rrs r)rNz2WhisperGenerationMixin.generate..sVCC !)>qABXBXDXBXYCCCCCCr+)>popwarningswarn FutureWarning_prepare_generation_configmodelencoderconv1strideconv2rmax_source_positions_retrieve_total_input_frames_set_return_outputs_set_return_timestamps_set_language_and_task_set_num_frames_set_thresholds_and_condition_set_prompt_condition_type_retrieve_init_tokens_check_decoder_input_idsrer"_retrieve_logit_processorsget_set_condition_on_prev_tokensrJrr_retrieve_max_frames_and_seekr _expand_variables_for_generation_prepare_segmentsany_maybe_reduce_batchclamp_get_input_segmentrTr_prepare_decoder_input_ids_set_max_new_tokens_and_lengthhasattrrgenerate_with_fallback enumerate_retrieve_segmentrryromax_new_tokensrYrhfull eos_token_idrirmr_stack_split_outputsrr5rgr)=rrrrQrrrrrrrrrrrrrrrrrrrrrrrrrrrrerrrrrrrrnrrrrrprocseek_sequencesr should_skipmodel_output_typer>rprev_irsegment_offsetfinal_segmentsru eos_tokensoutputsrrs= @@r)generatezWhisperGenerationMixin.generate!st v  #ZZ11N Mj    %DD$CDU$`$`Y_$`$`!6z)/6q9DJ   D 1E  1 1)!#!%=/ 2         ' 11!/2  j %%''p 3 6:5M5M-%+ 6N66 2NG]/,>K)D077 "?ARO)-(G(G'!1+,G%"3{") /)H ) ) % v  / /{"3"3 0    +,JJDt%677J,,->-DR-HIII+++"3+#5%)"3!1"3)A'(?,G)%-',  +!0%.n$=$= 3 3 =&q)q>LLLOF$;;LLL+/+A+A"/!- +$3$3#1!-#,C,B , ,(.!(((H4(((3LLLJqM1LLLLLLLN2LLLLaj %%''p 3l&+<+RVe+e+e . -, - - - -!  ' -:4;]d    H!*GG G "  /7J=BCCCCC%*3|/Q+R+R%S%SCCC>> :+S7>?Q7RL!34##Nr+c 4 tj|}dt|D}dt|D}dt|D}dt|D}tt|}| j|| |||t |D]\}}|duo|dk| _| jr|nd| _| jrd| _tj|}dD] }||vr||= |j d }| j d kr||krtj |d d d d d ||z fd }tj |d d d ||z f| j }|d 'tj |d d d d ||z fd |d <|d)tj |dd d d d d ||z fd |d<tj|f| | | | | ||d|}t#|}||||| |\}}||kr|d|}|d|}g} g}!g}"g}#t |D]\}$}%|||$}&||&|z||&k}'|'r>|%d| jkr-|%dd}%|r!|s||$ddd||$d<|%d| j krL|%| j k}(|%d|( }%|r"|s ||$dd|( ||$d<||%||$| | |jj|\||$<||$<|%|||$<||$|||$<|dup|dk})| jo|)|||$<||$rv| ||$|!||$|"||$d |vr!|#|d |$| }t5|d ks|t5|dz kr|}|}nEt7j|"}t7j|!}d |vrt7j|#|d <|||||fS)Ncg|]}dSrIr]rK_s r)r`zAWhisperGenerationMixin.generate_with_fallback..-s;;;qd;;;r+cg|]}dSrIr]r's r)r`zAWhisperGenerationMixin.generate_with_fallback...s:::aT:::r+cg|]}dSFr]r's r)r`zAWhisperGenerationMixin.generate_with_fallback../s888A%888r+cg|]}dSr+r]r's r)r`zAWhisperGenerationMixin.generate_with_fallback..0s555u555r+g?r) do_samplerrrstatic)rfdecoder_attention_maskTr)rrQrrrrr)rrrrrrrg?)copyr5rr_setup_no_speech_detectionrr.rrr"cache_implementationrlr$rorsuperr$type_postprocess_outputsrr_need_fallbackr vocab_sizerr6rgrhrm)+rrrrrrrrrrrQrrrrrrrrrseek_sequence_listseek_outputs_listneeds_fallbackrfallback_index_map fallback_idxrgenerate_kwargskeyrrrnew_fallback_index_mapnew_segment_inputnew_decoder_input_idsnew_decoder_attention_maskr>rr is_not_final num_paddingsis_low_temperature __class__s+ r)rz-WhisperGenerationMixin.generate_with_fallbacks,6""<;E'NN;;;::5>>:::88w88855eGnn555 !%..11  0 <  + +,J1oN`>abecebe>f Q(:;!$(9(FFF$15F5S$S#X#X#Z#ZL$1.L=.$AM.r|r>J1oN`>abpdpcpbp>q Q(:;594G4G! $%K*551q!;q>=J"#5a#89;G?!"4Q"78%0D%8%MK#UCU,,>q,AB"!$_*112DQ2GHHH%,,]1-=>>>)001B11EFFF/699299&AY:Z[\:]^^^!7 %&&!++|s.s!TTTQ(J!7 8TTTr+cg|]}gSr]r]r's r)r`z.s>>>q>>>r+)rrPr5)rrrrIrns` r)r z(WhisperGenerationMixin._prepare_segmentss  !&7&MQ`&`&` '(9;NPT U U +5a=>E*,=,=>>> r+c s |jdntjd}ttjrdd|dffS|rdt |drTt |dd}|j||jdd<ddd|dfd<ddd|dfd<d fd d} fd t|jdD|fS) Nrrrr)rrrruc $|;|dkr5dt||dt|DS|dvrfd|DS|dvrtfd|DS|dkr|sdSt|trg}t jjD]}g}|j|j fD]L}|j |j fD];} | | |d .split_by_batch_index..s*sssmq(( ))sssr+rNrrlogitscDg|]}|Sr]rPrKrQrs r)r`z]WhisperGenerationMixin._postprocess_outputs..split_by_batch_index..s);;;q) ((**;;;r+decoder_attentionsdecoder_hidden_statesrc3NK|]}tfd|DV dS)c3XK|]$}|dV%dSrIrP)rKwrs r)rNzfWhisperGenerationMixin._postprocess_outputs..split_by_batch_index...s8"G"G1Y<#5#9#9#;#;"G"G"G"G"G"Gr+N)rrVs r)rNz\WhisperGenerationMixin._postprocess_outputs..split_by_batch_index..s@XXAU"G"G"G"GQ"G"G"GGGXXXXXXr+past_key_valuesr)ziprgrrJr r5rrself_attention_cachecross_attention_cache key_cache value_cacher6r) valuesr?rrrall_past_key_values layer_idxlayer_past_key_values cache_clsrQr\rs ` r)split_by_batch_indexzIWhisperGenerationMixin._postprocess_outputs..split_by_batch_indexsn'C8OOsss6rrrhs r) zJWhisperGenerationMixin._postprocess_outputs...sU   Aq''1aLL\L\]kLlLlmmm   r+)items)rKr>rrrhs @r)r`z?WhisperGenerationMixin._postprocess_outputs..sh           (..00      r+rI) r"rhrkrJrrrPrrr5) rrrrrr start_idxrsequence_tokensrhs `` ` @r)r6z+WhisperGenerationMixin._postprocess_outputss8DX%+B//VW lEL 1 1 .'9:: 6L- - " _w/@BS'T'T _ !2L$GGJ/3/M/M!1%/5b9 0N00L+ , 0<.+I+I+IqAcF+I+I+Ir+rbrSc3K|]8tjfdDV9dS)c,g|]}|Sr]r])rKrQr>r?s r)r`zIWhisperGenerationMixin._stack_split_outputs...s! A A Aq3 A A Ar+N)rhrmtorKr>rer?rs @r)rNz>WhisperGenerationMixin._stack_split_outputs..se%%RSEK A A A A AL A A ABBEEfMM%%%%%%r+sequences_scoresc g|] }| Sr]r]rrs r)r`z?WhisperGenerationMixin._stack_split_outputs..rsr+rWc 3K|]NtfdttddDVOdS)c3K|]LtjfdDdVMdS)c8g|]}|Sr]r]rKrQr>r=r?s r)r`zSWhisperGenerationMixin._stack_split_outputs.... s%$H$H$HaQsVAYq\$H$H$Hr+rNrhrmsqueezervrKr=rer>r?rs @r)rNzHWhisperGenerationMixin._stack_split_outputs...s{ $H$H$H$H$H$H<$H$H$HIIQQRSTTWWX^__r+rNrr5rgrws @r)rNz>WhisperGenerationMixin._stack_split_outputs..s%%  !&s<?3+?+B'C'C!D!D%%%%%%r+r]c 3K|]NtfdttddDVOdS)c3K|]LtjfdDdVMdS)c8g|]}|Sr]r]r}s r)r`zSWhisperGenerationMixin._stack_split_outputs....s%(L(L(L!31(L(L(Lr+rNr~rs @r)rNzHWhisperGenerationMixin._stack_split_outputs...s{ !"K(L(L(L(L(L(L|(L(L(LMMUUVWXX[[\bccr+rNrrws @r)rNz>WhisperGenerationMixin._stack_split_outputs..s))  %*3|As/CA/F+G+G%H%H))))))r+r]) keysrhrmrvrr5rgrrJr from_legacy_cache)rrrrerr#past_key_value_typer?s ` ` @r)rz+WhisperGenerationMixin._stack_split_outputss?'')) ( (C333${+I+I+I+IL+I+I+IqQQQTTU[\\ [[[$%%%%%%W\]`amnoapqtau]v]vWwWw%%%   ***${+I+I+I+IL+I+I+IqQQQTTU[\\ [[[$%%%%%% #3|As';#<#<== %%%   )))&,jj1B&C&C#?3'3#()))))) "'s<?3+?'@'@!A!A )))$$GCL+6:FY[n;o;o6':'L'LWUX\'Z'Z #'GCL  ++7+++r+cd}d} |j#|||} | |jkrd}|jat|ddrd|D|} n+||d} || ||j|} | |jkrd}|j6t|td} | |jkr| ||jkrd}d} || fS)NFTrrxcg|] }|d S)rxr])rKss r)r`z9WhisperGenerationMixin._need_fallback..3sHHHaA01HHHr+rNno_speech_prob) r_retrieve_compression_ratiorr_retrieve_avg_logprobsrrrTr)rrrrrQrr8rr;rcompression_ratiologprobsrNrs r)r7z%WhisperGenerationMixin._need_fallbacks  8 D $ @ @PZ [ [  #4#PPP!%  . :|A(:;; HH<HHHO%e,X666M+<+I;+===!%  0 << ":>>"5),=,QQQ!&" {**r+cR|j|jdkrtt||jz}t|} fdtt|D} ||jd}||jd}||jd}||jd}d|_n9|} tt| }fdt| D} || ||||| fS)Nrcg|]}Sr]r]rKr(rs r)r`zKWhisperGenerationMixin._expand_variables_for_generation..Qs*g*g*g+C*g*g*gr+rrbcg|]}Sr]r]rs r)r`zKWhisperGenerationMixin._expand_variables_for_generation..Zs*\*\*\+C*\*\*\r+)rrr5rgrepeat_interleave) rrrrrrrrrrrs ` r)r z7WhisperGenerationMixin._expand_variables_for_generationKsN  1 =BSBhklBlBl z4E4Z'Z![![\\M-((G*g*g*g*gUSVWdSeSeMfMf*g*g*g '+==>O>djk=llN))*;*PVW)XXD#556G6\bc5ddJ%778I8^de7ffK56  2 2 G w00M*\*\*\*\USZ^^*\*\*\ '       '  r+ct|td}d|D}|||d|dS)N set_inputscBi|]\}}tj|||Sr])rh is_tensor)rKrkrQs r)rlzEWhisperGenerationMixin._setup_no_speech_detection..is-NNNA5?1;M;MN1NNNr+)rr)rTrrm)rQrrrr extra_kwargss r)r2z1WhisperGenerationMixin._setup_no_speech_detectionfsV45EG_amnn NNNNN  mBSddWcdeeeeer+c||jd|jdfSd|vrNt|dtr|ddjn |dj}|d|d|zfStd)NrrrrzPMake sure to provide either `input_features` or `encoder_outputs` to `generate`.)r"rJrr!)rrrencoder_outputs_shapes r)rz3WhisperGenerationMixin._retrieve_total_input_framesls  %!'*N,@,DD D  & &f%67II5()!,22-.4 " )+-B1-E -TT Tklllr+cd|d}|0t|d||0t|d||0t|d||0t|d|t|tt frt d|d|ddS) NzAudio input consists of only z@. Short-form transcription is activated.{}, but will be ignored.z#condition_on_prev_tokens is set to z&compression_ratio_threshold is set to zlogprob_threshold is set to zno_speech_threshold is set to zE. Short-form transcription is activated.temperature cannot be set to z which can only be used for temperature fallback for long-form generation. Make sure to set `temperature` to a float value or `None` for short-form generation.)loggerwarningformatrJrrr!)rrrrrrwarning_prefixs r)_maybe_warn_unused_inputsz0WhisperGenerationMixin._maybe_warn_unused_inputs{sG ',> ' ' '  $ / NN>001qWo1q1qrr s s s & 2 NN%%&lOj&l&lmm     ( NN>001cPa1c1cdd e e e  * NN>001gRe1g1ghh i i i kD%= 1 1 ]0B]]0;]]]   r+c||j}n||_||_|rd|_d|_d|_|d|_d|_|S)NT)rroutput_attentions output_scoresrs r)rz*WhisperGenerationMixin._set_return_outputssg " *&7&O # #8O  54K1 " 38<  526  /.2  +  (8<  5.2  +&&r+c>|t|dr|j}|s/|durtdtdd}|rt|dstd||_t|dr |jdz}n|jjdz}|S) NrFaYou have passed more than 3000 mel input features (> 30 seconds) which automatically enables long-form generation which requires the model to predict timestamp tokens. Please either pass `return_timestamps=True` or make sure to pass no more than 3000 mel input features.z:Setting `return_timestamps=True` for long-form generation.Tno_timestamps_token_idadYou are trying to return timestamps, but the generation config is not properly set. Make sure to initialize the generation config with the correct attributes that are needed such as `no_timestamps_token_id`. For more details on how to generate the approtiate config, refer to https://github.com/huggingface/transformers/issues/21878#issuecomment-1451902363r)rrr!rinforrr8)rrrrrs r)rz-WhisperGenerationMixin._set_return_timestampss  $1BDW)X)X $ 1 C  % E)) m KKT U U U $   W->@X%Y%Y g  /@+ $&> ? ? 9/FJOO#k4q8Or+cN|&t|dstd||_t|dr|js||td|&t|dstd||_|(t|dstd||_dSdS)NrzThe generation config is outdated and is thus not compatible with the `is_multilingual` argument to `generate`. Please update the generation config as per the instructions https://github.com/huggingface/transformers/issues/25084#issuecomment-1664398224zCannot specify `task` or `language` for an English-only model. If the model is intended to be multilingual, pass `is_multilingual=True` to generate, or update the generation config. lang_to_ida=The generation config is outdated and is thus not compatible with the `language` argument to `generate`. Either set the language using the `forced_decoder_ids` in the model config, or update the generation config as per the instructions https://github.com/huggingface/transformers/issues/25084#issuecomment-1664398224 task_to_ida5The generation config is outdated and is thus not compatible with the `task` argument to `generate`. Either set the task using the `forced_decoder_ids` in the model config, or update the generation config as per the instructions https://github.com/huggingface/transformers/issues/25084#issuecomment-1664398224)rr!rrrrs r)rz-WhisperGenerationMixin._set_language_and_tasks   &,.?@@  g 1@  - $&7 8 8 ARAb 8#7 n  ,l;;  _ *2  &  ,l;;  _ &*  " " "  r+c  dttdtdttfd}dtdtffd t dd}t dd} j} | -| *|(| d d t d nt|d r|j|j} | )|'t d |d| d|dd} n*| (| &t d| d| d| dd} j g| | d d d krd t| d krX| d d krF| d d gz | d d} d z t| d kr| d d kFt| d kr"td| d| d d dd_td kptd ko d du} t| ttfratd| Drt!dt| |kr#td|dt| d| } n | dg|z} n| g} fd| Dd} | fd| D} nQtdrA| r?|||dd|} | kt)tD]Ntd kr| d <-| O~ t)tD]||t,vrhjjjj}||jntd|d t,d!| itd"rYtfd#jDs&jd$jsHtd%r8d&jkr!jnKjrDd&jkr-t d'dd&<d(D<t;jt:j|j )!|d&S)*NlstnumitrctfdD}|rfdDnS)z/short function to replace num with a itr in lstc3 K|]}|vV dSrIr])rKr>rs r)rNzWWhisperGenerationMixin._retrieve_init_tokens..replace_or_add..s'..QS......r+c g|] }|vrn| Sr]r])rKr>rrs r)r`zXWhisperGenerationMixin._retrieve_init_tokens..replace_or_add..s%;;;!a3hhssA;;;r+)r r6)rrrfounds``` r)replace_or_addzDWhisperGenerationMixin._retrieve_init_tokens..replace_or_addsa....#.....E ;;;;;s;;; 3Jr+rrc |}|jvr|}n|tjvrdt|d}n|tjvrd|d}nit |dk}t d|d|r ttjnttjd|jvrt |dj|S)Nz<|z|>rzUnsupported language: z. Language should be one of: .z is not supported by this specific model as it is not in the `generation_config.lang_to_id`.(You should just add it to the generation config))lowerrrrrcrgr!r)rlanguage_tokenis_language_coders r)language_to_idzDWhisperGenerationMixin._retrieve_init_tokens..language_to_idsK~~''H,7<<>>>>!)-24444!D&6x&@!D!D!D-46666!2h!2!2!2#&x==A#5  rXrr;Kn-466777QUVfVkVmVmQnQnrrr%6%AAA %HHH %/? ?r+rrraiDue to a bug fix in https://github.com/huggingface/transformers/pull/28687 transcription using a multilingual Whisper will default to language detection followed by transcription instead of translation to English.This might be a breaking change for your use case. If you want to instead always translate your audio to English, make sure to pass `language='en'`.forced_decoder_idszYou have passed task=z,, but also have set `forced_decoder_ids` to zQ which creates a conflict. `forced_decoder_ids` will be ignored in favor of task=rzYou have passed language=zU which creates a conflict. `forced_decoder_ids` will be ignored in favor of language=zYou are using token ids in `forced_decoder_ids` that do not seem to correctly follow the prompt pattern of Whisper. Make sure that z) has an entry for all indices >= 1 and < c3K|]}|duV dSrIr])rKrs r)rNz?WhisperGenerationMixin._retrieve_init_tokens..As&//19//////r+zExpected `language` to be `None`, a single string (e.g. `'en'`), or a list of strings with length equal to the batch size (e.g. `('en', 'fr')` for a batch size of 2). Got a list containing `None`.zgWhen passing a list of languages, the length of the list must match the batch size. Expected length of z , but got z languages.c8g|]}tjSr])r1)rKr(rs r)r`z@WhisperGenerationMixin._retrieve_init_tokens..Rs#AAA!ty--AAAr+c&g|] }|Sr]r])rKrrs r)r`z@WhisperGenerationMixin._retrieve_init_tokens..Ws#===aq))===r+rr)rrrrzThe `z3`task is not supported. The task should be one of ``rc3,K|]}|vVdSrIr])rKtir>rs r)rNz?WhisperGenerationMixin._retrieve_init_tokens..vs,``B2Q/``````r+ transcriberrzm<|notimestamps|> prompt token is removed from generation_config since `return_timestamps` is set to `'True'`.cg|]}||SrIr])rKrCs r)r`z@WhisperGenerationMixin._retrieve_init_tokens..sIIIA1=a===r+r)"rrrstrrPrr warning_oncerdecoder_start_token_idrgr!rJrrr  TypeErrordetect_languagertolistr5r6rrrrcrrrrh as_tensorlongreexpand)rrrrrrrrrrris_lang_id_undefined languageslang_idstask_idr>rrs ` @@@r)rz,WhisperGenerationMixin._retrieve_init_tokenss> S   (3-     @S @S @ @ @ @ @ @,(&$77,j$??.A  )DL5G5J15M5U##kV1 2 2 ;v7P7\!'!:   )d.>   GGGZlGG@DGGG   "&    +0D   WHWWbtWWLTWWW   "& (?@  ).@.CA.F!.K.KA())A--2DQ2G2Ja2O2O 21 5a 899 %7%;"Q())A--2DQ2G2Ja2O2O %&&** s[mssXjklXmnoXpsss 04,";//14i[9I9IA9M9hR]^_R`dhRh hu . . #//h///// [8}} ** [*4[[@CH [[[!II  +II! IBAAAyAAA   ====9===HH & 5 5 :N ++- & +7P=P7P(QRF'-a0JJ  ('9F/9/?/[/[s"(++apqras .G1G J At{%* M M M6 7  ]__ W WTHHFHH6GHHHOPQPQPQSUPUVF W W W W W W W W W W W W W W Wq DDD EJ d,7>>@@AAB$&F7qqq- ==$$s C44C8;C8c|dd}|dd}||tddSdS)Nrassistant_modelzQPassing `decoder_input_ids` is deprecated. Consider passing `prompt_ids` instead.)rr!)rrrs r)rz/WhisperGenerationMixin._check_decoder_input_idssY"JJ':DAA **%6==  (_-Hc  ) (-H-Hr+c|rkt|dddkrtdt|dst d|dd|_dSdS)Nr translatez@Token-level timestamps may not be reliable for task 'translate'.rzModel generation config has no `alignment_heads`, token-level timestamps not available. See https://gist.github.com/hollance/42e32852f24243b748ae6bc1f985b13a on how to add this property to the generation config.r)rPrrrr!rrrs r)rz&WhisperGenerationMixin._set_num_framess " J(&$77;FFabbb,.?@@  R,2::lD+I+I  ( ( ( J Jr+c||nt|dd|_||nt|dd|_||nt|dd|_||nt|dd|_dS)Nrrrr)rPrrrrrs r)rz4WhisperGenerationMixin._set_thresholds_and_conditions!,  *,?FF ++6 ( '*,I4PP 5#. *,A4HH -(3 % $*,FMM 222r+cddg}|p|d}||vr(td|dd||jdur|dkrtd||_dS) Nr all-segmentsrz`prompt_condition_type=zD does not exist. Make sure to set `prompt_condition_type` to one of r0TzeMake sure to set `condition_on_prev_tokens=True` when setting `prompt_condition_type='all-segments'`.)r!joinrr)rrallowed_cond_typess r)rz1WhisperGenerationMixin._set_prompt_condition_types-~>!6 N9KA9N (: : :e*?eeFJFOFOPbFcFcee   5T A AF[_mFmFmw 3H///r+c>||nt|dd}||_dS)NrF)rPrrs r)r z4WhisperGenerationMixin._set_condition_on_prev_tokenss:(3 % $*,FNN ! 6N222r+c|dkr|s|td|dkrh|sf|dtj}t j|ftj}nEt j|ftj|z}t j|ftj}||fS)NrzWhen doing batched long-form audio transcription, make sure to pass an `attention_mask`. You can retrieve the `attention_mask` by doing `processor(audio, ..., return_attention_mask=True)` rr.)r!rrrvrhrrr2)rrrrrrs r)r z4WhisperGenerationMixin._retrieve_max_frames_and_seeks >>,>>3IO !^^L^'++B//335588DDJ; }EJ???DDZMDDDGYYJ; }EJ???D4r+c|jdurt||}||gn|g|z}|j(t|j|}||gn|g|z}d|_|j)t |j||}||gn|g|z}d|_|j>t|jdz ||dk} || gn| g|z}| ||S)NT)rrd)rrer)no_speech_tokenrscores_is_logprobs) rrrrbegin_suppress_tokensrrrr set_model) rrrQrrretimestamp_processorsuppress_tokens_processorbegin_suppress_processorno_speech_detectors r)rz1WhisperGenerationMixin._retrieve_logit_processors*s`  .$ 6 6"ABSal"m"m"m )9)A$%%H[G\_oGo   , 8(EFWFgpv(w(w(w %$++++/03CC  15  -  2 >'K!7[Y_((( $ $+***./2BB  7;  3  0 r cut_indexs r)rz*WhisperGenerationMixin._maybe_reduce_batchRsx 1 1A"1%FF|z&1118!34 1 !&N:I:,FW`cdWdWfWfHg+hno!p!p!p"((0000w(999r+c\|dSg}t|D]}||}|||dzdd||||||zf} | jd|kr&tj| d|| jdz f} || t j|d}|S)Nrrr)r$rb)r5r"rlr$r6rhri) rrrrrrrr>rsegment_input_slices r)rz)WhisperGenerationMixin._get_input_segmentbs  !4 w 6 6A"1%F"0QUAAAtF|dSYl]lms]tNt?t1t"u "(,/AAA&'e'a1CFYF_`bFc1c-d'''#  !4 5 5 5 5 -Q777 r+c  d| vr| d} | | fS|jdzdz } ||} t|dd} | | | dnd} trt ddkrfd|D}||jdkr|}n2t j|df|t j }| | |dznd}|j d krd nd }t||j |d ||| }t j || gd} | |j k| d<nn|V|d | jdd}t j || gd} | ddn| dd| | fS)NrrrrIrrc4g|]}|r|ndSrIr])rKr>rnrs r)r`zEWhisperGenerationMixin._prepare_decoder_input_ids..s2vvvcd6QRS6T^/22Z^vvvr+rrr/rYrVrX)rerprqrrrsrrbr0)rmax_target_positionsrPr rgrrhr2rr3ryrorirr")rrrnrrrrrrerrrrsprev_start_of_textactive_segmentsprev_ids one_tensorrq prev_tokenss ` ` r)rz1WhisperGenerationMixin._prepare_decoder_input_idsxs  & ( ( & +> ? ? $f, ,49A=' 6$%68KTRR  %8G8S!4!4Y]  * + + 74DQ4G0H0H10L0LvvvvvhuvvvO%*;*QUc*c*c%"Z! V5:VVV ASA_- 1 ==ei&7&LPX&X&Xll^gG,!.#!)-K!& ;8I*JPR S S S /@DUDb/bF+ , ,  #$T*112C2I!2LaPPK % ;8I*JPR S S S  JJ/ 6 6 6 6 JJ/ 6 6 6 &((r+c ||j|jnd}||jdz|jjkrLt d|jdd|d||jdzd|jjd|jjd t |jd zd z |jdd z }|jL|jEt |j|z|j}td |jd |d dS|j<|j|jdz|jkr |j|jdz }||_dSdSdS)NrrzjThe length of `decoder_input_ids`, including special start tokens, prompt tokens, and previous tokens, is z, and `max_new_tokens` is zL. Thus, the combined length of `decoder_input_ids` and `max_new_tokens` is: z@. This exceeds the `max_target_positions` of the Whisper model: z. You should either reduce the length of your prompt, or reduce the value of `max_new_tokens`, so that their combined length is less than rrrzIncrease max_length from z to z0 since input is conditioned on previous segment.) rr"rrr!minrYrr)rrrrrnum_initial_tokensrYs r)rz5WhisperGenerationMixin._set_max_new_tokens_and_lengths=N=]=i*99op -3B7 7$+:Z Z Zb~O~UVX~Ybb,:bb@NQbQhikQl@lbbAE @`bb ?Ck>^ bbb !!<!AA!EGXG^_aGbefGfgg  ' 38I8X8`.9.s'VVV 68 < <VVVr+)rmathlog2rrrgzlibcompress)r\r8 token_bytesrrs @r)rz2WhisperGenerationMixin._retrieve_compression_ratiosTYz**Q.//!3hhVVVVfmmooVVVWW  ,,s4=3M3M/N/NN  r+cX|dkr|nd}tj|j}|jdjdkr|djd}n|jd dt j||zd|jtfdtjdD}k dn jd}||dzz }|S)Nr-rrrrbc3ZK|]%}|||kzV&dSrIr])rKr>rrr\s r)rNz@WhisperGenerationMixin._retrieve_avg_logprobs..s?ttVWHQKq 2fQi<6OPttttttr+) rhrmrvrer"rl log_softmaxfloatr/rr5) rNr\rrrescale_temperature sum_logprobsr avg_logprobsrs `` @r)rz-WhisperGenerationMixin._retrieve_avg_logprobss5-83->->kkAV$$'' 66 <?V\!_ , ,-fl1o-.FFV\!_,../F=&+>">!E!E!G!GRPPPSSTZT`aatttttt[`aiaopqar[s[sttttt 5A5M&L(--b111SYS_`aSb#vz2 r+c ||} | ddddgk} tj| dd| ddzd} | d| r||dng} t | dkr| }g}| r"|t |d}|D]}|||}|d|z }|d|z }|||||zz||||zz|||d| r| ||||z|dd<|}| r ||}n||dz |z }||z}n||  }||}| dkr;|d|kr|d|z }||||||zz|||dg}| r| ||z|dd<||}||fS) NrFTrrrr)startendr\r() gerrhwhereadd_rgr6itemnonzeroflattennumel)rrrrrrrrrrtimestamp_tokenssingle_timestamp_endingtimestamp_segment_indicesrslicesr last_slice current_slice sliced_tokensstart_timestamp_posend_timestamp_posr last_timestamp_posrs r)rz(WhisperGenerationMixin._retrieve_segments*7)9)9/)J)J"2233"7">">"@"@UDM"Q$)K0@"0EHXYZY[Y[H\0\$]$]^_$`!!&&q)))D[c<,-?@@ac ( ) )A - -.5577FH& 2 c-00111J!' + + -j.F G &3A&6&;&;&=&=&O#$1"$5$:$:$<$<$N!!,X!69L~9]!]*847H>7YY"/".s"3 +(M)AB[QYEZZRL!34+ & C!0!: &3:>%B%G%G%I%IO%["!3l!B''7'?'?'A'A'I'I'K'KLJ!0!: !!A%%*R.*=*=*?*??*R*R%/^%8%8%:%:_%L")2&x03E3VV+*3/ H' \3CkRZF[3[ /0,X6N''r+)r|NN)NNNNNFNNNNNNNNNNNNNr|NFN)NNNr)3__name__ __module__ __qualname__rrrhrr rrrrrrrrr rr$r staticmethodr r6rr7r r2rrrrrr FloatTensorrrrrrrr r rrrrrrrr __classcell__)rGs@r)r{r{sWeioooof268<:><@W[!,0"48*.-1/337AE7;-1/3,015 $26 %261qq .q$$45q##67 q $$89 q #+8S%,4Gc4R+S"T qq$D>qsmq5d3i01q"$qU\*q (}q#+4.qeE5+<$<=>q &.e_!q"$E?#q$&e_%q&%SM'q(!.)q*+q,"*$-q./q0"*$1qqqqfViViViViVip  \ A-A-A-F#,#,#,J*+*+*+X   6ff\f  m m\ m  \ D''\'$@!*!*\!*FUiUiUir7;OS8<"& ?? !23?"%(9?(J"KL?$$45 ?  ?  ????B\ J J\ J  \ 8HH\H$NN\N   \  & & & P : :\ :\*;);)\;)z>>>6!!\!\$N(N(\N(N(N(N(N(r+r{)rUrVNN)1r1rrrtypingrrrrrrrr1rhtorch.nn.functionalr r#rltransformers.cache_utilsr generationr r generation.logits_processrrrrrgeneration.stopping_criteriarmodeling_outputsrutilsrtokenization_whisperrr get_loggerr"rrrr*rrFrTryr{r]r+r)r2s-  CCCCCCCCCCCCCCCC 888888;;;;;;;;A@@@@@//////<<<<<<<<  H % %5<su|*3&"*3&3&3&3&l ,,,,^M(M(M(M(M(_M(M(M(M(M(r+