gk&ddlZddlmZddlmZmZmZmZddlZ ddl Z ddl m Z ddl mZmZmZddlmZdd lmZerdd lmZdd lmZdd lmZejeZer ddlZdd lm Z dZ!ddZ"dZ#dZ$GddeZ%dZ&dS)N) defaultdict) TYPE_CHECKINGDictOptionalUnion)PreTrainedTokenizer)is_torch_availableis_torchaudio_availablelogging) ffmpeg_read) ChunkPipeline)BeamSearchDecoderCTC)SequenceFeatureExtractor)PreTrainedModel)(MODEL_FOR_SPEECH_SEQ_2_SEQ_MAPPING_NAMEScg}|D]\}}}tt||z}tt||z |z}tt||z |z}|||f}|||S)z Rescales the stride values from audio space to tokens/logits space. (160_000, 16_000, 16_000) -> (2000, 200, 200) for instance. )introundappend)strideratio new_stridesinput_nleftrighttoken_n new_strides o/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/pipelines/automatic_speech_recognition.pyrescale_strider!)sK &''ueGeO,,--5'12233E%'/G34455tU+ :&&&& c#^K|jd}||z |z }td||D]}||z} ||| } || |jd} || |} |dkrdn|} | |k} | rdn|}| jd}|| |f}| jd| kr | |d| V| rdSdS)Nrpt) sampling_ratereturn_tensorsdtype)is_lastr)shaperanger%to)inputsfeature_extractor chunk_len stride_left stride_rightr( inputs_lenstepchunk_start_idx chunk_end_idxchunk processed _stride_leftr) _stride_rightrs r chunk_iterr:=saJ { "\ 1D J55')3 }45%%e;L;Zkoppp  ! 5 11I+q00qqk :-$6, KN \=9 ;q>L ( (%EE9E E E E   EE r"c. t|}t| fdt|dzD}d}t|D]P}t D]>}||||kr*|||dz}|||dz|dz<||kr|}?Qtj|}|dkr'tj||kdd|z nd}|dkr'tj||kdd|z nd} || |fS)Nc"g|] }dgdzz S)rr ).0_ seq_len_rights r z6_fast_find_longest_common_sequence..Us%JJJQsma'(JJJr"r r)lenr+nparrayargwhere) sequence_leftsequence_right seq_len_leftcounterlongestijprevious_counter index_left index_rightr@s @r "_fast_find_longest_common_sequencerQRsL}%%L''MJJJJ% q8H2I2IJJJGG < //}%% / /AQ>!#444#*1:a=1#4 (8Aq1u%#g--.G  /hwGELPQ\\W/004Q7'AAWYJFMQRll"+g011"5a87BBXZK {G ++r"c Ffd|ddD}|ddD]}fd|dD}d}d}tdt|dzD]l}|dz }tjtj|| dtj|d|k} | |z |z} | dkr | |kr|}| }m|||dtj|S)Nc&g|] }|jv |Sr=all_special_idsr>tok_id tokenizers r rAz1_find_longest_common_sequence..ms&iii6yOhAhAhAhAhAhr"rr c&g|] }|jv |Sr=rTrVs r rAz1_find_longest_common_sequence..os'lll6FR[RkDkDkDkDkDkr"gg@)tolistr+rCrDsumrEextend) sequencesrXsequencenew_seq new_sequenceindexmax_rLepsmatchesmatchings ` r _find_longest_common_sequencerffs?jiiiYq\!_%;%;%=%=iiiHQRR= . .llllWQZ->->-@-@lll q#l++a/00  Ag+CfRXhrssm44bqbAQ8R8RRSSG{S(H{{x$ UVV,---- 8H  r"ceZdZdZ ddddedefdeedeed efd eed fd eeed ff fd Z dee j e efffd Z ddZddZddZ ddeefdZxZS)"AutomaticSpeechRecognitionPipelinea Pipeline that aims at extracting spoken text contained within some audio. The input can be either a raw waveform or a audio file. In case of the audio file, ffmpeg should be installed for to support multiple audio formats Example: ```python >>> from transformers import pipeline >>> transcriber = pipeline(model="openai/whisper-base") >>> transcriber("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac") {'text': ' He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick, peppered flour-fatten sauce.'} ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) Arguments: model ([`PreTrainedModel`] or [`TFPreTrainedModel`]): The model that will be used by the pipeline to make predictions. This needs to be a model inheriting from [`PreTrainedModel`] for PyTorch and [`TFPreTrainedModel`] for TensorFlow. feature_extractor ([`SequenceFeatureExtractor`]): The feature extractor that will be used by the pipeline to encode waveform for the model. tokenizer ([`PreTrainedTokenizer`]): The tokenizer that will be used by the pipeline to encode data for the model. This object inherits from [`PreTrainedTokenizer`]. decoder (`pyctcdecode.BeamSearchDecoderCTC`, *optional*): [PyCTCDecode's BeamSearchDecoderCTC](https://github.com/kensho-technologies/pyctcdecode/blob/2fd33dc37c4111417e08d89ccd23d28e9b308d19/pyctcdecode/decoder.py#L180) can be passed for language model boosted decoding. See [`Wav2Vec2ProcessorWithLM`] for more information. chunk_length_s (`float`, *optional*, defaults to 0): The input length for in each chunk. If `chunk_length_s = 0` then chunking is disabled (default). For more information on how to effectively use `chunk_length_s`, please have a look at the [ASR chunking blog post](https://huggingface.co/blog/asr-chunking). stride_length_s (`float`, *optional*, defaults to `chunk_length_s / 6`): The length of stride on the left and right of each chunk. Used only with `chunk_length_s > 0`. This enables the model to *see* more context and infer letters better than without this context but the pipeline discards the stride bits at the end to make the final reconstitution as perfect as possible. For more information on how to effectively use `stride_length_s`, please have a look at the [ASR chunking blog post](https://huggingface.co/blog/asr-chunking). framework (`str`, *optional*): The framework to use, either `"pt"` for PyTorch or `"tf"` for TensorFlow. The specified framework must be installed. If no framework is specified, will default to the one currently installed. If no framework is specified and both frameworks are installed, will default to the framework of the `model`, or to PyTorch if no model is provided. device (Union[`int`, `torch.device`], *optional*): Device ordinal for CPU/GPU supports. Setting this to `None` will leverage CPU, a positive will run the model on the associated CUDA device id. torch_dtype (Union[`int`, `torch.dtype`], *optional*): The data-type (dtype) of the computation. Setting this to `None` will use float32 precision. Set to `torch.float16` or `torch.bfloat16` to use half-precision in the respective dtypes. Nmodelrr.rrXdecoderrdevicez torch.device torch_dtypez torch.dtypec 6|jjdkrd|_n`|jjt jvrd|_n9|jr+|jdr|||_ d|_nd|_tj |||f||d|dS)Nwhisperseq2seq_whisperseq2seqWithLM ctc_with_lmctc)rkrl) config model_typetype __class____name__rvalues_processor_classendswithrjsuper__init__) selfrir.rXrjrkrlkwargsrws r r}z+AutomaticSpeechRecognitionPipeline.__init__s < "i / /)DII _ %)Q)X)Z)Z Z Z!DII  . !2;;HEE #"DL%DIIDI +<oVYdoohnooooor"r-c 8tj|fi|S)ae Transcribe the audio sequence(s) given as inputs to text. See the [`AutomaticSpeechRecognitionPipeline`] documentation for more information. Args: inputs (`np.ndarray` or `bytes` or `str` or `dict`): The inputs is either : - `str` that is either the filename of a local audio file, or a public URL address to download the audio file. The file will be read at the correct sampling rate to get the waveform using *ffmpeg*. This requires *ffmpeg* to be installed on the system. - `bytes` it is supposed to be the content of an audio file and is interpreted by *ffmpeg* in the same way. - (`np.ndarray` of shape (n, ) of type `np.float32` or `np.float64`) Raw audio at the correct sampling rate (no further check will be done) - `dict` form can be used to pass raw audio sampled at arbitrary `sampling_rate` and let this pipeline do the resampling. The dict must be in the format `{"sampling_rate": int, "raw": np.array}` with optionally a `"stride": (left: int, right: int)` than can ask the pipeline to treat the first `left` samples and last `right` samples to be ignored in decoding (but used at inference to provide more context to the model). Only use `stride` with CTC models. return_timestamps (*optional*, `str` or `bool`): Only available for pure CTC models (Wav2Vec2, HuBERT, etc) and the Whisper model. Not available for other sequence-to-sequence models. For CTC models, timestamps can take one of two formats: - `"char"`: the pipeline will return timestamps along the text for every character in the text. For instance, if you get `[{"text": "h", "timestamp": (0.5, 0.6)}, {"text": "i", "timestamp": (0.7, 0.9)}]`, then it means the model predicts that the letter "h" was spoken after `0.5` and before `0.6` seconds. - `"word"`: the pipeline will return timestamps along the text for every word in the text. For instance, if you get `[{"text": "hi ", "timestamp": (0.5, 0.9)}, {"text": "there", "timestamp": (1.0, 1.5)}]`, then it means the model predicts that the word "hi" was spoken after `0.5` and before `0.9` seconds. For the Whisper model, timestamps can take one of two formats: - `"word"`: same as above for word-level CTC timestamps. Word-level timestamps are predicted through the *dynamic-time warping (DTW)* algorithm, an approximation to word-level timestamps by inspecting the cross-attention weights. - `True`: the pipeline will return timestamps along the text for *segments* of words in the text. For instance, if you get `[{"text": " Hi there!", "timestamp": (0.5, 1.5)}]`, then it means the model predicts that the segment "Hi there!" was spoken after `0.5` and before `1.5` seconds. Note that a segment of text refers to a sequence of one or more words, rather than individual words as with word-level timestamps. generate_kwargs (`dict`, *optional*): The dictionary of ad-hoc parametrization of `generate_config` to be used for the generation call. For a complete overview of generate, check the [following guide](https://huggingface.co/docs/transformers/en/main_classes/text_generation). Return: `Dict`: A dictionary with the following keys: - **text** (`str`): The recognized text. - **chunks** (*optional(, `List[Dict]`) When using `return_timestamps`, the `chunks` will become a list containing all the various text chunks identified by the model, *e.g.* `[{"text": "hi ", "timestamp": (0.5, 0.9)}, {"text": "there", "timestamp": (1.0, 1.5)}]`. The original full text can roughly be recovered by doing `"".join(chunk["text"] for chunk in output["chunks"])`. )r|__call__)r~r-rrws r rz+AutomaticSpeechRecognitionPipeline.__call__s%z uww11&111r"c i} |,|jdkr|std|| d<||| d<tt} |t jdt|| d<|*|d|vrtd| |i} ||| d<||jdkr|rtd |jd kr|d krtd |jd kr|dvrtd|jdkr|dkrtd|| d<|| d<||jdkrtd|| d<| | | fS)NrpaUsing `chunk_length_s` is very experimental with seq2seq models. The results will not necessarily be entirely accurate and will have caveats. More information: https://github.com/huggingface/transformers/pull/20104. Ignore this warning with pipeline(..., ignore_warning=True)chunk_length_sstride_length_sz`max_new_tokens` is deprecated and will be removed in version 4.49 of Transformers. To remove this warning, pass `max_new_tokens` as a key inside `generate_kwargs` instead.max_new_tokenszp`max_new_tokens` is defined both as an argument and inside `generate_kwargs` argument, please use only 1 versiondecoder_kwargszEWe cannot return_timestamps yet on non-CTC models apart from Whisper!rrwordzRCTC with LM can only predict word level timestamps, set `return_timestamps='word'`rs)charrzCTC can either predict character level timestamps, or word level timestamps. Set `return_timestamps='char'` or `return_timestamps='word'` as required.rorzWhisper cannot return `char` timestamps, only word level or segment level timestamps. Use `return_timestamps='word'` or `return_timestamps=True` respectively.return_timestampsz)Only Whisper can return language for now.return_language) rvloggerwarningrdictwarningswarn FutureWarning ValueErrorupdate) r~rrignore_warningrrrgenerate_kwargsrpreprocess_paramsforward_paramspostprocess_paramss r _sanitize_parametersz7AutomaticSpeechRecognitionPipeline._sanitize_parameterss  %yI%%n%, 3A . /  &3B / 0$T**  % M   0>N+ ,  &).>/.Q.Q &  ! !/ 2 2 2  %3A / 0  (yI%%*;% !hiiiyM)).?6.I.I !uvvvyE!!&7?O&O&O `y---2Cv2M2M _3DN. /6G 2 3  &y--- !LMMM4C 0 1 .2DDDr"rc# Kt|tr|ds|drtj|j}nfr||g}tA|j!j"dd } t1t3||j j z| z | z} t1t3|d |j j z| z | z} t1t3|d |j j z| z | z}| | |zkrtdtG||j | | ||j$D]}|VdS|jdkrA|jd |j j%kr&| ||j j dddd}nm|jdkr?|=| ||j j ddd}| d|d<n#| ||j j dd}|j$|&|j$}||jdkrtd ||d<d!di||VdS)"Nzhttp://zhttps://rbrr%rawrEzWhen passing a dictionary to AutomaticSpeechRecognitionPipeline, the dict needs to contain a "raw" key containing the numpy array representing the audio and a "sampling_rate" key, containing the sampling_rate associated with that arraypathr) functionalztorchaudio is required to resample audio samples in AutomaticSpeechRecognitionPipeline. The torchaudio package can be installed through: `pip install torchaudio`.r zStride is too large for inputz)We expect a numpy ndarray as input, got ``zMWe expect a single channel audio input for AutomaticSpeechRecognitionPipelineinputs_to_logits_ratioz.Chunk length must be superior to stride lengthroFrKr$T)r% truncationpaddingr&return_attention_mask)r%r&return_token_timestampsr num_frames)r%r&rr'rpz8Stride is only usable with CTC models, try removing it !r))' isinstancestr startswithrequestsgetcontentopenreadbytesrr.r%rpoprr torchaudior ImportErrorresampletorch from_numpynumpyr*rrrDndarray TypeErrorrvrCfloatgetattrrirtr:rl n_samplesr,)r~r-rrfrextra_inputsin_sampling_rateFralign_tor/r0r1itemr7s r preprocessz-AutomaticSpeechRecognitionPipeline.preprocessas fc " " &  ++ &v/@/@/L/L &"f--5&$''&1VVXXF&&&&&&&&&&&&&&& fe $ $ O )?)MNNF fd # #* iZZ$//F$v--5F??gQWFWFW N jj--G 64((( **Wd33%zz/:: EF4#9#GGG*,,:::::::%e $V,,.>@V@d%''.d|vr|d}n$td ||rgjd kr\||d <|d krQd |d<d |d<|@t |t r|djjz|d<nfd|D|d<n||d<d|vr j|d<j j d||d|} |d kr?jd kr4d| vr| d| dd} n"d| dD} | d| d} nd| i} jd kr||| d<n‰j j |j j d|i}j di|} | j } jdkrd| i} nd| di} |Wdj jjz }t |t rt!|g|d| d<nt!||| d<|}d|i| |S) Nattention_maskrrr)z0num_frames must be used only when stride is Nonerproinput_features input_valueszhSeq2Seq speech recognition model requires either a `input_features` or `input_values` key, but only has rorrTrreturn_segmentsrc:g|]}|djjzS)r)r. hop_length)r>sr~s r rAz?AutomaticSpeechRecognitionPipeline._forward..s)segments r rAzJAutomaticSpeechRecognitionPipeline._forward...s"["["[77+=#>"["["[r")rcat)r> segment_lists r rAz?AutomaticSpeechRecognitionPipeline._forward.. sA(((( "["[l"["["[\\(((r"rrrlogitsrB)dimr r=)rrrvkeysrtupler.rrrigeneratemain_input_namerargmaxrtrr!)r~ model_inputsrrrrrr)r-routroutputsrrrs` r _forwardz+AutomaticSpeechRecognitionPipeline._forwardsx%))*:DAA!!(D11!%%lD99 ""9--  *"8OPP P 96 6 6 <//%))*:;;<//%)).99 bLXL]L]L_L_bb ! CTY2C%C%C7H 34$..AEO$=>9=O$56)%fe44x((($&,K%8N^__CC(y---%$*CM *L,<,rsrrrpro)rrtime_precisionr )axisrr)r start_offset end_offsetrs)skip_special_tokensT)routput_char_offsets char_offsetsrrrr)text timestampchunksr)rr)!rv frameworkr(rbfloat16float16r,float32rrrrfrXr. chunk_lengthrirtmax_source_positionsr% _decode_asrrD concatenatesqueezerj decode_beamsdecode_get_word_offsetsreplace_word_delimiter_charrrlistritems)$r~ model_outputsrrroptional final_itemskeyrrrtotal_nrrright_nrr%outputr/r0r1rbeams chunk_offsetoffsetsrrrrrrstartstoprkvs$ r postprocessz.AutomaticSpeechRecognitionPipeline.postprocess1s )}44hh($ & &G~%%'#,*<QVQ^@_*_*_  66<<>> **,,[[400F!di3I&I&I'-$u "E/aaago.   u % % % %  %di9,,1+t~NNEE Y+ + +!3@4:CTCiiN 2@M' L Lv%%;A(;K8I{L.I=0K M1L'0+|'KF8$!^77"3 /- 8ND((N;Q777EMM!$$E 9 % %%!#-DL-eFF~FFE8A;D  k %Qx{ 8Dkk4D4<NND,^h#i#ijjjj Y+ + +"&)u"4 >((DW(XXD  t.///BX\0 "%.."n>>wHrssG  (2P!P!PF ] ]^,tz/@/WW/==L)DJ,=,TT.<< t,='>eUY][[\\\\!'HX D!!# # #F JJx & & & JJx & & & JJy$ ' ' ' JJx & & & JJ)4 0 0 0  # #1a"""" #22E22r")NNNNN)NNNNNNNN)rN)F)NNN)rx __module__ __qualname____doc__rrrr rr}rDrrrrrrrr __classcell__)rws@r rhrhsAALEI37@D-1;?pp p!!;S!@Ap/0 p % 6 ;<= p c>)* peC$678pppppp6=2bj%,-=2=2=2=2=2=2BBEBEBEBEHw:w:w:w:rU4U4U4U4pmq]3]3-5d^]3]3]3]3]3]3]3]3r"rhc |ddz}g}|j|z }d}t|D]g\}} | \} } t| trt j| } | \} } }| d} || vr$t j| |kddnd}| |d} | |k}|dkrt|dkrt j|dd|ddzddz}t j|dd}||vrt j ||n|}|| |zz}t||j z |z }t| |j z |z }t j| |||zkd}|j ddkr|ddkr||ddz n|d}d}g}tt|D]F\}}|dd}|d||z|z kr |dkrnt|dkrt!| d||\}}}|dkr||kr|}|}t j| |dzd|kdddz}|dz|z}|dkrF|t|kr3t j| |dz|d|d}|d|d<|dkrO| |dz|}|dkr |d|dzn|dg} t j|d| }|dxx|z cc<Ht|dkr<||t||z dz <|dt||z }| |d} | |k}t j|dd|ddzddz}t|dkr>t j|dd}||vrt j ||dzn|}t|dkrhd}!|D]c}"|dks|!dkr|ddn| d}#| |!|"}$|$d|$dz }%|#|$d<|#|%z|$d<| |$|"}!d|| z }ig}&t%t|D]}'|&||'z }& |&S)a Computes the final sequences by merging the end of the nth sequence with the beginning of the n+1th sequence. Since `WhisperForConditionalGeneration` produces the timestamps pairwise, we filter the consecutive timestamps and only iterate over them. We keep track of the `time` which indicates the actual starting time of the chunk that is processed. We need to make sure to offset the timestamps tokens by the `time` in order for the tokenizer to properly compute the final `offset`. z<|notimestamps|>r rNrB)convert_tokens_to_idsr enumeraterrrDrErwherer[rrr%r*reversedrCrQinsertr+rZ)(r]rXr.rtimestamp_beginrrtimeseq_idxrr^rr/r0r1 begin_idxtimestamp_tokens consecutivelast_timestampoffset overlap_timerelevant_timestamp best_matchsliced_sequenceidxprevious_sequenceprevious_tokensrOrP match_lengthbest_idxend_of_curr_sequence_idxprevious_slice last_slice current_slice actual_offset sliced_tokensdurationresultrLs( r _find_timestamp_sequencer,s 556HIIAMO E&36JJN D"9--UU & h % % *x))H/5, ; ##A&&CRV^C^C^BHX899!NDRZeDeDe")K@@@kvK K,. .D$!2!@@NRSSF .?.M MQ__``L!#(;*??UaCa*a!b!bcd!e !'*Q..>PQR>SVW>W>WK 21 5 9::]hij]k# "$.7.H.H>>*C*&7"&=O(+/G,/VWW\_cd\d\d?++a//@b$Q'9%9:OAA= K(!++ z0I0I)5J'*H "*q.2B2B)C)V W WXY Z[\ ]`a a58PRS7SV`7`4)Q<3CWCW3W3W24)$,Z!^>V-V$WYZ\mno\p3"3"7H6K 3 3!+q2::>Ld;d2eMXZ[OO$56G a6G$H$Hbstubvaw!/35)OQP^2_2_ / 3 3 3v = 3 3 3''!++7FE#e**x/!34!"9CJJ$9"9:E'(@(A(ABH$6h/47G7KKLLQORSS  1 $ $X&677:2>N>LT_>_>_ +~'9:::ep  {  a  J!, + + 18Aqb " V^_`Va (M)A B (,}Q/??#0 a $1H$< b! ]+++*   F 3u::  $$%(//### Mr")N)'r collectionsrtypingrrrrrrDrtokenization_utilsr utilsr r r audio_utilsrbaser pyctcdecoder!feature_extraction_sequence_utilsrmodeling_utilsr get_loggerrxrrmodels.auto.modeling_autorr!r:rQrfrhr,r=r"r r8s######777777777777444444HHHHHHHHHH$$$$$$1000000LLLLLL000000  H % %ULLLTTTTTT(*,,,(2O3O3O3O3O3O3O3O3dgggggr"