Nga*ddlZddlZddlZddlZddlmZmZmZmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZejeZGddeZGd d eZGd d eZGd deZGddeZdS)N)AnyCallableDictIteratorLiteralOptionalTupleUnion)Document)BaseBlobParser)Blob) is_openai_v1c$eZdZdZdddddddddd deedeedeedeegefdfd eed eed eed dfd ee dede fdZ e de eeffdZdedeefdZdS)AzureOpenAIWhisperParserav Transcribe and parse audio files using Azure OpenAI Whisper. This parser integrates with the Azure OpenAI Whisper model to transcribe audio files. It differs from the standard OpenAI Whisper parser, requiring an Azure endpoint and credentials. The parser is limited to files under 25 MB. **Note**: This parser uses the Azure OpenAI API, providing integration with the Azure ecosystem, and making it suitable for workflows involving other Azure services. For files larger than 25 MB, consider using Azure AI Speech batch transcription: https://learn.microsoft.com/azure/ai-services/speech-service/batch-transcription-create?pivots=rest-api#use-a-whisper-model Setup: 1. Follow the instructions here to deploy Azure Whisper: https://learn.microsoft.com/azure/ai-services/openai/whisper-quickstart?tabs=command-line%2Cpython-new&pivots=programming-language-python 2. Install ``langchain`` and set the following environment variables: .. code-block:: bash pip install -U langchain langchain-community export AZURE_OPENAI_API_KEY="your-api-key" export AZURE_OPENAI_ENDPOINT="https://your-endpoint.openai.azure.com/" export OPENAI_API_VERSION="your-api-version" Example Usage: .. code-block:: python from langchain.community import AzureOpenAIWhisperParser whisper_parser = AzureOpenAIWhisperParser( deployment_name="your-whisper-deployment", api_version="2024-06-01", api_key="your-api-key", # other params... ) audio_blob = Blob(path="your-audio-file-path") response = whisper_parser.lazy_parse(audio_blob) for document in response: print(document.page_content) Integration with Other Loaders: The AzureOpenAIWhisperParser can be used with video/audio loaders and `GenericLoader` to automate retrieval and parsing. YoutubeAudioLoader Example: .. code-block:: python from langchain_community.document_loaders.blob_loaders import ( YoutubeAudioLoader ) from langchain_community.document_loaders.generic import GenericLoader # Must be a list youtube_url = ["https://your-youtube-url"] save_dir = "directory-to-download-videos" loader = GenericLoader( YoutubeAudioLoader(youtube_url, save_dir), AzureOpenAIWhisperParser(deployment_name="your-deployment-name") ) docs = loader.load() N) api_keyazure_endpoint api_versionazure_ad_token_providerlanguagepromptresponse_format temperature max_retriesrrrrrrrjsontextsrt verbose_jsonvttrdeployment_namerc |ptjd|_|ptjd|_|ptjd|_||_||_||_||_ ||_ | |_ | |_ ddl } n#t$rtdwxYwtr:| |j|j|j|j |j|_dS|jr |j| _|jr |j| _|jr |j| _d| _| |_dS) a Initialize the AzureOpenAIWhisperParser. Args: api_key (Optional[str]): Azure OpenAI API key. If not provided, defaults to the `AZURE_OPENAI_API_KEY` environment variable. azure_endpoint (Optional[str]): Azure OpenAI service endpoint. Defaults to `AZURE_OPENAI_ENDPOINT` environment variable if not set. api_version (Optional[str]): API version to use, defaults to the `OPENAI_API_VERSION` environment variable. azure_ad_token_provider (Union[Callable[[], str], None]): Azure Active Directory token for authentication (if applicable). language (Optional[str]): Language in which the request should be processed. prompt (Optional[str]): Custom instructions or prompt for the Whisper model. response_format (Union[str, None]): The desired output format. Options: "json", "text", "srt", "verbose_json", "vtt". temperature (Optional[float]): Controls the randomness of the model's output. deployment_name (str): The deployment name of the Whisper model. max_retries (int): Maximum number of retries for failed API requests. Raises: ImportError: If the required package `openai` is not installed. AZURE_OPENAI_API_KEYAZURE_OPENAI_ENDPOINTOPENAI_API_VERSIONrNEopenai package not found, please install it with `pip install openai`)rrrrazure_ad_tokenazure)osenvirongetrrrrrrrrr!ropenai ImportErrorr AzureOpenAI_clientapi_baseapi_type) selfrrrrrrrrr!rr,s n/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/document_loaders/parsers/audio.py__init__z!AzureOpenAIWhisperParser.__init__Vsm`H"*..1G"H"H ,W ?V0W0W&N"*..9M*N*N'>$   .&.&  MMMM   '   >> "!-- #2 , ,#; .DLLL| .!%" 6"&"5 6%)%5"%FO!DLLLs %B**Creturnct|j|j|j|jd}d|DS)Nrrrrci|] \}}||| SN.0kvs r3 z;AzureOpenAIWhisperParser._create_params..AAAA1=1===rrrritemsr2paramss r3_create_paramsz'AzureOpenAIWhisperParser._create_paramsE k#3+    BAAAAArAblobc#Ktt|jd} tr*|jjjjd|j|d|j }n*|jj j d|j|j|d|j }n#t$rwxYwtt|ts|jn|d|jiVdS)at Lazily parse the provided audio blob for transcription. Args: blob (Blob): The audio file in Blob format to be transcribed. Yields: Document: Parsed transcription from the audio file. Raises: Exception: If an error occurs during transcription. rbmodelfile)rL deployment_idrMsource page_contentmetadataNr:)openstrpathrr/audiotranscriptionscreater!rFAudio transcribe Exceptionr isinstancerrO)r2rHfile_obj transcripts r3 lazy_parsez#AzureOpenAIWhisperParser.lazy_parses""DI-- ~~ ET\/>E.!) ;T\/:."&"6!)       j#.. ,         s A"B B)__name__ __module__ __qualname____doc__rrTr rrfloatintr4propertyrrrFr rr r_r:rAr3rrslCCP"&(,%)BF"& $ '+U"U"U"#U"! U" c] U" "'xC'8$'>!? U"3-U" U" @ A4 G U"e_U"U"U"U"U"U"nBS#XBBBXB* t* (:* * * * * * rArceZdZdZ dddddddddeededeedeedfd eedfd eed dfd eedffd Z e de ee ffdZ dedeefdZdS)OpenAIWhisperParseraTranscribe and parse audio files. Audio transcription is with OpenAI Whisper model. Args: api_key: OpenAI API key chunk_duration_threshold: Minimum duration of a chunk in seconds NOTE: According to the OpenAI API, the chunk duration should be at least 0.1 seconds. If the chunk duration is less or equal than the threshold, it will be skipped. Ng?)chunk_duration_thresholdbase_urlrrrrrrirjrrrrrc||_||_||ntjd|_||_||_||_||_ dS)NOPENAI_API_BASE) rrir)r*r+rjrrrr)r2rrirjrrrrs r3r4zOpenAIWhisperParser.__init__s[ (@% ,HH"*..AR2S2S !  .&rAr5ct|j|j|j|jd}d|DS)Nr7ci|] \}}||| Sr9r:r;s r3r?z6OpenAIWhisperParser._create_params..r@rArBrDs r3rFz"OpenAIWhisperParser._create_paramsrGrArHc #K ddl}n#t$rtdwxYw ddlm}n#t$rtdwxYwt r"||j|j}n&|jr |j|_|jr |j|_| |j }d}|dzd z}ttdt||D]\}} || | |z} | j|jkr$t!j| d } |j|jd |d z| _n d|d | _t-d|dzdd} | dkr t r |jjjdd| d|j} n|jd| } ni#t:$rE}| dz } t-d| dt=|t?j dYd}~nd}~wwxYw| dkt-dVtCtE| t<s| j#n| |j|dVdS)Lazily parse the blob.rNr& AudioSegmentCpydub package not found, please install it with `pip install pydub`)rrj<mp3format_part_z.mp3part_Transcribing part !rz whisper-1rKzAttempt z failed. Exception: z&Failed to transcribe after 3 attempts.)rOchunkrPr:)$r,r-pydubrrrOpenAIrrjr0 from_filerU enumeraterangelenduration_secondsriioBytesIOexportreadrOnameprintrVrWrXrFrYrZr[rTtimesleepr r\r)r2rHr,rrclientrVchunk_durationchunk_duration_ms split_numberirr]attemptsr^es r3r_zOpenAIWhisperParser.lazy_parses7  MMMM   '     * * * * * * *   X   >> 0]]4<$-]PPFF| .!%} 0"&-&&ty11*R/$6 )q#e**>O)P)PQQ% % OL!!a"3334E%)FFFz%,,e,"<"<"A"A"C"CDDH{& $ .I|.I.I.I I : : : :  :|a'7::: ; ; ;HQ,, "#~~T%GV\%@%G&"-H&&@D@S&& &,\%<%<[(%S%S  """MHKXKK3q66KKLLLJqMMMMMMMM"Q,,>???!*c22 Z__$(K,GG     A% % s* #.AA G H,';H''H,r9)r`rarbrcrrTrdr rr4rfrrrFr rr r_r:rAr3rhrhs9  "&'+."&%)#' *.'''#'#( ' 3- ' T " 'c4i ' @ A4 G '5$;'''''.BS#XBBBXBItI(:IIIIIIrArhc zeZdZdZ ddedeeded ed eeef d Z d e d e e fdZ dS)OpenAIWhisperParserLocala;Transcribe and parse audio files with OpenAI Whisper model. Audio transcription with OpenAI Whisper model locally from transformers. Parameters: device - device to use NOTE: By default uses the gpu if available, if you want to use cpu, please set device = "cpu" lang_model - whisper model to use, for example "openai/whisper-medium" forced_decoder_ids - id states for decoder in multilanguage model, usage example: from transformers import WhisperProcessor processor = WhisperProcessor.from_pretrained("openai/whisper-medium") forced_decoder_ids = WhisperProcessor.get_decoder_prompt_ids(language="french", task="transcribe") forced_decoder_ids = WhisperProcessor.get_decoder_prompt_ids(language="french", task="translate") 0Ndevice lang_model batch_size chunk_lengthforced_decoder_idsc ddlm}n#t$rtdwxYw ddl}n#t$rtdwxYw|dkrd|_n"|jrdnd|_|jdkrd}|r|n||_nO|j|jj d z } | d krd} n| d krd } n | d krd} nd} |r|n| |_td|j||_ |d|j||j|_ |O ||j j j_dS#t $r)} t"d| dYd} ~ dSd} ~ wwxYwdS)aInitialize the parser. Args: device: device to use. lang_model: whisper model to use, for example "openai/whisper-medium". Defaults to None. forced_decoder_ids: id states for decoder in a multilanguage model. Defaults to None. batch_size: batch size used for decoding Defaults to 8. chunk_length: chunk length used during inference. Defaults to 30s. r)pipelinezQtransformers package not found, please install it with `pip install transformers`NCtorch package not found, please install it with `pip install torch`cpuzcuda:0zopenai/whisper-baseiXzopenai/whisper-smalli.zopenai/whisper-mediumzopenai/whisper-largezUsing the following model: zautomatic-speech-recognition)rLchunk_length_srzOUnable to set forced_decoder_ids parameter for whisper modelText of exception: z9Therefore whisper model will use default mode for decoder) transformersrr-torchrcuda is_availablerget_device_properties total_memoryrrpiperLconfigrr[loggerinfo) r2rrrrrrr default_modelmem rec_modelexception_texts r3r4z!OpenAIWhisperParserLocal.__init__us'*  - - - - - - -   -     LLLL   X   U??DKK&+j&=&=&?&?J((UDK ;%  1M,6IjjMDOO*224;??LPWXCTzz1 t2 u3 2 ,6EjjIDO +T_===$H */';      ) )sr)rrrOrP)rrrr-librosarrUrrrrrloadrcopyrr rO) r2rHrrrrVr]yr predictions r3r_z#OpenAIWhisperParserLocal.lazy_parsese  * * * * * * *   U     NNNN   (  &&ty11:ell%l88==??@@ /49///000 X% 002YYqvvxxDOYDDVL # ,        s %.A)rNrrN)r`rarbrcrTrrer rr4r rr r_r:rAr3rr^s0$(48 LLLSML L  L %U4[1 LLLL\! t! (:! ! ! ! ! ! rArc jeZdZdZddddddeedeeded efd Zd ed ee fd Z dS)YandexSTTParserzWTranscribe and parse audio files. Audio transcription is with OpenAI Whisper model.Ngeneralauto)r iam_tokenrLrrrrLrcp|du|dukrtd||_||_||_||_dS)aInitialize the parser. Args: api_key: API key for a service account with the `ai.speechkit-stt.user` role. iam_token: IAM token for a service account with the `ai.speechkit-stt.user` role. model: Recognition model name. Defaults to general. language: The language in ISO 639-1 format. Defaults to automatic language recognition. Either `api_key` or `iam_token` must be provided, but not both. Nz?Either 'api_key' or 'iam_token' must be provided, but not both.) ValueErrorrrrLr)r2rrrLrs r3r4zYandexSTTParser.__init__sO* tOd!2 3 3Q  "   rArHr5c#lK ddlm}m}m}ddlm}n#t $rt dwxYw ddlm}n#t $rt dwxYw|j r&|| |j n%|| |j | |j }|}|j|_|j|_|j|_||} | D]!} t)| jd |ji V"d S) rpr)configure_credentialscredsmodel_repository)AudioProcessingTypezYyandex-speechkit package not found, please install it with `pip install yandex-speechkit`rqrs)r)yandex_credentials)rrOrPN) speechkitrrr speechkit.sttrr-rrrrYandexCredentialsrrrUrecognition_modelrLrFullaudio_processing_typerZr normalized_textrO) r2rHrrrrrrrVrLresultress r3r_zYandexSTTParser.lazy_parse s  P P P P P P P P P P 9 9 9 9 9 9 9   1     * * * * * * *   X   <  ! !#(#:#:4<#:#P#P      " !#(#:#:T^#:#T#T    &&ty11 2244j &9&>#!!%((  C 0"DK0      s/:A r`rarbrcrrTr4r rr r_r:rAr3rrs99 "&#' !!!#!C= !  !  !!!!<)t)(:))))))rArc^eZdZdZddddeedeefdZded ee fd Z dS) FasterWhisperParseraTranscribe and parse audio files with faster-whisper. faster-whisper is a reimplementation of OpenAI's Whisper model using CTranslate2, which is up to 4 times faster than openai/whisper for the same accuracy while using less memory. The efficiency can be further improved with 8-bit quantization on both CPU and GPU. It can automatically detect the following 14 languages and transcribe the text into their respective languages: en, zh, fr, de, ja, ko, ru, es, th, it, pt, vi, ar, tr. The gitbub repository for faster-whisper is : https://github.com/SYSTRAN/faster-whisper Example: Load a YouTube video and transcribe the video speech into a document. .. code-block:: python from langchain.document_loaders.generic import GenericLoader from langchain_community.document_loaders.parsers.audio import FasterWhisperParser from langchain.document_loaders.blob_loaders.youtube_audio import YoutubeAudioLoader url="https://www.youtube.com/watch?v=your_video" save_dir="your_dir/" loader = GenericLoader( YoutubeAudioLoader([url],save_dir), FasterWhisperParser() ) docs = loader.load() rN)r model_sizerrc ddl}n#t$rtdwxYw|dkrd|_n"|jrdnd|_|jdkrd|_nX|j|jjdz }|dkrd|_n#|d krd |_n|d krd |_nd |_| |dvr ||_dSdSdS)a$Initialize the parser. Args: device: It can be "cuda" or "cpu" based on the available device. model_size: There are four model sizes to choose from: "base", "small", "medium", and "large-v3", based on the available GPU memory. rNrrrbaserrvi smallrmediumlarge-v3)rrrr)rr-rrrrrr)r2rrrrs r3r4zFasterWhisperParser.__init__Xs  LLLL   U   U??DKK$)J$;$;$=$=H&&5DK ;%  $DOO*224;??LPWXCTzz"(t")t"*",  !DDD", " !DDs!rHr5c #(K ddlm}n#t$rtdwxYw ddlm}n#t$rtdwxYwt |jtr-|tj |j}n8|j"|j r||j }ntdtj | d }||j|jd }||d \}}|D]W} t%| j|jd| j| jfz|jdt1|jdzzd|jVXdS)rprrqrs) WhisperModelzUfaster_whisper package not found, please install it with `pip install faster-whisper`NzUnable to get audio from blobrwrxfloat16)r compute_typer) beam_sizez[%.2fs -> %.2fs]z%d%%d)rO timestampsr probabilityrP)rrrr-faster_whisperrr\databytesrrrrUrrrrrrZr rrOstartendrroundlanguage_probabilityrR) r2rHrrrrVr]rLsegmentsrsegments r3r_zFasterWhisperParser.lazy_parses  * * * * * * *   U     3 3 3 3 3 3 3   /   di ' ' > **2:di+@+@AAEE Y 49  **4955EE<== =:ell%l88==??@@ ODKi   ))(a)@@$  G$\"k"4 w{7S"S $ #)E$2Kc2Q,R,R#R  m          s %0A rr:rAr3rr5s  J!'$( +-+-+- +-SM +-+-+-+-Z/t/(://////rAr)rloggingr)rtypingrrrrrrr r langchain_core.documentsr )langchain_community.document_loaders.baser 1langchain_community.document_loaders.blob_loadersr langchain_community.utils.openair getLoggerr`rrrhrrrr:rAr3rs  QQQQQQQQQQQQQQQQQQQQ------DDDDDDBBBBBB999999  8 $ $Q Q Q Q Q ~Q Q Q hwwwww.wwwtF F F F F ~F F F RKKKKKnKKK\.rA