Ng ddlZddlmZmZddlZddlmZmZddlm Z ddl m Z de de fdZ d e fd Zdd e dee de d e fdZdZdS)N)ListOptional) MarianMTModelMarianTokenizer) sent_tokenize)chunk_by_attention_window source_lang target_langcd|d|S)zjConstructs the name of the MarianMT machine translation model based on the source and target language.zHelsinki-NLP/opus-mt--)r r s [/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/cleaners/translate.py_get_opus_mt_model_namer s ?; > > > >> language_codec|t|trt|dkrtd|ddS)NzInvalid language code: z,. Language codes must be two letter strings.) isinstancestrlen ValueError)rs r_validate_language_codersO mS ) ) S-?-?1-D-D am a a a   .E-DrentextreturncD|dkr|S||ntj|}|drd}t |t |||kr|St ||} t j|}tj|}n!#t$rtd|dwxYwt||t}g}|D]&} | t|||'d|S)aTranslates the foreign language text. If the source language is not specified, the function will attempt to detect it using langdetect. Parameters ---------- text: str The text to translate target_lang: str The two letter language code for the target langague. Defaults to "en". source_lang: Optional[str] The two letter language code for the language of the input text. If source_lang is not provided, the function will try to detect it. Nzhz2Transformers could not find the translation model z>. The requested source/target language combo is not supported.)split_function )strip langdetectdetect startswithrrrfrom_pretrainedrOSErrorrrrappend_translate_textjoin) rr r _source_lang model_name tokenizermodelchunkstranslated_chunkschunks rtranslate_textr1s` zz||r '2'> JDUVZD[D[Lt$$ K(((L)))l"" ({CCJ #3J?? -j99     K K K K    2$ R_```F#%JJ  ui!H!HIIII 88% & &&s (B**Cc tj5tjd|jd i|gddd}dddn #1swxYwYfd|DdS) z8Translates text using the specified model and tokenizer.ignorept max_length)return_tensorspaddingr5Nc@g|]}|ddS)r6T)max_new_tokensskip_special_tokens)decode).0tr,s r z#_translate_text..Ts/ b b bRSI  Qs  M M b b brrr )warningscatch_warnings simplefiltergenerate)rr-r, translateds ` rr(r(Js  " "  h'''#U^  it\VYZZZ                 c b b bWa b b b  s0AAA)Nr)r@typingrrr" transformersrrunstructured.nlp.tokenizer unstructured.staging.huggingfacerrrrr1r(r rrrIs!!!!!!!!77777777333333FFFFFF??3????  3    /'/'/'8C=/'c/']`/'/'/'/'d     r