gyJddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZejeZdZdd iZgd ZGd d eZdS) N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)loggingu▁ vocab_filezsentencepiece.bpe.model)ace_Arabace_Latnacm_Arabacq_Arabaeb_Arabafr_Latnajp_Arabaka_Latnamh_Ethiapc_Arabarb_Arabars_Arabary_Arabarz_Arabasm_Bengast_Latnawa_Devaayr_Latnazb_Arabazj_Latnbak_Cyrlbam_Latnban_Latnbel_Cyrlbem_Latnben_Bengbho_Devabjn_Arabbjn_Latnbod_Tibtbos_Latnbug_Latnbul_Cyrlcat_Latnceb_Latnces_Latncjk_Latnckb_Arabcrh_Latncym_Latndan_Latndeu_Latndik_Latndyu_Latndzo_Tibtell_Grekeng_Latnepo_Latnest_Latneus_Latnewe_Latnfao_Latnpes_Arabfij_Latnfin_Latnfon_Latnfra_Latnfur_Latnfuv_Latngla_Latngle_Latnglg_Latngrn_Latnguj_Gujrhat_Latnhau_Latnheb_Hebrhin_Devahne_Devahrv_Latnhun_Latnhye_Armnibo_Latnilo_Latnind_Latnisl_Latnita_Latnjav_Latnjpn_Jpankab_Latnkac_Latnkam_Latnkan_Kndakas_Arabkas_Devakat_Georknc_Arabknc_Latnkaz_Cyrlkbp_Latnkea_Latnkhm_Khmrkik_Latnkin_Latnkir_Cyrlkmb_Latnkon_Latnkor_Hangkmr_Latnlao_Laoolvs_Latnlij_Latnlim_Latnlin_Latnlit_Latnlmo_Latnltg_Latnltz_Latnlua_Latnlug_Latnluo_Latnlus_Latnmag_Devamai_Devamal_Mlymmar_Devamin_Latnmkd_Cyrlplt_Latnmlt_Latnmni_Bengkhk_Cyrlmos_Latnmri_Latnzsm_Latnmya_Mymrnld_Latnnno_Latnnob_Latnnpi_Devanso_Latnnus_Latnnya_Latnoci_Latngaz_Latnory_Oryapag_Latnpan_Gurupap_Latnpol_Latnpor_Latnprs_Arabpbt_Arabquy_Latnron_Latnrun_Latnrus_Cyrlsag_Latnsan_Devasat_Bengscn_Latnshn_Mymrsin_Sinhslk_Latnslv_Latnsmo_Latnsna_Latnsnd_Arabsom_Latnsot_Latnspa_Latnals_Latnsrd_Latnsrp_Cyrlssw_Latnsun_Latnswe_Latnswh_Latnszl_Latntam_Tamltat_Cyrltel_Telutgk_Cyrltgl_Latntha_Thaitir_Ethitaq_Latntaq_Tfngtpi_Latntsn_Latntso_Latntuk_Latntum_Latntur_Latntwi_Latntzm_Tfnguig_Arabukr_Cyrlumb_Latnurd_Arabuzn_Latnvec_Latnvie_Latnwar_Latnwol_Latnxho_Latnydd_Hebryor_Latnyue_Hantzho_Hanszho_Hantzul_Latnc eZdZUdZeZddgZgZee e d<gZ ee e d< d3d e e eefffd ZdZdZedZedefdZejdedd fdZ d4dee de ee dedee ffd Z d5dee de ee dee fdZ d5dee de ee dee fdZdede ede efdZd Zd!edeefd"Zd#Zd$Zd%Z d5d&ed'e ede!efd(Z" d6d+eeded,e eedede#f fd- Z$d.Z%d/Z&d7d0Z'd1edd fd2Z(xZ)S)8 NllbTokenizeraN Construct an NLLB tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import NllbTokenizer >>> tokenizer = NllbTokenizer.from_pretrained( ... "facebook/nllb-200-distilled-600M", src_lang="eng_Latn", tgt_lang="fra_Latn" ... ) >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_french = "Le chef de l'ONU affirme qu'il n'y a pas de solution militaire en Syrie." >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_french, return_tensors="pt") ``` Args: vocab_file (`str`): Path to the vocabulary file. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token (`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. tokenizer_file (`str`, *optional*): The path to a tokenizer file to use instead of the vocab file. src_lang (`str`, *optional*): The language to use as source language for translation. tgt_lang (`str`, *optional*): The language to use as target language for translation. sp_model_kwargs (`Dict[str, str]`): Additional keyword arguments to pass to the model initialization. input_idsattention_mask prefix_tokens suffix_tokensNFsp_model_kwargsc | t} t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|dddn|}| in| |_||_t jd i|j|_|j t|||_ ||||d|_ d|_ t|j|_tjd |||||||| | | | |j|d || | nd|_||j|_| |_||jdS) NFT) normalizedspecial)rlstripr)rr r) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokentokenizer_filesrc_langtgt_langadditional_special_tokensrlegacy_behaviourr=)FAIRSEQ_LANGUAGE_CODES isinstancestrr rrspmSentencePieceProcessorsp_modelLoadr_added_tokens_decoderfairseq_offsetlen sp_model_sizesuper__init__ _src_langconvert_tokens_to_idscur_lang_code_idrset_src_lang_special_tokens)selfrrrrrrrrrrrrrrkwargs __class__s f/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/nllb/tokenization_nllb.pyrzNllbTokenizer.__init__qs$ % ,(> %MWXacfMgMgvJyUDIIIImv MWXacfMgMgvJyUDIIIImv MWXacfMgMgvJyUDIIIImv MWXacfMgMgvJyUDIIIImv *c** Jzd4 N N N N  &5% J J   ((88888cz|j}d|d<|j|d<|S)Nrsp_model_proto)__dict__copyrserialized_model_proto)r states r  __getstate__zNllbTokenizer.__getstate__s= ""$$ j"&-"F"F"H"H r c||_t|dsi|_tjdi|j|_|j|jdS)Nrr)rhasattrrrrrLoadFromSerializedProtor)r ds r  __setstate__zNllbTokenizer.__setstate__s` t.// &#%D 2JJT5IJJ  --d.ABBBBBr c:t|j|jzSN)rrrr s r  vocab_sizezNllbTokenizer.vocab_sizes4=!!D$777r returnc|jSr)rrs r rzNllbTokenizer.src_langs ~r  new_src_langcH||_||jdSr)rr)r r s r rzNllbTokenizer.src_langs%% ((88888r  token_ids_0 token_ids_1already_has_special_tokensc@|r$t||dSdgt|jz}dgt|jz}||dgt|zz|zS|dgt|zzdgt|zz|zS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)r"r#r$rNr)rget_special_tokens_maskrrr)r r"r#r$ prefix_ones suffix_onesr s r r&z%NllbTokenizer.get_special_tokens_masks& & 7722'[]a3 cC 2333 cC 2333  1#K(8(8"89KG GqcC $4$445!s;?O?O9OPS^^^r cT||j|z|jzS|j|z|z|jzS)aa Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An NLLB sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )rr)r r"r#s r  build_inputs_with_special_tokensz.NllbTokenizer.build_inputs_with_special_tokenss;,  % 3d6HH H!K/+=@RRRr c|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. nllb does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr) sep_token_id cls_token_idr)r r"r#sepclss r $create_token_type_ids_from_sequencesz2NllbTokenizer.create_token_type_ids_from_sequencessm$ ! !  s[(3.//1#5 53$s*S0;>DEEKKr return_tensorsrrc ||td||_||fd|d|}||}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensr1forced_bos_token_id) ValueErrorrr)r  raw_inputsr1rr extra_kwargsinputs tgt_lang_ids r _build_translation_inputsz'NllbTokenizer._build_translation_inputs'sg  x/`aa a  jiT.ii\hii00:: (3$% r c|fdtjD}|j|S)Nc<i|]}||Sr)convert_ids_to_tokens).0ir s r  z+NllbTokenizer.get_vocab..4s)RRRa++A..RRRr )rangerupdateadded_tokens_encoder)r vocabs` r  get_vocabzNllbTokenizer.get_vocab3s@RRRR5;Q;QRRR T./// r textcD|j|tS)N)out_type)rencoder)r rFs r  _tokenizezNllbTokenizer._tokenize8s}##D3#777r c\|j|}|r ||jzn|jS)z0Converts a token (str) in an id using the vocab.)r PieceToIdr unk_token_id)r tokenspm_ids r _convert_token_to_idz"NllbTokenizer._convert_token_to_id;s2((///5Lv+++4;LLr cF|j||jz S)z=Converts an index (integer) in a token (str) using the vocab.)r IdToPiecer)r indexs r _convert_id_to_tokenz"NllbTokenizer._convert_id_to_tokenAs }&&ut/B'BCCCr cd|td}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r tokens out_strings r convert_tokens_to_stringz&NllbTokenizer.convert_tokens_to_stringEs4WWV__,,-=sCCIIKK r save_directoryfilename_prefixctj|s td|ddStj||r|dzndt dz}tj|jtj|kr:tj |jrt|j|nztj |jsVt|d5}|j }||dddn #1swxYwY|fS)NzVocabulary path (z) should be a directory-rVrwb)ospathisdirloggererrorrXVOCAB_FILES_NAMESabspathrisfileropenrrwrite)r r_r`out_vocab_fileficontent_spiece_models r save_vocabularyzNllbTokenizer.save_vocabularyJsww}}^,,  LLT^TTT U U U F oM_s222QbcoQpp   7??4? + +rw~/N/N N NSUSZSaSabfbqSrSr N T_n 5 5 5 500 /nd++ /r'+}'K'K'M'M$-... / / / / / / / / / / / / / / /  s(/E##E'*E'r=rG src_texts tgt_textsc V||_||_tj||fi|Sr)rrrprepare_seq2seq_batch)r rrrrsrr r s r ruz#NllbTokenizer.prepare_seq2seq_batch[s4!   ,uww,Y LLVLLLr c6||jSr)rrrs r _switch_to_input_modez#NllbTokenizer._switch_to_input_modeg// >>>r c6||jSr)set_tgt_lang_special_tokensrrs r _switch_to_target_modez$NllbTokenizer._switch_to_target_modejrxr c|||_|jrg|_|j|jg|_dS|jg|_|jg|_dS)zReset the special tokens to the source lang setting. - In legacy mode: No prefix and suffix=[eos, src_lang_code]. - In default mode: Prefix=[src_lang_code], suffix = [eos] Nr cur_lang_coderr eos_token_idr)r rs r rz)NllbTokenizer.set_src_lang_special_tokensmsh "77AA   5!#D "&"3T5G!HD   "&"4!5D "&"3!4D   r langc|||_|jrg|_|j|jg|_dS|jg|_|jg|_dS)zReset the special tokens to the target lang setting. - In legacy mode: No prefix and suffix=[eos, tgt_lang_code]. - In default mode: Prefix=[tgt_lang_code], suffix = [eos] Nr})r rs r rzz)NllbTokenizer.set_tgt_lang_special_tokenszsh "77==   5!#D "&"3T5G!HD   "&"4!5D "&"3!4D   r ) rrrrrrrNNNNNF)NFr)r=NrG)rN)*__name__ __module__ __qualname____doc__rivocab_files_namesmodel_input_namesrrint__annotations__rrrrrrrrpropertyrrsetterboolr&r*r0r:rErJrPrTr^rrqr rurwr{rrz __classcell__)r s@r rr$sDDL*$&67!M49!!!!M49!!! 48"&E9E9"$sCx.1E9E9E9E9E9E9N CCC88X8#X_9S9T999_9 sx__9_3;DI3F_ko_ c______>JNSS9S3;DI3FS cSSSS8JNLL9L3;DI3FL cLLLL2 *- 9A# RZ[^R_     8c8d3i8888MMM DDD !!c!HSM!]bcf]g!!!!(#)-" M M9 M MDI& M  M  M M M M M M?????? 5 5 5 5 5 5 5 5 5 5 5 5 5 5r r)rdshutilrtypingrrrrr sentencepiecertokenization_utilsr r r utilsr get_loggerrrgrZrirrrr r rs  33333333333333PPPPPPPPPP  H % %!#<=R&R&R&a5a5a5a5a5'a5a5a5a5a5r