g7ddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZejeZdZdd iZgd ZGd d eZdS) N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)loggingu▁ vocab_filezsentencepiece.bpe.model)ar_ARcs_CZde_DEen_XXes_XXet_EEfi_FIfr_XXgu_INhi_INit_ITja_XXkk_KZko_KRlt_LTlv_LVmy_MMne_NPnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CNc eZdZUdZeZddgZgZee e d<gZ ee e d< d3d e e eefffd ZdZdZedZedefdZejdedd fdZ d4dee de ee dedee ffd Z d5dee de ee dee fdZ d5dee de ee dee fdZdede ede efdZd Zd!edeefd"Zd#Zd$Zd%Z d5d&ed'e ede!efd(Z" d6d+eeded,e eedede#f fd- Z$d.Z%d/Z&d7d0Z'd1edd fd2Z(xZ)S)8MBartTokenizeruT Construct an MBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizer >>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO") >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ``` input_idsattention_mask prefix_tokens suffix_tokensNsp_model_kwargsc ft|trt|ddn|}| in| _t jdij_jt||_ddddd_ d_ tj_ fd ttD_d jD_tjtjzj zj d <j jd j D_t)j| !fd | Dt/jd|||||||d| | jd || | nd_jj_| _jdS)NTF)lstrip normalizedrr )r.r1r/r0c<i|]\}}|j|zjzS) sp_model_sizefairseq_offset).0icodeselfs h/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/mbart/tokenization_mbart.py z+MBartTokenizer.__init__..hs<   CJ1dD$$q(4+>>   ci|]\}}|| Sr:r:r=kvs rArBz+MBartTokenizer.__init__..ksNNNA1NNNrCr2ci|]\}}|| Sr:r:rEs rArBz+MBartTokenizer.__init__..os%Z%Z%Ztq!a%Z%Z%ZrCcg|]}|v| Sr:r:)r=t_additional_special_tokenss rA z+MBartTokenizer.__init__..us$]]]qB\9\9\9\9\9\rC) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokentokenizer_filesrc_langtgt_langadditional_special_tokensr3rr:) isinstancestrr r3spmSentencePieceProcessorsp_modelLoadrfairseq_tokens_to_idsr<lenr; enumerateFAIRSEQ_LANGUAGE_CODESlang_code_to_iditemsid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextendsuper__init__ _src_langcur_lang_code_idrVset_src_lang_special_tokens)r@rrMrNrPrQrOrRrSrTrUrVr3rWkwargsrK __class__s` @rArkzMBartTokenizer.__init__?sx&FPPZ\_E`E` pJz$5 A A A Afp &5%???%Z%Zt7Q7W7W7Y7Y%Z%Z%Z"%)$*>*C*C*E*E%F%F" $ 0 & - -]]]]5]]]     !&@ 0     &.%9w $ 4T^ D   ((88888rCcz|j}d|d<|j|d<|S)Nr\sp_model_proto)__dict__copyr\serialized_model_proto)r@states rA __getstate__zMBartTokenizer.__getstate__s= ""$$ j"&-"F"F"H"H rCc||_t|dsi|_tjdi|j|_|j|jdS)Nr3r:)rshasattrr3rZr[r\LoadFromSerializedProtorr)r@ds rA __setstate__zMBartTokenizer.__setstate__s` t.// &#%D 2JJT5IJJ  --d.ABBBBBrCcjt|jt|jz|jzdzS)Nr7)r_r\rbr<r@s rA vocab_sizezMBartTokenizer.vocab_sizes04=!!C(<$=$==@SSVWWWrCreturnc|jSN)rlr~s rArUzMBartTokenizer.src_langs ~rC new_src_langcH||_||jdSr)rlrn)r@rs rArUzMBartTokenizer.src_langs%% ((88888rCF token_ids_0 token_ids_1already_has_special_tokensc@|r$t||dSdgt|jz}dgt|jz}||dgt|zz|zS|dgt|zzdgt|zz|zS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr7Nr)rjget_special_tokens_maskr_r,r-)r@rrr prefix_ones suffix_onesrps rArz&MBartTokenizer.get_special_tokens_masks& & 7722'[]a3 cC 2333 cC 2333  1#K(8(8"89KG GqcC $4$445!s;?O?O9OPS^^^rCcT||j|z|jzS|j|z|z|jzS)ab Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r,r-)r@rrs rA build_inputs_with_special_tokensz/MBartTokenizer.build_inputs_with_special_tokenss;,  % 3d6HH H!K/+=@RRRrCc|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr) sep_token_id cls_token_idr_)r@rrsepclss rA$create_token_type_ids_from_sequencesz3MBartTokenizer.create_token_type_ids_from_sequencessm$ ! !  s[(3.//1#5 53$s*S0;>DEEKKrCreturn_tensorsrUrVc ||td||_||fd|d|}||}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensrforced_bos_token_id) ValueErrorrUconvert_tokens_to_ids)r@ raw_inputsrrUrV extra_kwargsinputs tgt_lang_ids rA_build_translation_inputsz(MBartTokenizer._build_translation_inputssg  x/`aa a  jiT.ii\hii00:: (3$% rCc|fdtjD}|j|S)Nc<i|]}||Sr:)convert_ids_to_tokens)r=r>r@s rArBz,MBartTokenizer.get_vocab.. s)RRRa++A..RRRrC)rangerreadded_tokens_encoder)r@vocabs` rA get_vocabzMBartTokenizer.get_vocabs@RRRR5;Q;QRRR T./// rCtextcD|j|tS)N)out_type)r\encoderY)r@rs rA _tokenizezMBartTokenizer._tokenize s}##D3#777rCc||jvr |j|S|j|}|r ||jzn|jS)z0Converts a token (str) in an id using the vocab.)r^r\ PieceToIdr< unk_token_id)r@tokenspm_ids rA_convert_token_to_idz#MBartTokenizer._convert_token_to_idsP D. . .-e4 4((//06Lv+++4;LLrCcr||jvr |j|S|j||jz S)z=Converts an index (integer) in a token (str) using the vocab.)rfr\ IdToPiecer<)r@indexs rA_convert_id_to_tokenz#MBartTokenizer._convert_id_to_tokens< D. . .-e4 4}&&ut/B'BCCCrCcd|td}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r@tokens out_strings rAconvert_tokens_to_stringz'MBartTokenizer.convert_tokens_to_strings4WWV__,,-=sCCIIKK rCsave_directoryfilename_prefixctj|s td|ddStj||r|dzndt dz}tj|jtj|kr:tj |jrt|j|nztj |jsVt|d5}|j }||dddn #1swxYwY|fS)NzVocabulary path (z) should be a directory-rrwb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathrisfileropenr\ruwrite)r@rrout_vocab_fileficontent_spiece_models rAsave_vocabularyzMBartTokenizer.save_vocabulary$sww}}^,,  LLT^TTT U U U F oM_s222QbcoQpp   7??4? + +rw~/N/N N NSUSZSaSabfbqSrSr N T_n 5 5 5 500 /nd++ /r'+}'K'K'M'M$-... / / / / / / / / / / / / / / /  s(/E##E'*E'rr" src_texts tgt_textsc V||_||_tj||fi|Sr)rUrVrjprepare_seq2seq_batch)r@rrUrrVrorps rArz$MBartTokenizer.prepare_seq2seq_batch5s4!   ,uww,Y LLVLLLrCc6||jSr)rnrUr~s rA_switch_to_input_modez$MBartTokenizer._switch_to_input_modeA// >>>rCc6||jSr)set_tgt_lang_special_tokensrVr~s rA_switch_to_target_modez%MBartTokenizer._switch_to_target_modeDrrCc^|j||_g|_|j|jg|_dS)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].Nrb cur_lang_coder, eos_token_idr-)r@rUs rArnz*MBartTokenizer.set_src_lang_special_tokensGs3!1(;"/1CDrClangc^|j||_g|_|j|jg|_dS)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nr)r@rs rArz*MBartTokenizer.set_tgt_lang_special_tokensMs3!1$7"/1CDrC) r.r/r/r.r0r1r2NNNNN)NFr)rNr")rN)*__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr,rint__annotations__r-rrrYrrkrwr|propertyrrUsetterboolrrrrrrrrrrrr rrrrnr __classcell__)rps@rAr)r)$s  (*$&67!M49!!!!M49!!! 48"&L9L9"$sCx.1L9L9L9L9L9L9\ CCCXXXX#X_9S9T999_9 sx__9_3;DI3F_ko_ c______>JNSS9S3;DI3FS cSSSS8JNLL9L3;DI3FL cLLLL2 *- 9A# RZ[^R_     8c8d3i8888MMMDDD  !!c!HSM!]bcf]g!!!!( )- M M9 M MDI& M  M  M M M M M M??????EEEE EEEEEEEEEErCr))rshutilrtypingrrrrr sentencepiecerZtokenization_utilsr r r utilsr get_loggerrrrrrar)r:rCrArs   33333333333333PPPPPPPPPP  H % %!#<={{{mEmEmEmEmE(mEmEmEmEmErC