gIddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZejeZdZdd d Zgd gd d ZddddddddZGddeZdS)N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)loggingu▁zsentencepiece.bpe.modelztokenizer.json) vocab_filetokenizer_file)__java__ __python__ __en_XX__)rrr__javascript____php____ruby____go__)basemultirrrrrrr)javapythonen_XX javascriptphprubygoc eZdZUdZeZddgZgZee e d<gZ ee e d< d6de e eefffd ZdZdZedZedefdZejdedd fdZ d7dee de ee dedee ffd Z d8dee de ee dee fdZ d8dee de ee dee fdZdede ed e efd!Zd"Zd#edeefd$Zd%Zd&Zd'Z d8d(ed)e ede!efd*Z" d9d-eeded.e eed ede#f fd/ Z$d0Z%d1Z&d:d2Z'd3edd fd4Z(d3edefd5Z)xZ*S);PLBartTokenizera Construct an PLBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Args: vocab_file (`str`): Path to the vocabulary file. src_lang (`str`, *optional*): A string representing the source language. tgt_lang (`str`, *optional*): A string representing the target language. bos_token (`str`, *optional*, defaults to `""`): The start of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The cls token, which is a special token used as the first token for all tasks. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token(`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masking tasks. This is only used in the `"base"` tokenizer type. For `"multi"` tokenizer, masking is never done for the downstream tasks. language_codes (`str`, *optional*, defaults to `"base"`): What language codes to use. Should be one of `"base"` or `"multi"`. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import PLBartTokenizer >>> tokenizer = PLBartTokenizer.from_pretrained("uclanlp/plbart-python-en_XX", src_lang="python", tgt_lang="en_XX") >>> example_python_phrase = "def maximum(a,b,c):NEW_LINE_INDENTreturn max([a,b,c])" >>> expected_translation_english = "Returns the maximum value of a b c." >>> inputs = tokenizer(example_python_phrase, text_target=expected_translation_english, return_tensors="pt") ``` input_idsattention_mask prefix_tokens suffix_tokensrNTsp_model_kwargsc lt|trt|ddn|}| in| _| } | } t jdij_jt||_ | _ tj }ddddd_ d_ tj_fd t!|D_d jD_j d kr9tjtjzj zj d <j jd j D_t-j|!fd|Dj d kr-| _jjjnj_n"| | nd_jj_t7jd|||||||| | | | j|d|| _jdS)NTF)lstriprstriprr )r&r)r'r(c<i|]\}}|j|zjzS) sp_model_sizefairseq_offset).0icodeselfs j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/plbart/tokenization_plbart.py z,PLBartTokenizer.__init__..s<   CJ1dD$$q(4+>>   ci|]\}}|| Sr2r2r5kvs r9r:z,PLBartTokenizer.__init__..sNNNA1NNNr;rr*ci|]\}}|| Sr2r2r=s r9r:z,PLBartTokenizer.__init__..s%Z%Z%Ztq!a%Z%Z%Zr;cg|]}|v| Sr2r2)r5t_additional_special_tokenss r9 z,PLBartTokenizer.__init__..s$]]]qB\9\9\9\9\9\r;r) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokenlanguage_codesrsrc_langtgt_langadditional_special_tokensr+clean_up_tokenization_spacesr2) isinstancestrr r+!_convert_lang_code_special_formatspmSentencePieceProcessorsp_modelLoadrrLFAIRSEQ_LANGUAGE_CODESfairseq_tokens_to_idsr4lenr3 enumeratelang_code_to_iditemsid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextend _src_langcur_lang_code_idsuper__init__rNset_src_lang_special_tokens)r8rrErFrHrIrGrJrKrLrrMrNr+rOrPkwargsfairseq_language_codesrC __class__s` @r9rgzPLBartTokenizer.__init__us(KUU_adJeJeuZ 4FFFFku %4%???%Z%Zt7Q7W7W7Y7Y%Z%Z%Z"%)$*>*C*C*E*E%F%F" $ 0 & - -]]]]5]]]     & ( (%DN8<8R$T^44X\Xf  ! !*2)=XX;DN$($8$HD ! !))&@ 0)E     $!  ((88888r;cz|j}d|d<|j|d<|S)NrVsp_model_proto)__dict__copyrVserialized_model_proto)r8states r9 __getstate__zPLBartTokenizer.__getstate__s= ""$$ j"&-"F"F"H"H r;c||_t|dsi|_tjdi|j|_|j|jdS)Nr+r2)rnhasattrr+rTrUrVLoadFromSerializedProtorm)r8ds r9 __setstate__zPLBartTokenizer.__setstate__s` t.// &#%D 2JJT5IJJ  --d.ABBBBBr;c|jdkr4t|jt|jz|jzdzSt|jt|jz|jzS)Nrr/)rLrZrVr\r4r8s r9 vocab_sizezPLBartTokenizer.vocab_sizesj  & ( (DM""S)=%>%>>ATTWXX t}%%D,@(A(AADDWW Wr;returnc|jSN)rdrys r9rMzPLBartTokenizer.src_langs ~r; new_src_langcr||}||_||jdSr})rSrdrh)r8r~s r9rMzPLBartTokenizer.src_langs8==lKK % ((88888r;F token_ids_0 token_ids_1already_has_special_tokensc@|r$t||dSdgt|jz}dgt|jz}||dgt|zz|zS|dgt|zzdgt|zz|zS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrr/Nr)rfget_special_tokens_maskrZr$r%)r8rrr prefix_ones suffix_onesrks r9rz'PLBartTokenizer.get_special_tokens_masks& & 7722'[]a3 cC 2333 cC 2333  1#K(8(8"89KG GqcC $4$445!s;?O?O9OPS^^^r;cT||j|z|jzS|j|z|z|jzS)ac Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An PLBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r$r%)r8rrs r9 build_inputs_with_special_tokensz0PLBartTokenizer.build_inputs_with_special_tokenss;,  % 3d6HH H!K/+=@RRRr;c|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. PLBart does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr) sep_token_id cls_token_idrZ)r8rrsepclss r9$create_token_type_ids_from_sequencesz4PLBartTokenizer.create_token_type_ids_from_sequences/sm" ! !  s[(3.//1#5 53$s*S0;>DEEKKr;return_tensorsrMrNc ||td|||_|||_||fd|d|}||j}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)add_special_tokensrforced_bos_token_id) ValueErrorrSrMrNconvert_tokens_to_ids)r8 raw_inputsrrMrN extra_kwargsinputs tgt_lang_ids r9_build_translation_inputsz)PLBartTokenizer._build_translation_inputsGs  x/`aa a>>xHH >>xHH jiT.ii\hii00?? (3$% r;c|fdtjD}|j|S)Nc<i|]}||Sr2)convert_ids_to_tokens)r5r6r8s r9r:z-PLBartTokenizer.get_vocab..Us)RRRa++A..RRRr;)rangerzr_added_tokens_encoder)r8vocabs` r9 get_vocabzPLBartTokenizer.get_vocabTs@RRRR5;Q;QRRR T./// r;textcD|j|tS)N)out_type)rVencoderR)r8rs r9 _tokenizezPLBartTokenizer._tokenizeYs}##D3#777r;c||jvr |j|S|j|}|r ||jzn|jS)z0Converts a token (str) in an id using the vocab.)rYrV PieceToIdr4 unk_token_id)r8tokenspm_ids r9_convert_token_to_idz$PLBartTokenizer._convert_token_to_id\sP D. . .-e4 4((//06Lv+++4;LLr;cr||jvr |j|S|j||jz S)z=Converts an index (integer) in a token (str) using the vocab.)r`rV IdToPiecer4)r8indexs r9_convert_id_to_tokenz$PLBartTokenizer._convert_id_to_tokenes< D. . .-e4 4}&&ut/B'BCCCr;cd|td}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r8tokens out_strings r9convert_tokens_to_stringz(PLBartTokenizer.convert_tokens_to_stringks4WWV__,,-=sCCIIKK r;save_directoryfilename_prefixctj|s td|ddStj||r|dzndt dz}tj|jtj|kr:tj |jrt|j|nztj |jsVt|d5}|j }||dddn #1swxYwY|fS)NzVocabulary path (z) should be a directory-rrwb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathrisfileropenrVrpwrite)r8rrout_vocab_fileficontent_spiece_models r9save_vocabularyzPLBartTokenizer.save_vocabularypsww}}^,,  LLT^TTT U U U F oM_s222QbcoQpp   7??4? + +rw~/N/N N NSUSZSaSabfbqSrSr N T_n 5 5 5 500 /nd++ /r'+}'K'K'M'M$-... / / / / / / / / / / / / / / /  s(/E##E'*E'rr src_texts tgt_textsc |||_|||_tj||fi|Sr})rSrMrNrfprepare_seq2seq_batch)r8rrMrrNrirks r9rz%PLBartTokenizer.prepare_seq2seq_batchsP>>xHH >>xHH ,uww,Y LLVLLLr;c6||jSr})rhrMrys r9_switch_to_input_modez%PLBartTokenizer._switch_to_input_mode// >>>r;c6||jSr})set_tgt_lang_special_tokensrNrys r9_switch_to_target_modez&PLBartTokenizer._switch_to_target_moderr;c||}| |j|nd|_g|_|j|j|jg|_dS|jg|_dS)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].NrSr\ cur_lang_coder$ eos_token_idr%)r8rMs r9rhz+PLBartTokenizer.set_src_lang_special_tokenssn99(CC?G?ST1(;;Y]   )"&"3T5G!HD   "&"3!4D   r;langc||}| |j|nd|_g|_|j|j|jg|_dS|jg|_dS)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nrr8rs r9rz+PLBartTokenizer.set_tgt_lang_special_tokenssn55d;;;?;KT1$77QU   )"&"3T5G!HD   "&"3!4D   r;cZ|tvr t|n|}|S)z;Convert Language Codes to format tokenizer uses if required)FAIRSEQ_LANGUAGE_CODES_MAPrbrs r9rSz1PLBartTokenizer._convert_lang_code_special_formats/37;U;Z;Z;\;\3\3\)$//bf r;)r&r'r'r&r(r)r*rNNNNNT)NFr})rNr)r{N)+__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr$rint__annotations__r%rrrRrrgrrrwpropertyrzrMsetterboolrrrrrrrrrrrr rrrrhrrS __classcell__)rks@r9r!r!1s9; ; z*$&67!M49!!!!M49!!! 48"&%)!\9\9"$sCx.1\9\9\9\9\9\9| CCCXXXX#X_9S9T999_9 sx__9_3;DI3F_ko_ c______>JNSS9S3;DI3FS cSSSS8JNLL9L3;DI3FL cLLLL0 *- 9A# RZ[^R_     8c8d3i8888MMMDDD  !!c!HSM!]bcf]g!!!!( )- M M9 M MDI& M  M  M M M M M M??????5555 5 5 5 5 5 5ccr;r!)rshutilrtypingrrrrr sentencepiecerTtokenization_utilsr r r utilsr get_loggerrrrrrXrr!r2r;r9rs5  33333333333333PPPPPPPPPP  H % %#