gA,ddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlZddlmZddlmZejeZdd d d d d ZdZGddeZdede ee fdejfdZdeddfdZdedee e ffdZdS)N)Path)copyfile)AnyDictListOptionalTupleUnion)PreTrainedTokenizer)loggingz source.spmz target.spmz vocab.jsonztarget_vocab.jsonztokenizer_config.json) source_spm target_spmvocabtarget_vocab_filetokenizer_config_fileu▁c eZdZdZeZddgZejdZ d.d e e e e fd dffd ZdZde d e fdZdZde fdZde d ee fdZded e fdZfdZfdZdee d e fdZd/d eefdZdZdZed efdZd/de d e e d ee fd!Z d e fd"Z!d#Z"d$Z#d e fd%Z$d&e d dfd'Z%d(Z&d)Z' d0d*ed+e ed,e(d eefd-Z)xZ*S)1MarianTokenizeraB Construct a Marian tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: source_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the source language. target_spm (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .spm extension) that contains the vocabulary for the target language. source_lang (`str`, *optional*): A string representing the source language. target_lang (`str`, *optional*): A string representing the target language. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. model_max_length (`int`, *optional*, defaults to 512): The maximum sentence length the model accepts. additional_special_tokens (`List[str]`, *optional*, defaults to `["", ""]`): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Examples: ```python >>> from transformers import MarianForCausalLM, MarianTokenizer >>> model = MarianForCausalLM.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> src_texts = ["I am a small frog.", "Tom asked his teacher for advice."] >>> tgt_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."] # optional >>> inputs = tokenizer(src_texts, text_target=tgt_texts, return_tensors="pt", padding=True) >>> outputs = model(**inputs) # should work ``` input_idsattention_maskz>>.+<Fsp_model_kwargsreturnc p| in| |_t|s Jd|| |_t ||_t ||jvrtdt | |jvsJ| rDt ||_d|j D|_ g|_ n>d|j D|_ d|jD|_ ||_ ||_ ||g|_t||j|_t||j|_|j|_|j|_|t+jd||||| | |j|| d | dS)Nzcannot find spm source z token must be in the vocabci|]\}}|| Sr.0kvs j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/marian/tokenization_marian.py z,MarianTokenizer.__init__..sIIITQAqIIIci|]\}}|| Srrr s r$r%z,MarianTokenizer.__init__..sBBBTQAqBBBr&cfg|].}|d|d,|/S)z>>z<<) startswithendswithr!r"s r$ z,MarianTokenizer.__init__..s?2v2v2vall[_N`N`2vefeoeopteueu2v12v2v2vr&) source_lang target_lang unk_token eos_token pad_tokenmodel_max_lengthrrseparate_vocabsr)rrexistsr3 load_jsonencoderstrKeyErrortarget_encoderitemsdecodersupported_language_codesr-r. spm_filesload_spm spm_source spm_target current_spmcurrent_encoder_setup_normalizersuper__init__)selfrrrrr-r.r/r0r1r2rr3kwargs __class__s r$rEzMarianTokenizer.__init__ks &5%> - -=>> >9~~----  w"+,=">">D IIT-@-F-F-H-HIIIDL,.D ) )BBT\-?-?-A-ABBBDL2v2vdl2v2v2vD )&&$j1#:t/CDD":t/CDD?#|     ##- 0/+  r&c ddlm}||jj|_dS#t t f$r tjdd|_YdSwxYw)Nr)MosesPunctNormalizerz$Recommended: pip install sacremoses.c|SNr)xs r$z3MarianTokenizer._setup_normalizer..sQr&) sacremosesrJr- normalizepunc_normalizer ImportErrorFileNotFoundErrorwarningswarn)rFrJs r$rCz!MarianTokenizer._setup_normalizers} / 7 7 7 7 7 7#7#78H#I#I#SD ./ / / / M@ A A A#.;D  /s $-AArMc4|r||ndS)zHCover moses empty string edge case. They return empty list for '' input!)rQ)rFrMs r$rPzMarianTokenizer.normalizes *+3t##A&&&3r&cX|j||j|jSrL)rBgetr/)rFtokens r$_convert_token_to_idz$MarianTokenizer._convert_token_to_ids%#''t/CDN/STTTr&textc|j|}|r|dgng}||jd|fS)z6Remove language codes like >>fr<< before sentencepiecerrW)language_code_rematchgroupsub)rFr\r_codes r$remove_language_codez$MarianTokenizer.remove_language_codesR%++D11).6ekk!nn%%BT*..r48888r&c~||\}}|j|t}||zS)N)out_type)rcrAencoder7)rFr\rbpiecess r$ _tokenizezMarianTokenizer._tokenizes>..t44 d!(((<<f}r&indexcB|j||jS)z?Converts an index (integer) in a token (str) using the decoder.)r;rYr/)rFris r$_convert_id_to_tokenz$MarianTokenizer._convert_id_to_tokens|t~666r&c 8tj|fi|S)ao Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `List[str]`: The list of decoded sentences. )rD batch_decode)rF sequencesrGrHs r$rmzMarianTokenizer.batch_decodes$*$uww#I88888r&c 8tj|fi|S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces` (available in the `tokenizer_config`). use_source_tokenizer (`bool`, *optional*, defaults to `False`): Whether or not to use the source tokenizer to decode sequences (only applicable in sequence-to-sequence problems). kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. )rDdecode)rF token_idsrGrHs r$rpzMarianTokenizer.decodes#0uww~i226222r&tokenscJ|jr|jn|j}g}d}|D]A}||jvr!||||zdzz }g},||B|||z }|td}|S)zQUses source spm if _decode_use_source_tokenizer is True, and target spm otherwiserW ) _decode_use_source_tokenizerr?r@all_special_tokens decode_piecesappendreplaceSPIECE_UNDERLINEstrip)rFrrsp_modelcurrent_sub_tokens out_stringrZs r$convert_tokens_to_stringz(MarianTokenizer.convert_tokens_to_strings&*&G\4??T_  1 1E///h445GHH5PSVVV %'"""))%0000h,,-?@@@ ''(8#>> !!!r&c8| ||jgzS||z|jgzS)z=Build model inputs from a sequence by appending eos_token_id.) eos_token_id)rF token_ids_0 token_ids_1s r$ build_inputs_with_special_tokensz0MarianTokenizer.build_inputs_with_special_tokens s/  $"3!44 4[(D,=+>>>r&c6|j|_|j|_dSrL)r?rAr6rBrFs r$_switch_to_input_modez%MarianTokenizer._switch_to_input_modes?#|r&cH|j|_|jr|j|_dSdSrL)r@rAr3r9rBrs r$_switch_to_target_modez&MarianTokenizer._switch_to_target_modes2?   7#'#6D  7 7r&c*t|jSrL)lenr6rs r$ vocab_sizezMarianTokenizer.vocab_sizes4<   r&save_directoryfilename_prefixctj|s td|ddSg}|jrtj||r|dzndtdz}tj||r|dzndtdz}t|j |t|j || || |n_tj||r|dzndtdz}t|j || |ttdtdg|j |j|jgD];\}}} tj||r|dznd|z} tj|tj| krEtj|r&t%|| | | tj|sft'| d 5} | } | | dddn #1swxYwY| | =t-|S) NzVocabulary path (z) should be a directory-rWrrrrwb)ospathisdirloggererrorr3joinVOCAB_FILES_NAMES save_jsonr6r9rxzipr=r?r@abspathisfileropenserialized_model_protowritetuple) rFrr saved_filesout_src_vocab_fileout_tgt_vocab_fileout_vocab_filespm_save_filename spm_orig_path spm_model spm_save_pathficontent_spiece_models r$save_vocabularyzMarianTokenizer.save_vocabularysw}}^,,  LLT^TTT U U U F   /!#*9A3&&rEVW^E__"" "$*9A3&&rEVWjEkk""  dl$6 7 7 7 d)+= > > >   1 2 2 2   1 2 2 2 2W\\/!Q3!6!6rUfgnUo oN dlN 3 3 3   ~ . . .;> | ,.? .M N N _do .< <  2 2 7 }i GLL/!Q3!6!6rUf fMw}--1O1OOOTVT[TbTbcpTqTqO 666""=1111W^^M22 2-..3"+4+K+K+M+M(HH1222333333333333333""=111[!!!s*J<<K K c*|SrL) get_src_vocabrs r$ get_vocabzMarianTokenizer.get_vocabJs!!###r&c0t|jfi|jSrL)dictr6added_tokens_encoderrs r$rzMarianTokenizer.get_src_vocabMsDL>>D$=>>>r&c0t|jfi|jSrL)rr9added_tokens_decoderrs r$ get_tgt_vocabzMarianTokenizer.get_tgt_vocabPsD'EE4+DEEEr&cv|j}|ddD|S)Nci|]}|dSrLrr+s r$r%z0MarianTokenizer.__getstate__..Vs r r rQ r r rr&)r?r@rArQr)__dict__copyupdate)rFstates r$ __getstate__zMarianTokenizer.__getstate__SsD ""$$ r rq r r r    r&dc|_tdsi_fdjD\__j_dS)Nrc3BK|]}t|jVdSrL)r>r)r!frFs r$ z/MarianTokenizer.__setstate__..as1+f+fRSHQ8L,M,M+f+f+f+f+f+fr&)rhasattrrr=r?r@rArC)rFrs` r$ __setstate__zMarianTokenizer.__setstate__Zso t.// &#%D +f+f+f+fW[We+f+f+f(?      r&cdS)zJust EOSr)rFargsrGs r$num_special_tokens_to_addz)MarianTokenizer.num_special_tokens_to_addesqr&c|t|j|jfd|DS)Nc g|] }|vrdnd S)rrr)r!rMall_special_idss r$r,z7MarianTokenizer._special_token_mask..ls'>>>QQ/))q>>>r&)setrremove unk_token_id)rFseqrs @r$_special_token_maskz#MarianTokenizer._special_token_maskisDd233t0111>>>>#>>>>r&rralready_has_special_tokensc|r||S|||dgzS|||zdgzS)zCGet list where entries are [1] if a token is [eos] or [pad] else 0.Nr)r)rFrrrs r$get_special_tokens_maskz'MarianTokenizer.get_special_tokens_masknsb & M++K88 8  ++K88A3> >++K+,EFF!L Lr&) NNNrrrrNFrL)NF)+__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrecompiler^rrr7rrErCrPr[rcrrhintrkrmrprrrrpropertyrr rrrrrrrrboolr __classcell__)rHs@r$rr,sB8 8 t*$&67!rz(++48< < "$sCx.1<  < < < < < < |///43434444UUU99999 cd3i 7#7#777799999.333334"tCy"S"""" ??QUVYQZ????,,,777 !C!!!X!+"+"c+"HSM+"]bcf]g+"+"+"+"Z$4$$$$???FFFd !d !t ! ! ! !??? in M M M.6tn Mae M c M M M M M M M Mr&rrrrcRtjdi|}|||S)Nr) sentencepieceSentencePieceProcessorLoad)rrspms r$r>r>zs,  . A A A ACHHTNNN Jr&ct|d5}tj||dddddS#1swxYwYdS)Nw)indent)rjsondump)datarrs r$rrs dC%A $!$$$$%%%%%%%%%%%%%%%%%%s 6::c~t|d5}tj|cdddS#1swxYwYdS)Nr)rrload)rrs r$r5r5s| dCAy||s 266)rrrrTpathlibrshutilrtypingrrrrr r rtokenization_utilsr utilsr get_loggerrrrrzrr7rr>rr5rr&r$rs ::::::::::::::::555555  H % % ,4  KMKMKMKMKM)KMKMKM\ 3c3hM<` %#%$%%%% CE$*-r&