gP7ddlZddlmZddlmZmZmZmZmZm Z ddl Z ddl m Z mZddlmZerddlmZejeZdd iZd ZGd d eZd gZdS) N)copyfile) TYPE_CHECKINGAnyDictListOptionalTuple) AddedTokenPreTrainedTokenizer)logging) TextInput vocab_fileztokenizer.modelu▁c eZdZdZeZddgZ d%d eee e fffd Z d Z dZ edZdZdddee ffd ZdZdZdZdZd&dee dee fdZd&dZ d'deedeeededeeffd Z d&deedeeedeefd Z d(d!eed"ed#ede fd$ZxZS))GemmaTokenizera Construct a Gemma tokenizer. Based on byte-level Byte-Pair-Encoding. The default padding token is unset as there is no padding token in the original model. Args: vocab_file (`str`): Path to the vocabulary file. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. sp_model_kwargs (`Dict[str, Any]`, `Optional`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. add_bos_token (`bool`, *optional*, defaults to `True`): Whether or not to add an `bos_token` at the start of sequences. add_eos_token (`bool`, *optional*, defaults to `False`): Whether or not to add an `eos_token` at the end of sequences. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces. use_default_system_prompt (`bool`, *optional*, defaults to `False`): Whether or not the default system prompt for Gemma should be used. spaces_between_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add spaces between special tokens. input_idsattention_maskNTFsp_model_kwargsc  P|in||_t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}||_||_||_| |_tj di|j|_ |j |tj d|||||||| | | d | dS)NFT) normalizedspecial) bos_token eos_token unk_token pad_token add_bos_token add_eos_tokenrclean_up_tokenization_spacesuse_default_system_promptspaces_between_special_tokens)r isinstancestrr rr r!r#spmSentencePieceProcessorsp_modelLoadsuper__init__)selfrrrrrrr r!r"r#r$kwargs __class__s h/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/gemma/tokenization_gemma.pyr-zGemmaTokenizer.__init__\si&5%z,GemmaTokenizer.get_vocab..s)RRRa++A..RRRr2)ranger@updateadded_tokens_encoder)r.vocabs` r1 get_vocabzGemmaTokenizer.get_vocabs@RRRR5;Q;QRRR T./// r2textrreturnc 8tj|fi|S)ze Args: text: TextInput Simply calls PreTrainedTokenizer's method )r,tokenize)r.rLr/r0s r1rOzGemmaTokenizer.tokenizes$  uww/////r2c D|j|tS)z Args: text: TextInput Returns a tokenized string. The Gemma tokenizer never adds a prefix space. )out_type)r*encoder')r.rLr/s r1 _tokenizezGemmaTokenizer._tokenizes }##D3#777r2c6|j|S)z0Converts a token (str) in an id using the vocab.)r* piece_to_id)r.tokens r1_convert_token_to_idz#GemmaTokenizer._convert_token_to_ids}((///r2c:|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)r* IdToPiece)r.indexrVs r1_convert_id_to_tokenz#GemmaTokenizer._convert_id_to_tokens ''.. r2cg}d}|D]C}||jvr#||j||zz }g}.||D||j|z }|S)z:Converts a sequence of tokens (string) in a single string.)_added_tokens_encoderr*decodeappend)r.tokenscurrent_sub_tokens out_stringrVs r1convert_tokens_to_stringz'GemmaTokenizer.convert_tokens_to_strings  1 1E222dm223EFFNN %'"""))%0000dm**+=>>> r2filename_prefixctj|s td|ddStj||r|dzndt dz}tj|jtj|kr:tj |jrt|j|nztj |jsVt|d5}|j }||dddn #1swxYwY|fS)a Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. zVocabulary path (z) should be a directoryN-r]rwb)ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathrisfileropenr*r7write)r.save_directoryreout_vocab_fileficontent_spiece_models r1save_vocabularyzGemmaTokenizer.save_vocabularysyw}}^,,  LLT^TTT U U U F oM_s222QbcoQpp   7??4? + +rw~/N/N N NSUSZSaSabfbqSrSr N T_n 5 5 5 500 /nd++ /r'+}'K'K'M'M$-... / / / / / / / / / / / / / / /  s(/E##E'*E'ct|jr|jgng}|jr|jgng}||z|z}| ||z|z|z}|SN)r bos_token_idr! eos_token_idr. token_ids_0 token_ids_1r{r|outputs r1 build_inputs_with_special_tokensz/GemmaTokenizer.build_inputs_with_special_tokenss`.2.@H)**b .2.@H)**b  +l:  "l*[8<GF r2r~ralready_has_special_tokensc|r$t||dS|jrdgng}|jrdgng}||dgt |zz|zS|dgt |zz|z|zdgt |zz|zS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)r~rrNr)r,get_special_tokens_maskr r!len)r.r~rrr{r|r0s r1rz&GemmaTokenizer.get_special_tokens_masks$ & 7722'[]a3 #08ssb "08ssb  A3[)9)9#9:\I I sS%%% '  sS%%%  '    r2c|jr|jgng}|jr|jgng}dgt ||z|zz}||dgt ||z|zzz }|S)a Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rNr)r r{r!r|rr}s r1$create_token_type_ids_from_sequencesz3GemmaTokenizer.create_token_type_ids_from_sequencess./3.@H)**b .2.@H)**b s<+5 DEEE  " qcC { :\ IJJJ JF r2 token_idsskip_special_tokensr$c g}g}|D]}|r ||jvr||jvrW|r-||j|||j|jg}n|||r-||j||rd|}nd|}|tdS)N r]) all_special_ids_added_tokens_decoderr`r*r_contentrnreplaceSPIECE_UNDERLINE)r.rrr$r/ sub_textscurrent_sub_textidss r1_decodezGemmaTokenizer._decode/s"  - -C" sd.B'B'Bd000#M$$T]%9%9:J%K%KLLL  !;C!@!HIII#%   '',,,,  E   T]112BCC D D D ( +++II **I  !13777r2) rrrrNTFFFFrz)NF)FF) __name__ __module__ __qualname____doc__rovocab_files_namesmodel_input_namesrrr'rr-r9r=propertyr@rKrrOrSrWr[rdr rxrintboolrrr __classcell__)r0s@r1rr*s,,\*$&67 48%*"'&+( ( "$sCx.1( ( ( ( ( ( T CCC ..X. 0[0tCy000000888000   !!x}!X]^aXb!!!!6    sx# # 9# 3;DI3F# ko# c# # # # # # LJN93;DI3F cH%*.3 8898"8(, 8 88888888r2r)rishutilrtypingrrrrrr sentencepiecer(tokenization_utilsr r utilsr tokenization_utils_baser get_loggerrrlrorr__all__r%r2r1rs , BBBBBBBBBBBBBBBBAAAAAAAA5444444  H % %!#45`8`8`8`8`8(`8`8`8F  r2