g<dZddlZddlZddlZddlmZddlmZmZm Z m Z m Z ddl m Z mZddlmZejeZdd iZGd d ZGd d eZdS)z"Tokenization class for model MyT5.N) defaultdict)DictListOptionalTupleUnion) AddedTokenPreTrainedTokenizer)logging vocab_filezbyte_maps.jsonc 0eZdZdZdZdeeeeefffdZdeeee e effdedefdZ deeefd eeee e efffd Z d e ed ed e effd Z dde ed e efdZd S) ByteRewriteraZ Byte rewriter class for MyT5 tokenizer. This class is used to rewrite bytes using a hash tree. The hash tree is constructed from a set of rewriting rules. Args: rewriting_rules (`str` or `Dict[str, str]`): A path to a json file containing the rewriting rules or a dictionary containing the rewriting rules. z[LEAF]rewriting_rulesct|tr=t|d5}tj|}dddn #1swxYwYn4t|t st dt||||_ d| D}|||_ dS)NrzDrewriting_rules should be either a path to json file or a dict, got ci|]\}}|| Sr).0kvs f/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/myt5/tokenization_myt5.py z)ByteRewriter.__init__..8s"L"L"LDAq1a"L"L"L) isinstancestropenjsonloaddict ValueErrortypeconstruct_hash_tree hash_treeitemsreverse_hash_tree)selfrfreverse_rewriting_ruless r__init__zByteRewriter.__init__.s os + + os++ /q"&)A,, / / / / / / / / / / / / / / /OT22 nW[\kWlWlnn 11/BB"L"LO4I4I4K4K"L"L"L!%!9!9:Q!R!RsAA A r$byte_in_sequencebyte_out_sequencec|d}|d}|}|D]}||vri||<||}|||j<dS)zL Add a leaf with the output byte sequence to the hash tree.  N)splitLEAF)r'r$r+r, byte_in_list byte_out_list tree_pointerbs radd_leafzByteRewriter.add_leaf;so(--c22 )//44   + +A $$"$ Q'?LL"/ TYrreturnctt}dtdDD]}|g|||j<|D]\}}|||||S)zE Construct a hash tree for rewritten byte sequences. c3K|]}|dV dS)02xNr)rxs r z3ByteRewriter.construct_hash_tree..Os&11Q**111111r)rr ranger0r%r5)r'rr$r4 in_sequence out_sequences rr#z ByteRewriter.construct_hash_treeJs %% 11eCjj111 * *A'(cIaL # #)8)>)>)@)@ @ @ %K MM)[, ? ? ? ?r byte_sequenceNcR|j}|D]}||vr ||}dS||jS)zW Search the hash tree and return the rewritten byte sequence if found. N)r$r0)r'r@r3r4s rsearch_hash_treezByteRewriter.search_hash_treeWsD~   AL  +A ttDI&&rFin_bytescjg}d}d}|t|kr|s|jn|j}t|t|D]>}||}||vr ||}n||kr|g} |}nn|j|vr||j} |}?|| |dz}|t|k|S)a6 Rewrite a sequence of bytes using the hash tree. Args: in_bytes (`List[str]`): A list of bytes to be rewritten. reverse (`bool`): If True, decoding is performed with the reverse hash tree. Returns: `List[str]`: The rewritten byte sequence. r)lenr$r&r=r0extend) r'rCreverse out_bytesb_startb_endr3jr4cur_leafs r rewrite_byteszByteRewriter.rewrite_bytesds H %%18T4>>d>TL7CMM22  QK $$#/?LL'\\ !sHEEE9 ,,+DI6HE   X & & &aiG!H %%$r)F)__name__ __module__ __qualname____doc__r0rrrr*r rr5r#rBrNrrrrr!sB D Sc4S>.A(B S S S S 0$sE$S /,B'B"C 0WZ 0or 0 0 0 0 4S> d3PUVZ\`ad\eVePfKfFg     'd3i 'E$S /"`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. extra_ids (`int`, *optional*, defaults to 125): Add a number of extra ids added to the end of the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. Extra tokens are indexed from the end of the vocabulary up to beginning ("" is the last token in the vocabulary like in ByT5 preprocessing see [here](https://github.com/google-research/text-to-text-transfer-transformer/blob/9fd7b14a769417be33bc6c850f9598764913c833/t5/data/preprocessors.py#L2117)). additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. input_idsattention_mask}Nr6c d|dkr|dt|D}nb|dkr\|Zt|dkrGtttd|}||krt d|d|dt |t rt|ddn|}t |t rt|ddn|}t |t rt|ddn|}|||d |_t|j|_ d |_ tj t|d |_t|jd |_t|jd |_t%jd|||d|d|dS)Nrcg|]}d|d S)z rris r z*MyT5Tokenizer.__init__..s$(U(U(Uq):a):):):(U(U(Urc>tdt|vS)Nextra_id)boolr)r:s rz(MyT5Tokenizer.__init__..sDs1vv9M4N4NrzBoth extra_ids (z!) and additional_special_tokens (zm) are provided to MyT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensT)lstriprstrip)rrEr<r decompose_map merge_map) eos_token unk_token pad_token extra_idsadditional_special_tokensr)r=rFsetfilterr!rrr _added_tokens_decoderoffset_utf_vocab_sizerrr byte_mapsrdecompose_rewritermerge_rewritersuperr*) r'r rjrkrlrmrnkwargs extra_tokens __class__s rr*zMyT5Tokenizer.__init__s q==6>(U(UE)DTDT(U(U(U % % ]]8DMfIgIgjkIkIks6*N*NPi#j#jkkllLy(( (y((Sl((( HRR[]`GaGapJydCCCCgp GQR[]`GaGapJydCCCCgp GQR[]`GaGapJydCCCCgp )2yY%O%O"$455 #4 C#8#899".t~o/N"O"O*4>++FGG &?         rc|jSN)rs)r's r vocab_sizezMyT5Tokenizer.vocab_sizes ##rcfdtjjzD}|j|S)Nc<i|]}||Sr)convert_ids_to_tokens)rr_r's rrz+MyT5Tokenizer.get_vocab..s)```a++A..```r)r=r}rrupdateadded_tokens_encoder)r'vocabs` r get_vocabzMyT5Tokenizer.get_vocabsI````5SWS^A^;_;_``` T./// rF token_ids_0 token_ids_1already_has_special_tokensc|r$t||dS|dgt|zdgzSdgt|zdgzdgt|zzdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNrrE)rwget_special_tokens_maskrF)r'rrrrzs rrz%MyT5Tokenizer.get_special_tokens_masks$ & 7722'[]a3   C#k***qc1 1c+&&&1#-!s;7G7G1GHA3NNr token_idsct|dkr0|d|jkrtjd|jd|S||jgzS)z.Do not add eos again if user already added it.rzThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)rF eos_token_idwarningswarnrj)r'rs r_add_eos_if_not_presentz%MyT5Tokenizer._add_eos_if_not_presentsi y>>A  )B-43D"D"D M+T^+++     122 2rcz|jg}|t||zdgzSt||z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. MyT5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)rrF)r'rreoss r$create_token_type_ids_from_sequencesz2MyT5Tokenizer.create_token_type_ids_from_sequencessS  !  {S())QC/ /;${2S899QC??rch||}||S||}||zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )r)r'rrs r build_inputs_with_special_tokensz.MyT5Tokenizer.build_inputs_with_special_tokenssA&22;??   66{CCK, ,rtextc nd|dD}||}|S)zTake as input a string and return a list of strings (tokens) for words/sub-words. Represents tokens in two character hex formatcg|]}|dS)r9rr^s rr`z+MyT5Tokenizer._tokenize..:s;;;Q**;;;rutf-8)encodemorphological_encode)r'rrxtokenss r _tokenizezMyT5Tokenizer._tokenize6s;<;dkk'&:&:;;;**622 rcbt|dkrd}nt|d|jz}|S)z0Converts a token (str) in an id using the vocab.rgN)rFintrr)r'tokentoken_ids r_convert_token_to_idz"MyT5Tokenizer._convert_token_to_id>s3 u::??HH5"~~ 3Hrc||jz d}|S)z=Converts an index (integer) in a token (str) using the vocab.r9)rr)r'indexrs r_convert_id_to_tokenz"MyT5Tokenizer._convert_id_to_tokenHs4;&,, rindicescv|j|d}|j|d}|S)NFrH)rurNrvr'rs rrz"MyT5Tokenizer.morphological_encodeMs=)777OO%33GU3KKrcv|j|d}|j|d}|S)NTr)rvrNrurs rmorphological_decodez"MyT5Tokenizer.morphological_decodeSs=%33GT3JJ)777NNrcd}g}|D]`}||jvr!||j|,||jvr||K||a||}t |jt |jz}|D]7}||vr|t |dz }|t |z }8|dd}|S)z:Converts a sequence of tokens (string) in a single string.rrignore)errors) added_tokens_decoderappendrrrovaluesbytesfromhexdecode)r'rbstring out_tokensr _added_tokensstrings rconvert_tokens_to_stringz&MyT5Tokenizer.convert_tokens_to_stringYs%  ) )E111!!$";E"BCCCC$333!!%((((!!%((((..z:: D5<<>>??#dF_B`B``  0 0E %%50005==///99 rsave_directoryfilename_prefixc|tj|r6tj||r|dzndtdz}n |r|dznd|z}t |dd5}|tj|j dd dddn #1swxYwY|fS) N-r wr)encodingrgF)indent ensure_ascii) ospathisdirjoinVOCAB_FILES_NAMESrwriterdumpsrt)r'rrr writers rsave_vocabularyzMyT5Tokenizer.save_vocabularyps 7== ( ( ]/!Q3!6!6rUfgsUt tJJ4CJ/C//n\J *cG 4 4 4 S LLDN15QQQ R R R S S S S S S S S S S S S S S S}s40B00B47B4)rWrXrYrZN)r6N)NFr|)rOrPrQrRmodel_input_namesrvocab_files_namesr*propertyr}rrrrrcrrrrrrrrrrrrr __classcell__)rzs@rrTrTs4%&67) "&,  , , , , , , \$$X$sxOO9O3;DI3FOkoO cOOOOOO8 3c 3tCy 3 3 3 3JN@@9@3;DI3F@ c@@@@0JN--9-3;DI3F- c----4cS  DI$s) DI$s) .  c HSM ]bcf]g        rrT)rRrrr collectionsrtypingrrrrrtokenization_utilsr r utilsr get_loggerrOloggerrrrTrrrrs )( ######55555555555555AAAAAAAA  H % %"#34ccccccccLrrrrr'rrrrrr