gG$ UdZddlmZmZmZddlmZmZddlm Z e j e Z dZ dZdZdZd Zd Zd Zed ed edededediZeeefed<deDZeeefed<GddeZdS)z Tokenization classes for CANINE.)DictListOptional) AddedTokenPreTrainedTokenizer)loggingiiiiiiz[CLS]z[SEP]z[BOS]z[MASK]z[PAD]z [RESERVED]SPECIAL_CODEPOINTSci|]\}}|| Sr ).0 codepointnames j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/canine/tokenization_canine.py r7s-p-p-p/)TdI-p-p-pSPECIAL_CODEPOINTS_BY_NAMEc eZdZdZeeeeeeeeeeeeddffd Z e de fdZ dZ dedeefd Zd ede fd Zd e defd ZdZ ddee deee dee fdZ ddee deee dedee ffd Z ddee deee dee fdZddedeefdZxZS)CanineTokenizera Construct a CANINE tokenizer (i.e. a character splitter). It turns text into a sequence of characters, and then converts each character into its Unicode code point. [`CanineTokenizer`] inherits from [`PreTrainedTokenizer`]. Refer to superclass [`PreTrainedTokenizer`] for usage examples and documentation concerning parameters. Args: model_max_length (`int`, *optional*, defaults to 2048): The maximum sentence length the model accepts. Fic 2t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}t|trt|ddn|}i|_tD]\} } | |j| <d|jD|_t|_t|j|_ tj d||||||||d| dS)NF)lstriprstripTci|]\}}|| Sr r )r rrs rrz,CanineTokenizer.__init__..cs+; ; ; /iIt; ; ; r) bos_token eos_token sep_token cls_token pad_token mask_tokenadd_prefix_spacemodel_max_lengthr ) isinstancestrr_special_codepointsr items_special_codepoint_stringsUNICODE_VOCAB_SIZE_unicode_vocab_sizelen_num_special_tokenssuper__init__) selfrrrrrrr r!kwargsrr __class__s rr,zCanineTokenizer.__init__HsJTT]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir KUU_adJeJeuZ 4FFFFku 46 17799 7 7OIt-6D $T * *; ; 373K3Q3Q3S3S; ; ; '$6 #&t'?#@#@  !--  rreturnc|jSN)r()r-s r vocab_sizezCanineTokenizer.vocab_sizevs ''rcvdt|jD}||j|S)Nc.i|]}t||Sr )chr)r is rrz-CanineTokenizer.get_vocab..{s ;;;qQ;;;r)ranger3updateadded_tokens_encoder)r-vocabs r get_vocabzCanineTokenizer.get_vocabzs9;;E$/$:$:;;; T./// rtextc t|S)z5Tokenize a string (i.e. perform character splitting).)list)r-r=s r _tokenizezCanineTokenizer._tokenizesDzzrtokencd t|S#t$rtd|dwxYw)zaConverts a token (i.e. a Unicode character) in an id (i.e. its integer Unicode code point value).zinvalid token: '')ord TypeError ValueError)r-rAs r_convert_token_to_idz$CanineTokenizer._convert_token_to_idsH :u::  : : :888899 9 :s/indexc |tvr t|St|S#t$rtd|wxYw)z Converts a Unicode code point (integer) in a token (str). In case it's a special code point, convert to human-readable format. z invalid id: )r r6rErF)r-rHs r_convert_id_to_tokenz$CanineTokenizer._convert_id_to_tokens\  5***)%00u::  5 5 53E3344 4 5s ''Ac,d|S)N)join)r-tokenss rconvert_tokens_to_stringz(CanineTokenizer.convert_tokens_to_stringswwvrN token_ids_0 token_ids_1cJ|jg}|jg}||z|z}||||zz }|S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CANINE sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) sep_token_id cls_token_idr-rPrQsepclsresults r build_inputs_with_special_tokensz0CanineTokenizer.build_inputs_with_special_tokenssC& ! !{"S(  " kC' 'F ralready_has_special_tokensc|r$t||dSdgdgt|zzdgz}||dgt|zdgzz }|S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rPrQrZr)r+get_special_tokens_maskr))r-rPrQrZrXr/s rr]z'CanineTokenizer.get_special_tokens_masks$ & 7722'[]a3 c+.../1#5  " sS---!4 4F rc|jg}|jg}t||z|zdgz}||t||zdgzz }|S)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. A CANINE sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rNr\)rSrTr)rUs r$create_token_type_ids_from_sequencesz4CanineTokenizer.create_token_type_ids_from_sequencessa. ! !S;&,--3  " c++,,s2 2F rsave_directoryfilename_prefixcdS)Nr r )r-r`ras rsave_vocabularyzCanineTokenizer.save_vocabularysrrr2)NF)__name__ __module__ __qualname____doc__r6CLSSEPPADMASKr,propertyintr3r<r#rr@rGrJrOrrYboolr]r_rc __classcell__)r/s@rrr:sZ  #c((#c((#c((#c((#c((3t99, , , , , , \(C(((X( cd3i:#:#:::: 5# 5# 5 5 5 5JN93;DI3F c8sx93;DI3Fko c:JN93;DI3F c@cHSMrrN)rgtypingrrrtokenization_utilsrrutilsr get_loggerrdloggerr'rjrhriBOSrkRESERVEDr rmr#__annotations__r%rrr rrrxsK'&&''''''''''AAAAAAAA  H % %     ( l &DcN    .q-pUgUmUmUoUo-p-p-pDcNpppwwwww)wwwwwr