gP>dZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddlmZerdd lmZdd lmZmZejeZd d iZd ZGddeZdS)z$Tokenization class for SigLIP model.N)copyfile) TYPE_CHECKINGAnyDictListOptionalTuple)import_protobuf)PreTrainedTokenizer) AddedToken) TextInput)loggingrequires_backends vocab_filez spiece.modelu▁c 4eZdZdZeZddgZ d(d eee e fd dffd Z d Z e d ZdZ d)deedeeeded eeffd Zdeed eefdZ d*deedeeed eefdZ d*deedeeed eefdZdZdZde d e fdZdddZd+ddd ee ffd Ze d Zd!Zd"Zd#Zd$Z d*d%e d&ee d e!e fd'Z"xZ#S),SiglipTokenizera Construct a Siglip tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. model_max_length (`int`, *optional*, defaults to 64): The maximum length (in number of tokens) for model inputs. do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. input_idsattention_maskN@Tsp_model_kwargsreturnc t|dt|trt|ddddn|}t|trt|ddddn|}t|trt|ddddn|}|in||_||_||_||_||_tj d|||||j||d| dS)NprotobufTF)rstriplstrip normalizedspecial) eos_token unk_token pad_tokenadditional_special_tokensrmodel_max_length do_lower_case) r isinstancestrr rr&rget_spm_processorsp_modelsuper__init__) selfrr!r"r#r$rr%r&kwargs __class__s j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/siglip/tokenization_siglip.pyr-zSiglipTokenizer.__init__XsA $ +++)S)) JyduVZ [ [ [ [ )S)) JyduVZ [ [ [ [ )S)) JyduVZ [ [ [ [  &5%z-SiglipTokenizer.get_vocab..s)RRRa++A..RRRr2)rangerIupdateadded_tokens_encoder)r.vocabs` r1 get_vocabzSiglipTokenizer.get_vocabs@RRRR5;Q;QRRR T./// r2F token_ids_0 token_ids_1already_has_special_tokensc|r$t||dS|dgt|zdgzSdgt|zdgzdgt|zzdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rUrVrWNr)r,get_special_tokens_masklen)r.rUrVrWr0s r1rZz'SiglipTokenizer.get_special_tokens_masks$ & 7722'[]a3   C#k***qc1 1c+&&&1#-!s;7G7G1GHA3NNr2 token_idsct|dkr0|d|jkrtjd|jd|S||jgzS)z.Do not add eos again if user already added it.rzThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)r[ eos_token_idwarningswarnr!)r.r\s r1_add_eos_if_not_presentz'SiglipTokenizer._add_eos_if_not_presentsi y>>A  )B-43D"D"D M+T^+++     122 2r2cz|jg}|t||zdgzSt||z|z|zdgzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)r_r[)r.rUrVeoss r1$create_token_type_ids_from_sequencesz4SiglipTokenizer.create_token_type_ids_from_sequencessS  !  {S())QC/ /;${2S899QC??r2ch||}||S||}||zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. )rb)r.rUrVs r1 build_inputs_with_special_tokensz0SiglipTokenizer.build_inputs_with_special_tokenssA&22;??   66{CCK, ,r2cB|j}d|d<|S)Nr+)__dict__copy)r.states r1 __getstate__zSiglipTokenizer.__getstate__s$ ""$$ j r2c||_t|dsi|_tjdi|j|_|j|jdS)Nrr')rihasattrrr5r6r+Loadr)r.ds r1 __setstate__zSiglipTokenizer.__setstate__s_ t.// &#%D 2JJT5IJJ  4?+++++r2textct|tddtjS)N) translater) maketransstring punctuation)r.rrs r1remove_punctuationz"SiglipTokenizer.remove_punctuation s'~~cmmBF4FGGHHHr2keep_punctuation_exact_stringc|r5|fd||D}n|}tjdd|}|}|S)aReturns canonicalized `text` (puncuation removed). Args: text (`str`): String to be canonicalized. keep_punctuation_exact_string (`str`, *optional*): If provided, then this exact string is kept. For example providing '{}' will keep any occurrences of '{}' (but will still remove '{' and '}' that appear separately). c3BK|]}|VdSrF)ry)rMpartr.s r1 z4SiglipTokenizer.canonicalize_text..sB6626''--666666r2z\s+ )joinsplitryresubstrip)r.rrr{s` r1canonicalize_textz!SiglipTokenizer.canonicalize_texts ) 10556666:>**Eb:c:c666DD**400Dvfc4((zz|| r2rc tjt|tdzfi|}t |dkr*|dtkr|d|jvr |dd}|S)z8 Converts a string to a list of tokens. rrYrN)r,tokenizeSPIECE_UNDERLINEreplacer[all_special_tokens)r.rradd_special_tokensr/tokensr0s r1rzSiglipTokenizer.tokenize&s{"!"2T\\BRTW5X5X"Xcc\bcc v;;??vay,<<<dNeAeAeABBZF r2ctt|jt|jSrF)r[r+encoder)r"rHs r1unk_token_lengthz SiglipTokenizer.unk_token_length0s,4=''DN(;(;<<===r2c ||d}|j|t}|j|j|zt}t ||jkr||jdn|S)u* Returns a tokenized string. We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any SPIECE_UNDERLINE. For example: `self.sp_model.encode(f"{SPIECE_UNDERLINE}Hey", out_type = str)` will give `['H', 'e', 'y']` instead of `['▁He', 'y']`. Thus we always encode `f"{unk_token}text"` and strip the `unk_token`. Here is an example with `unk_token = ""` and `unk_token_length = 4`. `self.tokenizer.sp_model.encode(" Hey", out_type = str)[4:]`. Nrz)out_type)rr+rr)r"r[r)r.rrr/rs r1 _tokenizezSiglipTokenizer._tokenize5s%%d$%OO%%dS%99%%dnt&;c%JJ25f++AV2V2Vvd+--..\bbr2c6|j|S)z0Converts a token (str) in an id using the vocab.)r+ piece_to_id)r.tokens r1_convert_token_to_idz$SiglipTokenizer._convert_token_to_idJs}((///r2c:|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)r+ IdToPiece)r.indexrs r1_convert_id_to_tokenz$SiglipTokenizer._convert_id_to_tokenOs ''.. r2cg}d}d}|D]N}||jvr,|s|dz }||j||zz }d}g}7||d}O||j|z }|S)z:Converts a sequence of tokens (string) in a single string.rtFrT)rr+decodeappendr)r.rcurrent_sub_tokens out_stringprev_is_specialrs r1convert_tokens_to_stringz(SiglipTokenizer.convert_tokens_to_stringTs  ( (E///&&#%Jdm223EFFNN "&%'"""))%000"'dm**+=>>> !!!r2save_directoryfilename_prefixctj|s td|ddStj||r|dzndt dz}tj|jtj|kr:tj |jrt|j|nztj |jsVt|d5}|j }||dddn #1swxYwY|fS)NzVocabulary path (z) should be a directory-rtrwb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathrisfilerr7r+serialized_model_protowrite)r.rrout_vocab_fileficontent_spiece_models r1save_vocabularyzSiglipTokenizer.save_vocabularyhsww}}^,,  LLT^TTT U U U F oM_s222QbcoQpp   7??4? + +rw~/N/N N NSUSZSaSabfbqSrSr N T_n 5 5 5 500 /nd++ /r'+}'K'K'M'M$-... / / / / / / / / / / / / / / /  s(/E##E'*E')rrrNNrT)NFrF)F)$__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrrr)rr-r*propertyrIrTrintboolrZrbrergrlrqryrrrrrrrr r __classcell__)r0s@r1rr,s.&&P*$&67 "&48/ / "$sCx.1/  / / / / / / b   ..X.sxOO9O3;DI3FOkoO cOOOOOO: 3c 3tCy 3 3 3 3JN@@9@3;DI3F@ c@@@@0JN--9-3;DI3F- c----6 ,,,IsIsIIIIHL*[QUVYQZ>>X>ccc*000  """(!!c!HSM!]bcf]g!!!!!!!!r2r) rrrrwr`shutilrtypingrrrrrr sentencepiecer5convert_slow_tokenizerr tokenization_utilsr tokenization_utils_baser rutilsrr get_loggerrrrrrr'r2r1rsO+* BBBBBBBBBBBBBBBB5555555555551111115444444////////  H % %!>2K!K!K!K!K!)K!K!K!K!K!r2