g`P dZddlZddlZddlZddlmZddlmZmZm Z ddl Z ddl m Z mZmZmZmZddlmZejeZdd d Zed Zd Zd ZdZGddZGddeZdS)zTokenization classes for CLIP.N) lru_cache)ListOptionalTuple) AddedTokenPreTrainedTokenizer _is_control_is_punctuation_is_whitespace)loggingz vocab.jsonz merges.txt) vocab_file merges_filec \tttdtddztttdtddzztttdtddzz}|dd}d }td D]8}||vr2|||d |z|dz }9d |D}t t ||S) a8 Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. !~¡¬®ÿNrc,g|]}t|S)chr).0ns f/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/clip/tokenization_clip.py z$bytes_to_unicode..:s   Q#a&&   )listrangeordappenddictzip)bscsrbs rbytes_to_unicoder*%s U3s88SXX\ * *++d5TCIIPQM3R3R.S.SSVZ[`adeiajajloptluluxyly[z[zV{V{{ AAAB A 4[[ B;; IIaLLL IIdQh    FA  "   B B  r c~t}|d}|ddD]}|||f|}|S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairs prev_charchars r get_pairsr2>sP EEEQIQRR 9d#$$$ Lr cZtjdd|}|}|S)Nz\s+ )resubstrip)texts rwhitespace_cleanr9Ls' 6&#t $ $D ::<>>SWSc %%  & 6//55D"-"7t"D"D)*ABB    M MEK''%;!KKMME)66 $ 7 7 > >'; 33E::E    7 7{ K K L L L L+CHH\,B,BCC r ctjd|}g}|D]2}tj|}|dkr||3d|S)z$Strips accents from a piece of text.NFDMn)rNrOcategoryr$rT)rGr8outputr1cats rrQz!BasicTokenizer._run_strip_accentssi$UD11  D&t,,Cd{{ MM$    wwvr c|jr|||vr|gSt|}d}d}g}|t|kr|||}t|r||gd}n4|r|gd}|d||dz }|t|k|d|DS)z&Splits punctuation on a piece of text.NrTFrc8g|]}d|S)r^)rT)rxs rrz5BasicTokenizer._run_split_on_punc..s"+++q +++r )rFr!lenr r$)rGr8rCcharsistart_new_wordr`r1s rrSz!BasicTokenizer._run_split_on_puncs$ )@T[EXEX6MT  #e**nn8Dt$$ ( tf%%%!%!&MM"%%%!&r !!$''' FA#e**nn,+F++++r c,g}|D]{}t|}||r@|d|||df|||d|S)z)Adds whitespace around any CJK character.r4r^)r#_is_chinese_charr$rTrGr8r`r1cps rrMz&BasicTokenizer._tokenize_chinese_charss $ $DTB$$R(( $ c""" d### c"""" d####wwvr c|dkr|dksT|dkr|dksH|dkr|dks<|dkr|dks0|d kr|d ks$|d kr|d ks|d kr|dks |dkr|dkrdSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr)rGrms rrkzBasicTokenizer._is_chinese_chars6\\bFllf vg "--g "--g "--g "--f vg "--4ur cg}|D]g}t|}|dks|dkst|r-t|r|dR||hd|S)zBPerforms invalid character removal and whitespace cleanup on text.rir4r^)r#r r r$rTrls rrLzBasicTokenizer._clean_texts $ $DTBQww",,+d*;*;,d## $ c"""" d####wwvr )TNTNTrA) __name__ __module__ __qualname____doc__rHrZrQrSrMrkrLrr rr?r?]s0# 1111 $$$$L   ,,,,,   0     r r?c eZdZdZeZddgZ dfd ZedZ d Z dd e e d e e e d e e fdZ dd e e d e e e ded e e ffd Z dd e e d e e e d e e fdZdZdZdZdZdZddede ed eefdZxZS) CLIPTokenizera Construct a CLIP tokenizer. Based on byte-level Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|startoftext|>"`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskreplace <|endoftext|><|startoftext|>c t|trt|ddn|}t|trt|ddn|}t|trt|ddn|} ddl} | j|_nG#t $r:t dtdd|_ d|_YnwxYwt|d5} tj | |_ dddn #1swxYwYd|j D|_||_t#|_d |jD|_t|d5} | d d d } dddn #1swxYwYd | D} t/t1| t3t5| |_ddd|_t;jdt:j|_ tCj"d|||||d|dS)NF)lstriprstriprzKftfy or spacy is not installed using custom BasicTokenizer instead of ftfy.)rErFutf-8encodingci|]\}}|| Srrrkvs r z*CLIPTokenizer.__init__..4s>>>A1>>>r ci|]\}}|| Srrrs rrz*CLIPTokenizer.__init__..7sHHHdaQHHHr  ricPg|]#}t|$Sr)tupler;)rmerges rrz*CLIPTokenizer.__init__..:s(CCCueEKKMM**CCCr rzry)rzryz[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)errors unk_token bos_token eos_token pad_tokenr)# isinstancestrrftfyfix_text ImportErrorloggerinfor?nlpopenjsonloadencoderitemsdecoderrr* byte_encoder byte_decoderreadr7r;r%r&r"rf bpe_rankscacher5compile IGNORECASEpatsuperrH)rGrrrrrrrkwargsr vocab_handle merges_handle bpe_merges __class__s rrHzCLIPTokenizer.__init__sJTT]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir  ! KKK MDMM ! ! ! KKe f f f%EERRRDH DMMM ! *w / / / 3<9\22DL 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3>>););)=)=>>>  ,..HHd.?.E.E.G.GHHH + 0 0 0 [M&++--3355;;DAA!FYBYZJ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [CC CCC c*eC OO.D.DEEFF):_]] : n M             s8>BACC'D  DDAG!!G%(G%c*t|jSrA)rfrrGs r vocab_sizezCLIPTokenizer.vocab_sizeLs4<   r c0t|jfi|jSrA)r%radded_tokens_encoderrs r get_vocabzCLIPTokenizer.get_vocabPsDL>>D$=>>>r N token_ids_0 token_ids_1returncX|jg}|jg}|||z|zS||z|z|z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CLIP sequence has the following format: - single sequence: `<|startoftext|> X <|endoftext|>` Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) bos_token_id eos_token_idrGrrrrs r build_inputs_with_special_tokensz.CLIPTokenizer.build_inputs_with_special_tokensSsO(&' &'  {*Y6 6;&2Y>LyXXr Falready_has_special_tokensc|r$t||dS|dgdgt|zzdgzSdgdgt|zzdgzdgzdgt|zzdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNrr)rget_special_tokens_maskrf)rGrrrrs rrz%CLIPTokenizer.get_special_tokens_maskns& & 7722'[]a3   31#K 0 001QC7 7sqcC ,,,-3qc9aS3{CSCS=STXYWZZZr c|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed. CLIP does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)rrrfrs r$create_token_type_ids_from_sequencesz2CLIPTokenizer.create_token_type_ids_from_sequencessp &' &'  y;.:;;qcA A9{*Y6B[PS\\]]ab`cccr c|jvr j|St|dd|ddzfz}t|}|s|dzS t|fd}|jvrn8|\}}g}d}|t |kr |||} |||| | }n-#t$r |||dYnwxYw|||krC|t |dz kr-||dz|kr| ||z|dz }n | |||dz }|t |kt|}|}t |dkrnt|}Wd |}|j|<|S) NrcTcTj|tdS)Ninf)rgetfloat)pairrGs rz#CLIPTokenizer.bpe..s 1C1CD%PU,,1W1Wr keyrrr4) rrr2minrrfindexrR ValueErrorr$rT) rGrXr.r/bigramfirstsecondnew_wordrhjs ` rbpezCLIPTokenizer.bpes DJ  :e$ $U3B3Z  E"I$6#88$ "6> ! ($W$W$W$WXXXFT^++"ME6HAc$ii-- 5!,,A OOD1I...AA "OODH---E7e##CIIM(9(9d1q5kV>S>SOOEFN333FAAOODG,,,FAc$ii-- XHD4yyA~~!$9 (:xx~~  5 sC'C/.C/c g}j.dj|}n4t |}t jj|D]{}dfd| dD}| d | dD||S)zTokenize a string.Nr4r^c32K|]}j|VdSrA)r)rr)rGs r z*CLIPTokenizer._tokenize..s<)*!!$r r~c3K|]}|VdSrAr)r bpe_tokens rrz*CLIPTokenizer._tokenize..s"TTIiTTTTTTr ) rrTrrZr9rPr5findallrencoderRrr;)rGr8 bpe_tokensrXs` r _tokenizezCLIPTokenizer._tokenizes = 88DH--d3344DD#DMM$$7$788>>@@DZ$// U UEGG.3ll7.C.CE   TT%9N9Ns9S9STTT T T T Tr cr|j||j|jS)z0Converts a token (str) in an id using the vocab.)rrr)rGrXs r_convert_token_to_idz"CLIPTokenizer._convert_token_to_ids,|t|'7'7'G'GHHHr c6|j|S)z=Converts an index (integer) in a token (str) using the vocab.)rr)rGrs r_convert_id_to_tokenz"CLIPTokenizer._convert_id_to_tokens|&&&r cd|}tfd|D}|djdd}|S)z:Converts a sequence of tokens (string) in a single string.r^c*g|]}j|Sr)r)rcrGs rrz:CLIPTokenizer.convert_tokens_to_string..s!CCC 1! 4CCCr r~)rrr4)rT bytearraydecoderrxr7)rGr<r8 byte_arrays` rconvert_tokens_to_stringz&CLIPTokenizer.convert_tokens_to_stringsowwvCCCCdCCCDD    ==EEfcRRXXZZ r save_directoryfilename_prefixc tj|s/td|dStj||r|dzndtdz}tj||r|dzndtdz}t|dd5}| tj |j d d d d zdddn #1swxYwYd}t|dd5}| dt|jdD]j\}} || kr/td|| }| d|d z|dz }k dddn #1swxYwY||fS)Nz*Vocabulary path ({}) should be a directory-r^rrwr~rrTF)indent sort_keys ensure_asciirrz#version: 0.2 c|dS)Nrr)kvs rrz/CLIPTokenizer.save_vocabulary..s Y[\]Y^r rzqSaving vocabulary to {}: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!r4r)ospathisdirrerrorformatrTVOCAB_FILES_NAMESrwriterdumpsrsortedrrwarning) rGrrr merge_filefrwriterr token_indexs rsave_vocabularyzCLIPTokenizer.save_vocabularysw}}^,,  LLELL^\\ ] ] ] FW\\ oM_s222QbcoQpp  W\\ oM_s222QbcpQqq  *cG 4 4 4 c GGDJt|ATYZZZ]aa b b b c c c c c c c c c c c c c c c *cG 4 4 4  LL* + + ++1$.2F2F2H2HN^N^+_+_+_  ' KK''NNMMSVT^M_M_(E SXXj11D8999                 :%%s% 4D  DD*B*G!!G%(G%)rxryrzryryrA)NF)rprqrrrsrvocab_files_namesmodel_input_namesrHpropertyrrrintrrboolrrrrrrrrrr __classcell__)rs@rrurus'2*$&67 !#!!/ / / / / / b!!X!???JNYY9Y3;DI3FY cYYYY8sx[[9[3;DI3F[ko[ c[[[[[[:JNdd9d3;DI3Fd cdddd.(((T   III'''&&c&HSM&]bcf]g&&&&&&&&r ru)rsrrrN functoolsrtypingrrrregexr5tokenization_utilsrr r r r utilsr get_loggerrprrr*r2r9r=r?rurr rr s%$ ((((((((((oooooooooooooo  H % %   0   ^^^^^^^^BF&F&F&F&F&'F&F&F&F&F&r