gdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZddlmZmZmZmZd d lmZd d lmZd d l m!Z!d d l"m#Z#d dl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.d dl/m0Z0m1Z1m2Z2e2j3e4Z5dZ6dZ7dZ8dZ9dZ:e%dz Z%eeeedZ;e6e9dZ<e1e%Gdde*Z=dS)z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)AnyDictListOptionalTupleUnion)Encoding) Tokenizer)Decoder) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer)convert_slow_tokenizer)convert_gguf_tokenizer)load_gguf_checkpoint)PreTrainedTokenizer) INIT_TOKENIZER_DOCSTRING AddedToken BatchEncodingPreTokenizedInputPreTokenizedInputPairPreTrainedTokenizerBaseSpecialTokensMixin TextInput TextInputPairTruncationStrategy)PaddingStrategyadd_end_docstringsloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonztokenizer.modelzadded_tokens.jsonu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. )BPEUnigram WordLevel WordPiece)tokenizer_file vocab_filec)feZdZUdZeZdZeed<fdZ e de fdZ e de fdZ e defdZdeeeffd Ze deeeffd Ze deeeffd Ze deeeffd Zdeeeffd ZdefdZe defdZe defdZ dEdedee dee de de de de de deeee fe!effdZ"de#ee!efde#ee!effdZ$dedefdZ%d edeefd!Z&dFd"e!e#eefdefd#Z'dFd$e defd%Z( dFd&e#ee!efd'e de#ee!effd(Z)dGd)ed$eed*e de!efd+Z*d,e+d-e,d.ed/ed0eed1ee f d2Z-de+j.e,j/dd3ddddddddddddfd4e#e!e0e!e1e!e2e!e3fd*e d,e+d-e,d.eed/ed5e d0eed1ee d6eedee dee de de de de de d7e de4f&d8Z5dde+j.e,j/dd3ddddddddddddfd)e#e0e2fd9ee#e0e2fd*e d,e+d-e,d.eed/ed5e d0eed1ee d6ee dee dee de de de de de d7e de4f(d:Z6de!edefd;Z7 dHdZ8 dId?e#ee9j:fd@eedAee dBeedeef dCZ; dJdDZ<xZ=S)KPreTrainedTokenizerFastaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Nslow_tokenizer_classc |dd}|dd}|dd}|dd}|dd}|di}|r||jtd|tj|} nC||st j|} n)|rt|} n|t| d } | d d } | d } | d } t| | \} }| | t|dkr| |n|j!|dur|j|i|}t|} n[|sJ| d d|_ | dg|_t|d} d}ntd| |_|| |jd|_|jj}||jjd&i||d|d|d|d|d|d|d|dn|j|jj}||jjd&i||d|d|d|d|d|d|d|d|d|dt1jd&i||j|j_d|jDfd t;|d!"Dt?|j !d#Dzfd$|j"Dz tdkrg}|j#}D]}tI|tJr|j&ptO||vntO||v}tI|tNrtK||%}n||_&|(||r|)|dSdSdS)'Ntokenizer_object__slow_tokenizer gguf_filer' from_slowFadded_tokens_decoderzCannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.r(config model_type tokenizertokenizer_configradditional_special_tokensT) from_tiktokena9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one. max_lengthtruncation_side directionstridetruncation_strategystrategy pad_tokenpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofcFh|]}tt|Shashrepr.0tokens `/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py z3PreTrainedTokenizerFast.__init__..s&$^$^$^5T$u++%6%6$^$^$^cVg|]%\}}tt|v#|&SrErF)rJindexrKadded_tokens_decoder_hashs rL z4PreTrainedTokenizerFast.__init__..sA   uDKK  (AAA AAArNc|dSNrrE)xs rLz2PreTrainedTokenizerFast.__init__..s STUVSWrNkeyc,g|]}t|SrE)strrIs rLrRz4PreTrainedTokenizerFast.__init__..s;b;b;b5CJJ;b;b;brNc$g|] }|v|v | SrErE)rJrKencoder tokens_to_adds rLrRz4PreTrainedTokenizerFast.__init__..s7   5PWCWCW\aiv\v\vE\v\v\vrN)specialrE)*popr+ ValueErrorcopydeepcopy TokenizerFast from_filerrgetrupdatelenr(r6 _tokenizer init_kwargs_decode_use_source_tokenizer truncationenable_truncation setdefault no_truncationpaddingenable_paddingsuper__init__split_special_tokensencode_special_tokensr1sorteditemslistadded_tokens_encoderkeysall_special_tokens_extendedall_special_tokens isinstancerr^rZappend add_tokens)selfargskwargsr-slow_tokenizerr/fast_tokenizer_filer0r1fast_tokenizer gguf_param architecturetokenizer_dictr5additional_kwargs _truncation_paddingtokensspecial_tokensrK is_specialrQr\r] __class__s @@@rLrrz PreTrainedTokenizerFast.__init__bs!::&8$??$6==JJ{D11 $jj)94@@JJ{E22 %zz*@"EE  /D4M4U0   '!]+;< > >   3[5L M M M M O ) ) + + +?*   *DO * 6 6X 6 6 6   k8K+@ A A A   18M3J K K K   nh{.C D D D   lHX,> ? ? ?   2H=Q4R S S S ""6"""040I-$^$^DD]$^$^$^!    &';'A'A'C'C X X X   t0557788;b;bTa;b;b;bb     #?     }   ! !F!4N& % %"%446U]Bc%jjN&BU~5 eS))/&ujAAAEE$.EM e$$$$ (''''' " ! ( (rNreturncdS)NTrErs rLis_fastzPreTrainedTokenizerFast.is_faststrNcdS)z `bool`: Whether or not the slow tokenizer can be saved. Usually for sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. TrErs rLcan_save_slow_tokenizerz/PreTrainedTokenizerFast.can_save_slow_tokenizers trNc8|jdS)zP `int`: Size of the base vocabulary (without the added tokens). Fwith_added_tokensrhget_vocab_sizers rL vocab_sizez"PreTrainedTokenizerFast.vocab_sizes ---FFFrNc8|jdS)NTr)rh get_vocabrs rLrz!PreTrainedTokenizerFast.get_vocabs((4(@@@rNc*|SN)rrs rLvocabzPreTrainedTokenizerFast.vocabs~~rNchdt|jdDS)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. c$i|] \}}|j|SrEcontentrJvks rL z@PreTrainedTokenizerFast.added_tokens_encoder.. mmmA 1mmmrNc|dSrTrEitems rLrVz>PreTrainedTokenizerFast.added_tokens_encoder.. dhijdkrNrWrur1rvrs rLrxz,PreTrainedTokenizerFast.added_tokens_encoders9 nm0I0O0O0Q0QWkWk)l)l)lmmmmrNc4|jS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `Dict[str, int]`: The added tokens. )rhget_added_tokens_decoderrs rLr1z,PreTrainedTokenizerFast.added_tokens_decoders77999rNchdt|jdDS)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. c$i|] \}}|j|SrErrs rLrz;PreTrainedTokenizerFast.get_added_vocab..rrNc|dSrTrErs rLrVz9PreTrainedTokenizerFast.get_added_vocab..rrNrWrrs rLget_added_vocabz'PreTrainedTokenizerFast.get_added_vocabs9nm0I0O0O0Q0QWkWk)l)l)lmmmmrNc8|jdS)zD Size of the full vocabulary with the added tokens. Trrrs rL__len__zPreTrainedTokenizerFast.__len__s---EEErNc|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rhrs rLbackend_tokenizerz)PreTrainedTokenizerFast.backend_tokenizer s rNc|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rhdecoderrs rLrzPreTrainedTokenizerFast.decoders &&rNFTencodingreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec F| d|jv}| d|jv}|r|j |g|jz} n|g} tt} | D]} | d| j|r | d| j|r | d| j|r | d| j|r | d| j |r-| dt| j| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) Ntoken_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingrB) model_input_names overflowingrrwr}idstype_idsrroffsetsrg) rrrrrrrrr encodings encoding_dictes rL_convert_encodingz)PreTrainedTokenizerFast._convert_encodingsV( ! ($48N$N ! ($48N$N ! $ #)=)I! X%99II! I#D))  ; ;A + & - -ae 4 4 4$ C./66qzBBB$ I./66q7GHHH) S34;;A.Zs'TTTE88??TTTrN)r|rZrrrs` rLconvert_tokens_to_idsz-PreTrainedTokenizerFast.convert_tokens_to_idsIsP >4 fc " " D;;FCC CTTTTVTTTTrNrKcL|j|}||jS|Sr)rh token_to_id unk_token_id)rrKrPs rLrz;PreTrainedTokenizerFast._convert_token_to_id_with_added_voc\s*++E22 =$ $ rNrPcP|jt|Sr)rh id_to_tokenint)rrPs rL_convert_id_to_tokenz,PreTrainedTokenizerFast._convert_id_to_tokenbs**3u::666rN new_tokenscn|r|j|S|j|Sr)rhadd_special_tokensr~)rrrs rL _add_tokensz#PreTrainedTokenizerFast._add_tokenses7  B?55jAA A))*555rNpairc6|j|S)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rhnum_special_tokens_to_add)rrs rLrz1PreTrainedTokenizerFast.num_special_tokens_to_addks&88>>>rNrskip_special_tokensct|tr|j|Sg}|D]J}t|}|r ||jvr||j|K|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). )r|rrhrall_special_idsr})rrrrrPs rLconvert_ids_to_tokensz-PreTrainedTokenizerFast.convert_ids_to_tokenss c3   4?..s33 3 > >EJJE" u0D'D'D MM$/55e<< = = = = rNtextrc H|jd|||d|S)N)r text_pairrrE) encode_plusr)rrrrrs rLtokenizez PreTrainedTokenizerFast.tokenizes2tkTTN`kkdjkkrrtttrNpadding_strategyr<r8r;rCrAc |jj |jj}|tjkr |jn<|||j|jd} d} n fd|D} | |kr|jjdi||tj kr||j dSdS|tj kr|nd} | ||n|j |j|j|j|d}||kr|jjdi|dSdS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)r8r;r=r:c>i|]}||dSrre)rJrrs rLrzFPreTrainedTokenizerFast.set_truncation_and_padding..s)GGG11kooa66GGGrN)rBr:pad_idr>r@rCrE)rhrkrorDO_NOT_TRUNCATErnvaluer9rlr DO_NOT_PAD no_padding MAX_LENGTHrA pad_token_idr>r?rp) rrr<r8r;rCrArtargetcurrentrBrs @rLset_truncation_and_paddingz2PreTrainedTokenizerFast.set_truncation_and_paddingsiBo0 ?* "4"D D D&--///) /5!1 F"GGGGGGG&  11;;F;;; 9 9 9#**,,,,,$#$47Q#Q#QZZW[F -9-E\\4K\+!^#5&8 F6!!..8888888"!rNrbatch_text_or_text_pairsis_split_into_wordsreturn_tensorsrsc t|ttfs tdt |d|||||| jj|kr |j_j|||}   fd|D}i}|dd D]fd|D}||<d|D} r;g}t|D]$\}\}}||gt|d zz }%||d <|d D]} ||t||| S) Nz:batch_text_or_text_pairs has to be a list or a tuple (got ))rr<r8r;rCrA)ris_pretokenizedc Jg|]}|  S))rrrrrrrr)r) rJrrrrrrrrrs rLrRz>PreTrainedTokenizerFast._batch_encode_plus..sX   " "!&;&;*C+E'=+ #   rNrc0g|]\}}|D]}|SrErE)rJr_rrXs rLrRz>PreTrainedTokenizerFast._batch_encode_plus..3s.NNN74DINNqQNNNNrNc"g|] \}}|D]}| SrErE)rJrrrs rLrRz>PreTrainedTokenizerFast._batch_encode_plus..5s)SSSWQdSSqSSSSrNroverflow_to_sample_mapping) tensor_type)r|tuplerw TypeErrortyperrhrt encode_batchry enumeraterg&_eventual_warn_about_too_long_sequencer)rrrrr<r8r;rrCrArrrrrrrrrsrtokens_and_encodingssanitized_tokensstacksanitized_encodingsritoksrrrXs` ``````` @rL_batch_encode_plusz*PreTrainedTokenizerFast._batch_encode_pluss".2UDMBB nTRjMkMknnn  ''- 3!1% (    ? 04H H H4HDO 1O00 $1/1   & ('*1-2244 * *CNNNN&:NNNE$) S ! !SS0DSSS % X)+ & )*> ? ? K K 9D!*qcC[8I4J4J.JJ**=W 9 :)+6 X XI  7 7 :w W W W W-/BP^____rNrc @|r||fgn|g}|j|fid|d|d|d|d|d|d| d| d | d | d | d |d |d|d|d|d||}| 3|s1td|D|j}||d|||S)Nrrrr<r8r;rCrArrrrrrrrrsci|]>\}}|t|dkr#t|dtr|dn|?S)r)rgr|rw)rJrXrs rLrz8PreTrainedTokenizerFast._encode_plus..usW"USZZ!^^ 58T8R8R^qX]rNr)rrrvrr )rrrrrr<r8r;rrCrArrrrrrrrrsr batched_inputbatched_outputs rL _encode_plusz$PreTrainedTokenizerFast._encode_plusCs.09D$ *++tf 00    3 3  21 .-  !4 3  "z  6  21 & *> #8"7 #8"7 '@&? (B'A $:#9 (-! "G# $"6!5'  0  !*C !*&4&:&:&<&<( N 33N;4OQ[]deeerNc@|jj|Sr)rrdecoders rLconvert_tokens_to_stringz0PreTrainedTokenizerFast.convert_tokens_to_strings%-44V<<.s.vvv*#u]bfjfu]u]u3]u]u]urNwzutf-8)r)indent sort_keys ensure_ascii )r&)rZr+r`rospathjoinADDED_TOKENS_FILErxrvopenjsondumpswritesave_vocabularyTOKENIZER_FILErsave) rr#r$r%r& save_slow save_fastadded_tokens_file added_vocabfout_str vocab_filesr's ` rL_save_pretrainedz(PreTrainedTokenizerFast._save_pretraineds^,,  $ ,$1F1F`  d " ;mt&; -)5 -,  "T)C]e-C  I " /!Q3!6!6rUf f!! wvvv8Q8W8W8Y8YvvvK %+S7CCC%q"jQ$]bcccfjjGGGG$$$%%%%%%%%%%%%%%%..~.__K#k15F4HHJ  8W\\/!Q3!6!6rUc cN  " ' ' 7 7 7#~&77Js51C22C69C6c f tj|j}|d}|d} d} |dddkri|dd<g|dd<n|ddd kr^|dd O|dd } |dd| d }  | vr| } d |dd <| d gg|dd<n;|ddd vr i|dd<nt d|ddd7d|dvr-|ddvr|dd|dd<t jtj|g} |D]} | dd}| dd}|ddd kr|sC| dvr| d| d<| td*i| || ||dddkr#d|vr|dd|dd|d<|dddkr#d|vr|dd|dd|d<|ddd kr| | |d<|d|dddksA|dddkrPd|dvrFtd|ddDr!tj|d<t |dd}|d*|| d|}|||| ,tj}d| vr| dD]}| d|d}fd |D}|| d|d<|D](}|}|t d!)fd"|D| d|d#<d$D]L}|| vrF| |\}} |vr|}|}|t d!||g| |<M| |d<t jtj||j}t*j}|d%|D]}t1|d&|xt1||} |vr|}t1|d&|}t3|tr-t||j|j|j|jd'(||<|||<|j}|| |t?|d kr||d%<|j d*d)i|S)+uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `List[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`Dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokenspost_processorNmodelr r#rmergesr$unk_idrg)r%r&z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece. unk_tokenr^idrcontinuing_subword_prefixend_of_word_suffix pre_tokenizer ByteLevelSequence pretokenizersc3.K|]}|ddkVdS)r rPNrE)rJ pretokenizers rL zBPreTrainedTokenizerFast.train_new_from_iterator..4s@ V$ 3rNinitial_alphabet)rr)rBtrainerrrc<g|]}||SrEr)rJrKspecial_tokens_maps rLrRzCPreTrainedTokenizerFast.train_new_from_iterator..Fs*![![![5"4"8"8"F"F![![![rNzQAttempted to set a token in the post processor that does not exist in the mappingc:g|]}|SrE)r)rJrKr4s rLrRzCPreTrainedTokenizerFast.train_new_from_iterator..Os)CuCuCuejIDYDYZ_D`D`CuCuCurNr)clssepr6rT) single_wordlstriprstrip normalizedr^r-rE)!r7loadsrhto_strr_r`rcfrom_strr8r}rextendanypre_tokenizers_fastrPalphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrrirarSPECIAL_TOKENS_ATTRIBUTESremovegetattrr|r]r^r_r`r6rgr)r text_iteratorrrBnew_special_tokensrYrtokenizer_jsonrFrGrKrJr added_tokenr^r trainer_classrWtrained_tokenizer_jsonrXrrKtoken_id special_tokenspecial_tokens_listspecial_token_fullr6r4s ` @rLtrain_new_from_iteratorz/PreTrainedTokenizerFast.train_new_from_iterators=DDO$:$:$<$<==%)).99 '++,<== ' "6 *e 3 3/1N7 #G ,02N7 #H - - G $V , 9 9g&x0<'0:*73G4D3Ew'0 G $V ,0J J J/1N7 #G , ,>n]dNeflNm>>>   *~g666w' 48JJJ3EnU\F]^iFj3kN7 #K 0!*4:n+E+EFF ' = =K!ooi66Gd++Ag&v.);;G;!-+i2HL^2^2^);K /#BBB$2?$CO$TC *=)F)O)O)Q)QF% &01H1PQ -_:n__X^__%%mFG%TTT  %%)Z 0@0@0B0B%C%C ">11)*:; v vC+,<=cB8LF)5![![![![TZ![![![FLN#34S9(C!'#,#8#8#?#?#+", s##, DvCuCuCuntCuCuCuN#34S9%@@!/ F F  N22-m???( 2 2Et[[[))5 'e 4 4 %1mGY6Y6Y$6}$EM%,T;u;;%?%?"0*== 2$.%$6$B1818#5#@ $ %%%F5MM%2F5M$($B!  ) % , ,-? @ @ @ ( ) )A - -2KF. /t~CCyCFCCCrN)NNFFFFT)F)NF)FN)NN)NNN)>__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesr+r__annotations__rrpropertyboolrrrrrrZrrrxrr1rrrcr DecoderFastr EncodingFastrrrrrr rrrrrrrr rrrrrrrrrrrrr"r2PathLikerDrw __classcell__)rs@rLr*r*Qs  *04-444m(m(m(m(m(^XXGCGGGXG A4S>AAAA tCH~   X nd38nnnnXn:d3 ?&;:::X:nc3hnnnnFFFFF =X ''''X'1504*/+0',#-(-(-( (~-( (~ -( $( -( %) -(!%-(-(-( tCH~tL11 2-(-(-(-(^UE#tCy.,AUeCQUVYQZNF[UUUU& 7#7(3-777766d5j+A&B6]`6666 ??d?s????,GLd3i(?C sDI~ 4uuSu uRVumqrumvuuuuI9)I90I9 I9  I9 %SM I9tnI9I9I9I9`$(,;,F2D2T$($),0'+(,0404*/+0',#%*+Y`Y`"' OT-0$7H2I4PeKf f# Y` ! Y` * Y`0Y`SMY`Y`"Y`%SMY`tnY`! Y` (~Y` (~Y` $(!Y`"%)#Y`$!%%Y`&'Y`()Y`*#+Y`, -Y`Y`Y`Y`|DH#',;,F2D2T$($),0'+)-0404*/+0',#%*);;I001;E)->">?@;! ; * ; 0 ;SM;;";%SM;tn;!; (~; (~;$(; %)!;"!%#;$%;&';(#);, -;;;;z=tCy=S==== %*-1 d3i("'+  8)-)- //c2;.//#J/ ~ / "# / s ////j sDsDsDsDsDsDsDsDrNr*)>r{rar7r2 collectionsrtypingrrrrrr tokenizers.pre_tokenizerspre_tokenizersrf tokenizersr rr rctokenizers.decodersr rtokenizers.trainersr rrrrintegrations.ggmlrmodeling_gguf_pytorch_utilsrtokenization_utilsrtokenization_utils_baserrrrrrrrrrutilsr r!r" get_loggerrxloggerr;SPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILEr5rhr|r*rErNrLrs  ######::::::::::::::::777777//////111111666666^^^^^^^^^^^^::::::555555======333333                        @?????????  H % %"3/'( !! (6EXYY,--m Dm Dm Dm Dm D5m Dm D.-m Dm Dm DrN