g%dZddlZddlZddlmZmZmZddlmZm Z erddl Z ddl m Z ddl mZejeZdd iZd ZGd d ZGd de ZdS)z Tokenization classes for CPMAnt.N)ListOptionalTuple)is_jieba_availablerequires_backends)PreTrainedTokenizer)logging vocab_filez vocab.txtctj}t|dd5}|}dddn #1swxYwYt |D]\}}|d}|||< |S)z*Loads a vocabulary file into a dictionary.rutf-8encodingN ) collections OrderedDictopen readlines enumeraterstrip)r vocabreadertokensindextokens j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/cpmant/tokenization_cpmant.py load_vocabr$s  # % %E j# 0 0 0$F!!##$$$$$$$$$$$$$$$!&)) u T""e LsAA A ceZdZddZdZdS)WordpieceTokenizerc0||_||_||_dSN)r unk_tokenmax_input_chars_per_word)selfrr%r&s r__init__zWordpieceTokenizer.__init__0s "(@%%%ct|}t||jkr|jgSd}g}|t|krt|}d}||kr4d|||}||jvr|}n |dz}||k4| ||j|dz }n|||}|t|k|S)Nr)listlenr&r%joinrappend)r'rcharsstart sub_tokensend cur_substrsubstrs rtokenizezWordpieceTokenizer.tokenize5sU  u::5 5 5N# # c%jj  e**CJ#++uSy!122TZ''!'Jq #++ !!!$.111 !!*---c%jj  r)N)r!r")__name__ __module__ __qualname__r(r7r)rr r /s;AAAA r)r c eZdZdZeZddgZdZ d%fd Ze dZ e dZ e dZ e de fdZdZdZfdZdZdeedefdZdZdZd&dedeedeefdZd&d ee d!ee dee fd"Z d'd ee d!eee d#edee ffd$ ZxZS)(CpmAntTokenizera Construct a CPMAnt tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file (`str`): Path to the vocabulary file. bod_token (`str`, *optional*, defaults to `""`): The beginning of document token. eod_token (`str`, *optional*, defaults to `""`): The end of document token. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding. unk_token (`str`, *optional*, defaults to `""`): The unknown token. line_token (`str`, *optional*, defaults to `""`): The line token. space_token (`str`, *optional*, defaults to `""`): The space token. input_idsattention_maskFr!leftc .t|dg||_||_t||_|j| |jd<|j||jd<|j| =|j|=t jt|jd|_d|jD|_ t|j||_ tj d |||||||| | d | dS) Njieba rc|dSNr,r;xs rz*CpmAntTokenizer.__init__.. Z[\]Z^r)keyci|]\}}|| Sr;r;).0kvs r z,CpmAntTokenizer.__init__..s>>>A1>>>r))rr%) bod_token eod_token bos_token eos_token pad_tokenr% line_token space_token padding_sider;)rrXrYrencoderrrsorteditemsdecoderr wordpiece_tokenizersuperr() r'r rXrYrZr[r\r%r]r^r_kwargs __class__s rr(zCpmAntTokenizer.__init__ls. $ ***""!*--  L5 S!\*5 T L % L $".vdl6H6H6J6JP^P^/_/_/_`` >>););)=)=>>> #5DLT]#^#^#^  !#%  r)c&|j|jSr$)r`rXr's r bod_token_idzCpmAntTokenizer.bod_token_id|DN++r)c&|j|jSr$)r`rYris r eod_token_idzCpmAntTokenizer.eod_token_idrkr)c|jdS)Nrr`ris r newline_idzCpmAntTokenizer.newline_ids|D!!r)returnc*t|jSr$)r.r`ris r vocab_sizezCpmAntTokenizer.vocab_sizes4<   r)c0t|jfi|jSr$)dictr`added_tokens_encoderris r get_vocabzCpmAntTokenizer.get_vocabsDL>>D$=>>>r)cg}tj|dD]/}||j|0|S)zTokenize a string.F)cut_all)rIcutextendrdr7)r'text output_tokensrNs r _tokenizezCpmAntTokenizer._tokenizesU 4/// G GA  !9!B!B1!E!E F F F Fr)c nd|D}fd|D}tj|fi|S)zDecode ids into a string.cg|] }|dk| S)rr;)rTis r z+CpmAntTokenizer._decode..s4441Q!VVQVVVr)cVg|]%}|jk |jk|jk#|&Sr;) pad_token_id eos_token_id bos_token_id)rTrNr's rrz+CpmAntTokenizer._decode..sI   A):$:$:qDDU?U?UZ[_c_pZpZpAZpZpZpr))re_decode)r' token_idsrfrgs` rrzCpmAntTokenizer._decodes_44 444         uwwy33F333r)c||jvSr$ror'rs rcheckzCpmAntTokenizer.checks $$r)rc,d|S)Nr+)r/)r'rs rconvert_tokens_to_stringz(CpmAntTokenizer.convert_tokens_to_stringswwvr)cr|j||j|jS)z0Converts a token (str) in an id using the vocab.)r`getr%rs r_convert_token_to_idz$CpmAntTokenizer._convert_token_to_ids,|t|'7'7'G'GHHHr)cB|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rcrr%)r'rs r_convert_id_to_tokenz$CpmAntTokenizer._convert_id_to_tokens|t~666r)Nsave_directoryfilename_prefixctj|r6tj||r|dzndtdz}n |r|dznd|z}d}d|jvr|jd|jd<|jd=d|jvr|jd|jd<|jd=t jt|j d |_t|d d 5}|j D]H\}}||kr t d|d|}| |dz|dz }I dddn #1swxYwY|fS)N-r+r rrJrFrrEc|dSrLr;rMs rrOz1CpmAntTokenizer.save_vocabulary..rPr)rQwrrzSaving vocabulary to z\: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!r,)ospathisdirr/VOCAB_FILES_NAMESr`rrrarbrloggerwarningwrite)r'rrr rwriterr token_indexs rsave_vocabularyzCpmAntTokenizer.save_vocabularys 7== ( ( ]/!Q3!6!6rUfgsUt tJJ4CJ/C//n\J $,  #'<#4DL  S! 4<  #'<#5DL  T"".vdl6H6H6J6JP^P^/_/_/_`` *cG 4 4 4 &*l&8&8&:&:  "{K''NNN NNN(E UT\***                 }sA#E22E69E6 token_ids_0 token_ids_1cJ| |jg|zS|jg|z|jgz|zS)a1 Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CPMAnt sequence has the following format: - single sequence: `[BOS] Sequence`. Args: token_ids_0 (`List[int]`): The first tokenized sequence that special tokens will be added. token_ids_1 (`List[int]`): The optional second tokenized sequence that special tokens will be added. Returns: `List[int]`: The model input with special tokens. )r)r'rrs r build_inputs_with_special_tokensz0CpmAntTokenizer.build_inputs_with_special_tokenss<  %&4 4!"[0D4E3FFTTr)already_has_special_tokensc|r$t||dS|/dgdgt|zzdgzdgt|zzSdgdgt|zzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNr,r)reget_special_tokens_maskr.)r'rrrrgs rrz'CpmAntTokenizer.get_special_tokens_masks" & 7722'[]a3   "31#K 0 001QC7A3[AQAQ;QR RsqcC ,,,--r)) r@rArBrCrDr!rErFrGr$)NF)r8r9r:__doc__rvocab_files_namesmodel_input_namesadd_prefix_spacer(propertyrjrmrpintrsrwr~rrrstrrrrrrrrboolr __classcell__)rgs@rr=r=Os`0*$&67 ( ( ( ( ( ( T,,X,,,X,""X"!C!!!X!???44444%%%tCySIII777cHSM]bcf]g6UUDIUTXY\T]UimnqirUUUU&sx..9.3;DI3F.ko. c..........r)r=)rrrtypingrrrtransformers.utilsrrrItokenization_utilsr utilsr get_loggerr8rrrr r=r;r)rrs.'& ((((((((((DDDDDDDDLLL555555  H % %!;/@|.|.|.|.|.)|.|.|.|.|.r)