g3dZddlZddlZddlmZmZmZddlmZddl m Z e j e Z ddd Zd ZGd d eZdS) z Tokenization classes for BioGPT.N)ListOptionalTuple)PreTrainedTokenizer)loggingz vocab.jsonz merges.txt) vocab_file merges_filec~t}|d}|ddD]}|||f|}|S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairs prev_charchars j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/biogpt/tokenization_biogpt.py get_pairsr!sP EEEQIQRR 9d#$$$ Lc eZdZdZeZddgZ d fd Zed Z d Z d Z d Z d Z d!dZdZdZdZ d"deedeeedeefdZ d#deedeeededeeffd Z d"deedeeedeefdZd"dedeedeefdZdZdZxZS)$BioGptTokenizera: Construct an FAIRSEQ Transformer tokenizer. Moses tokenization followed by Byte-Pair Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc  ddl} n#t$rtdwxYwd|_| |_i|_i|_ t |d5} tj| |_ dddn #1swxYwYd|j D|_ t |d5} | ddd} dddn #1swxYwYd | D} tt| t!t#| |_i|_t)jd |||||d |dS) NrzqYou need to install sacremoses to use BioGptTokenizer. See https://pypi.org/project/sacremoses/ for installation.enutf-8encodingci|]\}}|| Sr$).0kvs r z,BioGptTokenizer.__init__..xs>>>A1>>>r c`g|]+}t|dd,S)N)tuplesplit)r%merges r z,BioGptTokenizer.__init__..{s1???u% bqb)**???r) bos_token eos_token sep_token unk_token pad_tokenr$) sacremoses ImportErrorlangsmcache_moses_tokenizercache_moses_detokenizeropenjsonloadencoderitemsdecoderreadr.dictziprangelen bpe_rankscachesuper__init__)selfr r r4r1r2r3r5kwargsr6 vocab_handle merges_handlemerges __class__s rrJzBioGptTokenizer.__init__\s         M    %'"')$ *w / / / 3<9\22DL 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3>>););)=)=>>> + 0 0 0 ;M"''))//55crc:F ; ; ; ; ; ; ; ; ; ; ; ; ; ; ;?????c&%F *<*<==>>           s,"A99A=A==0C99C=C=c*t|jS)zReturns vocab size)rFr?rKs r vocab_sizezBioGptTokenizer.vocab_sizes4<   rc0t|jfi|jSN)rCr?added_tokens_encoderrRs r get_vocabzBioGptTokenizer.get_vocabsDL>>D$=>>>rc||jvr%|j|}||j|<|j||dddS)Nr8TF)aggressive_dash_splits return_strescape)r:r9MosesTokenizertokenize)rKtextr8moses_tokenizers rmoses_tokenizezBioGptTokenizer.moses_tokenizesc t1 1 1"g44$4??O/>D &t ,)$/88 %9   rc||jvr%|j|}||j|<|j||S)NrY)r;r9MosesDetokenizer detokenize)rKtokensr8moses_detokenizers rmoses_detokenizez BioGptTokenizer.moses_detokenizesR t3 3 3 $ 8 8d 8 C C 1BD ( .+D1<TcTj|tdS)Ninf)rGgetfloat)pairrKs rz%BioGptTokenizer.bpe..s 1C1CD%PU,,1W1Wrkeyrr r, z z ) r-rHrminrGrFindexextend ValueErrorappendjoin) rKtokenrrbigramfirstsecondnew_wordijs ` rbpezBioGptTokenizer.bpesU3B3Z  E"I$6#88 DJ  :e$ $$ "6> ! ($W$W$W$WXXXFT^++"ME6HAc$ii-- 5!,,A OOD1I...AA "OODH---E7e##CIIM(9(9d1q5kV>S>SOOEFN333FAAOODG,,,FAc$ii-- XHD4yyA~~!$9 (:xx~~ :  D  5 sC'C/.C/Fc |r|}n|||j}g}|D]L}|rH|t ||dM|S)zReturns a tokenized string.rr)r.rar8rulistr)rKr_bypass_tokenizer split_tokensrys r _tokenizezBioGptTokenizer._tokenizes  8::<)>s)C)C$D$DEEErcr|j||j|jS)z0Converts a token (str) in an id using the vocab.)r?rlr4)rKrys r_convert_token_to_idz$BioGptTokenizer._convert_token_to_ids,|t|'7'7'G'GHHHrcB|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rArlr4)rKrts r_convert_id_to_tokenz$BioGptTokenizer._convert_id_to_tokens|t~666rcd|D}d|}|||j}|S)z:Converts a sequence of tokens (string) in a single string.cbg|],}|dddd-S)rrri)replace)r%ts rr0z.s6JJJa!))C$$,,VS99JJJrr)rxr.rgr8)rKrer_s rconvert_tokens_to_stringz(BioGptTokenizer.convert_tokens_to_stringsOKJ6JJJ&&(($$VTY77 rN token_ids_0 token_ids_1returncB| |jg|zS|jg}||z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BioGPT sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) sep_token_idrKrrseps r build_inputs_with_special_tokensz0BioGptTokenizer.build_inputs_with_special_tokenss;&  %&4 4 ![ 3&44ralready_has_special_tokensc|r$t||dS|/dgdgt|zzdgzdgt|zzSdgdgt|zzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNr r)rIget_special_tokens_maskrF)rKrrrrPs rrz'BioGptTokenizer.get_special_tokens_masks$ & 7722'[]a3   "31#K 0 001QC7A3[AQAQ;QR RsqcC ,,,--rc|jg}|t||zdgzSt||zdgzt||zdgzzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. A FAIRSEQ Transformer sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). Nrr )rrFrs r$create_token_type_ids_from_sequencesz4BioGptTokenizer.create_token_type_ids_from_sequencessd. !  {S())QC/ /;$%%+c+2C.D.Ds.JJJrsave_directoryfilename_prefixc ztj|s td|ddStj||r|dzndt dz}tj||r|dzndt dz}t|dd 5}|tj |j d d d dzdddn #1swxYwYd}t|dd 5}t|j dD][\}} || kr td|d| }|d|dz|dz }\ dddn #1swxYwY||fS)NzVocabulary path (z) should be a directory-rr r wr r!r,TF)indent sort_keys ensure_asciir)rc|dS)Nr r$)kvs rroz1BioGptTokenizer.save_vocabulary..Js Y[\]Y^rrpzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rrr )ospathisdirloggererrorrxVOCAB_FILES_NAMESr<writer=dumpsr?sortedrGr@warning) rKrrr merge_filefrtwriter bpe_tokens token_indexs rsave_vocabularyzBioGptTokenizer.save_vocabulary:sqw}}^,,  LLT^TTT U U U FW\\ oM_s222QbcoQpp  W\\ oM_s222QbcpQqq  *cG 4 4 4 c GGDJt|ATYZZZ]aa b b b c c c c c c c c c c c c c c c *cG 4 4 4 +1$.2F2F2H2HN^N^+_+_+_  ' KK''NNM MMM(E SXXj11D8999                 :%%s%<4C<<DDBF..F25F2cB|j}d|d<|S)Nr9)__dict__copy)rKstates r __getstate__zBioGptTokenizer.__getstate__Vs# ""$$d  rch||_ ddl}n#t$rtdwxYw||_dS)NrznYou need to install sacremoses to use XLMTokenizer. See https://pypi.org/project/sacremoses/ for installation.)rr6r7r9)rKdr6s r __setstate__zBioGptTokenizer.__setstate__[s]          M   s()rrrrr)FrU)NF)__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrJpropertyrSrWrargrrrrrrintrrboolrrstrrrrr __classcell__)rPs@rrr.sY((T*$&67 * * * * * * X!!X!???   EEE ***X    III777JN55953;DI3F5 c55552sx..9.3;DI3F.ko. c......8JNKK9K3;DI3FK cKKKK<&&c&HSM&]bcf]g&&&&8       rr)rr=rtypingrrrtokenization_utilsrutilsr get_loggerrrrrrr$rrrs'& ((((((((((555555  H % %   xxxxx)xxxxxr