g!KdZddlZddlZddlZddlZddlmZmZmZm Z ddl m Z ddl m Z e jeZddd d Zd Zd Zd ZGdde ZdS)zTokenization classes for FSMT.N)DictListOptionalTuple)PreTrainedTokenizer)loggingzvocab-src.jsonzvocab-tgt.jsonz merges.txt)src_vocab_filetgt_vocab_file merges_filec~t}|d}|ddD]}|||f|}|S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairs prev_charchars f/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/fsmt/tokenization_fsmt.py get_pairsr$sP EEEQIQRR 9d#$$$ Lc6|dd}tjdd|}|dd}|dd}|dd}|d d }|d d }|d d }|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|dd}|d d!}|d"d#}|d$d%}|d&d'}|d(d)}|d*d+}|d,d-}tjd.d|}|d/d0}|d1d2}|d3d4}|d5d6}|d7d8}|d9d:}|d;d<}|d=d>}|d?d@}|S)Azz Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/replace-unicode-punctuation.perl u，,u。\s*z. u、u”"u“u∶:u：u？?u《u》u）)u！!u（(u；;u１1u」u「u０0u３3u２2u５5u６6u９9u７7u８8u４4u．\s*u～~u’'u…z...u━-u〈u【[u】]u％%)replaceresub)texts rreplace_unicode_punctr71s <<s # #D 6)T4 ( (D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D 6)T4 ( (D <<s # #D <<s # #D <<u % %D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D <<s # #D Krcg}|D]A}tj|}|dr,||Bd|S)zw Port of https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/remove-non-printing-char.perl C) unicodedatacategory startswithappendjoin)r6outputrcats rremove_non_printing_charrB\sdF"4(( >>#     d 776??rc eZdZdZeZddgZ d'fd Zd ee e ffd Z e d e fd Z dZdZdZdZe dZe dZdZdZdZd(dZdZdZdZ d)dee deee d ee fdZ d*dee deee ded ee ffd Z d)dee deee d ee fd!Z d)d"e d#ee d e!e fd$Z"d%Z#d&Z$xZ%S)+ FSMTTokenizera Construct an FAIRSEQ Transformer tokenizer. Based on Byte-Pair Encoding. The tokenization process is the following: - Moses preprocessing and tokenization. - Normalizing all inputs text. - The arguments `special_tokens` and the function `set_special_tokens`, can be used to add additional symbols (like "__classify__") to a vocabulary. - The argument `langs` defines a pair of languages. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: langs (`List[str]`, *optional*): A list of two languages to translate from and to, for instance `["en", "ru"]`. src_vocab_file (`str`, *optional*): File containing the vocabulary for the source language. tgt_vocab_file (`st`, *optional*): File containing the vocabulary for the target language. merges_file (`str`, *optional*): File containing the merges. do_lower_case (`bool`, *optional*, defaults to `False`): Whether or not to lowercase the input when tokenizing. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskNFc  ddl} n#t$rtdwxYw| |_||_||_||_||_i|_i|_i|_ |r#t|dkr|\|_ |_ ntd|dt|d5} tj| |_dddn #1swxYwYt|d5} tj| }d|D|_dddn #1swxYwYt|d5}|d dd }dddn #1swxYwYd |D}t-t/|t1t||_i|_t7jd ||||||||| d | dS)NrnYou need to install sacremoses to use XLMTokenizer. See https://pypi.org/project/sacremoses/ for installation.zFarg `langs` needs to be a list of 2 langs, e.g. ['en', 'ru'], but got zw. Usually that means that tokenizer can't find a mapping for the given model path in and other maps of this tokenizer.utf-8encodingci|]\}}|| SrR.0kvs r z*FSMTTokenizer.__init__..s???TQAq???r c`g|]+}t|dd,S)NrM)tuplesplit)rTmerges r z*FSMTTokenizer.__init__..s1???u% bqb)**???r) langsr r r do_lower_case unk_token bos_token sep_token pad_tokenrR) sacremoses ImportErrorsmr r r r`cache_moses_punct_normalizercache_moses_tokenizercache_moses_detokenizerlensrc_langtgt_lang ValueErroropenjsonloadencoderitemsdecoderreadr\dictziprange bpe_rankscachesuper__init__)selfr_r r r r`rarbrcrdkwargsresrc_vocab_handletgt_vocab_handle tgt_vocab merges_handlemerges __class__s rr|zFSMTTokenizer.__init__s         M   ,,&*-/)%'"')$  SZZ1__+0 (DM4==8Y^888  .7 3 3 3 77G9%566DL 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 .7 3 3 3 @7G "233I??Y__->->???DL @ @ @ @ @ @ @ @ @ @ @ @ @ @ @+ 0 0 0 ;M"''))//55crc:F ; ; ; ; ; ; ; ; ; ; ; ; ; ; ;?????c&%F *<*<==>>  ))#'  s>"&C  CC(8D,,D03D00FF Freturnc*|SN) get_src_vocabr}s r get_vocabzFSMTTokenizer.get_vocabs!!###rc|jSr)src_vocab_sizers r vocab_sizezFSMTTokenizer.vocab_sizes ""rc||jvr%|j|}||j|<|j||SNlang)rhrgMosesPunctNormalizer normalize)r}r6rpunct_normalizers rmoses_punct_normzFSMTTokenizer.moses_punct_normsR t8 8 8#w;;;FF 6FD -d 306@@FFFrc||jvr%|j|}||j|<|j||dddS)NrTF)aggressive_dash_splits return_strescape)rirgMosesTokenizertokenize)r}r6rmoses_tokenizers rmoses_tokenizezFSMTTokenizer.moses_tokenizesc t1 1 1"g44$4??O/>D &t ,)$/88 %9   rc||jvr%|j|}||j|<|j||Sr)rjrgMosesDetokenizer detokenize)r}tokensrmoses_detokenizers rmoses_detokenizezFSMTTokenizer.moses_detokenizesR t3 3 3 $ 8 8d 8 C C 1BD ( .+D1<>D$=>>>rc0t|jfi|jSr)rvrtadded_tokens_decoderrs r get_tgt_vocabzFSMTTokenizer.get_tgt_vocabrrct|dd|ddzfz}|jvr j|St|}|s|dzS t|fd}|jvrn8|\}}g}d}|t |kr |||} |||| | }n-#t$r |||dYnwxYw|||krC|t |dz kr-||dz|kr| ||z|dz }n | |||dz }|t |kt|}|}t |dkrnt|}Wd |}|d krd }|j|<|S) NrYTcTj|tdS)Ninf)rygetfloat)pairr}s rz#FSMTTokenizer.bpe.. s 1C1CD%PU,,1W1WrkeyrrrM z z ) r[rzrminryrkindexextendrnr>r?) r}tokenrrbigramfirstsecondnew_wordijs ` rbpezFSMTTokenizer.bpesU3B3Z  E"I$6#88 DJ  :e$ $$ "6> ! ($W$W$W$WXXXFT^++"ME6HAc$ii-- 5!,,A OOD1I...AA "OODH---E7e##CIIM(9(9d1q5kV>S>SOOEFN333FAAOODG,,,FAc$ii-- XHD4yyA~~!$9 (:xx~~ :  D  5 sC'C/.C/encv|j}|jr|}|r|}n.|||}|||}g}|D]L}|rH|t||dM|S)av Tokenize a string given language code using Moses. Details of tokenization: - [sacremoses](https://github.com/alvations/sacremoses): port of Moses - Install with `pip install sacremoses` Args: - lang: ISO language code (default = 'en') (string). Languages should belong of the model supported languages. However, we don't enforce it. - bypass_tokenizer: Allow users to preprocess and tokenize the sentences externally (default = False) (bool). If True, we only apply BPE. Returns: List of tokens. rr) rlr`lowerr\rrrlistr)r}r6rbypass_tokenizer split_tokensrs r _tokenizezFSMTTokenizer._tokenizeBs*}   ::<)>s)C)C$D$DEEErcr|j||j|jS)z0Converts a token (str) in an id using the vocab.)rrrra)r}rs r_convert_token_to_idz"FSMTTokenizer._convert_token_to_idis,|t|'7'7'G'GHHHrcB|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rtrra)r}rs r_convert_id_to_tokenz"FSMTTokenizer._convert_id_to_tokenms|t~666rcd|D}d|}|||j}|S)z:Converts a sequence of tokens (string) in a single string.cbg|],}|dddd-S)rr:r)r3)rTts rr^z:FSMTTokenizer.convert_tokens_to_string..us6JJJa!))C$$,,VS99JJJrr:)r?r\rrm)r}rr6s rconvert_tokens_to_stringz&FSMTTokenizer.convert_tokens_to_stringqsOKJ6JJJ&&(($$VT];; r token_ids_0 token_ids_1c6|jg}|||zS||z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A FAIRSEQ Transformer sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) sep_token_idr}rrseps r build_inputs_with_special_tokensz.FSMTTokenizer.build_inputs_with_special_tokens{s7& !  $ $S ;.44ralready_has_special_tokensc|r$t||dS|/dgt|zdgzdgt|zzdgzSdgt|zdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNrr)r{get_special_tokens_maskrk)r}rrrrs rrz%FSMTTokenizer.get_special_tokens_masks& & 7722'[]a3   "C#k***qc1aS3{;K;K5KLPQsR Rc+&&&1#--rc|jg}|t||zdgzSt||zdgzt||zdgzzS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. A FAIRSEQ Transformer sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An FAIRSEQ_TRANSFORMER sequence pair mask has the following format: Nrr)rrkrs r$create_token_type_ids_from_sequencesz2FSMTTokenizer.create_token_type_ids_from_sequencessd4 !  {S())QC/ /;$%%+c+2C.D.Ds.JJJrsave_directoryfilename_prefixc tj|s td|ddStj||r|dzndt dz}tj||r|dzndt dz}tj||r|dzndt dz}t|dd 5}|tj |j d d d dzdddn #1swxYwYt|dd 5}d|j D}|tj |d d d dzdddn #1swxYwYd}t|dd 5} t|j dD][\} } || kr td|d| }| d| dz|dz }\ dddn #1swxYwY|||fS)NzVocabulary path (z) should be a directoryr-r:r r r wrNrOrMTF)indent sort_keys ensure_asciirXci|]\}}|| SrRrRrSs rrWz1FSMTTokenizer.save_vocabulary..s???$!QA???rrc|dS)NrrR)kvs rrz/FSMTTokenizer.save_vocabulary..s Y[\]Y^rrzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rr)ospathisdirloggererrorr?VOCAB_FILES_NAMESrowriterpdumpsrrrtrssortedrywarning) r}rrr r r frrwriter bpe_tokens token_indexs rsave_vocabularyzFSMTTokenizer.save_vocabularyspw}}^,,  LLT^TTT U U U F oM_s222QbcsQtt   oM_s222QbcsQtt  gll oM_s222QbcpQqq  .# 8 8 8 cA GGDJt|ATYZZZ]aa b b b c c c c c c c c c c c c c c c.# 8 8 8 `A??$,*<*<*>*>???I GGDJydQVWWWZ^^ _ _ _ ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` +sW 5 5 5 +1$.2F2F2H2HN^N^+_+_+_  ' KK''NNM MMM(E SXXj11D8999                 ~{::s814D11D58D5AF,,F03F0 BII"%I"cB|j}d|d<|S)Nrg)__dict__copy)r}states r __getstate__zFSMTTokenizer.__getstate__s# ""$$d  rch||_ ddl}n#t$rtdwxYw||_dS)NrrL)rrerfrg)r}dres r __setstate__zFSMTTokenizer.__setstate__s]          M   s() NNNNFrGrHrIrJ)rFr)NF)&__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr|rstrintrpropertyrrrrrrrrrrrrrrrrrboolrrrrrr __classcell__)rs@rrDrDrs++Z*$&67@ @ @ @ @ @ F$4S>$$$$#C###X#GGG    EEE  !!X!!!X!??????***X%%%%NIII777JN55953;DI3F5 c55556sx..9.3;DI3F.ko. c......:JNKK9K3;DI3FK cKKKKB";";c";HSM";]bcf]g";";";";H       rrD)rrprr4r;typingrrrrtokenization_utilsrutilsr get_loggerrrrrr7rBrDrRrrrs%$ ............555555  H % %'&   (((V   ,TTTTT'TTTTTr