g$dZddlZddlZddlZddlmZmZmZmZm Z m Z ddl m Z ddl mZmZmZerddlZerddlZejeZddiZd ZGd d e ZdS) zTokenization class for VITS.N)AnyDictListOptionalTupleUnion)PreTrainedTokenizer)is_phonemizer_availableis_uroman_availablelogging vocab_filez vocab.jsonc`tjd}||}|du}|S)Nz [^\x00-\x7F])recompilesearch) input_stringnon_roman_patternmatch has_non_romans f/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/vits/tokenization_vits.pyhas_non_roman_charactersr%s7 ?33  $ $\ 2 2E%M c 4eZdZdZeZddgZ d dfd Zed Z d Z d Z dZ dde dedeed ee ee efffdZde d ee fdZdee d e fdZdZdZdde dee d eee dffdZxZS) VitsTokenizera Construct a VITS tokenizer. Also supports MMS-TTS. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. language (`str`, *optional*): Language identifier. add_blank (`bool`, *optional*, defaults to `True`): Whether to insert token id 0 in between the other tokens. normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the input text by removing all casing and punctuation. phonemize (`bool`, *optional*, defaults to `True`): Whether to convert the input text into phonemes. is_uroman (`bool`, *optional*, defaults to `False`): Whether the `uroman` Romanizer needs to be applied to the input text prior to tokenizing. input_idsattention_maskNTFreturnc dt|d5} tj| |_dddn #1swxYwYd|jD|_||_||_||_||_ ||_ tj d|||||||d| dS)Nutf-8encodingci|]\}}|| Sr&).0kvs r z*VitsTokenizer.__init__..Ws>>>A1>>>r) pad_token unk_tokenlanguage add_blank normalize phonemize is_uromanr&) openjsonloadencoderitemsdecoderr-r.r/r0r1super__init__) selfrr+r,r-r.r/r0r1kwargs vocab_handle __class__s rr9zVitsTokenizer.__init__Hs*w / / / 3<9\22DL 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3?>););)=)=>>>   """"   s 9==c*t|jSN)lenr5)r:s r vocab_sizezVitsTokenizer.vocab_sizejs4<   rc|fdtjD}|j|S)Nc<i|]}||Sr&)convert_ids_to_tokens)r'ir:s rr*z+VitsTokenizer.get_vocab..os)RRRa++A..RRRr)rangerAupdateadded_tokens_encoder)r:vocabs` r get_vocabzVitsTokenizer.get_vocabns@RRRR5;Q;QRRR T./// rct|jt|jz}d}d}|t |krwd}|D];}|||t |z|kr||z }|t |z }d}n<|s"|||z }|dz }|t |kw|S)zfLowercase the input string, respecting any special token ids that may be part or entirely upper-cased.rFT)listr5keysrHr@lower)r:rall_vocabulary filtered_textrE found_matchwords rnormalize_textzVitsTokenizer.normalize_textssdl//1122T$:S:X:X:Z:Z5[5[[ #l####K&  AD M 12d::!T)MTNA"&KE ;  a!6!6!8!88 Q#l####rcH|jdkr|dd}|S)z4Special treatment of characters in certain languagesronuțuţ)r-replace)r:texts r_preprocess_charzVitsTokenizer._preprocess_chars' =E ! !<<d++D rrYis_split_into_wordsr/c ||nj}|r|}|}t|rXjrQt st dn(tj }| |}j rNtstdtj |ddddd}tjdd |}nG|rEd t%t'fd |}||fS) a Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. normalize (`bool`, *optional*, defaults to `None`): Whether or not to apply punctuation and casing normalization to the text inputs. Typically, VITS is trained on lower-cased and un-punctuated text. Hence, normalization is used to ensure that the input text consists only of lower-case characters. kwargs (`Dict[str, Any]`, *optional*): Keyword arguments to use for the tokenization. Returns: `Tuple[str, Dict[str, Any]]`: The prepared text and the unused kwargs. NaCText to the tokenizer contains non-Roman characters. To apply the `uroman` pre-processing step automatically, ensure the `uroman` Romanizer is installed with: `pip install uroman` Note `uroman` requires python version >= 3.10Otherwise, apply the Romanizer manually as per the instructions: https://github.com/isi-nlp/uromanzEPlease install the `phonemizer` Python package to use this tokenizer.zen-usespeakT)r-backendstrippreserve_punctuation with_stressz\s+ rLc|jvSr?)r5)charr:s rz8VitsTokenizer.prepare_for_tokenization..sTT\=Qr)r/rUrZrr1r loggerwarningurUromanromanize_stringr0r ImportError phonemizerrsubjoinrNfilterr_)r:rYr[r/r;rRuromans` rprepare_for_tokenizationz&VitsTokenizer.prepare_for_tokenizationse4"+!6IIDN  -&&t,,D--d33 #M 2 2 Ft~ F&(( Fy & 6 6} E E > l*,, k!"ijjj&0  %) MF63 >>MM  lGGD0Q0Q0Q0QS`)a)a$b$bcciikkMf$$rct|}|jr6|dgt|dzdzz}||ddd<|}|S)z]Tokenize a string by inserting the `` token at the boundary between adjacent characters.rrMN)rNr._convert_id_to_tokenr@)r:rYtokens intersperseds r _tokenizezVitsTokenizer._tokenizes[d > " 55a889S[[1_q=PQL!'LA !F rrucv|jrt|dkr |ddd}d|S)NrMrsrL)r.r@rn)r:rus rconvert_tokens_to_stringz&VitsTokenizer.convert_tokens_to_strings9 > "c&kkAooADqD\Fwwvrcr|j||j|jS)z0Converts a token (str) in an id using the vocab.)r5getr,)r:tokens r_convert_token_to_idz"VitsTokenizer._convert_token_to_ids,|t|'7'7'G'GHHHrc6|j|S)z=Converts an index (integer) in a token (str) using the vocab.)r7r{)r:indexs rrtz"VitsTokenizer._convert_id_to_tokens|&&&rsave_directoryfilename_prefixc tj|s td|ddStj||r|dzndt dz}t|dd5}|tj |j d d d d zdddn #1swxYwY|fS)NzVocabulary path (z) should be a directory-rLrwr"r#rsTF)indent sort_keys ensure_ascii ) ospathisdirrferrorrnVOCAB_FILES_NAMESr2writer3dumpsr5)r:rrrfs rsave_vocabularyzVitsTokenizer.save_vocabularys!w}}^,,  LLT^TTT U U U FW\\ oM_s222QbcoQpp  *cG 4 4 4 c GGDJt|ATYZZZ]aa b b b c c c c c c c c c c c c c c c}s4CC C )rrNTTTF)r N)FNr?)__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesr9propertyrArJrUrZstrboolrrrrrqrrwryr}rtrr __classcell__)r=s@rrr/s**$&67           D!!X! *Y]?%?%?%.2?%GOPT~?% sDcN" #?%?%?%?%B c d3i    tCyS III'''  c HSM ]bchilcmoscs]t        rr)rr3rrtypingrrrrrrtokenization_utilsr utilsr r r rlrprh get_loggerrrfrrrr&rrrs/#" ::::::::::::::::555555JJJJJJJJJJ  H % %!<0DDDDD'DDDDDr