g#3 d Z ddlZddlZddlmZ ddlmZmZmZ ddl m
Z
ddl m Z e j
e Zdd d
Zd Z G d d
e
ZdS )z Tokenization classes for PhoBERT N)copyfile)ListOptionalTuple )PreTrainedTokenizer)loggingz vocab.txtz bpe.codes)
vocab_file merges_filec t }| d }| dd D ]}| ||f |} t | }|S )z
Return set of symbol pairs in a word.
Word is represented as tuple of symbols (symbols being variable-length strings).
r N)setadd)wordpairs prev_charchars l/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/phobert/tokenization_phobert.py get_pairsr # s[
EEEQIQRR
9d#$$$ JJE L c
e Zd ZdZeZ d fd Z d d ee d
e ee d ee fd Z
d d ee d
e ee de d ee f fd
Z d d ee d
e ee d ee fdZ
ed Zd Zd Zd Zd Zd Zd Zd dede e d ee fdZd Z xZS ) PhobertTokenizeraO
Construct a PhoBERT tokenizer. Based on Byte-Pair-Encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
bos_token (`st`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) cls_token_id sep_token_id)rF rK rL clsseps r build_inputs_with_special_tokensz1PhobertTokenizer.build_inputs_with_special_tokens s[ ( %&48I7JJ J ! ![ 3&,{:S@@r Falready_has_special_tokensc |r$t ||d S |dgdgt | z z dgz S dgdgt | z z ddgz dgt | z z dgz S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rK rL rT Nr
r )rD get_special_tokens_maskrA )rF rK rL rT rJ s r rV z(PhobertTokenizer.get_special_tokens_mask s & & 7722 '[]a 3
31#K 0 001QC7 7sqcC , ,,-A61#K@P@P:PQUVTWWWr c | j g}| j g}|t ||z |z dgz S t ||z |z |z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. PhoBERT does not
make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )rP rO rA )rF rK rL rR rQ s r $create_token_type_ids_from_sequencesz5PhobertTokenizer.create_token_type_ids_from_sequences sm " ! ! s[(3.//1#5 53$s*S0;>DEEKKr c * t | j S N)rA r7 rF s r
vocab_sizezPhobertTokenizer.vocab_size s 4< r c 0 t | j fi | j S rZ )r> r7 added_tokens_encoderr[ s r get_vocabzPhobertTokenizer.get_vocab s DL>>D$=>>>r c | j v r
j | S t | }t t |d d |d dz gz }t | }|s|S t | fd }| j vrn8|\ }}g }d}|t
| k r | || } | ||| | }n-# t $ r | ||d Y nw xY w|| |k rC|t
| dz
k r-||dz |k r |
||z |dz
}n |
|| |dz
}|t
| k t | }|}t
| dk rnt | }Wd | }|d d
}| j |< |S ) Nr* zTc T j | t d S )Ninf)rB getfloat)pairrF s r z&PhobertTokenizer.bpe.. s 1C1CD%PU,,1W1W r )keyr r
r @@ ) rC r, listr minrB rA indexextend
ValueErrorappendjoin)
rF tokenr r bigramfirstsecondnew_wordijs
` r bpezPhobertTokenizer.bpe s# DJ :e$ $U||T$ss)__R6(9'::;;$ L ($W$W$W$WXXXFT^++ "ME6HAc$ii--
5!,,A
OOD1I...AA " OODH---E 7e##CIIM(9(9d1q5kV>S>S OOEFN333FAA OODG,,,FA c$ii-- XHD4yyA~~!$9 (: zz$ CRCy
5 s (C 'DDc g }t j d| }|D ]J}| t | | d K|S )zTokenize a string.z\S+\n? )refindallrm rj rx r- )rF text split_tokenswordsrq s r _tokenizezPhobertTokenizer._tokenize sf
9d++ B BE TXXe__%:%:3%?%? @ @ A A A Ar c r | j || j | j S )z0Converts a token (str) in an id using the vocab.)r7 rc r2 )rF rq s r _convert_token_to_idz%PhobertTokenizer._convert_token_to_id s, | t|'7'7'G'GHHHr c B | j || j S )z=Converts an index (integer) in a token (str) using the vocab.)r; rc r2 )rF rl s r _convert_id_to_tokenz%PhobertTokenizer._convert_id_to_token! s | t~666r c | d | dd }|S )z:Converts a sequence of tokens (string) in a single string.rz rh )rp replacestrip)rF tokens
out_strings r convert_tokens_to_stringz)PhobertTokenizer.convert_tokens_to_string% s5 XXf%%--eR88>>@@
r save_directoryfilename_prefixc t j | s t d| d d S t j ||r|dz ndt d z }t j ||r|dz ndt d z }t j | j t j | k r:t j | j rt | j | nzt j | j sVt |d 5 }| j
}| | d d d n # 1 swxY w Y t j | j t j | k rt | j | ||fS )NzVocabulary path (z) should be a directory-r r
r wb)ospathisdirloggererrorrp VOCAB_FILES_NAMESabspathr
isfiler r< sp_modelserialized_model_protowriter )rF r r out_vocab_fileout_merge_fileficontent_spiece_models r save_vocabularyz PhobertTokenizer.save_vocabulary* s w}}^,, LLT^TTT U U U F o M_s 2 22QbcoQp p
o M_s 2 22QbcpQq q
7??4? + +rw~/N/N N NSUSZSaSabfbqSrSr N T_n 5 5 5 5 00 /nd++
/r'+}'K'K'M'M$-...
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/ 7??4+ , ,0O0O O O T%~ 6 6 6 ~--s /FF F c t |t rs t |dd 5 }| | ddd n # 1 swxY w Y n0# t $ r}|d}~wt
$ r t
d| d w xY wdS | }|D ]f}| }| d }|dk rt d |d| }t | j | j |< gdS )
zi
Loads a pre-existing dictionary from a text file and adds its symbols to this instance.
rr& r' Nz Incorrect encoding detected in z , please rebuild the datasetrz r* z5Incorrect dictionary format, expected ' ')
isinstancer8 r< r9 FileNotFoundError UnicodeError Exception readlinesr rfindrn rA r7 ) rF ffdfnfelineslineTmplineidxr s r r9 z PhobertTokenizer.add_from_fileG sq a
c!S7333 +r&&r***+ + + + + + + + + + + + + + +$
c
c
c a! a a abbb
c F
3 3G==??D**S//Cbyy !XYYY:D!$T\!2!2DL
3 3s9 A A A AA AA
B!A##!B)r r r r r r r rZ )NF)__name__
__module__ __qualname____doc__r vocab_files_namesrE r intr rS boolrV rX propertyr\ r_ rx r r r r r8 r r r9
__classcell__)rJ s @r r r 3 s1 . .` * *
*
*
*
*
*
Z JNA A 9A3;DI3FA
cA A A A6 sxX X 9X3;DI3FXkoX
cX X X X X X: JNL L 9L3;DI3FL
cL L L L0 ! ! X!? ? ?* * *X I I I7 7 7
. .c .HSM .]bcf]g . . . .:3 3 3 3 3 3 3r r )r r r{ shutilr typingr r r tokenization_utilsr utilsr
get_loggerr r r r r r! r r r s ' & ( ( ( ( ( ( ( ( ( ( 5 5 5 5 5 5
H % %
i3 i3 i3 i3 i3* i3 i3 i3 i3 i3r