g0 d Z ddlZddlmZ ddlmZmZmZmZm Z ddl
Z ddl m
Z
ddlmZ ej e ZdZd d
iZ G d d e
ZdS )
zTokenization classes for . N)copyfile)AnyDictListOptionalTuple )PreTrainedTokenizer)loggingu ▁
vocab_filezsentencepiece.bpe.modelc
e Zd ZdZeZddgZ d d eee e
f d
df fd
Z d Z d
Z
d dee deee d
ee fdZ d!dee deee ded
ee f fd
Z d dee deee d
ee fdZed Zd Zde d
ee fdZd Zd Zd Zd d e d ee d
ee fd Z xZ S )"
XGLMTokenizera
Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on
[SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
input_idsattention_mask`
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) sep_token_idr* rS rT seps r build_inputs_with_special_tokensz.XGLMTokenizer.build_inputs_with_special_tokens s@ ( %&4 4 ![ 3&,{::r! Falready_has_special_tokensc |r$t ||d S |dgdgt | z z S dgdgt | z z ddgz dgt | z z S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rS rT rZ Nr& r )rC get_special_tokens_maskr? )r* rS rT rZ rF s r r\ z%XGLMTokenizer.get_special_tokens_mask s & & 7722 '[]a 3
31#K 0 001 1sqcC , ,,-A61#K@P@P:PQQr! c | j g}|t ||z dgz S t ||z |z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. XLM-RoBERTa does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )rV r? rW s r $create_token_type_ids_from_sequencesz2XGLMTokenizer.create_token_type_ids_from_sequences sX $ ! s[())QC/ /3$s*S0;>??1#EEr! c J t | j | j z | j z S N)r? r; r) r6 )r* s r
vocab_sizezXGLMTokenizer.vocab_size s" 4=!!D$77$:OOOr! c | fdt j D }| j |S )Nc < i | ]} | |S r )convert_ids_to_tokens)r r r* s r r, z+XGLMTokenizer.get_vocab.. s) RRRa++A..RRRr! )r7 ra r@ added_tokens_encoder)r* vocabs ` r get_vocabzXGLMTokenizer.get_vocab s@ RRRR5 ;Q;QRRR
T./// r! textc D | j |t S )N)out_type)r; encoder= )r* rh s r _tokenizezXGLMTokenizer._tokenize s }##D3#777r! c || j v r
| j | S | j | }|r
|| j z n| j S )z0Converts a token (str) in an id using the vocab.)r> r; PieceToIdr) unk_token_id)r* tokenspm_ids r _convert_token_to_idz"XGLMTokenizer._convert_token_to_id sP D. . .-e4 4 ((// 06Lv+++4;LLr! c r || j v r
| j | S | j || j z
S )z=Converts an index (integer) in a token (str) using the vocab.)rB r; IdToPiecer) )r* indexs r _convert_id_to_tokenz"XGLMTokenizer._convert_id_to_token s< D. . .-e4 4}&&ut/B'BCCCr! c d | t d }|S )zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r* tokens
out_strings r convert_tokens_to_stringz&XGLMTokenizer.convert_tokens_to_string s4 WWV__,,-=sCCIIKK
r! save_directoryfilename_prefixc t j | s t d| d d S t j ||r|dz ndt d z }t j | j t j | k r:t j | j rt | j | nzt j | j sVt |d 5 }| j
}| | d d d n # 1 swxY w Y |fS )NzVocabulary path (z) should be a directory-rx r wb)ospathisdirloggererrorrz VOCAB_FILES_NAMESabspathr isfiler openr; rK write)r* r r out_vocab_fileficontent_spiece_models r save_vocabularyz XGLMTokenizer.save_vocabulary sw w}}^,, LLT^TTT U U U F o M_s 2 22QbcoQp p
7??4? + +rw~/N/N N NSUSZSaSabfbqSrSr N T_n 5 5 5 5 00 /nd++
/r'+}'K'K'M'M$-...
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/ s (/E##E'*E')r r r r r r Nr` )NF) __name__
__module__ __qualname____doc__r vocab_files_namesmodel_input_namesr r r= r rD rM rR r intrY boolr\ r^ propertyra rg rl rr rv r r r
__classcell__)rF s @r r r " sz = =~ *$&67
486
6
"$sCx.16
6
6
6
6
6
6
p C C C JN; ; 9;3;DI3F;
c; ; ; ;4 sxR R 9R3;DI3FRkoR
cR R R R R R: JNF F 9F3;DI3FF
cF F F F0 P P XP
8c 8d3i 8 8 8 8M M MD D D
! !c !HSM !]bcf]g ! ! ! ! ! ! ! !r! r )r r shutilr typingr r r r r
sentencepiecer9 tokenization_utilsr
utilsr
get_loggerr r r| r r r r! r r s ! 3 3 3 3 3 3 3 3 3 3 3 3 3 3 5 5 5 5 5 5
H % % !#<= G! G! G! G! G!' G! G! G! G! G!r!