gN d Z ddlZddlZddlZddlmZ ddlmZmZm Z m
Z
m Z m Z ddl
ZddlmZ ddlmZ ddlmZ erdd lmZ dd
lmZ ej e Zd d iZd
Z G d de Z dS )z Tokenization class for model T5. N)copyfile)
TYPE_CHECKINGAnyDictListOptionalTuple )import_protobuf)PreTrainedTokenizer)
AddedToken) TextInput)logging
vocab_filez spiece.modelu ▁c
< e Zd ZdZeZddgZ d)d
eee e
f d df fd
Z d*dZ e
d Zed Zd Z d+dee deee ded ee f fd
Zd Zd Zdee d ee fdZ d,dee deee d ee fdZ d,dee deee d ee fdZd Zd Z d d d ee f fd Z ed! Z d" Z d# Z d$ Z!d% Z"d,d&e d'ee d e#e fd(Z$ xZ%S )- T5Tokenizera
Construct a T5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece).
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are
accessible as "" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be
retrieved by calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids
method
additional_special_tokens (`List[str]`, *optional*):
Additional special tokens used by the tokenizer.
sp_model_kwargs (`dict`, *optional*):
Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for
SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things,
to set:
- `enable_sampling`: Enable subword regularization.
- `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout.
- `nbest_size = {0,1}`: No sampling is performed.
- `nbest_size > 1`: samples from the nbest_size results.
- `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice)
using forward-filtering-and-backward-sampling algorithm.
- `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for
BPE-dropout.
legacy (`bool`, *optional*):
Whether or not the `legacy` behaviour of the tokenizer should be used. Legacy is before the merge of #24622
and #25224 which includes fixes to properly handle tokens that appear after special tokens. A simple
example:
- `legacy=True`:
```python
>>> from transformers import T5Tokenizer
>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-base", legacy=True)
>>> tokenizer.encode("Hello .")
[8774, 32099, 3, 5, 1]
```
- `legacy=False`:
```python
>>> from transformers import T5Tokenizer
>>> tokenizer = T5Tokenizer.from_pretrained("google-t5/t5-base", legacy=False)
>>> tokenizer.encode("Hello .") # the extra space `[3]` is no longer here
[8774, 32099, 5, 1]
```
Checkout the [pull request](https://github.com/huggingface/transformers/pull/24565) for more details.
add_prefix_space (`bool`, *optional*, defaults to `False`):
Whether or not to add an initial space to the input. This allows to treat the leading word just as any
other word.
Attributes:
sp_model (`SentencePieceProcessor`):
The *SentencePiece* processor that is used for every conversion (string, tokens and IDs).
input_idsattention_maskd NTsp_model_kwargsreturnc
t |t rt |d n|}t |t rt |d n|}t |t rt |d n|}|i n|| _ || _ || _ t
j di | j | _ | j | |ld |D } t | dk r |d t | D z
}nK|dk r)|t | k rt d| d| d nd
t | D } | }i | _
t t | D ]>} t d | d d
dddd
| j
t | j dz
|z | z
< ?|%t d| j d d}|| _ | |
dd
| _ || _ || _ | | _ t+ j d|||||| j || d|
d S )NT)specialc 4 g | ]}d t | v |S )
z(T5Tokenizer.__init__.. s, [[[!LTWXYTZTZDZDZADZDZDZ c g | ]}d | d S r > r is r" r# z(T5Tokenizer.__init__.. s$ -Z-Z-ZA.?1.?.?.?-Z-Z-Zr$ r zBoth extra_ids (z!) and additional_special_tokens (zk) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokensc g | ]}d | d S r' r) r* s r" r# z(T5Tokenizer.__init__.. s$ HHH! - - - -HHHr$ r r( F) single_wordlstriprstripr
normalizedz2You are using the default legacy behaviour of the a_ . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 from_slow) eos_token unk_token pad_token extra_idsadditional_special_tokensr legacyadd_prefix_spacer) )
isinstancer r
r r
_extra_idsspmSentencePieceProcessorsp_modelLoadlenrange
ValueError_added_tokens_decoderlogger warning_once __class__r7 get_spm_processorpopr8 super__init__)selfr r2 r3 r4 r5 r6 r r7 r8 kwargs extra_tokensr+ rE s r" rI zT5Tokenizer.__init__ s JT^ J J J
F ..vzz+u/M/MNN
$# 0
&? 0 -
r$ Fc ( t j di | j }| j s|r | | j |S t
| j d 5 }| }t d| j j
d }|j | }|
}d|_ |j | | }| | d d d n # 1 swxY w Y |S )NrbzThe new behaviour of z (with `self.legacy = False`)Fr) )r; r< r r7 r> r openreadr rE __name__
ModelProto
FromStringNormalizerSpecadd_dummy_prefixnormalizer_spec MergeFromSerializeToStringLoadFromSerializedProto)rJ r1 tokenizerfr= model_pb2modelrV s r" rF z T5Tokenizer.get_spm_processor sH .FF1EFF ; ) NN4? + + +
$/4
(
( 8AvvxxH'(v@W(v(v(vwwI (33H==E ' 6 6 8 8O/4O , ! + +O < < < ..00H - -h 7 7 7 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 s B*DD D c | t j v rEt j | }|||k r|S |'t j d| d| d| d| d t |S )NzGThis tokenizer was incorrectly instantiated with a model max length of z which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on z( automatically truncating your input to zM when padding/encoding.
- If you want to encode/pad to sequences longer than z you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.)r max_model_input_sizeswarningswarn
FutureWarning) pretrained_model_name_or_pathmax_model_lengthinit_max_model_lengthdeprecated_max_model_lengths r" !_eventually_correct_t5_max_lengthz-T5Tokenizer._eventually_correct_t5_max_length s (K,M M M*5*KLi*j '$05JN^5^5^,,&.
g3g g 6 g g
4 g g %@
g g g " r$ c 4 | j S N)r= get_piece_sizerJ s r"
vocab_sizezT5Tokenizer.vocab_size s }++---r$ c | fdt j D }| j |S )Nc < i | ]} | |S r) )convert_ids_to_tokens)r r+ rJ s r"
z)T5Tokenizer.get_vocab.. s) RRRa++A..RRRr$ )r@ rl updateadded_tokens_encoder)rJ vocabs ` r" get_vocabzT5Tokenizer.get_vocab s@ RRRR5 ;Q;QRRR
T./// r$ token_ids_0 token_ids_1already_has_special_tokensc |r$t ||d S |dgt | z dgz S dgt | z dgz dgt | z z dgz S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)ru rv rw Nr r% )rH get_special_tokens_maskr? )rJ ru rv rw rE s r" ry z#T5Tokenizer.get_special_tokens_mask s $ & 7722 '[]a 3
C#k***qc1 1c+&&&1#-!s;7G7G1GHA3NNr$ c b t t t d | j S )Nc J t t j d| d uS )Nz)boolresearch)r! s r" z1T5Tokenizer.get_sentinel_tokens.. s bi0A1&E&E!F!Fd!R r$ )listsetfilterr6 rk s r" get_sentinel_tokensz T5Tokenizer.get_sentinel_tokens s1 RRTXTrss t t
r$ c D fd D S )Nc : g | ]} | S r) )convert_tokens_to_ids)r tokenrJ s r" r# z6T5Tokenizer.get_sentinel_token_ids.. s' ZZZe**511ZZZr$ )r rk s `r" get_sentinel_token_idsz"T5Tokenizer.get_sentinel_token_ids s) ZZZZt?W?W?Y?YZZZZr$ token_idsc t | dk r0|d | j k r t j d| j d |S || j gz S )z.Do not add eos again if user already added it.r zThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.)r? eos_token_idr` ra r2 )rJ r s r" _add_eos_if_not_presentz#T5Tokenizer._add_eos_if_not_present si y>>A )B-43D"D"D M+T^ + + +
1 22 2r$ c z | j g}|t ||z dgz S t ||z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )r r? )rJ ru rv eoss r" $create_token_type_ids_from_sequencesz0T5Tokenizer.create_token_type_ids_from_sequences) sS ! {S())QC/ /;${2S899QC??r$ c h | | }||S | | }||z S )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
)r )rJ ru rv s r" build_inputs_with_special_tokensz,T5Tokenizer.build_inputs_with_special_tokens? sA & 22;?? 66{CCK , ,r$ c B | j }d |d< |S )Nr= )__dict__copy)rJ states r" __getstate__zT5Tokenizer.__getstate__Y s$
""$$ j r$ c || _ t | d si | _ t j di | j | _ | j | j d S )Nr r) )r hasattrr r; r< r= r> r )rJ ds r" __setstate__zT5Tokenizer.__setstate__^ s_
t .// &#%D 2JJT5IJJ
4?+++++r$ textr c v | j st | dk r t j |fi |S | t
d }| j r
t
|z } t j |fi |}t | dk r*|d t
k r|d | j v r
|dd }|S )z
Converts a string to a list of tokens. If `self.legacy` is set to `False`, a prefix token is added unless the
first token is special.
r r% N)r7 r? rH tokenizereplaceSPIECE_UNDERLINEr8 all_special_tokens)rJ r rK tokensrE s r" r zT5Tokenizer.tokenizeh s
; 4#d))q..#577#D33F33 3|| ,c22 +#d*D!!$11&11 v;;??vay,< < <dNeAeAeABBZF
r$ c t t | j t | j S ri )r? r= encoder r3 rk s r" unk_token_lengthz T5Tokenizer.unk_token_lengthz s* 4=''DN(;(;<<===r$ c . | j s | t df s!| j |t
S | j | j |z t
}t | | j k r|| j d n|S )u(
Returns a tokenized string.
We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any
SPIECE_UNDERLINE. For example: `self.sp_model.encode(f"{SPIECE_UNDERLINE}Hey", out_type = str)` will give
`['H', 'e', 'y']` instead of `['▁He', 'y']`. Thus we always encode `f"{unk_token}text"` and strip the
`unk_token`. Here is an example with `unk_token = ""` and `unk_token_length = 4`.
`self.tokenizer.sp_model.encode(" Hey", out_type = str)[4:]`.
r )out_typeN) r7
startswithr r= r r r3 r? r )rJ r rK r s r" _tokenizezT5Tokenizer._tokenize~ s ; >>
!!!r$ save_directoryfilename_prefixc t j | s t d| d d S t j ||r|dz ndt d z }t j | j t j | k r:t j | j rt | j | nzt j | j sVt |d 5 }| j
}| | d d d n # 1 swxY w Y |fS )NzVocabulary path (z) should be a directory-r r wb)ospathisdirrC errorjoinVOCAB_FILES_NAMESabspathr isfiler rO r= serialized_model_protowrite)rJ r r out_vocab_fileficontent_spiece_models r" save_vocabularyzT5Tokenizer.save_vocabulary sw w}}^,, LLT^TTT U U U F o M_s 2 22QbcoQp p
7??4? + +rw~/N/N N NSUSZSaSabfbqSrSr N T_n 5 5 5 5 00 /nd++
/r'+}'K'K'M'M$-...
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/ s (/E##E'*E')r r r r NNNT)F)NFri )&rQ
__module__ __qualname____doc__r vocab_files_namesmodel_input_namesr r r r rI rF staticmethodrg propertyrl rt r intr| ry r r r r r r r r r r r r r r r
__classcell__)rE s @r" r r - s- L L\ *$&67
"&48 H
H
"$sCx.1H
H
H
H
H
H
H
V " \ * . . X. sxO O 9O3;DI3FOkoO
cO O O O O O8
[ [ [ 3c 3tCy 3 3 3 3 JN@ @ 9@3;DI3F@
c@ @ @ @. JN- - 9-3;DI3F-
c- - - -4
, , ,[ tCy $ > > X>c c c$0 0 0
" " ".! !c !HSM !]bcf]g ! ! ! ! ! ! ! !r$ r ) r r r} r` shutilr typingr r r r r r
sentencepiecer; convert_slow_tokenizerr tokenization_utilsr tokenization_utils_baser
r utilsr
get_loggerrQ rC r r r r) r$ r" r s@ ' & B B B B B B B B B B B B B B B B 5 5 5 5 5 5 5 5 5 5 5 5 1 1 1 1 1 1 5444444
H % %!>2
R! R! R! R! R!% R! R! R! R! R!r$