gR z d Z ddlmZmZmZmZ ddlmZmZ ddl m
Z
e
j e Z
G d de ZdS ) z!Tokenization class for Perceiver. )DictListOptionalTuple )
AddedTokenPreTrainedTokenizer)loggingc
d e Zd ZdZddgZ d d fd
Zd eeef fdZ e
d Z d de e de
e e ded e e f fd
Z d!de e de
e e d e e fdZded e e fdZd Zd Zd Zd!ded e
e d ee fd Z xZS )"PerceiverTokenizeraS
Construct a Perceiver tokenizer. The Perceiver simply uses raw bytes utf-8 encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
pad_token (`str`, *optional*, defaults to `"[PAD]"`):
The token used for padding, for example when batching sequences of different lengths.
bos_token (`str`, *optional*, defaults to `"[BOS]"`):
The BOS token (reserved in the vocab, but not actually used).
eos_token (`str`, *optional*, defaults to `"[EOS]"`):
The end of sequence token (reserved in the vocab, but not actually used).
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
mask_token (`str`, *optional*, defaults to `"[MASK]"`):
The MASK token, useful for masked language modeling.
cls_token (`str`, *optional*, defaults to `"[CLS]"`):
The CLS token (reserved in the vocab, but not actually used).
sep_token (`str`, *optional*, defaults to `"[SEP]"`):
The separator token, which is used when building a sequence from two sequences.
input_idsattention_mask[PAD][BOS][EOS][MASK][CLS][SEP] returnNc t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}t |t rt |dd n|}d| _ ||||||d| _ t | j | _ t j d|||||||d| d S )NF)lstriprstrip )r r ) pad_token bos_token eos_token
mask_token cls_token sep_tokenmodel_max_length )
isinstancestrr _utf_vocab_size_added_tokens_decoderlen_num_special_tokenssuper__init__)
selfr r r! r" r# r$ r% kwargs __class__s
p/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/perceiver/tokenization_perceiver.pyr. zPerceiverTokenizer.__init__; s JTT]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir KUV`beKfKfvZ
5GGGGlv
IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir #
6
6
" $'t'A#B#B
! -
c i }t | j D ] }t | }|| j z ||< | | j |S N)ranger) chrr, updateadded_tokens_encoder)r/ vocabitokens r2 get_vocabz PerceiverTokenizer.get_vocabd sY t+,, 8 8AFFE t 77E%LL
T./// r3 c | j S r5 )r) )r/ s r2
vocab_sizez PerceiverTokenizer.vocab_sizel s ##r3 F token_ids_0 token_ids_1already_has_special_tokensc |r$t ||d S |dgdgt | z z dgz S dgdgt | z z dgz dgt | z z dgz S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)r@ rA rB Nr r )r- get_special_tokens_maskr+ )r/ r@ rA rB r1 s r2 rD z*PerceiverTokenizer.get_special_tokens_maskp s $ & 7722 '[]a 3
3!s; / ///1#5 5sqcC , ,,-3sS=M=M7MNRSQTTTr3 c n || j g|z | j gz S | j g|z | j gz |z | j gz S )af
Build model inputs from a sequence or a pair of sequence for sequence classification tasks. A sequence has the
following format:
- single sequence: `[CLS] X [SEP]`
- pair of sequences: `[CLS] A [SEP] B [SEP]`
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) cls_token_id sep_token_id)r/ r@ rA s r2 build_inputs_with_special_tokensz3PerceiverTokenizer.build_inputs_with_special_tokens sS & %&48I7JJ J%&48I7JJ[X\`\m[nn nr3 textc D d | d D }|S )zPTake as input a string and return a list of strings (tokens) for words/sub-wordsc , g | ]}t | S r& )r7 ).0r; s r2
z0PerceiverTokenizer._tokenize.. s 777Q#a&&777r3 utf-8)encode)r/ rI tokenss r2 _tokenizez PerceiverTokenizer._tokenize s& 77$++g"6"6777
r3 c j t | dk r| j }nt | | j z }|S )z0Converts a token (str) in an id using the vocab.r )r+ unk_token_idordr, )r/ r< token_ids r2 _convert_token_to_idz'PerceiverTokenizer._convert_token_to_id s3 u::??(HH5zzD$< $ $ X$ sxU U 9U3;DI3FUkoU
cU U U U U U: JNo o 9o3;DI3Fo
co o o o0c d3i
c HSM ]bcf]g r3 r N)rj typingr r r r tokenization_utilsr r utilsr
get_loggerrg loggerr r& r3 r2 ru s ( ' . . . . . . . . . . . . A A A A A A A A
H % %k k k k k, k k k k kr3