gji d Z ddlZddlZddlZddlmZ ddlmZmZm Z ddl
Z
ddl m Z ddl
mZ ej e Zdd d
Zd Z G d d
e Z dZdZededdddddf Z e
j dd e z e
j e
j z e
j z Z e
j d Z e
j ee
j e
j z e
j z Z e
j d Z d'd Z!d(d Z" G d! d" Z#d# Z$d$ Z%d)d&Z&dS )*z!Tokenization classes for BERTweet N)copyfile)ListOptionalTuple )PreTrainedTokenizer)loggingz vocab.txtz bpe.codes)
vocab_file merges_filec t }| d }| dd D ]}| ||f |} t | }|S )z
Return set of symbol pairs in a word.
Word is represented as tuple of symbols (symbols being variable-length strings).
r N)setadd)wordpairs prev_charchars n/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/bertweet/tokenization_bertweet.py get_pairsr & s[
EEEQIQRR
9d#$$$ JJE L c
e Zd ZdZeZ d fd Z d d
ee d e ee d ee fd
Z
d d
ee d e ee de d ee f fd
Z d d
ee d e ee d ee fdZ
ed Zd Zd Zd Zd Zd Zd Zd Zd Zd dede e d ee fd Zd Z xZ S )!BertweetTokenizera
Constructs a BERTweet tokenizer, using Byte-Pair-Encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
normalization (`bool`, *optional*, defaults to `False`):
Whether or not to apply a normalization preprocess.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
F X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) cls_token_id sep_token_id)rQ rV rW clsseps r build_inputs_with_special_tokensz2BertweetTokenizer.build_inputs_with_special_tokens s[ ( %&48I7JJ J ! ![ 3&,{:S@@r already_has_special_tokensc |r$t ||d S |dgdgt | z z dgz S dgdgt | z z ddgz dgt | z z dgz S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rV rW r_ Nr
r )rO get_special_tokens_maskrI )rQ rV rW r_ rU s r ra z)BertweetTokenizer.get_special_tokens_mask s & & 7722 '[]a 3
31#K 0 001QC7 7sqcC , ,,-A61#K@P@P:PQUVTWWWr c | j g}| j g}|t ||z |z dgz S t ||z |z |z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. BERTweet does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )r[ rZ rI )rQ rV rW r] r\ s r $create_token_type_ids_from_sequencesz6BertweetTokenizer.create_token_type_ids_from_sequences sm " ! ! s[(3.//1#5 53$s*S0;>DEEKKr c * t | j S N)rI r? rQ s r
vocab_sizez BertweetTokenizer.vocab_size s 4< r c 0 t | j fi | j S re )rF r? added_tokens_encoderrf s r get_vocabzBertweetTokenizer.get_vocab s DL>>D$=>>>r c | j v r
j | S t | }t t |d d |d dz gz }t | }|s|S t | fd }| j vrn8|\ }}g }d}|t
| k r | || } | ||| | }n-# t $ r | ||d Y nw xY w|| |k rC|t
| dz
k r-||dz |k r |
||z |dz
}n |
|| |dz
}|t
| k t | }|}t
| dk rnt | }Wd | }|d d
}| j |< |S ) Nr+ zTc T j | t d S )Ninf)rJ getfloat)pairrQ s r z'BertweetTokenizer.bpe.. s 1C1CD%PU,,1W1W r )keyr r
r @@ ) rK r- listr minrJ rI indexextend
ValueErrorappendjoin)
rQ tokenr r bigramfirstsecondnew_wordijs
` r bpezBertweetTokenizer.bpe s# DJ :e$ $U||T$ss)__R6(9'::;;$ L ($W$W$W$WXXXFT^++ "ME6HAc$ii--
5!,,A
OOD1I...AA " OODH---E 7e##CIIM(9(9d1q5kV>S>S OOEFN333FAA OODG,,,FA c$ii-- XHD4yyA~~!$9 (: zz$ CRCy
5 s (C 'DDc | j r| | }g }t j d| }|D ]J}| t | | d K|S )zTokenize a string.z\S+\n? )r2 normalizeTweetrefindallrx ru r r. )rQ text split_tokenswordsr| s r _tokenizezBertweetTokenizer._tokenize( s -&&t,,D
9d++ B BE TXXe__%:%:3%?%? @ @ A A A Ar c j D ]#}| | j | }$ j | }d fd|D }| dd dd dd dd d
d }| d d
dd dd dd dd dd }| dd dd d d d d }d | S ) z'
Normalize a raw Tweet
r c : g | ]} | S r" )normalizeToken)r# r| rQ s r r0 z4BertweetTokenizer.normalizeTweet..; s' M M MUd 1 1% 8 8 M M Mr zcannot zcan not zn't z n't zn 't zca n'tzcan'tzai n'tzain'tz'm z 'm z're z 're z's z 's z'll z 'll z'd z 'd z've z 've z p . m .z p.m.z p . m z p.m z a . m .z a.m.z a . m z a.m )rN replacerM tokenizer{ r. )rQ tweetpuncttokens normTweets ` r r z BertweetTokenizer.normalizeTweet3 s ( E EEMM%)>@@
r save_directoryfilename_prefixc t j | s t d| d d S t j ||r|dz ndt d z }t j ||r|dz ndt d z }t j | j t j | k r:t j | j rt | j | nzt j | j sVt |d 5 }| j
}| | d d d n # 1 swxY w Y t j | j t j | k rt | j | ||fS )NzVocabulary path (z) should be a directory-r r
r wb)ospathisdirr= errorr{ VOCAB_FILES_NAMESabspathr
isfiler rD sp_modelserialized_model_protowriter )rQ r r out_vocab_fileout_merge_fileficontent_spiece_models r save_vocabularyz!BertweetTokenizer.save_vocabularyu s w}}^,, LLT^TTT U U U F o M_s 2 22QbcoQp p
o M_s 2 22QbcpQq q
7??4? + +rw~/N/N N NSUSZSaSabfbqSrSr N T_n 5 5 5 5 00 /nd++
/r'+}'K'K'M'M$-...
/
/
/
/
/
/
/
/
/
/
/
/
/
/
/ 7??4+ , ,0O0O O O T%~ 6 6 6 ~--s /FF F c t |t rs t |dd 5 }| | ddd n # 1 swxY w Y n0# t $ r}|d}~wt
$ r t
d| d w xY wdS | }|D ]f}| }| d }|dk rt d |d| }t | j | j |< gdS )
zi
Loads a pre-existing dictionary from a text file and adds its symbols to this instance.
rr' r( Nz Incorrect encoding detected in z , please rebuild the datasetr r+ z5Incorrect dictionary format, expected ' ')
isinstancer@ rD rA FileNotFoundError UnicodeError Exception readlinesr rfindry rI r? ) rQ ffdfnfelineslineTmplineidxr s r rA z BertweetTokenizer.add_from_file sq a
c!S7333 +r&&r***+ + + + + + + + + + + + + + +$
c
c
c a! a a abbb
c F
3 3G==??D**S//Cbyy !XYYY:D!$T\!2!2DL
3 3s9 A A A AA AA
B!A##!B)Fr r r r r r r re )NF) __name__
__module__ __qualname____doc__r vocab_files_namesrP r intr r^ boolra rc propertyrg rj r r r r r r r r@ r r rA
__classcell__)rU s @r r r 6 sS 0 0d * :
:
:
:
:
:
z JNA A 9A3;DI3FA
cA A A A6 sxX X 9X3;DI3FXkoX
cX X X X X X: JNL L 9L3;DI3FL
cL L L L0 ! ! X!? ? ?* * *X + + +D &I I I7 7 7
. .c .HSM .]bcf]g . . . .:3 3 3 3 3 3 3r r ac
(?:
[<>]?
[:;=8] # eyes
[\-o\*\']? # optional nose
[\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
|
[\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
[\-o\*\']? # optional nose
[:;=8] # eyes
[<>]?
|
<3 # heart
)u # Capture 1: entire matched URL
(?:
https?: # URL protocol and colon
(?:
/{1,3} # 1-3 slashes
| # or
[a-z0-9%] # Single letter or digit or '%'
# (Trying not to match e.g. "URI::Escape")
)
| # or
# looks like domain name followed by a slash:
[a-z0-9.\-]+[.]
(?:[a-z]{2,13})
/
)
(?: # One or more:
[^\s()<>{}\[\]]+ # Run of non-space, non-()<>{}[]
| # or
\([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...)
|
\([^\s]+?\) # balanced parens, non-recursive: (...)
)+
(?: # End with:
\([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...)
|
\([^\s]+?\) # balanced parens, non-recursive: (...)
| # or
[^\s`!()\[\]{};:'".,<>?«»“”‘’] # not a space or one of these punct chars
)
| # OR, the following to match naked domains:
(?:
(?\s]+>z
[\-]+>|<[\-]+z (?:@[\w_]+)z (?:\#+[\w_]+[\w\'_\-]*[\w_]+)z#[\w.+-]+@[\w-]+\.(?:[\w-]\.?)+[\w-]a
(?:[^\W\d_](?:[^\W\d_]|['\-_])+[^\W\d_]) # Words with apostrophes or dashes.
|
(?:[+\-]?\d+[,/.:-]\d+[+\-]?) # Numbers, including fractions, decimals.
|
(?:[\w_]+) # Words without apostrophes or dashes.
|
(?:\.(?:\s*\.){1,}) # Ellipsis dots.
|
(?:\S) # Everything else that isn't whitespace.
z(%s)|z([^a-zA-Z0-9])\1{3,}z&(#?(x?))([^&;\s]+);strictc d |d}t | t r| || S | S )Nr' )r bytesdecode)r r) errorss r _str_to_unicoder \ s8 $ -{{8V,,, Kr r" Tr' c d fd}t |t | | S )u
Remove entities from text by converting them to their corresponding unicode character.
Args:
text:
A unicode string or a byte string encoded in the given *encoding* (which defaults to 'utf-8').
keep (list):
List of entity names which should not be replaced. This supports both numeric entities (`&#nnnn;` and
`&#hhhh;`) and named entities (such as ` ` or `>`).
remove_illegal (bool):
If `True`, entities that can't be converted are removed. Otherwise, entities that can't be converted are
kept "as is".
Returns: A unicode string with the entities removed.
See https://github.com/scrapy/w3lib/blob/master/w3lib/html.py
Examples:
```python
>>> from nltk.tokenize.casual import _replace_html_entities
>>> _replace_html_entities(b"Price: £100")
'Price: \xa3100'
>>> print(_replace_html_entities(b"Price: £100"))
Price: £100
```c P | d }| d r} | d rt |d }nt |d }d|cxk rdk r&n n#t |f d S nO# t $ r d }Y nAw xY w|v r| d S t
j j | }|' t | S # t t f$ r Y nw xY wrd
n| d S ) Nr r
r
cp1252r r ) groupr r r ry htmlentitiesname2codepointrn chr
OverflowError)match entity_bodynumberkeepremove_illegals r _convert_entityz/_replace_html_entities.._convert_entity sO kk!nn ;;q>> G
;;q>> 2 b 1 1FF b 1 1F
6))))T))))) &++228<<<
d"" {{1~~% 599+FF
6{{"
.
$7rrQ 7s$ A(B B&%B&)C8 8D D )ENT_REsubr )r r r r) r s `` r _replace_html_entitiesr d sB <8 8 8 8 8 8: ::otX'F'F G GGr c e Zd ZdZddZd ZdS )rL a
Examples:
```python
>>> # Tokenizer for tweets.
>>> from nltk.tokenize import TweetTokenizer
>>> tknzr = TweetTokenizer()
>>> s0 = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--"
>>> tknzr.tokenize(s0)
['This', 'is', 'a', 'cooool', '#dummysmiley', ':', ':-)', ':-P', '<3', 'and', 'some', 'arrows', '<', '>', '->', '<--']
>>> # Examples using *strip_handles* and *reduce_len parameters*:
>>> tknzr = TweetTokenizer(strip_handles=True, reduce_len=True)
>>> s1 = "@remy: This is waaaaayyyy too much for you!!!!!!"
>>> tknzr.tokenize(s1)
[':', 'This', 'is', 'waaayyy', 'too', 'much', 'for', 'you', '!', '!', '!']
```TFc 0 || _ || _ || _ d S re
preserve_case
reduce_len
strip_handles)rQ r r r s r rP zTweetTokenizer.__init__ s *$ *r c t | }| j rt | }| j rt | }t
d| }t | }| j s d |D }|S )z
Args:
text: str
Returns: list(str) A tokenized list of strings; concatenating this list returns the original string if
`preserve_case=False`
\1\1\1c n g | ]2}t | r|n| 3S r" ) EMOTICON_REsearchr )r# xs r r0 z+TweetTokenizer.tokenize.. s7 NNN1+,,Q//>QQQWWYYNNNr )
r r remove_handlesr reduce_lengtheningHANG_REr WORD_REr r )rQ r safe_textr s r r zTweetTokenizer.tokenize s &d++ (!$''D ? ,%d++DKK 400 **! ONNNNNE r NTFF)r r r r rP r r" r r rL rL sA &+ + + +
r rL c V t j d }| d| S )za
Replace repeated character sequences of length 3 or greater with sequences of length 3.
z (.)\1{2,}r regexcompiler r patterns r r r s' mL))G ;;y$ ' ''r c V t j d }| d| S )z4
Remove Twitter username handles from text.
zv(?r sI ( ' ( ( ( ( ( ( ( ( ( ( 5 5 5 5 5 5
H % %
q3 q3 q3 q3 q3+ q3 q3 q3~
L
$)\ (.
A+ ` %- chhw&7&77 9PSXS`9`
a
a %-/
0
0 emIu}uw'> 'NOO
. / / ;H ;H ;H ;HB0 0 0 0 0 0 0 0p( ( (" " " r