gandZddlmZmZmZddlmZddlmZm Z m Z m Z m Z ddl mZGddeZd S) z Processor class for Bros. )ListOptionalUnion)ProcessorMixin) BatchEncodingPaddingStrategyPreTokenizedInput TextInputTruncationStrategy) TensorTypec"TeZdZdZdgZdZdfd Z dd eee e ee e fd e d ee e e fd ee e efd eededeedee dee de de de de de deee efdef dZdZdZedZxZS) BrosProcessora Constructs a Bros processor which wraps a BERT tokenizer. [`BrosProcessor`] offers all the functionalities of [`BertTokenizerFast`]. See the docstring of [`~BrosProcessor.__call__`] and [`~BrosProcessor.decode`] for more information. Args: tokenizer (`BertTokenizerFast`, *optional*): An instance of ['BertTokenizerFast`]. The tokenizer is a required input. tokenizer) BertTokenizerBertTokenizerFastNc l|tdt|dS)Nz"You need to specify a `tokenizer`.) ValueErrorsuper__init__)selfrkwargs __class__s d/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/bros/processing_bros.pyrzBrosProcessor.__init__)s6  ABB B #####TFrtextadd_special_tokenspadding truncation max_lengthstridepad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosereturn_tensorsreturnc @|jd||||||||| | | | | ||d|}|S)z This method uses [`BertTokenizerFast.__call__`] to prepare text for the model. Please refer to the docstring of the above two methods for more information. )rrrrr r!r"r#r$r%r&r'r(r)r*)r)rrrrrr r!r"r#r$r%r&r'r(r)r*rencodings r__call__zBrosProcessor.__call__/s_0"4> 1!!1"7"7&?'A#9')  !  &rc&|jj|i|S)z This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoderargsrs rr1zBrosProcessor.batch_decode\s +t~*D;F;;;rc&|jj|i|S)z This method forwards all its arguments to BertTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoder2s rr5zBrosProcessor.decodecs %t~$d5f555rch|jj}tt|SN)rmodel_input_nameslistdictfromkeys)rtokenizer_input_namess rr8zBrosProcessor.model_input_namesjs( $ @DMM"788999rr7)NTFNNrNNNFFFFTN)__name__ __module__ __qualname____doc__ attributestokenizer_classrrr r rboolstrr r rintr rr/r1r5propertyr8 __classcell__)rs@rrrs  JrLs)(((((((((......wwwwwwwwwwwwwwS:S:S:S:S:NS:S:S:S:S:r