g^"dZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z m Z mZddlmZmZGd d e d ZGd de ZdS)z Processor class for Donut. N)contextmanager)ListOptionalUnion) ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInputceZdZiZdS)DonutProcessorKwargsN)__name__ __module__ __qualname__ _defaultsf/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/donut/processing_donut.pyrrsIIIrrF)totalc eZdZdZddgZdZdZdfd Z dded e e e e e e efd eefd Zd Zd ZedZddZedZedZxZS)DonutProcessora Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single processor. [`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and [`~DonutProcessor.decode`] for more information. Args: image_processor ([`DonutImageProcessor`], *optional*): An instance of [`DonutImageProcessor`]. The image processor is a required input. tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`], *optional*): An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input. image_processor tokenizerAutoImageProcessor AutoTokenizerNc ,d}d|vr/tjdt|d}||n|}|t d|t dt |||j|_d|_ dS)Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.F) warningswarn FutureWarningpop ValueErrorsuper__init__rcurrent_processor_in_target_context_manager)selfrrkwargsr __class__s rr&zDonutProcessor.__init__5s  & ( ( M    !' +> ? ? -<-H//N_  "HII I  ABB B )444!%!5*/'''rimagestextr*c *|jr|j||fi|S||td|jtfd|jji|}||j|fi|d}||j|fi|d}||S||S|d|d<|d|d<|S)a When used in normal mode, this method forwards all its arguments to AutoImageProcessor's [`~AutoImageProcessor.__call__`] and returns its output. If used in the context [`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's [`~DonutTokenizer.__call__`]. Please refer to the doctsring of the above two methods for more information. NzBYou need to specify either an `images` or `text` input to process.tokenizer_init_kwargs images_kwargs text_kwargs input_idslabels)r(r'r$ _merge_kwargsrr init_kwargsr) r)r,r-audiovideosr* output_kwargsinputs encodingss r__call__zDonutProcessor.__call__Is  * B)4)&$AA&AA A >dlabb b**   "&."<    )T)&SSM/4RSSF  &tLL}]/KLLI <M ^ (5F8 "+K"8F; Mrc&|jj|i|S)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r batch_decoder)argsr*s rr=zDonutProcessor.batch_decoders +t~*D;F;;;rc&|jj|i|S)z This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )rdecoder>s rrAzDonutProcessor.decodeys %t~$d5f555rc#Ktjdd|_|j|_dV|j|_d|_dS)z Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR. z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r r!r(rr'rr)s ras_target_processorz"DonutProcessor.as_target_processorsW   9   +/'!% !%!5*/'''rFc||j}i}|rtjd|tj}|nh|d}tj|}tjd|d|tj}|}|||d}n|}tj|} tj|} tj| d| |tjtjz} | | d } d| vr?d| vr;| | d | } | r t| dkr| d } | ||<ng||<| d D]Y} | } | |vr$| d d kr| dddkr | dd} || | Zt||dkr||d ||<|||t|zd }|ddd kr$|g| |ddd | zS|t|r|r|gn|S|rgnd|iS)zS Convert a (generated) token sequence into an ordered JSON format. Nz zz(.*?)z text_sequence)rget_added_vocabresearch IGNORECASEgroupescapereplaceDOTALLstrip token2jsonlensplitappendfind)r)tokensrIrJoutput start_tokenkey key_escaped end_tokenstart_token_escapedend_token_escapedcontentvalueleafs rrXzDonutProcessor.token2jsons  .88::K& p)L&"-HHK"##A&&C)C..K "8+"8"8"8&"-PPI%++--K  R88%OO-- &(i &<&<#$&Ii$8$8!)*DD1BDDfbm^`^gNg&%mmA..4466G((W-?-? $Ze f f 0"5zzQ(-a*/F3K&(s $+MM)$<$<55D#'::<rws  %%%%%%((((((((((%%%%%%HHHHHHHHHHCCCCCCCC+5s$s$s$s$s$^s$s$s$s$s$r