gbdZddlmZmZddlmZddlmZddlm Z m Z m Z GddeZ d S) z Processor class for MarkupLM. )OptionalUnion) TensorType)ProcessorMixin) BatchEncodingPaddingStrategyTruncationStrategyc eZdZdZdZdZdZ dded eee e fd eee e fd e e d e d e e de ede edededededede ee efdefdZdZdZedZdS)MarkupLMProcessoraJ Constructs a MarkupLM processor which combines a MarkupLM feature extractor and a MarkupLM tokenizer into a single processor. [`MarkupLMProcessor`] offers all the functionalities you need to prepare data for the model. It first uses [`MarkupLMFeatureExtractor`] to extract nodes and corresponding xpaths from one or more HTML strings. Next, these are provided to [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`], which turns them into token-level `input_ids`, `attention_mask`, `token_type_ids`, `xpath_tags_seq` and `xpath_subs_seq`. Args: feature_extractor (`MarkupLMFeatureExtractor`): An instance of [`MarkupLMFeatureExtractor`]. The feature extractor is a required input. tokenizer (`MarkupLMTokenizer` or `MarkupLMTokenizerFast`): An instance of [`MarkupLMTokenizer`] or [`MarkupLMTokenizerFast`]. The tokenizer is a required input. parse_html (`bool`, *optional*, defaults to `True`): Whether or not to use `MarkupLMFeatureExtractor` to parse HTML strings into nodes and corresponding xpaths. MarkupLMFeatureExtractor)MarkupLMTokenizerMarkupLMTokenizerFastTNFradd_special_tokenspadding truncation max_lengthstridepad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosereturn_tensorsreturnc |jrL|td|||td||}|d}|d}n$|td||td||jrt|tr|g}|jdid||n|d ||ndd|d |d |d |d |d| d| d| d| d| d|d|d|d|d|d||}|S)a This method first forwards the `html_strings` argument to [`~MarkupLMFeatureExtractor.__call__`]. Next, it passes the `nodes` and `xpaths` along with the additional arguments to [`~MarkupLMTokenizer.__call__`] and returns the output. Optionally, one can also provide a `text` argument which is passed along as first sequence. Please refer to the docstring of the above two methods for more information. NzDMake sure to pass HTML strings in case `parse_html` is set to `True`zUPlease don't pass nodes, xpaths nor node labels in case `parse_html` is set to `True`nodesxpathsz@You have passed HTML strings but `parse_html` is set to `False`.zIMake sure to pass nodes and xpaths in case `parse_html` is set to `False`text text_pair node_labelsrrrrrrrrrrrrrr) parse_html ValueErrorfeature_extractor isinstancestr tokenizer)self html_stringsr r!r$ questionsrrrrrrrrrrrrrrkwargsfeaturesencoded_inputss l/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/markuplm/processing_markuplm.py__call__zMarkupLMProcessor.__call__2sB ? n# !ghhh F$6+:Q k--l;;HW%Eh'FF' !cddd} !lmmm  T_ )S)) (&K '   '3 (4ee$ 6 $  21  G  "z "z 6  21 #8"7 #8"7 '@&? (B'A $:#9 (-! "G# $*>'  ,c&|jj|i|S)z This method forwards all its arguments to TrOCRTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r+ batch_decoder,argsr/s r2r6zMarkupLMProcessor.batch_decodes +t~*D;F;;;r4c&|jj|i|S)z This method forwards all its arguments to TrOCRTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )r+decoder7s r2r:zMarkupLMProcessor.decodes %t~$d5f555r4c|jj}|S)N)r+model_input_names)r,tokenizer_input_namess r2r<z#MarkupLMProcessor.model_input_namess $ @$$r4)NNNNNTFNNrNNNFFFFTN)__name__ __module__ __qualname____doc__feature_extractor_classtokenizer_classr&boolrr*r r rintrrr3r6r:propertyr<r%r4r2r r s&9DOJ#'5:;?$(,00404*/+0',#;?)NN!NtS/12N$%778NSMNN%SMN (~N (~N$(N %)!N"!%#N$%N&'N(!sJ!78)N, -NNNN`<<<666%%X%%%r4r N) rAtypingrr file_utilsrprocessing_utilsrtokenization_utils_baserr r r r%r4r2rKs#"""""""$$$$$$......YYYYYYYYYYy%y%y%y%y%y%y%y%y%y%r4