g9 4dZddlZddlZddlZddlZddlZddlZddlZddlm Z ddlm Z m Z m Z m Z mZmZmZddlZddlZddlmZddlmZmZmZerddlmZdd lmZmZmZmZm Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.e.j/e0Z1e*e e2j3Z4d d d dZ5ej6dkrej7Z7nej7Z7GddedZ8GddedZ9GddedZ:GddedZ;GddedZ<Gdde8e9e:e;ed Z?e)e>j@e>_@e>j@j.e>j@jAd!d"d#$e>j@_dSdS)%z8 Processing saving/loading class for common processors. N)Path)AnyDictListOptionalTuple TypedDictUnion)custom_object_save)ChannelDimensionis_valid_imageis_vision_available)PILImageResampling)PaddingStrategyPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategy) CHAT_TEMPLATE_NAMEPROCESSOR_NAMEPushToHubMixin TensorTypeadd_model_info_to_auto_map"add_model_info_to_custom_pipelines cached_file copy_funcdirect_transformers_import download_urlis_offline_mode is_remote_urlloggingrFeatureExtractionMixinImageProcessingMixin) AutoTokenizerAutoFeatureExtractorAutoImageProcessor) c$eZdZUdZeeeeeeeefe d<eeeeeeefe d<eeeeeeeefe d<ee e d<ee e e fe d<ee e e fe d<eee d<eee d <ee e d <eee d <ee e d <ee e d <ee e d<ee e d<ee e d<ee e d<ee e d<ee e d<dS) TextKwargsa Keyword arguments for text processing. For extended documentation, check out tokenization_utils_base methods and docstrings associated. Attributes: add_special_tokens (`bool`, *optional*) Whether or not to add special tokens when encoding the sequences. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*) Activates and controls padding. truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*): Activates and controls truncation. max_length (`int`, *optional*): Controls the maximum length to use by one of the truncation/padding parameters. stride (`int`, *optional*): If set, the overflowing tokens will contain some tokens from the end of the truncated sequence. is_split_into_words (`bool`, *optional*): Whether or not the input is already pre-tokenized. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_token_type_ids (`bool`, *optional*): Whether to return token type IDs. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. return_overflowing_tokens (`bool`, *optional*): Whether or not to return overflowing token sequences. return_special_tokens_mask (`bool`, *optional*): Whether or not to return special tokens mask information. return_offsets_mapping (`bool`, *optional*): Whether or not to return `(char_start, char_end)` for each token. return_length (`bool`, *optional*): Whether or not to return the lengths of the encoded inputs. verbose (`bool`, *optional*): Whether or not to print more information and warnings. padding_side (`str`, *optional*): The side on which padding will be applied. text_pair text_targettext_pair_targetadd_special_tokenspadding truncation max_lengthstrideis_split_into_wordspad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbose padding_sideN)__name__ __module__ __qualname____doc__rr rrr__annotations__boolstrrrintY/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/processing_utils.pyr+r+Qs##Ji):DOTRcMddeffffy"3T)_dK\F]]^^^^uY0A4 ?TXYjTk%klmmmm &&& 4o- ....dC!334444  SM!$'''  %%%#D>)))#D>)))'~--- (...$TN***D>!!! d^3-rGr+F)totalceZdZUdZeeed<eeee fed<ee ed<eeee fed<ee de fed<eeed<ee ed <eeed <ee e e e fed <ee e e e fed <eeed <eeee fed<eeed<ee ed<ee ee fed<dS) ImagesKwargsaW Keyword arguments for image processing. For extended documentation, check the appropriate ImageProcessor class methods and docstrings. Attributes: do_resize (`bool`, *optional*): Whether to resize the image. size (`Dict[str, int]`, *optional*): Resize the shorter side of the input to `size["shortest_edge"]`. size_divisor (`int`, *optional*): The size by which to make sure both the height and width can be divided. crop_size (`Dict[str, int]`, *optional*): Desired output size when applying center-cropping. resample (`PILImageResampling`, *optional*): Resampling filter to use if resizing the image. do_rescale (`bool`, *optional*): Whether to rescale the image by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*): Mean to use if normalizing the image. image_std (`float` or `List[float]`, *optional*): Standard deviation to use if normalizing the image. do_pad (`bool`, *optional*): Whether to pad the image to the `(max_height, max_width)` of the images in the batch. pad_size (`Dict[str, int]`, *optional*): The size `{"height": int, "width" int}` to pad the images to. do_center_crop (`bool`, *optional*): Whether to center crop the image. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the output image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. do_resizesize size_divisor crop_sizerresample do_rescalerescale_factor do_normalize image_mean image_stddo_padpad_sizedo_center_crop data_formatinput_data_formatN)r>r?r@rArrCrBrrDrEr floatrr rFrGrHrKrKsb##J~ 4S> """"3-S#X''''u13678888UO###4.   ud5k123333eT%[012222 TNtCH~&&&&TN"""*++++c+;&; <======rGrKceZdZUdZeeed<eeee fed<ee ed<eded<eeed<ee ed<eeed <ee e e e fed <ee e e e fed <eeed <eeed <ee ed<ee ee fed<dS) VideosKwargsa Keyword arguments for video processing. Attributes: do_resize (`bool`): Whether to resize the image. size (`Dict[str, int]`, *optional*): Resize the shorter side of the input to `size["shortest_edge"]`. size_divisor (`int`, *optional*): The size by which to make sure both the height and width can be divided. resample (`PILImageResampling`, *optional*): Resampling filter to use if resizing the image. do_rescale (`bool`, *optional*): Whether to rescale the image by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*): Mean to use if normalizing the image. image_std (`float` or `List[float]`, *optional*): Standard deviation to use if normalizing the image. do_pad (`bool`, *optional*): Whether to pad the image to the `(max_height, max_width)` of the images in the batch. do_center_crop (`bool`, *optional*): Whether to center crop the image. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the output image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. rLrMrNrrPrQrRrSrTrUrVrXrYrZN)r>r?r@rArrCrBrrDrEr[r rr rFrGrHr]r]s&@~ 4S> """"3-+,,,,UO###4.   ud5k123333eT%[012222 TNTN"""*++++c+;&; <======rGr]ceZdZUdZeeed<eedee edeee fed<eee e e fed<eeed<ee ed<eeed<ee ed <d S) AudioKwargsa Keyword arguments for audio processing. Attributes: sampling_rate (`int`, *optional*): The sampling rate at which the `raw_speech` input was sampled. raw_speech (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`): The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). truncation (`bool`, *optional*): Activates truncation to cut input sequences longer than *max_length* to *max_length*. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_attention_mask (`bool`, *optional*): Whether or not [`~ASTFeatureExtractor.__call__`] should return `attention_mask`. sampling_ratez np.ndarray raw_speechr0r2r1r5r7N) r>r?r@rArrErBr rr[rCrDrrFrGrHr_r_s:C=   |T%[$|:LdSWX]S^N__`aaaa eD#67 8888   %%%#D>)))))rGr_c6eZdZUeeeefed<dS) CommonKwargsreturn_tensorsN)r>r?r@rr rDrrBrFrGrHrcrcs,U3 ?3444444rGrcceZdZUdZiejZeed<iejZeed<ie jZ e ed<ie jZ e ed<ie jZe ed<dS)ProcessingKwargsa' Base class for kwargs passing to processors. A model should have its own `ModelProcessorKwargs` class that inherits from `ProcessingKwargs` to provide: 1) Additional typed keys and that this model requires to process inputs. 2) Default values for existing keys under a `_defaults` attribute. New keys have to be defined as follows to ensure type hinting is done correctly. ```python # adding a new image kwarg for this model class ModelImagesKwargs(ImagesKwargs, total=False): new_image_kwarg: Optional[bool] class ModelProcessorKwargs(ProcessingKwargs, total=False): images_kwargs: ModelImagesKwargs _defaults = { "images_kwargs: { "new_image_kwarg": False, } "text_kwargs": { "padding": "max_length", }, } ``` For Python 3.8 compatibility, when inheriting from this class and overriding one of the kwargs, you need to manually update the __annotations__ dictionary. This can be done as follows: ```python class CustomProcessorKwargs(ProcessingKwargs, total=False): images_kwargs: CustomImagesKwargs CustomProcessorKwargs.__annotations__["images_kwargs"] = CustomImagesKwargs # python 3.8 compatibility ```python common_kwargs text_kwargs images_kwargs videos_kwargs audio_kwargsN)r>r?r@rArcrBrgr+rhrKrir]rjr_rkrFrGrHrfrfs##J#  &#M<  $K#  &#M<#  &#M<!  %!L+rGrfceZdZUdZddgZdgZgZeee d<dZ dZ dZ gZ eee d<dZd eeeffd Zd efd Zd eeejffd ZdZd)defdZedeeejfd eeeefeeefffdZedeeeffdZ d*dedeed eeeffdZ e d+deeejfdeeeejfdededeeeefdef dZ!ed,d!Z"ed"Z#e$d#Z%e&d$Z'd%Z( d-d&eeeeefdeed'ed efd(Z)dS).ProcessorMixinza This is a mixin used to provide saving/loading functionality for all processor classes. feature_extractor tokenizer chat_templateoptional_call_argsN valid_kwargsc |jD]'}t||||d(|D]}||jvrt d|dt ||jD]!\}}||vrt d|d|||<"t |t |jkrPtdt |jdd|jdt |d| D]\}}t||d }t ||}t|trtd |D}ntt|}t||s+t d t!|jd |d |dt|||dS)NzUnexpected keyword argument .z!Got multiple values for argument zThis processor requires z arguments: , z. Got z arguments instead._classc3DK|]}|tt|VdSNgetattrtransformers_module.0ns rH z*ProcessorMixin.__init__..~s3$j$j\]\iW-@!%D%D\i\i\i\i$j$jrGz Received a z for argument z, but a z was expected.)optional_attributessetattrpop attributes TypeErrorziplen ValueErrorjoinitemsrzAUTO_TO_BASE_CLASS_MAPPINGget isinstancetupler{typer>) selfargskwargsoptional_attributekeyargattribute_name class_name proper_classs rH__init__zProcessorMixin.__init__csB#'": T T  D,fjj9KT.R.R S S S S G GC$/)) Es E E EFFF*#&tT_#=#= - - C'' UN U U UVVV),~&& v;;#do.. . .23t+?+?22TYYW[WfMgMg22t99222  $*<<>> / / NC .'@'@'@AAJ377 JOOJ*e,, H$$j$jj$j$j$jjj &':JGG c<00 v$s))"4vvNvv\fvvv D.# . . . . / /rGreturncvtjj}tjj}|jfdDdgz fd|D}jj |d<d|vr|d=d|vr|d=d|vr|d=d|vr|d=d |D}|S) z Serializes this instance to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this processor instance. c0g|]}|jjv|SrF) __class__r)r}xrs rH z*ProcessorMixin.to_dict..s(XXXqQdn>W5W5W5W5W5WrGauto_mapc$i|] \}}|v || SrFrF)r}kv attrs_to_saves rH z*ProcessorMixin.to_dict..s)HHH41aQ-5G5G!Q5G5G5GrGprocessor_classroimage_processorrnrpcdi|]-\}}t|ts|jjdk*||.S)BeamSearchDecoderCTC)rrrr>)r}rrs rHrz*ProcessorMixin.to_dict..sM   1q.11 67[5IMc5c5c q5c5c5crG) copydeepcopy__dict__inspect signaturer parametersrrr>)routputsigrs` @rHto_dictzProcessorMixin.to_dictst}-- .. XXXXMXXX *% HHHH6<<>>HHH$(N$; ! & {#  & &() & ( (*+ f $ $'        rGc^|}tj|dddzS)z Serializes this instance to a JSON string. Returns: `str`: String containing all the attributes that make up this feature_extractor instance in JSON format. Tindent sort_keys )rjsondumps)r dictionarys rHto_json_stringzProcessorMixin.to_json_strings-\\^^ z*Q$???$FFrGjson_file_pathct|dd5}||ddddS#1swxYwYdS)z Save this instance to a JSON file. Args: json_file_path (`str` or `os.PathLike`): Path to the JSON file in which this processor instance's parameters will be saved. wutf-8encodingN)openwriter)rrwriters rH to_json_filezProcessorMixin.to_json_files.# 8 8 8 0F LL,,.. / / / 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0s(AA A cfdjD}d|}jjd|dS)Nc Vg|]%}d|dtt|&S)z- z: )reprrz)r}namers rHrz+ProcessorMixin.__repr__..s<```CCCWT4-@-@(A(ACC```rGrz: z )rrrr>r)rattributes_reprs` rH__repr__zProcessorMixin.__repr__s^````PTP_```))O44.)ZZoZZ4CVCVCXCXZZZrGF push_to_hubc |dd}|Dtjdt|ddt d||d<t j|d|rw|dd}|d |t j j d }j |fi|} |}j Ffd jD}d |D} | t!|| jD]V} t#| } t%| dr| jj| |Wj 7jD]/} t#| } t/| t0r| jd=0t j |t6} t j |t8} }j}t?j djidddz}tC| dd5}|"|dddn #1swxYwYtF$d| tK|&dhkr2'| tF$d| |r-(|||||dtK|&dhkrgS| gS)a Saves the attributes of this processor (feature extractor, tokenizer...) in the specified directory so that it can be reloaded using the [`~ProcessorMixin.from_pretrained`] method. This class method is simply calling [`~feature_extraction_utils.FeatureExtractionMixin.save_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`]. Please refer to the docstrings of the methods above for more information. Args: save_directory (`str` or `os.PathLike`): Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.tokenV`token` and `use_auth_token` are both specified. Please set only the argument `token`.T)exist_okcommit_messagerepo_idc0g|]}t|SrF)rz)r}rrs rHrz2ProcessorMixin.save_pretrained..s#YYY~WT>22YYYrGcJg|] }t|tr|jn|!SrF)rr init_kwargs)r}as rHrz2ProcessorMixin.save_pretrained..s0ggg\]A7N)O)OV UVgggrG)config_set_processor_classrrprrrrrrzchat template saved in rzprocessor saved in )rr))rwarningswarn FutureWarningrrosmakedirssplitpathsep _create_repo_get_files_timestamps _auto_classrappendr rzhasattrrrr>save_pretrainedrrrrrrrrprrrrloggerinfosetkeysr_upload_modified_files)rsave_directoryrrrrrfiles_timestampsattrsconfigsr attributeoutput_processor_fileoutput_chat_template_fileprocessor_dictchat_template_json_stringrs` rHrzProcessorMixin.save_pretraineds0 $4d;;  % ME   zz'4((4 l-F7O NT2222  J#ZZ(8$??NjjN,@,@,M,Mb,QRRG'd'::6::G#99.II    'YYYYYYYEggafgggG NN4 t^G D D D D"o 6 6Nn55Iy"899 H..t~/FGGG  % %n 5 5 5 5   '"&/ : :#D.99 i)@AA:!-j9!# ^^ L L$&GLLAS$T$T!   ) OT-?@VZ[[[^bb &/wGGG 86 6777 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 KKM2KMM N N N ~""$$ % %*;)< < <   3 4 4 4 KKE.CEE F F F    ' ' -jj)) (    ~""$$ % %*;)< < <I%&&s.JJJpretrained_model_name_or_pathc |dd}|dd}|dd}|dd}|dd}|dd}|d d} |d d } |d d} |d d} d| d} | | | d<tr|stdd}t |}t j|}t j|rEt j|t}t j|d}t j |r|}d}d}nt|r|}t|}d}n|t}t} t||||||||| | | d }t||||||||| | | d }n6#t$rt $rtd|d|dtdwxYwd}|]t#|dd5}|}dddn #1swxYwYt'j|d}||d<|i|fS t#|dd5}|}dddn #1swxYwYt'j|}n&#t&j$rtd|dwxYw|rtd|n td|d |d|vr"|dtd!|s:d"|vrt/|d"||d"<d#|vrt1|d#||d#<||fS)$a From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a processor of type [`~processing_utils.ProcessingMixin`] using `from_args_and_dict`. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): The identifier of the pre-trained checkpoint from which we want the dictionary of parameters. subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. Returns: `Tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the processor object. cache_dirNforce_downloadFresume_downloadproxiesrlocal_files_onlyrevision subfolder_from_pipeline _from_auto processor) file_typefrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=TrueTzchat_template.json) rrrrrr user_agentrr%_raise_exceptions_for_missing_entrieszCan't load processor for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filerrrrpz"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at zChat templates should be in a 'chat_template.json' file but found key='chat_template' in the processor's config. Make sure to move your template to its own file.rcustom_pipelines)rr rrrDrrisdirrrisfiler!rrrEnvironmentError ExceptionrreadrloadsJSONDecodeError warning_oncerr)clsrrrrrrrrrr from_pipelinerris_localprocessor_filechat_template_fileresolved_processor_fileresolved_chat_template_filerpreadertextrs rHget_processor_dictz!ProcessorMixin.get_processor_dict2sI$JJ{D11 $4e<< **%6==**Y-- 7D))!::&8%@@::j$//JJ{B//  #3T::  **\599#.?SS  $+8J' (    $%5 $ KKE F F F# (+,I(J(J%7==!>?? 7==6 7 7 cW\\*GXXN!#.KMa!b!b  7>>7 8 8: &C #*. 'HH 8 9 95 :N&23P&Q&Q #*. ' '+N!3 - *51"'#1#$3%5)%':? + + +'$/:1&'#1#$3%5)%':? / / /++$      &E1NEE9VEE0>EEE  & 213III %V{{}} % % % % % % % % % % % % % % % Jt,,_=M&3F? # # *v:  -sWEEE %{{}} % % % % % % % % % % % % % % %!Z--NN#   "i5Liii    p KKO6MOO P P P P KKnnnnUlnn o o o n , ,1P1\   ^     ^++-G":.0M..z*"^335W"#568U6612v%%sN?6H663I)J##J'*J'L'&L; L'L  L'L L''#M rc 0|}|dd}|dd}d|vr|d=d|vr|d=|||j}||i|}|t |d|t |D]6}t||r$t ||||7||t d||r||fS|S) a Instantiates a type of [`~processing_utils.ProcessingMixin`] from a Python dictionary of parameters. Args: processor_dict (`Dict[str, Any]`): Dictionary that will be used to instantiate the processor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the [`~processing_utils.ProcessingMixin.to_dict`] method. kwargs (`Dict[str, Any]`): Additional parameters from which to initialize the processor object. Returns: [`~processing_utils.ProcessingMixin`]: The processor object instantiated from those parameters. return_unused_kwargsFrpNrr)processor_configrrz Processor ) rrvalidate_init_kwargsrrrrrrupdaterr) rrrrrrp unused_kwargsrrs rHfrom_args_and_dictz!ProcessorMixin.from_args_and_dictsD"(,,..%zz*@%HH ?D99   . .01  ' 'z*00._b_o0pp C000  $ I > > >v{{}}%% 9 9Cy#&& 9 3 3888 m$$$ ,,,---  f$ $ rGModelProcessorKwargstokenizer_init_kwargsc iiiiid}iiiiidt}D]}|j|i|<|j|jD]C}||vr=t |j|rt|j|n||}|||<D| t|t|z } |D]}|j|jD]}||vr=|| |d} | dkr|| vrtd|d|dn||vr||d} nd} | dkr | |||<| |tfd|Dr]|D]G\}} |vr>| D])\} } | |vr | ||| <| | *Hn|D]}||vr|||d|<|D]#}|| |d$|S)a Method to merge dictionaries of kwargs cleanly separated by modality within a Processor instance. The order of operations is as follows: 1) kwargs passed as before have highest priority to preserve BC. ```python high_priority_kwargs = {"crop_size" = {"height": 222, "width": 222}, "padding" = "max_length"} processor(..., **high_priority_kwargs) ``` 2) kwargs passed as modality-specific kwargs have second priority. This is the recommended API. ```python processor(..., text_kwargs={"padding": "max_length"}, images_kwargs={"crop_size": {"height": 222, "width": 222}}}) ``` 3) kwargs passed during instantiation of a modality processor have fourth priority. ```python tokenizer = tokenizer_class(..., {"padding": "max_length"}) image_processor = image_processor_class(...) processor(tokenizer, image_processor) # will pass max_length unless overriden by kwargs at call ``` 4) defaults kwargs specified at processor level have lowest priority. ```python class MyProcessingKwargs(ProcessingKwargs, CommonKwargs, TextKwargs, ImagesKwargs, total=False): _defaults = { "text_kwargs": { "padding": "max_length", "max_length": 64, }, } ``` Args: ModelProcessorKwargs (`ProcessingKwargs`): Typed dictionary of kwargs specifically required by the model passed. tokenizer_init_kwargs (`Dict`, *optional*): Dictionary of kwargs the tokenizer was instantiated with and need to take precedence over defaults. Returns: output_kwargs (`Dict`): Dictionary of per-modality kwargs to be passed to each modality-specific processor. )rhrirkrjrg __empty__zKeyword argument z+ was passed two times: in a dictionary for z and as a **kwarg.c3 K|]}|vV dSrxrF)r}rdefault_kwargss rHrz/ProcessorMixin._merge_kwargs..bs(77sn$777777rGrg)r _defaultsrrrBrrrorzr"rraddanyr)rr%r&r output_kwargs used_keysmodality modality_keyvaluenon_modality_kwargs kwarg_valuesubdictsubkeysubvaluerr*s @rH _merge_kwargszProcessorMixin._merge_kwargssJ^      EE ' C CH';'E'I'I(TV'W'W'\'\'^'^N8 $ 4 DX N ^ c c e e C C #888#4><@@A ===2<@ >CN8,\: C ^,,,"&kkC ,>,>>% 0 0H 4 DX N ^ c c e e 0 0 v%%"("2"6"6|["Q"QK"k11lFY6Y6Y(P PP3;PPP"V++#)**\;"G"GKK"-K+--FM(3F;%MM&111  2 F Fi'':@+M/237& K KH ( # * *=+I J J J JrGmainrrrrrc ||d<||d<||d<||d<|dd}|-tjdt|t d|}|||d <|j|fi|} |j|fi|\} }|j| | fi|S) a[ Instantiate a processor associated with a pretrained model. This class method is simply calling the feature extractor [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`], image processor [`~image_processing_utils.ImageProcessingMixin`] and the tokenizer [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] methods. Please refer to the docstrings of the methods above for more information. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on huggingface.co. - a path to a *directory* containing a feature extractor file saved using the [`~SequenceFeatureExtractor.save_pretrained`] method, e.g., `./my_model_directory/`. - a path or url to a saved feature extractor JSON *file*, e.g., `./my_model_directory/preprocessor_config.json`. **kwargs Additional keyword arguments passed along to both [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`]. rrrrrNrrr)rrrrr_get_arguments_from_pretrainedrr$) rrrrrrrrrrrs rHfrom_pretrainedzProcessorMixin.from_pretrainedusN({#1 %5!"%z$4d;;  % ME     l#E  #F7O1s12OZZSYZZ!7!78U!`!`Y_!`!`%s%dNEEfEEErG AutoProcessorct|ts|j}ddlmcm}t ||st|d||_dS)a Register this class with a given auto class. This should only be used for custom feature extractors as the ones in the library are already mapped with `AutoProcessor`. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoProcessor"`): The auto class to register this new feature extractor with. rNz is not a valid auto class.) rrDr>transformers.models.automodelsautorrr)r auto_class auto_modules rHregister_for_auto_classz&ProcessorMixin.register_for_auto_classsn *c** -#,J666666666{J// I GGGHH H$rGc pg}|jD]}t||d}t|trKtd|D}|dd}|r|d |d}n|d}ntt |}||j|fi||S)Nrvc3HK|]}|tt|ndVdSrxryr|s rHrz@ProcessorMixin._get_arguments_from_pretrained..s8oo_`1=(;Q ? ? ?VZoooooorGuse_fastTr r)rrzrrrr{rr<) rrrrrrclassesrGattribute_classs rHr;z-ProcessorMixin._get_arguments_from_pretraineds!n b bN &?&?&?@@J*e,, Koodnooooo!::j$771 6&-ajOO&-ajOO")*=z"J"J KK778U``Y_`` a a a a rGcZt||jd}t|ddS)Nrmodel_input_names)rzr)rfirst_attributes rHrKz ProcessorMixin.model_input_namess+!$(:;;(;TBBBrGc}i}t|t|z }|rAd|}td|dfd|D}|S)NruzISome kwargs in processor config are unused and will not have any effect: z. c"i|] }|| SrFrF)r}rr s rHrz7ProcessorMixin.validate_init_kwargs..s!IIIQ 0 3IIIrG)rrrrwarning)r rrkwargs_from_configr# unused_keysunused_key_strs` rHr!z#ProcessorMixin.validate_init_kwargss-2244 ,--L0A0AA  J!YY{33N NNn\jnnn   JIII[IIIMrGc pt|rtjdt|t|jkrPt dt|jdd|jdt|ddt ||jDS)a Matches optional positional arguments to their corresponding names in `optional_call_args` in the processor class in the order they are passed to the processor call. Note that this should only be used in the `__call__` method of the processors with special arguments. Special arguments are arguments that aren't `text`, `images`, `audio`, nor `videos` but also aren't passed to the tokenizer, image processor, etc. Examples of such processors are: - `CLIPSegProcessor` - `LayoutLMv2Processor` - `OwlViTProcessor` Also note that passing by position to the processor call is now deprecated and will be disallowed in future versions. We only have this for backward compatibility. Example: Suppose that the processor class has `optional_call_args = ["arg_name_1", "arg_name_2"]`. And we define the call method as: ```python def __call__( self, text: str, images: Optional[ImageInput] = None, *arg, audio=None, videos=None, ) ``` Then, if we call the processor as: ```python images = [...] processor("What is common in these images?", images, arg_value_1, arg_value_2) ``` Then, this method will return: ```python { "arg_name_1": arg_value_1, "arg_name_2": arg_value_2, } ``` which we could then pass as kwargs to `self._merge_kwargs` zPassing positional arguments to the processor call is now deprecated and will be disallowed in v4.47. Please pass all arguments as keyword arguments.zExpected *at most* zK optional positional arguments in processor callwhich will be matched with  z+ in the order they are passed.However, got z positional arguments instead.Please pass all arguments as keyword arguments instead (e.g. `processor(arg_name_1=..., arg_name_2=...))`.ci|]\}}|| SrFrF)r} arg_valuearg_names rHrzJProcessorMixin.prepare_and_validate_optional_call_args..+sbbb(; 8)bbbrG)rrrrqrrr)rrs rH'prepare_and_validate_optional_call_argsz6ProcessorMixin.prepare_and_validate_optional_call_argssX t99  MB    t99s4233 3 3}c$*A&B&B}}.1hht7N.O.O}} #D }}}  cbs4I`?a?abbbbrG conversationtokenizec n||j|j}ntd|jj|f||d|S)a Similar to the `apply_chat_template` method on tokenizers, this method applies a Jinja template to input conversations to turn them into a single tokenizable string. Args: conversation (`List[Dict, str, str]`): The conversation to format. chat_template (`Optional[str]`, *optional*): The Jinja template to use for formatting the conversation. If not provided, the tokenizer's chat template is used. tokenize (`bool`, *optional*, defaults to `False`): Whether to tokenize the output or not. **kwargs: Additional keyword arguments NzNo chat template is set for this processor. Please either set the `chat_template` attribute, or provide a chat template as an argument. See https://huggingface.co/docs/transformers/main/en/chat_templating for more information.)rprZ)rprroapply_chat_template)rrYrprZrs rHr\z"ProcessorMixin.apply_chat_template-sj.  !- $ 2  m 2t~1  (5  LR   rG)Frx)NFFNr9)r=)NF)*r>r?r@rArrrqrrDrBfeature_extractor_classtokenizer_classrrrrrrrrr rPathLikerrrCr classmethodrrr$rfrr8r<rDr;propertyrK staticmethodr!rXr\rFrGrHrmrmTs&{3J*+$&S &&&"OK L$s)   $/$/$/L&c3h&&&&P G G G G G 05bk1A+B 0 0 0 0[[[ d'd'4d'd'd'd'LV&,1#r{2B,CV& tCH~tCH~- .V&V&V&[V&p+d38n+++[+`15zz.z (~z c4i zzzzx8<$!&,0=F=F',S"+-='>=FE#r{"234=F =F  =F c4i() =F=F=F=F[=F~%%%[%2["CCXC  \ 8c8c8cz(, " " Dc3h01"  }"  " " " " " " " rGrmc dtfd  fdfdd}||}|}||}|}|r|r||fS||s||s|r |rtd||fStd) a For backward compatibility: reverse the order of `images` and `text` inputs if they are swapped. This method should only be called for processors where `images` and `text` have been swapped for uniformization purposes. Note that this method assumes that two `None` inputs are valid inputs. If this is not the case, it should be handled in the processor's `__call__` method before calling this method. rcVt|to|dS)Nhttp)rrD startswith)vals rHis_urlz1_validate_images_text_input_order..is_urlZs##s##>v(>(>>rGct|ttfr|D]}|sdSnt|s |sdSdS)NFT)rlistrr)imgsimg$_is_valid_images_input_for_processorrhs rHrmzO_validate_images_text_input_order.._is_valid_images_input_for_processor]sw dT5M * *  ! !;;C@@! 55! !!&& &&,, 5trGct|trdSt|ttfr&t |dkrdS|D]}|cSdS)NTrF)rrDrjrr)tt_s"_is_valid_text_input_for_processors rHrqzM_validate_images_text_input_order.._is_valid_text_input_for_processorhso a   ?4 D%= ) ) ?1vv{{u ? ?99#>>>>>urGc ||p|duSrxrF)input validators rH _is_validz4_validate_images_text_input_order.._is_validusy05D=0rGNzYou may have used the wrong order for inputs. `images` should be passed before `text`. The `images` and `text` inputs will be swapped. This behavior will be deprecated in transformers v4.47.zGInvalid input type. Check that `images` and/or `text` are valid inputs.)rCrrr) imagesrruimages_is_validimages_is_text text_is_validtext_is_imagesrmrqrhs @@@rH!_validate_images_text_input_orderr{Rs?t????           111 i(LMMO77??NId$FGGM99$??N=t| >t||Tb|gu| v   V| ^ _ __rGrr=zprocessor files)object object_class object_files)BrArrrrsystypingrpathlibrrrrrrr r numpynptyping_extensionsdynamic_module_utilsr image_utilsr rrrtokenization_utils_baserrrrrutilsrrrrrrrrrrr r!r" get_loggerr>r__file__parentr{r version_infoUnpackr+rKr]r_rcrfrmr{rformatrFrGrHrs&  EEEEEEEEEEEEEEEEEE444444NNNNNNNNNN0//////"  H % %10h1FGG/40 w ]FF  %F7 7 7 7 7 %7 7 7 7 t4>4>4>4>4>9E4>4>4>4>n->->->->->9E->->->->`$*$*$*$*$*)5$*$*$*$*N555559E555544444z<{L`e4444n{  {  {  {  {  ^{  {  {  |7`7`7`t'Y~'ABB%1)7)C)K)R)RGX*S**N&&&21rG