gcvdZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%ere!rddl&Z&e$j'e(Z)e dZ*Gd d eZ+Gd d eZ,ee,j-e,_-e,j-j.e,j-j.d dde,j-_dSdS)zH Feature extraction saving/loading class for common feature extractors. N)UserDict) TYPE_CHECKINGAnyDictOptionalTupleUnion)custom_object_save)FEATURE_EXTRACTOR_NAMEPushToHubMixin TensorTypeadd_model_info_to_auto_map"add_model_info_to_custom_pipelines cached_file copy_func download_urlis_flax_available is_jax_tensoris_numpy_arrayis_offline_mode is_remote_urlis_tf_availableis_torch_availableis_torch_deviceis_torch_dtypeloggingrequires_backendsSequenceFeatureExtractorceZdZdZddeeeefdedee fffd Z dedeefdZ defd Z d Z d Zd Zd ZdZddeeee ffdZddeeee ffdZddZxZS) BatchFeatureaU Holds the output of the [`~SequenceFeatureExtractor.pad`] and feature extractor specific `__call__` methods. This class is derived from a python dictionary and can be used as a dictionary. Args: data (`dict`, *optional*): Dictionary of lists/arrays/tensors returned by the __call__/pad methods ('input_values', 'attention_mask', etc.). tensor_type (`Union[None, str, TensorType]`, *optional*): You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at initialization. Ndata tensor_typecvt|||dS)N)r#)super__init__convert_to_tensors)selfr"r# __class__s a/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/feature_extraction_utils.pyr&zBatchFeature.__init__Ms8  K88888itemreturncdt|tr |j|Std)z If the key is a string, returns the value of the dict associated to `key` ('input_values', 'attention_mask', etc.). zRIndexing with integers is not available when using Python based feature extractors) isinstancestrr"KeyErrorr(r,s r* __getitem__zBatchFeature.__getitem__Qs2 dC  q9T? "opp pr+cH |j|S#t$rtwxYwN)r"r1AttributeErrorr2s r* __getattr__zBatchFeature.__getattr__[s4 !9T? " ! ! !  !s !cd|jiSNr"r"r(s r* __getstate__zBatchFeature.__getstate__as ""r+c,d|vr|d|_dSdSr9r:)r(states r* __setstate__zBatchFeature.__setstate__ds! U??f DIII ?r+c4|jSr5)r"keysr;s r*rAzBatchFeature.keysisy~~r+c4|jSr5)r"valuesr;s r*rCzBatchFeature.valuesmsy!!!r+c4|jSr5)r"itemsr;s r*rEzBatchFeature.itemsqsy   r+c|dSt|tst|}|tjkr0tst dddl}|j|j}n|tjkr.tst dddl fdj}nO|tj kr2tst dddl m}|jt }n dfd t"}|fS) NNNzSUnable to convert output to TensorFlow tensors format, TensorFlow is not installed.rzMUnable to convert output to PyTorch tensors format, PyTorch is not installed.c*t|ttfrt|dkrt|dtjrt j|}nut|dttfrSt|ddkr:t|ddtjrt j|}t|tjrj|Sj|S)Nr) r/listtuplelennpndarrayarray from_numpytensor)valuetorchs r* as_tensorz5BatchFeature._get_is_as_tensor_fns..as_tensorsedE]330E Q!%(BJ770 ""58dE];;0aMMA--&uQx{BJ??.!#eRZ00/+5+E222'5<...r+zEUnable to convert output to JAX tensors format, JAX is not installed.cVt|ttfrwt|dtttjfrJd|D}t t |dkr|d|Dt}tj||S)Nrc,g|]}t|S)rK.0vals r* zIBatchFeature._get_is_as_tensor_fns..as_tensor..s!.as_tensor..s *L*L*Ls2:c??*L*L*Lr+)dtype) r/rIrJrLrMrKsetobjectr\)rQr] value_lensrSs r*rSz5BatchFeature._get_is_as_tensor_fns..as_tensorsedE]33\ 58dTY[][eMf8g8g\!c$ii!mmq'Cc"" rC%% r)=)= rCQTAUAU r!!pcRUhh!p!p!pqqqJJLL  DAq!U\** /Fu/Fq/I/I "adD3F33 Au|,, 1Cdd&d11    r+rGr5)r-r!)__name__ __module__ __qualname____doc__rrr0rr rr&r3r7r<r?rArCrErlr'rw __classcell__)r)s@r*r!r!>sy  99Xd38n595QUWZ\fQfKg999999qqc qqqq!!!!! ###&&&    """!!!9$9$%Z:P1Q9$9$9$9$vhuS*_7M.N>********r+r!c,eZdZdZdZdZdefdZe dde ee j fd e e ee j fd e d e d e e ee fd ef dZd de ee j fde fdZede ee j fdeeeefeeefffdZedeeefdefdZdeeeffdZede ee j fdefdZdefdZde ee j ffdZdZed!dZdS)"FeatureExtractionMixinz This is a feature extraction mixin used to provide saving/loading functionality for sequential and image feature extractors. Nc |dd|_|D]N\}} t|||#t$r*}t d|d|d||d}~wwxYwdS)z'Set elements of `kwargs` as attributes.processor_classNz Can't set z with value z for )pop_processor_classrEsetattrr6loggererror)r(ryrprQerrs r*r&zFeatureExtractionMixin.__init__s!' + To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. return_unused_kwargs (`bool`, *optional*, defaults to `False`): If `False`, then this function returns just the final feature extractor object. If `True`, then this functions returns a `Tuple(feature_extractor, unused_kwargs)` where *unused_kwargs* is a dictionary consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of `kwargs` which has not been used to update `feature_extractor` and is otherwise ignored. kwargs (`Dict[str, Any]`, *optional*): The values in kwargs of any keys which are feature extractor attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are *not* feature extractor attributes is controlled by the `return_unused_kwargs` keyword parameter. Returns: A feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`]. Examples: ```python # We can't instantiate directly the base class *FeatureExtractionMixin* nor *SequenceFeatureExtractor* so let's show the examples on a # derived class: *Wav2Vec2FeatureExtractor* feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h" ) # Download feature_extraction_config from huggingface.co and cache. feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "./test/saved_model/" ) # E.g. feature_extractor (or model) was saved using *save_pretrained('./test/saved_model/')* feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("./test/saved_model/preprocessor_config.json") feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h", return_attention_mask=False, foo=False ) assert feature_extractor.return_attention_mask is False feature_extractor, unused_kwargs = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h", return_attention_mask=False, foo=False, return_unused_kwargs=True ) assert feature_extractor.return_attention_mask is False assert unused_kwargs == {"foo": False} ```rrrruse_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.r)rwarningswarn FutureWarningroget_feature_extractor_dict from_dict) clsrrrrrrryrfeature_extractor_dicts r*from_pretrainedz&FeatureExtractionMixin.from_pretrainedsp({#1 %5!"%z$4d;;  % ME     l#E  #F7O)G)GHe)p)pio)p)p&s}3>>v>>>r+save_directory push_to_hubc |dd}|Dtjdt|ddt d||d<t j|rtd|dt j |d |rw|d d}|d | t jj d }|j |fi|}||}|jt!||| t j|t$}||t(d||r-||||||d|gS)az Save a feature_extractor object to the directory `save_directory`, so that it can be re-loaded using the [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] class method. Args: save_directory (`str` or `os.PathLike`): Directory where the feature extractor JSON file will be saved (will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. rNrrrzProvided path (z#) should be a directory, not a fileT)exist_okcommit_messagerepo_id)configzFeature extractor saved in )rr)rrrrrsroospathisfileAssertionErrormakedirssplitsep _create_repo_get_files_timestamps _auto_classr joinr to_json_filerinfo_upload_modified_files) r(rrryrrrfiles_timestampsoutput_feature_extractor_files r*save_pretrainedz&FeatureExtractionMixin.save_pretraineds $4d;;  % ME   zz'4((4 l-F7O 7>>. ) ) h !f>!f!f!fgg g NT2222  J#ZZ(8$??NjjN,@,@,M,Mb,QRRG'd'::6::G#99.II    ' t^D A A A A)+ ^E[(\(\% 7888 Q2OQQRRR    ' ' -jj)) (   ...r+r-c |dd}|dd}|dd}|dd}|dd}|dd}|d d} |d d} |d d} | -tjd t|t d | }|dd} |dd} d| d}| | |d<t r| st dd} t|}tj |}tj |r%tj |t}tj |r|}d}nzt|r|}t!|}nYt} t#||||||| ||||  }n6#t$$rt&$rt%d|d|dtdwxYw t)|dd5}|}dddn #1swxYwYt-j|}n&#t,j$rt%d|dwxYw|rt d|n t d|d ||s:d!|vrt3|d!||d!<d"|vrt5|d"||d"<||fS)#a6 From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`] using `from_dict`. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): The identifier of the pre-trained checkpoint from which we want the dictionary of parameters. Returns: `Tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the feature extractor object. rNrFresume_downloadproxies subfolderrrrrrr_from_pipeline _from_autofeature extractor) file_typefrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=TrueT) rrrrrrr user_agentrz"Can't load feature extractor for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filerutf-8encodingz"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at auto_mapcustom_pipelines)rrrrrorrrr0rrisdirrr rrrrEnvironmentError ExceptionopenreadjsonloadsJSONDecodeErrorrr)rrryrrrrrrrrr from_pipelinerris_localfeature_extractor_fileresolved_feature_extractor_filereadertextrs r*rz1FeatureExtractionMixin.get_feature_extractor_dictsiJJ{D11 $4e<< **%6==**Y--JJ{D11  7D))$4d;;!::&8%@@::j$//  % ME     l#E #3T::  **\599#6?[[  $+8J' (    $%5 $ KKE F F F# (+,I(J(J%7==!>?? 7==6 7 7 i%'W\\2OQg%h%h " 7>>7 8 8" .K +HH 8 9 9 %B ".:;X.Y.Y + +%; " 2=1*'#1#$3%5')% 3 3 3//$      &M9VMM9VMM0FMMM  5sWMMM %QW{{}} % % % % % % % % % % % % % % %%)Z%5%5 " "#   "q5Tqqq     KKW6UWW X X X X KKv.DvvUtvv    3335O*:68U66&z2"%;;;=_*+=>@]>>&'9:&v--s<H443I'+J>=J J>J""J>%J"&J>>#K!rc 2|dd}g}|D]#\}}||vr|||<||$|D]}||d|di|}td||r||fS|S)a  Instantiates a type of [`~feature_extraction_utils.FeatureExtractionMixin`] from a Python dictionary of parameters. Args: feature_extractor_dict (`Dict[str, Any]`): Dictionary that will be used to instantiate the feature extractor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the [`~feature_extraction_utils.FeatureExtractionMixin.to_dict`] method. kwargs (`Dict[str, Any]`): Additional parameters from which to initialize the feature extractor object. Returns: [`~feature_extraction_utils.FeatureExtractionMixin`]: The feature extractor object instantiated from those parameters. return_unused_kwargsFNzFeature extractor rV)rrEappendrr)rrryr to_removerprQfeature_extractors r*rz FeatureExtractionMixin.from_dict5s$ &zz*@%HH  ,,.. & &JC,,,.3&s+  %%% " "C JJsD ! ! ! !C99"899 <):<<===  %$f, ,$ $r+crtj|j}|jj|d<d|vr|d=d|vr|d=|S)z Serializes this instance to a Python dictionary. Returns: `Dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance. feature_extractor_type mel_filterswindow)copydeepcopy__dict__r)r~)r(outputs r*to_dictzFeatureExtractionMixin.to_dictZsP t}--+/>+B'( F " "}% v  x  r+ json_filect|dd5}|}dddn #1swxYwYtj|}|di|S)a Instantiates a feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`] from the path to a JSON file of parameters. Args: json_file (`str` or `os.PathLike`): Path to the JSON file containing the parameters. Returns: A feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`]: The feature_extractor object instantiated from that JSON file. rrrNrV)rrrr)rrrrrs r*from_json_filez%FeatureExtractionMixin.from_json_filegs)S7 3 3 3 !v;;==D ! ! ! ! ! ! ! ! ! ! ! ! ! ! !!%D!1!1s,,+,,,s 488c.|}|D]6\}}t|tjr|||<7|dd}|||d<tj|dddzS)z Serializes this instance to a JSON string. Returns: `str`: String containing all the attributes that make up this feature_extractor instance in JSON format. rNrT)indent sort_keys ) rrEr/rLrMtolistrrdumps)r( dictionaryrprQrs r*to_json_stringz%FeatureExtractionMixin.to_json_stringzs\\^^ $**,, 1 1JC%,, 1"',,.. 3&>>* This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoFeatureExtractor"`): The auto class to register this new feature extractor with. rNz is not a valid auto class.) r/r0r~transformers.models.automodelsautohasattrror)r auto_class auto_modules r*register_for_auto_classz.FeatureExtractionMixin.register_for_auto_classsn *c** -#,J666666666{J// I GGGHH H$r+)NFFNr)F)r)r~rrrrr&r0r classmethodr rPathLikerboolrrrrrrPreTrainedFeatureExtractorrrrrrrrrVr+r*rrs K   0C00008<$!&,0m?m?',S"+-='>m?E#r{"234m? m?  m? c4i() m?m?m?m?[m?^;/;/eC4D.E;/TX;/;/;/;/zr.,1#r{2B,Cr. tCH~tCH~- .r.r.r.[r.h"%tCH~"%Lf"%"%"%["%H c3h    -uS"+-='>-C]---[-$GGGGG* 05bk1A+B 0 0 0 0DDD%%%[%%%r+rrrzfeature extractor file)r_ object_class object_files)/rrrrr collectionsrtypingrrrrrr rirLdynamic_module_utilsr utilsr r rrrrrrrrrrrrrrrrrrR get_loggerr~rrr!rrformatrVr+r*rsu  CCCCCCCCCCCCCCCC444444.   H % %"#=>zzzzz8zzzzz%z%z%z%z%^z%z%z%z &/Y/E/Q%R%R"%-91G1S1[1b1b"1GVn2c22&...:9r+