g'ddlmZmZddlmZmZmZmZmZm Z ddl m Z ddl m Z erddlmZe rddlZGdd eZdS) )ABCabstractmethod) TYPE_CHECKINGAnyDictListOptionalUnion)is_torch_available)QuantizationConfigMixin)PreTrainedModelNc eZdZdZdZdZdZdefdZd(d Z d e e e e fde e e e ffd Zd(d Zd ee de dee fdZdee dee dee fdZddde e dffdZde e eee ffde e eee fffdZddddde de e e fdef dZd)dZdZd*d Zd*d!Zd"Zd#Zed$Z ed%Z!ed+d&Z"e#ed'Z$dS), HfQuantizera Abstract class of the HuggingFace quantizer. Supports for now quantizing HF transformers models for inference and/or quantization. This class is used only for transformers.PreTrainedModel.from_pretrained and cannot be easily used outside the scope of that method yet. Attributes quantization_config (`transformers.utils.quantization_config.QuantizationConfigMixin`): The quantization config that defines the quantization parameters of your model that you want to quantize. modules_to_not_convert (`List[str]`, *optional*): The list of module names to not convert when quantizing the model. required_packages (`List[str]`, *optional*): The list of required pip packages to install prior to using the quantizer requires_calibration (`bool`): Whether the quantization method requires to calibrate the model before using it. requires_parameters_quantization (`bool`): Whether the quantization method requires to create a new Parameter. For example, for bitsandbytes, it is required to create a new xxxParameter in order to properly quantize the model. FNquantization_configc ||_|dg|_|dd|_|js|jrt d|jddSdS)Nmodules_to_not_convert pre_quantizedTzThe quantization method z does require the model to be pre-quantized. You explicitly passed `pre_quantized=False` meaning your model weights are not quantized. Make sure to pass `pre_quantized=True` while knowing what you are doing.)rpoprrrequires_calibration ValueError quant_method)selfrkwargss X/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/quantizers/base.py__init__zHfQuantizer.__init__4s#6 '-jj1I2&N&N##ZZ>>! d&? O+>+KOOO      torch_dtype torch.dtypereturnc|S)aU Some quantization methods require to explicitly set the dtype of the model to a target dtype. You need to override this method in case you want to make sure that behavior is preserved Args: torch_dtype (`torch.dtype`): The input dtype that is passed in `from_pretrained` rrs rupdate_torch_dtypezHfQuantizer.update_torch_dtypeB r device_mapc|S)a Override this method if you want to pass a override the existing device map with a new one. E.g. for bitsandbytes, since `accelerate` is a hard requirement, if no device_map is passed, the device_map is set to `"auto"`` Args: device_map (`Union[dict, str]`, *optional*): The device_map that is passed through the `from_pretrained` method. r")rr&s rupdate_device_mapzHfQuantizer.update_device_mapNs rc|S)a Override this method if you want to adjust the `target_dtype` variable used in `from_pretrained` to compute the device_map in case the device_map is a `str`. E.g. for bitsandbytes we force-set `target_dtype` to `torch.int8` and for 4-bit we pass a custom enum `accelerate.CustomDtype.int4`. Args: torch_dtype (`torch.dtype`, *optional*): The torch_dtype that is used to compute the device_map. r"r#s radjust_target_dtypezHfQuantizer.adjust_target_dtypeZr%r missing_keysprefixc|S)z Override this method if you want to adjust the `missing_keys`. Args: missing_keys (`List[str]`, *optional*): The list of missing keys in the checkpoint compared to the state dict of the model r")rmodelr+r,s rupdate_missing_keyszHfQuantizer.update_missing_keysfs r expected_keys loaded_keysc|S)aV Override this method if you want to adjust the `update_expected_keys`. Args: expected_keys (`List[str]`, *optional*): The list of the expected keys in the initialized model. loaded_keys (`List[str]`, *optional*): The list of the loaded keys in the checkpoint. r")rr.r0r1s rupdate_expected_keysz HfQuantizer.update_expected_keysps rcHfd|DS)a returns dtypes for modules that are not quantized - used for the computation of the device_map in case one passes a str as a device_map. The method will use the `modules_to_not_convert` that is modified in `_process_model_before_weight_loading`. Args: model (`~transformers.PreTrainedModel`): The model to quantize torch_dtype (`torch.dtype`): The dtype passed in `from_pretrained` method. c^i|](\}tfdjD%)S)c3 K|]}|vV dSNr").0mnames r zCHfQuantizer.get_special_dtypes_update...s'BB19BBBBBBr)anyr)r8_r:rrs @r z9HfQuantizer.get_special_dtypes_update..sV   aBBBBd&ABBBBB +   r)named_parameters)rr.rs` `rget_special_dtypes_updatez%HfQuantizer.get_special_dtypes_update|s@      1133    r max_memoryc|S)zaadjust max_memory argument for infer_auto_device_map() if extra memory is needed for quantizationr")rrAs radjust_max_memoryzHfQuantizer.adjust_max_memorysrr.r param_valuez torch.Tensor param_name state_dictc dS)a checks if a loaded state_dict component is part of quantized param + some validation; only defined if requires_parameters_quantization == True for quantization methods that require to create a new parameters for quantization. Fr")rr.rDrErFrs rcheck_quantized_paramz!HfQuantizer.check_quantized_params urtorch.nn.ParametercN|jstd|jjddS)z takes needed components from state_dict and creates quantized param; only applicable if requires_parameters_quantization == True zG`.create_quantized_param()` method is not supported by quantizer class .N) requires_parameters_quantizationAttributeError __class____name__rargsrs rcreate_quantized_paramz"HfQuantizer.create_quantized_params@ 4  tZ^ZhZqttt   rcdS)a& This method is used to potentially check for potential conflicts with arguments that are passed in `from_pretrained`. You need to define it for all future quantizers that are integrated with transformers. If no explicit check are needed, simply return nothing. Nr"rPs rvalidate_environmentz HfQuantizer.validate_environments rc Nd|_|jj|_|j|fi|S)aQ Setting model attributes and/or converting model before weights loading. At this point the model should be initialized on the meta device so you can freely manipulate the skeleton of the model in order to replace modules in-place. Make sure to override the abstract method `_process_model_before_weight_loading`. Args: model (`~transformers.PreTrainedModel`): The model to quantize kwargs (`dict`, *optional*): The keyword arguments that are passed along `_process_model_before_weight_loading`. T) is_quantizedrrquantization_method$_process_model_before_weight_loadingrr.rs rpreprocess_modelzHfQuantizer.preprocess_models5"$($<$I!8t8II&IIIrc |j|fi|S)a Post-process the model post weights loading. Make sure to override the abstract method `_process_model_after_weight_loading`. Args: model (`~transformers.PreTrainedModel`): The model to quantize kwargs (`dict`, *optional*): The keyword arguments that are passed along `_process_model_after_weight_loading`. )#_process_model_after_weight_loadingrYs rpostprocess_modelzHfQuantizer.postprocess_models8t7HHHHHrc4||}|`|S)z Potentially dequantize the model to retrive the original model, with some loss in accuracy / performance. Note not all quantization schemes support this. ) _dequantize hf_quantizerrr.s r dequantizezHfQuantizer.dequantizes$   ''   rc:t|jjd)NzH has no implementation of `dequantize`, please raise an issue on GitHub.)NotImplementedErrorrrras rr_zHfQuantizer._dequantizes&!'4 ~ ~ ~   rc dSr7r"rYs rrXz0HfQuantizer._process_model_before_weight_loadingsEHSrc dSr7r"rYs rr\z/HfQuantizer._process_model_after_weight_loadingsDGCrcdSr7r")rsafe_serializations ris_serializablezHfQuantizer.is_serializables8;rcdSr7r")rs r is_trainablezHfQuantizer.is_trainablesCr)rrr r)r rI)r.rr7)%rO __module__ __qualname____doc__rrequired_packagesrLr rr$r rstrrr(r*rr/r3r@r intrCboolrHrRrTrZr]rbr_rrXr\ripropertyrkr"rrrrs&!',$ ,C         HT#s(^,D RVWZ\_W_R`Ia        tCy#RVWZR[ c QUVYQZ _cdg_h     M dSVXeSeNf    &DeCHo1E,F4PSUZ[^`c[cUdPdKe   $   cN      JJJJ I I I I      HH^HGG^G;;;^; ^Xrr)abcrrtypingrrrrr r utilsr utils.quantization_configr modeling_utilsrtorchrr"rrrzs$#######BBBBBBBBBBBBBBBB&&&&&&??????1000000LLLQ Q Q Q Q #Q Q Q Q Q r