g>ddlZddlmZddlmZmZmZmZmZm Z ddl m Z ddl m Z ddlmZerdd lmZdd lmZmZmZmZmZmZer ddlZdd lmZejeZGd d e ZdS)N)cached_property) TYPE_CHECKINGAnyDictListOptionalUnion)version) HfQuantizer)get_module_from_name)PreTrainedModel)ACCELERATE_MIN_VERSIONis_accelerate_availableis_bitsandbytes_availableis_torch_availableis_torch_xpu_availablelogging)Conv1DceZdZdZdZdZdZddgZfdZdZ d$d Z d d ddde de e e fd ef dZ d%d d ddde ddde e e fdeee f dZde e eee ffd e e eee fffdZd&dZdZgfd d dee fdZd'dZd%d Zed efd!Zed efd"Zd#ZxZ S)(Bnb4BitHfQuantizera 4-bit quantization from bitsandbytes.py quantization method: before loading: converts transformer layers into Linear4bit during loading: load 16bit weight and pass to the layer object after: quantizes individual weights in Linear4bit into 4bit at the first .cuda() call saving: from state dict, as usual; saves weights and `quant_state` components loading: need to locate `quant_state` components and pass to Param4bit constructor TF bitsandbytes acceleratec ztj|fi||jj|jj|_dSdSN)super__init__quantization_configllm_int8_skip_modulesmodules_to_not_convert)selfrkwargs __class__s f/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/quantizers/quantizer_bnb_4bit.pyrzBnb4BitHfQuantizer.__init__>sL,77777  # 9 E*.*B*XD ' ' ' F Ecrtstdtdtstdddlm}ddlm}|}|d|d d s|d d rtd |d dttrj j sfdD}tdhkr|rn;d|vsd|vrtdt#jt&jdt#jdkrtddS)NzWUsing `bitsandbytes` 4-bit quantization requires Accelerate: `pip install 'accelerate>=z'`zrUsing `bitsandbytes` 4-bit quantization requires the latest version of bitsandbytes: `pip install -U bitsandbytes`r)!validate_bnb_backend_availability)'is_bitsandbytes_multi_backend_availableT)raise_exceptionfrom_tfF from_flaxzConverting into 4-bit or 8-bit weights from tf/flax weights is currently not supported, please make sure the weights are in PyTorch format. device_mapc4i|]}|jv ||S)r!).0keyr-r"s r% z;Bnb4BitHfQuantizer.validate_environment..`s4***),SPTPkEkEkZ_EkEkEkr&cpudiskaSome modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `llm_int8_enable_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details. rz0.39.0zYou have a version of `bitsandbytes` that is not compatible with 4bit inference and training make sure you have the latest version of `bitsandbytes` installed)r ImportErrorrr integrationsr(utilsr)get ValueError isinstancedictr llm_int8_enable_fp32_cpu_offloadkeyssetvaluesr parse importlibmetadata)r"argsr#r(r)bnb_multibackend_is_enableddevice_map_without_lm_headr-s` @r%validate_environmentz'Bnb4BitHfQuantizer.validate_environmentDs0&(( EkAEEE )** E  EDDDDDCCCCCC&M&M&O&O#))$???? ::i ' ' 6::k5+I+I ;  ZZ d33  ":t,, #,M #*****0:0A0A*** &:$$&&''E7227R24;;====KeKlKlKnKnAnAn ) =+33NCC D Dw}U]G^G^ ^ ^U  _ ^r& target_dtype torch.dtypereturnctjtjdtjdkr7ddlm}|t jkrt d|j Std)Nrz0.19.0r) CustomDtypezXtarget_dtype {target_dtype} is replaced by `CustomDtype.INT4` for 4-bit BnB quantizationaUYou are using `device_map='auto'` on a 4bit loaded version of the model. To automatically compute the appropriate device map, you should upgrade your `accelerate` library,`pip install --upgrade accelerate` or install it from source to support fp4 auto device mapcalculation. You may encounter unexpected behavior, or pass your own device map) r r@rArBaccelerate.utilsrKtorchint8loggerinfoINT4r9)r"rGrKs r%adjust_target_dtypez&Bnb4BitHfQuantizer.adjust_target_dtypeus =+33LAA B BW]S[E\E\ \ \ 4 4 4 4 4 4uz)) vwww# #b r&modelr param_valuez torch.Tensor param_name state_dictc ddl}t||\}}t|j|d|jjrdSt||jjr|dkrdSdS)NrTbiasF)rr r: _parametersr8nn Params4bit Linear4bit) r"rSrTrUrVr#bnbmodule tensor_names r%check_quantized_paramz(Bnb4BitHfQuantizer.check_quantized_params #"""25*EE f(,,[$??AR S S 4  1 2 2 {f7L7L45r&N target_devicez torch.deviceunexpected_keyscddl}t||\}} | |jvrt|d| dt || } | dkr_|| |} n||} t j| | j } | |j| <dSt|j| |jj std| j t j dkr/|dt j dfvr|t| d |d|j r|jstd |d z|vr|d z|vrtd |di} |D],\} }|dz| vr || | <|| |vr|| -i}|jr||d<|jj jd|| d|d|} ne|d} t'|jt*r| j} | j}|jj | fddi||} | |j| <dS)z combines logic from _load_state_dict_into_meta_model and .integrations.bitsandbytes.py::set_module_quantized_tensor_to_device() rNz- does not have a parameter or a buffer named .rX) requires_gradz0this function only loads `Linear4bit components`metaz7 is on the meta device, we need a `value` to put in on zDetected int4 weights but the version of bitsandbytes is not compatible with int4 serialization. Make sure to download the latest `bitsandbytes` version. `pip install --upgrade bitsandbytes`.z.quant_state.bitsandbytes__fp4z.quant_state.bitsandbytes__nf4zSupplied state dict for zT does not contain `bitsandbytes__*` and possibly other `quantized_stats` components.r^F)dataquantized_statsredevicer3rer/)rr rYr9getattrtorMrZ Parameterrer:r[ri pre_quantizedis_serializableitemsremove$is_bnb_supports_quant_storage_modulefrom_prequantized issubclass source_clsrT__dict__)r"rSrTrUrarVrbr]r^r_ old_value new_valuerhkv param_kwargsr#s r%create_quantized_paramz)Bnb4BitHfQuantizer.create_quantized_params #"""25*EE f0 0 0ccU`cccdd dFK00 & "%LL77 'NN=99 **9ID[*\\I.7F { + F&,[936;LMM QOPP P   V 4 4 4 4fel6.B.B%CCC# ttdqtttuu u  , f'  u ==ZOO==ZOO PzPPP!O"((** 2 21#q(()*OA&&2qO7K7K'..q111L8 0)/ X&;); /#$   II$u--I&+V44 (%K 'F)))SS5SFSSVVWdeeI*3;'''r& max_memorycBd|D}|S)Nc i|] \}}||dz S)g?r/)r0r1vals r%r2z8Bnb4BitHfQuantizer.adjust_max_memory..s"III(#sc3:IIIr&)ro)r"r}s r%adjust_max_memoryz$Bnb4BitHfQuantizer.adjust_max_memorys'IIj6F6F6H6HIII r& torch_dtypecX|'td|tj}|S)NaOverriding torch_dtype=%s with `torch_dtype=torch.float16` due to requirements of `bitsandbytes` to enable model loading in 8-bit or 4-bit. Pass your own torch_dtype to specify the dtype of the remaining non-linear layers or pass torch_dtype=torch.float16 to remove this warning.)rOrPrMfloat16)r"rs r%update_torch_dtypez%Bnb4BitHfQuantizer.update_torch_dtypes8   KKE     -Kr&c0|tjr!dtji}n6t r$ddtji}nddi}t d|d|S)Nzxpu:r3z:The device_map was not initialized. Setting device_map to zL. If you want to use the model for inference, please set device_map ='auto' )rMcuda is_availablecurrent_devicerxpurOrP)r"r-s r%update_device_mapz$Bnb4BitHfQuantizer.update_device_maps  z&&(( ) %*";";"="=> ')) ) "E)A)A)C)C"E"EF  %[ KK])3]]]    r&keep_in_fp32_modulesc ddlm}m}|jj}|jj|||_n|jj|_t|jts |jg|_|j |t|trt| dkr\d| D}t|dkr|std|j ||||j|j}|j|j_dS)Nr)get_keys_to_not_convertreplace_with_bnb_linearr c g|] \}}|dv | S))r4r3r/)r0r1values r% zKBnb4BitHfQuantizer._process_model_before_weight_loading../s'```:3uP_G_G_3G_G_G_r&rzIf you want to offload some keys to `cpu` or `disk`, you need to set `llm_int8_enable_fp32_cpu_offload=True`. Note that these modules will not be converted to 8-bit but kept in 32-bit.)r!r)r6rrrr<r r!r:listextendr;lenr=ror9config) r"rSr-rr#rrr< keys_on_cpus r%$_process_model_before_weight_loadingz7Bnb4BitHfQuantizer._process_model_before_weight_loadingsi TSSSSSSS+/+C+d(  # 9 A*A*A%*H*HD ' '*.*B*XD '$5t<< H+/+F*GD ' #**+?@@@ j$ ' '   ' . .{ ; ; ;'' $*E[_[s    ,0+C (((r&c Fd|_||_|SNT)is_loaded_in_4bitrnis_4bit_serializable)r"rSr#s r%#_process_model_after_weight_loadingz6Bnb4BitHfQuantizer._process_model_after_weight_loadingAs#"&%)%9%9%;%;" r&ctjtjdtjdk}|stddSdS)Nrz0.41.3zYou are calling `save_pretrained` to a 4-bit converted model, but your `bitsandbytes` version doesn't support it. If you want to save 4-bit models, make sure to have `bitsandbytes>=0.41.3` installed.FT)r r@rArBrOwarning)r"safe_serialization_is_4bit_serializables r%rnz"Bnb4BitHfQuantizer.is_serializableFse ' i.@.H.H.X.X Y Y]d]jks]t]t t$  NNh   5tr&ctjtjdtjdkS)z determines if the current version of bitsandbytes supports the `module` parameter in `Params4bit.from_prequantized` :return: rz0.43.3)r r@rArBr"s r%rqz7Bnb4BitHfQuantizer.is_bnb_supports_quant_storage_moduleRs5}Y/77GGHHGMZbLcLcccr&cdSrr/rs r% is_trainablezBnb4BitHfQuantizer.is_trainable[str&cBddlm}|||j|j}|S)Nr)dequantize_and_replace)r)r6rr!r)r"rSrs r% _dequantizezBnb4BitHfQuantizer._dequantize_s?999999&& 4.DD\    r&)rGrHrIrHr)rrHrIrH)rSr)!__name__ __module__ __qualname____doc__use_keep_in_fp32_modules requires_parameters_quantizationrequires_calibrationrequired_packagesrrFrRstrrrboolr`rrr|r intrrrrrrnrrqpropertyrr __classcell__)r$s@r%rr-s $'+$ '6YYYYY ///b     $  cN  804W4W4 W4$W4 W4 & W4 cN W4"$s),W4W4W4W4tDeCHo1E,F4PSUZ[^`c[cUdPdKe       (+- 'D'D 'D#3i 'D'D'D'DT    ddddd_ddXr&r) rA functoolsrtypingrrrrrr packagingr baser quantizers_utilsr modeling_utilsrr7rrrrrrrM pytorch_utilsr get_loggerrrOrr/r&r%rsu%%%%%%BBBBBBBBBBBBBBBB2222221000000'LLL&&&&&&  H % %xxxxxxxxxxr&