gZPddlZddlZddlmZddlmZmZmZmZddl Z ddl m Z ddl m Z mZddlmZmZddlmZdd lmZmZmZmZmZmZejeZ dd e j jd ed eeej fdeeeee!ee j"ffdeeedeeee!efee!effdeeeej fde#fdZ$ ddZ%ddZ& ddZ'dZ(dZ)de jfdZ*dZ+dS)N)deepcopy)DictListOptionalUnion)is_4bit_bnb_availableis_8bit_bnb_available)dispatch_modelinit_empty_weights)BnbQuantizationConfig)find_tied_parametersget_balanced_memoryinfer_auto_device_mapload_checkpoint_in_modeloffload_weightset_module_tensor_to_deviceFmodelbnb_quantization_configweights_location device_mapno_split_module_classes max_memoryoffload_folderoffload_state_dictc 4|j}|j} | rtstd|rt st dg} t trCt dkrd D} |j t||_ |r|j | |j } |jg|_|j} | | ||_| |_t#|} | jdkrt&dt+||| }|j}| D]\}t1fd | Dr|t4j|jt4jkr\d d d d t=|d}||t4jt5j|r||| jd krP| t4j !t4j "n_t4j #r2|t4j !ntIdt&%d| jd|S|tId|dtM5t+||| }dddn #1swxYwYtO|||||d(vrd}t1fddD}tS|||j|||j| o|tU||S)a} This function will quantize the input model with the associated config passed in `bnb_quantization_config`. If the model is in the meta device, we will load and dispatch the weights according to the `device_map` passed. If the model is already loaded, we will quantize the model and put the model on the GPU, Args: model (`torch.nn.Module`): Input model. The model can be already loaded or on the meta device bnb_quantization_config (`BnbQuantizationConfig`): The bitsandbytes quantization parameters weights_location (`str` or `os.PathLike`): The folder weights_location to load. It can be: - a path to a file containing a whole model state dict - a path to a `.json` file containing the index to a sharded checkpoint - a path to a folder containing a unique `.index.json` file and the shards of a checkpoint. - a path to a folder containing a unique pytorch_model.bin file. device_map (`Dict[str, Union[int, str, torch.device]]`, *optional*): A map that specifies where each submodule should go. It doesn't need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. no_split_module_classes (`List[str]`, *optional*): A list of layer class names that should never be split across device (for instance any layer that has a residual connection). max_memory (`Dict`, *optional*): A dictionary device identifier to maximum memory. Will default to the maximum memory available if unset. offload_folder (`str` or `os.PathLike`, *optional*): If the `device_map` contains any value `"disk"`, the folder where we will offload weights. offload_state_dict (`bool`, *optional*, defaults to `False`): If `True`, will temporarily offload the CPU state dict on the hard drive to avoid getting out of CPU RAM if the weight of the CPU state dict + the biggest shard does not fit. Returns: `torch.nn.Module`: The quantized model zYou have a version of `bitsandbytes` that is not compatible with 8bit quantization, make sure you have the latest version of `bitsandbytes` installed.zYou have a version of `bitsandbytes` that is not compatible with 4bit quantization,make sure you have the latest version of `bitsandbytes` installed.r c g|] \}}|dv | S))diskcpu).0keyvalues P/var/www/html/ai-engine/env/lib/python3.11/site-packages/accelerate/utils/bnb.py z+load_and_quantize_model..is&___*#ueF^F^#F^F^F^NmetazIt is not recommended to quantize a loaded model. The model should be instantiated under the `init_empty_weights` context manager.)modules_to_not_convertc3 K|]}|vV dSNr!)r"module_to_keep_in_fp32names r% z*load_and_quantize_model..s*ee6L)T1eeeeeer'.weight.biascuda/No GPU found. A GPU is needed for quantization.zThe model device type is zE. However, cuda is needed for quantization.We move the model to cuda.za`weights_location` needs to be the folder path containing the weights of the model, but we found  )rrrTc3^K|]'}|tvV(dSr+)listvalues)r"xrs r%r.z*load_and_quantize_model..s<NNa4 1 1 3 3444NNNNNNr'r r)dtyperrkeep_in_fp32_modulesoffload_8bit_bnb)r offload_dir)+ load_in_4bit load_in_8bitr ImportErrorr ValueError isinstancedictlenkeysitems skip_modulesget_keys_to_not_convertextendr;is_loaded_in_4bitis_loaded_in_8bitget_parameter_devicetypeloggerwarningreplace_with_bnb_layers torch_dtype state_dictanytotorchfloat32r:replacegetattris_floating_pointr2current_device empty_cache is_available RuntimeErrorinfor get_quantized_model_device_mapr7rr )rrrrrrrrr>r?modules_on_cpur)r; model_devicer:paramoffloadr-s ` @r%load_and_quantize_modelrd,sX+7L*7L 133  R    133  Q   N*d##`JOO,=,=(>(>(B(B__ 0@0@0B0B___+3/Fu/M/M,D,33NCCC4A3;7942G!!"6777+E*E'..LF"" _   (/F_uvvv'3 ++--3355 KD%eeeePdeeeee ''';%-//<< 266>>wKKD#E466E(///(//    & & JJuz0022 3 3 3 J " " $ $ $ $ Z $ $ & & R HHUZ..00 1 1 1 1PQQ Q  ) (9 ) ) )     ! DqA D D D    ! !  +.G]E                4  # !$;      %**@VzO`O`ObObEbEb!% NNNNoNNNNN   )5)1!8!M)5g e WWWWs0NNNc htjr!dtjint dt dttrdvrtdi}| fd| D| fd| Di}||d<||d <j |d <d krt|fd k|d |}||d<t|fi|ttr|jjz  fdD}dD]H}||vr0jrtdt dI~S)Nr0r3z\The device_map was not initialized.Setting device_map to `{'':torch.cuda.current_device()}`.)autobalancedbalanced_low_0 sequentialziIf passing a string for `device_map`, please choose 'auto', 'balanced', 'balanced_low_0' or 'sequential'.chi|]-\}tfdjD%j.S)c3 K|]}|vV dSr+r!r"mr-s r%r.z..s'OOQqDyOOOOOOr')rSrGrQr"_r-rs @r% z2get_quantized_model_device_map..s[   D!OOOO*A*NOOOOO -9   r'cri|]2\}tfdjD%tj3S)c3 K|]}|vV dSr+r!rls r%r.z..s'WWQqDyWWWWWWr')rSr;rUrVrns @r%rpz2get_quantized_model_device_map..sY   D!WWWW*A*VWWWWW em   r'special_dtypesrr:rirh)low_zerorrc*i|]}|v||Sr!r!)r"r#rmodules_not_to_converts r%rpz2get_quantized_model_device_map..s1+ + + %(LbAbAbCCAbAbAbr'r9aH Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in `torch_dtype`, you need to pass a custom `device_map` to `load_and_quantize_model`. Check https://huggingface.co/docs/accelerate/main/en/usage_guides/quantization#offload-modules-to-cpu-and-disk for more details. ziSome modules are are offloaded to the CPU or the disk. Note that these modules will be converted to 8-bit)rUr2r\rZr]rNr^rBstrrAupdatenamed_parameters target_dtyperrrCrGr;rEr7r>) rrrrrrskwargsdevice_map_without_some_modulesdevicervs `` @r%r_r_s : " " $ $ Rej7799:JJPQQ Q uvvv*c""&< Q Q Q       $5577           $5577      #1 ,C()1>w  % %,$(88% J *|*5;;F;; *d##,!8!EH_Ht!t+ + + + + ,6OO,=,=+ + + '&  F8??AAAA*7$   KKD , r'cp|g}t||||\}}|std|S)a, A helper function to replace all `torch.nn.Linear` modules by `bnb.nn.Linear8bit` modules or by `bnb.nn.Linear4bit` modules from the `bitsandbytes`library. The function will be run recursively and replace `torch.nn.Linear` modules. Parameters: model (`torch.nn.Module`): Input model or `torch.nn.Module` as the function is run recursively. modules_to_not_convert (`List[str]`): Names of the modules to not quantize convert. In practice we keep the `lm_head` in full precision for numerical stability reasons. current_key_name (`List[str]`, *optional*): An array to track the current key of the recursion. This is used to check whether the current key (part of it) is not in the list of modules to not convert. NaYou are loading your model in 8bit or 4bit but no linear modules were found in your model. this can happen for some architectures such as gpt2 that uses Conv1D instead of Linear layers. Please double check your model architecture, or submit an issue on github if you think this is a bug.)_replace_with_bnb_layersrNrO)rrr)current_key_namehas_been_replaceds r%rPrPs] %!#7 &(>@P  E       Lr'c ddl}d}|D]\}}|g}||t|tjr'||vr"d|}d} |D]} | |vr| dz|vs| |krd} n| r|jr6|j|j |j |j dud|j } nW|j rA|j|j |j |j du|j|j|j} nt%d|jj| j_|j |j j| j _| dt-||| d}t/t1|dkrt5||||\} } || z}|d ||fS) z Private method that wraps the recursion for module replacement. Returns the converted model and a boolean that indicates if the conversion has been successfull or not. rNF.T)has_fp16_weights threshold)compress_statistics quant_typez1load_in_8bit and load_in_4bit can't be both False) bitsandbytesnamed_childrenappendrBnnLinearjoinr? Linear8bitLt in_features out_featuresbiasllm_int8_thresholdr> Linear4bitbnb_4bit_compute_dtypebnb_4bit_use_double_quantbnb_4bit_quant_typerAweightdatarequires_grad_setattrrDr6childrenrpop)rrr)rbnbrr-modulecurrent_key_name_strproceedr# bnb_modulero_has_been_replaceds r%rr2sS,,../!/! f  #! %%% fbi ( ($ )T9O-O-O#&88,<#=#= G-  000sSyDX7X7X000#GE1 )*7Z!$!4!4*+ 4/)."9"L "5""JJ-9 Z!$!2!2*+ 4//F,C,]#:#N "3""JJ%%XYYY)/); !&;*+1;+;JO())%000tZ000$(! tFOO%%&& ' '! + +$</1GIY%% !A!!24F F R    # ##r'crt5t|}dddn #1swxYwYt|}t|trRt t |gt |z}nt |g}t|dk}d}t|drt||j  }|s|rgSt | }|ddg}t|t|z }t t|t |z} ddg} g} | D]6} | D]} | | vr| | d} | | 7| S) a An utility function to get the key of the module to keep in full precision if any For example for CausalLM modules we may want to keep the lm_head in full precision for numerical stability reasons. For other architectures, we want to keep the tied weights of the model. The function will return a list of the keys of the modules to not convert in int8. Parameters: model (`torch.nn.Module`): Input model NrFbase_model_prefixrr/r1r0)r rrrBrCsumr6r7rErDhasattrrrsetrWr)r tied_model tied_params tied_keyshas_tied_params is_base_model list_moduleslist_last_module intersectionlist_untouchednames_to_removefiltered_module_namesr-name_to_removes r%rHrHts    %%e__ %%%%%%%%%%%%%%%'z22K+t$$)[//1122B77${?O?O?Q?Q:R:RR  R(( )nnq(OMu)**D#E5+BCCC   ,,..//L$R(+,'((3y>>9L#i..))D,>,>>N!'*O++- 8 8N%%||NB77$$T****  s +//cvddl}|D]}t||jjrdS dS)zUCheck if we have `bnb.nn.Linear4bit` or `bnb.nn.Linear8bitLt` layers inside our modelrNTF)rmodulesrBrr)rrrms r%has_4bit_bnb_layersrsO ]]__ a* + + 44  5r' parametercNt|jSr+)next parametersr})rs r%rLrLs  $$&& ' ' ..r'c |t||d|||}|}d|vrS|d} | ddD]+} t|| } | t|d| d| },| d}d|j|_t |j||||t|j|dr7t |j|j| d d||n:t ||||t || d d||t||d |tj | dS) Nr)r:r$rrz has no attribute F)indexSCBrr() rsplitrXrA _parameters requires_gradrrrrWrUemptysize) rrb param_name new_dtyper offload_indexfp16_statistics tensor_namersplitsr new_modules r%quantize_and_offload_8bitrs#E:q QVWWWW  +   &&s++F $ $$VU33 %$%J%J%%J%J%JKKK# *K8=;'5v)+6 NZghhhh 6%k2E : :  ";/3""8U33#      uj. NNNN (:(:8U(K(K^cpqqqqz6RWR]_d_i_i_k_kRlmmmmmmr')NNNNNF)NNN)NN),loggingoscopyrtypingrrrrrUtorch.nnraccelerate.utils.importsrr big_modelingr r dataclassesrmodelingrrrrrr getLogger__name__rNModulerwPathLikeintr}boolrdr_rPrrHrrLrr!r'r%rs  ............  >=======......  8 $ $ 15EI37CG8<$UXUX 8?UX2UXC,-UXc5c5<)?#@@AB UX &d3i0 UX eCHouS#X>?@ UXU3 #345UXUXUXUXUXr_cKKKK\F  ?$?$?$?$D1!1!1!h/BI////nnnnnr'