gJLdZddlZddlmZddlZddlZddlm Z ddl m Z ddl m Z mZmZmZerddlZejeZdZe rerddlmZndd lmZGd d eZGd d eZdadZdZdZdZ dZ!ddZ"ddZ#dS)z Integration with Deepspeed N) partialmethod)dep_version_check)is_accelerate_availableis_torch_availableis_torch_mlu_availableloggingctjddu}|rQ trt jd}dSt jd}dS#tj$rYdSwxYwdS)N deepspeedz deepspeed-mluTF) importlibutil find_specrimportlib_metadatametadataPackageNotFoundError)package_exists_s _/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/integrations/deepspeed.pyis_deepspeed_availabler#s^--k::$FN %'' &/@@t"+K88A4!6   55 s"A AA21A2)HfDeepSpeedConfig)objectc"eZdZdZfdZxZS)raJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. ct|tdtdt|dS)N accelerater )set_hf_deepspeed_configrsuper__init__selfconfig_file_or_dict __class__s rrzHfDeepSpeedConfig.__init__LsL%%%,'''+&&& ,-----)__name__ __module__ __qualname____doc__r __classcell__r!s@rrr;sB .........r"rc^eZdZdZfdZdZdZd dZeed Z d d Z d Z xZ S)HfTrainerDeepSpeedConfigz The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the same lifespan as the latter. cft|d|_g|_dSN)rr_dtype mismatchesrs rrz!HfTrainerDeepSpeedConfig.__init__Zs. ,--- r"c<|jtd|jS)Nz8trainer_config_process() wasn't called yet to tell dtype)r- ValueError)rs rdtypezHfTrainerDeepSpeedConfig.dtype_s! ; WXX X{r"c@||}|dS|dkS)NFauto) get_value)r ds_key_longvals ris_autoz HfTrainerDeepSpeedConfig.is_autods'nn[)) ;5&= r"NTc ||\}}|dS||dkr|||<dS|sdS||}|.||kr*|jd|d|d|d|dSdSdS)a A utility method that massages the config file and can optionally verify that the values match. 1. Replace "auto" values with `TrainingArguments` value. 2. If it wasn't "auto" and `must_match` is true, then check that DS config matches Trainer config values and if mismatched add the entry to `self.mismatched` - will assert during `trainer_config_finalize` for one or more mismatches. Nr3z- ds =z vs hf )find_config_nodegetr.append)rr5hf_valhf_key must_matchconfigds_keyds_vals r fill_matchz#HfTrainerDeepSpeedConfig.fill_matchks..{;; > F ::f   ' '#F6N F  FF##  &F"2"2 O " "#Y;#Y#Y#Y#Y#Y#YQW#Y#Y Z Z Z Z Z  "2"2r"F)r?c|j|jz|jz}|d|jd| |d|jd|d|d| |d|jd|d|jd |d |j|jgd |d |jd |d|j d| dd|d|jd |j s|j r|j dkrdnd}nd}|jr8|jdi|jd<|j|jdd<|d|j s|j o|dkd|d|dkd|d|jd|d|jp|jd|drt*j|_dS|drt*j|_dSt*j|_dS) z Adjust the config with `TrainingArguments` values. This stage is run during `TrainingArguments` object creation. train_micro_batch_size_per_gpuper_device_train_batch_sizegradient_accumulation_stepstrain_batch_sizeztrain_batch_size (calculated)gradient_clipping max_grad_normzoptimizer.params.lr learning_ratezoptimizer.params.betaszadam_beta1+adam_beta2zoptimizer.params.eps adam_epsilonzoptimizer.params.weight_decay weight_decayzscheduler.params.warmup_min_lrrzscheduler.params.warmup_max_lrapexampN checkpointuse_node_local_storagez fp16.enabledz%fp16|fp16_full_eval+fp16_backend(amp)z amp.enabledzfp16+fp16_backend(apex)z amp.opt_levelfp16_opt_levelz bf16.enabledzbf16|bf16_full_eval) world_sizerFrGrCrJrK adam_beta1 adam_beta2rLrM fill_onlyfp16fp16_full_eval fp16_backendsave_on_each_noder@r;rRbf16bf16_full_evalis_truetorchbfloat16r-is_falsefloat32float16)rargsauto_find_batch_sizerHrYs rtrainer_config_processz/HfTrainerDeepSpeedConfig.trainer_config_processs ?T-MMPTPpp  ,  , )$ $     )  , )      +$ $    +T-?QQQ -t/A?SSS  $ _do . #   .0A>RRR 79JN[[[ 7;;; 8$:Lo^^^ 9 + %)%6&%@%@66eLLL  ! Y(,  b(I(IDK %BFBXDK %&> ?  i.4. ILE4I 3     |v'=?XYYY )<>NOOO )Id6ILabbb << ' ' (.DKKK ]]> * * (-DKKK-DKKKr"c$gd}fd|D}t|dkrbt|jdr |jj}nt|jdrt |jj}nt|jdr,t|jjdr|jjj}nat|jdr9t|jjdrt |jjj}ntd|dd ||z rBd td |z|zd d |z d|d d| |dtj dkr-dj }td|ddS)z This stage is run after we have the model and know num_training_steps. Now we can complete the configuration process. )$zero_optimization.reduce_bucket_size-zero_optimization.stage3_prefetch_bucket_size4zero_optimization.stage3_param_persistence_thresholdc>g|]}||S)r7).0xrs r zDHfTrainerDeepSpeedConfig.trainer_config_finalize..s) V V VqdllSToo V V V Vr"r hidden_size hidden_sizes text_configzThe model's config file has neither `hidden_size` nor `hidden_sizes` entry, therefore it's not possible to automatically fill out the following `auto` entries in the DeepSpeed config file: zb. You can fix that by replacing `auto` values for these keys with an integer value of your choice.rgrhg?ri z scheduler.params.total_num_stepsznum_training_steps (calculated)z!scheduler.params.warmup_num_steps warmup_steps z]Please correct the following DeepSpeed config values that mismatch TrainingArguments values: zF The easiest method is to set these DeepSpeed config values to 'auto'.N)lenhasattrr@romaxrprqr0rVis_zero3intrCget_warmup_stepsr.join)rrcmodelnum_training_stepshidden_size_based_keyshidden_size_auto_keysror.s` rtrainer_config_finalizez0HfTrainerDeepSpeedConfig.trainer_config_finalizesf" " "  !W V V V,B V V V $ % % ) )u|]33 #l6 ~66 !%,";<< }55 '%,BZ\i:j:j #l6B }55 '%,BZ\j:k:k !%,":"GHH  Y5JYYY NNA;Q\C\ ] ] ]}} Ck)K788J$  .  -    /  ! !"4 5 5    t  ! # #4?33Jq'qqq  $ #r")NTF) r#r$r%r&rr1r7rCrrVrerr'r(s@rr*r*Ts   !!![[[[4 jU;;;IH(H(H(H(T@@@@@@@r"r*c.tj|adSr,)weakrefref_hf_deepspeed_config_weak_ref)hf_deepspeed_config_objs rrrs %,K0G$H$H!!!r"c dadSr,)rrkr"runset_hf_deepspeed_configr s%)!!!r"cpt.t tSdS)NF)rrxrkr"ris_deepspeed_zero3_enabledr&s1$05R5T5T5`,..77999ur"cVt!ttjSdSr,)rr@rkr"rdeepspeed_configr-s($05R5T5T5`,..55tr"cddlm}m}|j}d}d|vr#|jrt d||}nG|rtd }d|d <d} d |vr ||} n;t||rfd } ||| } n | } || fS)zY A convenience wrapper that deals with optimizer and lr scheduler configuration. r) DummyOptimDummySchedulerN optimizerz|--adafactor was passed, but also found `optimizer` configured in the DeepSpeed config. Only one optimizer can be configured.)paramszDetected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)Tzero_allow_untested_optimizer schedulercltj}d|_||}|S)Nr}r)copy lr_schedulercreate_scheduler)r trainer_copyrr}trainers r_lr_scheduler_callablez5deepspeed_optim_sched.._lr_scheduler_callable_sC#y11 -1 )+<<'9Y =   $#r")lr_scheduler_callabler) accelerate.utilsrrr@ adafactorr0 is_offloadloggerinfocreate_optimizer isinstancer) rhf_deepspeed_configrcr}model_parametersrrr@rrrs ` ` rdeepspeed_optim_schedr4sN<;;;;;;; 'FIf > 8 J&6777  ) ) + +  KKV   ,,.. 26./Lf%~i00 i , , p $ $ $ $ $ $*>)KabbbLL"33GYen3ooL l ""r"Fc"ddlm}|j}|j}|jjjj}||||| | |rU| std| d| dd\}}d} nMd|_tt!d|} t%||||| \}}||fS) a Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args. If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made. Args: trainer: Trainer object num_training_steps: per single gpu resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load inference: launch in inference mode (no optimizer and no lr scheduler) auto_find_batch_size: whether to ignore the `train_micro_batch_size_per_gpu` argument as it's being set automatically by the auto batch size finder Returns: optimizer, lr_scheduler We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on: https://github.com/microsoft/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping https://github.com/microsoft/DeepSpeed/issues/1612 r)rzMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configrr)NNNc|jSr,) requires_grad)ps rz deepspeed_init..sr")deepspeed.utilsrr|rc acceleratorstatedeepspeed_plugin hf_ds_configrsetLevelget_process_log_levelrxr0del_config_sub_treerlistfilter parametersr) rr} inference ds_loggerr|rcrrrrs rdeepspeed_initrqs;*433333 ME rs //////######999999````````````LLL  H % %    3!7!7!9!93OOOOOOO322222........2}}}}}0}}}B!%III))) :#:#:#z6#6#6#6#rPPPPPPr"