g+ddlZddlZddlZddlmZddlmZddlmZddl m Z dZ Gd d Z Gd d Z Gd deZGddeZGddZGddZdS)N)deepcopy)AcceleratedOptimizer)AcceleratedScheduler)DistributedTypec|jtjkrtdt |jt s|jStd|jDS)z Returns the currently active DeepSpeedPlugin. Raises: ValueError: If DeepSpeed was not enabled and this function is called. a!Couldn't retrieve the active `DeepSpeedPlugin` as none were enabled. Please make sure that either `Accelerator` is configured for `deepspeed` or make sure that the desired `DeepSpeedPlugin` has been enabled (`AcceleratorState().select_deepspeed_plugin(name)`) before calling this function.c3(K|] }|j |VdSN)selected).0plugins V/var/www/html/ai-engine/env/lib/python3.11/site-packages/accelerate/utils/deepspeed.py z.get_active_deepspeed_plugin..)s)YY6YYYYYYY) distributed_typer DEEPSPEED ValueError isinstancedeepspeed_pluginsdictnextvalues)states rget_active_deepspeed_pluginrsx !::: ,   e-t 4 4'&& YYU%<%C%C%E%EYYY Y YYrcReZdZdZdZdZdZddZddZd Z d Z d Z d Z d Z dS)HfDeepSpeedConfigaJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. ct|trt|}ntj|r>t |d5}tj|}dddn #1swxYwYnj tj | d}tj |}n-#tttf$rtd|wxYw||_|dS)Nzutf-8)encodingzoExpected a string path to an existing deepspeed config, or a dictionary, or a base64 encoded string. Received: )rrrospathexistsopenjsonloadbase64urlsafe_b64decodedecodeloadsUnicodeDecodeErrorAttributeErrorrconfigset_stage_and_offload)selfconfig_file_or_dictr,fconfig_decodeds r__init__zHfDeepSpeedConfig.__init__=sV )4 0 0 122FF W^^/ 0 0 )G<<< &1 & & & & & & & & & & & & & & & !'!9:M!N!N!U!UV]!^!^N33& C    \GZ\\    ""$$$$$sA77A;>A;;C*C*cl|dd|_d|_|s|rit ddg}t |d|dg}t ||zdkr d |_dSdSdS) Nzzero_optimization.stageFcpunvmez*zero_optimization.offload_optimizer.devicez&zero_optimization.offload_param.devicerT) get_value_stage_offloadis_zero2is_zero3setlen)r.offload_devices_validoffload_devicess rr-z'HfDeepSpeedConfig.set_stage_and_offloadRsnn%>CC  ==?? %dmmoo %$'$8$8 !!NN#OPPNN#KLLO ?%::;;a?? $ @? % %rc|j}|d}|}|D]}||}|d|fcS ||fS)N.)r,splitpopget)r. ds_key_longr,nodesds_keynodes rfind_config_nodez"HfDeepSpeedConfig.find_config_nodeesq!!#&& $ $DZZ%%F~V|###v~rNcf||\}}||S|||S)zG Returns the set value or `default` if no value is set )rIrD)r.rEdefaultr,rGs rr7zHfDeepSpeedConfig.get_valuers9..{;; >Nzz&'***rFc|j}|d}|D]:}|}||}||rtd|d|jdS;|||dSdS)z Deletes a sub-section of the config file if it's found. Unless `must_exist` is `True` the section doesn't have to exist. rANz Can't find z entry in the config: )r,rBrDrrC)r.rE must_existr,rFrH parent_configs rdel_config_sub_treez%HfDeepSpeedConfig.del_config_sub_tree{s !!#&&  D"MZZ%%F~$%c;%c%cVZVa%c%cdddFF   $   d # # # # # % $rcR||}|dnt|S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `True` (and it's not set to `False`` or isn't set). NFr7boolr.rEvalues ris_truezHfDeepSpeedConfig.is_trues) {++ uu4;;6rcT||}|dnt| S)z Returns `True`/``False` only if the value is set, always `False` otherwise. So use this method to ask the very specific question of whether the value is set to `False` (and it's not set to `True`` or isn't set). NFrQrSs ris_falsezHfDeepSpeedConfig.is_falses+ {++ uutE{{?:rc|jdkS)Nrr8r.s rr:zHfDeepSpeedConfig.is_zero2{arc|jdkS)NrYrZs rr;zHfDeepSpeedConfig.is_zero3r[rc|jSr )r9rZs r is_offloadzHfDeepSpeedConfig.is_offloads }rr )F)__name__ __module__ __qualname____doc__r2r-rIr7rOrUrWr:r;r_rrrr,s %%%*%%%&   ++++$$$$.777;;;      rrceZdZdZdZdZdS)DeepSpeedEngineWrapperz Internal wrapper for deepspeed.runtime.engine.DeepSpeedEngine. This is used to follow conventional training loop. Args: engine (deepspeed.runtime.engine.DeepSpeedEngine): deepspeed engine to wrap c||_dSr )engine)r.rhs rr2zDeepSpeedEngineWrapper.__init__s  rc ^|jj|fi||jdSr )rhbackwardstep)r.losskwargss rrjzDeepSpeedEngineWrapper.backwards; T,,V,,, rN)r`rarbrcr2rjrdrrrfrfs<     rrfcFeZdZdZfdZddZdZedZxZ S)DeepSpeedOptimizerWrapperz Internal wrapper around a deepspeed optimizer. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. ct|ddt|jd|_dS)NF)device_placementscaleroverflow)superr2hasattr optimizer__has_overflow__)r.rv __class__s rr2z"DeepSpeedOptimizerWrapper.__init__s< U4HHH ' C CrNcdSr rd)r. set_to_nones r zero_gradz#DeepSpeedOptimizerWrapper.zero_grad rcdSr rdrZs rrkzDeepSpeedOptimizerWrapper.stepr|rc,|jr |jjSdS)zTWhether or not the optimizer step was done, or skipped because of gradient overflow.F)rwrvrsrZs rstep_was_skippedz*DeepSpeedOptimizerWrapper.step_was_skippeds   +>* *urr ) r`rarbrcr2r{rkpropertyr __classcell__rxs@rrorosDDDDD       Xrroc(eZdZdZfdZdZxZS)DeepSpeedSchedulerWrapperz Internal wrapper around a deepspeed scheduler. Args: scheduler (`torch.optim.lr_scheduler.LambdaLR`): The scheduler to wrap. optimizers (one or a list of `torch.optim.Optimizer`): cLt||dSr )rtr2)r. scheduler optimizersrxs rr2z"DeepSpeedSchedulerWrapper.__init__s# J/////rcdSr rdrZs rrkzDeepSpeedSchedulerWrapper.stepr|r)r`rarbrcr2rkrrs@rrrsQ00000       rrceZdZdZddZdS) DummyOptima Dummy optimizer presents model parameters or param groups, this is primarily used to follow conventional training loop when optimizer config is specified in the deepspeed config file. Args: lr (float): Learning rate. params (iterable): iterable of parameters to optimize or dicts defining parameter groups weight_decay (float): Weight decay. **kwargs (additional keyword arguments, *optional*): Other arguments. MbP?rc >||_||_||_||_dSr )paramslr weight_decayrm)r.rrrrms rr2zDummyOptim.__init__s# ( rN)rrr`rarbrcr2rdrrrrs2  rrceZdZdZddZdS)DummySchedulera Dummy scheduler presents model parameters or param groups, this is primarily used to follow conventional training loop when scheduler config is specified in the deepspeed config file. Args: optimizer (`torch.optim.optimizer.Optimizer`): The optimizer to wrap. total_num_steps (int, *optional*): Total number of steps. warmup_num_steps (int, *optional*): Number of steps for warmup. lr_scheduler_callable (callable, *optional*): A callable function that creates an LR Scheduler. It accepts only one argument `optimizer`. **kwargs (additional keyword arguments, *optional*): Other arguments. Nrc L||_||_||_||_||_dSr )rvtotal_num_stepswarmup_num_stepslr_scheduler_callablerm)r.rvrrrrms rr2zDummyScheduler.__init__s,". 0%:" r)NrNrrdrrrr s2"rr)r&r$r copyrrvrrr dataclassesrrrrfrorrrrdrrrs ,,,,,,,,,,,,((((((ZZZ&~~~~~~~~B8 46     4   ".r