g8ddlZddlmZmZGddZdS)N) AcceleratorDistributedTypec XeZdZdZdZdZd dedejj de de fd Z d Z d Zd S)LocalSGDu A helper class to support local SGD on top of Accelerator. It simply runs a given number of updates independently on each device, and averages model weights every K synchronization step. It should be used only in the multi-GPU (or multi-CPU) setup without extensions such as DeepSpeed. In particular, this is a simple implementation that cannot support scenarios such as model parallelism. Although we are not aware of the true origins of this simple approach, the idea of local SGD is quite old and goes back to at least: Zhang, J., De Sa, C., Mitliagkas, I., & Ré, C. (2016). [Parallel SGD: When does averaging help?. arXiv preprint arXiv:1606.07365.](https://arxiv.org/abs/1606.07365) We credit the term Local SGD to the following paper (but there might be earlier references we are not aware of). Stich, Sebastian Urban. ["Local SGD Converges Fast and Communicates Little." ICLR 2019-International Conference on Learning Representations. No. CONF. 2019.](https://arxiv.org/abs/1805.09767) c|jr7|j|_|j|SN)enabledmodelno_syncmodel_sync_obj __enter__selfs P/var/www/html/ai-engine/env/lib/python3.11/site-packages/accelerate/local_sgd.pyr zLocalSGD.__enter__)s< < ,"&*"4"4"6"6D    ) ) + + + cx|jr2||j|||dSdSr)r _sync_and_avg_model_paramsr __exit__)rtypevaluetbs rrzLocalSGD.__exit__0sI < :  + + - - -   ( (ub 9 9 9 9 9 : :rT acceleratorr local_sgd_stepsr c<|jtjtjtjtjtjtjfvrtd|o|jtjk|_ d|_ |j r||_ ||_ ||_ dSdS)a Constructor. Args: model (`torch.nn.Module): The model whose parameters we need to average. accelerator (`Accelerator`): Accelerator object. local_sgd_steps (`int`): A number of local SGD steps (before model parameters are synchronized). enabled (`bool): Local SGD is disabled if this parameter set to `False`. zILocalSGD is supported only for CPUs and GPUs (no DeepSpeed or MegatronLM)rN)distributed_typerNO MULTI_CPU MULTI_GPU MULTI_MLU MULTI_MUSA MULTI_NPUNotImplementedErrorr num_stepsrr r)rrr rr s r__init__zLocalSGD.__init__6s  '    %  %  %  &  % 0   &&qrr rU;#??CU#U  < 3*D DJ#2D  3 3rc|xjdz c_|jsdS|j|jzdkr|dSdS)z^ This function makes a "step" and synchronizes model parameters if necessary. Nr)r#r rrrs rstepz LocalSGD.stepTsX !|  F >D0 0A 5 5  + + - - - - - 6 5rc"|j|j5|jD](}|j|jd|_) ddddS#1swxYwYdS)zH Synchronize + Average model parameters across all GPUs mean) reductionN)rwait_for_everyoneautocastr parametersreducedata)rparams rrz#LocalSGD._sync_and_avg_model_params_s **,,,   & & ( ( S S..00 S S!-44UZ64RR  S S S S S S S S S S S S S S S S S S SsABB BN)T)__name__ __module__ __qualname____doc__r rrtorchnnModuleintboolr$r'rrrrrs*::: 33K33Z]3hl3333< . . .SSSSSrr)r5 acceleraterrrr:rrr<sn 33333333TSTSTSTSTSTSTSTSTSTSr