NgBdZddlZddlmZddlZGddeZdS)a6 Nvidia NovoGrad Optimizer. Original impl by Nvidia from Jasper example: - https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechRecognition/Jasper Paper: `Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks` - https://arxiv.org/abs/1905.11286 N) OptimizercdeZdZdZ d fd ZfdZejd d ZxZ S) NvNovoGrada( Implements Novograd algorithm. Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.95, 0.98)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) grad_averaging: gradient averaging amsgrad (boolean, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) MbP?gffffff?g\(\?:0yE>rFcd|ks"td|d|ks"td|d|dcxkrdks*ntd|dd|dcxkrdks*ntd|dt|||||| }tt|||dS) NgzInvalid learning rate: {}zInvalid epsilon value: {}rg?z%Invalid beta parameter at index 0: {}z%Invalid beta parameter at index 1: {})lrbetaseps weight_decaygrad_averagingamsgrad) ValueErrorformatdictsuperr__init__) selfparamsr r r rrrdefaults __class__s Q/var/www/html/ai-engine/env/lib/python3.11/site-packages/timm/optim/nvnovograd.pyrzNvNovoGrad.__init__ s$byy8??CCDD Dczz8??DDEE EeAh$$$$$$$$DKKERSHUUVV VeAh$$$$$$$$DKKERSHUUVV V2U%1'5 '))) j$((:::::ctt|||jD]}|dddS)NrF)rr __setstate__ param_groups setdefault)rstategrouprs rrzNvNovoGrad.__setstate__1sV j$,,U333& / /E   Y . . . . / /rNcd}|5tj5|}dddn #1swxYwY|jD]}|dD]}|j |j}|jrt d|d}|j|}t|dkrd|d<tj||d<tj g |dj |d<|r5tj g |dj |d <|d|d} }|r|d } |d \} } |dxxd z cc<tj tj |d } | dkr| | n-| | | d | z |rEtj| | | | |d}n-| |d}|||ddkr|||d |dr|d | z || ||||d |S)zPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrz#Sparse gradients are not supported.rrstepexp_avg exp_avg_sqmax_exp_avg_sqr r )alpha)outr rrr )torch enable_gradrgrad is_sparse RuntimeErrorr len zeros_likezerostodevicesumpowcopy_mul_add_maxsqrtdiv_)rclosurelossr!pr,rr r$r%r&beta1beta2normdenoms rr#zNvNovoGrad.step6s/  "$$ ! !wyy ! ! ! ! ! ! ! ! ! ! ! ! ! ! !&3 43 4E8_2 42 46>v>N&'LMMM * 1 u::??$%E&M','7':':E)$*/+b//*<*rLss ++++++ kkkkkkkkkkr