Ng:dZddlZddlmZGddeZdS)aL RMSProp modified to behave like Tensorflow impl Originally cut & paste from PyTorch RMSProp https://github.com/pytorch/pytorch/blob/063946d2b3f3f1e953a2a3b54e0b34f1393de295/torch/optim/rmsprop.py Licensed under BSD-Clause 3 (ish), https://github.com/pytorch/pytorch/blob/master/LICENSE Modifications Copyright 2021 Ross Wightman N) OptimizercdeZdZdZ d fd Zfd Zejdd ZxZ S) RMSpropTFaEImplements RMSprop algorithm (TensorFlow style epsilon) NOTE: This is a direct cut-and-paste of PyTorch RMSprop with eps applied before sqrt and a few other modifications to closer match Tensorflow for matching hyper-params. Noteworthy changes include: 1. Epsilon applied inside square-root 2. square_avg initialized to ones 3. LR scaling of update accumulated in momentum buffer Proposed by G. Hinton in his `course `_. The centered version first appears in `Generating Sequences With Recurrent Neural Networks `_. Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-2) momentum (float, optional): momentum factor (default: 0) alpha (float, optional): smoothing (decay) constant (default: 0.9) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-10) centered (bool, optional) : if ``True``, compute the centered RMSProp, the gradient is normalized by an estimation of its variance weight_decay (float, optional): weight decay (L2 penalty) (default: 0) decoupled_decay (bool, optional): decoupled weight decay as per https://arxiv.org/abs/1711.05101 lr_in_momentum (bool, optional): learning rate scaling is included in the momentum buffer update as per defaults in Tensorflow {Gz??绽|=rFTc d|ks"td|d|ks"td|d|ks"td|d|ks"td|d|ks"td|t|||||||| } tt||| dS)Nr zInvalid learning rate: {}zInvalid epsilon value: {}zInvalid momentum value: {}zInvalid weight_decay value: {}zInvalid alpha value: {})lrmomentumalphaepscentered weight_decaydecoupled_decaylr_in_momentum) ValueErrorformatdictsuperr__init__) selfparamsr r rrr rrrdefaults __class__s Q/var/www/html/ai-engine/env/lib/python3.11/site-packages/timm/optim/rmsprop_tf.pyrzRMSpropTF.__init__0sbyy8??CCDD Dczz8??DDEE Eh9@@JJKK Kl""=DD\RRSS Se||6==eDDEE EHEsX\h+NLLL i''99999ctt|||jD].}|dd|dd/dS)Nr rrF)rr __setstate__ param_groups setdefault)rstategrouprs rrzRMSpropTF.__setstate__Bsl i++E222& 0 0E   Z + + +   Z / / / / 0 0rNc*d}|5tj5|}dddn #1swxYwY|jD]}|dD]}|j |j}|jrt d|j|}t|dkr^d|d<tj||d<|ddkrtj ||d<|d rtj ||d <|d}d |d z }|dxxd z cc<|ddkrM|dr(| d |d|dzz n| ||d}| | d|z ||d rf|d } | || z ||| | d |d} n-| |d} |ddkr|d} |drO| |d|| |d| | U| |d|| | | |d ||| |d Ő|S)zPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrz)RMSprop does not support sparse gradientsrstep square_avgr momentum_bufferrgrad_avgg?r rrr )r )valuerr)torch enable_gradr grad is_sparse RuntimeErrorr"len ones_like zeros_likemul_addadd_powaddcmulsqrt_addcdiv_) rclosurelossr#pr/r"r&one_minus_alphar(avgbufs rr%zRMSpropTF.stepHs}  "$$ ! !wyy ! ! ! ! ! ! ! ! ! ! ! ! ! ! !&4 >4 >E8_3 >3 >6>v>T&'RSSS 1 u::??$%E&M*//!*<*>C$q(( 12C-.8z!233<j s /33)rrrrr FFT)N) __name__ __module__ __qualname____doc__rrr-no_gradr% __classcell__)rs@rrrsBej7;::::::$00000 U]__BBB_BBBBBrr)rEr- torch.optimrrrrrJsj !!!!!!}}}}} }}}}}r