NgBdZddlZddlZddlmZGddeZdS)z AdamW Optimizer Impl copied from PyTorch master NOTE: Builtin optim.AdamW is used by the factory, this impl only serves as a Python based reference, will be removed someday N) OptimizercdeZdZdZ d fd ZfdZejd d ZxZ S) AdamWaImplements AdamW algorithm. The original Adam algorithm was proposed in `Adam: A Method for Stochastic Optimization`_. The AdamW variant was proposed in `Decoupled Weight Decay Regularization`_. Arguments: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-3) betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay coefficient (default: 1e-2) amsgrad (boolean, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 .. _Decoupled Weight Decay Regularization: https://arxiv.org/abs/1711.05101 .. _On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ MbP?g?g+?:0yE>{Gz?Fcd|ks"td|d|ks"td|d|dcxkrdks*ntd|dd|dcxkrdks*ntd|dt||||| }tt|||dS) NgzInvalid learning rate: {}zInvalid epsilon value: {}rg?z%Invalid beta parameter at index 0: {}z%Invalid beta parameter at index 1: {})lrbetaseps weight_decayamsgrad) ValueErrorformatdictsuperr__init__) selfparamsr r rrrdefaults __class__s L/var/www/html/ai-engine/env/lib/python3.11/site-packages/timm/optim/adamw.pyrzAdamW.__init__'s!byy8??CCDD Dczz8??DDEE EeAh$$$$$$$$DKKERSHUUVV VeAh$$$$$$$$DKKERSHUUVV V2U%17DDD eT##FH55555ctt|||jD]}|dddS)NrF)rr __setstate__ param_groups setdefault)rstategrouprs rrzAdamW.__setstate__5sV eT''...& / /E   Y . . . . / /rNcd}|5tj5|}dddn #1swxYwY|jD]C}|dD]6}|j |jd|d|dzz |j}|jrtd|d}|j|}t|dkrLd|d <tj ||d <tj ||d <|rtj ||d <|d |d } }|r|d } |d \} } |d xxdz cc<d| |d zz } d| |d zz }||  |d| z | |  ||d| z |rZtj | | | | tj|z |d}nB| tj|z |d}|d| z }|||| 8E|S)zPerforms a single optimization step. Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrr r rzJAdam does not support sparse gradients, please consider SparseAdam insteadrrstepexp_avg exp_avg_sqmax_exp_avg_sqr )alpha)value)outr)torch enable_gradrgraddatamul_ is_sparse RuntimeErrorr len zeros_likeadd_addcmul_maxsqrtmathaddcdiv_)rclosurelossr!pr,rr r$r%r&beta1beta2bias_correction1bias_correction2denom step_sizes rr#z AdamW.step:s  "$$ ! !wyy ! ! ! ! ! ! ! ! ! ! ! ! ! ! !&1 =1 =E8_0 =0 =6> Ad eN.C CCDDDv>u&'sttt * 1 u::??$%E&M','7':':E)$*/*:1*=*=E,'F272B12E2E./&+I&6l8K=%*+;%rJss  ++++++nnnnnInnnnnr