Χg-O$dZddlmZmZmZmZddlZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZmZmZddgZGd deZd d e d ed e d e d ze_deedeedeedeedeedeededededededededededef dZdeedeedeedeedeedeededededededededededef dZe e  d$deedeedeedeedeedeed"eededededededededededef"d#ZdS)%z)Implementation for the RMSprop algorithm.)castListOptionalUnionN)Tensor) _capturable_doc_default_to_fused_or_foreach_differentiable_doc_disable_dynamo_if_unsupported _foreach_doc!_get_capturable_supported_devices_get_scalar_dtype _maximize_doc_use_grad_for_differentiable _view_as_real OptimizerParamsTRMSproprmspropceZdZ ddedeeefd ed ed ed ed eededeffd Z fdZ dZ e ddZ xZS)r{Gz?Gz?:0yE>rFNparamslralphaeps weight_decaymomentumforeachmaximizedifferentiablec t|tr'|dkrtdd|kstd|d|kstd|d|kstd|d|kstd|d|kstd|t |||||||| | | } t || dS) NrzTensor lr must be 1-elementgzInvalid learning rate: zInvalid epsilon value: zInvalid momentum value: zInvalid weight_decay value: zInvalid alpha value: ) rr rrcenteredr capturabler!r"r#) isinstancernumel ValueErrordictsuper__init__)selfrrrrrr r%r&r!r"r#defaults __class__s O/var/www/html/ai-engine/env/lib/python3.11/site-packages/torch/optim/rmsprop.pyr,zRMSprop.__init__s, b& ! ! is_complexappend is_sparse RuntimeErrorr;r=zerosrr6 zeros_likepreserve_format) r-rBparams_with_gradgrads square_avgsmomentum_buffer_list grad_avgs state_steps has_complexrCr;s r0 _init_groupzRMSprop._init_group[s x$ 4$ 4Av~ 5+A.. .K  # #A & & &v P"#NOOO LL JqME5zzQ\*DEK*;*=*=ahOOOOR/@/B/BCCCf ',&6U%:'''l#$q((/4/?)>000E+,$(-(8)>)))E*%   u\2 3 3 3   uV} - - -Z 1$$$++E2C,DEEEZ  4  z!2333r1c|d}|5tj5|}dddn #1swxYwY|jD]}g}g}g}g}g}g} |||||||| } t |||||| |d|d|d|d|d|d|d|d |d |d | |S) zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrr r%r!r"r#r&) rrrrr r%r!r"r#r&rZ) _cuda_graph_capture_health_checkr> enable_gradr9r[r) r-closurelossrBrTrUrVrXrWrYrZs r0r3z RMSprop.stepsv --///  "$$ ! !wyy ! ! ! ! ! ! ! ! ! ! ! ! ! ! !&$ $ E-/ "$E(*K&(I13 (*K** $K  $;Gn%L">2z*z*i(z*$%56 .'#     ( s AA A) rrrrrFFNFFN)__name__ __module__ __qualname__rrr@rrboolr,r8r[rr3 __classcell__)r/s@r0rrs$("&$'+'+'+ %- '+ '+  '+  '+'+$'+'+'+'+'+'+'+'+R*111f"444"!44444r1a Implements RMSprop algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \alpha \text{ (alpha)},\: \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)},\: \mu \text{ (momentum)},\: centered\\ &\textbf{initialize} : v_0 \leftarrow 0 \text{ (square average)}, \: \textbf{b}_0 \leftarrow 0 \text{ (buffer)}, \: g^{ave}_0 \leftarrow 0 \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}v_t \leftarrow \alpha v_{t-1} + (1 - \alpha) g^2_t \hspace{8mm} \\ &\hspace{5mm} \tilde{v_t} \leftarrow v_t \\ &\hspace{5mm}if \: centered \\ &\hspace{10mm} g^{ave}_t \leftarrow g^{ave}_{t-1} \alpha + (1-\alpha) g_t \\ &\hspace{10mm} \tilde{v_t} \leftarrow \tilde{v_t} - \big(g^{ave}_{t} \big)^2 \\ &\hspace{5mm}if \: \mu > 0 \\ &\hspace{10mm} \textbf{b}_t\leftarrow \mu \textbf{b}_{t-1} + g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \\ &\hspace{10mm} \theta_t \leftarrow \theta_{t-1} - \gamma \textbf{b}_t \\ &\hspace{5mm} else \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \hspace{3mm} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `lecture notes `_ by G. Hinton. and centered version `Generating Sequences With Recurrent Neural Networks `_. The implementation here takes the square root of the gradient average before adding epsilon (note that TensorFlow interchanges these two operations). The effective learning rate is thus :math:`\gamma/(\sqrt{v} + \epsilon)` where :math:`\gamma` is the scheduled learning rate and :math:`v` is the weighted moving average of the squared gradient. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, Tensor, optional): learning rate (default: 1e-2) momentum (float, optional): momentum factor (default: 0) alpha (float, optional): smoothing constant (default: 0.99) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) centered (bool, optional) : if ``True``, compute the centered RMSProp, the gradient is normalized by an estimation of its variance weight_decay (float, optional): weight decay (L2 penalty) (default: 0) z z rrUrVrXrWrYrrrrr r%r"r#r&rZc t|D]P\}}||}tjsF|rDt }|jj|jjkr|jj|vsJd|d||}| s|n| }||}|dz }| dkr||| }tj|}|r_utils is_compilingrr6typeaddrM view_as_realmul_addcmul_lerp_addcmulsqrt_sqrtadd_addcdiv_)rrUrVrXrWrYrrrrr r%r"r#r&rZiparamr3capturable_supported_devicesrLrIis_complex_paramrKavgbufs r0_single_tensor_rmsproprs&f%%111151~|((** {z {+L+N+N ( !T[%555L%)EEEEz[wzzzFEEQx#.tt$ ^    1  88E866D +E22  8&u--E%d++D+J77J''d!e)'DDD  $ |H 8 -h77 NN4U + + +$$Xxr$BBHHJJCC//##C  ''#,,CC((3--C a<<&q)C .(-- HHX   ' 'c 2 2 2 JJs2#J & & & & NN4RCN 0 0 0 0c1111r1c !t|dkrdS| r JdtjsG|rEt !t !fdt ||DsJd!dtj||||||g}| D]\\}}}}}}}ttt|}ttt|}ttt|}ttt|}|r||g}| dkr5ttt|}| || r5ttt|}| |t|g|R| rtj|}tjs9|djr,tj|tjddd ntj|d | dkr1| rtj||| ntj||| }tj||tj|||d |z | r{ttt|}tj||d |z tj|||d }tj|tj||n)tj|}tj||| dkrttt|}tj|| tj||||rGt7|tj r-tj|| } tj|| ;tj||| U|rHt7|tj r.tj|| tj|||tj|||| dS) Nrz#_foreach ops don't support autogradc3nK|]/\}}|jj|jjko |jjvV0dSra)r6rq).0rCr3r~s r0 z(_multi_tensor_rmsprop..is]  4 HMT[- - > !==      r1rhrig?cpu)r6rjrrkrm)r=r>rorprallzipr"_group_tensors_by_device_and_dtypevaluesrrrrNr _foreach_negis_cpu _foreach_add_rA _foreach_add _foreach_mul__foreach_addcmul__foreach_lerp__foreach_addcmul_foreach_sqrt_ _foreach_sqrt_foreach_addcdiv_r' _foreach_mul _foreach_div_)"rrUrVrXrWrYrrrrr r%r"r#r&rZgrouped_tensorsgrouped_params_grouped_grads_grouped_square_avgs_grouped_grad_avgs_grouped_momentum_buffer_list_grouped_state_steps__grouped_params grouped_gradsgrouped_square_avgsgrouped_state_stepsstate_and_gradsgrouped_momentum_buffer_listgrouped_grad_avgsr momentum_lrr~s" @r0_multi_tensor_rmsproprNs& 6{{aDDDDDD < $ $ & &w:w'H'J'J$    v{33      w w wWs v v v  w w w  B  Y0DkRO  " " $ $YWYW     ) d6lO<<T&\>:: "4<1EFF"4<1EFF  <,.ABO!||/3L"?00, &&'CDDD :$(f7I$J$J!&&'8999 . ;? ; ; ; ;  >!.}==M |((** 8/B1/E/L 8  #U\#e%D%D%DC        3Q 7 7 7 1   #M>VVVVV % 2!>!!!  /777   QY      * $T&\3E F F   !2M1u9 M M M(#%68IQSC   % % %  S ) ) ) )%&9::C  S ) ) ) a<<+/V ;,, (   r)rts r0rzrmsprop..s?33() 1el##333333r1zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNF) use_fusedz6torch.jit.script not supported with foreach optimizers) rrrrr r%r"r&r#rZ) r>rorprrPr jit is_scriptingrr)rrUrVrXrWrYr!r"r#r&rZrrrrr r%rfuncs r0rrs0: < $ $ & & s33-833300  ^   1 Ne   7U59))++USTTT&uy--//&$%D   !%!r1)NFFFF)__doc__typingrrrrr>r optimizerr r r r r rrrrrrr__all__rr@rerrrrGr1r0rsj10............                              i gggggigggV*V    W<BD1 LD1 <D1fD1F| D1 v, D1 f D1 D1 D1 D1D1D1D1D1D1 !D1"#D1D1D1D1N}W L}W <}Wf}WF| }W v, }W f }W }W }W }W}W}W}W}W}W !}W"#}W}W}W}W@ 1GHHH# AA LA <AfAF| A v, A f Ad^AAAAA A !A" #A$%A&'A()AAAIHAAAr1