ΧgDldZddlmZmZmZmZmZddlZddlmZddl m Z m Z m Z m Z mZmZmZmZmZmZmZmZddgZGd deZd d ed e d ed e d ze_deedeedeedeedeededededededededefdZdeedeedeedeedeededededededededefdZe e d!deedeedeedeedeedeededededededededefd ZdS)"z1Implementation for the Resilient backpropagation.)castListOptionalTupleUnionN)Tensor) _capturable_doc_default_to_fused_or_foreach_differentiable_doc_disable_dynamo_if_unsupported _foreach_doc!_get_capturable_supported_devices_get_scalar_dtype _maximize_doc_use_grad_for_differentiable _view_as_real OptimizerParamsTRproprpropceZdZ dddddddedeeefd eeefd eeefd ed e ed edeffdZ fdZ dZ e ddZxZS)r{Gz?g?g333333?gư>2FN) capturableforeachmaximizedifferentiableparamslretas step_sizesrrrr c t|tr'|dkrtdd|kstd|d|dcxkrdcxkr |dks#ntd|dd|dt ||||||| } t || dS) Nr zTensor lr must be 1-elementgzInvalid learning rate: r?zInvalid eta values: z, )r"r#r$rrr r) isinstancernumel ValueErrordictsuper__init__) selfr!r"r#r$rrrr defaults __class__s M/var/www/html/ai-engine/env/lib/python3.11/site-packages/torch/optim/rprop.pyr,zRprop.__init__s b& ! ! is_complexappend is_sparse RuntimeErrorr;r=zerosrr6 zeros_likepreserve_formatr5 full_likecomplex) r-rBr!gradsprevsr$ state_steps has_complexrCrLr;s r0 _init_groupzRprop._init_groupNs x . .Av~ 5+A.. .K MM!   6D~ N"#LMMM LL   JqME5zzQ\*DEK*;*=*=ahOOOOR/@/B/BCCCf !& 0%BW X X Xf 7%L*/geDk5;??**E+&&*/uT{)K)KE+& LLv ' ' '   eK0 1 1 1   uV} - - - -r1c|d}|5tj5|}dddn #1swxYwY|jD]u}g}g}g}g}g}|d\} } |d\} } |d} |d}|||||||}t |||||| | | | | ||d|d|v|S) zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nr#r$rrr r) step_size_min step_size_maxetaminusetaplusrrr rrY) _cuda_graph_capture_health_checkr> enable_gradr9rZr)r-closurelossrBr!rVrWr$rXr^r_r\r]rrrYs r0r3z Rprop.steptsj --///  "$$ ! !wyy ! ! ! ! ! ! ! ! ! ! ! ! ! ! !&  E#%F"$E"$E')J(*K %f Hg+0+> (M=I&GZ(H**vueZK ++!!$%56 .'     " s AA A)rrrN)__name__ __module__ __qualname__rrr@rrboolrr,r8rZrr3 __classcell__)r/s@r0rrs$($.*4 +!"&$++++ %- +E5L! + %,' ++$++++++++<&$$$L"///"!/////r1a Implements the resilient backpropagation algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \theta_0 \in \mathbf{R}^d \text{ (params)},f(\theta) \text{ (objective)}, \\ &\hspace{13mm} \eta_{+/-} \text{ (etaplus, etaminus)}, \Gamma_{max/min} \text{ (step sizes)} \\ &\textbf{initialize} : g^0_{prev} \leftarrow 0, \: \eta_0 \leftarrow \text{lr (learning rate)} \\ &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm} \textbf{for} \text{ } i = 0, 1, \ldots, d-1 \: \mathbf{do} \\ &\hspace{10mm} \textbf{if} \: g^i_{prev} g^i_t > 0 \\ &\hspace{15mm} \eta^i_t \leftarrow \mathrm{min}(\eta^i_{t-1} \eta_{+}, \Gamma_{max}) \\ &\hspace{10mm} \textbf{else if} \: g^i_{prev} g^i_t < 0 \\ &\hspace{15mm} \eta^i_t \leftarrow \mathrm{max}(\eta^i_{t-1} \eta_{-}, \Gamma_{min}) \\ &\hspace{15mm} g^i_t \leftarrow 0 \\ &\hspace{10mm} \textbf{else} \: \\ &\hspace{15mm} \eta^i_t \leftarrow \eta^i_{t-1} \\ &\hspace{5mm}\theta_t \leftarrow \theta_{t-1}- \eta_t \mathrm{sign}(g_t) \\ &\hspace{5mm}g_{prev} \leftarrow g_t \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to the paper `A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm `_. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-2) etas (Tuple[float, float], optional): pair of (etaminus, etaplus), that are multiplicative increase and decrease factors (default: (0.5, 1.2)) step_sizes (Tuple[float, float], optional): a pair of minimal and maximal allowed step sizes (default: (1e-6, 50)) z z r!rVrWr$rXr\r]r^r_rrr rYct|D];\} }|| }| s|n| }|| }|| }|| }tjsF| rDt }|jj|jjkr|jj|vsJd|d|dz }tj|rPtj|}tj|}tj|}tj|}| r:| |  }n'| | }| r| tj |d||| tj |d||| tj |dd|nH|||d<|||d<d||d<||||| tj}| r=| tj ||d|nd|||<|| |d| |=dS)NIIf capturable=True, params and state_steps must be on supported devices: .r rrHvalue) enumerater>_utils is_compilingrr6typerM view_as_realmulclonesigncopy_wheregtlteqmul_clamp_rSaddcmul_)r!rVrWr$rXr\r]r^r_rrr rYiparamrLrJrKr3capturable_supported_devicesrws r0_single_tensor_rproprs f%%115Qx#.tt$QxqM 1~|((** {z {+L+N+N ( !T[%555L%)EEEEz[wzzzFEE    E " " 6%d++D%d++D&u--E*955I  )88DJJLL))..00DD88D>>&&((D  ! JJu{4771::w== > > > JJu{4771::x>> ? ? ? JJu{4771::q$77 8 8 8 8&D 'D  D  t##M=AAAzz(=z>>  ( JJu{4778#4#4a>> ? ? ? ?&'D"" # tyy{{IR888 4c11r1c Vt|dkrdS| r JdtjsG| rEt t fdt ||DsJddtj|||||g} | D]r\\}}}}}}ttt|}ttt|}ttt|}ttt|}ttt|}tjs9|dj r,tj|tjddd ntj|d | rt!||||tj||}| rtj|tj||| rtj||}tj|| r|D]}|tj|d|||tj|d|||tj|dd |nM|D]J}|||d<|||d<d ||d<Ktj|||D]}|||t9|}t;t|D]P}||tj|||d||Q~d |D}tj|||d tdS)Nrz#_foreach ops don't support autogradc3nK|]/\}}|jj|jjko |jjvV0dSrd)r6rs).0rCr3rs r0 z&_multi_tensor_rprop..:s]  4 HMT[- - > !==      r1rkrlr&cpu)r6)alphar c6g|]}|SrG)rw)rrLs r0 z'_multi_tensor_rprop..s <<rqrrrallzipr"_group_tensors_by_device_and_dtypevaluesrrris_cpu _foreach_add_rAr _foreach_mul _foreach_neg__foreach_copy__foreach_sign_rxryrzr{r| _foreach_mul_r~listrange_foreach_addcmul_)r!rVrWr$rXr\r]r^r_rrr rYgrouped_tensorsgrouped_params_grouped_grads_grouped_prevs_grouped_step_sizes_grouped_state_steps__grouped_params grouped_grads grouped_prevsgrouped_step_sizesgrouped_state_stepssignsrwrKr grad_signsrs @r0_multi_tensor_rpropr"s  6{{aDDDDDD < $ $ & &w:w'H'J'J$    v{33      w w wWs v v v  w w w  B z;7O  " " $ $ J J   d6lO<<T&\>:: T&\>:: !$v,0CDD"4<1EFF |((** 8/B1/E/L 8  #U\#e%D%D%DC        3Q 7 7 7    }>P   "=-@@  '   & & & ]M:::  /   . . .%  U###  % = = 5;twwqzz7DAABBB 5;twwqzz8TBBCCC 5;twwqzz1d;;<<<< =  % %#*TWWQZZ #+TWWQZZ #$TWWQZZ   .666+ ; ;I   ]M : : : :]++ s=))**  A !  " " E!HKK111mA6FGG     =r)rts r0rzrprop..s?33() 1el##333333r1zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNF) use_fusedz6torch.jit.script not supported with foreach optimizers)r\r]r^r_rrr rY) r>rqrrrrPr jit is_scriptingrr)r!rVrWr$rXrrrr rYr\r]r^r_rfuncs r0rrs'4 < $ $ & & s33-833300  ^   1 Ne   7U59))++USTTT$uy--//$"#D  ##%r1)NFFFF)__doc__typingrrrrrr>r optimizerr r r r rrrrrrrr__all__rr@rhrrrrGr1r0rs9855555555555555                              G HHHHHIHHHX"F    G2 nA LA <A <AV A f AAAAAAAAAAAAHk Lk <k  <k V k f k k k k k k k k k k k k d 1EFFF# ;; L; <; <;V ; f ;d^;;;;;;; !;"#;;;GF;;;r1