Ng=@dZddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0ej1e2Z3dhZ4 d6de j5fdZ6dZ7d7dZ8 d8de j5d#e9d$ee:d%e9d&ee9d'e;f d(Z d:d/e:d0ee9d#e9d1e9d2ee;d3e;d%ee9d4eefd5Z?dS);z\ Optimizer Factory w/ Custom Weight Decay Hacked together by / Copyright 2021 Ross Wightman N)islice)OptionalCallableTuple)group_parameters) AdaBelief) Adafactor) Adahessian)AdamP)Adan)Lamb)Lars)Lion) Lookahead)MADGRAD)Nadam)NAdamW) NvNovoGrad)RAdam) RMSpropTF)SGDP)SGDWlionh㈵>modelct|}g}g}|D]\\}}|js |jdks|ds||vr||G||]|dd||dgS)Nrz.bias)params weight_decay)setnamed_parameters requires_gradndimendswithappend)rr!no_weight_decay_listdecayno_decaynameparams T/var/www/html/ai-engine/env/lib/python3.11/site-packages/timm/optim/optim_factory.pyparam_groups_weight_decayr.*s 344 EH--//  e"   :??dmmG44?@T8T8T OOE " " " " LL    R00,77 99cLttfddS)Nc>ttSN)tupleritsizesr-z_group..BsfR..//r/r)iterr4s``r-_groupr9@s, bB ///// 4 44r/ c d}t|didd}g}g}|D];\}}|||r||n||z0_layer_map.._in_head..Js%888c S))888r/) isinstancer3listanyr>)rAhps` r-_in_headz_layer_map.._in_headFs[ $4 UDM * * $8888R88899 9<<## #r/pretrained_cfg classifierc$i|] \}}|D]}||Srr)r?ilrAs r- z_layer_map..[s+DDD$!Q!DDQADDDDr/ci|]}|Srr)r?rAnum_trunk_groupss r-rMz_layer_map..\s>>>aa)>>>r/) getattrgetr#r'lenrDr9 enumerateupdate) rlayers_per_group num_groupsrG head_prefix names_trunk names_headrA_num_trunk_layers layer_maprOs @r- _layer_mapr]Es/$$$%!1266::<NNKKJ&&((TT1 (K 8 8S !k>P>PQR>S>S;''-*<=vk+;<<==K;''DD)K"8"8DDDI >>>>:>>>??? r/皙??Fr!r( layer_decayend_layer_decayverbosec|t|}i}i}t|dr't||dd}nt |}t |dz} | dz tfdt| D} | D]\} } | j s | j dks| |vrd} d }nd } |}| | }d || fz}||vr| |}||gd ||<||gd ||<||d | ||d | |r6ddl}td||dzt|S)z Parameter groups for layer-wise lr decay & weight decay Based on BEiT: https://github.com/microsoft/unilm/blob/master/beit/optim_factory.py#L58 group_matcherF)coarseT)reverserc3(K|] }|z zV dSr2r)r?rKr` layer_maxs r- z+param_groups_layer_decay..xs,RR1 A 6RRRRRRr/r*rr)z layer_%d_%s)lr_scaler! param_names)rjr!r rkr rNzparameter groups: %s)indent)r"hasattrrrdr]maxvaluesrDranger#r$r%rQr'json_loggerinfodumps)rr!r(r`rarbparam_group_names param_groupsr\ num_layers layer_scalesr+r,g_decay this_decaylayer_id group_name this_scalerrrhs ` @r-param_groups_layer_decayr`s344Luo&&&$UE,?,?u,?,M,MW[\\\ u%% Y%%''((1,JQIRRRRRj@Q@QRRRRRL--//99 e"   :??d&::: GJJG%J==y11"h%88 \ ) )%h/J& *!-- j ) ' *((L $ *%m4;;DAAAZ *11%8888Y  - ;LUV 0W0WWXXX  ##%% & &&r/ct|j|j|j|j}t |dd |j|d<t |dd |j|d<t |dd |j|d<t |dd| |j t |d d |j |d <|S) z cfg/argparse to kwargs helper Convert optimizer args in argparse args or cfg like object to keyword args for updated create fn. )optlrr!momentumopt_epsNeps opt_betasbetasr`opt_args opt_foreachforeach) dictrrr!rrPrrr`rTrr)cfgkwargss r-optimizer_kwargsrs G 6% F sIt$$0 u sK&&2-wsM4((4 #}sJ%%1 cl###sM4((4Oy Mr/Tc@t|fit|d|iS)zk Legacy optimizer factory for backwards compatibility. NOTE: Use create_optimizer_v2 for new code. )rfilter_bias_and_bn)create_optimizer_v2r)argsrrs r-create_optimizerrsE     t $ $ $  .   r/sgdr?rrrrrparam_group_fnc  rt|tjryi} t|dr|} |r ||} nG|t |||| } d}n/|r|rt ||| } d}n|} n|} |} | d} | d} | drL dd l m }m }m}m}d }n#t $rd }YnwxYw|rt"js Jd | d rD ddl}d }n#t $rd }YnwxYw|rt"js Jdt+d\d|i| }||d|| |t.vr|dd n||d<| dks| dkr-|ddt3j| f|d d|}n| dkr-|ddt3j| f|d d|}n| dkrt7| f|d d|}n| dks| dkr(|ddt9| f|d d|}n| dkr(|ddt9| f|d d|}nU| dkrt3j| fi|}n;| dkrt3j| fi|}n!| dkrt?| fdd d|}n | d kr3 t3j | fi|}n#tB$rtA| fi|}YnwxYw| d!krtE| fi|}n| d"krtG| fi|}n| d#krt3j$| fi|}n| d$krtK| fd%d i|}nu| d&krtK| fd%d i|}n^| d'krt3j&| fi|}nD| d(kr*|dd)t3j'| fi|}n| d*krtQ| fi|}n| d+krtS| fd,d i|}n| d-krtS| fd,d i|}n| d.krtU| fi|}n| d/krtU| fd0d i|}n| d1krtW| f|d d2|}n| d3krtW| fd|i|}nv| d4krtW| f|d d d5|}n]| d6krtW| f|d d|}nE| d7krtY| fd|i|}n.| d8krtY| f|d d9|}n| d:ks| d;krt[| fi|}n| d|}n| d?krt_| fd=|d>|}n| d@kr%|ddta| fi|}n| dAkrtc| fi|}n| dBkr$|dd|| f|d d|}n\| dCkr$|dd|| f|d d|}n2| dDkr || fdEd i|}n| dFkr || fdEd i|}n | dGkr || fi|}n| dHkr!|dIdJ|| fi|}n| dKkr.|dd|jj| f|d d|}n| dLkr.|dd|jj2| f|d d|}nl| dMkr-|dd|jj| fd|i|}n9| dNkr-|dd|jj2| fd|i|}n| dOkr|jj| fi|}n| dPkr|jj3| fi|}n| dQkr|jj| fi|}n| dRkr|jj4| fi|}n| dSkr|jj5| fi|}n| dTkr|jj6| fi|}nj| dUkr|jj7| fi|}nP| dVkr|jj6| fi|}n6| dWkr|jj0| fi|}n| dXkr|jj8| fi|}nJtu| dZkr| dd[krtw|}|S)]a Create an optimizer. TODO currently the model is passed in and all parameters are selected for optimization. For more general use an interface that allows selection of parameters to optimize and lr groups, one of: * a filter fn interface that further breaks params into groups in a weight_decay compatible fashion * expose the parameters interface and leave it up to caller Args: model_or_params (nn.Module): model containing parameters to optimize opt: name of optimizer to create lr: initial learning rate weight_decay: weight decay to apply in optimizer momentum: momentum for momentum based optimizers (others may use betas via kwargs) foreach: Enable / disable foreach (multi-tensor) operation if True / False. Choose safe default if None filter_bias_and_bn: filter out bias, bn and other 1d params from weight decay **kwargs: extra optimizer specific kwargs to pass through Returns: Optimizer no_weight_decayN)r!r`r(rrZfusedr) FusedNovoGrad FusedAdam FusedLAMBFusedSGDTFz+APEX and CUDA required for fused optimizersbnbz1bitsandbytes and CUDA required for bnb optimizersr!rrrnesterovr)rrrsgdpsgdw nesterovw momentumwadamadamwadampg{Gz?)wd_ratiornadamnadamwradamadamax adabeliefrectify radabeliefadadeltaadagradg:0yE> adafactoradanpno_proxadanwlamblambc trust_cliplarc)rrlarsnlarc)rrrnlarsmadgradmadgradw)rdecoupled_decaynovograd nvnovogradrmspropr)alphar rmsproptfr adahessianfusedsgd fusedmomentum fusedadam adam_w_mode fusedadamw fusedlamb fusednovogradr)gffffff?g\(\?bnbsgd bnbsgd8bit bnbmomentumbnbmomentum8bitbnbadam bnbadam8bitbnbadamw bnbadamw8bitbnblamb bnblamb8bitbnblars bnblarsb8bitbnblion bnblion8bitzInvalid optimizerr lookaheadr)apex.optimizersrrrr ImportErrortorchcuda is_available bitsandbytesr setdefault_DEFAULT_FOREACHpopoptimSGDrrAdamAdamWr rAttributeErrorrrAdamaxr AdadeltaAdagradr r rrrrRMSproprrr SGD8bitAdam8bit AdamW8bitLAMBLAMB8bitLARSLion8bit ValueErrorrRr)model_or_paramsrrr!rrrr`rrrr opt_lower opt_splitrrrrhas_apexrhas_bnbr optimizers r-rrs @/29--% ?$5 6 6 @-==??O  6'88JJ  $1)'%4 J LL  60 62?LRabbJLL(3355JJ%  I$$I" IG$$e  U U U U U U U U U U U UHH   HHH dEJ3355dd7ddddE""j  & & & &GG   GGG i5:2244ii6iiii88888H ~D"%%% " " "    4 0 0 0%EY*44 UD!!!IjW8dWWhWW j  UD!!!IjX8eXXxXX f  RhRRRR f   [ 8 8 UD!!!RhRRRR k ! ! UD!!!ShSS(SS  f  Jz66X66 g  K 77h77 g  *OtdOOhOO g   6 J;;(;;II 6 6 6j55H55III 6 h  :2222 g  *1111 h  L88x88 k ! !jDD%D8DD l " "jCC$C(CC j N::::: i  E4(((M*9999 k ! !j55H55 g  ??U?h?? g  >>T>X>> f  00x00 g  AAAAA f  Th4TT8TT f  CChC(CC g  ch4RVccZbcc g  RhRRRR i  JFFFXFF j J\4\\S[\\ j I$=$=z66X66 i  M*WC(WWhWW k ! !jShSS(SS f   UD!!!00x00  l " "z66X66  j  UD!!!HZV(TVVXVV o % % UD!!!HZW(UWWhWW k ! !IjHHeHxHH l " "IjGGdGhGG k ! !Ij55H55 o % %G\222!M*9999  h   UD!!!!CIM*[x$[[RZ[[ l " " UD!!!%CI%j_8d__V^__ m # # UD!!!!CIM*LLxL8LL ' ' ' UD!!!%CI%jPP8PxPP i  "CIN::::: m # #&CI&z>>X>> j #CIOJ;;(;; n $ $'CI' ??h?? i  "CIN::::: m # #&CI&z>>X>> i  "CIN::::: n $ $&CI&z>>X>> i  "CIN::::: m # #&CI&z>>X>>  -, 9~~ Q<; & &!),,I s6C,, C;:C;>E EELL65L6)rr)r:N)r^rr_NF)T)rNrrNTNN)@__doc__logging itertoolsrtypingrrrrtorch.nnr torch.optimr timm.modelsrrr rr rr rr adanr rrrrrrrrrrrrrrrrrr rmsprop_tfrrrrr getLogger__name__rsrrr.r9r]floatstrboolrrrrrr/r-rse,,,,,,,,,, (((((( """""" """"""!!!!!! ' H % %  99y9999,555 :#+- +/ ='='y='='$Cj=' =' "% ='  ='='='='@." "&#''+-1ee e UOe e  e $ e!ee_e!*eeeeeer/