Ngq^bdZddlmZddlmZmZmZddlZddlm Z ddl m cm Z ddl mZmZddlmZmZmZmZmZmZddlmZdd lmZdd lmZmZd gZGd d e jZ Gdde jZ!Gdde jZ"Gdde jZ#Gdde jZ$Gdd e jZ%d4dZ&d5dZ'ee'de'de'de'de'de'de'ddddd e'ddddd e'ddddd e'ddddd e'dd!e'dd!d" Z(d#e%fd$Z)d6d&Z*ed6d'e%fd(Z+ed6d'e%fd)Z,ed6d'e%fd*Z-ed6d'e%fd+Z.ed6d'e%fd,Z/ed6d'e%fd-Z0ed6d'e%fd.Z1ed6d'e%fd/Z2ed6d'e%fd0Z3ed6d'e%fd1Z4ed6d'e%fd2Z5ed6d'e%fd3Z6dS)7a[ FocalNet As described in `Focal Modulation Networks` - https://arxiv.org/abs/2203.11926 Significant modifications and refactoring from the original impl at https://github.com/microsoft/FocalNet This impl is/has: * fully convolutional, NCHW tensor layout throughout, seemed to have minimal performance impact but more flexible * re-ordered downsample / layer so that striding always at beginning of layer (stage) * no input size constraints or input resolution/H/W tracking through the model * torchscript fixed and a number of quirks cleaned up * feature extraction support via `features_only=True` )partial)CallableOptionalTupleNIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)MlpDropPath LayerNorm2d trunc_normal_ClassifierHeadNormMlpClassifierHead)build_model_with_cfg) named_apply)generate_default_cfgsregister_modelFocalNetcTeZdZdddddefdedededed ed ed ed effd ZdZ xZ S)FocalModulationTFdim focal_level focal_factorbias use_post_normnormalize_modulator proj_drop norm_layerc t||_||_||_||_||_||_|||jdzg|_tj |d|z|jdzzd||_ tj ||d||_ tj |_tj ||d|_tj||_tj|_g|_t+|jD]} |j| z|jz} |jtjtj ||| || dzdtj |j| |jr | |ntj|_dS)Nrr) kernel_sizer)r#F)r#groupspaddingr)super__init__r focal_windowrrrr input_splitnnConv2dfhGELUactprojDropoutr ModuleList focal_layers kernel_sizesrangeappend SequentialIdentitynorm) selfrr(rrrrrr r!kr# __class__s P/var/www/html/ai-engine/env/lib/python3.11/site-packages/timm/models/focalnet.pyr'zFocalModulation.__init__%s (&(*#6 d&6&:;3C4+;a+? @aVZ[[[3>>>799Ic3A666 I..MOOt'(( 2 2A+a/$2CCK   $ $R] #s CQ\`aQahmnnn &&      $ $[ 1 1 1 1'+'9LJJsOOOr{}} cF||}tj||jd\}}}d}t |jD]'\}}||}|||dd||dzfzz}(||dd}|||dd|jdfzz}|j r ||jdzz }|| |z} | | } | | } | | } | S)Nrr)rT)keepdim)r,torchsplitr) enumerater3r/meanrrr-r9r0r ) r:xqctxgatesctx_alll focal_layer ctx_globalx_outs r=forwardzFocalModulation.forwardMs= FF1II At'7;; 3'(9:: 8 8NA{+c""CeAAAqQwJ&7 77GGXXchhvth<<== Jqqq$2B2C2C/C)DDD  # 7!1A!56GDFF7OO# %   %  u%% r>) __name__ __module__ __qualname__r intboolfloatrr'rO __classcell__r<s@r=rr$s !""'(-!#.&M&M&M &M  &M  &M &M"&&M&M!&M&M&M&M&M&MPr>rc&eZdZdfd ZdZxZS) LayerScale2dh㈵>Fct||_tj|t j|z|_dSN)r&r'inplacer* ParameterrBonesgamma)r:r init_valuesr]r<s r=r'zLayerScale2d.__init__isB  \+ 3"?@@ r>c~|jdddd}|jr||n||zS)Nr)r`viewr]mul_)r:rFr`s r=rOzLayerScale2d.forwardns: 2q!,, $ ;qvve}}}!e);r>)rZF)rPrQrRr'rOrVrWs@r=rYrYhsRAAAAAA <<<<<<rYc|eZdZdZdddddddddejef ded ed ed ed e d e de dededede de ffd Z dZ xZ S) FocalNetBlockz% Focal Modulation Network Block. @rr@F-C6?rr mlp_ratiorr(ruse_post_norm_in_modulationrlayerscale_valuer drop_path act_layerr!c t||_||_||_||_||_|s | |ntj|_ t|||j||| | |_ |r | |ntj|_ |t||ntj|_| dkrt| ntj|_|s | |ntj|_t%|t'||z| | d|_|r | |ntj|_|t||ntj|_| dkrt| ntj|_dS)ap Args: dim: Number of input channels. mlp_ratio: Ratio of mlp hidden dim to embedding dim. focal_level: Number of focal levels. focal_window: Focal window size at first focal level. use_post_norm: Whether to use layer norm after modulation. use_post_norm_in_modulation: Whether to use layer norm in modulation. layerscale_value: Initial layerscale value. proj_drop: Dropout rate. drop_path: Stochastic depth rate. act_layer: Activation layer. norm_layer: Normalization layer. )r(rrrr r!NrT) in_featureshidden_featuresrndropuse_conv)r&r'rrjr(rrr*r8norm1r modulation norm1_postrYls1r drop_path1norm2r rSmlp norm2_postls2 drop_path2)r:rrjrr(rrkrrlr rmrnr!r<s r=r'zFocalNetBlock.__init__ws: "(&*,9LZZ___r{}} ) %(5 3!   .;M**S/// :J:V<%5666\^\g\i\i1:R(9---R[]],9LZZ___r{}} i00    .;M**S/// :J:V<%5666\^\g\i\i1:R(9---R[]]r>c |}||}||}||}||||z}|||||| |z}|Sr\) rtrurvrxrwr}r|r{rzry)r:rFshortcuts r=rOzFocalNetBlock.forwards JJqMM OOA   OOA   ttxx{{33 3 $**Q--9P9P)Q)Q R RSS Sr>)rPrQrR__doc__r*r.r rSrUrTrr'rOrVrWs@r=rgrgss  " !"'05(-&*!!"$'#.=S=S=S=S =S  =S  =S*.=S"&=S$=S=S=S =S!=S=S=S=S=S=S~       r>rgceZdZdZdddddddddddef ded ed ed ed ed edededededededededeffd Z e j j ddZ dZxZS) FocalNetStagez4 A basic Focal Transformer layer for one stage. rhTrFrirrout_dimdepthrj downsamplerr(use_overlap_downrrkrrlr rmr!c h t||_||_d|_|rt |d||_ntj|_tj    f dt|D|_ dS)a8 Args: dim: Number of input channels. out_dim: Number of output channels. depth: Number of blocks. mlp_ratio: Ratio of mlp hidden dim to embedding dim. downsample: Downsample layer at start of the layer. focal_level: Number of focal levels focal_window: Focal window size at first focal level use_overlap_down: User overlapped convolution in downsample layer. use_post_norm: Whether to use layer norm after modulation. use_post_norm_in_modulation: Whether to use layer norm in modulation. layerscale_value: Initial layerscale value proj_drop: Dropout rate for projections. drop_path: Stochastic depth rate. norm_layer: Normalization layer. Fr)in_chsout_chsstrideoverlapr!c~ g|]9}t   ttr|n :S)) rrjrr(rrkrrlr rmr!)rg isinstancelist) .0irmrr(rlrjr!rrr rrks r= z*FocalNetStage.__init__..sr%#%#%# #')+,G$7!1#*4Y*E*ET)A,,9%   %#%#%#r>N) r&r'rrgrad_checkpointing Downsamplerr*r8r2r5blocks)r:rrrrjrrr(rrrkrrlr rmr!r<s ` ` `` ```````r=r'zFocalNetStage.__init__sF  "'  ,((% DOO!kmmDOm%#%#%#%#%#%#%#%#%#%#%#%#%#%#5\\%#%#%#$$ r>c||_dSr\)r)r:enables r=set_grad_checkpointingz$FocalNetStage.set_grad_checkpointing s"(r>c||}|jD]H}|jr4tjst j||}=||}I|Sr\)rrrrBjit is_scripting checkpoint)r:rFblks r=rOzFocalNetStage.forwardsl OOA  ;  C& uy/E/E/G/G )#q11CFFr>T)rPrQrRrr rSrUrTrr'rBrignorerrOrVrWs@r=rrsR "# !%*"'05(-&*!!#.!B$B$B$B$ B$  B$  B$B$B$#B$ B$*.B$"&B$$B$B$B$ !!B$B$B$B$B$B$H Y))))r>rc NeZdZ d dededededeef fd Zd ZxZ S) rFNrrrrr!c"t||_d}|}|r|dvsJ|dkrd\}}n |dkrd\}}tj||||||_| ||ntj|_dS) a Args: in_chs: Number of input image channels. out_chs: Number of linear projection output channels. stride: Downsample stride. overlap: Use overlapping convolutions if True. norm_layer: Normalization layer. r)rrr)rr)r@r)r#rr%N)r&r'rr*r+r0r8r9) r:rrrrr!r%r#r<s r=r'zDownsample.__init__s"    ,V####{{'+$ WW1'+$ WIfg;v_fggg +5+AJJw'''r{}} r>cZ||}||}|Sr\)r0r9r:rFs r=rOzDownsample.forward;s% IIaLL IIaLLr>)rFN) rPrQrRrSrTrrr'rOrVrWs@r=rrs !-1 UUUU U  U !* UUUUUU<r>rc'eZdZdZdddddddd d d d d d d d d d deedfdededededeedfde deedfdeedfde de de de de ede d e e d!e d"e d#e d$e f&fd% Z ejjd&Zejjd1d'Zejjd2d)Zejjd*ejfd+Zd3dede efd,Zd-Zd1d.e fd/Zd0ZxZS)4rz," Focal Modulation Networks (FocalNets) r@avg`rrrrh)rrrrr@r@r@r@FN?rg?rZ)epsin_chans num_classes global_pool embed_dimdepths.rj focal_levels focal_windowsrrrkrhead_hidden_sizehead_init_scalerl drop_rateproj_drop_ratedrop_path_rater!cvtt||_fdt |jD||_|_dx|_|_g|_ t|d| ||_ d}dtj d|t|D}g}t |jD]}|}t||||||dk||||| | | | |||t|d|t|d|dz|}|}||gz }|xj t!|d d |zzd | gz c_ t#j||_| r?t#j|_| |_t-|j|| ||| |_n2||j|_t1|j||||_t3t5t6||dS)a` Args: in_chans: Number of input image channels. num_classes: Number of classes for classification head. embed_dim: Patch embedding dimension. depths: Depth of each Focal Transformer layer. mlp_ratio: Ratio of mlp hidden dim to embedding dim. focal_levels: How many focal levels at all stages. Note that this excludes the finest-grain level. focal_windows: The focal window size at all stages. use_overlap_down: Whether to use convolutional embedding. use_post_norm: Whether to use layernorm after modulation (it helps stablize training of large models) layerscale_value: Value for layer scale. drop_rate: Dropout rate. drop_path_rate: Stochastic depth rate. norm_layer: Normalization layer. c g|] }d|zz S)r)rrrs r=rz%FocalNet.__init__..ns"JJJaY!q&)JJJr>rcr)rrrr!c6g|]}|Sr)item)rrFs r=rz%FocalNet.__init__..}s PPPAqvvxxPPPr>Nr)rrrrjrrr(rrrkrrlr rmr!rrlayers.)num_chs reductionmodule) hidden_size pool_typerr!)rr)r)r&r'len num_layersr5rr num_featuresr feature_inforstemrBlinspacesumrdictr*r7layersr8r9rheadrrr _init_weights)r:rrrrrrjrrrrrkrrrrlrrrr!in_dimdprri_layerrlayerr<s ` r=r'zFocalNet.__init__EsL f++JJJJ53I3IJJJ &"4=bMAD1aL$!     1PP>3v;;!O!OPPPT_-- q qG(G!Wo#"Q;(1*73!1+,G$7!1(c&'"233C|! |8L4M4MMN%E"F ug F   $w!a7lBR[nel[n[n"o"o"o!p p   mV,   DI$4D !-!,%#% DII# 4#455DI&!%# DI GM?KKKTRRRRRr>cdhS)Nrr:s r=no_weight_decayzFocalNet.no_weight_decays t r>c4td|rddgngdS)Nz^stem)z^layers\.(\d+)Nz^norm)i))z^layers\.(\d+).downsample)r)z^layers\.(\d+)\.\w+\.(\d+)Nr)rr)r)r:coarses r= group_matcherzFocalNet.group_matchersE)$    r>TcT||_|jD]}||dS)N)r)rrr)r:rrKs r=rzFocalNet.set_grad_checkpointings?"( 4 4A $ $F $ 3 3 3 3 4 4r>returnc|jjSr\)rfcrs r=get_classifierzFocalNet.get_classifiers y|r>c>|j||dS)N)r)rreset)r:rrs r=reset_classifierzFocalNet.reset_classifiers   {;;;;;r>c||}||}||}|Sr\)rrr9rs r=forward_featureszFocalNet.forward_featuress4 IIaLL KKNN IIaLLr> pre_logitsc^|r|||n||S)N)r)r)r:rFrs r= forward_headzFocalNet.forward_heads-6@Rtyyzy222diiPQllRr>cZ||}||}|Sr\)rrrs r=rOzFocalNet.forwards-  ! !! $ $   a r>Frr\)rPrQrRrrr rSstrrrUrTrrr'rBrrrrrr*ModulerrrrrOrVrWs@r=rrAs #$&2!,8-9%*"'05(-.2%(04 #%#&#*7;D#A#A#A)gSgSgSgS gS  gS #s(O gSgS S/gS!c?gS#gS gS*.gS"&gS'smgS#gS 'uo!gS"#gS$!%gS&!'gS(!)gSgSgSgSgSgSR Y Y       Y4444  Y <rct|tjrEt|jd|j&tj|jdSdSt|tjrt|jd|j$tj|j|rFd|vrD|jj ||jj |dSdSdSdS)Ng{Gz?)stdhead.fc) rr*r+r weightrinitzeros_Lineardatare)rnamers r=rrs&")$$ 3fm---- ; " GNN6; ' ' ' ' ' # " FBI & &3fm---- ; " GNN6; ' ' '  3I%% M  # #O 4 4 4 K  ! !/ 2 2 2 2 2 33 3 3%%r>rc 6|dddddttdddd |S) Nr)r@r)rrg?bicubicz stem.projrmit) urlr input_size pool_sizecrop_pct interpolationrEr first_conv classifierlicenser)rkwargss r=_cfgrs8=v%.B!    #  r>ztimm/) hf_hub_id)r@r) riRU)rrrrr)rr) zfocalnet_tiny_srf.ms_in1kzfocalnet_small_srf.ms_in1kzfocalnet_base_srf.ms_in1kzfocalnet_tiny_lrf.ms_in1kzfocalnet_small_lrf.ms_in1kzfocalnet_base_lrf.ms_in1kzfocalnet_large_fl3.ms_in22kzfocalnet_large_fl4.ms_in22kzfocalnet_xlarge_fl3.ms_in22kzfocalnet_xlarge_fl4.ms_in22kzfocalnet_huge_fl3.ms_in22kzfocalnet_huge_fl4.ms_in22kmodelc|d|}d|vr|Sddl}i}|}|D]\}}|dd|}|dd}|dd |}d |vr||vr|d d |}|d d}|dd}||vrf|||kr6||j|jkr |||j}|||<|S)Nrzstem.proj.weightrz gamma_([0-9])z ls\1.gamma patch_embedrzlayers.(\d+).downsamplecTdt|ddzdS)Nrrz .downsample)rSgroup)rFs r=z&checkpoint_filter_fn.."s(9c3qwwqzz??UVCV9c9c9cr>r9z norm([0-9])z norm\1_postzln.znorm.rr) getre state_dictitemssubreplacenumelshapereshape)rrrout_dict dest_dictr;vs r=checkpoint_filter_fnrs^44JZ'' IIIH  ""I  ""  1 FF#]A 6 6 IImV , , FF-/c/cef g g Q;;1I--~~q99A IIeW % % IIfi ( ( >>il0022aggii??IaLDVZ[ZaDaDa )A,,--A Or>Fc tdt|ddD}|d|}t t ||ft td|d|}|S)Nc3 K|] \}}|V dSr\r)rr_s r= z#_create_focalnet...s&\\da\\\\\\r>r)rrr@r out_indicesT)flatten_sequentialr)pretrained_filter_fn feature_cfg)tuplerDr poprrrr)variant pretrainedrdefault_out_indicesrrs r=_create_focalnetr'-s\\i 8\8Z8Z.[.[\\\\\**],?@@K ': 1DkJJJ      E Lr>rc @tdgddd|}tdd|i|S)Nrrrrfocalnet_tiny_srfr%r)r*rr'r%r model_kwargss r=r*r*9s9D|||rDDVDDL  W WJ W, W WWr>c @tdgddd|}tdd|i|S)Nrrrrr)focalnet_small_srfr%r)r1r+r,s r=r1r1?s9E}}}EEfEEL  X XZ X< X XXr>c @tdgddd|}tdd|i|S)Nr/r)focalnet_base_srfr%r)r4r+r,s r=r4r4Es9F}}}FFvFFL  W WJ W, W WWr>c Ftdgddgdd|}tdd|i|S) Nrrrrrrfocalnet_tiny_lrfr%r)r7r+r,s r=r7r7Ks@_|||r __X^__L  W WJ W, W WWr>c Ftdgddgdd|}tdd|i|S) Nr/rrr6focalnet_small_lrfr%r)r9r+r,s r=r9r9Qs@`}}}``Y_``L  X XZ X< X XXr>c Ftdgddgdd|}tdd|i|S) Nr/r3rr6focalnet_base_lrfr%r)r;r+r,s r=r;r;Ws@a}}},,,aaZ`aaL  W WJ W, W WWr>c Vtd gddgddgdzdddd|}td d |i|S) Nr/rrTrirrrrrrrlfocalnet_large_fl3r%r)r@r+r,s r=r@r@^seT}},,,WXVY\]V]TDTTLRTTL  X XZ X< X XXr>c Ltd gddgddddd|}td d|i|S) Nr/r=rrrrTrirrrrrrlfocalnet_large_fl4r%r)rDr+r,s r=rDrDfsYT}},,,TDTTLRTTL  X XZ X< X XXr>c Vtd gddgddgdzdddd|}td d |i|S) Nr/rr>rTrir?focalnet_xlarge_fl3r%r)rGr+r,s r=rGrGnseT}},,,WXVY\]V]TDTTLRTTL  Y Yj YL Y YYr>c Ltd gddgddddd|}td d|i|S) Nr/rFrBTrirCfocalnet_xlarge_fl4r%r)rIr+r,s r=rIrIvsYT}},,,TDTTLRTTL  Y Yj YL Y YYr>c Xtd gddgddgdzddddd|}td d |i|S) Nr/`rr@rTri)rrrrrrkrrlfocalnet_huge_fl3r%r)rLr+r,s r=rLrL~shv}},,,WXVY\]V]tfjvvntvvL  W WJ W, W WWr>c Ntd gddgdddddd|}td d|i|S) Nr/rKrBTri)rrrrrkrrlfocalnet_huge_fl4r%r)rNr+r,s r=rNrNs\v}},,,tfjvvntvvL  W WJ W, W WWr>)Nr)rr)7r functoolsrtypingrrrrBtorch.nnr*torch.utils.checkpointutilsr timm.datarr timm.layersr r r r rr_builderr _manipulater _registryrr__all__rrrYrgrrrrr default_cfgsrr'r*r1r4r7r9r;r@rDrGrIrLrNrr>r=r[sc  &,,,,,,,,,, +++++++++AAAAAAAAhhhhhhhhhhhhhhhh******$$$$$$<<<<<<<< ,AAAAAbiAAAH<<<<<29<<<OOOOOBIOOOdSSSSSBISSSl########LXXXXXryXXXv 3 3 3 3%$!%""""&$###!%"""!%""""&$###!%"""$(4 HsPU$W$W$W$(4 HsPU$W$W$W%)D HsPU%W%W%W%)D HsPU%W%W%W#'$####'$###; & &   FH*    XXXXXXX YYhYYYY XXXXXXX XXXXXXX YYhYYYY XXXXXXX YYhYYYYYYhYYYYZZxZZZZZZxZZZZXXXXXXXXXXXXXXXXr>