Ng)J dZddlmZmZddlmZddlmZmZddl m Z m Z m Z m Z gZee e ddd ddd e ddd ddd e d ddddd e dddddd fdddddee e ddd ddd e d dgdd ddd e d dgddddd e ddddddfdddddd de! e e ddd ddd e d d"d"d ddd e d d#dddd e d ddddd fdddddde$e e ddd dd%d e ddd dd%d e d dddd%d e ddddd%d fdddddd&ded%'( e e dddddd) e ddd*ddd) e dd+d ddd) e ddd ddd) fdd,dd-ed.d/0e e ddd ddd e ddd ddd e d ddddd e dddddd fdddd-ed.d/0e e ddd ddd e d dgdd ddd e d dgddddd e ddddddfdddddd d-ed.d/! e e ddd ddd e d d"d"d ddd d-ed.dd"12 e d d#dddd e d ddddd fddddd-ed.d/3e e ddd dd%d e ddd dd%d e d dddd%d e ddddd%d fddddd&d-ed.dd%45e e ddd ddd e ddd ddd e d ddddd e dddddd fdddd6ed780e e ddd ddd e d d"d"d ddd e d d#dddd e d ddddd fddddd6ed783e e ddd ddd e ddd ddd e d ddddd e dddddd fdddd6ed980e e ddd:dd%d e dd#d;dd%d e d dddd%dfd?ddddd ed @eddAd-edBddCDE e e ddd ddd e d d"d ddd d6edF8Ge d d#dddd d-edHGe d ddddd deGfddddIe e ddd ddd e d d"d ddd d-edHGe d d#dddd d-edHGe d ddddd deGfddddIJZd~dLZddMZeedNdOddPdQRedSdOddPdQdTUeddPdQVedWdOddPdQRedPdQdPXedYdOdPdQdPZed[dOdPdQdPdT\ed]dOdPdQdPdT\ed^dOdPdQdPdT\ed_dOd`dPdQdTaedbdOd`dPdQcedddOddPdQdTUededOdfdfdgdhdidhdTj edkdOddPdQRedldOddPdQRdmZeddne fdoZeddne fdpZeddne fdqZeddne fdrZeddne fdsZeddne fdtZeddne fduZeddne fdvZeddne fdwZeddne fdxZeddne fdyZeddne fdzZeddne fd{Z eddne fd|Z!eddne fd}Z"d9S)a Bring-Your-Own-Attention Network A flexible network w/ dataclass based config for stacking NN blocks including self-attention (or similar) layers. Currently used to implement experimental variants of: * Bottleneck Transformers * Lambda ResNets * HaloNets Consider all of the models definitions here as experimental WIP and likely to change. Hacked together by / copyright Ross Wightman, 2021. IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)build_model_with_cfg)register_modelgenerate_default_cfgs) ByoBlockCfg ByoModelCfgByobNetinterleave_blocksbottleg?)typedcsgsbri)r self_attni)typesrrrrrri@tieredmaxpoolT bottleneck)blocksstem_chs stem_type stem_poolfixed_input_sizeself_attn_layerself_attn_kwargs)reveryrrrrrigZd;O?)rrrrrsiluise) rrrr act_layer num_features attn_layerr"r#)rrrr r)r!r"r#eca)dim_head) rrrr r!r)r+r"r#g? 7x7halo) block_size halo_size)rrrr r"r#)r6r7 num_heads) rr%rrrrrr"r#)rrrr r)r"r#)r6r7r0)rrrr r)r+r"r#lambda )rN0` i  )rd_ratio) bottle_in linear_outgQ?)r6r7qk_ratio) rrr downsampler*r)r+ attn_kwargs block_kwargsr"r# )rrrrrrr"r#)r7)rrrr r)) botnet26t sebotnet33ts botnet50tseca_botnext26ts halonet_h1 halonet26t sehalonet33ts halonet50tseca_halonext26tslambda_resnet26tlambda_resnet50tslambda_resnet26rpt_256 haloregnetz_blamhalobotnet50tshalo2botnet50tsFc tt||f|s t|n t|tdd|S)NT)flatten_sequential) model_cfg feature_cfg)rr model_cfgsdict)variant cfg_variant pretrainedkwargss O/var/www/html/ai-engine/env/lib/python3.11/site-packages/timm/models/byoanet.py_create_byoanetrdsT * -8U*W%%j>UD111      c 8|dddddttddddd |S) Nir$rhrDrDgffffff?bicubiczstem.conv1.convzhead.fcF) url num_classes input_size pool_sizecrop_pct interpolationmeanstd first_conv classifierr!min_input_sizer)rkrbs rc_cfgrvs:4}SY9%.B'y!]      rezshttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/botnet26t_c1_256-167a0e9f.pthztimm/)r$rr)r5r5)rk hf_hub_idr!rmrnzxhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/sebotnet33ts_a1h2_256-957e3c3e.pthgGz?)rkrwr!rmrnro)r!rmrnzxhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/eca_botnext26ts_c_256-95a898f6.pth)rmrnruzuhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/halonet26t_a1h_256-3083328c.pth)rkrwrmrnruzthttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/sehalonet33ts_256-87e053f9.pth)rkrwrmrnrurozwhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/halonet50ts_a1h2_256-f3a3daee.pthzyhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/eca_halonext26ts_c_256-06906299.pthzyhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/lambda_resnet26t_c_256-e5a5c857.pth)r$r1r1)rkrwrurmrnroz|https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/lambda_resnet50ts_a1h_256-b87370f7.pth)rkrwrurmrnz{https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/lambda_resnet26rpt_c_256-ab00292d.pthzxhttps://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/haloregnetz_c_raa_256-c8ad7616.pth)?rxrxz stem.convrgri) rkrwrqrrrsrmrnruroz}https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/lamhalobotnet50ts_a1h2_256-fe3d9445.pthz{https://github.com/rwightman/pytorch-image-models/releases/download/v0.1-attn-weights/halo2botnet50ts_a1h2_256-fd9c11a3.pth)zbotnet26t_256.c1_in1kzsebotnet33ts_256.a1h_in1kzbotnet50ts_256.untrainedzeca_botnext26ts_256.c1_in1kzhalonet_h1.untrainedzhalonet26t.a1h_in1kzsehalonet33ts.ra2_in1kzhalonet50ts.a1h_in1kzeca_halonext26ts.c1_in1kzlambda_resnet26t.c1_in1kzlambda_resnet50ts.a1h_in1kzlambda_resnet26rpt_256.c1_in1kzhaloregnetz_b.ra3_in1kzlamhalobotnet50ts_256.a1h_in1kzhalo2botnet50ts_256.a1h_in1kreturnc J|ddtdd|i|S)z4 Bottleneck Transformer w/ ResNet26-T backbone. img_sizer botnet26t_256rJra)r|rJ setdefaultrdrarbs rcr|r|\s4 j#&&&  Y YJ YRX Y YYrec tdd|i|S)zY Bottleneck Transformer w/ a ResNet33-t backbone, SE attn for non Halo blocks, SiLU, sebotnet33ts_256rKra)rrKrdrs rcrrds  _ _* _X^ _ __rec J|ddtdd|i|S)z> Bottleneck Transformer w/ ResNet50-T backbone, silu act. r{rbotnet50ts_256rLra)rrLr}rs rcrrks4 j#&&&  [ [j [TZ [ [[rec J|ddtdd|i|S)z> Bottleneck Transformer w/ ResNet26-T backbone, silu act. r{reca_botnext26ts_256rMra)rrMr}rs rcrrss5 j#&&&  e ePZ e^d e eerec tdd|i|S)za HaloNet-H1. Halo attention in all stages as per the paper. NOTE: This runs very slowly! rNra)rNrrs rcrNrN{s  I IJ I& I IIrec tdd|i|S)zJ HaloNet w/ a ResNet26-t backbone. Halo attention in final two stages rOra)rOrrs rcrOrOs  I IJ I& I IIrec tdd|i|S)zc HaloNet w/ a ResNet33-t backbone, SE attn for non Halo blocks, SiLU, 1-2 Halo in stage 2,3,4. rPra)rPrrs rcrPrP  L Lz LV L LLrec tdd|i|S)zT HaloNet w/ a ResNet50-t backbone, silu act. Halo attention in final two stages rQra)rQrrs rcrQrQs  J JZ J6 J JJrec tdd|i|S)zT HaloNet w/ a ResNet26-t backbone, silu act. Halo attention in final two stages rRra)rRrrs rcrRrR  O O* O O OOrec tdd|i|S)zG Lambda-ResNet-26-T. Lambda layers w/ conv pos in last two stages. rSra)rSrrs rcrSrSrrec tdd|i|S)zR Lambda-ResNet-50-TS. SiLU act. Lambda layers w/ conv pos in last two stages. rTra)rTrrs rcrTrTs  P P: P P PPrec J|ddtdd|i|S)zN Lambda-ResNet-26-R-T. Lambda layers w/ rel pos embed in last two stages. r{rrUra)rUr}rs rcrUrUs3 j#&&&  U U Uf U UUrec tdd|i|S)z Halo + RegNetZ rVra)rVrrs rcrVrVrrec tdd|i|S)z3 Combo Attention (Lambda + Halo + Bot) Network lamhalobotnet50ts_256rWra)rrWrrs rcrrs  i iT^ ibh i iirec tdd|i|S)z1 Combo Attention (Halo + Halo + Bot) Network halo2botnet50ts_256rXra)rrXrrs rcrrs  e ePZ e^d e eere)NF)r&)F)#__doc__ timm.datarr_builderr _registryrr byobnetr r r r __all__r^r]rdrv default_cfgsr|rrrrNrOrPrQrRrSrTrUrVrrrercrs  BAAAAAAA******<<<<<<<<IIIIIIIIIIII Tk KXcQ1 F F F KXcQ1 F F F  $;qDARSX\ ] ] ] K[AqT J J J  $    KXcQ1 F F F  $;A3!sVW\]bf g g g  $;A3!tWX]^cg h h h K qDA! F F F  $ { KXcQ1 F F F  $;1STUZ[`d e e e  $;qDARSX\ ] ] ]  $;qDARSX\ ] ] ]  $ K KXcQ2$ G G G KXcQ2$ G G G  $;qDARTY] ^ ^ ^ K[Ard K K K  $r***"{ K[AqQ3 G G G K[AaC H H H K[B#qS I I I K[AaC H H H  a888   { KXcQ1 F F F KXcQ1 F F F  $;qDARSX\ ] ] ] K[AqT J J J  a888   + KXcQ1 F F F  $;A3!sVW\]bf g g g  $;A3!tWX]^cg h h h K qDA! F F F  a888  KXcQ1 F F F  -Q!saARV &VWcd9e9e9e g g g  $;qDARSX\ ] ] ]  $;qDARSX\ ] ] ]  a888 ![ KXcQ2$ G G G KXcQ2$ G G G  $;qDARTY] ^ ^ ^ K[Ard K K K  a"EEE ![ KXcQ1 F F F KXcQ1 F F F  $;qDARSX\ ] ] ] K[AqT J J J      "k KXcQ1 F F F  $;1STUZ[`d e e e  $;qDARSX\ ] ] ]  $;qDARSX\ ] ] ]      '; KXcQ1 F F F KXcQ1 F F F  $;qDARSX\ ] ] ] K[AqT J J J      + KXbA" C C C KXbA" C C C  $;1cUV[]bc d d d K qC1 B B B  D$'''TDT:::a$GGG!("k KXcQ1 F F F  -cQ1 (44"::: G G G  -daA$ &9J9J9J L L L  -daA$ ,ttvv G G G  !$ K KXcQ1 F F F  -cQ1 &9J9J9J L L L  -daA$ &9J9J9J L L L  -daA$ ,ttvv G G G  !wmmm `%$!T B-6KKK"& G-6TX"Z"Z"Z!%-6!K!K!K#'4 G-6$K$K$K !DMV\ijjj4 D F=RRR#d C F=[_aaa!D F F=[_aaa!% H F=[_!a!a!a !% H$&[_!a!a!a#'$ K$&#R#R#R'+d J-6TX'Z'Z'Z #d G /=F[hsw yyy '+d L-6'K'K'K%)D J-6%K%K%Ku>&>&>> BZZZZZZ``G```` \\'\\\\ffwffffJJgJJJJJJgJJJJ MMMMMM KKwKKKK PPGPPPP PPGPPPP QQWQQQQ VV'VVVVMMMMMM jjjjjj ffwffffffre