NgdZddlZddlmZddlmcmZddlmZm Z m Z Gddej Z Gddej Z Gd d ej ZGd d ej ZdS) a Convolution with Weight Standardization (StdConv and ScaledStdConv) StdConv: @article{weightstandardization, author = {Siyuan Qiao and Huiyu Wang and Chenxi Liu and Wei Shen and Alan Yuille}, title = {Weight Standardization}, journal = {arXiv preprint arXiv:1903.10520}, year = {2019}, } Code: https://github.com/joe-siyuan-qiao/WeightStandardization ScaledStdConv: Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 Official Deepmind JAX code: https://github.com/deepmind/deepmind-research/tree/master/nfnets Hacked together by / copyright Ross Wightman, 2021. N) get_paddingget_padding_valuepad_samec.eZdZdZ dfd ZdZxZS) StdConv2dzConv2d with Weight Standardization. Used for BiT ResNet-V2 models. Paper: `Micro-Batch Training with Batch-Channel Normalization and Weight Standardization` - https://arxiv.org/abs/1903.10520v2 rNFư>c |t|||}t||||||||| |_dS)Nstridepaddingdilationgroupsbias)rsuper__init__eps) self in_channel out_channels kernel_sizer r rrrr __class__s P/var/www/html/ai-engine/env/lib/python3.11/site-packages/timm/layers/std_conv.pyrzStdConv2d.__init__ s^ ?!+vx@@G   k&hvD  J J Jc tj|jd|jddddd|j|j}tj|||j|j |j |j |j }|SNrT)trainingmomentumr) F batch_normweightreshaperr reshape_asconv2drr r rrrxr#s rforwardzStdConv2d.forward*s K  4#4b 9 94BDH6666@j6M6M  HQ 4; dmUYU` a ar)rNrrFr __name__ __module__ __qualname____doc__rr) __classcell__rs@rrrs` LP26rrc.eZdZdZ dfd ZdZxZS) StdConv2dSamezConv2d with Weight Standardization. TF compatible SAME padding. Used for ViT Hybrid model. Paper: `Micro-Batch Training with Batch-Channel Normalization and Weight Standardization` - https://arxiv.org/abs/1903.10520v2 rSAMEFr c t||||\}} t||||||||| |_| |_dS)Nr rr )rrrsame_padr) rrrrr r rrrr is_dynamicrs rrzStdConv2dSame.__init__8si0V^fggg   k&'\d  & & &# rc h|jr!t||j|j|j}t j|jd|j ddddd|j  |j}t j |||j |j|j|j|j}|Sr)r6rrr rr!r"r#r$rrr%r&rr rr's rr)zStdConv2dSame.forwardBs = JD,dk4=IIA K  4#4b 9 94BDH6666@j6M6M  HQ 4; dmUYU` a ar)rr3rrFr r*r0s@rr2r22s` LR26rr2c.eZdZdZ d fd ZdZxZS) ScaledStdConv2da4Conv2d layer with Scaled Weight Standardization. Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 NOTE: the operations used in this impl differ slightly from the DeepMind Haiku impl. The impact is minor. rNT?r c L|t|||}t||||||||tjt j|jdddf| |_| |j d dzz|_ | |_ dS)Nr rr) rrrnn Parametertorchfullrgainr#numelscaler) r in_channelsrrr r rrrgammar gain_initrs rrzScaledStdConv2d.__init__Us ?!+vx@@G  {67]e  & & &LT->1a,H)!T!TUU T[^1133t;; rc Vtj|jd|jddd|j|jzddd|j |j}tj |||j |j |j |j|jSNrrTr)r#rr r)r!r"r#r$rrBrDviewrr%r&rr r rrr's rr)zScaledStdConv2d.forwardas K  4#4b 9 94I *0044BDH6667Aj6M6M x649dk4<X\Xcdddr)rNrrTr;r r;r*r0s@rr:r:LsgMQLO      eeeeeeerr:c.eZdZdZ d fd ZdZxZS) ScaledStdConv2dSamea\Conv2d layer with Scaled Weight Standardization and Tensorflow-like SAME padding support Paper: `Characterizing signal propagation to close the performance gap in unnormalized ResNets` - https://arxiv.org/abs/2101.08692 NOTE: the operations used in this impl differ slightly from the DeepMind Haiku impl. The impact is minor. rr3Tr;r c `t||||\}} t||||||||tjt j|jdddf| |_| |j d dzz|_ | |_ | |_ dS)Nr5r rrr=)rrrr>r?r@rArrBr#rCrDr6r)rrErrr r rrrrFrrGr7rs rrzScaledStdConv2dSame.__init__rs0V^fggg  {67]e  & & &LT->1a,H)!T!TUU T[^1133t;; " rc |jr!t||j|j|j}t j|jd|j ddd|j |j z ddd|j |j}t j|||j|j|j|j|jSrI)r6rrr rr!r"r#r$rrBrDrJrr%r&rr rr's rr)zScaledStdConv2dSame.forward~s = JD,dk4=IIA K  4#4b 9 94I *0044BDH6667Aj6M6M x649dk4<X\Xcdddr)rr3rrTr;r r;r*r0s@rrLrLisgMSLO      eeeeeeerrL)r.r@torch.nnr>torch.nn.functional functionalr!r rrrConv2drr2r:rLrrrTs1$ ========== 0BI4eeeeebieee:eeeee")eeeeer