NgdZddlZddlmZddlZddlmZddlmcmZ ddl m Z m Z m Z mZGddejZGdd ejZGd d ejZGd d ejZdejdefdZdejdejdejdefdZdejdejdejdefdZGddejZGddejZGddejZdS)z Normalization layers and wrappers Norm layer definitions that support fast norm and consistent channel arg order (always first arg). Hacked together by / Copyright 2022 Ross Wightman N)Tuple) is_fast_normfast_group_normfast_layer_norm fast_rms_normc&eZdZdfd ZdZxZS) GroupNorm h㈵>Tcxt||||t|_dS)N)epsaffinesuper__init__r fast_norm)self num_channels num_groupsrr __class__s L/var/www/html/ai-engine/env/lib/python3.11/site-packages/timm/layers/norm.pyrzGroupNorm.__init__s3 \s6JJJ%c|jr't||j|j|j|jSt j||j|j|j|jSNrrrweightbiasrF group_normrxs rforwardzGroupNorm.forwardK > V"1dot{DItxXX X<4?DKDHUU Ur)r r T)__name__ __module__ __qualname__rr# __classcell__rs@rr r sS(((((( VVVVVVVrr cFeZdZdZfdZdejdejfdZxZS) GroupNorm1zL Group Normalization with 1 group. Input: tensor in shape [B, C, *] c dtjd|fi|t|_dS)Nrr)rrkwargsrs rrzGroupNorm1.__init__#s3L33F333%rr"returnc|jr't||j|j|j|jSt j||j|j|j|jSrrr!s rr#zGroupNorm1.forward'r$r r%r&r'__doc__rtorchTensorr#r(r)s@rr+r+sr(((((VV%,VVVVVVVVrr+cHeZdZdZdfd ZdejdejfdZxZS) LayerNormz# LayerNorm w/ fast norm option ư>Tcvt|||t|_dSN)relementwise_affinerrr _fast_normrrrrrs rrzLayerNorm.__init__11 36JJJ&..rr"r.c|jr(t||j|j|j|j}n,t j||j|j|j|j}|Sr)r;rnormalized_shaperrrr layer_normr!s rr#zLayerNorm.forward5sT ? Y4#8$+tyRVRZ[[AA Q 5t{DItxXXArr6Tr0r)s@rr5r5.sm))))))%,rr5cHeZdZdZdfd ZdejdejfdZxZS) LayerNorm2dz5 LayerNorm for channels of '2D' spatial NCHW tensors r6Tcvt|||t|_dSr8r:r<s rrzLayerNorm2d.__init__?r=rr"r.c|dddd}|jr(t||j|j|j|j}n,tj||j|j|j|j}|dddd}|SNrr) permuter;rr?rrrrr@r!s rr#zLayerNorm2d.forwardCs IIaAq ! ! ? Y4#8$+tyRVRZ[[AA Q 5t{DItxXXA IIaAq ! !rrAr0r)s@rrCrC=si??))))))%,rrCtensorr.ctjr|S|tjS)N) memory_format)r2jit is_scripting is_contiguouscontiguous_format)rJs r_is_contiguousrQMsE yK##%%%##%2I#JJJrr"rrrctj|ddd\}}||z tj||zz}||ddddfz|ddddfz}|S)NrFT)dimunbiasedkeepdim)r2var_meanrsqrt)r"rrrsus r_layer_norm_cfrZUsl >!UD A A ADAq Q%+a#g&&&A F111dD= !!DD$$77A Hrc<|dd}||zdd||zz d}||z tj||zz}||ddddz|ddddz}|S)NrT)rSrUr)meanclampr2rWview)r"rrrrYrXs r_layer_norm_cf_sqmr`\s 1d##A a%!T * *a!e 4;;A>>A Q%+a#g&&&A FKK2q! $ $$tyyB1'='==A Hrc:eZdZdZdfd ZdejfdZxZS)LayerNormExp2da_ LayerNorm for channels_first tensors with 2d spatial dimensions (ie N, C, H, W). Experimental implementation w/ manual norm for tensors non-contiguous tensors. This improves throughput in some scenarios (tested on Ampere GPU), esp w/ channels_last layout. However, benefits are not always clear and can perform worse on other GPUs. r6cNt||dS)N)r)rr)rrrrs rrzLayerNormExp2d.__init__ms& 3/////rr.ct|rYtj|dddd|j|j|j|jdddd}n!t||j|j|j}|SrF) rQrr@rIr?rrrrZr!s rr#zLayerNormExp2d.forwardps !   D  !Q1%%t'rsm  TTTTTTTTTTTT V V V V V V V V V V V V V V V V             ",    K5<KDKKKK el EL   SX     %,   EL W\    R\*!!!!!bi!!!H##### #####r