NgdZddlmZddlZddlmZddlmcmZddl m Z m Z ddl m Z ddlmZdeefd ZGd d ejZGd d ejZdS)aP Bottleneck Self Attention (Bottleneck Transformers) Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 @misc{2101.11605, Author = {Aravind Srinivas and Tsung-Yi Lin and Niki Parmar and Jonathon Shlens and Pieter Abbeel and Ashish Vaswani}, Title = {Bottleneck Transformers for Visual Recognition}, Year = {2021}, } Based on ref gist at: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 This impl is a WIP but given that it is based on the ref gist likely not too far off. Hacked together by / Copyright 2021 Ross Wightman )ListN) to_2tuplemake_divisible) trunc_normal_)_assert permute_maskc|j\}}}}||ddz}|d|d|zdz }tj|ddgd}tj|d|dz g}|d|dzd|zdz }|ddd||dz df}|||d||dd|dd}||S)a Compute relative logits along one dimension As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 Args: q: (batch, heads, height, width, dim) rel_k: (2 * width - 1, dim) permute_mask: permute output dim according to this rrN)shape transposereshapeFpadflattenexpandpermute) qrel_kr BHWdimxx_pads W/var/www/html/ai-engine/env/lib/python3.11/site-packages/timm/layers/bottleneck_attn.py rel_logits_1drs 7LAq!S U__R $ $ $A "aQ""A E!aV   $ $Q ' 'E E%!QU $ $E MM"a!eQUQY / /E aaa!QUVVmA !Q1a  ''B2r::A 99\ " ""c(eZdZdZfdZdZxZS) PosEmbedRelz Relative Position Embedding As per: https://gist.github.com/aravindsrinivas/56359b79f0ce4449bcb04ab4b56a57a2 Originally from: `Attention Augmented Convolutional Networks` - https://arxiv.org/abs/1904.09925 cvtt|\|_|_||_t jtj |jdzdz ||z|_ t jtj |jdzdz ||z|_ dS)Nr r) super__init__rheightwidthdim_headnn Parametertorchrandn height_rel width_rel)self feat_sizer(scale __class__s rr%zPosEmbedRel.__init__=s "+I"6"6 TZ  ,u{4;?Q3F'Q'QTY'YZZek$*q.12Dh&O&ORW&WXXr c |j\}}}|||j|jd}t ||jd}|dd}t ||jd}||z}||||}|S)Nr )rrr )r rr )rr4rr5r )rrr&r'rr.rr-)r/rrHW_ rel_logits_w rel_logits_h rel_logitss rforwardzPosEmbedRel.forwardDs72q IIadj" 5 5$Q_UUU  KK1  $QoVVV !L0 ''2r22 r )__name__ __module__ __qualname____doc__r%r; __classcell__r2s@rr"r"8sVYYYYY       r r"c4eZdZdZ d fd ZdZd ZxZS) BottleneckAttna Bottleneck Attention Paper: `Bottleneck Transformers for Visual Recognition` - https://arxiv.org/abs/2101.11605 The internal dimensions of the attention module are controlled by the interaction of several arguments. * the output dimension of the module is specified by dim_out, which falls back to input dim if not set * the value (v) dimension is set to dim_out // num_heads, the v projection determines the output dim * the query and key (qk) dimensions are determined by * num_heads * dim_head if dim_head is not None * num_heads * (dim_out * attn_ratio // num_heads) if dim_head is None * as seen above, attn_ratio determines the ratio of q and k relative to the output if dim_head not used Args: dim (int): input dimension to the module dim_out (int): output dimension of the module, same as dim if not set stride (int): output stride of the module, avg pool used if stride == 2 (default: 1). num_heads (int): parallel attention heads (default: 4) dim_head (int): dimension of query and key heads, calculated from dim_out * attn_ratio // num_heads if not set qk_ratio (float): ratio of q and k dimensions to output dimension when dim_head not set. (default: 1.0) qkv_bias (bool): add bias to q, k, and v projections scale_pos_embed (bool): scale the position embedding as well as Q @ K Nrr5?Fc xt| Jd|p|}||zdksJ||_|pt||zd|z|_||jz|_||jz|_||jz|_|jdz|_| |_ tj ||jdz|jzd||_ t||j|j |_|dkrtjddntj|_|dS) NzBA concrete feature size matching expected input (H, W) is requiredr)divisorr r)bias)r(r1)r$r% num_headsr dim_head_qk dim_head_v dim_out_qk dim_out_vr1scale_pos_embedr)Conv2dqkvr" pos_embed AvgPool2dIdentitypoolreset_parameters) r/rdim_outr0striderJr(qk_ratioqkv_biasrOr2s rr%zBottleneckAttn.__init__jsD $$&j$$$.S"a''''"#a~g6HRS'T'T'TXa'a!T^3#d&66"T_4%- .9S$/A"5"FPXYYY%Y9IQUQ[\\\*0A++BLA&&&2;==  r ct|jj|jjjddzt|jj|jt|jj|jdS)NrrH)std)rrQweightrrRr-r1r.)r/s rrVzBottleneckAttn.reset_parameterssddho48?+@+Ct+KLLLLdn/TZ@@@@dn.DJ??????r c|j\}}}}t||jjkdt||jjkd||}t j||j|j|j gd\}}}| ||j z|j d dd}| ||j z|j d}| ||j z|jd dd}|jr$||z||z|jz} n#||z|jz||z} | d} | |z dd ||j ||} || } | S)Nr)rr r )rrrRr&r'rQr+splitrMrNrrJrKrrLrOr1softmaxrU) r/rrCrrrkvattnouts rr;zBottleneckAttn.forwardsW 1aT^**B///T^))2... HHQKK+a$/4?DN!SYZ[[[1a IIa$.($*:B ? ? I I"b Q Q IIa$.($*:B ? ? IIa$.($/2 > > H HR P P   <EDNN1---;DDETZ'$..*;*;;D|||##ax""2r**221dnaKKiinn r )NNrr5NrDFF)r<r=r>r?r%rVr;r@rAs@rrCrCTsr,VZ:?      0@@@ r rC)r?typingrr+torch.nnr)torch.nn.functional functionalrhelpersrr weight_initr trace_utilsrintrModuler"rCr rrqs   ........&&&&&& #$s)####8")8IIIIIRYIIIIIr