Ngd8dZddlZddlmZmZmZmZddlZddlmZddl m Z ddl m Z  d1d e d ed ed eejfdZ d2d e dede d eejdejf dZddddejdfdee de dedededejd eejdejfdZddddddd dejdf dee deejd e de ded ed ed!ed"eee dejd eejdeejfd#ZGd$d%ejZd&Zd'ejfd(Zd'eejfd)Zd'ejfd*Zd+Zdddddd dejdf dee deejde de ded ed!ed"eee dejd eejfd,ZGd-d.ejZ Gd/d0ejZ!dS)3zv Sin-cos, fourier, rotary position embedding modules and functions Hacked together by / Copyright 2022 Ross Wightman N)ListTupleOptionalUnion)nn)ndgrid)_assertl@T num_bandsmax_freq linear_bandsdevicec|r'tjd|dz |tj|}nz,build_sincos2d_pos_embed..DsL   QvU[999<>"   2 2 2D(.QIk59T??EIdOO<)LLLTTUVWWG ::E: " ""rrmax_res include_grid in_pixelsref_feat_shapec D |7|r!t|t|| }n&t||d }n |j | |j} |r fd|D} n fd|D} |dt | ||D} t jt| d } | d} | |z} | | | | }}|r| ||gn||g}|S) a[ Args: feat_shape: Feature shape for embedding. bands: Pre-calculated frequency bands. num_bands: Number of frequency bands (determines output dim). max_res: Maximum resolution for pixel based freq. temperature: Temperature for non-pixel freq. linear_bands: Linear band spacing for pixel based freq. include_grid: Include the spatial grid in output. in_pixels: Output in pixel freq. ref_feat_shape: Reference feature shape for resize / fine-tune. dtype: Output dtype. device: Output device. Returns: N)rrrr1c Vg|]%}tjdd|tj&S)gr)stepsrr)rrrr5s rr8z+build_fourier_pos_embed..s1 f f fVWU^C1V5= Q Q Q f f frcg|]@}tj|tjtjASr4r$r5s rr8z+build_fourier_pos_embed..s= e e eUVU\!F%+ > > > A A%- P P e e erc&g|]\}}}||z |zSrQ)r6xfrs rr8z+build_fourier_pos_embed..s& I I I71aQUQY I I Irr2r9r:) rfloatr)rrziprr;r r>r?r'r@)r+rr rHr rrIrJrKrrtrBpospos_sinpos_cosouts ` rbuild_fourier_pos_embedr\Psc> }  $g) EE' EE >\F =KEf f f f f[e f f f e e e eZd e e e! I Ic!Z&H&H I I I ;vayyb ) ) )D >>"  D ,Cwwyy||%|00#'')),,u2E2EWG&2 J4' " "'8JC Jrc8eZdZ d dedeffd ZdZxZS) FourierEmbedrGr*TFrHr ct||_||_||_||_|dt||ddS)NrF persistent)super__init__rHr concat_grid keep_spatialregister_bufferr)selfrHr rdre __class__s rrczFourierEmbed.__init__ss  "&(   Wi 0 0      rc |jdd\}}|jdd}t||j|j|j|j}t j|d}|dd t|}|fd|j dz zz}|j rUt j|| d|dd ddgd}n}t j|ddd d| d|gd}|||d}|S) Nr)rIrrr2r9)r2rr)shaper\rrdrrrcatr=r<lenndimrer>expandpermutereshapenumel)rgrRBCr+emb batch_expands rforwardzFourierEmbed.forwardsiwrr{1WQRR[ %  J)'8    i$$$mmB##++C OO<<teqvz22    5 1cmmA..55lCCKKAqRSUVWWX^_```AA 199Q1a00#--2B2B2I2I,2W2WX^`aaaA !Z--//44Ar)rGr*TF)__name__ __module__ __qualname__intrcrx __classcell__rhs@rr^r^sq          $rr^ctj|ddddf |ddddfgd|jS)N.rrr2)rr;rrrl)rRs rrotrsG ;319 qccc{3R 8 8 @ @ I IIrrRc|jdkrd||d|zt||d|zzS||zt||zzS)Nrkr)ror> expand_asrrRsin_embcos_embs rapply_rot_embedrs~|q7$$Q''11!444s1vv@Q@QRS@T@T@^@^_`@a@a7aaa w;Q') ))rc^t|tjr|g}fd|DS)Nc@g|]}|zt|zzSrQ)r)r6rWrrs rr8z(apply_rot_embed_list..s- 6 6 6qAK#a&&7* * 6 6 6r) isinstancerTensorrs ``rapply_rot_embed_listrs<!U\"" C 6 6 6 6 6A 6 6 66rcB|dd\}}|jdkrd||d|zt ||d|zzS||zt ||zzS)Nrr2rkr) tensor_splitror>rr)rRrvrrs rapply_rot_embed_catrs''2..GW|q7$$Q''11!444s1vv@Q@QRS@T@T@^@^_`@a@a7aaa w;Q') ))rc |d|jddd}|d|ddd|jd}|S)Nrr2r)r>rprlgather)rR pos_embed keep_indicess rapply_keep_indices_nlcrsp##A&&--agaj"bAAI  L$:$:2$>$>$E$Eb"io^`Na$b$bccI rc t|||dz|||||| | \} } d} |D]} | | z} | | ddd} | | ddd} | | fS)a Args: feat_shape: Spatial shape of the target tensor for embedding. bands: Optional pre-generated frequency bands dim: Output dimension of embedding tensor. max_res: Maximum resolution for pixel mode. temperature: Temperature (inv freq) for non-pixel mode linear_bands: Linearly (instead of log) spaced bands for pixel mode in_pixels: Pixel vs language (inv freq) mode. dtype: Output dtype. device: Output device. Returns: r0) rr rHr rrJrKrrrr2r)r\rrrepeat_interleave)r+rr,rHr rrJrKrrrrnum_spatial_dimrRs rbuild_rotary_pos_embedrs8/(!%   GWO 1ooor22DDQKKGooor22DDQKKG G rc eZdZdZ ddedeeed eeeffd Zdd eeefd Z d Z xZ S)RotaryEmbeddinga Rotary position embedding NOTE: This is my initial attempt at impl rotary embedding for spatial use, it has not been well tested, and will likely change. It will be moved to its own file. The following impl/resources were referenced for this impl: * https://github.com/lucidrains/vit-pytorch/blob/6f3a5fcf0bca1c5ec33a35ef48d97213709df4ba/vit_pytorch/rvt.py * https://blog.eleuther.ai/rotary-embeddings/ rG'TFNrr+rKc t||_||_||_||_||_||_|b|r#t|dzt||}nt|dz|d}| d|dd|_ d|_ dSt||||||j\} } d|_| d | d| d | ddS) Nr0rrr r!rFr`r+r,rHrrJrK pos_embed_sin pos_embed_cos)rbrcr,rHr rJr+rKrrUr)rfrrrr) rgr,rHr rJrr+rKremb_sinemb_cosrhs rrczRotaryEmbedding.__init__sr  &"$,   (1H'NN!- #1H +    !    "&D !%D    6%)##2     GWDJ    !       !     rrlcl|j |Jt||j|jS|j|jfS)N)rJ)rrrJrr)rgrls r get_embedzRotaryEmbedding.get_embedXsM : !$$$) .  %t'99 9rcn||jdd\}}t|||SNr)rrlr)rgrRrrs rrxzRotaryEmbedding.forwardds4>>!'!""+66q'7333rrGrTFNNN ryrzr{__doc__boolrrr|rcrrxr}r~s@rrrs!&.226;;  ;!c+;%T#Y/;;;;;;z : :xS 2 : : : :4444444rrc eZdZdZ ddedeeed eeeffd Zdd eeefd Z d Z xZ S)RotaryEmbeddingCata Rotary position embedding w/ concatenatd sin & cos The following impl/resources were referenced for this impl: * https://github.com/lucidrains/vit-pytorch/blob/6f3a5fcf0bca1c5ec33a35ef48d97213709df4ba/vit_pytorch/rvt.py * https://blog.eleuther.ai/rotary-embeddings/ rGrTFNrr+rKct||_||_||_||_||_||_|[|r#t|dzt||}nt|dz|d}| d|dd|_ dSt||||||j} d|_| d tj| d ddS) Nr0rrrrFr`rrr2)rbrcr,rHr rJr+rKrrUr)rfrrrrrm) rgr,rHr rJrr+rKrembedsrhs rrczRotaryEmbeddingCat.__init__rsF  &"$,   (1H'NN!- #1H +    !    "DNNN,%)##2 FDJ   &"%%  !     rrlc|j9|7t||j|j|j}t j|dS|j|jSJd)N)rJrKr2FzPget_embed() requires pre-computed pos_embed or valid shape w/ pre-computed bands)rrrJrKrrmr)rgrlrs rrzRotaryEmbeddingCat.get_embedsi : !e&7+ .#2 F 9VR(( ( ^ '> ! ll l l lrcf||jdd}t||Sr)rrlr)rgrRrs rrxzRotaryEmbeddingCat.forwards-NN17122;// "1i000rrrrr~s@rrrjs!&.22655  5!c+5%T#Y/555555n m mxS 2 m m m m1111111rr)r TN)rrN)"rrtypingrrrrrrrBr trace_utilsr r|rUrrrrr)rrrFr\Moduler^rrrrrrrrrQrrrsl ////////////  !)-     &    $)- &   \ ###("])-&#&#I&# &#&# &# ! &# { &#&&# \&#&#&#&#V)-#"".2"])-CCIC %CC C  C  CCC!c+C{C&C %,CCCCL)))))29)))XJJJ*u|**** 7D.7777 *5<****)-#".2"])-..I. %.. .  .  ..!c+.{.&....bW4W4W4W4W4biW4W4W4tQ1Q1Q1Q1Q1Q1Q1Q1Q1Q1r