I gkdddlZddlmZddlmcmZddlmcmZddl Z ddl m Z m Z mZGddejZdZdZGddejZGd d ejZGd d ejZGd dejZGddejZGddejZddZddZddZddZy)N)DropPath to_2tuple trunc_normal_cFeZdZdZddej dffd ZdZxZS)Mlpz Multilayer perceptron.Nct||xs|}|xs|}tj|||_||_tj|||_tj||_yN) super__init__nnLinearfc1actfc2Dropoutdrop)self in_featureshidden_features out_features act_layerr __class__s X/var/www/html/transparent-background/transparent_background/backbones/SwinTransformer.pyr z Mlp.__init__s_ #2{ )8[99[/:;99_l;JJt$ c|j|}|j|}|j|}|j|}|j|}|Sr )rrrr)rxs rforwardz Mlp.forwardsH HHQK HHQK IIaL HHQK IIaLr) __name__ __module__ __qualname____doc__r GELUr r __classcell__rs@rrrs!!48tWYW^W^eg%rrc|j\}}}}|j|||z|||z||}|jddddddjjd|||}|S)z Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C) r)shapeviewpermute contiguous)r window_sizeBHWCwindowss rwindow_partitionr7$spJAq!Q q!{"Kk1A;PQRAii1aAq)446;;B [Z[\G Nrct|jd||z|z |z z }|j|||z||z||d}|jddddddj j|||d}|S)z Args: windows: (num_windows*B, window_size, window_size, C) window_size (int): Window size H (int): Height of image W (int): Width of image Returns: x: (B, H, W, C) rr,r'r(r)r*r+)intr-r.r/r0)r6r1r3r4r2rs rwindow_reverser:3s GMM! A 3k A BCA Q[(!{*:KVXYA !Q1a#..055aArBA Hrc,eZdZdZdfd ZddZxZS)WindowAttentiona Window based multi-head self attention (W-MSA) module with relative position bias. It supports both of shifted and non-shifted window. Args: dim (int): Number of input channels. window_size (tuple[int]): The height and width of the window. num_heads (int): Number of attention heads. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0 proj_drop (float, optional): Dropout ratio of output. Default: 0.0 ct|||_||_||_||z}|xs|dz|_t jtjd|dzdz d|dzdz z||_ tj|jd} tj|jd} tjtj| | g} tj| d} | dddddf| dddddfz } | jdddj!} | dddddfxx|jddz z cc<| dddddfxx|jddz z cc<| dddddfxxd|jdzdz zcc<| j#d}|j%d|t j&||dz||_t j*||_t j&|||_t j*||_t3|jd t j4d |_y) Ngr)rr'r,relative_position_indexr(bias{Gz?std)dim)r r rDr1 num_headsscaler Parametertorchzerosrelative_position_bias_tablearangestackmeshgridflattenr/r0sumregister_bufferrqkvr attn_dropproj proj_droprSoftmaxsoftmax)rrDr1rEqkv_biasqk_scalerRrThead_dimcoords_hcoords_wcoordscoords_flattenrelative_coordsr>rs rr zWindowAttention.__init__Rs# &")#1T!1 -/LL KK[^+a/A A4F4JKY W-Y)<< 0 0 34<< 0 0 34U^^Xx,@ABvq1(At4~aqj7QQ)11!Q:EEG1a D$4$4Q$7!$;; 1a D$4$4Q$7!$;; 1a A(8(8(;$;a$?? "1"5"5b"9 68OP99S#'9I.IIc3' I.d77SAzzb) rcT|j\}}}|j|j||d|j||jzj ddddd}|d|d|d} }}||j z}||j ddz} |j|jjdj|jd|jdz|jd|jdzd} | j dddj} | | jdz} ||jd} | j|| z| |j|||jdjdz} | jd|j||} |j| } n|j| } |j| } | | zj ddj|||}|j|}|j!|}|S)z Forward function. Args: x: input features with shape of (num_windows*B, N, C) mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None r(r)rr'r*r,)r-rQreshaperEr/rF transposerJr>r.r1r0 unsqueezerVrRrSrT) rrmaskB_Nr5rQqkvattnrelative_position_biasnWs rrzWindowAttention.forwardts77Aqhhqk!!"aDNNA>1a@4>>RSCTC^C^_`CaaD99RA6D<<%D<<%D~~d# AX A & . .r1a 8 IIaL NN1 r)TNrrr rr r!r"r rr$r%s@rr<r<Ds  *D rr<c feZdZdZddddddddej ej f fd Zd ZxZ S) SwinTransformerBlocka] Swin Transformer Block. Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. window_size (int): Window size. shift_size (int): Shift size for SW-MSA. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set. drop (float, optional): Dropout rate. Default: 0.0 attn_drop (float, optional): Attention dropout rate. Default: 0.0 drop_path (float, optional): Stochastic depth rate. Default: 0.0 act_layer (nn.Module, optional): Activation layer. Default: nn.GELU norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm r@TNrc t|||_||_||_||_||_d|j cxkr|jksJdJd| ||_t|t|j|||| ||_ | dkDr t| ntj|_| ||_t!||z} t#|| | ||_d|_d|_y)Nrz shift_size must in 0-window_size)r1rErWrXrRrTr)rrrr)r r rDrEr1 shift_size mlp_rationorm1r<rrjrr Identity drop_pathnorm2r9rmlpr3r4)rrDrEr1rsrtrWrXrrRrwr norm_layermlp_hidden_dimrs rr zSwinTransformerBlock.__init__s "&$"DOO6d&6&66Z8ZZ6Z8ZZ6_ # Yt'7'78IIQUW 1:B),BKKM_ S9_-3R[bfgrc |j\}}}|j|j}}|||zk(sJd|}|j|}|j ||||}dx} } |j ||j zz |j z} |j ||j zz |j z} t j|dd| | | | f}|j\} }}} |jdkDr3tj||j |j fd}|}n|}d}t||j }|j d|j |j z|}|j||}|j d|j |j |}t||j ||}|jdkDr/tj||j|jfd}n|}| dkDs| dkDr|ddd|d|ddfj}|j |||z|}||j|z}||j|j!|j#|z}|S)z Forward function. Args: x: Input feature, tensor size (B, H*W, C). H, W: Spatial resolution of the input feature. mask_matrix: Attention mask for cyclic shift. input feature has wrong sizer)r'r))shiftsdimsNr,)rd)r-r3r4rur.r1FpadrsrHrollr7rjr:r0rwryrx)rr mask_matrixr2Lr5r3r4shortcutpad_lpad_tpad_rpad_b_HpWp shifted_x attn_mask x_windows attn_windowss rrzSwinTransformerBlock.forwardse''1avvtvv1AEz999z JJqM FF1aA !!A(8(8$88D@P-QX^_I#III%Y0@0@A NN2t'7'7$:J:J'JAN yyy; $((T-=-=t?O?OQRS "<1A1A2rJ  ??Q  9doot-OV\]AA 19 !RaR!Q,**,A FF1a!eQ  t~~a( ( txx 1 67 7r) rr r!r"r r# LayerNormr rr$r%s@rroros5"45t"PR^`77r||09rroc@eZdZdZej ffd ZdZxZS) PatchMergingz Patch Merging Layer Args: dim (int): Number of input channels. norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm ct|||_tjd|zd|zd|_|d|z|_y)Nr*r)Fr?)r r rDr r reductionnorm)rrDrzrs rr zPatchMerging.__init__sA 1s7AG%@q3w' rc |j\}}}|||zk(sJd|j||||}|dzdk(xs|dzdk(}|r"tj|ddd|dzd|dzf}|ddddddddddf}|ddddddddddf} |ddddddddddf} |ddddddddddf} t j || | | gd}|j|dd|z}|j |}|j|}|S) Forward function. Args: x: Input feature, tensor size (B, H*W, C). H, W: Spatial resolution of the input feature. r}r)r'rNr,r*)r-r.rrrHcatrr) rrr3r4r2rr5 pad_inputx0x1x2x3s rrzPatchMerging.forward sL''1aAEz999z FF1aA UaZ0QUaZ a!Q1q5!QU34A q!$Q$1a  q!$Q$1a  q!$Q$1a  q!$Q$1a  IIr2r2& + FF1b!a%  IIaL NN1 r rr r!r"r rr rr$r%s@rrrs (*||( rrc ReZdZdZdddddddej ddf fd Zd ZxZS) BasicLayera A basic Swin Transformer layer for one stage. Args: dim (int): Number of feature channels depth (int): Depths of this stage. num_heads (int): Number of attention head. window_size (int): Local window size. Default: 7. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set. drop (float, optional): Dropout rate. Default: 0.0 attn_drop (float, optional): Attention dropout rate. Default: 0.0 drop_path (float | tuple[float], optional): Stochastic depth rate. Default: 0.0 norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm downsample (nn.Module | None, optional): Downsample layer at the end of the layer. Default: None use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False. rprqTNrFcjt|||_|dz|_||_| |_t jt|Dcgc]:}t||||dzdk(rdn|dz||||| t| tr| |n| |  <c}|_ | | || |_ yd|_ ycc}w)Nr)r) rDrEr1rsrtrWrXrrRrwrz)rDrz)r r r1rsdepthuse_checkpointr ModuleListrangero isinstancelistblocks downsample)rrDrrEr1rtrWrXrrRrwrzrrirs rr zBasicLayer.__init__;s &%* ,mm5\ %# !#'!"Q!1+2B#!!#*4Y*E)A,9% ' %# $  !(SZHDO"DO' %#s ?B0c ttj||jz |jz}ttj||jz |jz}t j d||df|j }td|j t|j |j t|j df}td|j t|j |j t|j df}d} |D]} |D]} | |dd| | ddf<| dz } t||j} | jd|j|jz} | jd| jdz } | j| dk7tdj| dk(td} |jD]>}||c|_|_|j"rt%j$||| }6||| }@|j&+|j'|||}|dzdz|dzdz}}||||||fS||||||fS) rr')devicerNr,r)gYr)r9npceilr1rHrIrslicersr7r.rc masked_fillfloatrr3r4r checkpointr)rrr3r4rrimg_maskh_slicesw_slicescnthw mask_windowsrblkx_downWhWws rrzBasicLayer.forwardesrT---. /$2B2B B T---. /$2B2B B;;2r1~ahh?!d.../4+++doo-=>4??*D13!d.../4+++doo-=>4??*D13 A '*Aq!$q   ($2B2BC #((T-=-=@P@P-PQ  **1- 0F0Fq0II )))q.%-HTTU^bcUcejkneop ;; &CaLCE35""))#q)<9%  & ?? &__Q1-F!e\AEa<BaFB* *aAq!# #rrr%s@rrr(s7,LL %(#T($rrc*eZdZdZdfd ZdZxZS) PatchEmbedaE Image to Patch Embedding Args: patch_size (int): Patch token size. Default: 4. in_chans (int): Number of input image channels. Default: 3. embed_dim (int): Number of linear projection output channels. Default: 96. norm_layer (nn.Module, optional): Normalization layer. Default: None ct|t|}||_||_||_t j|||||_||||_ yd|_ y)N) kernel_sizestride) r r r patch_sizein_chans embed_dimr Conv2drSr)rrrrrzrs rr zPatchEmbed.__init__s] z* $  "IIh zR\]  !"9-DIDIrc |j\}}}}||jdzdk7r8tj|d|jd||jdzz f}||jdzdk7r:tj|ddd|jd||jdzz f}|j |}|j |jd|jd}}|j djdd}|j |}|jddjd|j||}|S)Forward function.r'rr)r(r,) sizerrrrSrrNrbr.r)rrrr3r4rrs rrzPatchEmbed.forwards*VVX 1a tq! !Q &a!T__Q/!dooa6H2HHIJA tq! !Q &a!Q4??1#5DOOA>2rBAr)r*r(`Nrmr%s@rrrs rrceZdZdZddddgdgddd d d d d d ej dd dddffd ZdZddZdZ dfd Z xZ S)SwinTransformera Swin Transformer backbone. A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows` - https://arxiv.org/pdf/2103.14030 Args: pretrain_img_size (int): Input image size for training the pretrained model, used in absolute postion embedding. Default 224. patch_size (int | tuple(int)): Patch size. Default: 4. in_chans (int): Number of input image channels. Default: 3. embed_dim (int): Number of linear projection output channels. Default: 96. depths (tuple[int]): Depths of each Swin Transformer stage. num_heads (tuple[int]): Number of attention head of each stage. window_size (int): Window size. Default: 7. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4. qkv_bias (bool): If True, add a learnable bias to query, key, value. Default: True qk_scale (float): Override default qk scale of head_dim ** -0.5 if set. drop_rate (float): Dropout rate. attn_drop_rate (float): Attention dropout rate. Default: 0. drop_path_rate (float): Stochastic depth rate. Default: 0.2. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. ape (bool): If True, add absolute position embedding to the patch embedding. Default: False. patch_norm (bool): If True, add normalization after patch embedding. Default: True. out_indices (Sequence[int]): Output from which stages. frozen_stages (int): Stages to be frozen (stop grad and set eval mode). -1 means not freezing any parameters. use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False. r*r(rr)r)r)r(r rprqTNrg?F)rr'r)r(r,ct|||_t||_||_||_||_||_||_ t||||jr|nd|_ |j ryt|}t|}|d|dz|d|dzg}tjtj d||d|d|_t%|j"dtj&| |_tj*d| t-|Dcgc]}|j/}}tj0|_t5|jD]}t7t9|d|zz||||||| | | | |t-|d|t-|d|dz|||jdz krt:nd| }|j2j=|t5|jDcgc]}t9|d|zz}}||_|D]$}|||}d |}|jA||&|jCycc}wcc}w) N)rrrrzrr'rArB)pr)) rDrrEr1rtrWrXrrRrwrzrrr)"r r pretrain_img_sizelen num_layersrape patch_norm out_indices frozen_stagesr patch_embedrr rGrHrIabsolute_pos_embedrrpos_droplinspacerOitemrlayersrrr9rappend num_features add_module_freeze_stages)rrrrrdepthsrEr1rtrWrX drop_rateattn_drop_ratedrop_path_raterzrrrrrpatches_resolutionrdpri_layerlayerrr layer_namers rr zSwinTransformer.__init__sp( !2f+"$&*&!H %)__z$@ 88 )*; < ":.J"3A"6*Q-"GIZ[\I]aklmanIn!o &(ll5;;q)M_`aMbdvwxdy3z&{D # $11s ; Y/ "'>3v;!OPAqvvxPPmmo T__- &G AL01Wo#G,'#!!(c&'"23C|! 8L4MN%,3doo6I,I._init_weights@s!RYY'ahhC0a+0BGG%%affa01C+Ar||,!!!&&!,!!!((C0-rF)strictloggerNz pretrained must be a str or None)rstrapplyget_root_loggerload_checkpoint TypeError)r pretrainedrrs r init_weightszSwinTransformer.init_weights8sS 1 j# & JJ} %$&F D*U6 J   JJ} %>? ?rc|j|}|jd|jd}}|jr)tj|j ||fd}||z}|j g}|jdjdd}|j|}t|jD]}|j|}||||\}} } }}}||jvs2t|d|} | |}|jd| | |j |j#ddddj } |j%| t'|S) rr)r(bicubic)rmoder'rr,r)rrrr interpolaterr0rNrbrrrrrgetattrr.rr/rtuple) rrrrroutsrrx_outr3r4rzouts rrzSwinTransformer.forwardRsN   Q AFF1IB 88!"t/F/FbRTX\e!f ''A  IIaL " "1a ( MM! t' !AKKNE%*1b"%5 "E1aBD$$$$TT!:6 "5)jjQ4+<+gi jET ejj)gv{|~EFOT U Lrctdgdgdd}|dur+|jtjdd d d |S) Nr)rrr0rrTz=data/backbone_ckpt/swin_large_patch4_window12_384_22kto1k.pthrrrFrrrs rSwinLr*sU c-?hj kET ejj)hw|}FGPU V Lrr)rHtorch.nnr torch.nn.functional functionalrtorch.utils.checkpointutilsrnumpyr timm.layersrrrModulerr7r:r<rorrrrrr r&r*rrr4s ++::"))*  "PbiiPfc299cL(299(Ve$e$P((Vtbiitlr