Ng]HdZddlmZddlmZddlZddlmZddlmZm Z ddl m Z ddl m Z mZmZd d lmZd d lmZmZmZd gZGd d e Zd^dZd_dZeidedddedddedddedddddedd d!"d#edd$d!"d%edd&d!"d'edd(ddd!)d*edd+d,edd-ddd.edd/d0edd1d2edd3ddd4edd5d6edd7ddd8edd9d:edd;d<edd=ddedd>d<edd?d<edd@ddeddAd<eddBddeddCd<dDZed`dEe fdFZed`dEe fdGZed`dEe fdHZed`dEe fdIZed`dEefdJZed`dEefdKZ ed`dEefdLZ!ed`dEefdMZ"ed`dEe fdNZ#ed`dEe fdOZ$ed`dEe fdPZ%ed`dEe fdQZ&ed`dEe fdRZ'ed`dEe fdSZ(ed`dEe fdTZ)ed`dEe fdUZ*ee+d:dVdWdXdYdZd[d\d]dS)aa[ DeiT - Data-efficient Image Transformers DeiT model defs and weights from https://github.com/facebookresearch/deit, original copyright below paper: `DeiT: Data-efficient Image Transformers` - https://arxiv.org/abs/2012.12877 paper: `DeiT III: Revenge of the ViT` - https://arxiv.org/abs/2204.07118 Modifications copyright 2021, Ross Wightman )partial)OptionalN)nnIMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STD)resample_abs_pos_embed)VisionTransformer trunc_normal_checkpoint_filter_fn)build_model_with_cfg)generate_default_cfgsregister_modelregister_model_deprecationsVisionTransformerDistilledceZdZdZfdZdfd ZejjddZ ejjde j fdZ dd e d eefd ZejjddZdZddedejfdZxZS)rz Vision Transformer w/ Distillation Token and Head Distillation token & head support for `DeiT: Data-efficient Image Transformers` - https://arxiv.org/abs/2012.12877 c:|dd}tj|i|ddi|jdvsJd|_t jtjdd|j |_ t jtjd|j j |jz|j |_ |jdkrt j|j |jnt j|_d|_||dS) N weight_initskip)tokenr rF)popsuper__init__ global_poolnum_prefix_tokensr Parametertorchzeros embed_dim dist_token patch_embed num_patches pos_embed num_classesLinearIdentity head_distdistilled_training init_weights)selfargskwargsr __class__s L/var/www/html/ai-engine/env/lib/python3.11/site-packages/timm/models/deit.pyrz#VisionTransformerDistilled.__init__#s jj33 $=&==f====:----!",u{1a'H'HII K4+7$:PPRVR` a accHLHX[\H\H\4>43CDDDbdbmbobo"' +&&&&&rcxt|jdt|dS)Ng{Gz?)std)mode)r r#rr,)r-r5r0s r1r,z'VisionTransformerDistilled.init_weights1s9do3//// $'''''r2Fc(tdddgS)Nz+^cls_token|pos_embed|patch_embed|dist_token)z^blocks\.(\d+)N)z^norm)i)stemblocks)dict)r-coarses r1 group_matcherz(VisionTransformerDistilled.group_matcher5s'?)$&    r2returnc|j|jfSNheadr*)r-s r1get_classifierz)VisionTransformerDistilled.get_classifier>sy$.((r2Nr'rc||_|dkrtj|j|ntj|_|dkrtj|j|jntj|_dS)Nr)r'rr(r"r)r@r*)r-r'rs r1reset_classifierz+VisionTransformerDistilled.reset_classifierBsj&>IAooBIdnk:::SUS^S`S` HSVW4>43CDDD]_]h]j]jr2Tc||_dSr>)r+)r-enables r1set_distilled_trainingz1VisionTransformerDistilled.set_distilled_trainingGs"(r2c|jrK|j\}}}}t|j||f|jrdn|j}||d|}n|j}|jri||z}tj|j |jddd|j |jddd|fd}nhtj|j |jddd|j |jddd|fd}||z}| |S)Nr)rr )dim) dynamic_img_sizeshaper r&no_embed_classrviewr cat cls_tokenexpandr#pos_drop)r-xBHWCr&s r1 _pos_embedz%VisionTransformerDistilled._pos_embedKs_  'JAq!Q.A'+':"V!!@VI q"a  AAI   I A %%agaj"b99&&qwqz2r:: AA %%agaj"b99&&qwqz2r:: A I A}}Qr2 pre_logitsc |dddf|dddf}}|r||zdz S||}||}|jr)|jr"tjs||fS||zdz S)Nrr r)r@r*r+trainingr jit is_scripting)r-rRrXx_dists r1 forward_headz'VisionTransformerDistilled.forward_headjsaaadGQqqq!tW6  $J!# # IIaLL''  " $t} $UY=S=S=U=U $f9 J!# #r2rFr>)T)__name__ __module__ __qualname____doc__rr,r r[ignorer;rModulerAintrstrrCrFrWboolTensorr^ __classcell__)r0s@r1rrsE ' ' ' ' '(((((( Y     Y) ))))kkCkhsmkkkk  Y))))   > $ $$ $5< $ $ $ $ $ $ $ $r2Fc |dd}|rtnt}t|||ft t dt |dd|}|S)N out_indicesT)adapt_layer_scalegetter)rm feature_cls)pretrained_filter_fn feature_cfg)rrr rrr r9)variant pretrained distilledr/rm model_clsmodels r1 _create_deitryxs|**]A..K.7N**=NI  %%9TRRR[hGGG       E Lr2rc 6|ddddddttddd |S) Ni)rnr{g?bicubicTzpatch_embed.projr@) urlr' input_size pool_sizecrop_pct interpolationfixed_input_sizemeanr4 first_conv classifierr)r}r/s r1_cfgrs7=t%.B(      r2zdeit_tiny_patch16_224.fb_in1kztimm/zFhttps://dl.fbaipublicfiles.com/deit/deit_tiny_patch16_224-a1311bcf.pth) hf_hub_idr}zdeit_small_patch16_224.fb_in1kzGhttps://dl.fbaipublicfiles.com/deit/deit_small_patch16_224-cd65a155.pthzdeit_base_patch16_224.fb_in1kzFhttps://dl.fbaipublicfiles.com/deit/deit_base_patch16_224-b5f2ef4d.pthzdeit_base_patch16_384.fb_in1kzFhttps://dl.fbaipublicfiles.com/deit/deit_base_patch16_384-8de9b5d1.pth)rnrg?)rr}r~rz'deit_tiny_distilled_patch16_224.fb_in1kzPhttps://dl.fbaipublicfiles.com/deit/deit_tiny_distilled_patch16_224-b40b3cf7.pthr?)rr}rz(deit_small_distilled_patch16_224.fb_in1kzQhttps://dl.fbaipublicfiles.com/deit/deit_small_distilled_patch16_224-649709d9.pthz'deit_base_distilled_patch16_224.fb_in1kzPhttps://dl.fbaipublicfiles.com/deit/deit_base_distilled_patch16_224-df68dfff.pthz'deit_base_distilled_patch16_384.fb_in1kzPhttps://dl.fbaipublicfiles.com/deit/deit_base_distilled_patch16_384-d0272ac0.pth)rr}r~rrzdeit3_small_patch16_224.fb_in1kz;https://dl.fbaipublicfiles.com/deit/deit_3_small_224_1k.pthzdeit3_small_patch16_384.fb_in1kz;https://dl.fbaipublicfiles.com/deit/deit_3_small_384_1k.pthz deit3_medium_patch16_224.fb_in1kzhttps://dl.fbaipublicfiles.com/deit/deit_3_huge_224_21k_v1.pth)(deit3_small_patch16_384.fb_in22k_ft_in1k)deit3_medium_patch16_224.fb_in22k_ft_in1k'deit3_base_patch16_224.fb_in22k_ft_in1k'deit3_base_patch16_384.fb_in22k_ft_in1k(deit3_large_patch16_224.fb_in22k_ft_in1k(deit3_large_patch16_384.fb_in22k_ft_in1k'deit3_huge_patch14_224.fb_in22k_ft_in1kr<c ^tdddd}tdd|it|fi|}|S) z DeiT-tiny model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit.  rn patch_sizer"depth num_headsdeit_tiny_patch16_224ru)rr9ryrur/ model_argsrxs r1rrsI s"JJJJ  f fZ f4PZKeKe^dKeKe f fE Lr2c ^tdddd}tdd|it|fi|}|S) z DeiT-small model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrdeit_small_patch16_224ru)rrrs r1rrsI s"JJJJ  g gj gDQ[LfLf_eLfLf g gE Lr2c ^tdddd}tdd|it|fi|}|S)z DeiT base model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrdeit_base_patch16_224ru)rrrs r1rrI s"KKKJ  f fZ f4PZKeKe^dKeKe f fE Lr2c ^tdddd}tdd|it|fi|}|S)z DeiT base model @ 384x384 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrdeit_base_patch16_384ru)rrrs r1rrrr2c btdddd}t d |ddt|fi|}|S) z DeiT-tiny distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrnrdeit_tiny_distilled_patch16_224Trurv)rrrs r1rrs_ s"JJJJ ) p6@D p pTXYcTnTngmTnTn p pE Lr2c btdddd}t d |ddt|fi|}|S) z DeiT-small distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrr deit_small_distilled_patch16_224Tr)rrrs r1rr%s_ s"JJJJ * q7AT q qUYZdUoUohnUoUo q qE Lr2c btdddd}t d|ddt|fi|}|S) z DeiT-base distilled model @ 224x224 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrdeit_base_distilled_patch16_224Tr)rrrs r1rr0_ s"KKKJ ) p6@D p pTXYcTnTngmTnTn p pE Lr2c btdddd}t d|ddt|fi|}|S) z DeiT-base distilled model @ 384x384 from paper (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrdeit_base_distilled_patch16_384Tr)rrrs r1rr;rr2c btdddddd}td d |it|fi|}|S) z DeiT-3 small model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrTư>rr"rrrL init_valuesdeit3_small_patch16_224ru)rrrs r1rrFO s"Z^lpqqqJ  h hz hTR\MgMg`fMgMg h hE Lr2c btdddddd}td d |it|fi|}|S) z DeiT-3 small model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrrTrrdeit3_small_patch16_384ru)rrrs r1rrPrr2c btdddddd}td d |it|fi|}|S) z DeiT-3 medium model @ 224x224 (https://arxiv.org/abs/2012.12877). ImageNet-1k weights from https://github.com/facebookresearch/deit. rirTrrdeit3_medium_patch16_224ru)rrrs r1rrZsO s"Z^lpqqqJ  i i idS]NhNhagNhNh i iE Lr2c btdddddd}td d|it|fi|}|S) z DeiT-3 base model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrTrrdeit3_base_patch16_224ru)rrrs r1rrdO s"[_mqrrrJ  g gj gDQ[LfLf_eLfLf g gE Lr2c btdddddd}td d|it|fi|}|S) DeiT-3 base model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrTrrdeit3_base_patch16_384ru)rrrs r1rrnrr2c btdddddd}td d|it|fi|}|S) z DeiT-3 large model @ 224x224 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rTrrdeit3_large_patch16_224ru)rrrs r1rrxO t2\`nrsssJ  h hz hTR\MgMg`fMgMg h hE Lr2c btdddddd}td d|it|fi|}|S) z DeiT-3 large model @ 384x384 from paper (https://arxiv.org/abs/2204.07118). ImageNet-1k weights from https://github.com/facebookresearch/deit. rrrTrrdeit3_large_patch16_384ru)rrrs r1rrrr2c btdddddd}td d |it|fi|}|S) ri rTrrdeit3_huge_patch14_224ru)rrrs r1rrsO t2\`nrsssJ  g gj gDQ[LfLf_eLfLf g gE Lr2rrrrrrr) deit3_small_patch16_224_in21ft1k deit3_small_patch16_384_in21ft1k!deit3_medium_patch16_224_in21ft1kdeit3_base_patch16_224_in21ft1kdeit3_base_patch16_384_in21ft1k deit3_large_patch16_224_in21ft1k deit3_large_patch16_384_in21ft1kdeit3_huge_patch14_224_in21ft1k)FFr_r`),rd functoolsrtypingrr r timm.datarr timm.layersr timm.models.vision_transformerr r r _builderr _registryrrr__all__rryr default_cfgsrrrrrrrrrrrrrrrrrar2r1rs   AAAAAAAA......aaaaaaaaaa******YYYYYYYYYY ' (Y$Y$Y$Y$Y$!2Y$Y$Y$x    %$^&#TT T&V&V&V^& %dd U'W'W'W ^&$TT T&V&V&V^&$TT T 3&0&0&0^& .tt ^(0*0*0*!^&(/ _(1*1*1*)^&0.tt ^(0*0*0*1^&8.tt ^ 3( 0*0*0*9^&D&tt I(K(K(KE^&J&tt I 3(0(0(0K^&R' J)L)L)LS^&X%dd H'J'J'JY^&^%dd H 3'0'0'0_^&f&tt I(K(K(Kg^&l&tt I 3(0(0(0m^&t%dd H'J'J'Ju^&|/ J111}^&D15 J 310101026 K22204t I00004t I 300000015 J11115 J 310101004t L000u^&^&^&^^ B9J:K9J9JC]D^C]C];L;L