gͨ$dZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZdd lmZmZmZmZdd lmZdd lmZmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%d dl&m'Z'e#j(e)Z*dZ+dZ,gdZ-dZ.dZ/Gdde j0Z1Gdde j0Z2Gdde j0Z3Gdde3Z4Gdde j0Z5Gdde j0Z6Gdd e6Z7Gd!d"e j0Z8Gd#d$e j0Z9e6e7d%Z:Gd&d'e j0Z;Gd(d)e j0Z<Gd*d+eZ=d,Z>d-Z?e!d.e>Gd/d0e=Z@Gd1d2e j0ZAe!d3e>Gd4d5e=ZBe!d6e>Gd7d8e=ZCeGd9d:eZDe!d;e>Gd<d=e=ZEdS)>zPyTorch DeiT model.N) dataclass)OptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedImageModelingOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings torch_int) DeiTConfigrz(facebook/deit-base-distilled-patch16-224)riztabby, tabby catc eZdZdZddededdffd Zdejd e d e dejfd Z dd ejd e ej dedejfdZ xZS)DeiTEmbeddingszv Construct the CLS token, distillation token, position and patch embeddings. Optionally, also the mask token. Fconfiguse_mask_tokenreturnNczttjt jdd|j|_tjt jdd|j|_|r-tjt jdd|jnd|_ t||_ |j j }tjt jd|dz|j|_ tj|j|_|j|_dS)Nr)super__init__r Parametertorchzeros hidden_size cls_tokendistillation_token mask_tokenDeiTPatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropout patch_size)selfr!r"r1 __class__s b/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/deit/modeling_deit.pyr'zDeiTEmbeddings.__init__Cs ek!Q8J&K&KLL"$,u{1aAS/T/T"U"UQ_i",u{1a9K'L'LMMMei 3F ; ;+7 #%< A{QPVPb0c0c#d#d z&"<==  + embeddingsheightwidthc|jddz }|jjddz }tjs||kr ||kr|jS|jddddf}|jddddf}|jd}||jz} ||jz} t |dz} |d| | |}|dddd}tj || | fdd }|dddd dd|}tj ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing and 2 class embeddings. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rr%N?rr bicubicF)sizemode align_cornersdim)shaper2r)jit is_tracingr6rreshapepermuter functional interpolateviewcat) r7r;r<r=r1 num_positionsclass_and_dist_pos_embedpatch_pos_embedrF new_height new_widthsqrt_num_positionss r9interpolate_pos_encodingz'DeiTEmbeddings.interpolate_pos_encodingOst!&q)A- 06q9A= y##%% ,+*F*F6UZ??+ +#'#;AAArrE#B 2111abb59r"t. T_, &}c'9::)11!5GI[]`aa)11!Q1==-33 i( 4  *11!Q1==BB1b#NNy2OD!LLLLr: pixel_valuesbool_masked_posrVc6|j\}}}}||}|\}} }|R|j|| d} |d| } |d| z z| | zz}|j|dd} |j|dd} tj | | |fd}|j }|r| |||}||z}| |}|S)Nr??rrE)rGr0rBr.expand unsqueezetype_asr,r-r)rOr2rVr5)r7rWrXrV_r<r=r; batch_size seq_length mask_tokensmask cls_tokensdistillation_tokensposition_embeddings r9forwardzDeiTEmbeddings.forwardws4 +01fe**<88 $.OO$5$5! J  &/00ZLLK",,R0088EED#sTz2[45GGJ^**:r2>> "5<!>z6SX!Y!Y "44 \\*-- r:)FNF)__name__ __module__ __qualname____doc__rboolr'r)TensorintrVr BoolTensorrf __classcell__r8s@r9r r >s , ,z ,4 ,D , , , , , ,&M5<&M&MUX&M]b]i&M&M&M&MV7;). l"%"23#'   r:r cFeZdZdZfdZdejdejfdZxZS)r/z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct|j|j}}|j|j}}t |tjj r|n||f}t |tjj r|n||f}|d|dz|d|dzz}||_||_||_||_ tj |||||_ dS)Nrr) kernel_sizestride)r&r' image_sizer6 num_channelsr+ isinstance collectionsabcIterabler1rConv2d projection)r7r!rvr6rwr+r1r8s r9r'zDeiTPatchEmbeddings.__init__s !'!2F4EJ $*$79Kk #-j+/:R#S#SqZZZdfpYq #-j+/:R#S#SqZZZdfpYq !!} 1 5*Q-:VW=:XY $$(&)L+:^hiiir:rWr#c|j\}}}}||jkrtd||ddd}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r%r)rGrw ValueErrorr}flatten transpose)r7rWr_rwr<r=xs r9rfzDeiTPatchEmbeddings.forwardsl2>2D/ L&% 4, , ,w  OOL ) ) 1 1! 4 4 > >q! D Dr:) rhrirjrkr'r)rmrfrprqs@r9r/r/sm j j j j jELU\r:r/c eZdZdeddffd ZdejdejfdZ d deejd e de e ejejfe ejffd Z xZ S) DeiTSelfAttentionr!r#Nct|j|jzdkr1t |ds!t d|jfd|jd|j|_t |j|jz |_|j|jz|_tj |j|j|j |_ tj |j|j|j |_ tj |j|j|j |_tj|j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)bias)r&r'r+num_attention_headshasattrrrnattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer3attention_probs_dropout_probr5r7r!r8s r9r'zDeiTSelfAttention.__init__s1    : :a ? ?PVXhHiHi ?76#5"7773777  $*#= #&v'9FS"S%**+BCC 6G]=/22mM]r:rg)rhrirjrr'r)rmrrrlrrrfrprqs@r9rrsGzGdGGGGGG$%el%u|%%%% bg!!(0(>!Z^! uU\5</0% 2EE F!!!!!!!!r:rc eZdZdeddffd Z d dejdeejde de e ejejfe ejfffd Z xZ S) DeiTSdpaSelfAttentionr!r#Ncbt||j|_dSN)r&r'rrs r9r'zDeiTSdpaSelfAttention.__init__s,    ,2,O)))r:Frrrc |s|>tdt|||S||}|||}|||}||}tj j |||||j r|j nddd}|dddd }|dd |jfz} || }|dfS) Na`DeiTSdpaAttention` is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrrF) is_causalscalerr%rr r)logger warning_oncer&rfrrrrr)rrLscaled_dot_product_attentiontrainingrrKrrBrrN) r7rrrrrrrrrr8s r9rfzDeiTSdpaSelfAttention.forwardsp   5   w    77??+#"3#  !JJ}55--dhh}.E.EFF // =0I0IJJ //0ABB +HH    15 GD - -CI  &--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S%**+BCC d""r:rg)rhrirjrr'r) FloatTensorrrmrlrrrfrprqs@r9rrsPzPdPPPPPP-1"' '#'#('#EL)'# '# uU\5</0% 2EE F '#'#'#'#'#'#'#'#'#'#r:rc^eZdZdZdeddffd ZdejdejdejfdZxZ S) DeiTSelfOutputz The residual connection is defined in DeiTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r!r#Ncttj|j|j|_tj|j|_dSr) r&r'rrr+denser3r4r5rs r9r'zDeiTSelfOutput.__init__+sJ Yv163EFF z&"<== r:r input_tensorcZ||}||}|Srrr5r7rrs r9rfzDeiTSelfOutput.forward0s* =11  ]33 r:) rhrirjrkrr'r)rmrfrprqs@r9rr%s >z>d>>>>>> U\RWR^r:rc eZdZdeddffd ZdeeddfdZ d dej d e ej d e de e ej ej fe ej ffd ZxZS) DeiTAttentionr!r#Nctt||_t ||_t |_dSr)r&r'r attentionroutputset pruned_headsrs r9r'zDeiTAttention.__init__9sI *622$V,, EEr:headsct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)NrrrE)lenrrrrrrrrrrrrunion)r7rindexs r9 prune_headszDeiTAttention.prune_heads?s u::?? F7 4>5t~7Y[_[l  u 2$.2FNN/0BEJJ1$.2FNN.t{/@%QOOO .2^-ORUV[R\R\-\*'+~'IDNLn'n$ -33E::r:Frrrc||||}||d|}|f|ddz}|S)Nrr)rr)r7rrr self_outputsattention_outputrs r9rfzDeiTAttention.forwardQsM ~~mY@QRR ;;|A FF#% QRR(88r:rg)rhrirjrr'rrnrr)rmrrlrrrfrprqs@r9rr8s"z"d"""""" ;S;d;;;;*-1"'  | EL)   uU\5</0% 2EE F        r:rc(eZdZdeddffd ZxZS)DeiTSdpaAttentionr!r#Ncrt|t||_dSr)r&r'rrrs r9r'zDeiTSdpaAttention.__init__as.    .v66r:)rhrirjrr'rprqs@r9rr`sK7z7d7777777777r:rcLeZdZdeddffd ZdejdejfdZxZS)DeiTIntermediater!r#Ncttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r&r'rrr+intermediate_sizerrx hidden_actstrr intermediate_act_fnrs r9r'zDeiTIntermediate.__init__hsn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r:rcZ||}||}|Sr)rr)r7rs r9rfzDeiTIntermediate.forwardps, =11 00?? r: rhrirjrr'r)rmrfrprqs@r9rrgsq9z9d999999U\elr:rcZeZdZdeddffd ZdejdejdejfdZxZS) DeiTOutputr!r#Ncttj|j|j|_tj|j|_ dSr) r&r'rrrr+rr3r4r5rs r9r'zDeiTOutput.__init__ysJ Yv79KLL z&"<== r:rrcd||}||}||z}|Srrrs r9rfzDeiTOutput.forward~s4 =11  ]33 % 4 r:rrqs@r9rrxs|>z>d>>>>>> U\RWR^r:r)eagersdpac eZdZdZdeddffd Z d dejdeejd e de e ejejfe ejffd Z xZ S) DeiTLayerz?This corresponds to the Block class in the timm implementation.r!r#Nct|j|_d|_t |j||_t||_t||_ tj |j |j|_tj |j |j|_dS)Nreps)r&r'chunk_size_feed_forward seq_len_dimDEIT_ATTENTION_CLASSES_attn_implementationrr intermediaterrr LayerNormr+layer_norm_epslayernorm_beforelayernorm_afterrs r9r'zDeiTLayer.__init__s '-'E$/0KLVTT,V44 (( " V-?VEZ [ [ [!|F,>FDYZZZr:Frrrc|||||}|d}|dd}||z}||}||}|||}|f|z}|S)N)rrr)rrrrr)r7rrrself_attention_outputsrr layer_outputs r9rfzDeiTLayer.forwards "&  ! !- 0 0 /"0" "  2!4(,)=8 ++M:: ((66 {{<?? /G+r:rg)rhrirjrkrr'r)rmrrlrrrfrprqs@r9rrsII[z[d[[[[[[-1"' |EL)  uU\5</0% 2EE F r:rceZdZdeddffd Z d dejdeejd ed ed ede e e ff d Z xZ S) DeiTEncoderr!r#Nct|_tjfdt jD|_d|_dS)Nc.g|]}tS)r).0r^r!s r9 z(DeiTEncoder.__init__..s!#_#_#_!If$5$5#_#_#_r:F) r&r'r!r ModuleListrangenum_hidden_layerslayergradient_checkpointingrs `r9r'zDeiTEncoder.__init__s`  ]#_#_#_#_uVE]?^?^#_#_#_`` &+###r:FTrrroutput_hidden_states return_dictc|rdnd}|rdnd}t|jD]h\}} |r||fz}|||nd} |jr%|jr|| j|| |} n | || |} | d}|r || dfz}i|r||fz}|st d|||fDSt|||S)Nrrrc3K|]}||V dSrr)rvs r9 z&DeiTEncoder.forward..s(mmq_`_l_l_l_l_lmmr:)last_hidden_stater attentions) enumeraterrr_gradient_checkpointing_func__call__tupler) r7rrrrrall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r9rfzDeiTEncoder.forwardsI#7@BBD$5?bb4(44 P POA|# I$58H$H!.7.CillO* `t} ` $ A A )!#% !! !- ]OM^ _ _ )!,M  P&9]1=M