gDdZddlZddlmZmZmZmZddlZddlZddlm Z ddl m Z m Z ddl mZddlmZmZmZdd lmZdd lmZmZdd lmZmZmZmZmZd d lmZej e!Z"dZ#dZ$Gdde j%Z&Gdde j%Z'Gdde j%Z(Gdde(Z)Gdde j%Z*Gdde j%Z+Gdde+Z,Gdde j%Z-Gd d!e j%Z.e+e,d"Z/Gd#d$e j%Z0Gd%d&e j%Z1Gd'd(e j%Z2Gd)d*eZ3d+Z4d,Z5ed-e4Gd.d/e3Z6ed0e4Gd1d2e3Z7dS)3zPyTorch ViViT model.N)OptionalSetTupleUnion)nn)CrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings torch_int) VivitConfigzgoogle/vivit-b-16x2-kinetics400rc0eZdZdZfdZddefdZxZS)VivitTubeletEmbeddingsa Construct Vivit Tubelet embeddings. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]). ct|j|_|j|_|j|_|j|jdz|j|jdzz|j|jdzz|_|j|_tj |j |j|j|j|_ dS)Nrr) kernel_sizestride) super__init__ num_frames image_size tubelet_size patch_size num_patches hidden_size embed_dimrConv3d num_channels projectionselfconfig __class__s d/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/vivit/modeling_vivit.pyr zVivitTubeletEmbeddings.__init__8s  + + - _ 2 2$/!"44 6$/!"44 6   +)  !3AT]c]p   Finterpolate_pos_encodingc b|j\}}}}}|sH||jks ||jkr2td|d|d|jdd|jdd |ddddd }||}|ddd}|S) NzImage image size (*z) doesn't match model (rrz).rr )shaper" ValueErrorpermuter*flatten transpose) r, pixel_valuesr1 batch_sizer!r)heightwidthxs r/forwardzVivitTubeletEmbeddings.forwardHs>J>P; J fe' Vt-F-F%SWSbJbJbAVAAeAA_Q'AA*./!*<AAA  $++Aq!Q:: OOL ) ) IIaLL " "1a ( (r0F)__name__ __module__ __qualname____doc__r boolr? __classcell__r.s@r/rr-sb     dr0rc\eZdZdZfdZdejdededejfdZd d e fd Z xZ S) VivitEmbeddingsz Vivit Embeddings. Creates embeddings from a video using VivitTubeletEmbeddings, adds CLS token and positional embeddings. cttjt jdd|j|_t||_ tjt jd|j j dz|j|_ tj |j |_|jdd|_||_dS)Nr)rr r Parametertorchzerosr& cls_tokenrpatch_embeddingsr%position_embeddingsDropouthidden_dropout_probdropoutr#r$r-r+s r/r zVivitEmbeddings.__init__as ek!Q8J&K&KLL 6v > >#%< K40J>P; J fe**>J\\*-- r0r@) rArBrCrDr rLTensorintr1rEr?rFrGs@r/rIrIZs     &D5<&D&DUX&D]b]i&D&D&D&DPdr0rIc eZdZdeddffd ZdejdejfdZ d deejd e de e ejejfe ejffd Z xZ S) VivitSelfAttentionr-rUNct|j|jzdkr1t |ds!t d|jfd|jd|j|_t |j|jz |_|j|jz|_tj |j|j|j |_ tj |j|j|j |_ tj |j|j|j |_tj|j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)bias)rr r&num_attention_headshasattrr6rpattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluerQattention_probs_dropout_probrSr+s r/r zVivitSelfAttention.__init__s1    : :a ? ?PVXhHiHi ?76#5"7773777  $*#= #&v'9Fc|dd|j|jfz}||}|ddddS)NrWrrrr )rYrwryrcr7)r,r> new_x_shapes r/transpose_for_scoresz'VivitSelfAttention.transpose_for_scoressPffhhssmt'?AY&ZZ FF;  yyAq!$$$r0F head_maskoutput_attentionsc||}|||}|||}||}t j||dd}|tj|j z }tj |d} | | } || |z} t j| |} | dddd} | dd|jfz} | | } |r| | fn| f} | S)NrWr\rrrr )r}rr~rrLmatmulr9mathsqrtryrrasoftmaxrSr7 contiguousrYrzrc) r, hidden_statesrrmixed_query_layer key_layer value_layer query_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputss r/r?zVivitSelfAttention.forwardsr!JJ}55--dhh}.E.EFF // =0I0IJJ //0ABB !< Y5H5HR5P5PQQ+di8P.Q.QQ-//0@b/II,,77  - 9O _kBB %--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S%**+BCC 6G]=/22mM]r0NF)rArBrCrr rLrorrrErrr?rFrGs@r/rrrrsG{GtGGGGGG$%el%u|%%%% bg!!(0(>!Z^! uU\5</0% 2EE F!!!!!!!!r0rrc eZdZdeddffd Z d deejdede e ejejfe ejfffd Z xZ S) VivitSdpaSelfAttentionr-rUNcbt||j|_dSN)rr rr+s r/r zVivitSdpaSelfAttention.__init__s,    ,2,O)))r0Frrc |s|=tdt|||S||}|||}|||}||}tj j |||||j r|j nddd}|dddd}|dd |jfz} || }|dfS) NaVivitSdpaSelfAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True` or `head_mask`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.F) is_causalscalerrrr r)logger warning_oncerr?r}rr~rrLrrascaled_dot_product_attentiontrainingrr7rrYrzrc) r,rrrrrrrrrr.s r/r?zVivitSdpaSelfAttention.forwardsk   5   d    77??!  !JJ}55--dhh}.E.EFF // =0I0IJJ //0ABB +HH    15 GD - -CI  &--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S%**+BCC d""r0r) rArBrCrr rrLrorErrr?rFrGs@r/rrsP{PtPPPPPP bg$#$#(0(>$#Z^$# uU\5</0% 2EE F$#$#$#$#$#$#$#$#$#$#r0rc^eZdZdZdeddffd ZdejdejdejfdZxZ S) VivitSelfOutputz The residual connection is defined in VivitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. r-rUNcttj|j|j|_tj|j|_dSr) rr rr{r&denserQrRrSr+s r/r zVivitSelfOutput.__init__sJ Yv163EFF z&"<== r0r input_tensorcZ||}||}|SrrrSr,rrs r/r?zVivitSelfOutput.forwards* =11  ]33 r0) rArBrCrDrr rLror?rFrGs@r/rrs >{>t>>>>>> U\RWR^r0rc eZdZdeddffd ZdeeddfdZ d dej d e ej d e de e ej ej fe ej ffd ZxZS) VivitAttentionr-rUNctt||_t ||_t |_dSr)rr rr attentionroutputset pruned_headsr+s r/r zVivitAttention.__init__(sI +F33%f-- EEr0headsct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrrr\)lenrrrwryrrr}r~rrrrzunion)r,rindexs r/ prune_headszVivitAttention.prune_heads.s u::?? F7 4>5t~7Y[_[l  u 2$.2FNN/0BEJJ1$.2FNN.t{/@%QOOO .2^-ORUV[R\R\-\*'+~'IDNLn'n$ -33E::r0Frrrc||||}||d|}|f|ddz}|S)Nrr)rr)r,rrr self_outputsattention_outputrs r/r?zVivitAttention.forward@sM ~~mY@QRR ;;|A FF#% QRR(88r0r)rArBrCrr rrprrLrorrErrr?rFrGs@r/rr's"{"t"""""" ;S;d;;;;*-1"'  | EL)   uU\5</0% 2EE F        r0rc(eZdZdeddffd ZxZS)VivitSdpaAttentionr-rUNcrt|t||_dSr)rr rrr+s r/r zVivitSdpaAttention.__init__Ps.    /77r0)rArBrCrr rFrGs@r/rrOsK8{8t8888888888r0rc$eZdZfdZdZxZS)VivitIntermediatecJttj|j|j|_tj|j|_ t|j trt|j |_dS|j |_dSr)rr rr{r&intermediate_sizerrQrRrS isinstance hidden_actstrr intermediate_act_fnr+s r/r zVivitIntermediate.__init__Vs Yv163KLL z&"<== f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r0c||}||}||}|Sr)rrrS)r,rs r/r?zVivitIntermediate.forward_s= =11 00??  ]33 r0rArBrCr r?rFrGs@r/rrUsG99999r0rc$eZdZfdZdZxZS) VivitOutputcttj|j|j|_tj|j|_ dSr) rr rr{rr&rrQrRrSr+s r/r zVivitOutput.__init__hsJ Yv79KLL z&"<== r0cd||}||}||z}|Srrrs r/r?zVivitOutput.forwardms4 =11  ]33 % 4 r0rrGs@r/rrgsG>>>>> r0r)eagersdpac*eZdZdZfdZddZxZS) VivitLayerzNThis corresponds to the EncoderBlock class in the scenic/vivit implementation.ct|j|_d|_t |j||_t||_t||_ tj |j |j|_tj |j |j|_dS)Nreps)rr chunk_size_feed_forward seq_len_dimVIVIT_ATTENTION_CLASSES_attn_implementationrr intermediaterrr LayerNormr&layer_norm_epslayernorm_beforelayernorm_afterr+s r/r zVivitLayer.__init__s '-'E$01LMfUU-f55!&)) " V-?VEZ [ [ [!|F,>FDYZZZr0NFc|||||}|d}|dd}||z}||}||}|||}|f|z}|S)N)rrr)rrrrr)r,rrrself_attention_outputsrr layer_outputs r/r?zVivitLayer.forwards!%  ! !- 0 0 / "0" "  2!4(,)=8 ++M:: ((66 {{<?? /G+r0r)rArBrCrDr r?rFrGs@r/rr}sWXX[[[[[r0rc.eZdZfdZ ddZxZS) VivitEncoderct|_tjfdt jD|_d|_dS)Nc.g|]}tS)r).0_r-s r/ z)VivitEncoder.__init__..s!#`#`#`1Jv$6$6#`#`#`r0F) rr r-r ModuleListrangenum_hidden_layerslayergradient_checkpointingr+s `r/r zVivitEncoder.__init__s`  ]#`#`#`#`fF^@_@_#`#`#`aa &+###r0NFTc|rdnd}|rdnd}t|jD]h\}} |r||fz}|||nd} |jr%|jr|| j|| |} n | || |} | d}|r || dfz}i|r||fz}|st d|||fDSt|||S)Nrrrc3K|]}||V dSrr)rvs r/ z'VivitEncoder.forward..s(mmq_`_l_l_l_l_lmmr0)last_hidden_stater attentions) enumeraterrr_gradient_checkpointing_func__call__tupler ) r,rrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r/r?zVivitEncoder.forwardsI#7@BBD$5?bb4(44 P POA|# I$58H$H!.7.CillO* `t} ` $ A A )!#% !! !- ]OM^ _ _ )!,M  P&9]1=M>>>>.-  - - M K  " " $ $ $ M  $ $S ) ) ) ) )  - - M K  Sdk.K  L L L L L M Mr0N) rArBrCrDr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdparrr0r/rrsU L$O&*#NMMMMMr0raG This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`VivitConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_frames, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`VivitImageProcessor`]. See [`VivitImageProcessor.preprocess`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*, `False`): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z_The bare ViViT Transformer model outputting raw hidden-states without any specific head on top.ceZdZdfd ZdZdZeeee e  dde e j d e e j d e ed e ed ed e edeee j e ffdZxZS) VivitModelTcJt|||_t||_t ||_tj|j |j |_ |rt|nd|_ |dS)Nr)rr r-rIrTrencoderrrr&r layernormrpooler post_init)r,r-add_pooling_layerr.s r/r zVivitModel.__init__/s     )&11#F++ f&8f>STTT->Hk&)))D  r0c|jjSr)rTrO)r,s r/get_input_embeddingszVivitModel.get_input_embeddings<s //r0c|D]/\}}|jj|j|0dS)z Prunes heads of the model. Args: heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsr rrr)r,heads_to_prunerrs r/ _prune_headszVivitModel._prune_heads?sU+0022 C CLE5 L u % / ; ;E B B B B C Cr0 output_typerNFr:rrrr1rrUc||n |jj}||n |jj}||n |jj}|t d|||jj}|||}||||||}|d} | | } |j | | nd} |s| | f|ddzSt| | |j |j S)a Returns: Examples: ```python >>> import av >>> import numpy as np >>> from transformers import VivitImageProcessor, VivitModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 3137, 768] ```Nz You have to specify pixel_valuesrl)rrrrrr)r pooler_outputrr)r-rruse_return_dictr6 get_head_maskrrTr r!r"r rr) r,r:rrrr1rembedding_outputencoder_outputssequence_outputrs r/r?zVivitModel.forwardJs>n2C1N--TXT_Tq$8$D $+Jj &1% u 'tn u #' u d^u  uU&')CC Du u u edCBu u u u u r0ra ViViT Transformer model with a video classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for Kinetics-400. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. c*eZdZfdZeeeee dde e j de e j de e j de e d e e d e d e e d eee j effd ZxZS)VivitForVideoClassificationc:t||j|_t|d|_|jdkrt j|j|jnt j|_ | dS)NF)r$r) rr num_labelsrr rr{r&Identity classifierr#r+s r/r z$VivitForVideoClassification.__init__s     +%@@@ OUN_bcNcNc")F$68IJJJikitiviv r0r+NFr:rlabelsrrr1rrUcF||n |jj}|||||||}|d} || dddddf} d} ||jdkrAt } | | d|d} nFt} | | d|j|d} |s| f|ddz} | | f| zn| St| | |j |j S)a( labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> import av >>> import numpy as np >>> import torch >>> from transformers import VivitImageProcessor, VivitForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`List[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`List[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) LABEL_116 ```N)rrrr1rrrrWr)losslogitsrr) r-r/r r<r:r rcrrrr)r,r:rr=rrr1rrr3r@r?loss_fctrs r/r?z#VivitForVideoClassification.forwardsL@&1% @ 'tn @ #'@ d^@  uU&')>> ?@ @ @ `_CB@ @ @ @ @ r0r8)8rDrtypingrrrrrLtorch.utils.checkpointrtorch.nnrr activationsr modeling_outputsr r rmodeling_utilsr pytorch_utilsrrutilsrrrrrconfiguration_vivitr get_loggerrAr_CHECKPOINT_FOR_DOCr5ModulerrIrrrrrrrrrrrrrVIVIT_START_DOCSTRINGr4rr8rr0r/rPs{ ............ ........!!!!!!bbbbbbbbbb------QQQQQQQQ-,,,,,  H % %7*****RY***ZLLLLLbiLLL`99999999z)#)#)#)#)#/)#)#)#Zbi&$$$$$RY$$$P88888888  $     ")   "  $$$$$$$$N0 0 0 0 0 290 0 0 f     ")   MMMMM?MMM@ 2eS S S S S %S S S l O O O O O "6O O O O O r0