gxdZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!m"Z"m#Z#d dl$m%Z%e"j&e'Z(dZ)dZ*eGddeZ+Gdde j,Z-Gdde j,Z.Gdde j,Z/Gdde j,Z0Gdde j,Z1Gdde j,Z2Gdd e j,Z3Gd!d"e j,Z4Gd#d$e j,Z5Gd%d&e j,Z6Gd'd(eZ7d)Z8d*Z9d+Z:e d,e8Gd-d.e7Z;Gd/d0e j,Z<e d1e8Gd2d3e7Z=Gd4d5e j,Z>Gd6d7e j,Z?e d8e8Gd9d:e7Z@e d;e8Gd<d=e7ZAe d>e:Gd?d@e7ZBe dAe8GdBdCe7ZCdS)DzPyTorch ViLT model.N) dataclass)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesmeshgridprune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) ViltConfigrzdandelin/vilt-b32-mlmceZdZUdZdZeejed<dZ ejed<dZ ee e ejed<dZ ee e ejed<dS)(ViltForImagesAndTextClassificationOutputa Class for outputs of [`ViltForImagesAndTextClassification`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the attention weights of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlosslogits hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rr rrr!b/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/vilt/modeling_vilt.pyrr4s$)-D(5$ %,,, $FE $$$>BM8Du'8!9:;BBB;?JeE$5678?????r*rc4eZdZdZfdZddZ ddZxZS) ViltEmbeddingsz Construct the text and patch embeddings. Text embeddings are equivalent to BERT embeddings. Patch embeddings are equivalent to ViT embeddings. ctt||_t jt jdd|j|_ t||_ |j j }t jt jd|dz|j|_ t j|j|j|_t j|j|_||_dSNr)super__init__TextEmbeddingstext_embeddingsr Parameterr&zeros hidden_size cls_tokenViltPatchEmbeddingspatch_embeddings num_patchesposition_embeddings Embeddingmodality_type_vocab_sizetoken_type_embeddingsDropouthidden_dropout_probdropoutconfig)selfrBr: __class__s r+r1zViltEmbeddings.__init__Ws  .f55ek!Q8J&K&KLL 3F ; ;+7 #%< A{QPVPb0c0c#d#d %'\&2QSYSe%f%f"z&"<==  r*c   !"#|jjjj\}}}}||}|dddddddf}t j||jd|jdf}|dddf ddddf} |dddf ddddf} |j\} } #|j j |j j z} |j ddddddfddd| | | !t!j!#fdt%| | Dd}|ddd}|ddd}t!jt+t!j|jdt!j|jd d d |j }|ddddddddf}||jd|jdd d d }|dd}|d}dkst5t6s| | z}|n'| | z}t;||d "d|z d  "dddf}"fd|D} fd|D}d|D}d|D}fd|D}g}tAt%|||D]\}\}}}|dkr[t!j!t!j"|}|#|||jt!j!t!j"||d}|#t!j|||||gdt!j|d}||dddf|dddff| d | }||dddf|dddff| d }||dddf|dddff| d d}||dddf|dddff| d | }|j$| d d }t!j||fd}t!j|j dddddfdddddf| d d |fd}||z}|%|}t!jt!j"|jdd||gd}|||#fffS)Nr sizerrdimc g|]R\}}tjtj||fddd|z d|z fSS)bilinearT)rImode align_cornersr)r functionalpad interpolate).0hwheight spatial_poswidths r+ z/ViltEmbeddings.visual_embed..ss   Aq !!M--#V'&* .  1fqj1   r*ij)indexingdeviceF)as_tuplec<g|]}dddf|kSNrr))rSu valid_idxs r+rYz/ViltEmbeddings.visual_embed..s/NNNQ9QQQT?a#78NNNr*c<g|]}dddf|kSrbr))rSrc non_valid_idxs r+rYz/ViltEmbeddings.visual_embed..s0ZZZ]=A+>!+CDZZZr*c8g|]}|dSrrHrSvs r+rYz/ViltEmbeddings.visual_embed..s"777AaffQii777r*c8g|]}|dSrhrHris r+rYz/ViltEmbeddings.visual_embed..s"???!&&))???r*cg|]}|z Sr)r))rSrjmax_image_lengths r+rYz/ViltEmbeddings.visual_embed..s===Q$q(===r*T) replacement)&r9 projectionweightshapefloatrrPrRlongsumrB image_size patch_sizer; transposeviewr&catzipflattenstackrarangetor_expand isinstanceintmaxminnonzerounique enumerate multinomialonesappendr7rA)$rC pixel_values pixel_maskrm_phpwxx_maskx_hx_w batch_size num_channels patch_dim pos_embed patch_indexeffective_resolution unique_rows valid_row_idxnon_valid_row_idx valid_numsnon_valid_numspad_numsselectirjnvp valid_choice pad_choice cls_tokensrVrfrWrdrXs$ ` @@@@@r+ visual_embedzViltEmbeddings.visual_embedfs,7>D 1b"  ! !, / /AAAtQQQM*0022**6QWQZ8P*QQVVXXQQQTl1%%aaad+QQQTl1%%aaad+23'/ L&%K*dk.DD .qqq!""aaax8BB1aHHMMaQ]_hjstt I       SMM      %%a((221a88 IIaLL " "1a ( (k U\&,r"233U\&,rBR5S5S^b c c cik   "FM" " " "$aaaAAA"56 !((a&,q/2rSUVV !))!Q// "" a  #3#;:N^`cCdCd#; $'9 37799  #&9 "#7#;#;#=#=?OPP NNEN22 V,,e,<< 1o,,.. NNNN+NNN ZZZZkZZZ77777 ??->???====*===&s:~x'P'PQQ f fMAz2qAvv$0A1D1D1F1FHXYY  mA.|<====".uz"~~/C/C/E/EqVZ[[[  eiq)9;LQ;OPZ;[(\bcdddeeee6q))) fQQQTlF111a4L( ) . .z2| L Lqqq!t fQQQTl2388RHH!&A,qqq!t "<=BB:rSTUU fQQQTlF111a4L89>>z2|\\ ^**:r2>> Iz1o1 - - -I  %aaaAAAg .qqq$z : A A*bRT U UW` agh     M LLOOEJv|A::==fEEvNTUVVV&;888r*rc ||||} |'||||jj\}} } n|d} |d}| |t j|t j| j z} ||t j | |t j| j z}t j | |gd} t j || gd} | | fS)N) input_idstoken_type_ids inputs_embeds)rmrdtyper_rJ) r3rrBrmr{r>r& zeros_likersr_ full_likery)rCrattention_maskrrrr image_embedsimage_token_type_idx text_embeds image_masksr embeddingsmaskss r+forwardzViltEmbeddings.forwards0**m+   595F5Fj4;;W6G66 2L+{{%,,Q//K '#$ !D$>$>  ^5:kFX Y Y Y% %  $d&@&@ OK)=UZXcXj k k k' '  Y \:BBB  >;7Q???5  r*)rE)r)r"r#r$r%r1rr __classcell__rDs@r+r-r-Nst     V9V9V9V9B'!'!'!'!'!'!'!'!r*r-c*eZdZdZfdZddZxZS)r2zGConstruct the embeddings from word, position and token_type embeddings.cttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|_t#|dd|_|dt)j|jdd|d t)j|jt(j ddS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rr[F) persistentrr)r0r1rr< vocab_sizer6 pad_token_idword_embeddingsmax_position_embeddingsr;type_vocab_sizer> LayerNormlayer_norm_epsr?r@rAgetattrrregister_bufferr&r}rr5rrIrsrCrBrDs r+r1zTextEmbeddings.__init__sK !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2H&J\%]%]"f&8f>STTTz&"<== '.v7PR\']']$  EL)GHHOOPWXXej      ek$*;*@*@*B*B%*UUUbg      r*Nc||}n|dd}|d}||jddd|f}|mt|dr2|jddd|f}||d|}|}n+t j|t j|jj}|| |}| |} || z} |j dkr| |} | | z } | | } || } | S)Nr[rrrrr)rIrhasattrrrr&r5rsr_rr>rr;rrA) rCrrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr>rr;s r+rzTextEmbeddings.forwardsb  #..**KK',,..ss3K ^  ,QQQ  ^ J J"%::  ': 5 5"&":":<"H"H  - -J^^J// \\*-- r*)NNNNr"r#r$r%r1rrrs@r+r2r2sRQQ     &        r*r2c(eZdZdZfdZdZxZS)r8z# Image to Patch Embedding. ct|j|j}}|j|j}}t |tjj r|n||f}t |tjj r|n||f}|d|dz|d|dzz}||_||_||_||_ tj |||||_ dS)Nrr) kernel_sizestride)r0r1rurvrr6r collectionsabcIterabler:rConv2dro)rCrBrurvrr6r:rDs r+r1zViltPatchEmbeddings.__init__&s !'!2F4EJ $*$79Kk #-j+/:R#S#SqZZZdfpYq #-j+/:R#S#SqZZZdfpYq !!} 1 5*Q-:VW=:XY $$(&)L+:^hiiir*c|j\}}}}||jkrtd|jjj}|||}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rqr ValueErrorrorprr~)rCrrrrVrX target_dtypers r+rzViltPatchEmbeddings.forward5si2>2D/ L&% 4, , ,w -3 OOLOO,O?? @ @r*rrs@r+r8r8!sV j j j j jr*r8c,eZdZfdZdZddZxZS)ViltSelfAttentionct|j|jzdkr1t |ds!t d|jfd|jd|j|_t |j|jz |_|j|jz|_tj |j|j|j |_ tj |j|j|j |_ tj |j|j|j |_tj|j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r0r1r6num_attention_headsrrrattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer?attention_probs_dropout_probrArs r+r1zViltSelfAttention.__init__As1    : :a ? ?PVXhHiHi ?76#5"7773777  $*#= #&v'9F>,,77  - 9O _kBB %--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S* *,CD 6G]=/22mM]r*NNF)r"r#r$r1rrrrs@r+rr@s`GGGGG$%%% !!!!!!!!r*rc^eZdZdZdeddffd ZdejdejdejfdZxZ S) ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rBreturnNcttj|j|j|_tj|j|_dSN) r0r1rrr6denser?r@rArs r+r1zViltSelfOutput.__init__sJ Yv163EFF z&"<== r*r input_tensorcZ||}||}|Sr r rArCr r s r+rzViltSelfOutput.forwards* =11  ]33 r*) r"r#r$r%rr1r&Tensorrrrs@r+rr}s >z>d>>>>>> U\RWR^r*rc,eZdZfdZdZddZxZS) ViltAttentionctt||_t ||_t |_dSr )r0r1r attentionroutputset pruned_headsrs r+r1zViltAttention.__init__sI *622$V,, EEr*ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)NrrrJ)lenrrrrrrrrrrr runion)rCheadsindexs r+ prune_headszViltAttention.prune_headss u::?? F7 4>5t~7Y[_[l  u 2$.2FNN/0BEJJ1$.2FNN.t{/@%QOOO .2^-ORUV[R\R\-\*'+~'IDNLn'n$ -33E::r*NFc|||||}||d|}|f|ddz}|S)Nrr)rr)rCr rrr self_outputsattention_outputrs r+rzViltAttention.forwardsM~~m^YPabb ;;|A FF#% QRR(88r*r)r"r#r$r1rrrrs@r+rrs[""""" ;;;$r*rcLeZdZdeddffd ZdejdejfdZxZS)ViltIntermediaterBrNcttj|j|j|_t|jtrt|j|_ dS|j|_ dSr ) r0r1rrr6intermediate_sizer r hidden_actstrr intermediate_act_fnrs r+r1zViltIntermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r*r cZ||}||}|Sr )r r&rCr s r+rzViltIntermediate.forwards, =11 00?? r* r"r#r$rr1r&rrrrs@r+r!r!sq9z9d999999U\elr*r!cZeZdZdeddffd ZdejdejdejfdZxZS) ViltOutputrBrNcttj|j|j|_tj|j|_ dSr ) r0r1rrr#r6r r?r@rArs r+r1zViltOutput.__init__sJ Yv79KLL z&"<== r*r r cd||}||}||z}|Sr r rs r+rzViltOutput.forwards4 =11  ]33 % 4 r*r)rs@r+r+r+s|>z>d>>>>>> U\RWR^r*r+c*eZdZdZfdZddZxZS) ViltLayerz?This corresponds to the Block class in the timm implementation.czt|j|_d|_t ||_t ||_t||_ tj |j |j |_tj |j |j |_dS)Nrr)r0r1chunk_size_feed_forward seq_len_dimrrr! intermediater+rrrr6rlayernorm_beforelayernorm_afterrs r+r1zViltLayer.__init__s '-'E$&v..,V44 (( " V-?VEZ [ [ [!|F,>FDYZZZr*NFcH||||||}|d}|dd}|||jz}||}||}|||}|f|z}|S)N)rrr)rr4r~r_r5r3r) rCr rrrself_attention_outputsrr layer_outputs r+rzViltLayer.forwards!%  ! !- 0 0  / "0" "  2!4(,)=+;+;.s!#_#_#_!If$5$5#_#_#_r*F) r0r1rBr ModuleListrangenum_hidden_layerslayergradient_checkpointingrs `r+r1zViltEncoder.__init__s`  ]#_#_#_#_uVE]?^?^#_#_#_`` &+###r*NFTc|rdnd}|rdnd}t|jD]j\} } |r||fz}||| nd} |jr&|jr|| j||| |} n| ||| |} | d}|r || dfz}k|r||fz}|st d|||fDSt|||S)Nr)rrc3K|]}||V dSr r)ris r+ z&ViltEncoder.forward..%s(mmq_`_l_l_l_l_lmmr*)last_hidden_stater r!)rr@rAtraining_gradient_checkpointing_func__call__tupler ) rCr rrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputss r+rzViltEncoder.forwardsN#7@BBD$5?bb4(44 P POA|# I$58H$H!.7.CillO* pt} p $ A A )!"#% !! !- ]NO]n o o )!,M  P&9]1=M>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r*N) r"r#r$r%r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modulesr^r)r*r+rSrS-sL L&*#)+>?*****r*rSaH This model is a PyTorch `torch.nn.Module `_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ViltConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViltImageProcessor.__call__`] for details. pixel_mask (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_images, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViltImageProcessor.__call__`] for details. pixel_mask (`torch.LongTensor` of shape `(batch_size, num_images, height, width)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. image_embeds (`torch.FloatTensor` of shape `(batch_size, num_images, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare ViLT Model transformer outputting raw hidden-states without any specific head on top.c eZdZdfd ZdZdZdZeee e e  dde e jd e e jd e e jd e e jd e e jd e e jde e jde e jde ede ede ede edee ee jffdZxZS) ViltModelTcJt|||_t||_t ||_tj|j |j |_ |rt|nd|_ |dSNr)r0r1rBr-rr:encoderrrr6r layernorm ViltPoolerpooler post_init)rCrBadd_pooling_layerrDs r+r1zViltModel.__init__s     (00"6** f&8f>STTT,=Gj(((4  r*c$|jjjSr rr3rrCs r+get_input_embeddingszViltModel.get_input_embeddingss.>>r*c(||jj_dSr rn)rCrs r+set_input_embeddingszViltModel.set_input_embeddingss:?'777r*c|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrgr@rr)rCheads_to_pruner@rs r+ _prune_headszViltModel._prune_headssU +0022 C CLE5 L u % / ; ;E B B B B C Cr* output_typer_NrrrrrrrrrrrJrKrc \| | n |jj} | | n |jj} | | n |jj} ||t d|+||||} n.||dd} nt d| \}}||jn|j}|tj ||f|}||t d||t d| |j dn |j d}||krt d |-tj ||jj |jj f|}| ||jj }||||||||| \}}||| }||||| | | }|d}||}|j||nd}| s||f|d dzSt'|||j|j S)a Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer[z5You have to specify either input_ids or inputs_embedsr^zFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrrJrKr)rE pooler_outputr r!)rBrrJuse_return_dictr%warn_if_padding_and_no_attention_maskrIr_r&rrqru get_head_maskr?rget_extended_attention_maskrgrhrjr r r!)rCrrrrrrrrrrrJrKrtext_batch_sizerr_image_batch_sizeembedding_outputextended_attention_maskencoder_outputssequence_output pooled_outputs r+rzViltModel.forwardsN2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U&1#%.%:!! @T  !"Z/:)FPVWWWN  # (@eff f  !l&:VWW W4@4L<-a00R^RdefRg  . .`aa a  %5t{7Mt{Oe$fouvvvJ&&y$+2OPP +/??       !5,; , , (.150P0PQ_al0m0m,, 2/!5# '  *!,..998< 8O O444UY  J#]3oabb6II I)-')7&1     r*)T NNNNNNNNNNNN)r"r#r$r1rprrrvrVILT_INPUTS_DOCSTRINGrr _CONFIG_FOR_DOCrr& LongTensorr'rboolrrrrrs@r+rdrds      ???@@@CCC+*+@AA+ETcddd156:594815155948.2,0/3&*p p E,-p !!23p !!12 p u01 p U-. p E-.p   12p u01p 'smp $D>p 'tnp d^p  )51B+CC Dp p p edBAp p p p p r*rdc$eZdZfdZdZxZS)ricttj|j|j|_tj|_dSr )r0r1rrr6r Tanh activationrs r+r1zViltPooler.__init__YsC Yv163EFF '))r*cr|dddf}||}||}|Srb)r r)rCr first_token_tensorrs r+rzViltPooler.forward^s@+111a40 #566  66 r*rQrs@r+ririXsG$$$$$ r*rizU ViLT Model with a language modeling head on top as done during pretraining. c eZdZddgZfdZdZdZee de e e  dd e ejd e ejd e ejd e ejd e ejde ejde ejde ejde ejde ede ede edee eejffdZxZS)ViltForMaskedLMzmlm_score.decoder.weightzmlm_score.decoder.biasct|t||_t ||_|dSr )r0r1rdrT ViltMLMHead mlm_scorerkrs r+r1zViltForMaskedLM.__init__psQ    f%% $V,, r*c|jjSr )rdecoderros r+get_output_embeddingsz%ViltForMaskedLM.get_output_embeddingsys ~%%r*c@||j_|j|j_dSr )rrr)rCnew_embeddingss r+set_output_embeddingsz%ViltForMaskedLM.set_output_embeddings|s!/,1r*zbatch_size, sequence_lengthrwNrrrrrrrrlabelsrrJrKrc V| | n |jj} |||||||||| | |  } | dd\}}| |jdn |jd}|ddd|f|dd|df}}||}d}| et }| |j} ||d|jj | d}| s|f| ddz}||f|zn|St||| j | j S)a labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import requests >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrrrrrrrJrKrGrr[rrr r!) rBr{rTrqrr r~r_rxrrr r!)rCrrrrrrrrrrrJrKrrr text_seq_len text_featuresr mlm_logitsmasked_lm_lossloss_fctrs r+rzViltForMaskedLM.forwardsR&1%XXX156:594815155948-1,0/3&*n n E,-n !!23n !!12 n u01 n U-. n E-.n   12n u01n )*n $D>n 'tnn d^n  ~uU%677 8n n n YXhgn n n n n r*rc$eZdZfdZdZxZS)ViltPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSrf) r0r1rrr6r rr$r%r transform_act_fnrrrs r+r1z$ViltPredictionHeadTransform.__init__s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTr*c||}||}||}|Sr )r rrr(s r+rz#ViltPredictionHeadTransform.forwards= =11 --m<< }55 r*rQrs@r+rrsLUUUUUr*rc,eZdZdfd ZdZdZxZS)rNcht||_t||_t j|j|jd|_ t j tj |j|_ | ||j _|j |j _ dS)NFr)r0r1rBr transformrrr6rrr4r&r5rrp)rCrBrprDs r+r1zViltMLMHead.__init__s  4V<<y!3V5FUSSS LV->!?!?@@  "(DL !I r*c(|j|j_dSr )rrros r+ _tie_weightszViltMLMHead._tie_weightss I r*cZ||}||}|Sr )rr)rCrs r+rzViltMLMHead.forwards' NN1   LLOOr*r )r"r#r$r1rrrrs@r+rrs[ & & & & & &&&&r*rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. c eZdZfdZeeeee dde e j de e j de e j de e j de e j d e e j d e e j d e e j d e e j d e e de e de e deeee j ffdZxZS)ViltForQuestionAnsweringc t||j|_t||_t jt j|j|jdzt j |jdzt j t j|jdz|j|_ | dS)NrG) r0r1 num_labelsrdrTr Sequentialrr6rGELU classifierrkrs r+r1z!ViltForQuestionAnswering.__init__"s     +f%% - If(&*>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrrrGr)rBr{rTrzrr~r_rrP binary_cross_entropy_with_logitsrqrr r!)rCrrrrrrrrrrrJrKrrzrrrs r+rz ViltForQuestionAnswering.forward3sb&1%S 'tnS d^S  'u/@)AA BS S S cbBAS S S S S r*rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. c eZdZfdZeeeee dde e j de e j de e j de e j de e j d e e j d e e j d e e j d e e j d e e de e de e deeee j ffdZxZS)ViltForImageAndTextRetrievalct|t||_t j|jd|_|dSr/) r0r1rdrTrrr6 rank_outputrkrs r+r1z%ViltForImageAndTextRetrieval.__init__s[    f%% 9V%7;; r*rwNrrrrrrrrrrrJrKrc B| | n |jj} d} | td|||||||||| | |  }| r|jn|d}||}| s|f|ddz}| | f|zn|St | ||j|jS)a' labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NzTraining is not yet supported.rrrGr) rBr{NotImplementedErrorrTrzrrr r!)rCrrrrrrrrrrrJrKrrrzrrs r+rz$ViltForImageAndTextRetrieval.forwardsZ&1%L 'tnL d^L  'u/@)AA BL L L cbBAL L L L L r*rzq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. c eZdZfdZeeeee dde e j de e j de e j de e j de e j d e e j d e e j d e e j d e e j d e e de e de e deeee j ffdZxZS)"ViltForImagesAndTextClassificationc t||j|_t||_|j}t jt j|j |z|j |zt j |j |zt j t j|j |z|j|_ | dSr )r0r1rrdrT num_imagesrrrr6rrrrk)rCrBrrDs r+r1z+ViltForImagesAndTextClassification.__init__s     +f%% & - If(:5v7IJ7V W W L+j8 9 9 GII If(:5v7H I I    r*rwNrrrrrrrrrrrJrKrc | | n |jj} | | n |jj} | | n |jj} | |jdkr|d}| |jdkr|d}| |jdnd} | | |jdnd} | |jjkrtdg}| rgnd}| rgnd}t| D]}| |||||dd|ddddddfnd||dd|ddddfnd||||dd|ddddfnd|dz| | |  }| r|j n|d}| || r| |j | r| |jtj|d}||}d}| `t%}| |j} ||d|j| d}| s|||f}||f|zn|St/|||| S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import requests >>> from PIL import Image >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image1, image2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrr z\Make sure to match the number of images in the model with the number of images in the input.) rrrrrrrrrrJrKr[rJr)rBrrJr{ndim unsqueezerqrrr>rTrzrr r!r&ryrr r~r_rxrr)rCrrrrrrrrrrrJrKrpooler_outputsr r!rrrzrrrrrs r+rz*ViltForImagesAndTextClassification.forwards^2C1N--TXT_Tq$8$D $+Jj &1%(>'11!44L  # (9Q(>(>'11!44L.:.F\'**D  2>2J+A..PTJ / / /n 2< ,6RR$ z"" 6 6Aii--o 'tno d^o  7u?P9QQ Ro o o srBAo o o o o r*rz ViLT Model with a token classification head on top (a linear layer on top of the final hidden-states of the text tokens) e.g. for Named-Entity-Recognition (NER) tasks. c eZdZfdZeeeee dde e j de e j de e j de e j de e j d e e j d e e j d e e j d e e j d e e de e de e deeee j ffdZxZS)ViltForTokenClassificationc:t||j|_t|d|_t j|j|_t j |j |j|_ | dS)NF)rl) r0r1rrdrTrr?r@rArr6rrkrs r+r1z#ViltForTokenClassification.__init__s~     +f>>> z&"<== )F$68IJJ r*rwNrrrrrrrrrrrJrKrc L| | n |jj} |||||||||| | |  } | d}| |jdn |jd}||}||ddd|f}d}| `t }| |j} || d|j | d}| s|f| ddz}||f|zn|St||| j | j S)z labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Returns: Nrrrr[rGr)rBr{rTrqrArr r~r_rxrrr r!)rCrrrrrrrrrrrJrKrrtext_input_sizerrrrs r+rz"ViltForTokenClassification.forwardsa0&1%= 'tn= d^=  $eE,=&>> ?= = = `_BA= = = = = r*r)Dr%collections.abcrr dataclassesrtypingrrrrr&torch.utils.checkpointrtorch.nnr activationsr modeling_outputsr r rrrrmodeling_utilsr pytorch_utilsrrrutilsrrrrconfiguration_viltr get_loggerr"loggerr_CHECKPOINT_FOR_DOCrModuler-r2r8rrrr!r+r/r:rSVILT_START_DOCSTRINGr4VILT_IMAGES_AND_TEXT_CLASSIFICATION_INPUTS_DOCSTRINGrdrirrrrrrrr)r*r+rs !!!!!!//////////// %%%%%%!!!!!!.----- uttttttttttt******  H % %- @@@@@{@@ @2W!W!W!W!W!RYW!W!W!t66666RY666r")>99999 999zRY$BIF     ry   "        ##### ###L2 2 2 2 2 ")2 2 2 j*****/***8 5n584pdN N N N N #N N N b          C C C C C )C C  C L")""), g g g g g 2g g g T Z Z Z Z Z #6Z Z Z z9  D D D D D )<D D  D N L L L L L !4L L L L L r*