g dZddlmZddlmZmZmZmZddlZddl Zddlm Z ddl m Z ddl mZdd lmZdd lmZmZdd lmZdd lmZdd lmZmZddlmZmZmZmZm Z m!Z!ddl"m#Z#ddl$m%Z%m&Z&erddl'm(Z(e j)e*Z+dZ,eGddeZ-eGddeZ.Gdde j/Z0Gdde j/Z1Gdde1Z2e1e2dZ3Gd d!e j/Z4Gd"d#e j/Z5Gd$d%e j/Z6Gd&d'e j/Z7d(ej8d)e9d*ej8fd+Z:Gd,d-e j/Z;Gd.d/e j/Zd4Z?ed5e?Gd6d7e>Z@d8ZAed9e=Gd:d;e>ZBede>eZCdS)?zPyTorch Idefics3 model.) dataclass)ListOptionalTupleUnionN)nn)CrossEntropyLoss)PreTrainedModel)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)BaseModelOutput ModelOutput)add_start_docstrings%add_start_docstrings_to_model_forwardis_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10loggingreplace_return_docstrings) AutoModel)Idefics3ConfigIdefics3VisionConfig)_flash_attention_forwardrceZdZUdZdZejed<dZe e e ejed<dZ e e ejed<dZ e e ejed<dZ e e ejed<dS)Idefics3BaseModelOutputWithPasta Base class for Idefics3 model's outputs that may also contain a past key/values (to speed up sequential decoding). Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and optionally if `config.is_encoder_decoder=True` 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder Nlast_hidden_statepast_key_values hidden_states attentionsimage_hidden_states)__name__ __module__ __qualname____doc__r!torch FloatTensor__annotations__r"rrr#r$r%j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/idefics3/modeling_idefics3.pyr r 4s<,0u(///AEOXeE%*;$<=>EEE8B%(9":;BBBBBr.r ceZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZee ejed<dS) Idefics3CausalLMOutputWithPasta Base class for Idefics causal language model (or autoregressive) outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder Nlosslogitsr"r#r$r%)r&r'r(r)r2rr*r+r,r3r"rr#rr$r%r-r.r/r1r1[s8)-D(5$ %,,, $FE $$$9=OXd5#456===8B%(9":;BBBBBr.r1cZeZdZdZdeffd Zdejdejdej fdZ xZ S)Idefics3VisionEmbeddingsaP This is a modified version of `siglip.modelign_siglip.SiglipVisionEmbeddings` to enable images of variable resolution. The modifications are adapted from [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://arxiv.org/abs/2307.06304) which allows treating images in their native aspect ratio and without the need to resize them to the same fixed size. In particular, we start from the original pre-trained SigLIP model (which uses images of fixed-size square images) and adapt it by training on images of variable resolutions. configct|j|_|j|_|j|_t j|j|j|j|jd|_ |j|jz|_ |j dz|_ |j |_ t j |j |j|_dS)Nvalid) in_channels out_channels kernel_sizestridepaddingr)super__init__ hidden_size embed_dim image_size patch_sizerConv2d num_channelspatch_embeddingnum_patches_per_side num_patches num_positions Embeddingposition_embeddingselfr6 __class__s r/r?z!Idefics3VisionEmbeddings.__init__s + + +!y+?    %)Ot$F!4a7!-"$,t/A4>"R"Rr. pixel_valuespatch_attention_maskreturnc|j\}}}}||}|ddd}||jz||jz} } t jd|jz dd|jz } t j|| | zfd} t|D]\} }|dddf }|d }t jddd|z }t jddd|z }t j || d}t j || d}|dddf|jz|z}|| | | d  <| |jjj} ||| z}|S) Nrrg?r)size fill_valueg !?T)right)shaperFflatten transposerCr*arangerGfull enumeratesum bucketizeviewcputorKweightdevice)rMrOrP batch_size_max_im_hmax_im_w patch_embeds embeddingsmax_nb_patches_hmax_nb_patches_w boundaries position_ids batch_idx p_attn_mask nb_patches_h nb_patches_wfractional_coords_hfractional_coords_wbucket_coords_hbucket_coords_wpos_idss r/forwardz Idefics3VisionEmbeddings.forwards,8,>) Ax++L99 !))!,,66q!<< -5-H(VZVeJe*\!d&?"?a$JcFcdd z 4DGW4W'Xefggg &/0D&E&E J J "I{&qqq!t,0022L&q>--//L"',q(A $> ? ?#t'>'E'LMM $"9"9,"G"GG r.) r&r'r(r)rr?r*r+ BoolTensorTensorrw __classcell__rNs@r/r5r5sS3SSSSSS&E$5UM]bgbnr.r5c eZdZdZfdZ d dejdeejdeede ejeejffd Z xZ S) Idefics3VisionAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct||_|j|_|j|_|j|jz|_|j|jz|jkr td|jd|jd|jdz|_ |j |_ tj |j|j|_tj |j|j|_tj |j|j|_tj |j|j|_d|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)r>r?r6r@rAnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutdropoutrLineark_projv_projq_projout_proj is_causalrLs r/r?z Idefics3VisionAttention.__init__s  +3$.8 =4> )T^ ; ;'dn''N''' ]D( / i?? i?? i??  $.$.AA r.NFr#attention_maskoutput_attentionsrQc(|\}}}||}||}||} ||||j|jdd}||||j|jdd}| |||j|jdd} |jd} tj ||dd|j z} | ||j|| fkr0td||j|| fd| |L||d|| fkr+td|d|| fd|| |z} tj| d tj |j} tj| |j|j } tj | | } | ||j||jfkr5td ||j||jfd| | dd} | |||j} || } | | fS) z#Input shape: Batch x Time x Channelrrr z$Attention weights should be of size z , but is Nz!Attention mask should be of size rV)dimdtype)ptrainingz `attn_output` should be of size )rSrrrr_rrrYrWr*matmulrrr functionalsoftmaxfloat32rarrr contiguousreshaperAr) rMr#rrrdq_lenre query_states key_states value_states k_v_seq_len attn_weights attn_outputs r/rwzIdefics3VisionAttention.forwards -1133 E1{{=11 [[// {{=11 #((UDNDMZZddefhijj __Z VV``abdeff #((UDNDMZZddefhijj  &r* |L*2F2Fq!2L2LMMPTPZZ     :t~uk"R R R* DNTY[f7g** %%''**   %""$$Q{(KKK }Q{8[}}ftfyfyf{f{}}(.8L},,\r,WWZZ[g[mnn },,\T\TXTa,bb l<>>     *dneT]!S S S)JPUW[Wd3e))$$&&))  "++Aq11<<>> !))*eT^LL mmK00 L((r.)NF) r&r'r(r)r?r*ryrboolrrwrzr{s@r/r}r}sGG226,1 2)2)|2)!.2)$D> 2) u|Xel33 4 2)2)2)2)2)2)2)2)r.r}ceZdZdZfdZ d dejdeejdeejdee d e d e d e ejeejee ejffd Z xZ S)Idefics3VisionFlashAttention2aX Idefics3Vision flash attention module. This module inherits from `Idefics3VisionAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cbtj|i|t |_dSN)r>r?r_flash_attn_uses_top_left_mask)rMargskwargsrNs r/r?z&Idefics3VisionFlashAttention2.__init__s9$)&))) 3V2W2W.W+++r.NFr#rrmpast_key_valuer use_cacherQc d}|\}} } ||} ||} ||} | || |j|j} | || |j|jdd} | || |j|jdd} | jd}||| ||j z }| dd} | dd} |j r|j nd}| j }|tjkrtjrtj}n3t%|jdr |jj}n|jjj }t,d|d| |} | |} | |} t3| | | || ||j|j }||| |j}||}|sd}||fS) NFrrr_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)rruse_top_left_mask) rSrrrr_rrrYrWget_usable_length layer_idxrrrr*ris_autocast_enabledget_autocast_gpu_dtypehasattrr6rrblogger warning_oncerarrrrrArr)rMr#rrmrrrrbszrrerrr kv_seq_len dropout_rate input_dtype target_dtyperrs r/rwz%Idefics3VisionFlashAttention2.forwards"%**,, UA{{=11 [[// {{=11 $((eT^T]SS __S%OOYYZ[]^__ #((eT^T]SS]]^_abcc %b)  % .:::t~VV VJ ))!Q// #--a33 '+}=t||# #( %- ' '(** 8$;== &?@@ 8#{B #{17   $ $$$    (??<88L#|44J'??<88L.      n"A    "))#udnEEPPRR mmK00   LL((r.)NNNFF)r&r'r(r)r?r*ryr LongTensorr rrrwrzr{s@r/rr sXXXXX6:37*."'O)O)|O)!!12O)u/0 O) ! O)  O)O) u|Xel3XeEL>Q5RR SO)O)O)O)O)O)O)O)r.r)eagerflash_attention_2cBeZdZfdZdejdejfdZxZS)Idefics3VisionMLPct||_t|j|_t j|j|j |_ t j|j |j|_ dSr) r>r?r6r hidden_act activation_fnrrr@intermediate_sizefc1fc2rLs r/r?zIdefics3VisionMLP.__init__wsf  #F$569V/1IJJ9V5v7IJJr.r#rQc||}||}||}|Sr)rrr)rMr#s r/rwzIdefics3VisionMLP.forward~s=// **=99 // r.)r&r'r(r?r*ryrwrzr{s@r/rrvscKKKKKU\elr.rc$eZdZfdZdZxZS)Idefics3SimpleMLPct|jj|jdzz}|jj}t j||d|_dS)NrFbias) r>r? vision_configr@ scale_factor text_configrrproj)rMr6 input_size output_sizerNs r/r?zIdefics3SimpleMLP.__init__sX )59La9OP (4 Ij+EBBB r.c,||Sr)r)rMxs r/rwzIdefics3SimpleMLP.forwardsyy||r.)r&r'r(r?rwrzr{s@r/rrsLCCCCC r.rc veZdZdeffd Z d dejdejdeede ej fdZ xZ S) Idefics3EncoderLayerr6cZt|j|_t |j||_tj|j|j |_ t||_ tj|j|j |_ dS)Neps)r>r?r@rA IDEFICS_VISION_ATTENTION_CLASSES_attn_implementation self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rLs r/r?zIdefics3EncoderLayer.__init__s +9&:UVW]^^<F $ u ! $$$$$$$$r.rc eZdZdZdeffd Z d deejdee dee dee d e e e ff d Z xZS) Idefics3Encoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`Idefics3EncoderLayer`]. Args: config: Idefics3Config r6ct|_tjfdt jD|_d|_dS)Nc.g|]}tSr-)r).0rer6s r/ z,Idefics3Encoder.__init__..s"$k$k$ka%9&%A%A$k$k$kr.F) r>r?r6r ModuleListrangenum_hidden_layerslayersgradient_checkpointingrLs `r/r?zIdefics3Encoder.__init__sa  m$k$k$k$k5QWQiKjKj$k$k$kll &+###r.Nrroutput_hidden_states return_dictrQc||n |jj}||n |jj}||n |jj}|rdnd}|rdnd}|}|jD]Z} |r||fz}|jr%|jr|| j|||} n| |||} | d}|r || dfz}[|r||fz}|std|||fDSt|||S)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr-)rrrc3K|]}||V dSrr-rvs r/ z*Idefics3Encoder.forward..s(eeqWXWdWdWdWdWdeer.r!r#r$) r6rruse_return_dictrrr_gradient_checkpointing_func__call__tupler) rM inputs_embedsrrrrencoder_statesall_attentionsr# encoder_layer layer_outputss r/rwzIdefics3Encoder.forwardsx<2C1N--TXT_Tq$8$D $+Jj &1%>N fee]NN$Seeeee e+>Vd    r.NNNN)r&r'r(r)rr?rr*ryrrrrrwrzr{s@r/rrs,~,,,,,,26,0/3&* E E !.E $D> E 'tn E d^ E  uo% &E E E E E E E E r.rr#n_reprQc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rWexpandr)r#rbatchnum_key_value_headsslenrs r/ repeat_kvrs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr.c,eZdZdfd ZdZdZxZS)Idefics3RMSNormư>cttjt j||_||_dS)z> Idefics3RMSNorm is equivalent to T5LayerNorm N)r>r?r Parameterr*onesrbvariance_epsilon)rMr@rrNs r/r?zIdefics3RMSNorm.__init__*sD l5:k#:#:;; #r.c|j}|tj}|ddd}|tj||jzz}|j||zS)NrrVT)keepdim) rrar*rpowmeanrsqrtrrb)rMr#rvariances r/rwzIdefics3RMSNorm.forward2s|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r.cHt|jjd|jS)Nz, eps=)rrbrWrrMs r/ extra_reprzIdefics3RMSNorm.extra_repr9s& )**II$2GIIIr.)r )r&r'r(r?rwrrzr{s@r/r r )sb$$$$$$;;;JJJJJJJr.r c,eZdZfdZddZdZxZS)Idefics3Connectorct|j|_t||_dSr)r>r?rrmodality_projectionrLs r/r?zIdefics3Connector.__init__>s: "/#4V#<#<   r.rc |\}}}t|dzx}}|||||}|||t||z ||z}|dddd}||t||z t||z ||dzz}|dddd}||t||dzz ||dzz}|S)Ng?rrrr )rSintr_permuter)rMrrrseqrAheightwidths r/ pixel_shufflezIdefics3Connector.pixel_shuffleCs ffhhS)S#X& FF3y 1 1 FF3EL$8 9 99|;S T T IIaAq ! ! IIc3u|344c&<:O6P6PR[_kmn_nRo p p IIaAq ! ! IIc3slAo677lTUo9V W Wr.cf|||j}||}|Sr)r#rr)rMr%s r/rwzIdefics3Connector.forwardNs7"001DdFWXX"667JKK""r.)r)r&r'r(r?r#rwrzr{s@r/rr=s[=====    #######r.rai This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Idefics3Config`] or [`Idefics3VisionConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zVThe bare Idefics3 Model outputting raw hidden-states without any specific head on top.c8eZdZeZdZdZddgZdZdZ dZ dZ dZ dS)Idefics3PreTrainedModelmodelTr}Idefics3DecoderLayerr"ct|jdr|jjjn|jjj}t|dr!|jjd|t|tj tj frJ|j jd||j |j j dSdSt|tjrS|j jd||j-|j j|j dSdSdS)Ninitializer_rangeclass_embeddingr)rstd)rr6rr*r+datanormal_ isinstancerrrDrbrzero_rJ padding_idx)rMmoduler,s r/ _init_weightsz%Idefics3PreTrainedModel._init_weightstsAt{$788 ;DK # 5 5(: 6, - - C  " ' / /Sc / B B B fry")4 5 5 ? M  & &CS & 9 9 9{&  &&((((('&  - - ? M  & &CS & 9 9 9!- "6#56<<>>>>> ? ?--r.N) r&r'r(r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_cache_classr3r-r.r/r&r&esY "L&*#24JK"3!N ?????r.r&aS This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Idefics3VisionConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zEThe Idefics3 Vision Transformer Model outputting raw image embedding.c eZdZeZdZdeffd ZdZdZ d de e j de e d e e d e e d e eeff d ZxZS)Idefics3VisionTransformerFr6c t||j}t||_t ||_|j|_tj ||j |_ |j dk|_ dS)Nrr)r>r?r@r5rirencoderrCrrrpost_layernormr_use_flash_attention_2)rMr6rArNs r/r?z"Idefics3VisionTransformer.__init__s|    & 26::&v..  + l9&:OPPP&,&AEX&X###r.c|jSrrirs r/get_input_embeddingsz.Idefics3VisionTransformer.get_input_embeddingss r.c||_dSrrCrMvalues r/set_input_embeddingsz.Idefics3VisionTransformer.set_input_embeddingss r.NrPrrrrQc||n |jj}||n |jj}||n |jj}|d}|p|j}t j||d|z|d|zf}|t j |j }| ||}| |d}t j |sd}n|jst||j}||||||} | d} || } |s| f| ddzSt'| | j| j S) Nrrr rrcrOrPrV)rrrrrrr)r6rrrrSrCr*rrarrcrir_anyrArrr?r@rr#r$) rMrOrPrrrrdrCr#encoder_outputsr!s r/rwz!Idefics3VisionTransformer.forwards2C1N--TXT_Tq$8$D $+Jj &1%RTaTg#h#h ,,'//!5# '  ,A. //0ABB >%'/!""*== =/)7&1    r.r)r&r'r(rr4r:r?rDrHrr*rxrrrrrwrzr{s@r/r=r=s (LNY3YYYYYY   <@,0/3&* 7 7 'u'787 $D> 7 'tn 7 d^ 7  uo% &7 7 7 7 7 7 7 7 r.r=a, Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)): The tensors corresponding to the input images. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details ([]`LlavaProcessor`] uses [`CLIPImageProcessor`] for processing images). pixel_attention_mask (`torch.Tensor` of shape `(batch_size, image_size, image_size)`, *optional*): Mask to avoid performing attention on padding pixel indices. image_hidden_states (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The hidden states of the image encoder after modality projection. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zPIdefics3 model consisting of a SIGLIP vision encoder and Llama3 language decoderceZdZdeffd ZdZdZdZdZde j de e j d e e j fd Z ed e dde j d e e j de e j de ee jde e jde e jde e jd e e jde ede ede ede edeeeffdZxZS) Idefics3Modelr6c4t||jjj|_|jjj|_t|j |_ t||_ tj|j|_t!|j j|j jzdz|jdzz |_|jj|_|jjdk|_|dS)Nrr)r>r?r6r pad_token_idr1 vocab_sizer= _from_configr vision_modelr connectorr from_config text_modelrrBrCr image_seq_lenimage_token_idrrA post_initrLs r/r?zIdefics3Model.__init__8s    ;2?+1<5BB6CWXX*622#/0BCC "-1E1PPUV V[a[npq[q r  #k8&,&8&MQd&d# r.cfdd}|||_|j||_dS)aE Enables the gradients for the input embeddings. This is useful for lora when using gradient checkpointing. c.f. https://github.com/huggingface/peft/issues/1402#issuecomment-1913675032 Override to set output.requires_grad = True for both the decoder's and vision model's embeddings. ctt|dkr|St|dS)Nr)lenlistchildren)r2get_lowest_modules r/r`zCIdefics3Model.enable_input_require_grads..get_lowest_moduleUsW4))**++q00 )(foo.?.?)@)@)CDDDr.c0|ddSNTrequires_grad_r2inputoutputs r/make_inputs_require_gradszKIdefics3Model.enable_input_require_grads..make_inputs_require_grads]  ! !$ ' ' ' ' 'r.N)rDregister_forward_hook_text_require_grads_hookrT_vision_require_grads_hook)rMrhr`s @r/enable_input_require_gradsz(Idefics3Model.enable_input_require_gradsKs E E E E E ( ( ()-(A(A(C(C(Y(YZs(t(t%*;*;D