g TdZddlZddlmZmZmZmZddlZddlm cm Z ddl Zddlm Z ddl mZddlmZddlmZmZdd lmZdd lmZdd lmZmZmZdd lmZdd lmZm Z m!Z!m"Z"ddl#m$Z$m%Z%m&Z&e!j'e(Z)dej*de+de,deej*ej*ffdZ-dej*de+de+dej.dej*f dZ/Gdde j0Z1Gdde j0Z2Gdde j0Z3Gdd e j0Z4Gd!d"e4Z5e4e5d#Z6Gd$d%e j0Z7Gd&d'e j0Z8Gd(d)e j0Z9Gd*d+e j0Z:Gd,d-e:Z;d.ZGd3d4e j0Z?Gd5d6e?Z@e:e;d#ZAe?e@d#ZBGd7d8e j0ZCGd9d:e j0ZDGd;de j0ZFGd?d@eZGdAZHdBZIdCZJdDZKedEeHGdFdGeGZLedHeHGdIdJeGZMedKeHGdLdMeGeZNedNeHGdOdPeGeZOdS)RzPyTorch Mllama model.N)ListOptionalTupleUnion)nn)PreTrainedModel)ACT2FN)Cache StaticCache)GenerationMixin)AttentionMaskConverter)BaseModelOutputBaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONS)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) MllamaConfigMllamaTextConfigMllamaVisionConfigcross_attention_masknum_vision_tokensdtypereturnc |j^}}}||d}|||d}|d}d|z |}||t jt j|j }t j|j }||k d |d}||z}||fS)Nrdimrg?).N) shaperepeat_interleaveview unsqueezeto masked_filltorchboolfinfominanytype_as) rrr batch_sizetext_total_length_inverted_cross_attn_masknegative_inf_valuefull_text_row_masked_out_masks f/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/mllama/modeling_mllama.py_prepare_cross_attention_maskr6,s )=(B%J!A/AABSYZA[[/44ZARTVWW/99!<<!$&: :>>uEE3?? ##EJ//U1C1C1G U++/ !3 388R8@@HHI]^^_hi"99 !> >>aspect_ratio_mask num_patches target_lengthc|j\}}|||dd|}|dd|d}||z }d|dddd| df<d|z }||||zd}||ddzt j|jz}| d}|S)Nrrr") r#r%r'repeatreshape transposer)r+r,r&)r8r9r:rr/ max_num_tilesattention_mask pad_patchess r5$_prepare_aspect_ratio_attention_maskrCHs!2 7J &++J q!LLOOPUVVN#**1aBBN +-K*+N111aaa+&''N$++J 8UWXYYN#n&>&>r2&F&FFUZI[I[I__N#--a00N r7c\eZdZd dedeffd ZdejdejdejfdZxZ S) %MllamaPrecomputedAspectRatioEmbeddingTconfigis_gatedcZt|j|_|j|_|j|_||_t j|jdz|j|jz|_|r-t j tj d|_ dSdSNr) super__init__r@ hidden_sizemax_aspect_ratio_idrGr Embedding embedding Parameterr)zerosgateselfrFrG __class__s r5rKz.MllamaPrecomputedAspectRatioEmbedding.__init__ds #1!-#)#=   d&>&BDDVY]YiDijj  5 U[^^44DIII 5 5r7 hidden_stateaspect_ratio_idsrc||}|d|jd|j}|jr||jz}||z}|S)Nr"r)rOr>r@rLrGrRtanh)rTrVrW embeddingss r5forwardz-MllamaPrecomputedAspectRatioEmbedding.forwardosc^^$455 ''D,>4CSTT = 7#dinn&6&66J#j0 r7)T) __name__ __module__ __qualname__rr*rKr)Tensorr[ __classcell__rUs@r5rErEcs 5 51 5T 5 5 5 5 5 5ELELUZUar7rEcVeZdZdeffd ZdejdejdejfdZxZS)"MllamaPrecomputedPositionEmbeddingrFc&t|j|_|j|_|j|jzdzdz|_|j|_|jdz|_tj tj d|_ tj|j|j}tj |j|z|_tj|jdz|j|jz|jz|_dS)Nr)rJrKr@rM image_size patch_sizer9rLscalerrPr)rQrRrandnrOrNtile_embedding)rTrFposition_embeddingrUs r5rKz+MllamaPrecomputedPositionEmbedding.__init__{s #1#)#= "-1BBqH1L!-'- LQ00 #[)94;KLLdj3E&EFF!l  $q ($*r@)rTrVrWgated_position_embeddingtile_position_embeddingr/gated_tile_position_embeddings r5r[z*MllamaPrecomputedPositionEmbedding.forwards$% (8(8$8DN#J #&>&C&CAq$JZ\`\l&m&mm #'"5"56F"G"G!'* "9"A"A *D,N# # )- (8(8;R(R%#&CC r7) r\r]r^rrKr)r_r[r`ras@r5rcrczsv 1      &ELELUZUar7rccBeZdZfdZdejdejfdZxZS)MllamaVisionMLPct||_t|j|_t j|j|j |_ t j|j |j|_ dSN) rJrKrFr hidden_act activation_fnrLinearrLintermediate_sizefc1fc2rTrFrUs r5rKzMllamaVisionMLP.__init__sf  #F$569V/1IJJ9V5v7IJJr7 hidden_statesrc||}||}||}|Srt)ryrvrz)rTr|s r5r[zMllamaVisionMLP.forwards=// **=99 // r7)r\r]r^rKr)r_r[r`ras@r5rrrrscKKKKKU\elr7rrc leZdZdeffd Z d dejdeejdedejfdZ xZ S) MllamaVisionAttentionrFct|j|_|j|_|j|jz|_tj|j|j|jzd|_ tj|j|j|jzd|_ tj|j|j|jzd|_ tj|j|jz|jd|_ dSNFbias) rJrKrL embed_dimattention_heads num_headshead_dimrrwq_projk_projv_projo_projr{s r5rKzMllamaVisionAttention.__init__s +/*f.DD i0NUZ[[[ i0NUZ[[[ i0NUZ[[[ i >UZ[[[ r7NrVrAoutput_attentionsrc,||}||}||}|j\}}} |j\} } } ||||j|jdd}||| |j|jdd}||| |j|jdd}tj ||ddtj |jz } |$|ddddddd|jdf} | | z} tj | dtj|j} tj | |} | dd} | ||d} || }|sd} || fS)Nrrerr<r"r!r)rrrr#r%rrr?r)matmulmathsqrtr functionalsoftmaxfloat32r'r contiguousr>r)rTrVrArquerykeyvaluer/ q_seq_lenr1 kv_seq_len attn_weights causal_mask attn_outputoutputs r5r[zMllamaVisionAttention.forwards  L))kk,'' L))#(; Iq9:q :y$.$-PPZZ[\^_``hhz:t~t}MMWWXY[\]] :z4>4=QQ[[\]_`aa|E3==A+>+>??$)DMBZBZZ  %(AAAqqq/CIbM/)ABK'+5L},,\r,WWZZ[`[fgg l<77 !++Aq11<<>> !))*iDD [))  L|##r7NN) r\r]r^rrKr)r_rr*r[r`ras@r5rrs \1 \ \ \ \ \ \26"& #$#$l#$!.#$ #$  #$#$#$#$#$#$#$#$r7rc `eZdZ ddejdeejdedejffd ZxZS)MllamaVisionSdpaAttentionNrVrArrc^|r>tdt|||S||}||}||}|j\}}} |j\} } } ||||j |j }||| |j |j }||| |j |j }| dd}| dd}| dd}tj ||||} | dd} | ||d} || } | dfS)NaMllamaModel is using MllamaVisionSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rVrArrre) attn_maskr")logger warning_oncerJr[rrrr#r%rrr?Fscaled_dot_product_attentionrr>r)rTrVrArrrrr/rr1rrrrUs r5r[z!MllamaVisionSdpaAttention.forwards     [   77??)-"3#   L))kk,'' L))#(; Iq9:q :y$.$-PPhhz:t~t}MM :z4>4=QQ1%%mmAq!!1%%4UCR`aaa !++Aq11<<>> !))*iDD [))t|r7r) r\r]r^r)r_rr*r[r`ras@r5rrs{ 26"& ((l(!.( (  ((((((((((r7r)eagersdpacdeZdZd dedeffd Z d dejdeejdefd Z xZ S) MllamaVisionEncoderLayerFrFrGct|j|_|j|_||_|j|_t|j||_ t||_ tj |j|j|_tj |j|j|_|rxtjt%jdt(jzdz |_tjt%jdt(jzdz |_dSdS)Nepsr)rJrKrLrnum_attention_headsrGrxMLLAMA_VISION_ATTENTION_CLASSES_attn_implementation self_attnrrmlpr LayerNormnorm_epsinput_layernormpost_attention_layernormrPr)onesrpi gate_attngate_ffnrSs r5rKz!MllamaVisionEncoderLayer.__init__s !-#)#9   !'!989TUV\]]"6**!|D,<&/RRR(* T5E6?([([([%  F\%*Q--$'*AA*EFFDNLA)@1)DEEDMMM F Fr7NrVrArcv|}||}|||\}}|jr|j|z}||z}|}||}||}|jr|j|z}||z}|f}|r||fz }|S)N)rA)rrrGrrYrrr)rTrVrArresidualroutputss r5r[z MllamaVisionEncoderLayer.forward%s ++L99 %)^^LQ_^%`%`" l = @>..00<?L,.  44\BB xx -- = ?=--//,>L,. /  '   &Gr7)Fr) r\r]r^rr*rKr)r_rr[r`ras@r5rrsFF1FTFFFFFF*26"& l!. r7rceZdZdZddeffd Z ddejdeejd ee d ee d ee d e e e ff d Z xZS)MllamaVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MllamaEncoderLayer`]. Args: config: MllamaConfig FrFct|_tjfdt |D|_d|_|_dS)Nc0g|]}tS)r).0r1rFrGs r5 z0MllamaVisionEncoder.__init__..Os%$k$k$kTU%=fh%O%O$k$k$kr7F)rJrKrFr ModuleListrangelayersgradient_checkpointing)rTrF num_layersrGrUs ` `r5rKzMllamaVisionEncoder.__init__Lsh  m$k$k$k$k$kY^_iYjYj$k$k$kll &+# r7Nr|rAroutput_hidden_states return_dictrc||n |jj}||n |jj}||n |jj}|rdnd}|rdnd}|jD]Z}|r||fz}|jr%|jr||j|||} n||||} |r || dfz}| d}[|r||fz}|std|||fDSt|||S)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nrrrrc3K|]}||V dSrtrrvs r5 z.MllamaVisionEncoder.forward..s(eeqWXWdWdWdWdWdeer7last_hidden_stater| attentions) rFrruse_return_dictrrtraining_gradient_checkpointing_func__call__tupler) rTr|rArrrencoder_statesall_attentions encoder_layer layer_outputss r5r[zMllamaVisionEncoder.forwardSsr<2C1N--TXT_Tq$8$D $+Jj &1%>N fee]NN$Seeeee e+>Vd    r7)rF)NNNN)r\r]r^__doc__rrKr)r_rr*rrrr[r`ras@r5rrCs126,0/3&* D D |D !.D $D> D 'tn D d^ D  uo% &D D D D D D D D r7rc,eZdZdfd ZdZdZxZS)MllamaTextRMSNormư>cttjt j||_||_dS)z@ MllamaTextRMSNorm is equivalent to T5LayerNorm N)rJrKrrPr)rweightvariance_epsilon)rTrLrrUs r5rKzMllamaTextRMSNorm.__init__sD l5:k#:#:;; #r7c|j}|tj}|ddd}|tj||jzz}|j||zS)Nrer"T)keepdim) rr'r)rpowmeanrsqrtrr)rTr| input_dtypevariances r5r[zMllamaTextRMSNorm.forwards|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r7cHt|jjd|jS)Nz, eps=)rrr#rrTs r5 extra_reprzMllamaTextRMSNorm.extra_reprs& )**II$2GIIIr7)r)r\r]r^rKr[rr`ras@r5rrsb$$$$$$;;;JJJJJJJr7rc,eZdZdZ ddeedeeffd Z ddej deej d ee d eej d e d e d eej de ej eej ee ej ffdZxZS)MllamaTextCrossAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNrF layer_idxct||_|jj|_|jj|_|j|_|j|_|j|jz|_||_ |j|jz|_ tj |j|j|jzd|_ tj |j|j|jzd|_tj |j|j|jzd|_tj |j|jz|jd|_t#|j|j|_t#|j|j|_dS)NFrr)rJrKrFrrnum_key_value_headsdropoutrLrrnum_key_value_groupsrrwrrrrr rms_norm_epsq_normk_normrTrFrrUs r5rKz!MllamaTextCrossAttention.__init__sE  8#';#B ~ !-*dn< "$(Nd6N$N!i 0$.4=2PW\]]] i 0$2JT]2Zafggg i 0$2JT]2Zafggg i >@PW\]]] ' 6;NOOO ' 6;NOOO r7Fr|cross_attention_statespast_key_valuerAr use_cachecache_positionrc|\}} } ||} | || |j|jdd} || } |||} ||} | |d|j |jdd} | |d|j |jdd} t| |j } t| |j } | | } |"| | | |jd|i\} } n@|ddkr%|j|j|j|j} } nt#dt%j| | ddt)j|jz }|$|ddddddd| jd f}||z}t.j|dt$j | j}t.j||j|j }t%j|| }|dd}| || d}|!|}|sd}|||fS) #Input shape: Batch x Time x ChannelrreNr"rr^Cross attention layer can't find neither `cross_attn_states` nor cached values for key/values!rr<rpr)"sizerr%rrr?rrrr repeat_kvrrupdater key_cache value_cache ValueErrorr)rrrr#rrrrr'rrrrr>r)rTr|rrrArrrbszq_lenr1 query_states key_states value_statesrrrs r5r[z MllamaTextCrossAttention.forwards&**,, UA{{=11 #((eT^T]SS]]^_abcc {{<00 ! -%;<>L#b$2JDMZZddefhijjJ',,S"d6NPTP]^^hhijlmnnL":t/HIIJ$\43LMMLZ00J),:+@+@ dn?OQ_>`,,( LA ! # #(8*4>:%JJ p |L*2F2Fq!2L2LMMPTPYZ^ZgPhPhh  %(AAAqqq2HJ4DR4H2H)HIK'+5L},,\r,WWZZ[g[mnn },,\T\TXTa,bb l<>> !++Aq11<<>> !))#ub99 kk+..   LL.88r7rNNNFNN)r\r]r^rrrintrKr)r_r r* LongTensorrr[r`ras@r5rrs-GG.2#'PP)*PC=PPPPPP4:>*.15"'599999|99!) 699! 99 !. 99  9999!!1299 u|Xel3XeEL>Q5RR S9999999999999999r7rceZdZdZ d dejdeejdeedeejded ed eej d e ejeejee ejfffd Z xZ S)MllamaTextCrossSdpaAttentiona Mllama attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `MllamaTextCrossAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. NFr|rrrArrrrc |rBtdt|||||||S|\}} } ||} | || |j|j dd} | | } || |} | |} | |d|j |j dd} | |d|j |j dd} |"|| | |jd|i\} } n@|ddkr%|j|j|j|j} } nt%d t'| |j} t'| |j} || } | jjd kr>|<| } | } | } || dkrd nd }t2jj| | | ||jr|jnd |}| dd}||| d}| |}|d|fS)raMllamaModel is using MllamaTextCrossSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)r|rrArrrrrreNr"rrrcudaTFr dropout_p is_causal)!rrrJr[rrr%rrr?rrrrrrr r r rrrdevicetyperr)rrrrrr>r)rTr|rrrArrrr r r1rrrrrrUs r5r[z$MllamaTextCrossSdpaAttention.forward s     [   77??+'=--"3#-# &**,, UA{{=11 #((eT^T]SS]]^_abcc {{<00 ! -%;<>L#b$2JDMZZddefhijjJ',,S"d6NPTP]^^hhijlmnnL),:+@+@ dn?OQ_>`,,( LA ! # #(8*4>:%JJ p z4+DEE  t/HII [[,,    #v - -.2L'2244L#..00J'2244L+2uqyyDDe h)FF   $&*m> !))#ub99 kk+.. D.00r7r) r\r]r^rr)r_rr r*rrr[r`ras@r5rrs:>*.15"'59S1S1|S1!) 6S1! S1 !. S1  S1S1!!12S1 u|Xel3XeEL>Q5RR SS1S1S1S1S1S1S1S1S1S1r7rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Nr"rer )r#r)cat)xx1x2s r5 rotate_halfr#ds] 3"!'"+"" " #B 3 q "" " #B 9rc2YB ' ' ''r7c||}||}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. )r&r#)qkcossin position_ids unsqueeze_dimq_embedk_embeds r5apply_rotary_pos_embr-lsc( -- & &C -- & &C3w;q>>C/0G3w;q>>C/0G G r7r|n_repc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r#expandr>)r|r.batchrslenrs r5rrs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr7c leZdZdedeffd Z d dejdejdejd ed ef d Z xZ S) MllamaTextSelfAttentionrFrct||_|j|_|j|_|j|_|j|_|j|jz|_|j|jz|_ |j |_ ||_ tj |j|j|jzd|_tj |j|j|jzd|_tj |j|j|jzd|_tj |j|jz|jd|_dSr)rJrKrFrrrrLrrr rope_thetarrrwrrrrrs r5rKz MllamaTextSelfAttention.__init__s  3~ !-#)#= *dn< $(Nd6N$N! +"i 0$.4=2PW\]]] i 0$2JT]2Zafggg i 0$2JT]2Zafggg i >@PW\]]] r7FNr|rAposition_embeddingsrrc b|\} } } ||} ||} ||}| | | |j|jdd} | | | |j|jdd} || | |j|jdd}|\}}t| | ||\} } |&|||d}| | ||j |\} }t| |j } t||j }tj| | ddt!j|jz }|$|ddddddd| jdf}||z}t&j|dtj| j}t&j||j|j}tj||}|dd}|| | d}||}|sd}|||fS) Nrrer(r'rrr<r"rr)rrrrr%rrr?rr-rrrrr)rrrr#rrrrr'rrrrr)rTr|rAr7rrrrkwargsr r r1rrrr'r( cache_kwargsrrrs r5r[zMllamaTextSelfAttention.forwards&**,, UA{{=11 [[// {{=11 #((eT^T]SS]]^_abcc __S%1I4=YYccdeghii #((eT5Mt}]]gghiklmm &S#7 jRUWZ#[#[ j  %#&snUUL'5'<'> !++Aq11<<>> !&&sE266 kk+..   LL.88r7FFNN) r\r]r^rrrKr)r_r*r[r`ras@r5r4r4s^/^C^^^^^^,#(3939|39 39#\ 39  39  3939393939393939r7r4c \eZdZ d dejdejdejdedef fd ZxZS) MllamaTextSelfSdpaAttentionFNr|rAr7rrc d|r;tdtjd |||||||d|S|\} } } ||} ||} ||}| | | |j |j  dd} | | | |j |j  dd} || | |j |j  dd}|\}}t| | ||\} } |&|||d}|| ||j|\} }t!| |j} t!||j}|}||ddddddd| jdf}| jjdkr>|<| } | } |}|| dkrdnd }t,jj| | |||jr|jnd | }| dd}|| | d }||}|d|fS)NaMllamaModel is using MllamaTextSelfSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)r|rAr7rrrrrrer9r<rTFrrr"r)rrrJr[rrrrr%rrr?rr-rrrrr#rrrr)rrrrrr)rTr|rAr7rrrrr:r r r1rrrr'r(r;rrrrUs r5r[z#MllamaTextSelfSdpaAttention.forwards     [   #577? +-$7-"3#-     &**,, UA{{=11 [[// {{=11 #((eT^T]SS]]^_abcc __S%1I4=YYccdeghii #((eT5Mt}]]gghiklmm &S#7 jRUWZ#[#[ j  %#&snUUL'5'<'> !&&sE266 kk+.. D.00r7r<)r\r]r^r)r_r*r[r`ras@r5r>r>s#(M1M1|M1 M1#\ M1  M1  M1M1M1M1M1M1M1M1M1M1r7r>c$eZdZfdZdZxZS) MllamaTextMLPct||_|j|_|j|_t j|j|jd|_t j|j|jd|_t j|j|jd|_ t|j |_ dSr) rJrKrFrLrxrrw gate_projup_proj down_projr ruact_fnr{s r5rKzMllamaTextMLP.__init__4s  !-!'!94#3T5KRWXXXy!143IPUVVV 4#94;KRWXXXV./ r7c||||||zSrt)rErFrCrD)rTr s r5r[zMllamaTextMLP.forward?s;~~dkk$..*;*;<7;15UY37*.,1$)59KOCC|C!) 6C'u|4 C !. C (0elEL6P0Q'R Cu/0C!C$D>CD>C!!12C&eEL%,,F&GHC u (51BEDU1U+V"WW XCCCCCCCCr7rIc>eZdZdZdededdffd Z ddejd ejd ejd ejd e ejejfd e ej de e de e de e de ej de ejde ejfdZxZS) MllamaCrossAttentionDecoderLayerzLCross-attention transformer block with tanh-gated attention and feedforward.rFrrNct||_t|j|||_t |j|j|_ tj tj d|_t||_t |j|j|_tj tj d|_dS)N)rrr)rJrKr#MLLAMA_TEXT_CROSS_ATTENTION_CLASSESr cross_attnrrLrrr)rrPrQcross_attn_attn_gaterArrcross_attn_mlp_gaters r5rKz)MllamaCrossAttentionDecoderLayer.__init__s "=f>YZ[amvwww01CI\]]]$)H$6$6u{1~~$F$F! (((9&:LRXRe(f(f(f%#(8#5#5ek!nn#E#E   r7Fr|rrrAr4r)rrrrr7c |} ||}||||||| \}} }| |j|zz}|} ||}||}||dddf|z}| |j|zz}|f}|r|| fz }| r||fz }|S)N)r|rArrrrr)rrTrUrYrrrV)rTr|rrrAr4r)rrrrr7rrrs r5r[z(MllamaCrossAttentionDecoderLayer.forwards!,,];; 6:oo'/#9)/) 7F7 7 3 |^!4#<#A#A#C#Cm#SS  55mDD // ( 49!!!Q$?-OM 4#;#@#@#B#B]#RR  "  '   &G  ) ( (Gr7)NNFFNN)r\r]r^rrrrKr)r_rrrr r*r[r`ras@r5rQrQsEVV F/ FC FD F F F F F F&48*.,1$)596:**|*!& *$l *  * (-U\5<-G'H *u/0*!*$D>*D>*!!12*&el3* u| ********r7rQc\eZdZddeffd ZdZejdZxZ S)MllamaRotaryEmbeddingNrFcft|jd|_|j|_|j|_||_t|j|_ | |j|\}|_ | d|d|j |_ dS)N rope_typeinv_freqF persistent)rJrK rope_scalingr[max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrFr rope_init_fnattention_scalingregister_bufferr\original_inv_freq)rTrFrr\rUs r5rKzMllamaRotaryEmbedding.__init__s ,[9"("@$*$B! /?+/+<+>@@GG HZ[\H]_acdee ,QQQaaaZ 8 > > @ @hm %/ S%A%AekUZFZFZkk`e ^ U C C C  &,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))C''))C                D**D**vvAGv$$cff17f&;&;;;s A>EEErt) r\r]r^rrKrkr)no_gradr[r`ras@r5rYrYs / // / / / / / /@@@$U]__<<_<<<<>>>>.-  - - ; K  Sc  2 2 2 2 2  1 2 2 ; GOOF27SO A A A A A  B C C ; GOOF,1sO ; ; ; ; ;  8 9 9 ;fo ; GOOF,1sO ; ; ; GOOFO0cO : : : : : ; ; ; ;r7N) r\r]r^r config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_cache_class_supports_static_cache_supports_sdpa_supports_quantized_cacherrr7r5r}r} s_L&*# !"N $;;;;;r7r}aK This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MllamaConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, max_num_images, max_num_tiles, channels, image_size, image_size)): The tensors corresponding to the input images. Pixel values can be obtained using [`AutoImageProcessor`]. See [`MllamaImageProcessor.__call__`] for details ([]`MllamaProcessor`] uses [`MllamaImageProcessor`] for processing images). aspect_ratio_mask (`torch.Tensor` of shape `(batch_size, max_num_images, max_num_tiles)`, *optional*): Mask to avoid performing attention on padding tiles. Mask values selected in `[0, 1]`: - 1 for tiles that are **not masked**, - 0 for tiles that are **masked**. aspect_ratio_ids (`torch.Tensor` of shape `(batch_size, max_num_images)`, *optional*): Aspect ratio ids used to select the appropriate precomputed tile embeddings based on the aspect ratio of each input image. These ids correspond to indices in the model's list of supported aspect ratios, offset by 1. For example, if the model supports aspect ratios [[1, 1], [1, 2], [2, 1]]: - An image with aspect ratio [1, 1] would have ID 1 - An image with aspect ratio [1, 2] would have ID 2 - An image with aspect ratio [2, 1] would have ID 3 The id 0 is reserved for padding (i.e., no image). If an image has aspect ratio [1, 2], that means it was split into 2 tiles horizontally, and its `aspect_ratio_id` would be 2. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*): Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Two formats are allowed: - a [`~cache_utils.Cache`] instance, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache); - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy cache format. The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned. If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, max_num_images, max_num_tiles, channels, image_size, image_size)): The tensors corresponding to the input images. Pixel values can be obtained using [`AutoImageProcessor`]. See [`MllamaImageProcessor.__call__`] for details ([]`MllamaProcessor`] uses [`MllamaImageProcessor`] for processing images). aspect_ratio_mask (`torch.Tensor` of shape `(batch_size, max_num_images, max_num_tiles)`, *optional*): Mask to avoid performing attention on padding tiles. Mask values selected in `[0, 1]`: - 1 for tiles that are **not masked**, - 0 for tiles that are **masked**. aspect_ratio_ids (`torch.Tensor` of shape `(batch_size, max_num_images)`, *optional*): Aspect ratio ids used to select the appropriate precomputed tile embeddings based on the aspect ratio of each input image. These ids correspond to indices in the model's list of supported aspect ratios, offset by 1. For example, if the model supports aspect ratios [[1, 1], [1, 2], [2, 1]]: - An image with aspect ratio [1, 1] would have ID 1 - An image with aspect ratio [1, 2] would have ID 2 - An image with aspect ratio [2, 1] would have ID 3 The id 0 is reserved for padding (i.e., no image). If an image has aspect ratio [1, 2], that means it was split into 2 tiles horizontally, and its `aspect_ratio_id` would be 2. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`Cache` or `tuple(tuple(torch.FloatTensor))`, *optional*): Pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used to speed up sequential decoding. This typically consists in the `past_key_values` returned by the model at a previous stage of decoding, when `use_cache=True` or `config.use_cache=True`. Two formats are allowed: - a [`~cache_utils.Cache`] instance, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache); - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`). This is also known as the legacy cache format. The model will output the same cache format that is fed as input. If no `past_key_values` are passed, the legacy cache format will be returned. If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. z>The Mllama Vision Model which consists of two vision encoders.c6eZdZeZdZdeffd ZdZdej dej fdZ e e e ed  dd ej d ej d ej deedeedeedeeeej dfffdZxZS)r vision_modelrFct||j|_|j|_|j|_|j|_|j|_|j|_|j|jzdzdz|_|jdz|_ tj |j|j|j|jdd|_ tj |j tj|jz|_t#||_t'|d|_t'|d|_tj|j|_tj|j|_t3||jd|_t3||jd|_|dS) NrerrfvalidF) in_channels out_channels kernel_sizestridepaddingrT)rG)rJrKrgrhr@rL num_channelsintermediate_layers_indicesr9rirrpatch_embeddingrPr)rjrrcgated_positional_embeddingrEpre_tile_positional_embeddingpost_tile_positional_embeddingr layernorm_prelayernorm_postrnum_hidden_layers transformernum_global_layersglobal_transformer post_initr{s r5rKzMllamaVisionModel.__init__'s     + +#1!-"/+1+M( Ot>1DqH'- !y+)?     "|DJTEU9V9V,VWW*LV*T*T'-RSYdh-i-i-i*.STZei.j.j.j+ \$*:;; l4+;<</vv7OZ_```"5ff>Vae"f"f"f r7c|jS)zg This function is used to fetch the first embedding layer to activate grads on inputs. )rrs r5get_input_embeddingsz&MllamaVisionModel.get_input_embeddingsLs ##r7rVrc|j\}}}|j|d|}tj||gd}|S)Nrr )r#rr0r)r)rTrVr/r1rLrs r5apply_class_embeddingz'MllamaVisionModel.apply_class_embeddingRsJ%1%7" A{.55j![QQy/>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, MllamaVisionModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaVisionModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 1, 4, 1025, 7680]) ``` Nr"rerr<rconstant)moder)r8r9r:rT)rArrr .c3K|]}||V dSrtrrs r5rz,MllamaVisionModel.forward..s(__qQRQ^Q^Q^Q^Q^__r7r)rFrrrr#r>rr'rrflattenr?rrrrrpadrCr9r%rrrrr)stackrrrr)rTrrWr8rrrr/num_concurrent_media num_tilesrheightwidth patch_embedsrVr1r9r!num_padding_patchesr slice_indexrAr global_outputall_intermediate_hidden_statesintermediate_hidden_statesr| global_attnrs r5r[zMllamaVisionModel.forwardXsH2C1N--TXT_Tq$8$D $+Jj &1%> +0;#++J9M,MyZ\^abb 99,HXYY $++J9M,MPY,Y[fhkll 11,?? q $++J9M,MyZegjkk 66|EUVV )),77  !L$6r$:Q$>?1Da/0u\71MMM .AA.E.E***4 +22:@T3TVXYY=,(&,Q/*    $((6J)JBPSTT !! )!%/ "   ay **<88 $++ - -y+H[:[]`  :: Y 'tn Y d^Y  elC&7 88 9Y Y Y _^KJY Y Y Y Y r7rzYThe Mllama Text Model which consists of transformer with self and cross attention layers.c!eZdZeZdZdeffd ZdZdZe e e e d d d e ejd e ejd e ejd e ejd e ejde eejejfde eeeejfde ejde ede ede ede ede ejdeee ffdZd ejdejdejdedef dZed ejdededejdejdejdefdZxZS)!MllamaTextModelzlanguage_model.modelrFct||j|_|j|_t j|jdz|j|j|_|j |_ g}t|j D]R}||j vr$| t||/| t||St j||_t#|j|j|_t)||_d|_|dS)Nrr)rFF)rJrK pad_token_idr vocab_sizerrNrL embed_tokenscross_attention_layersrrappendrQrIrrrrnormrY rotary_embrr)rTrFrrrUs r5rKzMllamaTextModel.__init__s*    !. +L):Q)>@RTXTdee&,&C#v788 R RID777 >vyQQRRRR =fiPPQQQQmF++ %f&8f>QRRR /v>>>&+# r7c|jSrtrrs r5rz$MllamaTextModel.get_input_embeddingss   r7c||_dSrtrrTrs r5set_input_embeddingsz$MllamaTextModel.set_input_embeddingss!r7rrN input_idsrAr)rrr4past_key_values inputs_embedsrrrrrrc| | n |jj} | | n |jj} | | n |jj} | | n |jj} |du|duz rt d|jr%|jr| rt dd} || |}|}| B|| nd}tj |||jdz|j} || d}|||| || }|||}| rdnd}| rdnd}d}t'|jD]\}}| r||fz }||jv}|dup|duo| |dk}|r||r>|jr-|jr&||j|||||||| | | | }n||||||||| | | | }|d}| r || rd nd}| r ||dfz }||}| r||fz }| r|nd}| st3d ||||fDSt5|||| S) aN Returns: Example: ```python >>> from transformers import AutoProcessor, MllamaTextModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaTextModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> text = "<|image|>If I had to write a haiku for this one" >>> inputs = processor(text=text, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 13, 4096]) ``` N:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Frrrnr) rrrAr4r)rrrrr7rec3K|]}||V dSrtrrs r5rz*MllamaTextModel.forward..s(ttqfgfsfsfsfsfsttr7)rrr|r)rFrrrrr rrrrrget_seq_lengthr)aranger#rr&_update_causal_maskr enumeraterrrrrrr)rTrrAr)rrr4rrrrrrrr|past_seen_tokensrr7all_hidden_statesall_self_attnsnext_decoder_cacheidx decoder_layeris_cross_attention_layeris_cross_attention_cache_emptyr next_caches r5r[zMllamaTextModel.forwardsP2C1N--TXT_Tq$8$D $+Jj "+!6IIDK?L]  #oom\JJ#7@BBD0:d!"+DK"8"83 63 6 C# 6!m%55! (+d.I'I $-<-D.t+X0N0Ns0S0SWX0X +( ,B,JOm,J* t}  $ A A!*!*(1 #%"' ! ! !. !+A)=#.2O!-#2&7'#1(; ! ! ! *!,M R%28I3P11q%Q"  6=#3"55 -00   2 -!1 1 +4>''$  utt]J@QSa$bttttt t&+&+%     r7 input_tensorc |jjdkr |d|vr|SdS||nd}t|t}|jjdkr#|s!|st j||||jrdS|j|j } }|j d} |r| } n/t|tj r |j dn|| zdz} ||| | || ||j d} |jjdkrB|@|j jd kr0|s.tj|j} t j| | } | S) Nflash_attention_2rrr)rpast_key_values_length is_trainingrr")sequence_lengthr:rrrr/r)rFrrrtr r_ignore_causal_mask_sdparrrr#get_max_cache_shaper)r_5_prepare_4d_causal_attention_mask_with_cache_positionrr+r,_unmask_unattended)rTrArrrrrusing_static_cacherrrr:r min_dtypes r5rz#MllamaTextModel._update_causal_masks ; +/B B B)c^.C.C%%4 @O?Z?99;;;`a'EE ; +v 5 5>P 5Yj 5%>*'7 M   t$*L,?v&,Q/  +??AAMMnel;;<$R((%7!; PP +')#)!,Q   K , 6 6*%*f44%5  E**.I0CKQZ[[Kr7rr:rrr/c ||dkr|}n+tj|j} tj||f| ||}|dkrtj|d}|tj|||ddkz}|ddddddf|ddd}|| }|j d} |ddddddd| f|ddddddfz} | dk} |ddddddd| f | | |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. device (`torch.device`): The device to plcae the 4D attention mask on. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr) fill_valuerrr)diagonalrnr"r) r!r)r+r,fulltriurr>r0cloner#r() rArr:rrrr/r:rr mask_length padding_masks r5rzEMllamaTextModel._prepare_4d_causal_attention_mask_with_cache_positionsD  %.*<*<*>*>!*C*C(KK E**.I* -0Ye\bK!###jqAAA 5< fEEEH^H^_acdHeHee eK%dD!!!QQQ&67>>z1bRTUUK))//11 ,226 *111aaaL[L+@ANSTSTSTVZ\`bcbcbcScDdd +q0 5@AAAqqq,;,AV5W5c5c )66 AAAqqq!!!\k\12r7) NNNNNNNNNNNNN)r\r]r^rrrrKrrrMLLAMA_TEXT_INPUTS_DOCSTRINGrrrr)rr_rOrrr rr*r[r staticmethodrrrrr`ras@r5rrs $L./(!!!"""+*+GHH+BQcddd151537>B7;UYKO59$(,0/3&*59S S E,-S !.S u/0 S !)): ; S 'u|4 S (0elEL6P0Q'RS "%tE4E/F(F"GHS   12S D>S $D>S 'tnS d^S !!12S  u-- .S S S edIHS l? ?l? ?  ?  ????B5 555{ 5  5  5555\55555r7rz;The Mllama Text Model with a language modeling head on top.c%FeZdZeZdZdgZfdZdZdZ dZ dZ dZ d Z eeeed  ddejdeejdeejdeejdeejdeeejejfdeeeeejfdeejdeejdeedeedeedeedeejdedeeeff dZxZS) MllamaForCausalLMr~zlm_head.weightct|||_|jj|_t |j|j|_tj |jj |jd|_ | dS)N)attn_implementationFr)rJrKr text_configrr _from_configrr~rrwrLlm_headrr{s r5rKzMllamaForCausalLM.__init__4s //11222!1133*5$11$2BX^Xs1tt y!1!=tUZ[[[  r7c|jjSrtr~rrs r5rz&MllamaForCausalLM.get_input_embeddings=s z&&r7c||j_dSrtrrs r5rz&MllamaForCausalLM.set_input_embeddings@s"' r7c|jSrtrrs r5get_output_embeddingsz'MllamaForCausalLM.get_output_embeddingsCs |r7c||_dSrtr rTnew_embeddingss r5set_output_embeddingsz'MllamaForCausalLM.set_output_embeddingsFs % r7c||_dSrtr~rTdecoders r5 set_decoderzMllamaForCausalLM.set_decoderIs  r7c|jSrtrrs r5 get_decoderzMllamaForCausalLM.get_decoderLs zr7rrNrrrAr)rrr4rrlabelsrrrrrnum_logits_to_keeprc | | n |jj} | | n |jj} | | n |jj} |||||||||| | | | | }|d}||dd| dddf}d}| |j|| |jfi|}| s|f|ddz}||f|zn|St|||j |j |j S)a Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. num_logits_to_keep (`int`, *optional*): Calculate logits for the last `num_logits_to_keep` tokens. If `0`, calculate logits for all `input_ids` (special case). Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences or large vocabulary size. Returns: Example: ```python >>> from transformers import AutoTokenizer, MllamaForCausalLM >>> model = MllamaForCausalLM.from_pretrained("Llama-3.2-11B-Vision") >>> tokenizer = AutoTokenizer.from_pretrained("Llama-3.2-11B-Vision") >>> prompt = "If I had to write a haiku, it would be:" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=40, do_sample=True, temperature=0.6) >>> result = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] >>> print(result) If I had to write a haiku, it would be: "Snowflakes gently fall" - simple, yet peaceful. I love the idea of snowflakes gently falling, each one ``` N) rrrAr)rr4rrrrrrrrr)losslogitsrr|r) rFrrrr~rrs loss_functionrrrr|r)rTrrAr)rrr4rrrrrrrrr loss_kwargsrr|rrrs r5r[zMllamaForCausalLM.forwardOs`l2C1N--TXT_Tq$8$D $+Jj &1%\ $D>\ 'tn\ d^\ !!12\  !\ $ u,, -%\ \ \ dcDC\ \ \ \ \ r7rzIThe Mllama model which consists of a vision encoder and a language model.c)eZdZdZdeffd ZdZdZdZdZ dZ d Z d Z e eeed  d$deejdeejdeejdeejdeejdeejdeejdeejdeeejdeejdeejdeedeedeedeedeejded eeeff$d!Z d%d"Zfd#ZxZS)&MllamaForConditionalGenerationFrFc(t||jj|_|jj|_|jj|_|jj|_|jj |jj nd|_ t |j|_ t |j|_tj|jj|jjd|_|dS)Nr"Tr)rJrKrrrL vision_configr@vision_output_dimrFrrrrrlanguage_modelrrwmulti_modal_projectorrr{s r5rKz'MllamaForConditionalGeneration.__init__s     ,7!-9#1?!'!5!G8< 8P8\DK44bd-::6;OPP/<>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, MllamaForConditionalGeneration >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaForConditionalGeneration.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> prompt = "<|image|>If I had to write a haiku for this one" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=prompt, images=image, return_tensors="pt") >>> # Generate >>> output = model.generate(**inputs, max_new_tokens=15) >>> prompt_len = inputs.input_ids.shape[-1] >>> generated_ids = output[:, prompt_len:] >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False) >>> print(generated_text) [', it would be:.\\nA stop sign in Chinatown.\\n'] ``` NrzdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either onezM`pixel_values` and `cross_attention_states` cannot be provided simultaneouslyzA`aspect_ratio_ids` must be provided if `pixel_values` is provided)rrWr8rrrrr"r<)rr)rrAr)rrr4rrrrrrrrr)rFrrrr rr%r>r#rLr6r9rr$)rTrrr8rWrArrr)rrrrrrrrrvision_outputsr4rs r5r[z&MllamaForConditionalGeneration.forwards @2C1N--TXT_Tq$8$D $+Jj &1%(Jlmm m  #' !deee!..)!1"3%9"3' /N&4A%6 "%)%?%?@V%W%W%_%_*04d6F&& " +B_$"&"3"?jCCC ? "?"? -1 )  +0J#7111n8L#M ,I!!!QQQP^J^,_ )%%)%#9!5*G+'!5/#)1&  $r7c | E||dd| jd df}n(|jd| jdkr |dd| f}|||ddz }||dkd| r:|dd|jd df}|t j}|| ddkr|dd}n#|t jdd}| | |d<||| | | ||d| ddkr||d <||d <||d <|S) Nrrr") memory_format)rr)rrr)r)rrrrArrrWr8)r#longcumsum masked_fill_rr)contiguous_formatr)rTrrrAr)rrWr8rrrrrr: model_inputss r5prepare_inputs_for_generationzq*A)A)C)C&CD #~';A'>>>%aaa&78   %,*>)..0077;;a?L  % %n&91 = = = Y+AAA 0B/B/D/D,DE  ,11@W1XX   $):a)?)?-:NNLL*3uG^)_)_rvwwL  )1CL- . ,"0#2&"0(<     !  ! !+7L (/?L+ ,0AL, -r7c |dd}tjd|||d|}|(tj||dddddfgd|d<|S)Nr)r model_kwargsis_encoder_decoderr".rr r)getrJ#_update_model_kwargs_for_generationr)r)rTrr:r;r:cross_attention_mask_prevrUs r5r=zBMllamaForConditionalGeneration._update_model_kwargs_for_generations$0$4$45KT$R$R!BuwwB %1      % 0389*,Eaaack,RSYZ444L/ 0r7)NNNNNNNNNNNNNNNNr) NNNNNNNNNFNN)r\r]r^rrrKrrr rrrr.rrrrrr)rrOr_rr*rrrr[r8r=r`ras@r5r r s !&|":::888;;;BBB111111111+*+BCC+AP^___15484837157;9=37=A59-1$(,0/3&*59"#%E,-u01$EL1  #5<0  !. 'u|4!) 6u/0"$u'8"9:  12)*D>$D>'tn d^!"!!12#$ %& u,, -'`_DCF!BBBBHr7r rI)Prrtypingrrrrr)torch.nn.functionalrrrtorch.utils.checkpointr activationsr cache_utilsr r generationr modeling_attn_mask_utilsrmodeling_outputsrrrmodeling_rope_utilsrutilsrrrrconfiguration_mllamarrr get_loggerr\rr_rrur6rrCModulerErcrrrrrrrrrrr#r-rr4r>rSrKrArIrQrYr}MLLAMA_START_DOCSTRINGrrrrrrr rr7r5rNsY //////////// !!!!!!--------))))))>>>>>>``````````666666 UTTTTTTTTT  H % %?,?? ? 5< %& ????8| ;   \ 6BI.""""""""L     bi   0$0$0$0$0$BI0$0$0$f***** 5***Z-BKd"e"e.....ry...bT T T T T ")T T T pJJJJJ JJJ(S9S9S9S9S9ryS9S9S9l[1[1[1[1[1#;[1[1[1~(((8 UU\ U# U%, U U U UE9E9E9E9E9biE9E9E9PO1O1O1O1O1"9O1O1O1d1IRn&o&o#*AKf g g P P P P PBI P P P"PPPPPbiPPPf99999ux999x4<4<4<4<4