g9: dZddlZddlmZmZmZmZmZmZddl Z ddl m cm Z ddlZ ddl m Z ddlmZddlmZmZddlmZdd lmZdd lmZmZmZdd lmZdd lmZm Z m!Z!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)e%rddl*m+Z+e'rddl,m-Z-m.Z.ddl/m0Z0nd\Z0Z.Z-e$r ddl1m2Z2m3Z3nd\Z3Z2e4e0e.e2e3e-fZ5e!j6e7Z8dZ9 dBdee j:ee j:dfdee;dee j:dee j:e;ffdZ<Gdde j=Z>de j:d e;de j:fd!Z?Gd"d#eZ@Gd$d%e j=ZAGd&d'eAZBGd(d)eAZCeAeBeCd*ZDGd+d,e j=ZEGd-d.e j=ZFGd/d0e j=ZGGd1d2e j=ZHGd3d4e j=ZId5ZJed6eJGd7d8eZKd9ZLeHeId:ZMed6eJGd;deKeZOed?eJGd@dAeKZPdS)CzPyTorch Jamba model.N)AnyDictListOptionalTupleUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)MoeCausalLMOutputWithPastMoeModelOutputWithPast SequenceClassifierOutputWithPast)PreTrainedModel)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)is_causal_conv1d_availableis_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10is_mamba_ssm_available) JambaConfig)_flash_attention_forward)mamba_inner_fnselective_scan_fn)selective_state_update)NNN)causal_conv1d_fncausal_conv1d_updateNNr router_logits num_expertsattention_maskreturnc|t|tsdSt|tr/|djtjfd|Dd}tjj|d}tj||d\}}tjj ||}|@tj | d} tj |d} n.|j \} } |j d| | zz} |dddddddf | | | ||fd||}tj| |zdtj|dz } |ddddddf | | | |fd|}tj||zdtj|dz } tj| | dz}||zS)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://arxiv.org/abs/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: router_logits: Logits from the `router`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrc:g|]}|S)to).0 layer_routercompute_devices d/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/jamba/modeling_jamba.py z,load_balancing_loss_func..vs% O O O\__^ , , O O Odim) isinstancetupledevicetorchcatr functionalsoftmaxtopkone_hotmeanfloatshapeexpandreshaper-sum unsqueeze)r&r'top_kr(concatenated_router_logitsrouting_weights_selected_experts expert_masktokens_per_expertrouter_prob_per_expert batch_sizesequence_lengthnum_hidden_layersexpert_attention_mask router_per_expert_attention_mask overall_lossr0s @r1load_balancing_loss_funcrUSs:J}e$D$Dq-'' &q)0%*Y O O O O O O OUV& & & "h)112LRT1UUO*_eDDDA(%--.> LLK!J{'8'8':':BBB"'O!C!C!C&4&:# O6cttjt j||_||_dS)z; JambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parameterr:onesweightvariance_epsilon)self hidden_sizeeps __class__s r1r[zJambaRMSNorm.__init__sD l5:k#:#:;; #r3c|j}|tj}|ddd}|tj||jzz}|j||zS)Nr%r6T)keepdim) dtyper-r:float32powr@rsqrtr_r^)r` hidden_states input_dtypevariances r1forwardzJambaRMSNorm.forwards|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r3cHt|jjd|jS)Nz, eps=)r8r^rBr_r`s r1 extra_reprzJambaRMSNorm.extra_reprs& )**II$2GIIIr3)rX)__name__ __module__ __qualname__r[rmrp __classcell__rcs@r1rWrWsb$$$$$$;;;JJJJJJJr3rWrjn_repc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rBrCrD)rjrvbatchnum_key_value_headsslenhead_dims r1 repeat_kvr|s 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr3c reZdZdZejdffd Z ddejdejdede e e e fde ejejff d Zd ejfd Zdde edefd Zde e eje ejffdZedde e e ejddfdZxZS) HybridMambaAttentionDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. Nc Ft||_|j|_d|_|j|jz}|j}|j}g|_ g|_ g|_ t|j D]}|j|dkrQ|xj tj|||gz c_ |xj tj|||gz c_ d|xj tjggzgz c_ |xj tjggzgz c_ |j |шfdt|j D|_fdt|j D|_dS)NFmambar9rfr9cDg|]}tjggzSrr:tensorr.rJrOr9s r1r2z=HybridMambaAttentionDynamicCache.__init__..s/rrrQ%,tj'8HHHrrrr3cDg|]}tjggzSrrrs r1r2z=HybridMambaAttentionDynamicCache.__init__..s/tttqEL" ):6JJJtttr3)rZr[rflayers_block_typehas_previous_state mamba_expandra mamba_d_state mamba_d_conv conv_states ssm_statestransformer_layersrangerQr:zerosrappend key_cache value_cache) r`configrOrfr9intermediate_sizessm_state_sizeconv_kernel_sizeircs ` ` r1r[z)HybridMambaAttentionDynamicCache.__init__s  !'!9"'"/&2DD-!."$v/00 2 2A%a(G33  K ,=?OX^fklll%  K ,=~V\dijjj$  U\2$2CF%S%S%S$TT  EL" 1B6$R$R$R#SS'..q1111rrrrrRWX^XpRqRqrrrtttttTYZ`ZrTsTstttr3 key_states value_states layer_idx cache_kwargsr)cD|j|jddkr||j|<||j|<nVtj|j||gd|j|<tj|j||gd|j|<|j||j|fS)Nr6rr%r4)rrBrr:r;)r`rrrrs r1updatez'HybridMambaAttentionDynamicCache.updates >) $ *2 .! 3 3(2DN9 %*6D Y ' '(- 4>)3Lj2Y_`(a(a(aDN9 %*/)T5Ei5PR^4_ef*g*g*gD Y '~i($*:9*EEEr3beam_idxctt|jD];}|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<=dS)zDReorders the cache for beam search, given the selected beam indices.rN) rlenrr9 index_selectr-rrr)r`rrr9s r1 reorder_cachez.HybridMambaAttentionDynamicCache.reorder_cachesEs4>2233 i iI^I.5F(,y(A(N(NqRZR]R]^dReRe(f(fDN9 %%i07F*.*:9*E*R*RSTV^VaVabhViVi*j*jD Y '%i07F*.*:9*E*R*RSTV^VaVabhViVi*j*jD Y '_Y/6F)-)C)P)PQRT\T_T_`fTgTg)h)hDOI & & i ir3rc||jvr |jdn|}t|j|krdS|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.r)rrrrB)r`rs r1get_seq_lengthz/HybridMambaAttentionDynamicCache.get_seq_lengthsS3<4CZ2Z2ZD+A..`i t~  ) + +1~i(.r22r3c tdNzIHybridMambaAttentionDynamicCache does not have a legacy cache equivalent.NotImplementedErrorros r1to_legacy_cachez0HybridMambaAttentionDynamicCache.to_legacy_caches!"mnnnr3past_key_valuesr c tdrr)clsrs r1from_legacy_cachez2HybridMambaAttentionDynamicCache.from_legacy_caches!"mnnnr3N)r)rqrrrs__doc__r:float16r[Tensorintrrstrrrr LongTensorrrr classmethod FloatTensorrrtrus@r1r~r~s  27tuuuuuu@26 FFLFlF F tCH~. F u|U\) * FFFF" ie&6 i i i i33 3c3333ouU\':E%,@PW\]]] r3Frjr( position_idspast_key_valueoutput_attentions use_cachecache_positionr)c|\}} } ||} ||} ||} | || |j|jdd} | || |j|jdd} | || |j|jdd} || | | |j \} } t| |j } t| |j } tj| | ddtj|jz }|$|ddddddd| jdf}||z}t$j|dtj| j}t$j||j|j}tj|| }|||j| |jfkr5t7d||j| |jfd ||dd}||| |j}||}|sd}|||fS) Nrr%r rr6r5rf)ptrainingz `attn_output` should be of size z , but is ) sizerrrviewrr{ transposeryrrr|rr:matmulmathsqrtrBr r<r=rgr-rfdropoutrrr contiguousrDrar)r`rjr(rrrrrbszq_lenrJ query_statesrr attn_weights causal_mask attn_outputs r1rmzJambaAttention.forwardDs&**,, UA{{=11 [[// {{=11 #((eT^T]SS]]^_abcc __S%1I4=YYccdeghii #((eT5Mt}]]gghiklmm  %'5'<'>     #t~udm!L L L)CPTP]3^))$$&&))  "++Aq11<<>> !))#ud6FGG kk+..   LL.88r3rNNNFFN)rqrrrsrrrrr[r:rrr~boolrrmrtrus@r1rr!s# ^^{^x}^^^^^^@2637EI"'594949|49!.49u/0 49 !!AB 49  4949!!1249 u|Xel3XeEL>Q5RR S4949494949494949r3rceZdZdZfdZ d dejdeejdeejdee d e d e d eejfd Z xZ S)JambaFlashAttention2aF Jamba flash attention module. This module inherits from `JambaAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cbtj|i|t |_dSr)rZr[r_flash_attn_uses_top_left_mask)r`argskwargsrcs r1r[zJambaFlashAttention2.__init__s9$)&))) 3V2W2W.W+++r3NFrjr(rrrrrc z|\} } } ||} ||} ||}| | | |j|j} | | | |j|jdd} || | |j|jdd}|| | ||j \} }t| |j } t||j }|j sdn|j}| j}|t jkrt!jrt!j}n3t)|jdr |jj}n|jjj}t0d|d| |} | |} ||}| dd} |dd}t7| | ||| |t9|jdd|j|j }|| | |j !}|"|}|sd}|||fS) Nrr%_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .sliding_window)rrruse_top_left_mask)#rrrrrrr{ryrrrr|rrrrfr:rgis_autocast_enabledget_autocast_gpu_dtypehasattrrrr^rrr-rgetattrrrrDrarr)r`rjr(rrrrrrrrrJrrr dropout_raterk target_dtyperrs r1rmzJambaFlashAttention2.forwards&**,, UA{{=11 [[// {{=11 $((eT^T]SS __S%1I4=YYccdeghii #((eT5Mt}]]gghiklmm  %'5'<'|<| } | } | } |jr || dkrdnd}t,jj| | | ||jr|jnd | }| dd}||| |j}||}|d|fS) NaJambaModel is using JambaSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rjr(rrrrrr%rcudaTFr) attn_mask dropout_pr)rrrZrmrrrrrrr{rryrrr|rrBr9typerrr:r r<scaled_dot_product_attentionrrrar)r`rjr(rrrrrrrrJrrrrrrrcs r1rmzJambaSdpaAttention.forwards     [   77??+-)-"3# # &**,, UA{{=11 [[// {{=11 #((eT^T]SS]]^_abcc __S%1I4=YYccdeghii #((eT5Mt}]]gghiklmm  %'5'<'> !&&sE43CDD kk+.. D.00r3r) rqrrrsrr:rrrr~rrrmrtrus@r1rrs2637EI"'59G1G1|G1!.G1u/0 G1 !!AB G1  G1G1!!12G1 u|Xel3XeEL>Q5RR SG1G1G1G1G1G1G1G1G1G1r3r)eagerflash_attention_2sdpaceZdZdZdeffd Z d dejdede ej fdZ d dede ej fd Z d dede ej fd Z xZS) JambaMambaMixeru Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) rc t||_||_|j|_|j|_|j|_|j |jz|_ |j |_ |j |_|j|_t#j|j |j |j|j|j |jdz |_|j|_t,|j|_|j|_t#j|j|j dz|j|_t#j|j |j |jdzzd|_t#j|j |j d|_t=jd|jdzdddf}| |j d!}t#j"t=j#||_$t#j"t=j%|j |_&t#j|j |j|j|_'tQ|j |j)|_*tQ|j|j)|_+tQ|j|j)|_,tZst\/d dSdS) Nr) in_channels out_channelsr kernel_sizegroupspaddingr%rFTr6rbapThe fast path is not available because on of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)0rZr[rrrarrrrrr mamba_dt_ranktime_step_rankmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasr Conv1dconv1d hidden_act activationr actuse_mamba_kernelsuse_fast_kernelsrin_projx_projdt_projr:arangerCrr\logA_logr]Dout_projrW rms_norm_eps dt_layernorm b_layernorm c_layernormis_fast_path_availablerr)r`rrArcs r1r[zJambaMambaMixer.__init__?s  "!-$2 & 3!'!4v7I!I$2#3. i./#-))A-     !+&+, & 8y!143IA3MTXTabbb i 68KdNadeNe8elqrrr y!4d6LSWXXX  LD/!3 4 4T111W = HHT+R 0 0 ; ; = =\%)A,,// ej)?@@AA $"8$:JQUQ^___ ()<&BUVVV'(;ATUUU'(;ATUUU%    ^       r3Nrj cache_paramsr(c  |j\}}}|duoR|joK|dkoE|j|jjd|j|jjdcxko|knc}||dd}|dd\}} |||dz}|j j |j j d|j j d} |r[t|d|j|j| |j j|j}|d}n~|Zt"j||j|jdz df} |j|j| t-|| |j j|j}|||dz}||dd} t1j| |j|j|jgd\} }}|| } ||}||}|jjj }t1j!5t1j"|jjj |jj_ dddn #1swxYwY|| dd}t1j!5||jj_ dddn #1swxYwYt1j#|j$% }||%nd}|rhtM|j|j|d|d||dddf|dddf|j'| d|d d}ntQ||||dd|dd|j'%| |dd \}}|'|%|j|j||)|dd}|S) Nrrr%r4r6)r).rT) dt_softplus)delta_softplusreturn_last_state)*rBrrrrrrchunkrFrr^rrr#squeezerrr r<padrcopy_r"rr:splitr rr"r#r$rdatano_grad zeros_likeexprrAr!rr r )r`rjr'r(rOseq_lenrJuse_precomputed_statesprojected_statesgate conv_weightsrssm_parameters time_stepBCtime_proj_biasdiscrete_time_stepr& scan_outputs ssm_statecontextualized_statess r1cuda_kernels_forwardz$JambaMambaMixer.cuda_kernels_forwardssX "/!4 GQ  $ / 1  (8>qA&t~6> t  %)N,D,DQ,G,GGM{)..t{/A/F/Fq/I/I4;K]KbKbcdKeKeff ! x0%%b))(8   M*33B77MM' m// @UXeXklnXo@oqr?stt (8>>{KKK,]L$+JZgkgvwwwM  %)N,D,DQ,G,GGM]%<%>q!DD ]__ 4 4%3DL  " 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4Ytz'')) * * *3A3M--///SW ! I1'7f%"6*!!!Q$!!!Q$V     imm L'8" Aq!! Aq!! #"& ' ' ' #L)$)A'7==iHHH!% l.D.DQ.J.J K K$$s$3LL L M,,M03M0c f |j\}}}|j}||dd}|dd\} } || |dz} t |t} | r|j|j jd|kr|j r%|j|j  } n|j|j } | | j } |jr|dkr|j|j jd|kr|j|j } t!j| dd} | dddddf| dddddf<| |j|j <t!j| |jjdddddfzd} |jr| |jjz } ||  |d} nt0j| |j| jdz df} | |j|j <||| dd|f} n[t!j||j|jf| j |} ||| dd|f} || |dz} || dd}t!j ||j!|j|jgd\}}}|"|}|#|}|$|}|%|}t0j&|dd}t!j'|j() }t!j'|ddddddf|dddddddfz}|dddddddf|dddddddf)z}|| dddddddf)z}g}tU|D]}|dddd|ddf| z|dddd|ddfz} t!j+| ||dd|ddfd}|,|dddddft!j-|d}|| |j.ddddfzz}||| z}| r| |j|j <|/|dd}|S) Nrr%r4rr6)shiftsdims.r)0rBrfrrr,rFr7r~rrrcloner-r9rrr:rollrErr^rrrr r<r.rrrrrr0r r"r#r$rsoftplusr4rrArrrstackrr )r` input_statesr'r(rOr5rJrfr7rjr8rrA conv_stater:r;r<r=r?r& discrete_A discrete_BdeltaB_ur@r scan_outputrBs r1 slow_forwardzJambaMambaMixer.slow_forwards!-!3 GQ"<< 55??1EE.44QA4>> t  %)N,D,DQ,G,GGM|-MNN  P0@FqIZWW} D(3DNCIIKK (3DNC ! ]%9::I. T7a<< ,T^<B1ESS)5dnE "Z 2BGGG '4QQQ1W'= 111aaa8$;E (8 % *t{7I!!!QPQPQPQ'7R*RXZ [ [ [ %6!T[%55M $ 7 7 : :5 A A K KB O O ]..!*]-@-DDaH 59 h%h%|h%7h%!!12 h%h%h%h%VR%R%7WR%pxy~zJqKR%R%R%R%p:>59 N N7 N!!12 N N N N N N N Nr3rc$eZdZfdZdZxZS)JambaMLPct|j|_|j|_t j|j|jd|_t j|j|jd|_t j|j|jd|_t|j |_ dSNFr) rZr[rarr r gate_projup_proj down_projr ract_fnr`rrcs r1r[zJambaMLP.__init__Ds !-!'!94#3T5KRWXXXy!143IPUVVV 4#94;KRWXXXV./ r3c||||||zSr)rYrZrWrX)r` hidden_states r1rmzJambaMLP.forwardMs>~~dkk$..*F*FGG$,,WcJdJddeeer3)rqrrrsr[rmrtrus@r1rTrTCsN00000fffffffr3rTcfeZdZdZdeffd ZdejdeejejffdZ xZ S)JambaSparseMoeBlocka This implementation is strictly equivalent to standard MoE with full capacity (no dropped tokens). It's faster since it formulates MoE operations in terms of block-sparse operations to accomodate imbalanced assignments of tokens to experts, whereas standard MoE either (1) drop tokens at the cost of reduced performance or (2) set capacity factor to number of experts and thus waste computation and memory on padding. rcdtj|_j|_j|_j|_tj |j|jd|_ tj fdt|jD|_dS)NFrc.g|]}tSr,)rT)r.rJrs r1r2z0JambaSparseMoeBlock.__init__..fs!%X%X%X1hv&6&6%X%X%Xr3)rZr[ra hidden_dimrffn_dimr'num_experts_per_tokrGr rrouter ModuleListrexpertsr[s `r1r[zJambaSparseMoeBlock.__init__^s  ,/ !-/ i1ANNN }%X%X%X%XdFV@W@W%X%X%XYY r3rjr)c|j\}}}|d|}||}tj|dt j}t j||jd\}}| |j }t j ||z|f|j |j }t j j||jddd} t%|jD]} |j| } t j| | \} } | jddkr>|d | fd|}| ||| | d fz}|d| | |j ||||}||fS)  r6rrr4)rfr9) num_classesr%rN)rBrreFr=r:rAr>rGr-rfrr9r r<r?r'permuterrgwhererD index_add_)r`rjrOrPrbr&rIrKfinal_hidden_statesrL expert_idx expert_layeridxtop_x current_statecurrent_hidden_statess r1rmzJambaSparseMoeBlock.forwardhs2?2E/ OZ%**2z::  M22 )Mq LLL,1J XZ,[,[,[))),,]-@AA#k / ): 6m>QZgZn    h)112BPTP`1aaiijkmnpqrr   011 d dJ< 3L[%<==JC{1~"" *$+6>>r:NNM$0L$?$?/RWY\^bRbBc$c ! * *1e5J5M5MmNa5b5b c c c c199*oWabb"M11r3) rqrrrsrrr[r:rrrmrtrus@r1r_r_Rs  Z{ZZZZZZ&2U\&2eEL%,FDWXXX ,V-?VEX Y Y Yr3NFrjr(rrroutput_router_logitsrrr)c T|} ||}||||||||\}} } | |z}|} ||}||} t | t r| \}} n| d} }| |z}|f}|r|| fz }|r|| fz }|r|| fz }|S) Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_value (`HybridMambaAttentionDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. )rjr(rrrrrN)r~r|rr}r7r8)r`rjr(rrrrrrresidualself_attn_weightspresent_key_value ff_outputsr&outputss r1rmz"JambaAttentionDecoderLayer.forwards <!,,];; >Bnn')%)/)?M? ? ; (*;!=0 !--m<< &&}55 j% ( ( <+5 (M==+5t=M =0  "  , )+ +G  , )+ +G  (  ' 'Gr3NNNFFFNrqrrrsrrr[r:rrrr~rrrrmrtrus@r1rwrws:Z{ZsZZZZZZ2637EI,1/4$)59DD|D!.Du/0 D !!AB D $D> D'tnDD>D!!12D u (51BEDU1U+V"WW XDDDDDDDDr3rwc*eZdZdedeffd Z ddejdeejdeej d ee d ee d ee d ee d eej de ej ee ej ej fffdZxZS)JambaMambaDecoderLayerrrcVt|j|}t|||_|dkrt nt }|||_t|j |j |_ t|j |j |_ dS)N)rrrr ) rZr[ryrrr_rTr}rWrar!r~rrs r1r[zJambaMambaDecoderLayer.__init__s / : $FiHHH 1FDWXXX ,V-?VEX Y Y Yr3NFrjr(rrrrrrr)c H|} ||}||||}d} | |z}|} ||}||} t | t r| \}} n| d} }| |z}|f} |r| | fz } |r| |fz } |r| | fz } | S)r)rjr'r(N)r~rrr}r7r8)r`rjr(rrrrrrrrrr&rs r1rmzJambaMambaDecoderLayer.forwards<!,,];;  '')#  !!=0 !--m<< &&}55 j% ( ( <+5 (M==+5t=M =0  "  , )+ +G  ) ( (G  (  ' 'Gr3rrrus@r1rrs:Z{ZsZZZZZZ2637EI,1/4$)59AA|A!.Au/0 A !!AB A $D> A'tnAD>A!!12A u (51BEDU1U+V"WW XAAAAAAAAr3raJ This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`JambaConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zSThe bare Jamba Model outputting raw hidden-states without any specific head on top.c<eZdZeZdZdZddgZdZdZ dZ dZ dZ dZ dS)JambaPreTrainedModelmodelTrwrrc|jj}t|tjtjfrJ|jjd||j |j j dSdSt|tj rS|jjd||j -|jj|j  dSdSdS)Nr)r@std) rinitializer_ranger7r rrr^r1normal_rzero_ Embedding padding_idx)r`modulers r1 _init_weightsz"JambaPreTrainedModel._init_weightsRsk+ fry")4 5 5 ? M  & &CS & 9 9 9{&  &&((((('&  - - ? M  & &CS & 9 9 9!- "6#56<<>>>>> ? ?--r3N)rqrrrsr config_classbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_2_supports_sdpa_supports_cache_class _is_statefulrr,r3r1rrCs^ L&*#57OP"3!N L ? ? ? ? ?r3ra Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`HybridMambaAttentionDynamicCache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): A HybridMambaAttentionDynamicCache object containing pre-computed hidden-states (keys and values in the self-attention blocks and convolution and ssm states in the mamba blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. Key and value cache tensors have shape `(batch_size, num_heads, seq_len, head_dim)`. Convolution and ssm states tensors have shape `(batch_size, d_inner, d_conv)` and `(batch_size, d_inner, d_state)` respectively. See the `HybridMambaAttentionDynamicCache` class for more details. If `past_key_values` are used, the user can optionally input only the last `input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. ) attentionrcleZdZdZdeffd ZdZdZee  dde j de e j d e e j d e ed e e jd e ed e ede ede ede ede e j deeeffdZdZdZxZS) JambaModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`JambaDecoderLayer`] Args: config: JambaConfig rc0t||j|_|j|_t j|j|j|j|_g}t|j D]:}t|j |}| |||;t j||_|j|_t#|j|j|_d|_|dS)N)rr F)rZr[ pad_token_idr vocab_sizer rra embed_tokensrrQALL_DECODER_LAYER_TYPESrrrflayersr{rWr!final_layernormgradient_checkpointing post_init)r`rdecoder_layersr layer_classrcs r1r[zJambaModel.__init__s    !. +L):FFDWXXX&+# r3c|jSrrros r1get_input_embeddingszJambaModel.get_input_embeddingss   r3c||_dSrrr`values r1set_input_embeddingszJambaModel.set_input_embeddingss!r3N input_idsr(rr inputs_embedsrroutput_hidden_statesr return_dictrr)c ||n |jj}| | n |jj} ||n |jj}||n |jj}| | n |jj} |du|duz rt d|jr%|jr|rt dd}|| |}|} |r|t d| &tj | jd| j} || d}|||| } ||| }|rdnd}|rdnd}| rdnd}|jD]}t)|t*r|n| }|r|| fz }|jr*|jr#||j| ||||| || }n|| ||||| || }|d} |r|d ||dfz }| r|d  ||d fz }|| } |r|| fz }|r|jsd |_|sdn|}| st5d | ||||fDSt7| |||| S)Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FzJamba requires an initialized `HybridMambaAttentionDynamicCache` to return a cache. None was provided, so no cache will be returned.rrrr,)r(rrrrrrr6Tc3K|]}||V dSrr,)r.vs r1 z%JambaModel.forward..>s0= ===r3)last_hidden_staterrj attentionsr&)rrrrruse_return_dictrrrrrrr:rrBr9rF_update_causal_mask_update_mamba_maskrr7r_gradient_checkpointing_func__call__rrr8r)r`rr(rrrrrrrrrrjr mamba_maskall_hidden_statesall_self_attnsall_router_logits decoder_layer layer_mask layer_outputs next_caches r1rmzJambaModel.forwards2C1N--TXT_Tq$8$D $+Jj %9$D $+Jj "+!6IIDK) >M'1-AW'X'Xi^iJ# 6!m%55!* t}  $ A A!*! #%(" ! ! !. !#-!-#2&7)='#1 ! ! ! *!,M  : #/"}Q'7&99N# > $0%-*;)==%,,];;   2 -!1 1   6?#E 615O .!*?TT  '5FXij  &+&+%+     r3c|jjdkr |d|vr|SdS|j|j}}t j|j}|jd}|ddz}t j||f|||} |dkrt j | d} | t j ||| ddkz} | ddddddf |jdddd} || } |d krw|jd} | d d| fd|ddddddfdz} | d d| f| || d d| f<|jjd kr'|%|jjd krt%j| |} | S) Nrrrr6) fill_valuerfr9)diagonalrrr%.rr)rr{rfr9r:finfominrBfulltriurrDrCrGr5eq masked_fillrr_unmask_unattended) r`r( input_tensorrrfr9 min_dtyperP target_lengthr mask_length padding_masks r1rzJambaModel._update_causal_maskKs ; +/B B B)c^.C.C%%4$*L,?vK&&* &,Q/&r*Q. j/=!Ai_dmsttt a  *[1===Ku|M&AAANDZDZ[]_`DaDaaa !$aaa"23::<;Ma;PRSUWY[\\  %%++--K!!##q((,226 *3  +<=@@EEWXWXWXZ^`dfgfgfgWgHhHkHkloHpHpp 1 $ ! q ^%?EIn`aNaDbDb%?Jr3) NNNNNNNNNNN)rqrrrsrrr[rrrJAMBA_INPUTS_DOCSTRINGr:rrrr~rrrrrrmrrrtrus@r1rrs {&!!!"""+*+ABB'+1537FJ59$(,0/3/3&*59z z #z !.z u/0 z ""BC z   12 z D>z $D>z 'tnz 'tnz d^z !!12z  u,, -z z z CBz x!!!F       r3rc!eZdZdgZdeffd ZdZdZdZdZ dZ d Z e e eee  dd ejd eejdeejdeedeejdeejdeedeedeedeedeedeejdeeed fdeeeffdZ ddZxZS) JambaForCausalLMzlm_head.weightrcFt|t||_|j|_t j|j|jd|_|j |_ |j |_ |j |_ | dSrV) rZr[rrrr rralm_headrouter_aux_loss_coefr'rdrr[s r1r[zJambaForCausalLM.__init__~s    ''  +y!3V5FUSSS $*$?!!-#)#=  r3c|jjSrrrros r1rz%JambaForCausalLM.get_input_embeddings z&&r3c||j_dSrrrs r1rz%JambaForCausalLM.set_input_embeddings"' r3c|jSrrros r1get_output_embeddingsz&JambaForCausalLM.get_output_embeddingss |r3c||_dSrr)r`new_embeddingss r1set_output_embeddingsz&JambaForCausalLM.set_output_embeddingss % r3c||_dSrr)r`decoders r1 set_decoderzJambaForCausalLM.set_decoders  r3c|jSrrros r1 get_decoderzJambaForCausalLM.get_decoders zr3) output_typerNrr(rrrlabelsrrrrrrnum_logits_to_keepr)c ||n |jj}| | n |jj} | | n |jj} | | n |jj} ||||||||| | | |  }|d}| ||}n#||d| dddf}d}||j|||jfi|}d}| rRt| r|j n|d|j |j |}|%||j ||jzz }| s |f|ddz}| r|f|z}||f|zn|St!||||j|j|j|j S)a Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. num_logits_to_keep (`int` or `None`, *optional*): Calculate logits for the last `num_logits_to_keep` tokens. If `None`, calculate logits for all `input_ids`. Only last token logits are needed for generation, and calculating them only for that token can save memory, which becomes pretty significant for long sequences. Returns: Example: ```python >>> from transformers import AutoTokenizer, JambaForCausalLM >>> model = JambaForCausalLM.from_pretrained("ai21labs/Jamba-v0.1") >>> tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rr(rrrrrrrrrr.r6r)lossaux_losslogitsrrjrr&)rrrrrrr loss_functionrrUr&r'rdrr-r9rrrjr)r`rr(rrrrrrrrrrr loss_kwargsrrjrrroutputs r1rmzJambaForCausalLM.forwardsf2C1N--TXT_Tq$8$D $+Jj  %9$D $+Jj &1%q*A)A)C)C&CD #~';A'>>>%aaa&78 > Y_Q/DKO  %,*>)..0077;;a?L  % %n&91 = = =  F+AAA 0B/B/D/D,DE   $ $+];LL')=)=)?)?@L ,#2&"0(<&*k&D"0   r3) NNNNNNNNNNNNN)NNNFNNT)rqrrrs_tied_weights_keysrr[rrrrrrrrrr_CONFIG_FOR_DOCr:rrrr~rrrrrrmr rtrus@r1rr{sG*+ {      '''(((&&&+*+ABB+DSbccc'+1537FJ59-1$(,0/3/3&*599=l l #l !.l u/0 l ""BC l   12 l )*l D>l $D>l 'tnl 'tnl d^l !!12l %U39%56l u// 0!l l l dcCBl b"55555555r3ra The Jamba Model with a sequence classification head on top (linear layer). [`JambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cveZdZfdZdZdZee ddee j dee j dee j dee e ee jfd ee jd ee j d eed eed eedeede eeffdZxZS)JambaForSequenceClassificationct||j|_t||_t j|j|jd|_| dSrV) rZr[ num_labelsrrr rrascorerr[s r1r[z'JambaForSequenceClassification.__init__Usi     +'' Yv14?OOO  r3c|jjSrrros r1rz3JambaForSequenceClassification.get_input_embeddings^rr3c||j_dSrrrs r1rz3JambaForSequenceClassification.set_input_embeddingsarr3Nrr(rrrrrrrrr)c "| | n |jj} ||||||||| |  } | d} || } ||jd}n |jd}|jj|dkrt d|jjd}nv|rtj||jj ddz }||jdz}| | j }nd}| tj || j |f}d}||| |||j}| s|f| ddz}||f|zn|St||| j| j| j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)r(rrrrrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r6r)rr pooled_logitsr)rrrrjr)rrrrrBrrr:rrargmaxr-r9rrrrrjr)r`rr(rrrrrrrrtransformer_outputsrjrrOsequence_lengthsrrrs r1rmz&JambaForSequenceClassification.forwardds(&1%E $D>E 'tnE d^E  u66 7E E E CBE E E E E r3r )Nr%N)Qrrtypingrrrrrrr:torch.nn.functionalr r<rktorch.utils.checkpoint activationsr cache_utilsr r generationrmodeling_attn_mask_utilsrmodeling_outputsrrrmodeling_utilsrutilsrrrrutils.import_utilsrrrrconfiguration_jambarmodeling_flash_attention_utilsr&mamba_ssm.ops.selective_scan_interfacerr +mamba_ssm.ops.triton.selective_state_updater! causal_conv1dr"r#rr% get_loggerrqrr rrrUModulerWr|r~rrrrzrrTr_rwrJAMBA_START_DOCSTRINGrrrrrr r,r3r1r+s( :::::::::::::::: !!!!!!........)))))) .-----  -,,,,,KJJJJJJQXXXXXXXXRRRRRRR@P=-~8DDDDDDDDD-7**.0@BVXfg  H % % "& -1 Q&Q&uU\':D@AQ&#Q&U\* Q&  5<  Q&Q&Q&Q&jJJJJJ29JJJ* UU\ U# U%, U U U UToToToToTo|ToToTopW9W9W9W9W9RYW9W9W9v]9]9]9]9]9>]9]9]9BO1O1O1O1O1O1O1O1f- HNHNHNHNHNbiHNHNHNX f f f f fry f f f<2<2<2<2<2")<2<2<2~OOOOOOOOdLLLLLRYLLL^"Y???????? ?.EN)CMcddY JJJJJ%JJ  J\FFFFF+_FFFR  V V V V V %9V V ! V V V r3