gfdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZmZm Z ddl!m"Z"ej#e$Z%e rddl&m'Z'ndZ'erddl(m)Z)m*Z*ddl+m,Z,nd\Z,Z*Z)er ddl-m.Z.m/Z/nd\Z/Z.e0e,e*e.e/e)fZ1dZ2dZ3Gdde j4Z5Gdde j4Z6Gdde j4Z7GddeZ8eGd d!eZ9eGd"d#eZ:d$Z;d%Z<ed&e;Gd'd(e8Z=ed)e;Gd*d+e8eZ>dS),zPyTorch MAMBA model.N) dataclass)AnyDictOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN) MambaCache)GenerationMixin)PreTrainedModel) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging)is_causal_conv1d_availableis_mamba_ssm_availableis_mambapy_available) MambaConfig)pscan)mamba_inner_fnselective_scan_fn)selective_state_updateNNN)causal_conv1d_fncausal_conv1d_update)NNzstate-spaces/mamba-130m-hfrc 0eZdZdZdedeffd Z d dejde e de ej d e ej fd Z d de e de ej d e ej fd Z d de e de ej d e ej fd ZxZS) MambaMixeru Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) config layer_idxc t||_|j|_|j|_|j|_|j|_t|j |_ ||_ |j |_ tj|j|j|j |j|j|jdz |_|j|_t$|j|_|j|_tj|j|jdz|j|_tj|j|j |jdzzd|_tj|j |jd|_t5jd|jdzt4jdddf}||jd}tjt5j ||_!tjt5j"|j|_#tj|j|j|j|_$|j|_tJs\|jr9tMrtN(d dStSd tN(d dSdS) Nr) in_channels out_channelsbias kernel_sizegroupspaddingr(FTdtypea7The fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. Falling back to the mamba.py backend. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1dzuse_mambapy is set to True but the mambapy package is not installed. To install it follow https://github.com/alxndrTL/mamba.py.aThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. Falling back to the sequential implementation of Mamba, as use_mambapy is set to False. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. For the mamba.py backend, follow https://github.com/alxndrTL/mamba.py.)*super__init__r# hidden_size state_sizessm_state_size conv_kernelconv_kernel_sizeintermediate_sizeinttime_step_rankr$ use_conv_biasr Conv1dconv1d hidden_act activationr act use_mambapyLinearuse_biasin_projx_projdt_projtorcharangefloat32expand contiguous ParameterlogA_logonesDout_projis_fast_path_availablerlogger warning_once ImportError)selfr#r$A __class__s d/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/mamba/modeling_mamba.pyr2zMambaMixer.__init__Ks  !-$/ & 2!'!9!&"788"#1i./%*)&*     !+&+,!-y!143IA3MTZTcddd i 68KdNadeNe8elqrrr y!4d6LSWXXX  LD/!35= I I I$PQPQPQ' R HHT+R 0 0 ; ; = =\%)A,,// ej)?@@AA $"8$:JQWQ`aaa  %  ')) ''F &Z##J  N hidden_states cache_paramscache_positionattention_maskc ||dd}|jr|t||jj|jr |jjnd|jj|j j|j j|j r|j j ndtj|j  dd|j |j j d }ny|dd\}}|||dz}|jj|jjd|jjd}|g|ddkr[t+|d|j|j||jj|j}|d}nu|Qt4j||j|jdz df} ||j| |tA|||jj|j}|||dz}||dd} tj!| |j"|j#|j#gd\} } } |j j| ddz}tj|j  }tI|j d r|j j nd}|t|ddkrhtK|j&|j|d |d || dddf| dddf|j|d |d d}nztO|||| dd| dd|j ||dd \}}|||(|j|| |dd}|S) Nrr,T) delta_biasdelta_softplusdimrr0)r?r().r) dt_softplus)rareturn_last_state))rD transposetrainingrr=weightr;r(rErFrQrCfloatrGexprNrPchunk unsqueezeviewsizer squeeze conv_statesr$r?r functionalpadr7shapeupdate_conv_statersplitr:r5hasattrr ssm_statesrupdate_ssm_state)rVr[r\r]r^projected_statescontextualized_statesgate conv_weightsrpssm_parameters time_stepBCdiscrete_time_steprWtime_proj_bias scan_outputs ssm_states rYcuda_kernels_forwardzMambaMixer.cuda_kernels_forwardsr << 66@@AFF =X P\1$2  "$($6@   D " # $.2mE "((***4:++--... <,2244#%%% ! !"#3"8"8"8"B"B M4) -0H0H0K0K K  ;-224;3E3J3J13M3Mt{OaOfOfghOiOijjL'N1,=,A,A 4!))"-- ,T^< K$O !! !. 7 7 ; ; +"$-"3"3%(= @STV@W(WYZ'[##K!224>;P^___ 0!<1Ado!!! ) -0H0H0K0K K "[[)@)@A)F)FGGN#k!4d6I4K^ _egOIq!"&!4y7J7J1a7P7P!P 4:++--...A:A$,PV:W:WaT\.44666]aN'N1,=,A,A5 +DN;!&)&v.aaadGaaadGFL" $   )B-- +<!&KK1%%KK1%%FLLNN"#'&* + + +' i(\-E 11$.)LLL%)MM,2H2HA2N2N$O$O !$$rZc d |j\}}}|j}||dd} | dd\} } || |dz} |v|j|j} | | j } |jd|j krtj | |j | jdz df} ||j| |||| dd|f} n||j| |} t%j| |jjdddddfzd} |jr| |jjz } ||  |d} n[t%j||j|jf| j |} ||| dd|f} || |dz} || dd}t%j||j|j|jgd\}}}||}tj |dd}t%j|j ! }t%j|ddddddf|dddddddfz}|dddddddf|dddddddf!z}|| dddddddf!z}|j"r|j#r|tI|dd|dd}||dz%ddd}|| |j&ddddfzz}||| z}ng}tO|D]}|dddd|ddf| z|dddd|ddfz} t%j(| ||dd|ddfd}|)|dddddft%j*|d}|| |j&ddddfzz}||| z}|%|j|j+| |,|dd}|S) Nrr,rbrr0.devicer/r )-rsr/rDrfrkrlrwr$clonetorr7r rqrrrtr@r=rGsumrhr;r(zerosr8r5rErur:rFsoftplusrjrNrirArgrrorPrangematmulappendstackcopy_rQ)rV input_statesr\r]r^ batch_sizeseq_len_r/ryr[r{r conv_stater}r~rrrrW discrete_A discrete_BdeltaB_uhs scan_outputrirzs rY slow_forwardzMambaMixer.slow_forwards!-!3 GQ"<< 55??1EE.44QA4>> t  %)N,D,DQ,G,GGM  #$/?EEGGI! ]%9::I#A&$*???]..!*]-@-DDaH ..t~z>ZZZ $])C)CC'M)R S S );;DNM[ijj % *t{7I!!!QPQPQPQ'7R*RXZ [ [ [ %6!T[%55M $ 7 7 : :5 A A K KB O O  T3T5HI$+5I!HHT[[%?%?XgX %NOOM  %)N,D,DQ,G,GGM]%<%!!!QQQPQSTSTST*AUU #l9<<+>+>!!!Q' @T@TUW@X@XYY ##K111a$89999+l;;;K%aaa 9N)NOK&$7K''7==iHHH!% k.C.CAq.I.I J J$$rZctrNd|jjjjvr6t js|||||S| ||||S)Ncuda) rRrErhrtyperG_dynamo is_compilingrr)rVr[r\r]r^s rYforwardzMambaMixer.forward>sl " jf 0B0I0N&N&NW\WdWqWqWsWs&N,,]L.Zhii i   nn]]]rZr)__name__ __module__ __qualname____doc__rr9r2rGTensorrr LongTensorrrr __classcell__rXs@rYr"r"Cs:{:s::::::~.25959 c%c%|c%z*c%!!12 c% !!12 c%c%c%c%LN%N%x 7KN%aijojza{N%S[\a\lSmN%N%N%N%h.25959 ^ ^z* ^!!12 ^ !!12 ^ ^ ^ ^ ^ ^ ^ ^rZr"c,eZdZdfd ZdZdZxZS) MambaRMSNormư>cttjt j||_||_dS)zL MambaRMSNorm is equivalent to T5LayerNorm and LlamaRMSNorm N)r1r2r rLrGrOrhvariance_epsilon)rVr3epsrXs rYr2zMambaRMSNorm.__init__KsD l5:k#:#:;; #rZc|j}|tj}|ddd}|tj||jzz}|j||zS)Nr,r0T)keepdim) r/rrGrIpowmeanrsqrtrrh)rVr[ input_dtypevariances rYrzMambaRMSNorm.forwardSs|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::rZc:|jjdd|jS)Nrz, eps=)rhrsrrVs rY extra_reprzMambaRMSNorm.extra_reprZs#+#A&EEd.CEEErZ)r)rrrr2rrrrs@rYrrJsb$$$$$$;;;FFFFFFFrZrcreZdZfdZ ddeedeejdeejfdZxZ S) MambaBlockct||_||_|j|_t |j|j|_t|||_ dS)Nrr$) r1r2r#r$residual_in_fp32rr3layer_norm_epsilonnormr"mixer)rVr#r$rXs rYr2zMambaBlock.__init___sd  " & 7 !39RSSS )<<< rZNr\r]r^c|}|||jjj}|jr|t j}|||||}||z}|S)Nr.r\r]r^)rrrhr/rrGrIr)rVr[r\r]r^residuals rYrzMambaBlock.forwardgs! -"2"29I9O"2"P"PQQ   2{{5=11H  ^dr#  !=0 rZr) rrrr2rr rGrrrrs@rYrr^s=====.25959 z*!!12  !!12 rZrc0eZdZdZeZdZddgZdZdZ dZ dS)MambaPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. backbonerr"Tct|trd|j_d|j_|jjdz|jjz}|jjdkr+tj |j j |n<|jjdkr,tj |j j | |tjtj|jjt'j|jjt'j|jjz zt'j|jjz|jj}|tjtj|  z}tj5|j j|dddn #1swxYwYd|j j_t|tjrB|j:t?|jdds$tj |jnJt|tj!r0tj "|j |jj# |jj$r|%D]\}}|d vrtj &|t'j'd  tj5|t'j'|jj(z}dddn #1swxYwYdSdS) zInitialize the weights.Tgconstantrandom)minN _no_reinitF)std)zout_proj.weight)a)) isinstancer"rN_no_weight_decayrPr#r:time_step_scaletime_step_init_schemer init constant_rFrhuniform_rGrjrandr8mathrM time_step_max time_step_minclamptime_step_floorexpm1no_gradr(rrrBgetattrzeros_ Embeddingnormal_initializer_rangerescale_prenorm_residualnamed_parameterskaiming_uniform_sqrtnum_hidden_layers)rVmodule dt_init_stddtinv_dtnameps rY _init_weightsz"MambaPreTrainedModel._init_weightssQ fj ) ) 2,0FL )(,FH %+4d:T[=XXK{0J>>!!&."7EEEE2h>>  !6 kRRR 4;8998DK566$+B[9\9\\^(4;4556e 3e44  %)U["%5%5$5666F 2 2#))&111 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2-1FN  * fbi ( ( N{&v{L%@@0GNN6;///  - - N GOOFMt{/LO M M M ; / F"2244 F Fa... G,,Q$)A,,,???FFTYt{'DEEEFFFFFFFFFFFFFFF F F F Fs$+ GGG"L66L: =L: N) rrrrr config_classbase_model_prefix_no_split_modulessupports_gradient_checkpointing _is_statefulrrZrYrrzsU L"%|4&*#L-F-F-F-F-FrZrceZdZUdZdZeejed<dZ ee ed<dZ ee ejed<dS) MambaOutputa# Class for the MAMBA model outputs. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. cache_params (`MambaCache`): The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to avoid providing the old `input_ids`. Includes both the State space model state matrices after the selective scan, and the Convolutional states hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. Nlast_hidden_stater\r[) rrrrrrrG FloatTensor__annotations__r\r r[rrrZrYrrsi$6:x 12999)-L(:&---80`, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. cache_params (`MambaCache`, *optional*): If passed along, the model uses the previous state in all the blocks (which will give the output for the `input_ids` provided as if the model add `state_input_ids + input_ids` as context). use_cache (`bool`, *optional*): If set to `True`, the `cache_params` is returned and can be used to quickly generate the next logits. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. Contrarily to `position_ids`, this tensor is not affected by padding. It is used to update the cache in the correct position and to infer the complete sequence length. z_The bare MAMBA Model transformer outputting raw hidden-states without any specific head on top.cPeZdZfdZdZdZdZeee e e e  dde ejde ejd e ed e ed e ed e ed e ejde ejdeee ffdZxZS) MambaModelcttjjj|_tjfdtj D|_ d|_ tjj |_||j|dS)Nc2g|]}t|S)r)r).0idxr#s rY z'MambaModel.__init__..%s&$r$r$r3Z#%F%F%F$r$r$rrZFr)r1r2r r vocab_sizer3 embeddings ModuleListrrlayersgradient_checkpointingrrnorm_f"_register_load_state_dict_pre_hook load_hook post_initrVr#rXs `rYr2zMambaModel.__init__!s    ,v'8&:LMMm$r$r$r$rRWX^XpRqRq$r$r$rss &+#"6#56;TUUU  //??? rZcv|D]5}d|vr/||||dd<dS6dS)Nz embedding.z embeddings.)popreplace)rV state_dictprefixargsks rYrzMambaModel.load_hook-sW  Aq  EO^^TUEVEV 199\=AAB!  rZc|jSNr rs rYget_input_embeddingszMambaModel.get_input_embeddings3s rZc||_dSrrrVnew_embeddingss rYset_input_embeddingszMambaModel.set_input_embeddings6s (rZ checkpoint output_typerN input_ids inputs_embedsr\ use_cacheoutput_hidden_states return_dictr]r^returnc ,||n |jj}||n|js |jjnd}||n |jj}|du|duz rt d|||}|jr |jr|rd}|rp|\t|j| d|j |j }tj d|jj|j }n|t dnd}|} |rdnd} |jD]F} |jr&|jr|| j| |||} n| | |||} |r| | fz} G|| } |r| | fz} |st'd | || fDSt)| |r|nd| S) NFz:You must specify exactly one of input_ids or inputs_embedsrrrzYou have to specify the `cache_position` manually when `use_cache=True` and `cache_params` is passed, you don't have to pass a `cache_params` if you are in prefilling stage because in that case it will be initialized for you automaticallyrrc3K|]}||V dSrr)rvs rY z%MambaModel.forward..s(ffqXYXeXeXeXeXeffrZ)rr\r[)r#r)rgr(use_return_dict ValueErrorr rr rnrr/rGrHr6r _gradient_checkpointing_func__call__rtupler) rVr&r'r\r(r)r*r]r^r[all_hidden_states mixer_blocks rYrzMambaModel.forward9s_$%9$D $+Jj "+!6IIZ^Zg=rT[=R=Rmr %0% J 'tn J d^J !!12J !!12J  uk! "J J J CB J J J J J rZrz The MAMBA Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). c eZdZdgZfdZdZdZdZdZ dde d e e e fd e d e e e ffd Z ddeedeejdeejfdZeeeeee ddeejdeejdeejdeedeejdeedeedeedeejd eeeffdZxZ S)MambaForCausalLMzlm_head.weightct|t||_t j|j|jd|_| dS)NFr-) r1r2rrr rBr3r lm_headrrs rYr2zMambaForCausalLM.__init__s^    "6** y!3V5FUSSS  rZc|jSrr?rs rYget_output_embeddingsz&MambaForCausalLM.get_output_embeddingss |rZc||_dSrrAr s rYset_output_embeddingsz&MambaForCausalLM.set_output_embeddingss % rZc4|jSr)rrrs rYrz%MambaForCausalLM.get_input_embeddingss}11333rZc6|j|Sr)rr"r s rYr"z%MambaForCausalLM.set_input_embeddingss}11.AAArZroutputs model_kwargsnum_new_tokensr+c 6|dd|d<|ddr"d|vr|d|ddd|z|d<d|vrC|d}tj|||jddfgd |d<|S) Nr\r(Tr]r0r^rrrb)getrGcatnew_onesrs)rVrGrHrIkwargsr^s rY#_update_model_kwargs_for_generationz4MambaForCausalLM._update_model_kwargs_for_generations(/{{>4'H'H ^$   [$ / / b L00-.:-9:J-KBCC-PSa-aL) * | + +)*:;N-2Y!8!8.:Nq:QST9U!V!VW]_...L) *rZNr\r]r^c J|rg|td|ddkr$|dddfd}|d}n&tjd|jj|j}||d|i}nd|i}|||||d|S)Nz`cache_position` should not be None as it should have been initialized in `model.generate`, you are responsible for passing in a valid `cache_position` if you are calling `prepare_inputs_for_generation` directly with `use_cache=True`rr0r-r'r&)r\r(r]r^) r2rlrGrHr#r6rrKupdate) rVr&r'r(r\r]r^rN model_inputss rYprepare_inputs_for_generationz.MambaForCausalLM.prepare_inputs_for_generations  c% e a 1$$%aaae,66r:: !-%)N"'a1HQZQa!b!b!b  $)=+];LL')=)=)?)?@L ,&"0"0      rZr#r&r'labelsr)r*r(c ||n |jj}|||||||| |} | d} || |jjj} d}||| j}| dddddf }|dddf }t}|| d| d| d}|s| f| ddz}||f|zn|St|| | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)r\r'r)r*r(r]r^r.r0r)rrr\r[)r#r1rr?rrhr/rirrKr rmrnrr\r[)rVr&r^r'r\rTr)r*r(r]rN mamba_outputsr[rr shift_logits shift_labelsloss_fctoutputs rYrzMambaForCausalLM.forwards2&1%7 !.7  u)) *7 7 7 CB 7 7 7 7 7 rZr=)?rr dataclassesrtypingrrrrrrGtorch.utils.checkpointr torch.nnr activationsr cache_utilsr generationrmodeling_utilsrutilsrrrrrutils.import_utilsrrrconfiguration_mambar get_loggerrrS mambapy.pscanr&mamba_ssm.ops.selective_scan_interfacerr+mamba_ssm.ops.triton.selective_state_updater causal_conv1drr allrRr9r:Moduler"rrrrrMAMBA_START_DOCSTRINGr8rr=rrZrYrpsK !!!!!!44444444444444 %%%%%%!!!!!!%%%%%%))))))------kjjjjjjjjj,,,,,,  H % %####### EQXXXXXXXXRRRRRRR@P=-~8DDDDDDDDD-7**.0@BVXfg3D^D^D^D^D^D^D^D^NFFFFF29FFF(89F9F9F9F9F?9F9F9Fx =====+== =0 =====+== =6 Bei i i i i %i i i X V V V V V +_V V V V V rZ