g+dZddlmZmZmZer ddlmZddlmZej e Z GddeZ Gdd eZ d S) zMpt configuration) TYPE_CHECKINGOptionalUnion)PretrainedConfig)loggingcPeZdZdZ d fd Zedd ZxZS)MptAttentionConfiga This is the configuration class to store the configuration of a [`MptAttention`] class. It is used to instantiate attention layers according to the specified arguments, defining the layers architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the MPT [mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b) architecture. Most of the arguments are kept for backward compatibility with previous MPT models that are hosted on the Hub (previously with `trust_remote_code=True`). Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: attn_type (`str`, *optional*, defaults to `"multihead_attention"`): type of attention to use. Options: `"multihead_attention"`, `"multiquery_attention"`. attn_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability for the attention layers. attn_impl (`str`, *optional*, defaults to `"torch"`): The attention implementation to use. One of `"torch"`, `"flash"`, or `"triton"`. clip_qkv (`float`, *optional*): If not `None`, clip the queries, keys, and values in the attention layer to this value. softmax_scale (`float`, *optional*, defaults to `None`): If not `None`, scale the softmax in the attention layer by this value. If `None`, will default to `1/sqrt(hidden_size)`. prefix_lm (`bool`, *optional*, defaults to `False`)): Whether the model should operate as a Prefix LM. This requires passing an extra `prefix_mask` argument which indicates which tokens belong to the prefix. Tokens in the prefix can attend to one another bi-directionally. Tokens outside the prefix use causal attention. qk_ln (`bool`, *optional*, defaults to `False`): Whether to apply layer normalization to the queries and keys in the attention layer. attn_uses_sequence_id (`bool`, *optional*, defaults to `False`)): Whether to restrict attention to tokens that have the same token_type_ids. When the model is in `train` mode, this requires passing an extra *token_type_ids* argument which indicates which sub-sequence each token belongs to. Defaults to `False` meaning any provided *token_type_ids* will be ignored. alibi (`bool`, *optional*, defaults to `True`): Whether or not to use the alibi bias instead of positional embedding. alibi_bias_max (`int`, *optional*, defaults to 8): The maximum value of the alibi bias. multihead_attentionrtorchNFTc  t||_||_||_||_||_||_||_| |_ ||_ | |_ |dvrtd|dS)N)r multiquery_attentionzX`attn_type` has to be either `multihead_attention` or `multiquery_attention`. Received: ) super__init__ attn_type attn_pdrop attn_implclip_qkv softmax_scale prefix_lmattn_uses_sequence_idalibiqk_lnalibi_bias_max ValueError) selfrrrrrrrrrrkwargs __class__s e/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/mpt/configuration_mpt.pyrzMptAttentionConfig.__init__Es "$"  *"%:"  , K K Kvktvv  L Kreturnrc N|||j|fi|\}}|ddkr|d}d|vrMt|dr=|d|jkr,t d|dd|jd|j|fi|S)N model_typempt attn_configzYou are using a model of type z to instantiate a model of type zN. This is not supported for all configurations of models and can yield errors.)_set_token_in_kwargsget_config_dictgethasattrr$loggerwarning from_dict)clspretrained_model_name_or_pathr config_dicts r from_pretrainedz"MptAttentionConfig.from_pretrainedds   (((1c12OZZSYZZ V ??< ( (E 1 1%m4K ; & &73 +E+E &+VbJcgjguJuJu NNr\1Jrr>rrr    s}[33F333r!) r rr NNFFFTr )r"r)__name__ __module__ __qualname____doc__r classmethodr1 __classcell__rs@r r r s$$P(#>444[44444r!r c%eZdZdZdZddddZ d&dedededededededededede de de e ee fd ed!ed"ed#e d$ef$fd% Z xZS)' MptConfiga7 This is the configuration class to store the configuration of a [`MptModel`]. It is used to instantiate a Mpt model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to the Mpt-7b architecture [mosaicml/mpt-7b](https://huggingface.co/mosaicml/mpt-7b). Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: d_model (`int`, *optional*, defaults to 2048): Dimensionality of the embeddings and hidden states. n_heads (`int`, *optional*, defaults to 16): Number of attention heads for each attention layer in the Transformer encoder. n_layers (`int`, *optional*, defaults to 24): Number of hidden layers in the Transformer encoder. expansion_ratio (`int`, *optional*, defaults to 4): The ratio of the up/down scale in the MLP. max_seq_len (`int`, *optional*, defaults to 2048): The maximum sequence length of the model. vocab_size (`int`, *optional*, defaults to 50368): Vocabulary size of the Mpt model. Defines the maximum number of different tokens that can be represented by the `inputs_ids` passed when calling [`MptModel`]. Check [this discussion](https://huggingface.co/bigscience/mpt/discussions/120#633d28389addb8530b406c2a) on how the `vocab_size` has been defined. resid_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability applied to the attention output before combining with residual. layer_norm_epsilon (`float`, *optional*, defaults to 1e-05): The epsilon to use in the layer normalization layers. emb_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability for the embedding layer. learned_pos_emb (`bool`, *optional*, defaults to `True`): Whether to use learned positional embeddings. attn_config (`dict`, *optional*): A dictionary used to configure the model's attention module. init_device (`str`, *optional*, defaults to `"cpu"`): The device to use for parameter initialization. Defined for backward compatibility logit_scale (`float`, *optional*): If not None, scale the logits by this value. no_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in all linear layers. verbose (`int`, *optional*, defaults to 0): The verbosity level to use for logging. Used in the previous versions of MPT models for logging. This argument is deprecated. embedding_fraction (`float`, *optional*, defaults to 1.0): The fraction to scale the gradients of the embedding layer by. norm_type (`str`, *optional*, defaults to `"low_precision_layernorm"`): Type of layer norm to use. All MPT models uses the same layer norm implementation. Defined for backward compatibility. use_cache (`bool`, *optional*, defaults to `False`): Whether or not the model should return the last key/values attentions (not used by all models). initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. Example: ```python >>> from transformers import MptConfig, MptModel >>> # Initializing a Mpt configuration >>> configuration = MptConfig() >>> # Initializing a model (with random weights) from the configuration >>> model = MptModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` r%n_headsd_modeln_layers)num_attention_heads hidden_sizenum_hidden_layersh㈵>TNcpur?low_precision_layernormF{Gz?expansion_ratio max_seq_len vocab_size resid_pdroplayer_norm_epsilon emb_pdroplearned_pos_embr& init_device logit_scaleno_biasverboseembedding_fraction norm_type use_cachec | t|_n.t| trtdi| |_n| |_||_||_||_||_||_||_ ||_ | |_ | |_ | |_ | |_||_||_||_||_||_||_||_t-jdi|dS)N)r r& isinstancedictr<r;r=rLrMrNrOrQrRrSrTrUrVrWrXrPrYinitializer_rangerr)rr<r;r=rLrMrNrOrPrQrRr&rSrTrUrVrWrXrYr^rrs r rzMptConfig.__init__s.  133D    T * * +1@@K@@D  *D     .&$&".&&  "4""4"!2""6"""""r!)rArBrCrDrArErFrGrFTNrHNTrrIrJFrK)r2r3r4r5r$ attribute_mapintfloatboolr strrrrr7r8s@r r:r:vsfEENJ( 'M  $( $*. 37$'2)/#/#/#/# /#  /#  /#/#/#"/#/#/#(/#/#eE3J/0/#/# !/#""#/#$%/#&'/#/#/#/#/#/#/#/#/#/#r!r:N)r5typingrrrconfiguration_utilsrutilsr get_loggerr2r+r r:r[r!r rhs1111111111 333333  H % %U4U4U4U4U4)U4U4U4p~#~#~#~#~# ~#~#~#~#~#r!