g* f d Z ddlZddlmZ ddlmZ ej e Z G d de Z dS )zXLNet configuration N )PretrainedConfig)loggingc e Zd ZdZdZdgZdddddZ d fd Zed Z e j
d Z xZ S )! XLNetConfiga
This is the configuration class to store the configuration of a [`XLNetModel`] or a [`TFXLNetModel`]. It is used to
instantiate a XLNet model according to the specified arguments, defining the model architecture. Instantiating a
configuration with the defaults will yield a similar configuration to that of the
[xlnet/xlnet-large-cased](https://huggingface.co/xlnet/xlnet-large-cased) architecture.
Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
documentation from [`PretrainedConfig`] for more information.
Args:
vocab_size (`int`, *optional*, defaults to 32000):
Vocabulary size of the XLNet model. Defines the number of different tokens that can be represented by the
`inputs_ids` passed when calling [`XLNetModel`] or [`TFXLNetModel`].
d_model (`int`, *optional*, defaults to 1024):
Dimensionality of the encoder layers and the pooler layer.
n_layer (`int`, *optional*, defaults to 24):
Number of hidden layers in the Transformer encoder.
n_head (`int`, *optional*, defaults to 16):
Number of attention heads for each attention layer in the Transformer encoder.
d_inner (`int`, *optional*, defaults to 4096):
Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
ff_activation (`str` or `Callable`, *optional*, defaults to `"gelu"`):
The non-linear activation function (function or string) in the If string, `"gelu"`, `"relu"`, `"silu"` and
`"gelu_new"` are supported.
untie_r (`bool`, *optional*, defaults to `True`):
Whether or not to untie relative position biases
attn_type (`str`, *optional*, defaults to `"bi"`):
The attention type used by the model. Set `"bi"` for XLNet, `"uni"` for Transformer-XL.
initializer_range (`float`, *optional*, defaults to 0.02):
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
layer_norm_eps (`float`, *optional*, defaults to 1e-12):
The epsilon used by the layer normalization layers.
dropout (`float`, *optional*, defaults to 0.1):
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
mem_len (`int` or `None`, *optional*):
The number of tokens to cache. The key/value pairs that have already been pre-computed in a previous
forward pass won't be re-computed. See the
[quickstart](https://huggingface.co/transformers/quickstart.html#using-the-past) for more information.
reuse_len (`int`, *optional*):
The number of tokens in the current batch to be cached and reused in the future.
bi_data (`bool`, *optional*, defaults to `False`):
Whether or not to use bidirectional input pipeline. Usually set to `True` during pretraining and `False`
during finetuning.
clamp_len (`int`, *optional*, defaults to -1):
Clamp all relative distances larger than clamp_len. Setting this attribute to -1 means no clamping.
same_length (`bool`, *optional*, defaults to `False`):
Whether or not to use the same attention length for each token.
summary_type (`str`, *optional*, defaults to "last"):
Argument used when doing sequence summary. Used in the sequence classification and multiple choice models.
Has to be one of the following options:
- `"last"`: Take the last token hidden state (like XLNet).
- `"first"`: Take the first token hidden state (like BERT).
- `"mean"`: Take the mean of all tokens hidden states.
- `"cls_index"`: Supply a Tensor of classification token position (like GPT/GPT-2).
- `"attn"`: Not implemented now, use multi-head attention.
summary_use_proj (`bool`, *optional*, defaults to `True`):
Argument used when doing sequence summary. Used in the sequence classification and multiple choice models.
Whether or not to add a projection after the vector extraction.
summary_activation (`str`, *optional*):
Argument used when doing sequence summary. Used in the sequence classification and multiple choice models.
Pass `"tanh"` for a tanh activation to the output, any other value will result in no activation.
summary_proj_to_labels (`boo`, *optional*, defaults to `True`):
Used in the sequence classification and multiple choice models.
Whether the projection outputs should have `config.num_labels` or `config.hidden_size` classes.
summary_last_dropout (`float`, *optional*, defaults to 0.1):
Used in the sequence classification and multiple choice models.
The dropout ratio to be used after the projection and activation.
start_n_top (`int`, *optional*, defaults to 5):
Used in the SQuAD evaluation script.
end_n_top (`int`, *optional*, defaults to 5):
Used in the SQuAD evaluation script.
use_mems_eval (`bool`, *optional*, defaults to `True`):
Whether or not the model should make use of the recurrent memory mechanism in evaluation mode.
use_mems_train (`bool`, *optional*, defaults to `False`):
Whether or not the model should make use of the recurrent memory mechanism in train mode.
For pretraining, it is recommended to set `use_mems_train` to `True`. For fine-tuning, it is recommended to
set `use_mems_train` to `False` as discussed
[here](https://github.com/zihangdai/xlnet/issues/41#issuecomment-505102587). If `use_mems_train` is set to
`True`, one has to make sure that the train batches are correctly pre-processed, *e.g.* `batch_1 = [[This
line is], [This is the]]` and `batch_2 = [[ the first line], [ second line]]` and that all batches are of
equal size.
Examples:
```python
>>> from transformers import XLNetConfig, XLNetModel
>>> # Initializing a XLNet configuration
>>> configuration = XLNetConfig()
>>> # Initializing a model (with random weights) from the configuration
>>> model = XLNetModel(configuration)
>>> # Accessing the model configuration
>>> configuration = model.config
```xlnetmems
vocab_sized_modeln_headn_layer)n_token hidden_sizenum_attention_headsnum_hidden_layers } geluTbi{Gz?-q=皙? NFlasttanh c || _ || _ || _ || _ ||z dk rt d||z d d| v r.| d ||z k r t d| d d||z d ||z | _ || _ || _ || _ || _ | | _
|
| _ | | _ | | _
|
| _ || _ || _ || _ || _ || _ || _ || _ || _ || _ || _ || _ || _ d| v r"t7 j d t: | d }|| _ || _ tA j! d |||d
| d S )
zConstructs XLNetConfig.r z'd_model % n_head' (z) should be equal to 0d_headz
`d_head` (z*) should be equal to `d_model // n_head` () use_cachezlThe `use_cache` argument is deprecated and will be removed in a future version, use `use_mems_eval` instead.) pad_token_id bos_token_id eos_token_idN )"r
r r
r
ValueErrorr$
ff_activationd_inneruntie_r attn_typeinitializer_rangelayer_norm_epsdropoutmem_len reuse_lenbi_data clamp_len same_length summary_typesummary_use_projsummary_activationsummary_last_dropout start_n_top end_n_topr( r' r) warningswarn
FutureWarning
use_mems_evaluse_mems_trainsuper__init__) selfr
r r
r r- r, r. r/ r0 r1 r2 r3 r4 rA rB r5 r6 r7 r8 r9 r: r; r< r= r' r( r) kwargs __class__s i/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/xlnet/configuration_xlnet.pyrD zXLNetConfig.__init__ s @ % V q \Gf4D \ \ \]] ] v h 7f#444 q!1qq]dhn]nqqq ' * "!2 , " "& ( 0 "4 $8!&" ( ( ( & M
#; /M * ,sl \hsslrsssss c L t d| j d dS )N
The model < is one of the few models that has no sequence length limit.r )loggerinfo
model_type)rE s rH max_position_embeddingsz#XLNetConfig.max_position_embeddings s' n nnnooorrI c 2 t d| j d )NrK rL )NotImplementedErrorrO )rE values rH rP z#XLNetConfig.max_position_embeddings s& " f f f f
rI )r r r r r r Tr r r r r NTFFr Fr Tr r r r r r! r" ) __name__
__module__ __qualname____doc__rO keys_to_ignore_at_inference
attribute_maprD propertyrP setter
__classcell__)rG s @rH r r s j j X J#)( ' & M ! 9Pt Pt Pt Pt Pt Ptd X #
$#
rI r )
rW r> configuration_utilsr utilsr
get_loggerrT rM r r* rI rH r` s 3 3 3 3 3 3
H % %R
R
R
R
R
" R
R
R
R
R
rI