g6ddlZddlZddlmZmZddlZddlmcmZ ddl m Z m Z e r2ddl mZmZmZddlmZmZdeejejvZdejd eejejeffd Zd ejd ejd ejdejdef dZdZ ddejdejdejdejdedededeejdeedeededeedefdZdS) N)OptionalTuple)is_flash_attn_2_availableis_flash_attn_greater_or_equal)index_first_axis pad_input unpad_input)flash_attn_funcflash_attn_varlen_func window_sizeattention_maskreturncv|dtj}tj|d}|}tjtj |dtjd}|||fS)aq Retrieves indexing data required to repad unpadded (ragged) tensors. Arguments: attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: indices (`torch.Tensor`): The indices of non-masked tokens from the flattened input sequence. cu_seqlens (`torch.Tensor`): The cumulative sequence lengths, used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). max_seqlen_in_batch (`int`): Maximum sequence length in batch. )dimdtypeF)as_tupler)rr) sumtorchint32nonzeroflattenmaxitemFpadcumsum)rseqlens_in_batchindicesmax_seqlen_in_batch cu_seqlenss g/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/modeling_flash_attention_utils.py_get_unpad_datar$!s &))b )DDmN2244uEEEMMOOG*..005577u|$4!5;OOOQWXXJ  query_layer key_layer value_layer query_lengthcLt|\}}}|j\}} } } t||| z| | |}t||| z| | |}|| kr/t||| zd| |}|} |} |}nu|dkrKd} t j|dztj|j} | dd}|d}n$|dd| df}t||\}}} } ||||| |f| |ffS)a Unpads query, key, and values tensors, using a single dimension for all tokens even though they belong to different batches. This function is used instead of `flash_attn.bert_padding.unpad_input` in order to avoid the recomputation of the same intermediary tensors for query, key, value tensors. Arguments: query_layer (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). attention_mask (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. query_length (`int`): Target length. Return: query_layer (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key_layer (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value_layer (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rr)rdeviceN) r$shaperreshaperarangerr+squeezer )r&r'r(rr) indices_k cu_seqlens_kmax_seqlen_in_batch_k batch_size kv_seq_lennum_key_value_headshead_dim cu_seqlens_qmax_seqlen_in_batch_q indices_qs r# _upad_inputr:<sN6E^5T5T2I|2>(9:FQR]_mFnFnC Y .C  |$  56  r%c|d|d|d}|d|d|d}|d|d|d}|}tj|d|jtj}tj||dktj||jtjf}| dz}||||||f||ffS)aK This function returns necessary arguments to call `flash_attn_varlen_func`. All three query, key, value states will be flattened. Cummulative lengths of each examples in the batch will be extracted from position_ids. NOTE: ideally cummulative lengths should be prepared at the data collator stage Arguments: query (`torch.Tensor`): Query state with padding. Shape: (batch_size, query_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (batch_size, kv_seq_len, num_key_value_heads, head_dim). position_ids (`torch.Tensor`): Boolean or int tensor of shape (batch_size, sequence_length), 1 means valid and 0 means not valid. Return: query (`torch.Tensor`): Query state without padding. Shape: (total_target_length, num_heads, head_dim). key (`torch.Tensor`): Key state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). value (`torch.Tensor`): Value state with padding. Shape: (total_source_length, num_key_value_heads, head_dim). indices_q (`torch.Tensor`): The indices of non-masked tokens from the flattened input target sequence. (cu_seqlens_q, cu_seqlens_k) (`Tuple[int]`): The cumulative sequence lengths for the target (query) and source (key, value), used to index into ragged (unpadded) tensors. `cu_seqlens` shape is (batch_size + 1,). (max_seqlen_in_batch_q, max_seqlen_in_batch_k) (`Tuple[int]`): Maximum sequence length in batch (`max_seqlen_in_batch_q` for the target sequence i.e. query, `max_seqlen_in_batch_k` for the source sequence i.e. key/value). rr)r+rr) viewsizerrr.r+rcattensorr)querykeyvalue position_idsr9 cu_seq_lens max_lengths r#prepare_fa2_from_position_idsrGs<@ JJr5::b>>5::b>> : :E ((2sxx||SXXb\\ 2 2C JJr5::b>>5::b>> : :E''))L \..q11,:MUZU`aaaI) la' ( L**,,\5HPUP[ \ \ \ K!!##a'J 3y; *DzS]F^ __r%F query_states key_states value_states is_causaldropoutrD softmax_scalesliding_windowuse_top_left_masksoftcap deterministicc R| s|} n|o|dk} to| duo|jd| k}|rd| | fini}tdr+| $tjdddk} | |d<| | |d <|[|jd }t |||||\}}}}}}|\}}|\}}t|||f||||||| d |}t||||}n||dkrtj |d d k s| d }t||||\}}}}}}|\}}|\}}t|||f||||||| d |}||d | d| d }nt||||f|| d|}|S)a Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, then computes the attention scores and pad the final attention scores. Args: query_states (`torch.Tensor`): Input query states to be passed to Flash Attention API key_states (`torch.Tensor`): Input key states to be passed to Flash Attention API value_states (`torch.Tensor`): Input value states to be passed to Flash Attention API attention_mask (`torch.Tensor`): The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the position of padding tokens and 1 for the position of non-padding tokens. dropout (`float`): Attention dropout softmax_scale (`float`, *optional*): The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim) use_top_left_mask (`bool`, defaults to `False`): flash_attn<2.1 generates top-left aligned causal mask, while what is needed here is bottom-right alignement, that was made default for flash_attn>=2.1. This attribute is used to handle this difference. softcap (`float`, *optional*): Softcap for the attention logits, used e.g. in gemma2. deterministic (`bool`, *optional*): Determines if the deterministic option introduced in flash_attn>=2.4.1 is enabled. rNr z2.4.1FLASH_ATTENTION_DETERMINISTIC01rRrQr)r7r1 max_seqlen_q max_seqlen_k dropout_prNcausalr)rr<)rNrZ)_flash_supports_window_sizer,rosenvirongetr:r r rdiffallr>rGr=r )rIrJrKrr)rLrMrDrNrOrPrQrRrZuse_sliding_windows flash_kwargsr3r9rE max_seq_lensr7r1r8r2attn_output_unpad attn_outputs r#_flash_attention_forwardrfsP 10|q0 $kd(BkzGWXYGZ]kGkI\cMNN#CDDacL%g..6  JNN+JCPPTWWM(5 _%") Y!!'* Wb *lNLX X T j, ; &1" l7C442     &%..'       19j,WW  !la&7&7L^`AaAaAaefAf@k@k@m@m&7!&&q)) Wt *lLX X T j, ; &1" l7C44,     &%..'      "&&z2{7G7G7K7K[M]M]^`MaMabb & *lG KXag  kw   r%)rHNNNFNN) inspectr\typingrrrtorch.nn.functionalnn functionalrutilsrrflash_attn.bert_paddingrr r flash_attnr r list signature parametersr[Tensorintr$r:rGboolfloatrfr%r#rws`  """""""" LLLLLLLLgPPPPPPPPPPBBBBBBBB"/448I8I/8Z8Z8e3f3f"fELU5<WZ;Z5[6FF|FFL F  FFFFR/`/`/`r+/%)$(##vv,v v,vL v  v  vv5<(vE?vSMvve_vvvvvvvr%