ΧgdZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z mZmZmZmZddlZddlmZddlmZmZddlmZdd lmZdd lmZmZdd lm Z gd Z!e eeeeegefZ"e eeeegefZ#Gd de Z$de de$fdZ% dPde deee&dfdeee&dfdee&eee&fde'f dZ(dedededededef dZ)dededededef d Z*d!Z+d"Z,d#ed$efd%Z-defd&Z.d#ed$efd'Z/Gd(d)Z0d*Z1d+Z2e+e+dfd,ed-e'deeeeffd.Z3d/e#de#fd0Z4d/e#de#fd1Z5e+e+fdefd2Z6e+e+fd3eeeefd4ee d5e&d6e&de0f d7Z7 dQd9ee"e#fd:ee&d;ee&de8d?e'defd@Z9d4e d:e&d;e&de8d5e&d6e&fdAZ:d8e+dfd4e#d:ee&d;ee&de8dBee&ee&e&ffde0fdCZ;dDedEede0fdFZ dRdDedEedGedKee"d3ee0dLee?dMe'dHe'dNee e8e fdeeeeefffdOZdS)SzIThis module implements the user facing API for flex_attention in PyTorch.N) nullcontext)Enum)AnyCallableDictListOptionalTupleUnion)Tensor)flex_attentionTransformGetItemToIndex)_set_compilation_env)-_temp_remove_pre_dispatch_torch_function_mode)_supported_head_dim_validate_sdpa_input) tree_map_only) BlockMaskr create_block_mask create_maskor_masks and_masks noop_maskceZdZdZdZdZdZdS)_ModificationTypezEnum for the type of modification function. - SCORE_MOD: score_mod function which accepts a score as the first argument - mask_mod: mask function which does not accept a score and is only used for generating block mask N)__name__ __module__ __qualname____doc__ SCORE_MODMASK_MODUNKNOWN]/var/www/html/ai-engine/env/lib/python3.11/site-packages/torch/nn/attention/flex_attention.pyrr*s) IHGGGr'rfnreturnctdtj|jD}|dks|dksJ|dkr t jS|dkr t jSt jS)a\Get the type of modification function. This function inspects the number of positional arguments of the function to determine the type of modification function. If the function has 5 positional arguments, it is considered as a score_mod function. If the function has 4 positional arguments, it is considered as a mask function. c3NK|] }|jtjjkdV!dS)rN)defaultinspect Parameterempty).0params r( z _get_mod_type..=sA  =G-3 3 3 3 3 3 3r') sumr. signature parametersvaluesrr#r$r%)r)num_positional_argss r( _get_mod_typer;6s&r**5<<>> ! # #':a'?'?'?'?a **  ! ! )) ((r'r&Fprefix.suffixout_dims group_dimcrg}gd}|r|dgz }|dgz }|D]}tj|||z|z|} |S)aUsed to vmap both score_mods and mask_mods over 4-dimensional/5-dimension inputs. Mapping over the [b, hq, q_idx, kv_idx] or [b, hkv, g, q_idx, kv_idx] dimensions. Args: fn (callable): The function to vmap. prefix (tuple): The prefix of the vmap. For score mod functions, this should be set to (0,). For mask_mods = () suffix (tuple): We need to add (0,) if gradOut is being mapped over, and (None,) * len(other_buffers). out_dims (tuple): For forward cases, keep this as the default 0 since we are only returning 1 output. For backwards, the joint graph returns grads for B, H, Q_idx, KV_idx and other_buffers, so we set this to (0, None, None, None, None) + (None,) * len(other_buffers). Returns: callable: The vmapped function. ))NNNr)NNrNNrNNrA)rNNN)in_dimsr>)torchvmap)r)r<r=r>r? dimensionsdimss r(_vmap_for_bhqkvrGLs2OQJJ   !  JOO ZFTMF$:X N N N Ir'scorebatchheadtoken_qtoken_kvc|SNr&)rHrIrJrKrLs r( _identityrOzs  Lr'cP|dtj|jS)zReturns a noop mask_modr&)sizedtypedevice)new_onesrCboolrS)rIrJrKrLs r(rrs  >>rEL> I IIr'i@num_blocks_in_row col_indicesc |jd |jd|jdd}|j fd}|}tt|D]}t j|d}|||}|S)Nc| dztj}tj tjd}tjtj}||dk}tj||}d|||f<|dddfS)NrrRrRrSr[) new_zerosrCint32arangeint unsqueezewhere contiguous) kv_num_blocks kv_indices dense_mask row_indices col_range index_mask valid_indicesrSnum_colsnum_rowss r(create_dense_onez+_ordered_to_dense..create_dense_ones))(HqL )TT l859VLLLVV   L6JJJ !8!8!OOO  0+ > >C Jr'cb|tj}|d}tj|ddd}|tj|tjfS)Nr]r[dimT)rz descendingstable)torCr`r6argsortre)rhrWrXs r(_dense_to_orderedrsU[11J"2..- tDQQQKU[))4466u{##..00 r'cht||}t|ddS)NrZr[)rwr transpose)rWrXdenses r(_transpose_orderedrs. / = =E U__R44 5 55r'c^eZdZUdZeed<eed<eeed<eeed<eeed<eeed<eeed<eeed <eeefed <e ed <dededeedeedeedeedeed eed eeefd e fd Z e d d e d fdededeedeed e eeeeffd ee f dZddefdZdZd dZdZedZdZdefdZdefdZd!dZde ejefddfdZd S)"ra< BlockMask is our format for representing a block-sparse attention mask. It is somewhat of a cross in-between BCSR and a non-sparse format. Basics ------ A block-sparse mask means that instead of representing the sparsity of individual elements in the mask, a KV_BLOCK_SIZE x Q_BLOCK_SIZE block is considered sparse only if every element within that block is sparse. This aligns well with hardware, which generally expects to perform contiguous loads and computation. This format is primarily optimized for 1. simplicity, and 2. kernel efficiency. Notably, it is *not* optimized for size, as this mask is always reduced by a factor of KV_BLOCK_SIZE * Q_BLOCK_SIZE. If the size is a concern, the tensors can be reduced in size by increasing the block size. The essentials of our format are: num_blocks_in_row: Tensor[ROWS]: Describes the number of blocks present in each row. col_indices: Tensor[ROWS, MAX_BLOCKS_IN_COL]: `col_indices[i]` is the sequence of block positions for row i. The values of this row after `col_indices[i][num_blocks_in_row[i]]` are undefined. For example, to reconstruct the original tensor from this format: .. code-block:: python dense_mask = torch.zeros(ROWS, COLS) for row in range(ROWS): for block_idx in range(num_blocks_in_row[row]): dense_mask[row, col_indices[row, block_idx]] = 1 Notably, this format makes it easier to implement a reduction along the *rows* of the mask. Details ------- The basics of our format require only kv_num_blocks and kv_indices. But, we have up to 8 tensors on this object. This represents 4 pairs: 1. (kv_num_blocks, kv_indices): Used for the forwards pass of attention, as we reduce along the KV dimension. 2. [OPTIONAL] (full_kv_num_blocks, full_kv_indices): This is optional and purely an optimization. As it turns out, applying masking to every block is quite expensive! If we specifically know which blocks are "full" and don't require masking at all, then we can skip applying mask_mod to these blocks. This requires the user to split out a separate mask_mod from the score_mod. For causal masks, this is about a 15% speedup. 3. [GENERATED] (q_num_blocks, q_indices): Required for the backwards pass, as computing dKV requires iterating along the mask along the Q dimension. These are autogenerated from 1. 4. [GENERATED] (full_q_num_blocks, full_q_indices): Same as above, but for the backwards pass. These are autogenerated from 2. rfrgfull_kv_num_blocksfull_kv_indices q_num_blocks q_indicesfull_q_num_blocksfull_q_indices BLOCK_SIZEmask_modc |dkrtd| Jd| Jd| Jd| Jd|du|duks Jd|du|duks Jd||_||_||_||_||_||_||_||_ | |_ | |_ dS) Nr)BlockMask must have at least 2 dimensionszkv_num_blocks must be providedzkv_indices must be providedzq_num_blocks must be providedzq_indices must be providedGfull_kv_num_blocks and full_kv_indices must be both provided or omittedzEfull_q_num_blocks and full_q_indices must be both provided or omitted) rz RuntimeErrorrfrgrrrrrrrr) selfrfrgrrrrrrrrs r(__init__zBlockMask.__init__s2 >>  a  JKK K((*J(((%%'D%%%'')H'''$$&B$$$"d* t #   T   "T) d "   R   +$"4.("!2,$  r'Nc N|dkrtd|du|duks Jdt||\}}||Jt||\} } nd\} } t|tr||f}||nt }|||||||| | || S)a Creates a BlockMask instance from key-value block information. Args: kv_num_blocks (Tensor): Number of kv_blocks in each Q_BLOCK_SIZE row tile. kv_indices (Tensor): Indices of key-value blocks in each Q_BLOCK_SIZE row tile. full_kv_num_blocks (Optional[Tensor]): Number of full kv_blocks in each Q_BLOCK_SIZE row tile. full_kv_indices (Optional[Tensor]): Indices of full key-value blocks in each Q_BLOCK_SIZE row tile. BLOCK_SIZE (Union[int, Tuple[int, int]]): Size of KV_BLOCK_SIZE x Q_BLOCK_SIZE tiles. mask_mod (Optional[Callable]): Function to modify the mask. Returns: BlockMask: Instance with full Q information generated via _transposed_ordered Raises: RuntimeError: If kv_indices has < 2 dimensions. AssertionError: If only one of full_kv_* args is provided. rrNrNN) rfrgrrrrrrrr)rzrr isinstancerbr) clsrfrgrrrrrrrrs r(from_kv_blockszBlockMask.from_kv_blocks+s 8 >>  a  JKK K"d* t #   T   #5]J"O"O i  )"...0B"O11 - ~~1; - ~ j# & & 2$j1J'388s'!1+%/)!    r'Tflattenc|r|jd|jdfn|jf}|j|j|j|j|j|j|j|jg||j RS)z Returns a tuple of the attributes of the BlockMask. Args: flatten (bool): If True, it will flatten the tuple of (KV_BLOCK_SIZE, Q_BLOCK_SIZE) rr) rrfrgrrrrrrr)rr block_sizes r(as_tuplezBlockMask.as_tuplejs9@ WT_Q !3 4 4doEW    O  #    N  "       M   r'cd|jd|dd}|}||z }|dz }|S)NzBlockMask(shape=z , sparsity=.2fz%,  ))rpsparsity to_stringstrip)rsmask_strs r(__str__zBlockMask.__str__sY Ptz P Pdmmoo P P P P>>##))++ X  U r'r*c|j|}|j|}|j$|jJ|j|}|j|}nd}d}t|||||jdS)a Returns a new BlockMask instance by getting the mask for the given index position. Args: index: Index to apply to all attributes. Example Usage: .. code-block:: python def causal_mask(b, h, q_idx, kv_idx): return q_idx >= kv_idx block_mask = create_block_mask(causal_mask, 4, 2, 512, 512, device="cuda") assert block_mask.kv_num_blocks.shape == (4,2,4) assert block_mask.kv_indices.shape == (4,2,4,4) # Index on batch dimension new_block_mask = block_mask[0] assert new_block_mask.kv_num_blocks.shape == (2,4) assert new_block_mask.kv_indices.shape == (2,4,4) # Index on batch and head dimension new_block_mask = block_mask[0, 1] assert new_block_mask.kv_num_blocks.shape == (4,) assert new_block_mask.kv_indices.shape == (4,4) # slicing on batch and head dimension new_block_mask = block_mask[0:2, 1:2] assert new_block_mask.kv_num_blocks.shape == (2,1,4) assert new_block_mask.kv_indices.shape == (2,1,4,4) # slicing on batch, head, and query dimension new_block_mask = block_mask[0:2, 1:2, torch.tensor([1], dtype=torch.int32)] assert new_block_mask.kv_num_blocks.shape == (2,1,1) assert new_block_mask.kv_indices.shape == (2,1,1,4) Nrr)rfrgrrrrr)rindexnew_kv_num_blocksnew_kv_indicesnew_full_kv_num_blocksnew_full_kv_indicess r( __getitem__zBlockMask.__getitem__sJ!.u5/  " .'333%)%.shape_or_nonesm177 5r'zBlockMask( kv_num_blocks=z, kv_indices=z, full_kv_num_blocks=z, full_kv_indices=z, q_num_blocks=z, q_indices=z, full_q_num_blocks=z, full_q_indices=z, BLOCK_SIZE=z , shape=z, sparsity=rz%, mask_mod=rr)r rCr rfrprgrrrrrrrrhasattrrr)rrs r(__repr__zBlockMask.__repr__si 6Xel3 6 6 6 6 !%!3!9  "o3  '4mD4K&M&M  $1=1E#F#F   !. d.? @ @   +]4>::  &3]43I%J%J  #0-0C"D"D  #o    !MMOO 5  7>dmZ6X6XkDM22^b^k    r'c|jj^}}}|jjd|jdz}|jjd|jdz}t|||gzS)zReturns the shape of the mask.rZrr[r)rgrprtuple)rrsq_lengthru kv_lengths r(rpzBlockMask.shapesc$(?#8 Xq?(,tq/AAO)"-0BB Z8Y"77888r'c,|j}d}||S)zIReturns the number of elements (not accounting for sparsity) in the mask.cBtjtj|dSNr) functoolsreduceoperatormul)xss r(_prodzBlockMask.numel.._prods#HL"a88 8r'r)rrprs r(numelzBlockMask.numels(  9 9 9uU||r'c|}|j}|j||jz }||jdz|jdz}||z }dd|z zS)zEComputes the percentage of blocks that are sparse (i.e. not computed)Nrrd)rrfr6ritemr)r total_sizecomputed_blocks computed_size dense_ratios r(rzBlockMask.sparsitysZZ\\ ,0022  " . t6::<< .create_block_vis..summarize_sectionsH$]]__113388:: ?? 51__3 5r'c||dz z|zSrr&)abs r(cdivz;BlockMask.to_string..create_block_vis..cdivsQU ))r'rrr)appendjoinreversedmaxrq) batch_idx descriptorsvisrrrow_stepcol_steprccur_maskidxcharrhmax_colsmax_rowsrmrns r(create_block_visz-BlockMask.to_string..create_block_vis s9K   )~ . . .))H[1122T9C ! ! ! * * *1dd8X6677H1dd8X6677H1h11  q(H55$$A)H(11#+C=,, Q\!11q8|3C!CDD4!8OCCt Jr'c,g|]}t|Sr&)rq)r1is r( z'BlockMask.to_string../s===Qa===r'z...z3To print out more, set BlockMask.to_string(limit=N)zNYou can also index (BlockMask[batch, head]) to choose a specific batch or headr) rrprrb enumerate itertoolsproductrr)r grid_sizelimitrsr total_visrr block_visrhrrrmrns @@@@@r(rzBlockMask.to_stringsZ ]]__ *4*:'Xx i % % + H HH "__HHH!* Hh         D '  ==*=== >  ( (NCe||  '''  !VWWW  d(()4I   Y ' ' ' 'yy###r'rSczttjfd|d}t |S)aMoves the BlockMask to the specified device. Args: device (torch.device or str): The target device to move the BlockMask to. Can be a torch.device object or a string (e.g., 'cpu', 'cuda:0'). Returns: BlockMask: A new BlockMask instance with all tensor components moved to the specified device. Note: This method does not modify the original BlockMask in-place. Instead, it returns a new BlockMask instance where invidual tensor attributes may or may not be moved to the specified device, depending on their current device placement. c.|SrN)r})rrSs r(zBlockMask.to..Psadd6llr'F)r)rrCr rr)rrSmapped_attributess ` r(r}z BlockMask.to=sE"* L " " " " MM%M ( (   +,,r')T)r*r)rr5)rr r!r"r __annotations__r r rb_mask_mod_signaturer classmethod_DEFAULT_SPARSE_BLOCK_SIZEr rrUrrrrpropertyrprrrrrrCrSstrr}r&r'r(rrs::v ((((f%%%%6""""''''V$$$$c3h!!!!#!#!#!%V, #! "&) #! v& #!F##!$F+#!!(#!#s(O#!&#!#!#!#!J 04,02L26< < < < %V, < "&) < #uS#X./ < ./< < < [< |      05 5 5 5 n   *99X9 '% ' ' ' '&@$@$@$@$D-u|S01-k------r'rc||kr-|d}||k-|S)Nr)rzrc)rrzs r(_broadcast_to_dimrVs: %%''C-- KKNN %%''C-- Hr'c||zdz |z|zSrr&)rmultiples r(_round_up_to_multipler\s L1  )H 44r'maskseparate_full_blockscH|jtjksJt|d}|j\}}}}||zdksJ||zdksJ|||||z|||z|}|dddddd}|ddg }|r\||z} || k} |dk|| kz} | tj } | tj } | | fS|dk} | tj } | dfS) Nr5rrrrr4rZr[ryr]) rRrCrUrrpviewpermuter6r}int8) r KV_BLOCK_SIZE Q_BLOCK_SIZErBHQKVmask_block_sumfull_block_sum full_blockspartial_blockss r(_convert_mask_to_block_maskr`sg : # # # # T1 % %D*KAq!R | q  " " " " 99 1a<r]/BM  D << 1aAq  DXX HN $% 5$6 (1,.1PQ'***<<!nn5:n66 {**'!+'***<<t##r' mask_modschtdDstdfd}|S)z9Returns a mask_mod that's the union of provided mask_modsc34K|]}t|VdSrNcallabler1args r(r3zor_masks..(22x}}222222r')All inputs should be callable mask_mods: cv|dtj}D]}||||||z}|SNr&r])r_rCrUrhq_idxkv_idxresultrrs r(or_maskzor_masks..or_masksJRuz22 8 8Ddd1a777FF r'allr)rr's` r(rrsZ 22 222 2 2TRyRRSSS Nr'chtdDstdfd}|S)z@Returns a mask_mod that's the intersection of provided mask_modsc34K|]}t|VdSrNrrs r(r3zand_masks..rr'rcv|dtj}D]}||||||z}|Sr!)rTrCrUr"s r(and_maskzand_masks..and_masksJBej11 8 8Ddd1a777FF r'r()rr-s` r(rrsZ 22 222 2 2TRyRRSSS Or'c|dksJ|j\}}}}|j||g|jR}|dddddd||||z||z}|S)Nr5rrrr4r)rzrpexpandr reshape) block_maskr r rrrrs r(_convert_block_mask_to_maskr2s >>  q "KAq!R""<RAQRRRJ##Aq!Q155== 1a,] 2J r'r1rr r c|\}}t|}|t|}nd}t|d|d|d|d||f|S)Nrrrr)rrr)r1rr r rr partial_bmfull_bms r($_create_sparse_block_from_block_maskr6sy #-NK">22J#K00  # #1 1   !<0 $  r'cudamod_fnrrQ_LENKV_LENrS_compilec |d}|d}tjd||}tjd||}tjd||} tjd||} |rt} nt} t |} | 5| t jkro|} t| d} | tj|||||||| | }tj tj |dd}|cdddS| t j kr/|}t|d }|||| | }|cdddSt#1swxYwYdS) aThis function creates a mask tensor from a mod_fn function. Args: mod_fn (Union[_score_mod_signature, _mask_mod_signature]): Function to modify attention scores. B (int): Batch size. H (int): Number of query heads. Q_LEN (int): Sequence length of query. KV_LEN (int): Sequence length of key/value. device (str): Device to run the mask creation on. Returns: mask (Tensor): A mask tensor with shape (B, H, M, N). Nrr)rS)r)r<FTr&) rCrarrr;rr#rGzerosrdisneginfr$AssertionError)r8rrr9r:rSr;rr#mnctxmod_type score_modrvrrs r(rrs, y y  Q&)))A Q&)))A Qf---A Qv...A(mm%''V$$H ! ! (2 2 2I' $???I)EK1eVFKKKQPQSTVWXXC;u~c22E4@@D ! ! ! ! ! ! ! !*3 3 3H&x;;;H8Aq!Q''D ! ! ! ! ! ! ! !!  ! ! ! ! ! ! ! ! ! !sA3E2EEE"Ec bt||||||d}t|||d\} } | | fS)zWork around for being unable to instantiate __torch_function__ mode under compile. `create_block_mask` will compile this inner function and wrap the call to this with the __torch_function__ mode. T)r;)r r r)rr) rrrr9r:rSr r  mask_tensorpartial_block_maskfull_block_masks r(_create_block_mask_innerrIsSh1eVVdSSSK*E#!! +++'  ..r'rc t|}|tjks Jd|t} |d}|d}t |t r|} |} n|\} } |dkr|} nt || }t || }|rtj| dd} t5| ||||||| | \} } t| | f|}dddn #1swxYwY|S)a%This function creates a block mask tuple from a mask_mod function. Args: mask_mod (Callable): mask_mod function. This is a callable that defines the masking pattern for the attention mechanism. It takes four arguments: b (batch size), h (number of heads), q_idx (query index), and kv_idx (key/value index). It should return a boolean tensor indicating which attention connections are allowed (True) or masked out (False). B (int): Batch size. H (int): Number of query heads. Q_LEN (int): Sequence length of query. KV_LEN (int): Sequence length of key/value. device (str): Device to run the mask creation on. KV_BLOCK_SIZE (int): Block size of block mask for each query. Q_BLOCK_SIZE (int): Block size of block mask for each key/value. _compile (bool): Whether to compile the mask creation. Returns: BlockMask: A BlockMask object that contains the block mask information. Example Usage: .. code-block:: python def causal_mask(b, h, q_idx, kv_idx): return q_idx >= kv_idx block_mask = create_block_mask(causal_mask, 1, 1, 8192, 8192, device="cuda") query = torch.randn(1, 1, 8192, 64, device="cuda", dtype=torch.float16) key = torch.randn(1, 1, 8192, 64, device="cuda", dtype=torch.float16) value = torch.randn(1, 1, 8192, 64, device="cuda", dtype=torch.float16) output = flex_attention(query, key, value, block_mask=block_mask) z4create-block_mask requires a mask_mod function! Got NrrVTF) fullgraphdynamic) r;rr$rIrrbrrCcompilerr6)rrrr9r:rSrr;rC inner_funcr r rGrHr1s r(rrstTX&&H%....HhHH /..)Jy y *c""1! " &0# m s{{ %e\:: "6= 9 9FN]:uMMM " "  .8j aE66=,/ / +O:  18                  s+(CC#&C#querykeyc|j}ttjgdtj|tjgdtj|tS)zDefault block mask for flex attention. If users don't specify any block sparse mask info, we create this empty block sparse mask. Which creates a BlockMask with 1 block that is the full length of the query and key tensors. )rrrr^)rrrr)rfrgr)rSrrrConesr`r=_LARGE_SPARSE_BLOCK_SIZE)rOrPrSs r(_create_empty_block_maskrT[s_ \F  # #j%+fMMM;|||5;vNNN+ $  r'value return_lsec|int|}|dd|ddd|vsJd|d<|s/|jp |jp|j}|otj}||d<|S)NROWS_GUARANTEED_SAFEF PRESCALE_QKOUTPUT_LOGSUMEXPT)dict setdefault requires_gradrCis_grad_enabled)rOrPrUrVkernel_optionsany_inputs_require_gradoutput_logsumexps r(_apply_kernel_optionsrbis*1RRtN7K7KN4e<<<mU333 ^ 3 3 3 3)-N%& >   K3#4 K8K 3Nu7L7N7N-=)* r'c|d|dkr Tensor: Where: - ``score``: A scalar tensor representing the attention score, with the same data type and device as the query, key, and value tensors. - ``batch``, ``head``, ``q_idx``, ``k_idx``: Scalar tensors indicating the batch index, query head index, query index, and key/value index, respectively. These should have the ``torch.int`` data type and be located on the same device as the score tensor. Args: query (Tensor): Query tensor; shape :math:`(B, Hq, L, E)`. key (Tensor): Key tensor; shape :math:`(B, Hkv, S, E)`. value (Tensor): Value tensor; shape :math:`(B, Hkv, S, Ev)`. score_mod (Optional[Callable]): Function to modify attention scores. By default no score_mod is applied. block_mask (Optional[BlockMask]): BlockMask object that controls the blocksparsity pattern of the attention. scale (Optional[float]): Scaling factor applied prior to softmax. If none, the default value is set to :math:`\frac{1}{\sqrt{E}}`. enable_gqa (bool): If set to True, enables Grouped Query Attention (GQA) and broadcasts key/value heads to query heads. return_lse (bool): Whether to return the logsumexp of the attention scores. Default is False. kernel_options (Optional[Dict[str, Any]]): Options to pass into the Triton kernels. Returns: output (Tensor): Attention output; shape :math:`(B, Hq, L, Ev)`. Shape legend: - :math:`N: \text{Batch size} ... : \text{Any number of other batch dimensions (optional)}` - :math:`S: \text{Source sequence length}` - :math:`L: \text{Target sequence length}` - :math:`E: \text{Embedding dimension of the query and key}` - :math:`Ev: \text{Embedding dimension of the value}` .. warning:: `torch.nn.attention.flex_attention` is a prototype feature in PyTorch. Please look forward to a more stable implementation in a future version of PyTorch. Read more about feature classification at: https://pytorch.org/blog/pytorch-feature-classification-changes/#prototype r5z-NYI: query, key, and value must be 4D tensorszGExpect query and key/value to have the same number of heads but got Hq=z and Hkv=z&. Try setting enable_gqa=True for GQA.rrzMExpect number of query heads to be a multiple of kv heads for GQA but got Hq=rdNg?r[rz&flex_attention requires dynamo supportct|i|SrN)flex_attention_hop)argskwargss r(_flex_attention_hop_wrapperz3flex_attention.._flex_attention_hop_wrappers!426222r'eagerT)backendrK)rrfrzNotImplementedErrorrQrerOrTmathsqrtrbrCcompileris_dynamo_compiling_dynamo mark_staticrlrlogis_dynamo_supportedrrutilsdisable_cache_limitrrM)rOrPrUrDr1rgrhrVr_HqHkvrrvlseros r(r r s @U+++sE*** yy{{a37799>>UYY[[A-=-=!"QRRR  EJJrNNchhrll:: 4**R.. 4 43688B<< 4 4 4    ZZ]]hhqkk 8q==2 22+.222   -eS99  }di 2///*    N ~))++ e$ - -A M % %a , , , M % %a , , , ,% 3y**=*=*?*?  S  dhqkk)) )J = , , . .ECDDD333    ] 4 4 6 6  >@@  5=/D''))"  Sdhqkk 11              "                                      ss4$M8M 'AM= M  M8"M# M / M8M M M M  M8 M$ $M8'M$ (M88M<?M<)r&rF)r7F)NNNFFN)@r"rr.rrsr contextlibrenumrtypingrrrrr r r rCr &torch._higher_order_ops.flex_attentionr rlrtorch._higher_order_ops.utilsr"torch.fx.experimental.proxy_tensorrtorch.nn.attention._utilsrrtorch.utils._pytreer__all___score_mod_signaturerrr;rbrUrGrOrrrSrwrrrrrrrrr2r6rrrIrrTrbrfrr&r'r(rsPO """"""DDDDDDDDDDDDDDDDDD ?>>>>>POOOOOOO------     H& PQ?GH        )h)#4))))2)+01 +++ (3-$ %+ (3-$ %+Chsm,,- +  ++++\        J J JJ J  JJJJ!"f<U6&6v6666 T-T-T-T-T-T-T-T-n    555 -+!& $$ $ $  68F# #$ $$$$B , 1D     - 2E     -+      "42 fhv../x    <3!3! &(;; <3!}3!}3!  3!  3!  3!3! 3!3!3!3!l// / /  /  /  //////<.H GG!G}G}G  G  G  Gc5c?*+GGGGGT F  I     '-;?* v F 6    415&*!/3HH H H H,- H # H E? HHHT#s(^,H 65( ()HHHHHHr'