Χg~PddlZddlmZmZddlZddlZddlmcmZ ddl m Z m Z m Z mZmZmZddlmZddlmZejeZ d-d ejd ejd ejd eejfd Zd.dedefdZd.dedefdZ d.dejdedefdZd.defdZd.dedefdZ d.dedefdZ!d.dedefdZ"d.dedefdZ#dZ$dejdeeje%e%ffdZ&dejfdZ'dejde%d e%d!e%dejf d"Z(d#Z)dejd$e%d%edejfd&Z*d'Z+d(ejfd)Z,d*Z-d ejd ejd ejd eejdeejejejeejff d+Z. d/d ejd ejd ejd eejfd,Z/dS)0N)OptionalTuple)can_use_efficient_attentioncan_use_flash_attentionflash_sdp_enabledmath_sdp_enabledmem_efficient_sdp_enabled SDPAParams) SDPBackend) NestedTensorFquerykeyvalue attn_maskc t|tr*t|trt|ts(td|jd|jd|jd|j|jks|j|jkr(td|jd|jd|jd|j|jks|j|jkr(td|jd |jd |jd|d ks0|d ks|d krOtd |d |d|d|j|jks|j|jkr(td|jd|jd|jd|tddS)NzNExpected query, key, and value to be nested tensors, but got query.is_nested: z, key.is_nested: z, and value.is_nested: z instead.zLExpected query, key, and value to have the same dtype, but got query.dtype: z , key.dtype: z, and value.dtype: zSExpected query, key, and value to have the same device type, but got query.device: z, key.device: z, and value.device: zUExpected query, key, and value to all be at least 3 dimensional, but got query.dim: z , key.dim: z and value.dim: z[Expected query, key, and value to all be ragged on the same dimension, but got ragged dims z, z, and z, respectively.zMasks are not yet supported!) isinstancer ValueError is_nesteddtypedevicedim _ragged_idxtorchbool)rrrr dropout_p is_causalscales W/var/www/html/ai-engine/env/lib/python3.11/site-packages/torch/nested/_internal/sdpa.py_validate_sdpa_inputr"s ul + +  #|,,  %..   ?(- ? ?JM- ? ?$)O ? ? ?    {ci5;%+#=#= 7$)K 7 7>Ai 7 7 %  7 7 7    |sz!!U\U\%A%A 9%*\ 9 9AD 9 9!& 9 9 9    yy{{Q#''))a--599;;?? Yyy{{ Y Y'*wwyy Y YBG))++ Y Y Y    CO++u/@EDU/U/U c% c c), c c@E@Q c c c   7888paramsreturnc|jd}|jd}|jd}||ko||kS)Nr)rsizerr)r$debug q_batch_size k_batch_size v_batch_sizes r!_check_batch_size_nestedr,KsX<$$Q''L:??1%%L<$$Q''L < ' HLL,HHr#c"d}|jd}|jd}|jd}||ko||k}|r|dzdkr||ks!|rtd|||dSdS)NrzFor NestedTensor inputs, Flash attention requires q,k,v to have the same last dimension and to be a multiple of 8 and less than or equal to 256. Got Query.size(-1): %d, Key.size(-1): %d, Value.size(-1): %d instead.FT)rr'rrlogwarning)r$r(max_sizequery_size_last key_size_lastvalue_size_lastsame_head_dim_sizes r!!_check_head_dim_size_flash_nestedr8XsHl''++OJOOB''Ml''++O=(O_-O  q A % %  ( (   KKX     u 4r#param param_namect|ts Jd|jdkr|rtd|dS|dkr|rtd|dSdS)Nzparam should be a jagged NTr zMFused kernels do not support ragged num_head_dims, %s has a ragged num_heads.FrzAFused kernels do not support seq_len == 0, %s has a seq len of 0.T)rr rr1r2_get_min_seqlen)r9r:r(s r!:_check_for_seq_len_0_and_consistent_head_dim_nested_helperr=rs e\ * *II,IIII A   KK_   u !##   KKS   u 4r#c t|||}||kr|dks||kr|dks ||kr+|dkr%|r!td|||||||dSdS)Nr zzBoth fused kernels require query, key and value to have broadcastable %s, got Query %s %d, Key %s %d, Value %s %d instead.FT)maxr1r2)q_sizek_sizev_sizer:r(r3s r!_try_broadcast_param_sizerCs666**H 8  ! h  6Q;; h  6Q;;   KKC   u 4r#cX|jjrt|jd|nd}|sdS|jjrt|jd|nd}|sdS|jjrt|jd|nd}|sdS|jd}|jd}|jd}||ko||k}|sU|jjs|jjs |jjr|rtddSt|||d|SdS) NrTFrrr zFBoth fused kernels do not support training with broadcasted NT inputs.z num heads) rrr=rrr' requires_gradr1r2rC) r$r( q_is_safe k_is_safe v_is_safe q_num_heads k_num_heads v_num_headssame_num_headss r!_check_for_seq_len_0_nestedrMs < ! B L'5     u :  B Ju     u < ! B L'5     u,##A&&K*//!$$K,##A&&K K/NK;4NN   L & z' |)    \5( k;    4r#cVtttf}|D]}|||sdSdSNFT)r,r8rMr$r( constraints constraints r!_can_use_flash_sdpa_jaggedrSsH )#K " z&%(( 55  4r#cJttf}|D]}|||sdSdSrO)r,rMrPs r!_can_use_efficient_sdpa_jaggedrUsE #K" z&%(( 55  4r#c|jddrZ|jddr-|jdds|rt ddS|jr|rt ddSdS)Nr zGIf inputs are nested tensors they must be contiguous after transposing.FzENested tensors for query / key are not supported when is_causal=True.T)r transpose is_contiguousrrr1r2r)r$r(s r!_can_use_math_sdpa_jaggedrZs L " "1a ( ( 6 6 8 8 z##Aq))7799 |%%a++99;;    KKY   u    KKW   u 4r#c ts(tsts tjStjtjtjf}t|||||||}|D]} | tjkr,t|rt|rtjcS| tjkr,t|rt|rtjcS| tjkr+trt|rtjcStdt|dt|dtdt|dt|dtdt|dtjS)Nz)Memory efficient kernel not used because:T)r(z(Flash attention kernel not used because:z'Math attention kernel not used because:)rr rr ERRORFLASH_ATTENTIONEFFICIENT_ATTENTIONMATHr rrSrrUrZr1r2) rrrrdropoutr enable_gqaorderingr$backends r!_select_sdp_backendrd s    )++ !""  "&H sE9gy* U UF ' ' j0 0 0&v.. 23Mf3U3U 2!1111 j4 4 4*622 67U88 6"5555 jo % %!! '&?&G&G '!&&&KK;<<<d3333"66666KK:;;;F$////vT2222KK9:::fD1111  r#qkvct|tstd|l|t j|j}| }| j d}n| dt j|j}| d}| }t|d}|||fS)Nzz+_is_computing_meta_flops..qsI   GGu{/? ?      r#F) rjit is_scriptingrrr_python_dispatch _get_current_dispatch_mode_stackany)rtorch_dispatch_mode_stacks r!_is_computing_meta_flopsrjsx 9 ! ! # #  (?(? K ( I I K K "  .       5r#c|jjt|stjs||||fSfd}||||||||fS)a* [Autocasting SDPA for NJT] Normal autocasting doesn't work for NJT+SDPA right now: * NJT intercepts the __torch_function__ call for scaled_dot_product_attention, which happens before we get to any aten ops or dispatcher logic; then the torch_function logic calls into efficient attention or flash attention. So, autocasting on the scaled_dot_product_attention op won't work because we never see that aten op. * If we put autocasting on `_flash_attention_forward`, then we'll get autocasting to run, but the kernel selection logic in torch_function handling (ie. jagged_scaled_dot_product_attention) won't work correctly: the kernel selection logic will run before autocasting, and choose a kernel based on the un-autocasted dtypes; but then autocasting will run and the actual attention computation will happen in a different dtype. An alternative is to just change the backend selection logic for SDPA+NJT to be autocast-aware and rely on autocasting to do the actual conversions for flash attention / efficient attention. However, by manually doing the actual autocast before the backend selection, we ensure that the autocast handling for backend selection doesn't diverge from the autocast handling for the actual dtype conversions. c||Stj}|jjr |j|ks|jtjkr|S||Sr)rget_autocast_dtyperis_floating_pointfloat64ri)r target_dtype device_types r!cvtz_autocast..cvts] 9H/ << * w,&&w%-''HttL!!!r#)rrrris_autocast_enabled)rrrrrrs @r! _autocastrxs4,#K&&,e.G .T.T,c5)++ " " " " " 3u::ss3xxUSS^^ ;;r#c t||||\}}}}t|||||||t|tr*t|trt|tsJddlm}|dkr|dkr|dkr|jdkrtj | | | t|tr| n||||} || | S|j p |j p|j } t|||||||} t|r tj} | tjkr|d} t%|dd} t%|dd}t%|dd}t'||}t)| ||\}}}}}}}}t*jj|||||||||d| \}}}}}|||d |d |d  dd}t5|| S| tjkrt)|||\}}} }}}}}t*jj|d|d| dd|||||t=|| | \}}!}"}#}$}||d|d |d |d  ddS| tj kr?| }%|j!d}&|j!d}'|j"#dd}(|j"#dd})d}*|*|}|*|}|*|}t+j$||||||| d}+ddlm%},|+dd& }+|+'d|&|'}+||+|%|(dn |,|(|)dn |,|) dd}+|+StQd)Nr)nested_view_from_values_offsetsrr )rrrr r/r0F)r rhrr) min_seqlenrrrWrrrc|jdd|jddz }tj|dd}|t |d}tjt |}|dd}|S)Nr r/rWr)r) _offsetsrrXrlsplitlistnestedas_nested_tensorr)jagged_layout_ntrgrX tensor_list strided_nts r! get_strided_layout_nested_tensorzMjagged_scaled_dot_product_attention..get_strided_layout_nested_tensor?s&/36F6OPSQSPS6TTG(8!Q??I#**,,224==a2HHK66tK7H7HIIJ#--a33>>@@J r#)_load_val_from_tensorz=No viable backend for scaled_dot_product_attention was found.))rr"rr $torch.nested._internal.nested_tensorrrrFscaled_dot_product_attentionrlrhrErdrr r]r'rrrropsaten_flash_attention_forwardrXrr^_efficient_attention_forward unsqueezerosqueezer__size_metadata_cacheget"_scaled_dot_product_attention_mathrrrr)-rrrrrrr raroutputcompute_logsumexpbackend_choicer query_padded key_padded value_paddedog_scalerrrrrrrr attention logsumexp philox_seed philox_offsetdebug_attn_maskquery_reshaped key_reshapedvalue_reshaped log_sumexpseedoffset max_seqlen_qrhd1d2min_seqlen_tensormax_seqlen_tensorrattn_outrs- r!#jagged_scaled_dot_product_attentionrs$-UC #J#J E3yUIy)USSS 5,'' sL ) ) ul + + UTTTTT  yy{{Q37799q==UYY[[1__ARVWAWAW/ LLNN JJLL LLNN&0L&I&IX   """y    /.vu}}GGG+Ws/@WEDW( sE9iJN &&4 $3333**R..$UAu55 "3511 $UAu55 #E511 '|Z N N !  ! ( )    IN 3 3 !  ! ( )      4       43  9 %%m4%m4    )Aq//  *)W=== :9 9 9 'uc5 9 9    ( )    IN 7 7  $ $Q ' '  " "1 % %  $ $Q ' '  ( )     NN  8       "/.   a 9 %%m4%m4    )Aq//   :? * *--// [^ [_!155 $  "155 $      1077..s330077; 3y)Ye     ONNNNN%%a++6688??AA==R,,22  %,**+<==%,**+<==     )Aq//  K   r#)NrFN)F)NrFNF)0loggingtypingrrrtorch.nntorch.nn.functionalrrrtorch.backends.cudarrrrr r torch.nn.attentionr nested_tensorr getLogger__name__r1Tensorr"rr,r8strr=rCrMrSrUrZrdrorur}rrrrrrrrr#r!rs"""""""" *)))))''''''g!!)- 00 <0 0 <0 % 0000f I IZ I I I I Ij$616 <%( 4RV.88 8D8888v  z 4    :tj$(%%%P11% cSV@V:W11118el>1 L1"1/21>A1 \1111lFFFR L*-6: \& EL   +< <+< +< <+< % +<  5<u|Xel5K KL +<+<+<+