NgddlmZddlmZmZmZmZddlZddl m Z ddl m Z ddl mZmZddlmZmZmZmZddlmZeeZGd d ZGd d e ZdS) ) getLogger)ListOptionalTupleUnionN)Fusion)AttentionMaskFormat) FusionUtils NumpyHelper) NodeProto TensorProtohelper numpy_helper) OnnxModelcFeZdZdZdefdZdefdZdZdZ de d e fd Z d S) AttentionMask: Fuse Attention subgraph into one Attention node. modelc||_i|_i|_t||_t j|_||_ dSN) r mask_indice mask_castedr utilsr MaskIndexEnd mask_formatget_opset_version opset_version)selfrs e/var/www/html/ai-engine/env/lib/python3.11/site-packages/onnxruntime/transformers/fusion_attention.py__init__zAttentionMask.__init__sM  '' .;"4466rc||_dSr)r)rrs rset_mask_formatzAttentionMask.set_mask_format!s&r!cR||jvr||j|ksJ||j|<dSr)r)rmask mask_indexs rset_mask_indicezAttentionMask.set_mask_indice$s< 4# # #!1$!77777!+r!cxt|jdksJtt|jSNr)lenrnextiter)rs rget_first_maskzAttentionMask.get_first_mask)s74#$$q((((D)**+++r!inputreturnc ^|jtjkrdS||jvr |j|S|j|r|j|\}}n|j|\}}d}|r ||j |<|jtj kr ||j|<|S|j d}|j dkrwtjd|g|g|j dd}|jtjddgtjd d gnd }|j|=|jtj|t*jdgdgd tjd||g|g|j dd}|jtjd d g|j|||j|<|S)NTr& ReduceSum MaskReduceSuminputsoutputsnameaxeskeepdimsrort_const_1_reduce_sum_axesFr7 data_typedimsvalsraw)rr NoMaskrrfind_graph_inputrcast_graph_input_to_int32cast_input_to_int32rrcreate_node_namerr make_node attributeextendmake_attributeget_initializeradd_initializer make_tensorr INT64add_node)rr.casted input_name cast_node output_namemask_index_node axes_names r process_maskzAttentionMask.process_mask-sW  29 9 94 D$ $ $#E* * : & &u - - !%!E!Ee!L!L FJJ$(J$B$B5$I$I !J F  1&0D U #  2@ @ @&0D U # j11,??   " "$."|$ Z00oNN O  % , ,f.CFQC.P.PRXRghrtuRvRv-w x x x x6Iz)))44< **&&"-"3SS! %."I.$ Z00oNN O  % , ,f.CJPQ.R.R-S T T T O,,,"-r!N) __name__ __module__ __qualname____doc__rr r r#r'r-strrUr!rrrs7i7777'+>'''',,, ,,,8#8#888888r!rc)eZdZdZdddddgfdededed eed ed ed e e ffd Z de de eeffdZde de eeffdZde fdZde fdZde de de fdZde de de e ffdZde de de fdZde de fdZde d ee dfd!ee dfd"e dee dff d#Zd$e d%e d&e de d ee dfd!ee dfdedee dffd'Z d6d$e d%ee e dfd&ee e dfde d ee dfd!ee dfdeded)e d*e de de de d+e d,e d-edee dff"d.Z d7d/e d$e d%e d&e de d e d!e deded0e d)e d1e de de d+e d,e d2eed3edee dff&d4Zd5ZxZS)8FusionAttentionrNFSkipLayerNormalizationLayerNormalizationr hidden_size num_headsattention_maskuse_multi_head_attention!disable_multi_head_attention_biassearch_op_typesc|rdnd}t|||||_||_|r|nt ||_||_||_d|_d|_ d|_ d|_ d|_ dS)NMultiHeadAttention AttentionT) superr r`rarrbrcrdmask_filter_valuenum_heads_warninghidden_size_warning shape_infershape_infer_done) rrr`rarbrcrdreattention_op_name __class__s rr zFusionAttention.__init__ms5M]00R]  1?CCC&"0>XnnMRWDXDX(@%1R.!%"&#'  $r!concatr/ct|jdkr|j|jd}|j|jd}t |t jrI|jdkr>t |t jr$|jdkr|d|d|dzfS|j|j fS)aU Detect num_heads and hidden_size from Concat node in the following subgraph: SkipLayerNormalization or EmbedLayerNormalization / | MatMul Shape | | Add Gather(indices=0) | | | Unsqueeze | | | Concat (*, -1, 12, 64) | / Reshape | Transpose r9r) r*r.rget_constant_value isinstancenpndarraysizerar`)rrqra head_sizes r)get_num_heads_and_hidden_size_from_concatz9FusionAttention.get_num_heads_and_hidden_size_from_concats$ v|   ! ! 55fl1oFFI 55fl1oFFI9bj11 ANa''y"*55(Na'' |Yq\IaL%@@@~t///r! reshape_qcf|j|jd}|s|j|d}| |jdkr||St |jdd|j|j fStj |}t|dks|ddks |ddkr,t d |d |j|j fS|d}|d}||z}|jdkr?||jkr4|j r-t d |jd |d d|_ |j dkr?||j kr4|jr-t d|j d |d d|_||fS)zDetect num_heads and hidden_size from a reshape node. Args: reshape_q (NodeProto): reshape node for Q Returns: Tuple[int, int]: num_heads and hidden_size r9NConcatz is not initializer.rsrtrruzq_shape_value=z7. Expected value are like [0, 0, num_heads, head_size].z--num_heads is z. Detected value is z. Using detected value.Fz--hidden_size is )rrJr. get_parentop_typer|loggerdebugrar`r to_arrayr*rkwarningrl)rr}q_shaperq q_shape_valuerar{r`s rget_num_heads_and_hidden_sizez-FusionAttention.get_num_heads_and_hidden_sizes*,,Y_Q-?@@ ?Z**9a88F!fn&@&@EEfMMM LLIOA.DDD E E E>4#33 3#,W55 }   " "}Q'71'<'< a@PTU@U@U LLp-ppp q q q>4#33 3!!$ !!$ )+ >A  )t~"="=% /wwwU^wwwxxx).&  a  K43C$C$C' 1r(8rrkrrr,1(+%%r!add_qkc|js'|jd|_d|_|jdS|j|jd}|j|jd}|| t d|ddS||kr t d|ddS|jdS) NT)updaterr9zone of the inputs of z is Nonezthe shape of two inputs of z is not same)rnrinfer_runtime_shapermget_edge_shaper.rr)rr input_0_shape input_1_shapes rget_add_qk_strzFusionAttention.get_add_qk_strs$ )#z==T=JJD $(D !   #4(77 QHH (77 QHH  M$9 LLAAAA B B B4 M ) ) LLKvKKK L L L4|Ar!cdzttfd|j}t|dkrSt|dksJ|jd}t jdfdt|j Dg|d}|j ||j |j |<S)N_maskc&|jdkSr))output)nodemask_output_names rz0FusionAttention.reshape_add_qk..st{1~AQ/Qr!r9rrcg|]}Sr[r[).0_rs r z2FusionAttention.reshape_add_qk..s:::qF:::r!r5r6r7axis) listfilter nodes_to_addr*rrErrFrangeraappendthis_graph_namenode_name_to_graph_name)rr concat_nodeconcat_node_nameconcat_add_qk_fp32rs ` @rreshape_add_qkzFusionAttention.reshape_add_qks"G+6"Q"Q"Q"QSWSdeeff {  q # #;1$$$$:66x@@#- ::::E$.$9$9:::%&!       !34449=9M$%56r!past_kpast_vc |jd}|jd}|dzdd}|dzdd}tjd|g|g|dg}tjd|g|g|dg}|j||j||j|j|<|j|j|<|jd} |dd ddd d } tjd||g| g| d } |j| |j|j| <| S) zConcatenate past_k and past_v inputs to create past_kv input. Args: past_k (str): name of past K value past_v (str): name of past V value Returns: kv_output_name (str): name of past KV value Unsqueeze_5d.rr)r5r6r7r8rz.valuez.kv_value_kvr) rrEreplacerrFrrrr) rrrunsqueeze_k_nameunsqueeze_v_name k_5d_name v_5d_namek_5dv_5drkv_output_name concat_kvs rrzFusionAttention.concat_kvs :66{CC:66{CCe^,,S#66 e^,,S#66  8K!     8K!       &&&   &&&9=9M$%569=9M$%56 :66x@@%88@@cJJRRS[]bcc$ y)#$!       +++9=9M$%56r!cd}|j|}|Wtjt jdd|jjgd|}|j||j|j d}|j d}|d z d d }|d z d d }tj d||g|g| } tj d||g|g| } |j | |j | |j|j|<|j|j|<||fS) ahReshape past_k and past_v from 4D to 3D to use as inputs for multihead attention node. Args: past_k (str): name of past K value of shape 4D past_v (str): name of past V value of shape 4D Returns: k_3d (str): name of past K value of shape 3D v_3d (str): name of past V value of shape 3D kv_4d_to_3dNrint64dtyper7Reshape_3drrr4)rrJr from_arrayrxarrayr`rKrrErrrFrrr) rrr new_dims_namenew_dimsreshape_k_namereshape_v_name k_3d_name v_3d_namek_3dv_3ds r reshape_kvzFusionAttention.reshape_kv4s& :--m<<  #.!R!78HHH}H J & &x1E F F F44Y??44Y??e^,,S#66 e^,,S#66  M*K      M*K       &&&   &&&7;7K$^47;7K$^4)##r!present_k_namepresent_v_namekv_nodec$d\}}|j|}|j|}|Jtjt jdd|}|j||j|Jtjt jdd|}|j||j|jd}|jd} tj d||g|g|d } tj d||g|g| d } |j | |j | |j|j |<|j|j | <dS) a?Split kv_node containing present KV values into separate present K and present V values. Args: present_k_name (str): name of output to store present K value in present_v_name (str): name of output to store present V value in kv_node (str): name of present KV values )index_0index_1Nrrrrr9Gatherr)rrJrrrxrrKrrErrFrrr) rrrrk_indexv_indexk_dimv_dim gather_k_name gather_v_name present_k present_vs rsplit_kvzFusionAttention.split_kves0 **733 **733 = +BHQg,F,F,FWUUUE J & &ud.B C C C = +BHQg,F,F,FWUUUE J & &ud.B C C C 33H==  33H== $ W%#$    $ W%#$       +++   +++6:6J$]36:6J$]333r!c|dzdd}|dzdd}|jd}|jd}tjd|g|g|gd}tjd|g|g|gd}|j||j||j|j|<|j|j|<||fS)a}Transpose past_k and past_v from (B,N,P,H) to (B,P,N,H) Args: past_k (str): name of past K value of shape (B,N,P,H) past_v (str): name of past V value of shape (B,N,P,H) Returns: past_k_transpose (str): name of past K value of shape (B,P,N,H) past_v_transpose (str): name of past V value of shape (B,P,N,H) _transposedrr Transpose)rrtr9ru)r5r6r7perm) rrrErrFrrrr) rrrpast_k_transposepast_v_transposetranspose_k_nametranspose_v_name transpose_k transpose_vs r transpose_kvzFusionAttention.transpose_kvs$#]2;;CEE"]2;;CEE:66{CC:66{CC& 8%&!    & 8%&!       ---   ---9=9M$%569=9M$%56!111r!q_addk_addv_add name_prefixcD|j|jdp$|j|jd}tj|}t j|}t j|}|^|j|jdp$|j|jd} tj| }|^|j|jdp$|j|jd} tj| }t j|||fd} dt j|j z} |dz} | | |j | g| | S)Nr9rrru _qkv_biasr7r=r>r?) rrJr.r rrx zeros_likestackprodshaperKr=)rrrrrq_biasqbkbvbk_biasv_biasqkv_bias qkv_bias_dim bias_names rcreate_combined_qkv_biasz(FusionAttention.create_combined_qkv_biass|++EKN;;itz?Y?YZ_ZefgZh?i?i  !& ) ) ]2   ]2    Z// A??m4:C]C]^c^ijk^lCmCmF%f--B  Z// A??m4:C]C]^c^ijk^lCmCmF%f--B8RRLq1112728,,, +-  &    r!q_matmulk_matmulv_matmulc |jd}|jd|jdkr|jd|jdksJ|j|jd} |j|jd} |j|jd} t j| } t j| } t j| }| j| jkr| j|jksJ| jd}tj| | |fd |d|zf}|dz}| || j |jd|jdg||dz}tj d|jd|g|g| }|j|j|<|g}|d z}| |t jdgdgd |d z}| |t jdg|gd |dz}| |t jdgd|zgd |dz}| |t jdgd|zgd |dz}| |t jdgdgd |dz}tj d||||g|g|jd }|j|j|j<|dz}tj d||||g|g|jd }|j|j|j<|dz}tj d||||g|g|jd }|j|j|j<|} |}!|}"||||g|jr||j|jdrdnd}#tjt j|j|j|#r8||jd|#z <|} |||j|j|j<||j|jdrdnd}#tjt j|j|j|#r8||jd|#z <|}!|||j|j|j<||j|jdrdnd}#tjt j|j|j|#r8||jd|#z <|}"|||j|j|j<|j|| |!|"fS)aCreate packed QKV MatMul node before MultiHeadAttention node. This is for the scenario where an Attention node should be created but cannot be created because past_key and past_value are separate inputs and not one concatenated input. Args: q_matmul (NodeProto): name of MatMul from Q path - (batch_size, sequence_length, hidden_size) k_matmul (NodeProto): name of MatMul from K path - (batch_size, sequence_length, hidden_size) v_matmul (NodeProto): name of MatMul from V path - (batch_size, sequence_length, hidden_size) q_add (NodeProto): name of Add from Q path k_add (NodeProto): name of Add from K path v_add (NodeProto): name of Add from V path num_heads (int): number of heads Returns: Union[NodeProto, None]: the node created or None if failed. MatMulrr9rru _qkv_weightr_qkv_outr4_q_start_indexFr<_k_start_index_v_start_indexrt_end_of_qkv_index_qkv_last_axisr_q_outSlice_k_out_v_out)rrEr.rJr rrrxrreshaperKr=rrFrrr rMr7rHrdanyrr)$rrrrrrrramatmul_node_nameq_weightk_weightv_weightqwkwvwd qkv_weightqkv_weight_nameqkv_matmul_output qkv_matmul qkv_nodes q_slice_name k_slice_name v_slice_nameend_of_qkv_nameqkv_last_axis_nameq_slice_outputq_slicek_slice_outputk_slicev_slice_outputv_sliceq_outputk_outputv_outputinitializer_inputs$ rcreate_packed_qkv_matmul_nodez-FusionAttention.create_packed_qkv_matmul_nodesP4 :66x@@~a HN1$555(.:Kx~^_O`:`:`:`:`:--hnQ.?@@:--hnQ.?@@:--hnQ.?@@  !( + +  !( + +  !( + +x28##BH(<(<(<(< HQKXr2rl333;;QAJGG *]:  ("1%z'7':;    "-z9% N1%7&'!    :>9M$%56L (*::  ,+:KSTRU]^\_ejkkk'*::  ,+:KSTRU]^\_ejkkk'*::  ,+:KSTRU]^ab]b\cinooo*-@@ /[=NVWUX`ade`e_flqrrr-0@@ "4 @QYZX[cebflqrrr)H4" %|\CUV#$,,W55     6:5I$W\2)H4" %|\CUV#$,,W55     6:5I$W\2)H4" %|_FXY#$,,W55     6:5I$W\2'7G4555  1 T )-)C)CEKPQN)S)S$ZAAYZ!6+.tz/I/I%+VgJh/i/ijjkkT9GEK$5 56$H$$U+++?C?SD0< )-)C)CEKPQN)S)S$ZAAYZ!6+.tz/I/I%+VgJh/i/ijjkkT9GEK$5 56$H$$U+++?C?SD0< )-)C)CEKPQN)S)S$ZAAYZ!6+.tz/I/I%+VgJh/i/ijjkkT9GEK$5 56$H$$U+++?C?SD0<   +++8++r!rkey_padding_maskrr packed_qkvc |dksJ|dkr+||zdkr"td|d|dStd|jjD}|jd}g}|rZ||||||||\}}}||j d|j d|j dgn0t|turt|tur{|j r:||j d|j d|j dgn||j d|j d|j dgnt|tkrot|tkrW||vrS||vrO|j r$||j d||gn&||j d||gndS|j s.|||||}||n|d| r| r|| | | | gn| s| r|| | g| g}|r|r|||gt!jd||| }d |_|jt!jd |g|S) a[Create a MultiHeadAttention node. Args: q_matmul (NodeProto): name of MatMul from Q path - (batch_size, sequence_length, hidden_size) k_matmul (NodeProto): name of MatMul from K path - (batch_size, sequence_length, hidden_size) or (batch_size, num_heads, past_sequence_length, head_size) v_matmul (NodeProto): name of MatMul from V path - (batch_size, sequence_length, hidden_size) or (batch_size, num_heads, past_sequence_length, head_size) q_add (NodeProto): name of Add from Q path k_add (NodeProto): name of Add from K path v_add (NodeProto): name of Add from V path num_heads (int): number of attention heads. If a model is pruned, it is the number of heads after pruning. hidden_size (int): hidden dimension. If a model is pruned, it is the hidden dimension after pruning. output (str): output name of MHA key_padding_mask (str): name of key padding mask add_qk (str): name of add after Q x K' past_k (str): name of past K value - (batch_size, num_heads, past_sequence_length, head_size) past_v (str): name of past V value - (batch_size, num_heads, past_sequence_length, head_size) present_k (str): name of present K value - (batch_size, num_heads, sequence_length, head_size) present_v (str): name of present V value - (batch_size, num_heads, sequence_length, head_size) packed_qkv (bool): whether to combine MatMuls from Q, K, V paths Note: This is for the scenario where an Attention node should be created but cannot be created because past_key and past_value are separate inputs and not one concatenated input. Returns: Union[NodeProto, None]: the node created or None if failed. rinput hidden size # is not a multiple of num of heads Ncg|] }|j Sr[r)rrs rrzCFusionAttention.create_multihead_attention_node..s P P Pt P P Pr!rhr.rgr4 com.microsoftra)rrsetrgraphr.rEr-rHrtyper rdrZrrrrFdomainrGrI)rrrrrrrrar`rr/rrrrrr0graph_input_names mha_node_name mha_inputsr$r&r(r mha_outputsmha_nodes rcreate_multihead_attention_nodez/FusionAttention.create_multihead_attention_nodefsbZ1}}}} ?? i 7A== LLikii^gii j j j4 P Ptz7G7G7I7I7O P P PQQ 33K@@   (,(J(J(HeUE9)) %GWg   w~a0'.2CW^TUEVW X X X X (^^y ( (T(^^y-H-H5 `!!5<?HOA4F UV"XYYYY!!8?1#5xq7I8?[\K]"^____ NNc ! !X#%%------5 L!!5<?Hh"GHHHH!!8?1#5x"JKKKK4 5 "55eUE=YYI   i ( ( ( (   b ! ! !  :f :   /H I I I I  : :   /8 9 9 9h  7 7    95 6 6 6#      *!!6#8i#P#P"QRRRr!r&r. add_qk_strscalecausalc|dksJ| dkr+| |zdkr"td| d|dSd}|||d}|j|jd}|j|jd}|j|jd}d\}}}|r|j|jdp$|j|jd}|j|jdp$|j|jd}|j|jdp$|j|jd}|r|r|r|sdS|t |jdd dSt j|}t j|}t j|}|j|jksJ|jd}|jd}|jd}||cxkr|ksnJ| dkr'| |kr!t d | d |d d} |j|jkrd} tj |jdd}!tj |jdd}"tj |jdd}#d}$| r"tj |||fd }%|!|"z|#z}$ntj |||fd }%d|!z}$|rt j|}&t j|}'t j|}(tj |&j})tj |'j}*tj |(j}+|)|*cxkr|!ksnJ|+|#ksJ| r"tj |&|'|(fd },|)|*z|+z}-ntj |&|'|(fd },d|)z}-|jd}.|js#||.dz|j||$g|%|r"||.dz|j|-g|,|jrz| rtddS|jd|jd|jd|.dzg}/||/|t)jd|/| g|.}0n9| |.dz|r|.dzndg}/||/|n|/d| o|}1|1r+|| |}2|/|2| A|| }3|1s|/d|/|3| g}4|rl|rj|dddddd}5|4|5||||5t)jd|/|4|.}0d|0_|0jt)jd|g|r.|0jt)jddg|.|0jt)jd|g| r1|0jt)jd|!|"|#gg|j@|0jt)jd t?|jg|0S)!a+Create an Attention node. Args: mask_index (str): mask input q_matmul (NodeProto): MatMul node in fully connection for Q k_matmul (NodeProto): MatMul node in fully connection for K v_matmul (NodeProto): MatMul node in fully connection for V q_add (NodeProto): Add bias node in fully connection for Q k_add (NodeProto): Add bias node in fully connection for K v_add (NodeProto): Add bias node in fully connection for V num_heads (int): number of attention heads. If a model is pruned, it is the number of heads after pruning. hidden_size (int): hidden dimension. If a model is pruned, it is the hidden dimension after pruning. input (str): input name output (str): output name add_qk_str (str): name of Add node after Q x K' past_k (str): name of input for past K value past_v (str): name of input for past V value present_k (str): name of output to store present K value present_v (str): name of output to store present V value scale: scale before softmax causal: whether it is uni-directional mask. Returns: Union[NodeProto, None]: the node created or None if failed. rr2r3NTFr9)NNNzl is not an initializer. Please set do_constant_folding=True in torch.onnx.export to unblock attention fusionzInput hidden size (z3) is not same as weight matrix dimension of q,k,v (z:). Please provide a correct input hidden size or pass in 0rrurhrrrzVMultiHeadAttention does not support relative_position_bias: cannot fuse the attention.rgr4r.z.key_keyrrr5raunidirectionalrAqkv_hidden_sizesrj) rrrrJr.printr rrrrxr concatenaterrErcrKr=rrrrFrrrrr9rGrHrIrjfloat)6rr&rrrrrrrar`r.rr@rrrrrArBhas_biasrrrrrrrrr qw_in_size kw_in_size vw_in_sizeis_qkv_diff_dims qw_out_size kw_out_size vw_out_sizeqkv_weight_dimrrrr q_bias_shape k_bias_shape v_bias_shaperrattention_node_nameattention_inputsattention_node past_existspast_kvrattention_outputs present_kvs6 rcreate_attention_nodez%FusionAttention.create_attention_nodes@\1}}}} ?? i 7A== LLikii^gii j j j4 =U]u}H:--hnQ.?@@:--hnQ.?@@:--hnQ.?@@!1  Z// A??m4:C]C]^c^ijk^lCmCmFZ// A??m4:C]C]^c^ijk^lCmCmFZ// A??m4:C]C]^c^ijk^lCmCmF  f  t   >!$ggg   4  !( + +  !( + +  !( + +x28####Xa[ Xa[ Xa[ Z5555:555555 ??{j88 NNJkJJfpJJJ    ! 8rx  #  gbhqrrl++ gbhqrrl++ gbhqrrl++   -R 1===J(;6DNN2r2,Q777J_N  0%f--B%f--B%f--B728,,L728,,L728,,L<>>>>;>>>>>>;.... 0>2r2,Q???+l:\I 8RRLq999 </ "j99+FF,   (=8", .1 !       (;6 *"^ !     (8   uvvvt"""#k1   % '' 333#-$'( NN#m35=E#k112   % '' 3333 ''+++ +VK 1..88 ''000%#'#6#6z#B#B #0$++B/// ''(8999!'  @Y @&..vr::BB62NNVVWZ\_`` !((444 iJ???#-')( N!0 '')>{I)V)V(WXXX  Z  $ + +V-BCSUV-W-W,X Y Y Y    $ + +V-B7E-R-R,S T T T    $ + +&'9KVa;bccd     ! -  $ + +V-BCVX]^b^tXuXu-v-v,w x x xr!c|}|jdkr#|j|dd}||}ndS|j|gdgd}d}| |\}}} } } n,|j|gdgd}||\}}} } ndSg} t |jD]7\} }||vr ||djdkr"| |8t| dkrdS| d} |j|d d}|p||jd}|6t|d kr#|d}|jdkr|jd}nWdS|!t|d kr|jd}n2dS|jdkr%||}|D]}|jdkr |jd} ||}|jd kr%t|jd kr |jd}||}d|D}| ddkrdS|j| gdgd}|t ddS|\}}}}d}d}d}gdgdfgdgdfgdgdfgdgdfgdgdfd}d}| D]H\}} |j| | d| d}|0|dkrd }|d!krd }|d"krd }|t d#dSd}!d}"d}#|r|\}}#}"}n|r|\}}!}#}"n|r|\}}}"n|\}}!}}"|j|"gdgd$}$|$>|j|"gd%gd&}$|$t d'dS|$d(}%|$d)}&|$d*}'|j|"gdgd}(|(>|j|"gd+gd,}(|(t d-dS|(d)})|(d*}*d}+d},|r6|j |#gd.gdfgd/gdfgd0gd1fg|\}}+}n|rg|j |#gd2gd1fgd/gdfg|\}}+}|!6||!},|,t d3|!dSn1|rn.|j |!gd4gd5fgd6gd7fg|\}}+}|s|+t d8dS|sTt|+dkrA|+djd kr0|j|+d\}}-|-d9kr|-|_|jd|krn|'jd|kr^|*jd|krN|s+|j|+d*jdnd}.|| n| }/||%\}0}1|0dks|1dkrtd:dS||.|'|*||&|)||0|1||/jd|, }2|2dS|j|2|j|j|2j<||jd}3d;|3z}4|d<|3zt6jd gt;jdd|0t?|1|0z gd=}5|j tCj"d>|/jd|5jg|4gd?|3z|j|4|jd<|j#$|/| | g|j#$||j#$|j%s|$n |$dd*|j#$|j%s|(n |(dd*|j#$|j%s|n |dd*d |_&dSdSdSdS)@Nr_Addr)r_rrrr)NNrrr)r_Einsumrr)r9Nrrr9Mulrtr^rscg|] }|j Sr[)r)rchilds rrz(FusionAttention.fuse..s>>>E%->>>r!rru)rrr_r)r9rrNz&fuse_attention: failed to match v pathF)Softmaxr_Divr)rrNr)rer_rar)reWhererrf)rrrtr)rer_rgr)rrrrt)rerfr)rrr)path1path2path3path4path5rjTrkrlz'fuse_attention: failed to match qk path)rrrN)rfrrr_r)rrrrNz&fuse_attention: failed to match q pathr)rrrr_r)r9rrrNz&fuse_attention: failed to match k path)ExpandrEqual)rprr)Castrorrp)rrrr)rqrprrz4fuse_attention: failed to verify shape inference of )raSubrqrr)Nrr9rr)rarrrr)Nrr9rz)fuse_attention: failed to match mask pathizmFailed to detect num_heads and hidden_size for Attention fusion. Please specify those parameters in argument.edge_modified_shape_modified_tensorr<rreshape_modified_)'rr match_parentmatch_parent_path enumerater.rrr*countrritemsmatch_parent_pathsrget_constant_inputrjrbrUrrr]rrrr7rKr rMrxrintrNrrFnodes_to_removerHrc prune_graph)6rnormalize_nodeinput_name_to_nodesoutput_name_to_node start_nodeadd_before_layernormr einsum_noder reshape_qkv transpose_qkv matmul_qkv other_inputs_ir. root_inputmul_before_layernorm mul_childrenlayernorm_nodechildrenrd parent_nodechildren_typesv_nodesadd_vmatmul_v is_distillis_distill_addis_no_mask_attentionqk_pathsqk_nodeskvr matmul_qkwhere_qkq_nodesr}add_qmatmul_qk_nodesadd_kmatmul_k mask_nodesr@mul_valr&attention_last_node q_num_heads q_hidden_sizenew_node unique_indexnew_edge shape_tensors6 rfusezFusionAttention.fuses $  !%9 9 9#':#:#:>5RS#T#T #/1 J00  ? ? ? ! ! !    =F :Q; zz 44DDDoooI$>G;K  ":#344 ' 'IB/// ! +A...    & & & & |   ! ! F!!_   $z66z5!LL  +./C/J1/MNL'C ,=,=,B,B!-a!)-AAA!/!6q!9JJF)c,.?.?1.D.D18;   #'; ; ;*:6H! 1 1=$888!&aJ **5  ": : :s;CU?V?VZ[?[?[$+A.J&z2>>X>>>    ) )Q . . F*..z;d;d;dfufufuvv ? LLA B B B F")Auh $999???K999???K;;;\\\J;;;\\\J222III>   NN$$  DAqz33J!adKKHG||! G||!%G||'+$    LLB C C C F   1*2 'Q)QQ  1/7 ,Q)) ! 1 ( Q99(0 %Q9*..y:c:c:cetetetuu ?j22@@@"""G  EFFFBK  2;*..y:c:c:cetetetuu ?j22FFF"""G  EFFF 2;  % #z<<333YYY?888)))D;;;\\\J $   Az11 #z<<@@@,,,O888)))D$    Az1!!0088 %LL!`X^!`!`aaaF !  #z<<IHH***>==O $   Az1$  (: LLD E E E F# 1J!(;(; 1 @UY^@^@^66z!}EEJAw&  )0& >!  * *x~a/@J/N/NS[SabcSdhrSrSrZnx,99*R.:Nq:QRRRtxJ1<1D++- )-)K)KI)V)V &Ka=A#5#5C11#*1-  H   $ $X . . .:>:ND ( 7&*03 +l:#330<?)/1ac-+:U6V6V"WXX 4    ##$!,3A6 8IJ! +l:  ((0 !!$  ' ')@T%U%%%% % !/ % #' %,0%c%%%%%%40 0eTWY\T\o0000>'&y'&U3PS8_'&'&'&'&RY* S    255S5S5555n/$/$c/$sCj/$/$/$/$b+Ks+KC+K#+K+K+K+KZ%23%2%2%2%2%2NY_%Y_%    y$  rs9 ////////////......11111111============ 8  SSSSSSSSlk$k$k$k$k$fk$k$k$k$k$r!