Χgg UddlZddlmZmZmZddlmZddlmZm Z m Z m Z m Z m Z mZddlmZddlmZddlmZdd lmZddlZd d gZejjZd ZiZe e e fed <dZd4dZeej ddde!fdZ"eej#d5de!fdZ$eej%d5de!fdZ&eej'd5de!fdZ( d4dee!dee!dee!de)de!f dZ*eej+ej,gddde!fdZ-eej.de!fdZ/dZ0eej1ej2ej3gddde!fdZ4d Z5dd!dee e e!d"fe e!d"fe e!d"fe e e!d"fffd#Z6dd!dee e e!d"fe e!d"fe e!d"fe e e!d"fffd$Z7eej8d%&ddde!fd'Z9eej:d%&de!fd(Z;d)Z<eej=ej>ej?gddde!fd*Z@eejAd%&de!fd+ZBeejCd%&de!fd,ZDiej e"ej#e$ej%e&ej'e(ej+e-ej,e-ej.e/ej1e4ej2e4ej3e4ej=e@ej>e@ej?e@ej8e9ej:e;ejAeBejCeDZd-ZEgd.ZFd/ZGd0ZHd1ZId2ZJGd3d eZKdS)6N)tree_map tree_flattentree_unflatten) ModuleTracker)ListAnyDictOptionalUnionTupleIterator) defaultdict)TorchDispatchModeprodwrapsFlopCounterModeregister_flop_formulacHt|tjr|jS|SN) isinstancetorchTensorshape)is T/var/www/html/ai-engine/env/lib/python3.11/site-packages/torch/utils/flop_counter.py get_shapers"!U\""w H flop_registrycBtddfd }|S)N)out_valcPtt|||f\}}}|d|i|S)N out_shape)rr)r#argskwargsr%fs rnfzshape_wrapper..nfs:"*9tVW6M"N"Nfiq$6)6v666r rr(r)s` r shape_wrapperr+s@ 1XX777777X7 Ir Fcfd}|S)Ncstfd}tjj|S)Nct|tjjs"t d|dt ||t vrtd|t |<dS)Nzlregister_flop_formula(targets): expected each target to be OpOverloadPacket (i.e. torch.ops.mylib.foo), got z which is of type zduplicate registrations for )rr_opsOpOverloadPacket ValueErrortyper! RuntimeError)target flop_formulas rregisterz=register_flop_formula..register_fun..register&sfej&ABB A @@@15f@@AAA&&"#J&#J#JKKK$0M& ! ! !r )r+rutils_pytree tree_map_)r5r6get_rawtargetss` r register_funz+register_flop_formula..register_fun"sU 7(66L 1 1 1 1 1  %%h888r )r;r:r<s`` rrr!s*& r )r%returnc<|\}}|\}}||ksJ||zdz|zS)zCount flops for matmul.r=) a_shapeb_shaper%r&r'mkk2ns rmm_floprG7s5 DAq EB 7777 q519q=r c "t||S)zCount flops for addmm.)rG self_shaperArBr%r's r addmm_floprKBs 7G $ $$r c Z|\}}}|\}}} ||ksJ||ksJ||z| zdz|z} | S)z"Count flops for the bmm operation.r@r=) rArBr%r'brCrDb2rErFflops rbmm_floprPGsO GAq!IBA 7777 7777 q519q=1 D Kr c "t||S)z&Count flops for the baddbmm operation.rPrIs r baddbmm_floprSTs GW % %%r x_shapew_shaper% transposedc|d}|r|n|dd}|^}}} t|t|z|z|z|zdz} | S)aCount flops for convolution. Note only multiplication is counted. Computation for bias are ignored. Flops for a transposed convolution are calculated as flops = (x_shape[2:] * prod(w_shape) * batch_size). Args: x_shape (list(int)): The input shape before convolution. w_shape (list(int)): The filter shape. out_shape (list(int)): The output shape after convolution. transposed (bool): is the convolution transposed Returns: int: the number of flops rr@Nr) rTrUr%rV batch_size conv_shapec_outc_in filter_sizerOs rconv_flop_countr]\sj*J'6''Y;J 'E4+   d;// /* .ts(a%(#d59oo55r rrFr_)rr])grad_out_shaperTrUr`rarbrcrV_output_padding_groups output_maskr%rh flop_countgrad_input_shapegrad_weight_shapes rconv_backward_floprps666JDL1~a$Yq\22ong?OU_Q_``` 1~q%il33  q /!!N*;*;QQwZZK\I]I]joppp pJJ /!!G**aa6G6GK\I]I]joppp pJ r c|\}}}}|\}}} } |\} } } }||cxkr| kr%nn"||cxkr| krnn|| kr | | kr|| ksJd}|t||z||f||z|| fz }|t||z|| f||z| |fz }|S)z^ Count flops for self-attention. NB: We can assume that value_shape == key_shape rrR) query_shape key_shape value_shaperMhs_qd_q_b2_h2s_k_d2_b3_h3_s3d_v total_flopss rsdpa_flop_countrs !NAq#s"Cc3$Cc3 ????s?????qC33#::#**QTX[Q[Q[Q[Q[K8QUC-AsC/@AAAK8QUC-AsC/@AAAK r c$t|||S)Count flops for self-attention.r)rrrsrtr%r&r's r sdpa_floprs ; ; ? ??r cddlm}ddlm}t |||fs&|S|g|ddz zS)z If the offsets tensor is fake, then we don't know the actual lengths. In that case, we can just assume the worst case; each batch has max length. r) FakeTensor)FunctionalTensorr)torch._subclasses.fake_tensorr#torch._subclasses.functional_tensorrrdifftolistsize)offsetsmax_lenrrs r_offsets_to_lengthsr sv 988888DDDDDD g ,<= > >'||~~$$&&& 9 Q!+ ,,r )grad_out.c#K|t|jdksJt|jdksJ||j|jksJ|j\}} } |j\}} } |j\}} }|J|J|j|jksJt||}t||}t||D]%\}}d| || f}d| || f}d| ||f}||nd}||||fV&dS|j|j|j||jndfVdS)a; Given inputs to a flash_attention_(forward|backward) kernel, this will handle behavior for NestedTensor inputs by effectively unbinding the NestedTensor and yielding the shapes for each batch element. In the case that this isn't a NestedTensor kernel, then it just yields the original shapes. Nrlenrrzip)querykeyvaluer cum_seq_q cum_seq_kmax_qmax_k_h_qrwh_kd_kh_vr seq_q_lengths seq_k_lengths seq_q_len seq_k_lennew_query_shape new_key_shapenew_value_shapenew_grad_out_shapes r%_unpack_flash_attention_nested_shapesrst$ 39~~""""5;1$$$$8>U[#@#@#@#@k 3i 3k 3$$$$$$)/1111+Iu== +Iu== &)-&G&G V V "Y  #y#6OY4M #y#6O4<4Hd !=/CUU U U U U +sy%+AUx~~[_ ______r c# K|t|jdksJt|jdksJ||j|jksJ|j\}}} } |j\}}} } |j\}}} }|J|J|j|jksJt||}t||}t||D]%\}}d| || f}d| || f}d| ||f}||nd}||||fV&dS|j|j|j||jndfVdS)a? Given inputs to a efficient_attention_(forward|backward) kernel, this will handle behavior for NestedTensor inputs by effectively unbinding the NestedTensor and yielding the shapes for each batch element. In the case that this isn't a NestedTensor kernel, then it just yields the original shapes. Nrr)rrrr cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_krrrwrrrr seqlens_q seqlens_klen_qlen_krrrrs r)_unpack_efficient_attention_nested_shapesrFs{$39~~""""5;1$$$$8>U[#@#@#@#@1c31c31c3''''''!\%77777' lCC ' lCC  955 V VLE5 #uc2OUC0M #uc2O4<4Hd !=/CUU U U U U +sy%+AUx~~[_ ______r T)r:c `t|||||||} td| DS)r)rrrrrrrc3BK|]\}}}}t|||VdSrr.0rrrsrtrs r z0_flash_attention_forward_flop..J 2KK  Y <<r rsum) rrrrrrrr%r&r'sizess r_flash_attention_forward_floprvs]" 2    E 6;  r c `t|||||||} td| DS)r)rrrrrrrc3BK|]\}}}}t|||VdSrrrs rrz4_efficient_attention_forward_flop..rr rr) rrrbiasrrrrr&r'rs r!_efficient_attention_forward_floprs]" 6 !!!!   E 6;  r cd}|\}}}}|\} } } } |\} }}}|\}}}}|| cxkr | cxkr|kr nn|| cxkr |cxkr|kr nn|| ksJ||kr | |kr||ksJd}|t||z||f||z|| fz }|t||z||f||z|| fz }|t||z| |f||z||fz }|t||z|| f||z| |fz }|t||z||f||z|| fz }|S)NrrR)rirrrsrtrrMrurvrwrxryrzr{r|r}r~r_b4_h4_s4_d4s rsdpa_backward_flop_countrsK NAq#s"Cc3$Cc3'Cc3  ! ! ! !s ! ! ! !c ! ! ! ! !a3&<&<&<&<#&<&<&<&<&<&<&<&<&< #::#**K8QUC-AsC/@AAAK8QUC-AsC/@AAAK8QUC-AsC/@AAAK8QUC-AsC/@AAAK8QUC-AsC/@AAAK r c&t||||S)z(Count flops for self-attention backward.r)rirrrsrtr%r&r's rsdpa_backward_floprs $NKK X XXr c bt|||||||| } td| DS)N)rrrrrrrrc3DK|]\}}}}t||||VdSrrrrrrsrtris rrz1_flash_attention_backward_flop..L ?KK !iUUr r) rrrrout logsumexprrrrr&r'shapess r_flash_attention_backward_floprs`"3    F CI  r c bt|||||||| } td| DS)N)rrrrrrrrc3DK|]\}}}}t||||VdSrrrs rrz5_efficient_attention_backward_flop..rr r) rrrrrrrrrrr&r'rs r"_efficient_attention_backward_floprs`"7 !!!!   F CI  r c6t|ts|fS|Sr)rtuple)xs rnormalize_tupler.s a  t Hr )KMBTc tdtttdz tt |dz dz}t|S)Nrrr@r)maxminrsuffixesstr)numberindexs rget_suffix_strr7sJ 3s8}}q(3s6{{+;+;a+?A*EFF G GE E?r cjt|}|d|zz d}|t|zS)Niz.3f)rr)rsuffixrrs rconvert_num_with_suffixr>s6 NN6 " "E % + +E 8E? ""r c |dkrdS||z dS)Nr0%z.2%r=)numdenoms rconvert_to_percent_strrEs  zztEk  r c<tfd}|S)NcRt|\}}|}t||Sr)rr)r& flat_argsspecrr(s rr)z)_pytreeify_preserve_structure..nfKs/&t,, 4amc4(((r rr*s` r_pytreeify_preserve_structurerJs3 1XX))))X) Ir c eZdZdZ ddeeejje ejjfde de dee e e fffd Zd e fd Zd e ee e e fffd Zdd ZfdZfdZddZdZxZS)ra ``FlopCounterMode`` is a context manager that counts the number of flops within its context. It does this using a ``TorchDispatchMode``. It also supports hierarchical output by passing a module (or list of modules) to FlopCounterMode on construction. If you do not need hierarchical output, you do not need to use it with a module. Example usage .. code-block:: python mod = ... with FlopCounterMode(mod) as flop_counter: mod.sum().backward() Nr@Tmodsdepthdisplaycustom_mappingcDttd|_||_||_|i}|t jdditd| D|_t|_ dS)Nc*ttSr)rintr=r rz*FlopCounterMode.__init__..os+VYJZJZr zz,FlopCounterMode.__init__..xs<nnntqRSqwq*e44J!!-:J:Jnnnr ) super__init__r flop_countsrrwarningswarnr!itemsr mod_tracker)selfrrrr __class__s rr zFlopCounterMode.__init__hs 6ABZBZ6[6[   !N   MXef g g g g  nnWeWkWkWmWmnnn )??r r>cZt|jdS)NGlobal)rr valuesrs rget_total_flopszFlopCounterMode.get_total_flops|s$4#H-4466777r cHd|jDS)aReturn the flop counts as a dictionary of dictionaries. The outer dictionary is keyed by module name, and the inner dictionary is keyed by operation name. Returns: Dict[str, Dict[Any, int]]: The flop counts as a dictionary. c4i|]\}}|t|Sr=)dictrs rrz3FlopCounterMode.get_flop_counts..s$@@@tq!477@@@r )r r rs rget_flop_countszFlopCounterMode.get_flop_countss(A@t'7'='='?'?@@@@r c |j}|d}ddl}d|_gd}g} t  d fd}t jD]L}|dkr |dd z}||kr(|||d z }| |MdjvrJ sHtt|D]} d || dz|| d<|dd|z}t|dkrgd g}|||d S)Ni?BrT)ModuleFLOPz% TotalFc t j|} |kz d|z}g}|||zt |t |g j|D]L\}}||dzt|zt |t |gM|S)N z - )rr rappendrrr r) mod_namerrpaddingrrDr global_flops global_suffixis_global_subsumedrs r process_modz.FlopCounterMode.get_table..process_modsd.x8??AABBK +"= = EkGF MM("' ]CC&{LAA    (288::  1 eOc!ff,+A}==*1l;; Mr r.rr)r0r)leftrightr))headerscolalign) rtabulatePRESERVE_WHITESPACErrsortedr keyscountextendranger) rrr,headerrr%mod mod_depth cur_valuesidxr"r#r$s ` @@@r get_tablezFlopCounterMode.get_tables =JE =E'+$...++-- &|44 "        ,$*//1122 & &Ch #*I5  $S)a-88J MM* % % % % t' ' '0B 'S[[)) 6 6!$vc{1~!5s A [1--6F v;;!  +++,F  B\ ]]]r c|j|jt |Sr)r clearr __enter__r)rrs rr;zFlopCounterMode.__enter__sH     ""$$$  r ctj||j|jr)t ||jdSdSr)r__exit__rrprintr8r)rr&rs rr=zFlopCounterMode.__exit__sb$ !!### < . $..,, - - - - - . .r r=cX|r|ni}||i|}||j|||Sr) _count_flops_overloadpacket)rfunctypesr&r'rs r__torch_dispatch__z"FlopCounterMode.__torch_dispatch__sA!)rdD#F##  !5sD&IIIr c||jvrP|j|}||i|d|i}t|jjD]}|j||xx|z cc<|S)Nr#)r!setrparentsr )r func_packetrr&r'flop_count_funcrmpars rr@zFlopCounterMode._count_flopss $, , ,"0=O($F&FF#FFFJ4+344 A A %k222j@2222 r )Nr@TNr)r=N)__name__ __module__ __qualname____doc__r r rnnrrrboolr r r rrrr8r;r=rDr@ __classcell__)rs@rrrTsj*MQ 7; ++5$ux2G!GHI++ + %T#s(^4 ++++++(88888 Ac4S>&9!: A A A A:^:^:^:^x ..... JJJJ r )Fr)Lrtorch.utils._pytreerrrmodule_trackerrtypingrr r r r r r collectionsrtorch.utils._python_dispatchrmathr functoolsrr __all__opsatenrr!__annotations__r+rmmrrGaddmmrKbmmrPbaddbmmrSrPr] convolution _convolutionrdconvolution_backwardrpr'_scaled_dot_product_efficient_attention#_scaled_dot_product_flash_attention#_scaled_dot_product_cudnn_attentionrrrr_flash_attention_forwardr_efficient_attention_forwardrr0_scaled_dot_product_efficient_attention_backward,_scaled_dot_product_flash_attention_backward,_scaled_dot_product_cudnn_attention_backwardr_flash_attention_backwardr_efficient_attention_backwardrrrrrrrrr=r rrns FFFFFFFFFF))))))DDDDDDDDDDDDDDDDDD######:::::: 5 6 y~   !# tCH~""",tw/3# tz""%%#%%%#"%tx    C   !  t|$$&&C&&&%$& %% #Y% #Y%Cy% %  %%%%N($*;<==bfOOOuxOOO>=O t011eeee21eN$D@@BCCEI@@@WZ@@@CC@ - - -" +`+`+`eE#s(OU38_eCHoxPUVY[^V^P_G``ab+`+`+`+`f -`-`-`eE#s(OU38_eCHoxPUVY[^V^P_G``ab-`-`-`-``t4dCCC DC>t8$GGG HG>6MIIKLL^bYYYpsYYYLLYt5tDDD ED@t94HHH IH@GWJ  Hh L,   i   y  1 0) ,i ,i 9;M 57I 57I !#@ %'H  "$B!" &(J# (    $ # ####   KKKKK'KKKKKr