קg6ddlZddlZddlmZmZmZddlZddlmZddl m Z m Z ddl m Z ddlmZddlmZdd lmZmZdd lmZdd lmZd d lmZd dlmZerddlmZGddeZdZ dZ!GddeZ"GddeZ#GddeZ$GddeZ%GddeZ&GddeZ'dS) N)DictList TYPE_CHECKING) BackwardState)compiled_autograd variables) trace_wrapped) unimplemented)%call_module_hooks_from_backward_state) GuardBuilder install_guard) AttrSource)istype)VariableTrackerConstantVariable)InstructionTranslatorc@eZdZdZdfd ZdZedZxZS)DistributedVariablea{ The base distributed variable that encapsulates common methods for the distributed objects (i.e. ProcessGroup, DeviceMesh, etc.). Concrete distributed objects could inherit this class and add object specific logic. i.e. It provides the check on the distributed package existance and hold the tracking value for the corresponding distributed object. returnNc tjdi|tst d||_dS)Nz+torch.distributed package is not available!)super__init__r is_availabler value)selfrkwargs __class__s _/var/www/html/ai-engine/env/lib/python3.11/site-packages/torch/_dynamo/variables/distributed.pyrzDistributedVariable.__init__#sO""6""""//11 I G H H H c*t|jSN)typerrs r" python_typezDistributedVariable.python_type)sDJr#c>tjSr%)torch distributedrrr#r"rz DistributedVariable.is_available,s --///r#)rN) __name__ __module__ __qualname____doc__rr( staticmethodr __classcell__r!s@r"rrsq    00\00000r#rc~tsdSddlm}t j|o||juS)NFr)DTensor)rrtorch.distributed.tensorr4inspect isfunction from_local)rr4s r" is_from_localr92sK  + + - -u000000  e $ $ D'2D)DDr#ctsdSddlm}m}m}m}m}|||||g}tj |o||vS)NFr)_get_group_size_by_name_get_group_tag_rank_not_in_group$_resolve_group_name_by_ranks_and_tagget_process_group_ranks) rr"torch.distributed.distributed_c10dr;r<r=r>r?r6r7)rr;r<r=r>r?constant_processgroup_functionss r"is_constant_pg_functionsrB:s  + + - -u , '#  e $ $ Q2Q)QQr#cFeZdZdZedZdddedeffd ZxZ S)WorldMetaClassVariablez Tracks torch.distributed.GroupMember and torch.distributed.group, which are instances of the metaclass _WorldMeta. c\|sdSddlm}t||uS)NFr) _WorldMeta)rr@rFr&)clsrrFs r"is_group_member_typez+WorldMetaClassVariable.is_group_member_typeWs>!! 5AAAAAAE{{j((r#txrnamerc |dkr[t|jd}t|tjt |jjSt ||S)NWORLD)basemember) rsourcer make_guardr ID_MATCHProcessGroupVariablerrLr var_getattr)rrIrJrOr!s r"rSz"WorldMetaClassVariable.var_getattr`sl 7??T[AAAF &++L,ABB C C C' (899 9ww""2t,,,r#) r,r-r.r/ classmethodrHstrrrSr1r2s@r"rDrDQsx ))[)-5-S-_----------r#rDcLeZdZedZdZ d fd ZxZS) PlacementClassVariablectsdSddlm}t |tuot ||SNFr) Placement)rr(torch.distributed.tensor.placement_typesrZr& issubclassrrZs r"is_placement_typez(PlacementClassVariable.is_placement_typeisO#//11 5FFFFFFE{{d"Cz%'C'CCr#c|jSr%rr's r"as_python_constantz)PlacementClassVariable.as_python_constants zr#rIrargsList[VariableTracker]r Dict[str, VariableTracker]rrcntj|jddtjfvrj|jrct|j}t |}tj|jddr||d|||St |||S)N__new__r) r6getattr_staticrobjectrgrOPlacementVariable call_methodr call_function)rrIrcr new_objvarr!s r"rlz$PlacementClassVariable.call_functionvs  "4:y$ ? ?FNCT T T  U nnTZ00G#G,,C%dj*dCC Jf=== ww$$Rv666r#)rIrrcrdr rerr)r,r-r.r0r^rarlr1r2s@r"rWrWhsDD\D7 #7&7- 7  7777777777r#rWc`eZdZedZdZdddedeffd Z dfd Z xZ S)rjcdtsdSddlm}t ||SrY)rrr[rZ isinstancer]s r" is_placementzPlacementVariable.is_placements?#//11 5FFFFFF%+++r#c|jSr%r`r's r"raz$PlacementVariable.as_python_constantrbr#rIrrJrc|dkrtj|jjSt ||S)Ndim)rcreaterrurrSrrIrJr!s r"rSzPlacementVariable.var_getattrs; 5==#*4:>:: :ww""2t,,,r#rcrdr rerc&ddlm}gd}||vr t|j}t j|dd Jdt j||}n#t $rd}YnwxYw|tjur|j dSd|D}d| D}|dkr||jg|Ri||S||jg|Ri|} |j | St ||||S) Nrr)r __setattr__is_shard is_partial is_replicate __getattr__zno custom getattr allowed!c6g|]}|Srra.0xs r" z1PlacementVariable.call_method..s$999qA((**999r#c>i|]\}}||Srrrkvs r" z1PlacementVariable.call_method..s*KKKDAqa--//KKKr#ry) rr&rr6rhAttributeErrorrirrvitemsrrk) rrIrJrcr rconstant_fold_functions value_typemethod constant_valr!s r"rkzPlacementVariable.call_methods~ '&&&&&# # #  * * * !$*-- *:}dKKSS/TSS / DAA!    ((.'.t44499D999DKKFLLNNKKKF}$$tz3D333F333 !6$*>t>>>v>>L*#*<88 8ww""2tT6:::sA A A*)A*rcrdr rerr) r,r-r.r0rrrarUrrSrkr1r2s@r"rjrjs,,\,-5-S-_------ *;& *; - *;  *;*;*;*;*;*;*;*;*;*;r#rjc`eZdZedZdZdddedeffd Z dfd Z xZ S)DeviceMeshVariablecdtsdSddlm}t ||S)NFr) DeviceMesh)rrtorch.distributed.device_meshrr)rrs r"is_device_meshz!DeviceMeshVariable.is_device_meshs?#//11 5<<<<<<eZ(((r#c|jSr%r`r's r"raz%DeviceMeshVariable.as_python_constantrbr#rIrrJrc|dkrtj|jjS|dkrtj|jjSt ||S)Nndim device_type)rrvrrrrrSrws r"rSzDeviceMeshVariable.var_getattrs\ 6>>#*4:?;; ; = #*4:+ABB Bww""2t,,,r#rcrdr rerc|dkrNd|D}d|D}tj|jj|i|S|dkr+tj|jS|dkr+tj|jS|dkr&t|jSt ||||S)Nsizec6g|]}|Srrrs r"rz2DeviceMeshVariable.call_method..s$???Q!..00???r#c>i|]\}}||Srrrs r"rz2DeviceMeshVariable.call_method..s*QQQ$!QAq3355QQQr#get_coordinate get_group_get_or_create_default_group) rrrvrrrrrRrrrk)rrIrJrcr const_args const_kwargsr!s r"rkzDeviceMeshVariable.call_methods 6>>??$???JQQ&,,..QQQL#*?4:?J+W,+W+WXX X # # ##*4:+D+D+F+FGG G ;  #*4:+?+?+A+ABB B 1 1 1' (O(O(Q(QRR Rww""2tT6:::r#r) r,r-r.r0rrarUrrSrkr1r2s@r"rrs))\)-5-S-_------;& ; - ;  ;;;;;;;;;;r#rcXeZdZdZdZ dfd Zdfd Zed ZxZ S)rRa We don't want a ProcessGroup object to end up in our output graph. But it's common for dynamo to intercept a PG that is then used to get info like rank() or world_size(), as well as passed to utility functions in distributed_c10d which desugar it into plain types like a ranklist and tag. For convenience and proper guarding, we construct a variable type. TODO: make it possible to use ProcessGroupVariable as input to simple functions like _expand_group without dynamo complaining about making a proxy for it. It is not a tensor-like type, and we don't want a proxy- but dynamo assumes torch library functions are dealing with tensor-like types and would have proxies for their args. TODO: should we make this inherit VT instead of UDOV? Do we want any of the default behaviors or just graph-break whenever one of our special cases is not hit? c|jSr%r`r's r"raz'ProcessGroupVariable.as_python_constantrbr#rcrdr rerrc|dkr6tj|jS|dkr6tj|jS|dkr6tj|jSt||||S)Nrankr_get_backend_name) r rrvrrrrrrkrrIrJrcr r!s r"rkz ProcessGroupVariable.call_method s 6>>-44TZ__5F5FGG G 6>>-44TZ__5F5FGG G & & &-44TZ5Q5Q5S5STT Tww""2tT6:::r#rIrcdkr)tjjjSdvrtjfdSt S)N group_name)rrc4||Sr%)rk)rcr rJrrIs r"z2ProcessGroupVariable.var_getattr..s(8(8T4(P(Pr#)r rrvrrLambdaVariablerrSrws```r"rSz ProcessGroupVariable.var_getattrs| <  -44TZ5JKK K # # #+PPPPPP ww""2t,,,r#cttsdSddlm}ddlm}t |||fS)NFr) ProcessGroup)FakeProcessGroup)rrtorch._C._distributed_c10dr+torch.testing._internal.distributed.fake_pgrr)rrrs r"is_process_groupz%ProcessGroupVariable.is_process_group$sW#//11 5;;;;;;PPPPPPel,<=>>>r#rrIr) r,r-r.r/rarkrSr0rr1r2s@r"rRrRs$;& ; - ;  ;;;;;; ------??\?????r#rRc eZdZdZedededefdZdejj dedededdf fd Z d Z d e ed e eefdeffd ZddZxZS)BackwardHookVariablezU Handles torch.utils.hooks.BackwardHook for module-level backward hooks. module user_hooksuser_pre_hooksc tjstddtf fd }|j|d\}|j|\ }|j|\ }|jd||fi}tjj ddd|j j d<t||||S)Nz6module-level backwards hooks require compiled autogradbw_statec tjjdt jt t|ft jt t|fS)a  Rather than installing the user hooks in the graph (which don't survive AotAutograd), we install hooks that will call trace_wrapped in the backward pass that CompiledAutograd can turn into actual hook calls. N)fnr hooks_name module_name)r*utilshooks BackwardHook functoolspartialr r )rruser_hooks_nameuser_pre_hooks_names r"_in_graph_bw_hooksz7BackwardHookVariable.create.._in_graph_bw_hooks?sy;$11%%@!)#2$/ %%@!)#6$/  r#modrlr example_value)rcompiled_autograd_enabledr routputadd_backward_state_hook create_proxyr*rrrnodemetar) rIrrrrbw_state_proxy_proxyrrrs @@@r"rvzBackwardHookVariable.create5s!: T R S S S         :')i&G&GPU&V&V# ^!#!B!B>!R!RQY>>zJJ &&         ,1;+<+I+I$PRTV+W+W (#E6:~NNNr#rrNc rtjdi|||_||_||_||_dS)Nr)rrrrrr)rrrrroptionsr!s r"rzBackwardHookVariable.__init__hsE ##7###  $,r#c|jSr%)rr's r"as_proxyzBackwardHookVariable.as_proxyvrbr#rcr cx|dvr|j||g|Ri|St||||S)N)setup_input_hooksetup_output_hook) _setup_hookrrkrs r"rkz BackwardHookVariable.call_methodysT < < <#4#B>t>>>v>> >ww""2tT6:::r#rIrc ddlm}|||jd|||fiS)Nr) wrap_fx_proxyrk)builderrrrr)rrIhook_method_namercrs r"rz BackwardHookVariable._setup_hooksa******}  I " " $--//2      r#r)r,r-r.r/r0rrvr*fxProxyrrrrrUrkrr1r2s@r"rr/s- 0O0O$0O( 0O0O0O\0Od -x~ -  -$ - ( -  - - - - - - ;?# ; S/)* ;  ; ; ; ; ; ;         r#r)(rr6typingrrrr*%torch.fx.experimental._backward_staterrrr _trace_wrapped_higher_order_opr excr external_utilsr guardsr rrOrrrrMrconstantrtorch._dynamo.symbolic_convertrrr9rBrDrWrjrrRrrr#r"rs,,,,,,,,,, ??????++++++++::::::BBBBBB00000000!!!!!!&&&&&&EDDDDDD00000/0004EEERRR.-----0---. 7 7 7 7 70 7 7 7F=;=;=;=;=;+=;=;=;@&;&;&;&;&;,&;&;&;R8?8?8?8?8?.8?8?8?v` ` ` ` ` ?` ` ` ` ` r#