קg fUddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZddlmZddgZes:ddlZGd d Zd Zeejd _eejd _dSdd lmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%ej&e'Z(e r( ddl)m*Z*n #e+$re(,dYnwxYwGddej-Z.e.Z/e.e0d<dde1fdZ2GddZddde1de e3dfde e e1dfdefdZdS)N)reduce)chain)DictListOptionalTuple TYPE_CHECKINGUnion) is_available)not_noneinit_device_mesh DeviceMeshceZdZdS)_DeviceMeshStubN)__name__ __module__ __qualname__Y/var/www/html/ai-engine/env/lib/python3.11/site-packages/torch/distributed/device_mesh.pyrrs rrcdSNrrrr_init_device_mesh_stubrs rztorch.distributed.device_mesh) _find_pg_by_ranks_and_tag_get_default_group_get_group_tag get_backendget_process_group_ranksget_rankget_world_sizeinit_process_groupis_initialized new_group ProcessGroup) ArrayLikezCDeviceMesh requires numpy >= 1.21 to be installed for type checkingc xeZdZddZddZdddeedfd eeedfddfd Z dddd e eddfd Z dd Z ddde efdZ ededefdZededefdZddd edefdZ ddedede ejddfdZdeeedffdZddd ededfdZdS)_MeshEnvreturnNcLg|_i|_i|_i|_i|_dSr) mesh_stackchild_to_root_mappingmesh_dim_group_optionsroot_to_flatten_mappingflatten_name_to_root_dimsselfs r__init__z_MeshEnv.__init__As=02DOGID &  'UWD (  * * *rrcjt|jdkrtd|jdS)Nrz#No device mesh is currently active!)lenr* RuntimeErrorr/s rget_current_meshz_MeshEnv.get_current_meshMs34?##q(("#HIII?2& &r device_meshsubmesh_dim_names. submesh_dimscpfd|D}j}g}g}d}t||D]\} } t| dkr|| d|z | d|z }|| d|z |t| dz z }||j| jd|| d|z |j| dtt|j } |D]} | | |j g| |Rj dg|R}  }| D] }tj||d}||vr|}!||_|j|<|S)Ncg|]H}t|dkrtfd|nj|dIS)cnj|j|zSr)meshsize)xyr7s rz5_MeshEnv.create_sub_mesh...^s/!1!6!6q!9!9Kr?).0mesh_dimr7s r z,_MeshEnv.create_sub_mesh..\svx==1$$ TTTT !%**8A;77 rrr<r3) start_dimend_dimFmesh_dim_names _init_backend)r>zipr4flattenappendr-_dim_group_infoslistrangendimremovepermutereshaperr device_typer+)r0r7r8r9slice_dim_size mesh_tensor slice_dim_idxslice_dim_group_infonum_dims_flattenmesh_dim_indices mesh_dim_namemesh_dims_remained_idxidxpg_ranks_by_dimcur_rankmesh_ndsubmesh res_submeshs ` rcreate_sub_meshz_MeshEnv.create_sub_meshRsy!-N&*KM#%  ! 36|EV3W3W  / -'((1,,"-"5"5"21"58H"H 0 47G G#6##K"(()9!).sC$$$+122889PQQ$$$r_cDg|]}tj|Sr)r rI)rCdimris rrEz0_MeshEnv.create_flatten_mesh..s9!!9::3?rz# already exists for submesh of the . z5The mesh_dim_names of submesh and flattened mesh are z-. Please specify another valid mesh_dim_name.r3rI)_mesh_resources get_root_meshr rIjoinr. setdefaultrrOkeysr5r-mathprodr>r?rPrQrRrSrTrrrUr+tuple)r0r7r\flatten_dims_in_rootinvalid_dim_namesflattened_mesh_dim_sizeremained_dims_in_rootflatten_dim_in_rootr_r`raflattened_meshres_flattened_meshris @rcreate_flatten_meshz_MeshEnv.create_flatten_meshs(55kBBI$$$$/7 8R/S/S$$$ !  ##7!!   * 5 5i D D D %!hy78899!/ :??AA!!!  111"$VVVVVCL]CCCT999!T%A)%LLL3I>}MM&*i 0@0E0E0G0G&H&H #$(y~/B)C)C$D$D !'; B B#%,,-@AAAA4in4&)=gb122 !))++H* 8 8!+)$1#3""" w&&)7&=FD &'9 :TfD ( 3 3Ir B B= QGLMaGbGbD *9 5m D% %rcD|j|d}|s|n|Sr)r+get)r0r7ris rrpz_MeshEnv.get_root_meshs+266{DIII&/>;;Y >rc||}|j}|r=|r;t|dks Jd|d}|||SdS)z Returns the index of the mesh dim in the root mesh. The device_mesh passed in needs to be sliced out from the root mesh or submesh of the root mesh. r<z"The submesh can only be a 1D mesh.rN)rprIr4get_mesh_dim_by_name)r0r7richild_mesh_dim_nameschild_mesh_dim_names rget_root_mesh_dimz_MeshEnv.get_root_mesh_dims~ **;77I#.#=  Q1 Q,--2227322&:1&=#00>>G}GGGa[E_aaK6<<]KKLL Lrrlbackend pg_optionsc||f|j|<dSr)r,)r0rlrrs r_set_mesh_dim_group_optionsz$_MeshEnv._set_mesh_dim_group_optionss 18/DD ' , , ,rc( |||krtd|j|i|j|}g|j| t fd|Dst d|d dd}g}|D]}}||vr&||}|d}||n0|j|}||f||krt d|dd|d d |}~|S) z Validate whether the mesh_dim_names is valid for slicing the given device_mesh. If valid, return dim indexes of the slice mesh in the device mesh. z'Cannot create a submesh from a submesh.c3 K|]}|vV dSrr)rCr\valid_mesh_dim_namess r z0_MeshEnv._get_slice_mesh_dims..1s=!!55rzInvalid mesh_dim_names z% specified. Valid mesh_dim_names are .r3z specified. z!Found mesh dim indices to slice: rmz.Mesh dim indices should be in ascending order.) rpr5r.rrrIallrrMrg) r0r7rIr.curr_idxslice_mesh_dimsr\ mesh_indicesnext_idxrs @r_get_slice_mesh_dimsz_MeshEnv._get_slice_mesh_dimssd00===="#LMMM  * 5 5k2 F F F(,(F{(S %$+$*$ %3 HnHH0DHHHH O!/ $ $  $===#<]#KL ,B/H#**<8888*9?? NNH#**H;777x''"N.NNNOOOOOH $" "rc|||}|jd|d|j|}|}g}|D]I}t |j||fd}||vr|j|gng|_| |J|S)z` Return all the submeshes of a given mesh dimension of the device mesh. r3FrH) rr>swapdimsrTr?rrrUrNrM) r0r7r\rDr_r` res_submeshesmesh_1drbs r_get_all_submeshesz_MeshEnv._get_all_submeshesRs 00mLLH).77HEEMMK$))(33O#++--HM* . .$+$1#3"'  7**!1(;<<( $$W---- rr(Nr(rr)r7rr(r)rrrr1r6rstrrintrdrr~rpr staticmethodrrrr$Optionsrrrrrrr'r'@s^     ' ' ' ' N %N  %S#XN uS#X/ N   N N N N bMQ@ &@ &+@ &  E E E E!!56  E   E E E E2 # %S/ "2 #2 #2 #2 #h !+ !>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) rUr>.rINTrHr%rJr(c||_t|tjr"|jjdkrt d|t|tjr2|tj n tj |dtj |_ |rt|nd|_ t|j |_d|_|dkr|r(||t)r*t+dkrt-j|_|j t1k}|ddvsJ|ddkr|dnd|_dSdS) Ncpuz!`mesh` must be a CPU tensor, got dtypedevicerxlathreadedr)rr<)rU isinstancerTensorrtype ValueErrordetachtortensorr>rvrIrLtolist_flatten_mesh_list _thread_id_get_or_create_default_group_init_process_groupsr"r threading get_identrnonzeror?_coordinate_on_dim)r0rUr>rIrJ rank_coordss rr1zDeviceMesh.__init__s +D $ -- M$+2Be2K2K !KT!K!KLLLdEL11G   uy 111\$uEIFFF I numelr5rrUr set_devicerr)r0default_initialized world_size device_handlers rrz'DeviceMesh._get_or_create_default_groupsJ"0"2"2 & %"$$$'))Jy  :--"{{{aeajapaparar{{{/t/?@@M& L= L(5'A'A'C'C$!555"%99Q>>&e%ee2FeeIMIYeee((6J)JKKK%'' 'rc Xg}|jjdkr|jtkrt }t t t}tj r$t|dkrtd|n|}| t|||jfnAt |jjD]&}|jd|d|j|}|D]}|}|t&jvrt&j|\} } nd\} } t|| | }||vrrt-||kr(t/d|d |d | tt1|||jfڐ(||_dS) Nr<gloozcpu:gloo,cuda:nccl)rranksr3)NN)rrrzFEach device mesh dimension should get only one process group, but got z in !)r>rQrr rrOrPrrr rr#rMr group_namerrTr?rror,rr4r5r rN) r0dim_group_infos default_groupr dim_grouprlr_dim_meshsubgroup_ranksrrs rrzDeviceMesh._init_process_groupsskACOy~""ty'8'8N#3#34455z..00'#M22f<<&:%HHHH'   &&&y11!,!00++C'+i&8&8S&A&A&I&IDINN3//''O %4##)1):):/"HHH!0 Fs K ' * 3=/GZ %."0$+'1%%%  ==??n<<"?33c99&2%s  & * * , , , , ,rc|js%d|jd|jdn,d|jd|jd|jd}|S)Nz DeviceMesh('z', )z, mesh_dim_names=)rIrUr>r)r0device_mesh_reprs r__repr__zDeviceMesh.__repr__Bsz*uIt/IIDI4D4D4F4FIIIItD$4tt9I9I9K9Ktt^b^qttt  $ #rct|dd|_|js7t|j|jj|j|j|jf|_|jS)N_hash) rrhashrr>shaperUrIrr/s r__hash__zDeviceMesh.__hash__Js] w55DJ: !/ (+  : rotherc(t|tsdSt|t|krdS|j|jkoI|jj|jjko/|j|jko|j|jko|j|jkS)NFT) rridrr>rrUrIr)r0rs r__eq__zDeviceMesh.__eq__YseZ00 u$xx2e99$$t+u/GG< 5:+;;<(E,==<+u/CC<5+;; rc|jstdt|tr|fn|}||jkr|St||}t|||}|S)aU Slice the current DeviceMesh based on the mesh_dim_names given to create a submesh. The submesh created consists of the dimensions and the communicators indicated by ``mesh_dim_names`` Args: mesh_dim_names (Union[str, Tuple[str]]): the name or the tuple of names of the mesh dimension of the DeviceMesh to create the submesh for. Returns: A :class:`DeviceMesh` object The following program runs on each process/rank in an SPMD manner in a world size of 8. In the first example: Calling mesh_2d["tp"] on rank 0, 1, 2, 3 returns a 1D submesh of DeviceMesh:([0, 1, 2, 3]). Calling mesh_2d["tp"] on rank 4, 5, 6, 7 returns a 1D submesh of DeviceMesh:([4, 5, 6, 7]). Calling mesh_2d["dp"] on rank 0, 4 returns a 1D submesh of DeviceMesh:([0, 4]). Calling mesh_2d["dp"] on rank 1, 5 returns a 1D submesh of DeviceMesh:([1, 5]). Calling mesh_2d["dp"] on rank 2, 6 returns a 1D submesh of DeviceMesh:([2, 6]). Calling mesh_2d["dp"] on rank 3, 7 returns a 1D submesh of DeviceMesh:([3, 7]). In the second example: Calling mesh_3d["dp", "cp"] on rank 0, 1, 4, 5 returns a 2D submesh of DeviceMesh:([[0, 1], [4, 5]]). Calling mesh_3d["dp", "cp"] on rank 2, 3, 6, 7 returns a 2D submesh of DeviceMesh:([[2, 3], [6, 7]]). Calling mesh_3d["cp", "dp"] on rank 0, 1, 4, 5 returns a 2D submesh of DeviceMesh:([[0, 4], [1, 5]]). Calling mesh_3d["cp", "dp"] on rank 2, 3, 6, 7 returns a 2D submesh of DeviceMesh:([[2, 6], [3, 7]]). Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize a 2D device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh_2d = init_device_mesh(device_type="cuda", (2,4), mesh_dim_names=("dp", "tp")) >>> tp_mesh = mesh_2d["tp"] >>> dp_mesh = mesh_2d["dp"] >>> >>> # Initialize a 3D mesh. >>> mesh_3d = init_device_mesh(device_type="cuda", (2,2,2), mesh_dim_names=("dp", "pp", "cp")) >>> # The order of the mesh_dim_names provided deteremines the order of dimensions in the submesh. >>> dp_cp_mesh = mesh_3d["dp", "cp"] >>> cp_dp_mesh = mesh_3d["cp", "dp"] z1Cannot slice a DeviceMesh without mesh_dim_names!)rIr5rrrorrd)r0rIrrbs r __getitem__zDeviceMesh.__getitem__gsZ& X"#VWWW&0%D%DX!!. !444 "1"F"F.##*99./rrDct|dstd|jjdkr!|td|jjddd|jjdkr+|)t t |jd dd St|}tj |d}|rG|| vr1||jd dd }t t |St|trt||n|}t t |j|dd S) a Returns the single ProcessGroup specified by mesh_dim, or, if mesh_dim is not specified and the DeviceMesh is 1-dimensional, returns the only ProcessGroup in the mesh. Args: mesh_dim (str/int, optional): it can be the name of the mesh dimension or the index of the mesh dimension. Default is None. Returns: A :class:`ProcessGroup` object. rNz*DeviceMesh process groups not initialized!r<NFound the DeviceMesh have dimensionsJOptional kwarg `mesh_dim` needs to be specified when device_mesh.ndim > 1.zmIf you want to get the list of all the ProcessGroups in the DeviceMesh,please use `get_all_groups()` instead.r)hasattrr5r>rQr rrNrorpr-rrsrrr)r0rDrir-rs r get_groupzDeviceMesh.get_groups4!344 Q"#OPPPy~!!h&6"LLLL`=y~""x'7-t/DQ/G/KL(55d;;I&5&M&Q&Q4'' #' 87N7S7S7U7U+U+U"9("C"TUV"WXZYZXZ"[ 9? KLLL"(C00"O88xHHH!  -t/DX/NrPQr/RSrcNfdtjjDS)z Returns a list of ProcessGroups for all mesh dimensions. Returns: A list of :class:`ProcessGroup` object. c:g|]}|Sr)r)rCir0s rrEz-DeviceMesh.get_all_groups..s%EEE!DNN1%%EEEr)rPr>rQr/s`rget_all_groupszDeviceMesh.get_all_groupss,FEEEuTY^/D/DEEE Errngroupct|trt|}t|tjr||ks|(||kr"t dt|d|tj|dtj }t|||d}t|||j fg|_ |St|}t|dkrt d |t d t|tjr3|tj d n tj|dtj }|jt|kr5t d |d t|dt|||d}d|D|_ |S)aM Constructs a :class:`DeviceMesh` with ``device_type`` from an existing :class:`ProcessGroup`. The constructed device mesh has number of dimensions equal to the number of groups passed. If more than one group is passed, then the ``mesh`` argument is required. Nz Invalid mesh z for ProcessGroup with ranks rrFrHrz.Expects at least one ProcessGroup to be passedz0Must pass mesh if passing multiple ProcessGroups)rrzEExpects mesh with ndim equal to number of ProcessGroups but got mesh z and z ProcessGroupscVg|]&}t|t||jf'Sr)rrr)rCrs rrEz)DeviceMesh.from_group..sH,,,  #5))+E22$,,,r)rr$rrrrrrrrrrrrNrOr4rrrQ)rrUr>rI group_ranksr7groupss r from_groupzDeviceMesh.from_groups& %.. #5e<< tU\227;{{}} 7S7S&4;+>+>$]D ]]P[]]|KUYOOO(#1"'  $E**K9IJ0 ,#"%[[F6{{a !QRRR| !STTTdEL11G   uy ???\$uEIFFF  yCKK'' L KKMMLL03F LLL%T.PUK,, $ ,,,K ( rcl||jn|j|Sr)r>rr?)r0rDs rr?zDeviceMesh.sizes,(0(849??$$$dinnX>V>V Vrc|jjSr)r>rQr/s rrQzDeviceMesh.ndims 9> !rc4t|jjSr)rvr>rr/s rrzDeviceMesh.shape#s)) )rctS)z: Returns the current global rank. )rr/s rrzDeviceMesh.get_rank's:: rc|jdkr |td|jjdd|d}t||}t |t s Jdtt|S)a{ Returns the local rank of the given mesh_dim of the DeviceMesh. Args: mesh_dim (str/int, optional): it can be the name of the mesh dimension or the index of the mesh dimension. Default is None. Returns: An integer denotes the local rank. The following program runs on each process/rank in an SPMD manner. In this example, we have 2 hosts with 4 GPUs each. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 0, 1, 2, 3 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=0) on rank 4, 5, 6, 7 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 0, 4 would return 0. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 1, 5 would return 1. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 2, 6 would return 2. Calling mesh_2d.get_local_rank(mesh_dim=1) on rank 3, 7 would return 3. Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import DeviceMesh >>> >>> # Initialize device mesh as (2, 4) to represent the topology >>> # of cross-host(dim 0), and within-host (dim 1). >>> mesh = DeviceMesh(device_type="cuda", mesh=[[0, 1, 2, 3],[4, 5, 6, 7]]) r<Nrrrrz1We expect ProcessGroup before calling `get_rank`!)rQr5r>r rrr$r)r0rDmesh_dim_groups rget_local_rankzDeviceMesh.get_local_rank-s8y1}}!1"LLLL`!%dnnX&>&>??N  C CB C C CH^4455 5rc"|jr|jndS)z Return the relative indices of this rank relative to all dimensions of the mesh. If this rank is not part of the mesh, return None. N)rr/s rget_coordinatezDeviceMesh.get_coordinateWs /3.EO4**4 Orr\cd|jstdt||S)a\ Returns a 1D DeviceMesh by flattening the current DeviceMesh. If no mesh_dim_name is provided, the default is a string concatentaing the mesh_dim_names of the given submesh with each mesh_dim_name separated by "_". For example, if we have a 3D mesh DeviceMesh([[[0, 1], [2, 3]], [[4, 5], [6, 7]]], mesh_dim_names=("dp", "cp", "tp")), calling mesh_3d["dp", "cp"]._flatten() will create a 1D submesh DeviceMesh([0, 1, 2, 3], mesh_dim_names=("dp_cp",)) on rank 0, 1, 2, 3 and a 1D submesh DeviceMesh([4, 5, 6, 7], mesh_dim_names=("dp_cp",)) on rank 4, 5, 6, 7. After the flattened dimension is created, to access the flattened dimesnion in mesh_3d, one can use the existing slicing method to obtain the flattened mesh through calling mesh_3d["dp_cp"]. z3Cannot flatten a DeviceMesh without mesh_dim_names!)rIr5ror~)r0r\s r_flattenzDeviceMesh._flatten^s<& "I#66t]KK Krrrr)%rrr__doc__r__annotations__rrrrr boolr1rrrrrrobjectrrrr$rrrrrr?propertyrQrrrrr rrrrrzs% % Nl sCx11119="& ( ( ( (  k12(  %U38_5 (   ( ( ( ( ( T ( ( (:I 4I 4I 4V      - - - - $c $ $ $ $     4    = "'U38_(<"==  = = = = ~, , huS#X&?, <, , , , \ FD$6 F F F F @D<  9= < < < tL'99:< < 5{!:;<<  %U38_5 <   < < <  < | W W# W# W W W W  "# " " "  "  *5c? * * *  * c     ( 6( 68E#s(O+D( 6PS( 6( 6( 6( 6T PHT#Y$7 P P P P L L(3- L< L L L L L Lrrn mesh_shape.rIr(c |tt|t|krtdd|t|t|kr1tddt|dt|d|r(|std|d d t jd 5t jtj|tj  |}dddn #1swxYwYt||| }|S)a Initializes a `DeviceMesh` based on `device_type`, `mesh_shape`, and `mesh_dim_names` parameters. This creates a DeviceMesh with an n-dimensional array layout, where `n` is the length of `mesh_shape`. If `mesh_dim_names` is provided, each dimension is labeled as `mesh_dim_names[i]`. .. note:: `init_device_mesh` follows SPMD programming model, meaning the same PyTorch Python program runs on all processes/ranks in the cluster. Ensure `mesh_shape` (the dimensions of the nD array describing device layout) is identical across all ranks. Inconsistent `mesh_shape` may lead to hanging. .. note:: If no process group is found, init_device_mesh will initialize distributed process group/groups required for distributed communications behind the scene. Args: device_type (str): The device type of the mesh. Currently supports: "cpu", "cuda/cuda-like". Passing in a device type with a GPU index, such as "cuda:0", is not allowed. mesh_shape (Tuple[int]): A tuple defining the dimensions of the multi-dimensional array describing the layout of devices. mesh_dim_names (Tuple[str], optional): A tuple of mesh dimension names to assign to each dimension of the multi-dimensional array describing the layout of devices. Its length must match the length of `mesh_shape`. Each string in `mesh_dim_names` must be unique. Returns: DeviceMesh: A :class:`DeviceMesh` object representing the device layout. Example:: >>> # xdoctest: +SKIP("no rank") >>> from torch.distributed.device_mesh import init_device_mesh >>> >>> mesh_1d = init_device_mesh("cuda", mesh_shape=(8,)) >>> mesh_2d = init_device_mesh("cuda", mesh_shape=(2, 8), mesh_dim_names=("dp", "tp")) Nz"Each mesh_dim_name must be unique.z/Found repeated mesh_dim_name in mesh_dim_names z6mesh_shape and mesh_dim_names should have same length!zFound len(mesh_dim_names): z and len(mesh_shape):rz4Device type with GPU index is not supported but got rmzUIf you maintained a 'torch.device' object, it's recommended to pass in 'device.type'.rr)rUr>rI) r4setr5isalpharrarangertrurviewr)rUrrIr>r7s rr r rsR  %3~&&''3~+>+>>>"8VnVV :#n"5"555"Ln#n2E2Enn\_`j\k\knnn  {2244 V{VVVg \%  Y Y< * 5 5UYGGGLLZXXD Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y #)   sAD$$D(+D()r)4loggingrtr functoolsr itertoolsrtypingrrrrr r rtorch.distributedr torch.utils._typing_utilsr __all__sysrrmodulesrr "torch.distributed.distributed_c10drrrrrrr r!r"r#r$ getLoggerrlogger numpy.typingr% ImportErrorwarninglocalr'ror rrrrrrr&sE DDDDDDDDDDDDDDDD ******...... | ,|~~gJJJ           ?NCK/0;0K'                          W x ( (F  . . . . . . .    NNU       m!m!m!m!m!9?m!m!m!^ !) OX***111111vLvLvLvLvLvLvLvLx59 GGGG#s(OG!sCx1 G  GGGGGGs/B66CC