קg1ddlZddlmZmZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZgdZGdd eZGd d eZGd d eZGddeZGddeZGddeZeGddZGddZGddeZGddZeegefZGddZeZdS)N)ABCabstractmethod) dataclass)AnyCallableClassVarDictOptional)Store) get_free_port) RendezvousClosedErrorRendezvousConnectionErrorRendezvousErrorRendezvousGracefulExitErrorRendezvousHandlerRendezvousHandlerCreatorRendezvousHandlerRegistryRendezvousInfoRendezvousParametersRendezvousStateErrorRendezvousStoreInfoRendezvousTimeoutErrorrendezvous_handler_registryceZdZdZdS)rz/Represents the base type for rendezvous errors.N__name__ __module__ __qualname____doc__d/var/www/html/ai-engine/env/lib/python3.11/site-packages/torch/distributed/elastic/rendezvous/api.pyrr"s9999r!rceZdZdZdS)r z#Raised when a rendezvous is closed.Nrr r!r"r r &s----r!r ceZdZdZdS)rz2Raised when a rendezvous did not complete on time.Nrr r!r"rr*s<<<Raised when the connection to a rendezvous backend has failed.Nrr r!r"rr.sHHHHr!rceZdZdZdS)rz1Raised when the state of a rendezvous is corrupt.Nrr r!r"rr2s;;;;r!rceZdZdZdS)rzRaised when node wasn't not included in rendezvous and gracefully exits. Exception is a mechanism to exit the stack, however does not mean a failure. Nrr r!r"rr6sr!rc eZdZUdZdZeeed<dZeeed<eed<e ed<e de d e d e ed dfd Z d S)rzNStore address and port that can be used to bootstrap trainer distributed comms MASTER_ADDRMASTER_ADDR_KEY MASTER_PORTMASTER_PORT_KEY master_addr master_portrankstore local_addrreturncF|dkr|ptj}t}|tj|d|tjt|d| tj d}t| tj d}t ||S)aFactory method, finds unused new port on rank0 host and addr/port info with all ranks. If master_addr/master_port is knowns (useful when sharing existing tcp store server) use the constructor. Args: rank: rank of the current node store: store to use for rendezvous local_addr: address of the current node, if not provided will be resolved from hostname rzUTF-8)encoding)r-r.) socketgetfqdn_get_free_portsetrr*encoder,strgetdecodeint)r/r0r1addrports r"buildzRendezvousStoreInfo.buildFs 1991!1!1D!##D II)94;;PW;;X;X Y Y Y II)93t99;K;KU\;K;];] ^ ^ ^yy,<==DDgDVV II)9 : : A A7 A S S  #tFFFFr!N)rrrrr*rr:__annotations__r,r= staticmethodr r r@r r!r"rr=sXX%2OXc]222%2OXc]222GGG-5c]G GGG\GGGr!rceZdZdZdedededefdZedefdZ edefd Z edefd Z ede efd Z d S) rz+Holds the information about the rendezvous.r0r/ world_sizebootstrap_store_infoc>||_||_||_||_dSN)_store_rank _world_size_bootstrap_store_info)selfr0r/rDrEs r"__init__zRendezvousInfo.__init__ds(  %%9"""r!r2c|jS)z(Store used by torchelastic control plane)rHrLs r"r0zRendezvousInfo.storeps {r!c|jS)zRank within a group)rIrOs r"r/zRendezvousInfo.rankus zr!c|jS)zGlobal group size)rJrOs r"rDzRendezvousInfo.world_sizezs r!c|jS)zOStore information that can used by trainer code to bootstrap distributed comms.)rKrOs r"rEz#RendezvousInfo.bootstrap_store_infos ))r!N)rrrrr r=rrMpropertyr0r/rDr rEr r!r"rras55 : : : : 2 : : : :uXcX C   X *h/B&C***X***r!rceZdZdZedefdZedefdZ ede fdZ edefdZ edZ edefdZedefd Zedefd Zd S) rzMain rendezvous interface. Note: Distributed Torch users normally **do not** need to implement their own ``RendezvousHandler``. An implementation based on C10d Store is already provided, and is recommended for most users. r2cdS)z*Return the name of the rendezvous backend.Nr rOs r" get_backendzRendezvousHandler.get_backendr!cdS)aIndicates that store reference returned by :py:meth:`next_rendezvous` can be shared with user applications and will be available during application lifecyle. Rendezous handler impl will share store details as instance of :py:class:`RendezvousStoreInfo`. Applications as a convention use `MASTER_ADDR`/`MASTER_PORT` env variables to lookup the store. Fr rOs r"use_agent_storez!RendezvousHandler.use_agent_stores ur!cdS)aMain entry-point into the rendezvous barrier. Blocks until the rendezvous is complete and the current process is included in the formed worker group, or a timeout occurs, or the rendezvous was marked closed. Returns: Instance of :py:class:`RendezvousInfo`. Raises: RendezvousClosedError: The rendezvous is closed. RendezvousConnectionError: The connection to the rendezvous backend has failed. RendezvousStateError: The rendezvous state is corrupt. RendezvousTimeoutError: The rendezvous did not complete on time. Nr rOs r"next_rendezvousz!RendezvousHandler.next_rendezvousrWr!cdS)aCheck whether the rendezvous has been closed. A closed rendezvous means all future attempts to re-rendezvous within same job will fail. ``is_closed()`` and :py:meth:`set_closed` have semantics of eventual propagation and should not be used for synchronization. The intention is that if at least one node decides the job is finished, it will close the rendezvous, and other nodes will soon observe this and stop running as well. Nr rOs r" is_closedzRendezvousHandler.is_closedrWr!cdS)zMark the rendezvous as closed.Nr rOs r" set_closedzRendezvousHandler.set_closedrWr!cdS)aWReturn the number of nodes who arrived late at the rendezvous barrier, hence were not included in the current worker group. Callers should periodically call this method to check whether new nodes are waiting to join the job and if so admit them by calling :py:meth:`next_rendezvous()` (re-rendezvous). Nr rOs r"num_nodes_waitingz#RendezvousHandler.num_nodes_waitingrWr!cdS)aReturn the run id of the rendezvous. The run id is a user-defined id that uniquely identifies an instance of a distributed application. It typically maps to a job id and is used to allow nodes to join the correct distributed application. Nr rOs r" get_run_idzRendezvousHandler.get_run_idrWr!cdS)a Close all resources that were open for the rendezvous. Example:: rdzv_handler = ... try: store, rank, world_size = rdzv_handler.next_rendezvous() finally: rdzv_handler.shutdown() Nr rOs r"shutdownzRendezvousHandler.shutdownrWr!N)rrrrrr:rVrSboolrYrr[r]r_r=rarcrer r!r"rrsJ9S999^9X    ^ *  4    ^  --^- 3   ^  C   ^   $    ^    r!rceZdZdZ ddedededededeef d Zdd ed ed efd Z dd ed ee d ee fdZ dd ed eed eefdZ dS)ra|Hold the parameters to construct a :py:class:`RendezvousHandler`. Args: backend: The name of the backend to use to handle the rendezvous. endpoint: The endpoint of the rendezvous, usually in form [:]. run_id: The id of the rendezvous. min_nodes: The minimum number of nodes to admit to the rendezvous. max_nodes: The maximum number of nodes to admit to the rendezvous. local_addr: The address of the local node. **kwargs: Additional parameters for the specified backend. Nbackendendpointrun_id min_nodes max_nodesr1c |std|dkrtd|d||krtd|d|d||_||_||_||_||_||_||_dS)N7The rendezvous backend name must be a non-empty string.z(The minimum number of rendezvous nodes (z) must be greater than zero.z(The maximum number of rendezvous nodes (zK) must be greater than or equal to the minimum number of rendezvous nodes (z).) ValueErrorrhrirjrkrlconfigr1)rLrhrirjrkrlr1kwargss r"rMzRendezvousParameters.__init__s XVWW W q==b9bbb  y R9RRDMRRR      "" $r!keydefaultr2c8|j||S)zAReturn the value for ``key`` if ``key`` exists, else ``default``.)rqr;)rLrsrts r"r;zRendezvousParameters.gets{sG,,,r!c\|||}|t|tr|St|tr|dkrdS|dkrdSnEt|tr0|dvrdS|dvrdSt d|d ) z+Return the value for ``key`` as a ``bool``.NroTrF)1truetyesy)0falsefnon%The rendezvous configuration option 'z+' does not represent a valid boolean value.)r; isinstancerfr=r:lowerrp)rLrsrtvalues r" get_as_boolz RendezvousParameters.get_as_boolsg&& =Jud33=L eS ! ! zztzzu s # # {{}} >>>t{{}} >>>u dC d d d   r!c|||}||S t|S#t$r}td|d|d}~wwxYw)z+Return the value for ``key`` as an ``int``.Nrz+' does not represent a valid integer value.)r;r=rp)rLrsrtres r" get_as_intzRendezvousParameters.get_as_int/s|g&& =L u::      s+ AA  ArG) rrrrr:r=r rMrr;rfrrr r!r"rrs4%)%%%% %  %  %SM%%%%>--s-S-C----  s Xd^ xPT~    &  c HSM Xc]      r!rcZeZdZUdZeeefed<d dZdededdfdZ d e de fd Z dS) rz?Represent a registry of :py:class:`RendezvousHandler` backends. _registryr2Nci|_dSrG)rrOs r"rMz"RendezvousHandlerRegistry.__init__Es r!rhcreatorc |std |j|}n#t$rd}YnwxYw|||krtd|d|d|d||j|<dS)zRegister a new rendezvous backend. Args: backend: The name of the backend. creator: The callback to invoke to construct the :py:class:`RendezvousHandler`. rnNThe rendezvous backend 'z' cannot be registered with 'z$' as it is already registered with ''.)rprKeyError)rLrhrcurrent_creators r"registerz"RendezvousHandlerRegistry.registerHs XVWW W #"nW5OO # # #"OOO #  &?g+E+EC7CCQXCC/>CCC  #*ws ! 00paramscH |j|j}n8#t$r+}td|jd|jjd|d}~wwxYw||}||jkr-td|d|jd|S)z+Create a new :py:class:`RendezvousHandler`.rz-' is not registered. Did you forget to call `z`?Nz(' does not match the requested backend 'r)rrhrrprrrV RuntimeError)rLrrrhandlers r"create_handlerz(RendezvousHandlerRegistry.create_handlercs nV^4GG   76>77 M2777   '&//    FN 2 2/7+>+>+@+@//"N///  s A &AA )r2N) rrrrr r:rrArMrrrrr r!r"rr@sIIC112222**.F*4****6%9>Or!r) r5abcrr dataclassesrtypingrrrr r torch.distributedr +torch.distributed.elastic.utils.distributedr r7__all__ Exceptionrr rrrrrrrrrrrr r!r"rs ########!!!!!!::::::::::::::######WWWWWW   ":::::i:::.....O...=====_===IIIIIIII<<<<