gyhdZddlZddlZddlmZmZddlmZmZddl m Z m Z ddl Z ddl mZmZdd lmZdd lmZmZdd lmZmZmZmZmZer ddlmcmZerdd lm Z d Z!dZ"dZ#dZ$dZ%e!dddZ&dHdZ'dZ(dZ)dZ*dZ+dZ,dZ-dZ.dZ/Gdde0Z1d Z2d!Z3e2d"Z4d#e fd$Z5d#e fd%Z6dId&Z7dJd(Z8e j9de j:de j;d)e j<d*e j=d+e j>d,e j?d-e j@d.e jAd/e jBd0i ZCd1eCDDZEd2ZFdKd3e jGfd4ZHe2dId5eIfd6ZJdId5eIfd7ZKdLd8ZLdId9ZMGd:d;eNZOe3dMd<ZPdId=ZQe2dNd@ZRdAZSGdBdCZTdDZUdEZVedOdGZWdS)PzB A set of basic tensor ops compatible with tpu, gpu, and multigpu N)contextmanager nullcontext)update_wrapperwraps)AnyMapping)AcceleratorState PartialState)!TORCH_DISTRIBUTED_OPERATION_TYPES)DistributedTypeTensorInformation)is_npu_availableis_torch_distributed_availableis_torch_versionis_torch_xla_availableis_xpu_available)ReduceOpc6t|tjSN) isinstancetorchTensortensors W/var/www/html/ai-engine/env/lib/python3.11/site-packages/accelerate/utils/operations.pyis_torch_tensorr-s fel + ++c t|tjjtjjtjjtjjtjjtjjtjj Sr) rrxpu FloatTensor ByteTensor IntTensor LongTensor HalfTensor DoubleTensorBFloat16Tensorrs ris_torch_xpu_tensorr)1sO            rc,t|tSr)rr tensor_infos ris_tensor_informationr->s k#4 5 55rclt|tot|dot|dS)z Checks if `data` is a `namedtuple` or not. Can have false positives, but only if a user is trying to mimic a `namedtuple` perfectly. _asdict_fields)rtuplehasattrdatas r is_namedtupler5Bs3 dE " " \wtY'?'? \GDR[D\D\\rct|rt|t|St||S)zO Cast a generator to the same type as obj (list, tuple, or namedtuple) )r5typelist)obj generators r honor_typer;JsC S$tCyy$y//**tCyy###rF test_typeerror_on_other_typec t|ttfr t|fd|DSt|tr:t |fd|DS|r |gRiSr0tdt |djdjd|S)ad Recursively apply a function on a data structure that is a nested list/tuple/dictionary of a given base type. Args: func (`callable`): The function to recursively apply. data (nested list/tuple/dictionary of `main_type`): The data on which to apply `func` *args: Positional arguments that will be passed to `func` when applied on the unpacked data. main_type (`type`, *optional*, defaults to `torch.Tensor`): The base type of the objects to which apply `func`. error_on_other_type (`bool`, *optional*, defaults to `False`): Whether to return an error or not if after unpacking `data`, we get on an object that is not of type `main_type`. If `False`, the function will leave objects of types different than `main_type` unchanged. **kwargs (additional keyword arguments, *optional*): Keyword arguments that will be passed to `func` when applied on the unpacked data. Returns: The same data structure as `data` with `func` applied to every object of type `main_type`. c3@K|]}t|gRdVdS)r<Nrecursively_apply).0oargsr>funckwargsr=s r z$recursively_apply..nsm  "!".7M`dj      rc @i|]\}}|t|gRdS)r<rA)rCkvrEr>rFrGr=s r z%recursively_apply..wsf   Aq$!".7M`dj   rzUnsupported types (z ) passed to `z?`. Only nested list/tuple/dicts of objects that are valid for `z` should be passed.) rr1r8r;rr7items TypeError__name__)rFr4r=r>rErGs` ````rrBrBUsw,$ &&                  D' " " tDzz        !JJLL        4 tD*4***6***   S$t** S S4= S S+4+= S S S    Krcrt|st|drdkrddkrd |S#t$r|cYSt$r^}t rt trdn+trt trdn|Yd }~nd }~wwxYw |S#t$r|cYSwxYwt |ttfrt|fd |DSt |trUt trgngt|fd |DS|S) a Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to send to a given device. device (`torch.device`): The device to send the data to. Returns: The same data structure as `tensor` with all tensors sent to the proper device. tonpuznpu:0r!zxpu:0) non_blockingznpu:zxpu:Nc3>K|]}t|VdS)rS skip_keysNsend_to_device)rCtdevicerSrVs rrHz!send_to_device..s6oocd^AvLT]^^^oooooorc Ji|]\}}||vr|nt| S)rUrW)rCrJrYrZrSrVs rrLz"send_to_device..sN   AqY11N1fS_kt,u,u,u   r)rr2rQrNAssertionErrorrrintrr1r8r;rstrr7rM)rrZrSrVerrors ``` rrXrXs6v)'&$"7"7) U??F U??F 99V,9?? ? % % %99V$$ $ $ $    !! fc**-,F__F!## fc**-,F__F   %99V,9?? ? % % %99V$$ $ $ $ % FUDM * * oooooohnooo    FG $ $  i % % " II  ItF||      "LLNN       s0A C, C5ACCC..DDc(d}t||S)aK Recursively gathers the information needed to rebuild a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with [`~utils.TensorInformation`] instead of tensors. c8t|j|jS)N)shapedtype)rrbrcrs r_get_data_structurez/get_data_structure.._get_data_structures v|6<HHHHrrA)r4rds rget_data_structureres'III 0$ 7 77rc(d}t||S)a: Recursively gathers the shape of a nested list/tuple/dictionary of tensors as a list. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with lists of tensor shapes instead of tensors. c*t|jSr)r8rbrs r _get_shapezget_shape.._get_shapesFL!!!rrA)r4rhs r get_shaperis#""" Z . ..rc6d}t||tS)z Recursively initializes tensors from a nested list/tuple/dictionary of [`~utils.TensorInformation`]. Returns: The same data structure as `data` with tensors instead of [`~utils.TensorInformation`]. c8tj|jd|jiSNrc)remptyrbrcr+s r_initialize_tensorz.initialize_tensors.._initialize_tensors{K-G[5FGGGrr=)rBr-)data_structurerns rinitialize_tensorsrqs-HHH /K` a a aarc"t|tttfr3t |dkr t dt |dt|ttfrt|dSt|tr.|D]}t||cSn:t|tj s tdt |d|j dS)a Recursively finds the batch size in a nested list/tuple/dictionary of lists of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. rz&Cannot find the batch size from empty .z0Can only find the batch size of tensors but got ) rr1r8rlen ValueErrorr7find_batch_sizekeysrrrNrb)r4rJs rrvrvs$g.//QSYY!^^O$t**OOOPPP$ &&ZtAw''' D' " "Z , ,A"47++ + + + , el + +ZX4PT::XXXYYY :a=rcT t|S#ttf$rYnwxYwdS)a Same as [`utils.operations.find_batch_size`] except will ignore if `ValueError` and `TypeErrors` are raised Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. N)rvrurNr3s rignorant_find_batch_sizery s? t$$$  "     4s %%c(d}t||S)aS Recursively finds tensors in a nested list/tuple/dictionary and converts them to a list of numbers. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to convert to regular numbers. Returns: The same data structure as `data` with lists of numbers instead of `torch.Tensor`. c|}|jtjkr|tj}|Sr)detachcpurcrbfloat16rQfloat32tolistrs r_convert_to_listz!listify.._convert_to_list&sM$$&& <5> ) )YYu}--F}}rrA)r4rs rlistifyrs$ -t 4 44rcVd}t||d}tj|S)Nc|jdkr|d}|s|}t j|S)Nr)ndimclone is_contiguous contiguousxm all_gatherrs r_tpu_gather_onez$_tpu_gather.._tpu_gather_one3sX ;!  \\^^D)F##%% )&&((F}V$$$rTr>)rBr mark_step)rrress r _tpu_gatherr2s7%%% OV N N NCLNNN Jrcttddrtjjntjjfd}t ||dS)Nz>=z1.13cNjdkrdsjzjdkrot jjzj j }||j dg ddRSfdtjD}t j|t j|dS)NrgloorcrZr c8g|]}tjS)r empty_like)rC_rs r z8_gpu_gather.._gpu_gather_one..`s$[[[1e.v66[[[rdim)rrrrbackendrrm num_processesnumelrcrZviewsizerange distributedrcat)routput_tensors gather_opstates` r_gpu_gather_onez$_gpu_gather.._gpu_gather_oneHs( ;!  \\^^D)F##%% )&&((F = $&)@)@ #[#fllnn4l|N Inf - - -&>&r>FKKMM!"",=>>> > \[[[eFY@Z@Z[[[N   ( ( @ @ @9^333 3rTr)r rrrall_gather_into_tensor_all_gather_baserB)rrrrs @@r _gpu_gatherrAsl NNEf%%7%< %6 4444448 _f$ O O OOrceZdZdZdS)DistributedOperationExceptionz An exception class for distributed operations. Raised if the operation cannot be performed due to the shape of the tensors. N)rO __module__ __qualname____doc__rrrrrgs  Drrc<tfd}|S)zv Verifies that `tensor` is the same shape across all processes. Only ran if `PartialState().debug` is `True`. c  tjtjkstjs|i|Sjdj}d|vr |d}n|d}tjjt|jkrUtd|d|jjdtjjdtjjd|d t|}t|g}|do| |dt|k}|sAd d t!|D}td |d ||i|S) Nrsrrz%One or more of the tensors passed to z were not on the z+ while the `Accelerator` is configured for z. Please move it to the z before calling z - c$g|] \}}d|d|S)zProcess z: r)rCirbs rrz5verify_operation..wrapper..s.2m2m2mxqRW3Ja3J3J53J3J2m2m2mrznCannot apply desired operation due to shape mismatches. All shapes across devices must be valid. Operation: `z` Input shapes: - )r distributed_typerNOdebugrrOrZr7 find_devicerri gather_objectcountrtjoin enumerate) rErG operationrshapesoutputare_sameprocess_shape_strfunctions rwrapperz!verify_operation..wrapperus >> *o.@ @ @ H\ @8T,V,, ,*@@X->@@ v  H%FF!WF >> %V)<)<)A A A/b bbTZTaTfbbT`TbTbTiTnbb)5)>)CbbU^bbb 6""x(( !9 ||F1I..#f++=H $,MM2m2m[dek[l[l2m2m2m$n$n!3\'0\\HY\\ x((((rrrrs` rverify_operationrps5  8__))))_)4 Nrc<tfd}|S)z Checks that `verify_operation` failed and if so reports a more helpful error chaining the existing `DistributedOperationException`. c |i|S#t$r*}jdj}td|d|d}~wwxYw)NrszError found while calling `z1`. Please see the earlier error for more details.)rrrO)rErGerrs rrz"chained_operation..wrappersx 8T,V,, ,,   #.DD1BDDI/jijjj  s ?%:?rrs` rchained_operationrs5  8___ Nrctjtjkrt |Stjt vrt |S|S)a4 Recursively gather tensor in a nested list/tuple/dictionary of tensors from all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. Returns: The same data structure as `tensor` with all tensors sent to the proper device. )r rrXLArr rrs rgatherrsM~~&/*===6"""  (,M M M6""" robjectcdttjD}tj||d|DS)Ncg|]}dSrr)rCrs rrz&_gpu_gather_object..sHHHqdHHHrcg|] }|D]}| Srr)rCyxs rrz&_gpu_gather_object..s% 1 1 1!q 1 1!A 1 1 1 1r)rr rrrall_gather_object)routput_objectss r_gpu_gather_objectrsQHHE,..*F$G$GHHHN ''??? 1 1~ 1 1 11rctjtjkrt dtjt vrt |S|S)a5 Recursively gather object in a nested list/tuple/dictionary of objects from all devices. Args: object (nested list/tuple/dictionary of picklable object): The data to gather. Returns: The same data structure as `object` with all the objects sent to every device. z&gather objects in TPU is not supported)r rrrNotImplementedErrorr r)rs rrrsN~~&/*===!"JKKK  (,M M M!&))) rc0dd}t||d|S)NrcHtj|||S)Nsrc)rr broadcast)rrs r_gpu_broadcast_onez*_gpu_broadcast.._gpu_broadcast_ones# ##F#444 rT)r>rrrA)r4rrs r_gpu_broadcastrs1 /4UX Y Y YYrbroadcast tensorcXt|ttfr)t|fdt |DSt|t r6t |fd|DStj |fdS)Nc3JK|]\}}t|d|VdS)rnameN_tpu_broadcast)rCrrYrs rrHz!_tpu_broadcast..s?"g"gTQPQ>!T--A--#H#H#H"g"g"g"g"g"grc Di|]\}}|t|d|S)rrr)rCrJrKrs rrLz"_tpu_broadcast..s6aaa$!QQq$}}}} E E Eaaarc|Srr)rrs rz _tpu_broadcast..s !C&r) rr8r1r;rrr7rMr mesh_reduce)rrrs ``rrrs&4-((c&"g"g"g"gU^_eUfUf"g"g"ghhh FG $ $ctF||aaaaRXR^R^R`R`aaabbb >$(8(8(8(8 9 99r ci|]\}}|| Srr)rCrJrKs rrLrLsCCC1q!CCCrcd}t}tj|tj|j}|Z|j}t |j}tjt||gzt|dt|dz<t|d}|| }t|ddd }|dd}||fS) ze Grabs the shape of `tensor` only available on one process and returns a tensor of its shape irNrcr sum reductionrr) r rrmr]rZrbTENSOR_TYPE_TO_INTrcrr8rtreducenonzero)rmax_tensor_dimensionr base_tensorrb tensor_dtypercs rgather_tensor_shapers ! NNE+2%)ELYYYK  )&,7 (- T%[[L>5QY\(](](] $c%jj1n$%666Kk11334K  BCC # $ $Ecrc"K  rreturnct}t|\}}|9tj|t||j}t|dS)a Copys a tensor that only exists on a single device and broadcasts it to other devices. Differs from `broadcast` as each worker doesn't need to know its shape when used (and tensor can be `None`) Args: tensor (`torch.tensor`): The tensor that should be sent to all devices. Must only have it be defined on a single device, the rest should be `None`. Nrrr)r rrzerosTENSOR_INT_TO_DTYPErQrZr)rrrbrcs rcopy_tensor_to_devicesr s` NNE&v..LE5 ~U*=e*DEEEHHVV &E * * **r from_processctjtjkrt ||dStjt vrt ||S|S)a Recursively broadcast tensor in a nested list/tuple/dictionary of tensors to all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. from_process (`int`, *optional*, defaults to 0): The process from which to send the data Returns: The same data structure as `tensor` with all tensors broadcasted to the proper device. zaccelerate.utils.broadcast)rrr)r rrrrr r)rrs rrrsZ~~&/*===f,=YZZZZ  (,M M Mf,7777 rc$tjtjkr2t |D]!\}}t jd|fd||<"n;tjtvr!tj ||S)a Broadcast a list of picklable objects form one process to the others. Args: object_list (list of picklable objects): The list of objects to broadcast. This list will be modified inplace. from_process (`int`, *optional*, defaults to 0): The process from which to send the data. Returns: The same list containing the objects from process 0. z&accelerate.utils.broadcast_object_listc|Srr)rrs rrz'broadcast_object_list..Csefgsetrr) r rrrrrrr rrbroadcast_object_list) object_listrrr9s ` rr r 4s~~&/*=== ,, v vFAs^,TVY[t[t[t[tuuKNN v  (,M M M // /NNN rc*d}t|||S)aN Recursively takes a slice in a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to slice. tensor_slice (`slice`): The slice to take. Returns: The same data structure as `data` with all the tensors slices. c||Srr)r tensor_slices r _slice_tensorz$slice_tensors.._slice_tensorWs l##rrA)r4r  process_indexrrs r slice_tensorsrIs%$$$ ]D, ? ??rc BtdttfrCtdfdt t dDStdt rCtdfddDStdtj s%tdtdtj S)a Recursively concatenate the tensors in a nested list/tuple/dictionary of lists of tensors with the same shape. Args: data (nested list/tuple/dictionary of lists of tensors `torch.Tensor`): The data to concatenate. dim (`int`, *optional*, defaults to 0): The dimension on which to concatenate. Returns: The same data structure as `data` with all the tensors concatenated. rc3TK|]!tfdDV"dS)c g|] }| Srr)rCdrs rrz)concatenate...ks0D0D0D!10D0D0DrrN concatenate)rCrr4rs @rrHzconcatenate..ksG#l#lSTK0D0D0D0Dt0D0D0D#$N$N$N#l#l#l#l#l#lrcNi|] tfdD!S)c g|] }| Srr)rCrrJs rrz*concatenate...ms-A-A-Aqad-A-A-Arrr)rCrJr4rs @rrLzconcatenate..ms?dddPQa-A-A-A-AD-A-A-As!K!K!Kdddrz%Can only concatenate tensors but got r) rr1r8r;rrtrr7rwrrrNr)r4rs``rrr]s$q'E4=))Q$q'#l#l#l#l#lX]^abfghbi^j^jXkXk#l#l#lmmm DGW % %QtDG}}dddddUYZ[U\UaUaUcUcdddeee Q . .QOT!W OOPPP 9Ts # # ##rceZdZdS)CannotPadNestedTensorWarningN)rOrrrrrrrssDrrc4dd}t||d|||S)a3 Recursively pad the tensors in a nested list/tuple/dictionary of tensors from all devices to the same size so they can safely be gathered. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. dim (`int`, *optional*, defaults to 0): The dimension on which to pad. pad_index (`int`, *optional*, defaults to 0): The value with which to pad. pad_first (`bool`, *optional*, defaults to `False`): Whether to pad at the beginning or the end. rFc t|ddrtjdt|St |jkr|St j|j|jd}t| }tfd|D |jkr|S|j t } |<| t||z}|r8t fdtt |D}n6t fdtt |D}|||<|S)N is_nestedFzHCannot pad nested tensors without more information. Leaving unprocessed.)rZc3(K|] }|V dSrr)rCsrs rrHzFpad_across_processes.._pad_across_processes..s'--!qv------rc3tK|]2}|krtz ntdV3dSrslice)rCrrmax_sizeold_sizes rrHzFpad_across_processes.._pad_across_processes..s[[\Q#XXh#.9995QU;;rc3nK|]/}|krtdntdV0dSrNr!rCrrr$s rrHzFpad_across_processes.._pad_across_processes..sEooUVqCxxE!Xc]333U4[[oooooor)getattrwarningswarnrrtrbrrrZrr}maxr8 new_zerosr1r) rr pad_index pad_firstrsizesnew_size new_tensorindicesr#r$s ` @@r_pad_across_processesz3pad_across_processes.._pad_across_processess 6; . .  MZ,   M #fl## # #M|FL???Et   ""----u----- v|C( ( (M<>>  %%eHoo66B  p`efijrfsfs`t`tGGoooooZ_`cdl`m`mZnZnoooooG$ 7rT)r>rr-r.rrFrA)rrr-r.r3s rpad_across_processesr5ws>"> v4ST]ir   rc4dd}t||d|||S)z Takes a `tensor` of arbitrary size and pads it so that it can work given `num_processes` needed dimensions. New tensors are just the last input repeated. E.g.: Tensor: ([3,4,4]) Num processes: 4 Expected result shape: ([4,4,4]) rcx ||z}|||zz }||zdkr||z }n|||zz }|||zcxkrdkrnn||z }|j t }||z|d<|t|}t fdt t |D} ||| <|S)Nrr c3nK|]/}|krtdntdV0dSr&r!r's rrHz@pad_input_tensors.._pad_input_tensors..sEkkQR188a#///tkkkkkkr)rbr8r,r1rrt) r batch_sizerr remainder last_inputsto_padr0r1r2r$s ` @r_pad_input_tensorsz-pad_input_tensors.._pad_input_tensorss-/  I $=>  &! + +"Z/FF"jM&ABF & , , , ,1 , , , , , 6)F<>> 6) %%eHoo66 kkkkkV[\_`h\i\iVjVjkkkkk$ 7rT)r>r9rrrrA)rr9rrr=s rpad_input_tensorsr>sA&  #     rmean?c2dd}t||d||S)aX Recursively reduce the tensors in a nested list/tuple/dictionary of lists of tensors across all processes by the mean of a given operation. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to reduce. reduction (`str`, *optional*, defaults to `"mean"`): A reduction method. Can be of "mean", "sum", or "none" scale (`float`, *optional*): A default scaling value to be applied after the reduce, only valied on XLA. Returns: The same data structure as `data` with all the tensors reduced. r?r@ct}|}|jtjkr|S|jtjkrHt jt jt j |g|t jn=|jj tvr*tj |tj|dkr ||jz}|S)Nr?)r rrrrrrr all_reduce REDUCE_SUMvaluer rrrSUMr)rrscaler cloned_tensors r_reduce_across_processesz(reduce.._reduce_across_processess   !_%7 7 7  !_%8 8 8 LNNN M"--% @ @ @ LNNNN  # )-N N N   ( ( E E E    U0 0MrT)r>rrGr?r@rA)rrrGrIs rrrs;$&  &di_d   rc2d}d}t|||S)av Recursively converts the elements nested list/tuple/dictionary of tensors in FP16/BF16 precision to FP32. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to convert from FP16/BF16 to FP32. Returns: The same data structure as `tensor` with all tensors that were in FP16/BF16 precision converted to FP32. c*|Sr)floatrs r_convert_to_fp32z)convert_to_fp32.._convert_to_fp32 s||~~rc~t|st|do|jtjtjfvSrl)rr2rcrfloat16r~rs r_is_fp16_bf16_tensorz-convert_to_fp32.._is_fp16_bf16_tensor s@''C767+C+C  M NZ J  rrorA)rrNrQs rconvert_to_fp32rRs9   -vAU V V VVrc$eZdZdZdZdZdZdS)ConvertOutputsToFp32ad Decorator to apply to a function outputing tensors (like a model forward pass) that ensures the outputs in FP16 precision will be convert back to FP32. Args: model_forward (`Callable`): The function which outputs we want to treat. Returns: The same function as `model_forward` but with converted outputs. c4||_t||dSr) model_forwardr)selfrVs r__init__zConvertOutputsToFp32.__init__#s *t]+++++rc6t|j|i|Sr)rRrV)rWrErGs r__call__zConvertOutputsToFp32.__call__'s#1t14B6BBCCCrc*tjd)NzCannot pickle a prepared model with automatic mixed precision, please unwrap the model with `Accelerator.unwrap_model(model)` before pickling it.)pickle PicklingError)rWs r __getstate__z!ConvertOutputsToFp32.__getstate__*s" `   rN)rOrrrrXrZr^rrrrTrTsN  ,,,DDD     rrTc>tfd}|_|S)Nc|i|Srr)rErGrVs rforwardz(convert_outputs_to_fp32..forward3s}d-f---r)rT __wrapped__)rVras` rconvert_outputs_to_fp32rc0s8(77M.....(G Nrc>t|tr.|D]}t|}||cSdSt|tt fr|D]}t|}||cSdSt|t jr|jSdS)z Finds the device on which a nested dict/list/tuple of tensors lies (assuming they are all on the same device). Args: (nested list/tuple/dictionary of `torch.Tensor`): The data we want to know the device of. N) rrvaluesrr1r8rrrZ)r4r9rZs rrr<s$  ;;==  C %%F! "   D5$- ( (  C %%F! "   D%, ' '{rTc#\Ktjtjks8tj4tjst }n"ddl}|j ||||}|5dVddddS#1swxYwYdS)z Wrapper around `deepspeed.runtime.zero.GatheredParameters`, but if Zero-3 is not enabled, will be a no-op context manager. Nr) modifier_rank fwd_moduleenabled) r rr DEEPSPEEDdeepspeed_pluginis_zero3_init_enabledr deepspeedzeroGatheredParameters)paramsrgrhrigather_param_contextrms rroroQs*o.GGG+7 ""3IIKK 8 +}}(~@@ -JPW A    sB!!B%(B%)FNr)rrr)NNr4rJ)NNT)Xrr\r) contextlibrr functoolsrrtypingrrrrr r constantsr dataclassesrrimportsrrrrrtorch_xla.core.xla_modelcore xla_modelrtorch.distributedrrr)r-r5r;rBrXrerirqrvryrrr ExceptionrrrrrrrrrMdoublehalfr~uint8int8int16int32int64boolrrMrrrrr]rr rr UserWarningrr5r>rrRrTrcrrorrrrs< 22222222++++++++ 22222222888888;;;;;;;;*)))))))))!!##+******,,,   666]]]$$$4CX]00000f6666r888$///$ b b b."555.   #P#P#PL     I      F&&2s2222#&ZZZZ:::: K L! J NA K J K K K J DC(:(@(@(B(BCCC2++5<++++"C*S*@@@@($$$$,     ;   1111h%%%%P&&&&RWWW0        4   *r