g/ddlZddlmZmZmZmZmZmZmZm Z m Z m Z m Z m Z ddlmZmZmZmZmZmZddlmZddlmZmZmZmZmZmZmZmZdZ dS) N) ComputeEnvironmentDistributedTypeis_deepspeed_availableis_fp8_availableis_mlu_availableis_mps_availableis_msamp_availableis_musa_availableis_npu_availableis_transformer_engine_availableis_transformers_availableis_xpu_available)DEEPSPEED_MULTINODE_LAUNCHERSFSDP_AUTO_WRAP_POLICYFSDP_BACKWARD_PREFETCHFSDP_SHARDING_STRATEGYFSDP_STATE_DICT_TYPETORCH_DYNAMO_MODES) ClusterConfig)DYNAMO_BACKENDS _ask_field _ask_options_convert_distributed_mode_convert_dynamo_backend_convert_fp8_backend_convert_mixed_precision_convert_yes_no_to_boolc9tdgdt}d}d}d}d}d}d}d}d}d} |tjtjtjtjtjtjfvrtdtd }|dkrtd tt|t}td }td t}td tdd}|stdd }tdtdd} |tjkrtdtdd} n|tjkrd} nd} i} i} | rtdtdd| d<|tjkrytdtdd} | r_tdtd }t j|| d<tdd | d<| sdt)rV|tjtjtjtjtjfvrtdtdd| d<i}tdtdd}|rd}tdd t,Dt.d! ||d"z<td#tdd}|r[td$t0d%d ||d&z<td'tdd||d(z<td)tdd||d*z<| o t3}i}|tjtjtjtjtjtjfvr|std+tdd}|r$tj}t7s Jd,|tjkrytd-tdd}|rtd.td/ |d0<ntd1gd2td! |d3<gd49|d3d!krztd599fd6|d7<td899fd9|d:<|d:d;krtd<|d7d;krtd?td= |d@<tdAtd |dB<tdCtdd}|rtdDt8dE |dF<|d3dGkrtdHtdd|dI<tdJtdd|dK<|dKrt;st=dLtdMtdd}|rtdNt|dO<|dkrdP}t|t>dQ|dR<|dRt>dkr|tdSt|dT<tdUtdd}|rtdVt|dW<tdXtdd}|rtdYt|dZ<i}|tjtjtjtjtjfvrtd[tdd}|r tj }|tj krd\}t|tBd]|d^<td_tdd|d`<da}t|tDdb|dc<|dctDdkr3tddtdd} | stdet|df<n1|dctDdkrtdgtdh |di<dj}!t|!tFdk|dl<dm}"t|"tHdnd! |do<tdptdd|dq<tdrtdd|ds<tdttdd|du<|durd|dv<ntdwtdd|dv<tdxtdd|dy<i}#|tjfvr'tdztdd}$|$r tj%}|tj%krd{}td|tdd}|#|d~z<|#|d~zdkrtdtdd|#|dz<tdtdd}|#|dz<|#|dzdkrtdtdd}|#|dz<tdtdd|#|dz<tdtdd|#|dz<tdt8dE |#|dFz<d}%d}&d}'g}(d})d}*d}+d},d}-|tjtjtjtjtjtjtjfvrst|&dd'dd}.|.dkr|.dz }.n|.dkrd}.n|.dz }.td|.dtdd}}n@|tj tjtj%fvrtdtdd}}nd}|tjkr|dkr|dkrtQd|d|tjtjtjtjtjtjfvrN| sL|sJtSrd}.n$tUrd}.ntWrd}.nd}.td|.dd }d}/|tjtjfvr| s|stdtdd}/d}0|tjkrhd}1tdd }2tdtdd}-|-r8tddd})tddd}+tddd},tdtdd}3|3rtdtdd}4|4r2tddd}&t j,|&}&nTt[dg}%d}5|5r?|%.tdddtdtdd}5|5?tdd &d}*tdd &d}(nd}2|tjkr|rd}1ntdgdt^}1|1dkrtastQdi}0tdddgtb|0d"<|0d"dkr'testQdtdtd |0d<tdtd |0d<tdtd |0d<tdddgdńd |0d<tdtdȬ |0d<tdddgd̈́d |0d<tdtd |0d<|0drMtdtd }6tdtd }7tdtd }8|6|7|8f|0d<nB|0d"dkr6tgstQdզtdddgdلd |0d<|r|1dkr| st[dۦ|tjkr|1dkrtdd }'tididtjj6d|d|d|d|1d|'d|d|d|d|d|2d|0d|d|d|#d| d| d| d|d|d|%d|&d|(d|)d|*d|+d|,d|-d|d| d|/S)Nz$Which type of machine are you using?)zNo distributed trainingz multi-CPUz multi-XPUz multi-GPUz multi-NPUz multi-MLUz multi-MUSATPUrrstaticTFzYHow many different machines will you use (use more than 1 for multi-node training)? [1]: )defaultz!What is the rank of this machine?zGWhat is the IP address of the machine that will host the main process? zDWhat is the port you will use to communicate with the main process? zAre all the machines on the same local network? Answer `no` if nodes are on the cloud and/or on different network hosts [YES/no]: zPlease enter yes or no.)r# error_messagez?What rendezvous backend will you use? ('static', 'c10d', ...): zShould distributed operations be checked while running for errors? This can avoid timeout issues but will be slower. [yes/NO]: zxDo you want to run your training on CPU only (even if a GPU / Apple Silicon / Ascend NPU device is available)? [yes/NO]:zXDo you want to use Intel PyTorch Extension (IPEX) to speed up training on CPU? [yes/NO]:ipexz3Do you want accelerate to launch mpirun? [yes/NO]: zGPlease enter the path to the hostfile to use with mpirun [~/hostfile]: z ~/hostfilempirun_hostfilez/Enter the number of oneCCL worker threads [1]: mpirun_cclzDDo you want to use XPU plugin to speed up training on XPU? [yes/NO]:use_xpuz?Do you wish to optimize your script with torch dynamo?[yes/NO]:dynamo_z+Which dynamo backend would you like to use?c6g|]}|S)lower).0xs ^/var/www/html/ai-engine/env/lib/python3.11/site-packages/accelerate/commands/config/cluster.py z%get_cluster_input..s 0 0 01QWWYY 0 0 0backendzGDo you want to customize the defaults sent to torch.compile? [yes/NO]: zWhich mode do you want to use?c6tt|SN)rintr.s r/z#get_cluster_input..s,SVV4r1modez\Do you want the fullgraph mode or it is ok to break model into several subgraphs? [yes/NO]: use_fullgraphz7Do you want to enable dynamic shape tracing? [yes/NO]: use_dynamicz(Do you want to use DeepSpeed? [yes/NO]: zRDeepSpeed is not installed => run `pip3 install deepspeed` or build it from sourcezDDo you want to specify a json file to a DeepSpeed config? [yes/NO]: z9Please enter the path to the json DeepSpeed config file: nonedeepspeed_config_filez8What should be your DeepSpeed's ZeRO optimization stage?)rrr2r zero_stage)r<cpunvmez"Where to offload optimizer states?c.t|Sr5r6r.deepspeed_devicess r/r8z#get_cluster_input..s[lmpqrmsms[tr1offload_optimizer_devicezWhere to offload parameters?c.t|Sr5rBrCs r/r8z#get_cluster_input..sUfgjklgmgmUnr1offload_param_devicer@z Nvme Path to offload parameters?z/nvmeoffload_param_nvme_pathz&Nvme Path to offload optimizer states?offload_optimizer_nvme_pathzIHow many gradient accumulation steps you're passing in your script? [1]: gradient_accumulation_stepsz0Do you want to use gradient clipping? [yes/NO]: z,What is the gradient clipping value? [1.0]: g?gradient_clippingrzLDo you want to save 16-bit model weights when using ZeRO Stage-3? [yes/NO]: zero3_save_16bit_modelzoDo you want to enable `deepspeed.zero.Init` when using ZeRO Stage-3 for constructing massive models? [yes/NO]: zero3_init_flagzpWhen `zero3_init_flag` is set, it requires Transformers to be installed. Please run `pip3 install transformers`.zCDo you want to enable Mixture-of-Experts training (MoE)? [yes/NO]: zSpecify the comma-separated list of transformers MoE layer class names (case-sensitive), e.g : `MixtralSparseMoeBlock`, `Qwen2MoeSparseMoeBlock`, `JetMoEAttention,JetMoEBlock` ... : deepspeed_moe_layer_cls_namesz*Which Type of launcher do you want to use?c6tt|Sr5)rr6r7s r/r8z#get_cluster_input..Us;CFFCr1deepspeed_multinode_launcheraEDeepSpeed configures multi-node compute resources with hostfile. Each row is of the format `hostname slots=[num_gpus]`, e.g., `localhost slots=2`; for more information please refer official [documentation](https://www.deepspeed.ai/getting-started/#resource-configuration-multi-node). Please specify the location of hostfile: deepspeed_hostfilez:Do you want to specify exclusion filter string? [yes/NO]: z#DeepSpeed exclusion filter string: deepspeed_exclusion_filterz:Do you want to specify inclusion filter string? [yes/NO]: z#DeepSpeed inclusion filter string: deepspeed_inclusion_filterz7Do you want to use FullyShardedDataParallel? [yes/NO]: z&What should be your sharding strategy?c6tt|Sr5)rr6r7s r/r8z#get_cluster_input..0Q8r1fsdp_sharding_strategyzBDo you want to offload parameters and gradients to CPU? [yes/NO]: fsdp_offload_paramsz%What should be your auto wrap policy?c6tt|Sr5)rr6r7s r/r8z#get_cluster_input..s/A7r1fsdp_auto_wrap_policyulDo you want to use the model's `_no_split_modules` to wrap. Only applicable for 🤗 Transformers [yes/NO]: zSpecify the comma-separated list of transformer layer class names (case-sensitive) to wrap ,e.g, :`BertLayer`, `GPTJBlock`, `T5Block`, `BertLayer,BertEmbeddings,BertSelfOutput` ...? : "fsdp_transformer_layer_cls_to_wrapzaWhat should be your FSDP's minimum number of parameters for Default Auto Wrapping Policy? [1e8]: ifsdp_min_num_paramsz4What should be your FSDP's backward prefetch policy?c6tt|Sr5)rr6r7s r/r8z#get_cluster_input..rUr1fsdp_backward_prefetchz+What should be your FSDP's state dict type?c6tt|Sr5)rr6r7s r/r8z#get_cluster_input..s.s1vv6r1fsdp_state_dict_typez@Do you want to enable FSDP's forward prefetch policy? [yes/NO]: fsdp_forward_prefetchzBDo you want to enable FSDP's `use_orig_params` feature? [YES/no]: fsdp_use_orig_paramsuoDo you want to enable CPU RAM efficient model loading? Only applicable for 🤗 Transformers models. [YES/no]: fsdp_cpu_ram_efficient_loadingfsdp_sync_module_stateszsDo you want each individually wrapped FSDP unit to broadcast module parameters from rank 0 at the start? [YES/no]: z?Do you want to enable FSDP activation checkpointing? [yes/NO]: fsdp_activation_checkpointingz+Do you want to use Megatron-LM ? [yes/NO]: megatron_lm_z0What is the Tensor Parallelism degree/size? [1]:zPlease enter an integer. tp_degreez6Do you want to enable Sequence Parallelism? [YES/no]: sequence_parallelismz2What is the Pipeline Parallelism degree/size? [1]: pp_degreez)What is the number of micro-batches? [1]:num_micro_batcheszDDo you want to enable selective activation recomputation? [YES/no]: recompute_activationszzDo you want to use distributed optimizer which shards optimizer state and gradients across data parallel ranks? [YES/no]: use_distributed_optimizerzSWhat is the gradient clipping value based on global L2 Norm (0 to disable)? [1.0]: no.MULTI_r!z coresCPU processesz(s)z How many z. should be used for distributed training? [1]:z=How many GPU(s) should be used for distributed training? [1]:zSpecified distributed type zy but only using 1 GPU on a single machine. Please select `No distributed training` for the type of machine you are using.zNPU(s)zMLU(s)zMUSA(s)zGPU(s)zWhat zV (by id) should be used for training on this machine as a comma-seperated list? [all]:allzcWould you like to enable numa efficiency? (Currently only supported on NVIDIA hardware). [yes/NO]: ziWhat is the name of the function in your script that should be launched in all parallel scripts? [main]: mainz'Are you using a TPU cluster? [yes/NO]: z&What is the name of your TPU cluster? z*Please enter the name of your TPU cluster.z&What is the zone of your TPU cluster? z*Please enter the zone of your TPU cluster.zFTo run a python script in a TPU pod, should `sudo` be used? [yes/NO]: zCDo you have code you wish to run on startup in each pod? [yes/NO]: z1Is this code located in a bash script? [yes/NO]: z&What is the path to your bash script? z*Please enter the path to your bash script.zLPlease enter each command seperately you wish to run on startup in each pod.z(Please enter a single command to be ran zTPlease enter the commands you wish to run on startup in each pod as a single string.z.Do you wish to add another command? [yes/NO]: zqIf not using an instance group, what are the names of the Compute VM instances to be used, seperated by a comma: ,zQWhat environment variables do you wish to set in each pod, seperated by a comma: z#Do you wish to use mixed precision?)rlfp16bf16fp8rwzJFP8 (either Transformer Engine or MSAMP) is not installed on this machine.z%Which FP8 backend do you want to use?temsampTEzITransformersEngine was selected, but it is not installed on this machine.zuDo you want to use FP8 autocast during eval mode? Generally better metrics are found when this is disabled [yes/NO]: use_autocast_during_evalz6What margin should be used for gradient scaling? [0]: marginzVWhat interval should be used for for how often the scaling factor is recomputed? [1]: intervalz#Which weight format should be used?HYBRIDE4M3c|dkrdndS)Nrr~rr+r7s r/r8z#get_cluster_input..sa1ff((&r1 fp8_formatzWWhat length of history should be used for the amax scaling factor computation? [1024]: iamax_history_lengthzGWhich algorithm should be used for the amax scaling factor computation?max most_recentc|dkrdndS)Nrrrr+r7s r/r8z#get_cluster_input..s166%%}r1amax_compute_algorithmz]Do you want to to execute `fprop`, `dgrad`, and `wgrad` GEMMS in higher precision? [yes/NO]: override_linear_precisionz:Should `fprop` be executed in higher precision? [yes/NO]: z:Should `dgrad` be executed in higher precision? [yes/NO]: z:Should `wgrad` be executed in higher precision? [yes/NO]: MSAMPz.s!q&&$$dr1optimization_levelzTorch dynamo used without mixed precision requires TF32 to be efficient. Accelerate will enable it by default when launching your scripts.rvzWShould `torch.float` be cast as `bfloat16` and `torch.double` remain `float32` on TPUs?compute_environmentdistributed_type num_processesgpu_idsmixed_precision downcast_bf16 machine_rank num_machinesmain_process_ipmain_process_portmain_training_function fp8_configdeepspeed_config fsdp_configmegatron_lm_config ipex_config mpirun_configuse_cpu rdzv_backend same_networkcommands command_filetpu_envtpu_nametpu_vmtpu_zone tpu_use_sudotpu_use_cluster dynamo_configdebugenable_cpu_affinityr+)7rrr MULTI_GPU MULTI_MLU MULTI_MUSA MULTI_NPU MULTI_XPU MULTI_CPUrr6listrangerNOstrospath expanduserstriprXLArrrr DEEPSPEEDrfloatr ExceptionrFSDPrrrr MEGATRON_LMsplitreplace ValueErrorr rr abspathprintappendrrrr r rr LOCAL_MACHINE):rrrrrrrrrrrrr use_mpirunr&r use_dynamoprefixuse_custom_optionsuse_mpsr use_deepspeeduse_deepspeed_configuse_gradient_clippinguse_moelauncher_queryis_exclusion_filteris_inclusion_filterruse_fsdpsharding_strategy_queryfsdp_wrap_queryuse_no_split_modulesfsdp_backward_prefetch_queryfsdp_state_dict_type_queryruse_megatron_lm tpu_commandstpu_command_filetpu_downcast_bf16rrrrrr machine_typerrrr run_commandsuse_command_fileanother_commandfpropdgradwgradrDs: @r/get_cluster_inputr6s#.  "  LLMGOLL E!!"!!! " g    !  '3U<(())L )YO!+V!! &U'7 L   )U_g     N #3    ?--- G #3     _6 6 6KMw( f #3    F 8 8 8#E'7 J  w",](### 4673E3EoF[F[F]F]3^3^ /0.89jtu.v.v.v l+        %  %  %    &    ", R #3 " " " IMI/ J !,8 9 0 0 0 0 0 # - - - fy() ( U #3      -90"44 ...M&6/ * 7An'7 777M&?2 3 5?I'7 555M&=0 1k0.00G  %  %  %  %  &       # 6 #3     d.8 &(( d dc d d d 8 8 8#-V'7 $$$ $9 N LL 222 .%<$;$;!#L1Q66CO<>OQtQtQtQtDD$%?@@L68IKnKnKnKn@@$%;<((>?6IIFP>$+GGG()BC ((BCvMMJTD$+KKK()FG CM_CCC !>? )3F+!"; )))% ) 3=B'7 333 . /  12 022#B!U'7 G  DNoEE !@A a!MCO"1CCDD !?@ $$BCGdefGggg=GD  >>$%9:+5T/ %&? +++' +ISAJJ()EF +5T/ %&? +++' +ISAJJ()EF K!!!"!  E #3      4.3  3 3 3&N #4@'&8855K0 1 2<T'7 222K- . FO3?%7744K/ 0 237LQ7OOO'1C+!"; ((($ ,HRqIIK DE 459Nq9QQQ5?w%666 12 ,b (4@,&8855K0 1 *W &2>*$66 333K. / 4>R'7 444K/ 0 3=T'7 333K. / =GB'7 ===K8 9 ;< 9= 5669CJ+ "; ::: 56 >>J9: ,6P !,,,Jx( .8p !...Jz* 0<=!6*@@ ! 000J|, 9Cq $999J45 X *X3$)!!! !+X3$)!!! !+X3$)!!! DI%QVBW #>? *g55-//i()ghhh7CBt :: ! 888J34 o--g-  Y   ?...?f3L3L& eos       .<< )) $m   (  (' "\ "\ ( ,+  65 : *)  K .-   K! "$m# $% &"\' ("\) *+ ,&%- ./ 01 2v3 45 6"\7 8(9 :$m; <e= >0/?  r1)!rutilsrrrrrr r r r r rrutils.constantsrrrrrr config_argsr config_utilsrrrrrrrrrr+r1r/rs"                             '&&&&&                    F F F F F r1