ge(ddlZddlZddlmZmZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZmZmZmZe rddlZd Zd Zd ZdS) N) SAGEMAKER_PARALLEL_EC2_INSTANCESTORCH_DYNAMO_MODES)ComputeEnvironmentSageMakerDistributedType)is_boto3_available)SageMakerConfig)DYNAMO_BACKENDS _ask_field _ask_options_convert_dynamo_backend_convert_mixed_precision#_convert_sagemaker_distributed_mode_convert_yes_no_to_boolcftjd}ddddiddgd} ||tj|d  ddgd d dgd}|||dtj|d dS#|jj$rtd|dYdSwxYw)Niamz 2012-10-17AllowServicezsagemaker.amazonaws.comzsts:AssumeRole)Effect PrincipalAction)Version Statement)indent)RoleNameAssumeRolePolicyDocument)z sagemaker:*zecr:GetDownloadUrlForLayerzecr:BatchGetImagezecr:BatchCheckLayerAvailabilityzecr:GetAuthorizationTokenzcloudwatch:PutMetricDatazcloudwatch:GetMetricDatazcloudwatch:GetMetricStatisticszcloudwatch:ListMetricszlogs:CreateLogGroupzlogs:CreateLogStreamzlogs:DescribeLogStreamszlogs:PutLogEventszlogs:GetLogEventszs3:CreateBucketz s3:ListBucketzs3:GetBucketLocationz s3:GetObjectz s3:PutObject*)rrResource_policy_permission)r PolicyNamePolicyDocumentzrole z# already exists. Using existing one) boto3client create_rolejsondumpsput_role_policy exceptionsEntityAlreadyExistsExceptionprint) role_name iam_clientsagemaker_trust_policypolicy_documents `/var/www/html/ai-engine/env/lib/python3.11/site-packages/accelerate/commands/config/sagemaker.py_create_iam_role_for_sagemakerr2&sBe$$J i9R-S_o p p  *FDZcd9e9e9e    $&*!$/  > ""#777:oa@@@ #      =FFF DiDDDEEEEEEFsA&B "B0/B0cntjd}||ddS)Nr)rRoleArn)r$r%get_role)r-r.s r1_get_iam_role_arnr7\s2e$$J     2 26 :5 AActdddgt}d}|dkr!tdd}|tjd<nKt d td }|tjd <td }|tjd <tdd}|tjd<tdddgt}|dkrtd}n$d}t d|dt |tdtdd}d}|rtdd}tdtdd} d} | rtdd } td!tdd} d} | rtd"d#} td$d%d&gt} i}td'tdd}|rd(}td)d*tDtd+||d,z<td-tdd}|r[td.td/d||d0z<td1tdd||d2z<td3tdd||d4z<d5}| tj krt|td6}n|d7z }t|d8d9}d}| tj krtd:tdd}d;}| tjtjfvrtdt"}|r|d?krt d@t%|t&j| d|||||||| | |AS)BNzHow do you want to authorize?z AWS Profilez7Credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) rz'Enter your AWS Profile name: [default] default)r: AWS_PROFILEzNote you will need to provide AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY when you launch you training script with,`accelerate launch --aws_access_key_id XXX --aws_secret_access_key YYY`zAWS Access Key ID: AWS_ACCESS_KEY_IDzAWS Secret Access Key: AWS_SECRET_ACCESS_KEYz"Enter your AWS Region: [us-east-1]z us-east-1AWS_DEFAULT_REGIONzMDo you already have an IAM Role for executing Amazon SageMaker Training Jobs?zProvide IAM Role namez%Create new IAM role using credentialszEnter your IAM role name: #accelerate_sagemaker_execution_rolez$Accelerate will create an iam role "z " using the provided credentialsz2Do you want to use custom Docker image? [yes/NO]: FzPlease enter yes or no.)r: error_messagezEnter your Docker image: cDt|SNstrlowerxs r1z%get_sagemaker_input..sQr8zODo you want to provide SageMaker input channels with data locations? [yes/NO]: z\Enter the path to the SageMaker inputs TSV file with columns (channel_name, data_location): cDt|SrBrCrFs r1rHz%get_sagemaker_input..c!ffllnnr8z3Do you want to enable SageMaker metrics? [yes/NO]: z[Enter the path to the SageMaker metrics TSV file with columns (metric_name, metric_regex): cDt|SrBrCrFs r1rHz%get_sagemaker_input..rJr8zWhat is the distributed mode?zNo distributed trainingzData parallelismz?Do you wish to optimize your script with torch dynamo?[yes/NO]:dynamo_z+Which dynamo backend would you like to use?c6g|]}|S)rE).0rGs r1 z'get_sagemaker_input..s 0 0 01QWWYY 0 0 0r8rbackendzGDo you want to customize the defaults sent to torch.compile? [yes/NO]: zWhich mode do you want to use?c6tt|SrB)rintrFs r1rHz%get_sagemaker_input..s,SVV4r8modez\Do you want the fullgraph mode or it is ok to break model into several subgraphs? [yes/NO]: use_fullgraphz7Do you want to enable dynamic shape tracing? [yes/NO]: use_dynamicz:Which EC2 instance type you want to use for your training?c6tt|SrB)rrSrFs r1rHz%get_sagemaker_input..sLlmpqrmsmsLtr8z? [ml.p3.2xlarge]:cDt|SrBrCrFs r1rHz%get_sagemaker_input..sSVV\\^^r8z ml.p3.2xlargezShould distributed operations be checked while running for errors? This can avoid timeout issues but will be slower. [yes/NO]: r z(How many machines do you want use? [1]: z2Do you wish to use FP16 or BF16 (mixed precision)?)nofp16bf16fp8rYzTorch dynamo used without mixed precision requires TF32 to be efficient. Accelerate will enable it by default when launching your scripts.) image_uricompute_environmentdistributed_typeuse_cpu dynamo_configec2_instance_typeprofileregion iam_role_namemixed_precision num_machinessagemaker_inputs_filesagemaker_metrics_filedebug)r rSr osenvironr,r2rrr rrrNOr DATA_PARALLELMODEL_PARALLELrr rAMAZON_SAGEMAKER)credentials_configuration aws_profileaws_access_key_idaws_secret_access_key aws_regionrole_managementreis_custom_docker_image docker_imageis_sagemaker_inputs_enabledrhis_sagemaker_metrics_enabledrir_ra use_dynamoprefixuse_custom_optionsec2_instance_queryrbrjrgrfs r1get_sagemaker_inputras ,' QR !! K A%% !JT]^^^ $/ =!!  V   ''<==*; &' *+D E E.C *+@+VVVJ'1BJ#$"W "IJ O !"#?@@ =  d]dddeee&}555'</  LY!"=?W?WXX ",Y/ ### !" * j $ $! !  $.=/ $$$ "# !+ i $ $" "  $' "$67+ MI/ J !,8 9 0 0 0 0 0 # - - - fy() ( U #3      -90"44! ...M&6/ * 7An'7 777M&?2 3 5?I'7 555M&=0 1 V3666(  @BtBt   22&'9;S;S]lmmm E3666 N #3    L4BD\Dklll! 6    #<%%% O  o--  Y    .?)#+#'!35   r8)r'rkutils.constantsrrutils.dataclassesrr utils.importsr config_argsr config_utilsr r r rrrrr$r2r7rrNr8r1rs&  SSSSSSSSMMMMMMMM//////((((((LLL3F3F3FlBBB jjjjjr8