קgdZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z m Z ddl m Z ddlmZmZddlmZmZmZdZejejeejeZGd d ZGd d Zd ZdZdZ ddZ!dZ"edkr9e de Z#e!e#e#$Z%e"e%dSdS)a This is a script for launching PyTorch inference on Intel(R) Xeon(R) Scalable Processors with optimal configurations. Single instance inference, multi-instance inference are enabled. Note: term "instance" here doesn't refer to a cloud instance. This script is executed as a single process. It invokes multiple "instances" which are formed from multiple threads for each. "instance" is kind of group of threads in this context. Illustrated as below: :: +-----------------------------+----------------------+-------+ | process | thread | core | +=============================+======================+=======+ | torch.backends.xeon.run_cpu | instance 0: thread 0 | 0 | | | thread 1 | 1 | | +----------------------+-------+ | | instance 1: thread 0 | 2 | | | thread 1 | 3 | | +----------------------+-------+ | | ... | ... | | +----------------------+-------+ | | instance N: thread 0 | M | | | thread 1 | M+1 | +-----------------------------+----------------------+-------+ To get the peak performance on Intel(R) Xeon(R) Scalable Processors, the script optimizes the configuration of thread and memory management. For thread management, the script configures thread affinity and the preload of Intel OMP library. For memory management, it configures NUMA binding and preload optimized memory allocation library (e.g. tcmalloc, jemalloc). Environment variables that will be set by this script: +------------------+-------------------------------------------------------------------------------------------------+ | Environ Variable | Value | +==================+=================================================================================================+ | LD_PRELOAD | Depending on knobs you set, /libiomp5.so, /libjemalloc.so, /libtcmalloc.so might | | | be appended to LD_PRELOAD. | +------------------+-------------------------------------------------------------------------------------------------+ | KMP_AFFINITY | If libiomp5.so is preloaded, KMP_AFFINITY could be set to "granularity=fine,compact,1,0". | +------------------+-------------------------------------------------------------------------------------------------+ | KMP_BLOCKTIME | If libiomp5.so is preloaded, KMP_BLOCKTIME is set to "1". | +------------------+-------------------------------------------------------------------------------------------------+ | OMP_NUM_THREADS | value of ncores_per_instance | +------------------+-------------------------------------------------------------------------------------------------+ | MALLOC_CONF | If libjemalloc.so is preloaded, MALLOC_CONF will be set to | | | "oversize_threshold:1,background_thread:true,metadata_thp:auto". | +------------------+-------------------------------------------------------------------------------------------------+ *Note*: This script respects environment variables set preliminarily. I.e. If you set the environment variables mentioned above before running the script, the script will not overwrite the values in the script. How to use this module: ~~~~~~~~~~~~~~~~~~~~~~~ Single instance inference ------------------------- 1. Run single-instance inference on a single node with all CPU nodes. :: python -m torch.backends.xeon.run_cpu --throughput-mode script.py args 2. Run single-instance inference on a single CPU node. :: python -m torch.backends.xeon.run_cpu --node-id 1 script.py args Multi-instance inference ------------------------ 1. Multi-instance By default this tool runs one process per node. If you want to set the instance numbers and core per instance, --ninstances and --ncores-per-instance should be set. :: python -m torch.backends.xeon.run_cpu -- python_script args eg: on an Intel(R) Xeon(R) Scalable Processor with 14 instance, 4 cores per instance :: python -m torch.backends.xeon.run_cpu --ninstances 14 --ncores-per-instance 4 python_script args 2. Run single-instance inference among multiple instances. By default, runs all ninstances. If you want to independently run a single instance among ninstances, specify rank. eg: run 0th instance on an Intel(R) Xeon(R) Scalable Processor with 2 instance (i.e., numactl -C 0-27) :: python -m torch.backends.xeon.run_cpu --ninstances 2 --rank 0 python_script args eg: run 1st instance on an Intel(R) Xeon(R) Scalable Processor with 2 instance (i.e., numactl -C 28-55) :: python -m torch.backends.xeon.run_cpu --ninstances 2 --rank 1 python_script args eg: run 0th instance on an Intel(R) Xeon(R) Scalable Processor with 2 instance, 2 cores per instance, first four cores (i.e., numactl -C 0-1) :: python -m torch.backends.xeon.run_cpu --core-list "0, 1, 2, 3" --ninstances 2 --ncores-per-instance 2 --rank 0 python_script args 3. To look up what optional arguments this module offers: :: python -m torch.backends.xeon.run_cpu --help Memory allocator ---------------- "--enable-tcmalloc" and "--enable-jemalloc" can be used to enable different memory allcator. N)ArgumentParserRawTextHelpFormatter REMAINDER) expanduser)DictList)DefaultLogsSpecsstart_processesStdz4%(asctime)s - %(name)s - %(levelname)s - %(message)s)levelformatcDeZdZdZd dZdZdZdZdZdZ d Z d Z d S) _CPUinfoz=Get CPU information, such as cores list and NUMA information.cFg|_tjdvr#ttjdtjdkrG|dkr.ddg}t j|dd }n|d }|D]m}d }tj||}|rR|j | d  d nttd |jDd z|_g|_g|_i|_i|_t'|jD]+}g}g} |jD]} | ddkr| dnd} |t| krt| d |vrR| t| d t||jt| d <| t| dt||jt| d<|j ||j | +dSdS)NWindowsDarwin is not supported!!!Linuxrlscpuz--parse=CPU,Core,Socket,NodeT)universal_newlines z^([\d]+,[\d]+,[\d]+,[\d]?),c3&K|] }|dV dS)N).0lines W/var/www/html/ai-engine/env/lib/python3.11/site-packages/torch/backends/xeon/run_cpu.py z$_CPUinfo.__init__..s&$F$FT!W$F$F$F$F$F$Fr0r)cpuinfoplatformsystem RuntimeError subprocess check_outputsplitresearchappendgroupstripintmax node_numsnode_physical_coresnode_logical_coresphysical_core_node_maplogical_core_node_maprange) self test_input lscpu_cmd lscpu_infor pattern regex_outnode_idcur_node_physical_corecur_node_logical_corer%nids r!__init__z_CPUinfo.__init__s ?   5 5 5(/"3"3IIIJJ J _  ' ) )R$&DE '4$%++ (--d33 # O O7Igt44 OL'' (:(:(@(@(B(B(H(H(M(MNNN!$F$F$F$F$F!F!FGG!KDN8:D $79D #*,D ')+D & 00 F F)+&(*%#|SSG(/ b(8(8'!**cC#c((**wqz??2HHH299#gaj//JJJKNw<= 1)r7r.rFloggerwarningstrr()r9 core_listcores_numa_mapnuma_idscorenuma_ids r!numa_aware_checkz_CPUinfo.numa_aware_checks3 ) )D$T*Gh&&((( x==1   NNCIH     x==A  N r#Nr) __name__ __module__ __qualname____doc__rCrHrJrPrRrYr[rerr#r!rrsGG1F1F1F1FfPPPNNN111000  r#rcveZdZdZdeddZddZdZd Z dd Z ddZ dZ ddZ dZ dS) _LauncherzClass for launcher.zUnable to find the {0} library file lib{1}.so in $CONDA_PREFIX/lib or $VIRTUAL_ENV/lib or /.local/lib/ or /usr/local/lib/ or /usr/local/lib64/ or /usr/lib or /usr/lib64 or ~zD/.local/lib/ so the LD_PRELOAD environment variable will not be set.returnNc,t|_dSrT)rr%rGs r!rCz_Launcher.__init__szz r#cg}dtjvr(|tjdddtjvr(|tjdd|tdddddd gz }d }d }tjd d d D]}|d|drd}n |s|D]}tj|d|d}tj |}t|dkrX|dtjd d g} tj d| Dtjd <d}n|p|S)z&Enable TCMalloc/JeMalloc/intel OpenMP. CONDA_PREFIXz/lib VIRTUAL_ENVrmz /.local/libz/usr/local/libz/usr/local/lib64z/usr/libz /usr/lib64F LD_PRELOADr:libz.soTrcPg|]#}||tj$Sr)r0ospathsep)rps r! z-_Launcher.add_lib_preload...s+GGGQG,,GGGr#) rwenvironr.rgetenvr+endswithpathjoinglobrFrx) r9lib_type library_pathslib_findlib_setitemlib_path library_filematches ld_preloadss r!add_lib_preloadz_Launcher.add_lib_preloads RZ ' '  BJ~$>!D!D!D E E E BJ & &  BJ}$=!C!C!C D D D# + + +       IlB//55c::  D}}0800011   )  !w||H6IH6I6I6IJJ )L11w<>I 3.aaaszz&'::;;;; 0111#QLL1OPPP _c222 &&&&&r#c  g}d}d}|jrd|jdD}|jdkrtd|jdkrX|j|jzt |kr8t d|j|jzt |nBt ||jz|_n$|jrG|j dkr |j |j }n`|j }d}nD|j dkr |j |j }n|j }|js3|jdkr(|jdkrd|_t ||_n|jr|jdkr|jdkr d|_nn|jdkri|jdkr^|jt |kr(td t |d |jd t ||jz|_n|jdkr|jdkr|jst ||jz|_nt |j jd }||jz}|j|kr7t d ||jt'jdnR|d kr8t dt ||jz|_n|jdkrt dd}t-}||zt |krA||||z|z ||z|dz }||zt |kAt1t-||z }t ||jzd ksJt ||jz|_n/|j|jzt |krtd|jrVt dd|_|j }t ||jz|_|jr`t d|j j|_|j }t ||jz|_|jdkr1|jdkr&t d|j|j|jso|} | sY|js$t dd|_d}n.t dt'jd|jsd}||j|j ||j!|j"|j#d} i} i} i} tItJj&'dd}tItJj&'dd }tQ|jD]}g}d}|jr|r|jsdg}n|rdg}tS|}|jdkr|||jz|dz|jz}n'||j|jz|jdz|jz}g}|dkr7t |}||z}|dks Jd|||z||dzz}|D]k}t |d kr||d}|*|0||ddz dkr ||dd<Q||d}|*|l|D]}||dd |dd}|dd}|jsmd!|d"}d+d#|j ,|D}|d$|z }|-|n/|r-d%|d"}|-||j. }|r4|*t&j/|*d&|j0r|*d'|*|j1|-|j2d"+|}t || dkr|d } |d =tg|| |<i| |<thj5| |<|jdkrntm|j7| | | tq|j9| ()}|:dS)*NTFc,g|]}t|Sr)r1)rxs r!rzz$_Launcher.launch..s???SVV???r#rzRplease specify the "--ncores-per-instance" if you have pass the --core-list paramsrzGonly first %s cores will be used, but you specify %s cores in core_listz there are z total cores but you specify z8 ninstances; please make sure ninstances <= total_cores)rzthere are %s core(s) per socket, but you specify %s ncores_per_instance and skip_cross_node_cores. Please make sure --ncores-per-instance < core(s) per socketzB--skip-cross-node-cores is set, but there are no cross-node cores.zr--skip-cross-node-cores is exclusive to --ninstances. --ninstances won't take effect even if it is set explicitly.z@Please make sure ninstances * ncores_per_instance <= total_coresz--latency-mode is exclusive to --ninstances, --ncores-per-instance, --node-id and --use-logical-core. They won't take effect even they are set explicitly.z--throughput-mode is exclusive to --ninstances, --ncores-per-instance, --node-id and --use-logical-core. They won't take effect even they are set explicitly.z"assigning %s cores for instance %szCore binding with numactl is not available. Disabling numactl and using taskset instead. This may affect performance in multi-socket system; please use numactl if memory binding is needed.zCore binding with numactl is not available, and --disable_taskset is set. Please unset --disable_taskset to use taskset instead of numactl.rLOCAL_WORLD_SIZE LOCAL_RANKrtasksetz3At least one core needs to be assigned to each rank)startendrr-z-C  c,g|]}t|Sr)r_)rrds r!rzz$_Launcher.launch..s. ' LLr#z-m z-c z-ur)log_dirtee)name entrypointargsenvs logs_specs);r`r+rr( ninstancesrFr]r^use_logical_corer?r%rRr[rPrYmulti_instancethroughput_modeskip_cross_node_coresr4sysexitrsetupdaterN latency_moder3rankdisable_numactlrdisable_tasksetrrrrrr1rwr{getr8sortedr.rrerU no_python executablemoduleprogram program_argstupler ALLr log_file_prefix_DefaultLogsSpecslog_pathwait)r9rrXrenable_tasksetncore_per_nodenum_leftover_coresileftover_coresrr launch_args launch_envs launch_tee local_size local_rankrcur_process_coresr` core_rangestotal_num_corescores_per_rankrc range_elemr numa_paramsrbtaskset_params with_pythoncmd_sctxs r!launchz_Launcher.launchs  >v I??T^%9%9#%>%>???E'2--"h!##,t>UKK',t>JJ #&e**0H"H$ B<2%% L?? MMEE L>>@@E(-$$<2%% L@@NNEE L??AAE'D Or)),22"#+.u::((#= Or)),22'+$$)R//DOr4I4I?SZZ//&-SZZ--do--- 035zzT_/LD,,)R//DOr4I4I1)Q&)%jjD4L&LDOO%()I!)L%M%MN)7$:R)R&/.@@+ 4  +q00 `+.e**8P*P ?b00"NN!1 ),,q0CJJ>>*11 %$2Q$6&8%9;IA;M%N!" FA-q0CJJ>>!%SZZ.%@ A A"5zzD,DDIIII*-e**8P*P?T%==E JJ&Z  IJ,-( ;;=="%e**0H"H# IJ#',"8 ;;==+.u::+H( ?Q  49?? KK4(     # ! $ 9 9 ; ; $ !+ !NNy,0D(%)NNNNWHRLLL# "!N ++  $       &      ')  (:A>>??  a8899 t''N N AC " '8 7>8 7+&$+CC#&$+Cu IOO %2367!e2633!II !& 236:i!m2633!I +- >>&))nnO%4 %BN&!+++L,++ )&3n UV6WW!I& ; ;D;''1,,/3D%A%A #**:6666+b/%"88A==59KOE2237)E)EJ'..z::::$WWA+<(Vaj(V(V1U8(V(V(V%%$5crc$:!+ 7"<(9"<"<"*** 4   { ! 4   JJt| $ $ $ JJt( ) ) )HHSMME KK   R V A"3ZZKNKNGJqMyB%!(JOOO      r#)rnN)TFFrf)FTTFF)rgrhrirjrrrCrrrrrrrrr#r!rlrlsW CWWW""""!#!#!#F!!!$RW4444lIIII # # #*#''''> }}}}}r#rlc|d}|ddddd|dd ddd |d d ddd dS)NzMemory Allocator Parametersz--enable-tcmallocz--enable_tcmalloc store_trueFzEnable tcmalloc allocatoractiondefaulthelpz--enable-jemallocz--enable_jemalloczEnable jemalloc allocatorz--use-default-allocatorz--use_default_allocatorzUse default memory allocatoradd_argument_group add_argumentparserr/s r!_add_memory_allocator_paramsr s  % %&C D DE  (   (  !! + r#c$|d}|ddddtd|dddtd |d d d d d|dddtd|ddd d d|ddd d d|ddddtd|ddd d d|dd d d d!|d"d#d d d$|d%d&ddtd'|d(d)dd*td+|d,d-dd.td/dS)0NzMulti-instance Parametersz--ncores-per-instancez--ncores_per_instancerzCores per instance)metavarrtyperz --ninstanceszOFor multi-instance, you should give the cores number you used for per instance.z--skip-cross-node-coresz--skip_cross_node_coresrFz;If specified --ncores-per-instance, skips cross-node cores.rz--rankz-1a Specify instance index to assign ncores_per_instance for rank; otherwise ncores_per_instance will be assigned sequentially to ninstances. Please refer to https://github.com/intel/intel-extension-for-pytorch/blob/master/docs/tutorials/performance_tuning/launch_script.mdz--latency-modez--latency_modez9By default 4 core per instance and use all physical coresz--throughput-modez--throughput_modez;By default one instance per node and use all physical coresz --node-idz --node_idz=node id for multi-instance, by default all nodes will be usedz--use-logical-corez--use_logical_corezWhether only use physical coresz--disable-numactlz--disable_numactlzDisable numactlz--disable-tasksetz--disable_tasksetzDisable tasksetz --core-listz --core_listzYSpecify the core list as "core_id, core_id, ....", otherwise, all the cores will be used.z --log-pathz --log_pathrzOThe log file directory. Default path is , which means disable logging to files.z--log-file-prefixz--log_file_prefixrzlog file prefix)r r r1r_r s r!_add_multi_instance_paramsrs  % %&A B BE   !    ^  !! J   u   H   J    L   .          h   2     r#cd|d}|ddddddS)NzIOMP Parametersz--disable-iompz--disable_iomprFzIBy default, we use Intel OpenMP and libiomp5.so will be add to LD_PRELOADrrr s r!_add_kmp_iomp_paramsr6sL  % %&7 8 8E  X r#cn|ddddd|ddddd |d d ddd t|t|t||dtd|dt dS)zE Parse the command line options. @retval ArgumentParser z--multi-instancez--multi_instancerFz7Enable multi-instance, by default one instance per noderrz--modulezwChanges each process to interpret the launch script as a python module, executing with the same behavior as"python -m".)rrrz --no-pythonz --no_pythonzyDo not prepend the --program script with "python" - just exec it directly. Useful when the script is not a Python script.rz`The full path to the program/script to be launched. followed by all the arguments for the script)rrr)nargsN)r r rrr_r)r s r! create_argsrAs   F     F !(((   v&&&  7 i88888r#cttj}t jdvr#t t jd|jrtj|jdntj |_|j r|j rt d|j s)|j dst ddtjvrg}tjd}|d D]Z}t!j|}t#|d kr||?t&d |[t#|d kr#d |tjd<nd tjd<t-}||t1ttj|z D].}t&d |tj|/dS)NrrT)exist_okz>Either args.latency_mode or args.throughput_mode should be setz.pyz>For non Python script, you should use "--no-python" parameter.rsrtrz.%s doesn't exist. Removing it from LD_PRELOAD.rr)rrwr{keysr&r'r(rmakedirsdevnullrrrrr}r+rrFr.r]r^rrlrrdebug)r env_before lst_valid tmp_ldpreloadrrlauncherrs r!mainr"rsRZ__&&''J111ho//EEEFFF }# DMD11111    T1  L    > $,"7"7">">  L   rz!!  <0 !'',, W WDiooG7||a  &&&&OQUVVVV y>>A  '*xx ':':BJ| $ $')BJ| ${{H OOD C ))**Z7 8 800 WaA////00r#__main__aThis is a script for launching PyTorch inference on Intel(R) Xeon(R) Scalable Processors with optimal configurations. Single instance inference, multi-instance inference are enable. To get the peak performance on Intel(R) Xeon(R) Scalable Processors, the script optimizes the configuration of thread and memory management. For thread management, the script configures thread affinity and the preload of Intel OMP library. For memory management, it configures NUMA binding and preload optimized memory allocation library (e.g. tcmalloc, jemalloc) ################################# Basic usage ############################# 1. single instance >>> python -m torch.backends.xeon.run_cpu python_script args 2. multi-instance >>> python -m torch.backends.xeon.run_cpu --ninstances xxx --ncores-per-instance xx python_script args ############################################################################# ) descriptionformatter_classrT)&rjrloggingrwr&r,r)rargparserrros.pathrtypingrr)torch.distributed.elastic.multiprocessingr rr r format_str basicConfigINFO getLoggerrgr]rrlr rrrr"r  parse_argsrrr#r!r0s&zzx    DDDDDDDDDDD ',z::::  8 $ $oooooooodjjjjjjjjZ 4eeeP.9.9.9.9b&0&0&0R z ^ ]-F"K     DDJJJJJ)r#