g@ d dl Z d dlZd dlmZ d dlZddlmZ ddlmZm Z m
Z
m Z ddl m
Z
ddlmZ dd lmZ ee Zd
Zd Zdd
ZddZd dZd dZd dZd dZd de de de fdZ d de de de de fdZ dS ) N)Path )
get_logger )FSDP_MODEL_NAMEOPTIMIZER_NAMESAFE_WEIGHTS_NAME WEIGHTS_NAME)
is_peft_model)save)is_torch_versionc ^ dt j vrdt j d< dt j d< dS )z[
Enables RAM efficient loading of Hugging Face models for FSDP in the environment.
ACCELERATE_USE_FSDPTrue FSDP_CPU_RAM_EFFICIENT_LOADINGNosenviron W/var/www/html/ai-engine/env/lib/python3.11/site-packages/accelerate/utils/fsdp_utils.py!enable_fsdp_ram_efficient_loadingr s1
BJ..,2
()39BJ/000r c $ dt j d< dS )z\
Disables RAM efficient loading of Hugging Face models for FSDP in the environment.
Falser Nr r r r "disable_fsdp_ram_efficient_loadingr ( s 4;BJ/000r Fc | |r't | rddlm} || | j S | S )Nr )get_peft_model_state_dict adapter_name)r peftr active_adapter
state_dict)model adapter_onlyr s r _get_model_state_dictr% / sV "
e,, "222222((U=QRRRR !!!r c |r(t | rddlm} || || j S | | S )Nr )set_peft_model_state_dictr )r r r' r! load_state_dict)r# r" r$ r' s r _set_model_state_dictr) 8 sZ 1
e,, 1222222((
I]^^^^$$Z000r c dd l mc m} ddlm} ddlm} ddlm} t j |d | j
| j k r#|j dk }
|
| j
_ |
| j
_ |
|| j
| j
| j 5 t# || } | j
| j k r|dk r
t$ d n t$ d
| d } t j || }
|j dk rOt, d |
t1 j | |
t, d |
nn| j
| j k r|dk rt$ d
|j d nt$ d
| d
|j d } t j || }
t, d |
t1 j | |
t, d |
n| j
| j k rt j |t$ d
| }t j |d t, d | d| i} | | | | | t, d | d d d d S # 1 swxY w Y d S )Nr DefaultSavePlannerFullyShardedDataParallel
StateDictTypeTexist_okr r$ .bin_zSaving model to zModel saved to _rankr# r" storage_writerplanner) torch.distributed.checkpoint distributed
checkpoint,torch.distributed.checkpoint.default_plannerr, 2torch.distributed.fsdp.fully_sharded_data_parallelr. r0 r makedirsstate_dict_typeFULL_STATE_DICT
num_processesstate_dict_configoffload_to_cpu
rank0_onlyoptim_state_dict_configr% r pathjoin
process_indexloggerinfotorchr LOCAL_STATE_DICTSHARDED_STATE_DICTsave_state_dictFileSystemWriter) fsdp_plugin acceleratorr#
output_dir model_indexr$ dist_cpr, FSDPr0 is_multi_processr" weights_nameoutput_model_fileckpt_dirs r save_fsdp_modelr[ A s} 222222222OOOOOOccccccPPPPPPK
T****"m&CCC '4q87G %43C %0
{*K,I;Kn
6 6 +5|LLL
&-*G G G7Ba7G7Go3333P_MsMsbmMsMsMsL " Z F F (A-- B/@ B BCCC
:'8999 A.? A ABBB
(M,J
J
J !## #HH)BHHHH'[[+[[K+<>> ? ? ? Jz#4 5 5 5 KK=*;== > > > >
(M,L
L
Lw||J?0R0R[0R0RSSH K4 0 0 0 0 KK5855 6 6 6!:.J # #% & 7 7 A A**,,
$
KK4(44 5 5 5A 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6s H)KK Kc dd l mc m} ddlm} ddlm} ddlm} | | j | j
k r#|j dk }
|
| j _
|
| j _ | || j | j | j 5 | j | j
k rt! | |ur/|j dk r$| j st' d d d d d S |dk r
t( dn t( d| d} t* j || } t0 d | t5 j | }
t0 d
| nz| j | j k r|dk rt( d |j dnt( d| d |j d} t* j || } t0 d | t5 j | }
t0 d
| n| j | j k rt( |vr*t* j |t( d| n|}t0 d | d t= ||
i}
| |
| | | |
d }
t0 d
| tC ||
|
}d d d n # 1 swxY w Y |S )Nr )DefaultLoadPlannerr- r/ r zzSet the `sync_module_states` flag to `True` so that model states are synced across processes when initializing FSDP objectr4 r5 zLoading model from zModel loaded from r6 r# r3 )r" storage_readerr9 )"r: r; r< r= r] r> r. r0 wait_for_everyoner@ rA rB rC rD rE rF typerI sync_module_states
ValueErrorr r rG rH rJ rK rL loadrM rN r% r( FileSystemReaderr) )rQ rR r# input_dirrT r$ rU r] rV r0 rW rX input_model_filer" rZ load_results r load_fsdp_modelrh s s
222222222OOOOOOccccccPPPPPP!!###"m&CCC '4q87G %43C %0
{*K,I;Kn
)Z )Z &-*G G GE{{$&&;+D+I+I"5 $3 )Z )Z )Z )Z )Z )Z )Z )Z 8Ca7G7Go3333P_MsMsbmMsMsMsL !w||I| D D KK@.>@@ A A A $455J KK?-=?? @ @ @ @
(M,J
J
J !## #HH)BHHHH'[[+[[K@@ A A A $455J KK?-=?? @ @ @ @
(M,L
L
L &'y88 Y?(J(J[(J(JKKK
KK8h88 9 9 9!#8\#Z#Z#Z[J # #% & 7 7 A A**,,
$
$G,J KK7X77 8 8 8+E:LYYY S)Z )Z )Z )Z )Z )Z )Z )Z )Z )Z )Z )Z )Z )Z )ZT s
AL HL LLc dd l mc m} ddlm} ddlm} ddlm} t j |d |
|| j
| j | j 5 |
|| }
| j
| j k r|j dk r|dk r
t dn t d| d} t j || } t& d | t+ j |
| t& d
| nt j |t d| }
t j |
d t& d |
| d |
i| |
| t& d
|
d d d d S # 1 swxY w Y d S )
Nr r+ r- r/ Tr1 r4 r5 zSaving Optimizer state to zOptimizer state saved in optimizerr7 )r: r; r< r= r, r> r. r0 r r? r@ rC rF optim_state_dictrA rI r rG rH rJ rK rL r rO rP )rQ rR rj r# rS optimizer_indexrU r, rV r0 optim_stateoptim_state_nameoutput_optimizer_filerZ s r save_fsdp_optimizerrp s 222222222OOOOOOccccccPPPPPPK
T****
{*K,I;Kn
@ @ ++E9== &-*G G G(A--/>!/C/C~++++NIrIr]lIrIrIr ! )+ ZAQ(R(R% P9N P PQQQ
;(=>>> O8M O OPPPw||J>0U0UO0U0UVVH K4 0 0 0 0 KK?X?? @ @ @ # # '5 & 7 7 A A**,,
$
KK>H>> ? ? ?-@ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @ @s E.GGGc dd l mc m} ddlm} ddlm} ddlm}
| | || j | j
| j 5 | j |
j k rd } |j
dk s | j j s|dk r
t dn t d| d} t j || }
t& d|
t+ j |
} t& d|
nt |vr*t j |t d| n|}t& d | |t/ ||
d | | } | d } t& d
| | ||| }| | d d d d S # 1 swxY w Y d S )Nr )!load_sharded_optimizer_state_dictr- r/ r4 r5 z Loading Optimizer state from z Optimizer state loaded from zLoading Optimizer from r3 rj )model_state_dict
optimizer_keyr^ zOptimizer loaded from )r# optimrk )r: r; r< &torch.distributed.checkpoint.optimizerrr r> r. r0 r_ r@ rC rF rA rI rE r r rG rH rJ rK rL rc r% rd optim_state_dict_to_loadr( )rQ rR rj r# re rl r$ rU rr rV r0 rm optimizer_nameinput_optimizer_filerZ
flattened_osds r load_fsdp_optimizerr{ s 222222222XXXXXXccccccPPPPPP!!###
{*K,I;Kn
1 1 &-*G G G K(A--[5X5c-/>!/C/C~++++NIrIr]lIrIrIr (*w||I~'N'N$ R > Q;O Q QRRR %&i77 Y>(M(MO(M(MNNN
KK<(<< = = =;;!6u=z2.3.0z/`merge_fsdp_weights` requires PyTorch >= 2.3.0`pytorch_model_fsdp_0 optimizer_0zTried to load from z) but couldn't find a valid metadata file.zE However, potential model and optimizer checkpoint directories exist.zPlease pass in either z/pytorch_model_fsdp_0 or z /optimizer_0zinstead.z8 However, a potential model checkpoint directory exists.zPlease try passing in z /pytorch_model_fsdp_0 instead.z< However, a potential optimizer checkpoint directory exists.z/optimizer_0 instead.zMerging FSDP weights from z.Successfully merged FSDP weights and saved to z"Removing old checkpoint directory N)
r accelerate.stater r
rb existsis_main_processrJ rK r shutilrmtreer_ )
r| r r~ r r model_path_existsoptimizer_path_existserrstater} s
r merge_fsdp_weightsr s ( .))N------ D' * * LJKKK " " +.D D L L N N!/-!? G G I I ]N]]] R!6 R Z ZC qNqqUcqqq qC : CC
R M MC ZNZZZ ZCC
" R Q QC QNQQQ QCoo
LNNE * A AABBB=nk[mnn PYPPQQQ * KKM^MM N N N M. ) ) ) r )F)r F)r )T)TF) r r pathlibr rL loggingr constantsr r r r
modelingr otherr versionsr
__name__rJ r r r% r) r[ rh rp r{ strboolr r r r r r s
W W W W W W W W W W W W # # # # # # & & & & & &
H : : :; ; ;" " " "1 1 1 1/6 /6 /6 /6d8 8 8 8v @ @ @ @B$1 $1 $1 $1N c c gk : kp4 4 4 &)4 ?C4 cg4 4 4 4 4 4 r