gL2 R d dl Z d dlmZ d dlmZ d dlZd dlZd dlm Z d dl
m Z ddl m
Z
mZmZmZmZmZmZmZmZmZmZmZmZmZ e r d dlm c m Z ddl m Z dd l!m"Z" e e# Z$ dd e%d
ee& de'de'de'de(de(de de)de)fdZ* d dZ+d de(de)fdZ,d de(fdZ-dS ) N)Path)List)
load_model)
GradScaler )
MODEL_NAMEOPTIMIZER_NAMERNG_STATE_NAMESAFE_MODEL_NAMESAFE_WEIGHTS_NAME SAMPLER_NAME SCALER_NAMESCHEDULER_NAME WEIGHTS_NAMEget_pretty_nameis_mlu_availableis_torch_xla_availableis_xpu_availablesave)
get_logger) PartialStateFT
output_dir model_states
optimizers
schedulers dataloaders
process_indexstepscalersave_on_each_nodesafe_serializationc
t | } t | D ]z\ }
} | st nt } |
dk r| dd|
d } | | }
t
| |
|| t d|
{t | D ]{\ }
}| } |
dk r
t dn t d|
d}| | }t
| ||d t d| |t | D ]{\ }
}| } |
dk r
t dn t d|
d}| | }t
| ||d t d | |t | D ] \ }
}|
dk r
t dn t d|
d}| | }d
d l
m}m} t! |j | r7| }t! || rt
|||d t' |d d rL|
dk rd
nd|
d}| | }| } t) j | | t d|
d| |`| } | t* } t) j | | t d| i } t, d| d} || d< t/ j | d< t2 j | d< t) j | d< t9 r!t( j | d< t? r"t( j | d< n!t( j! | d< tE rtG j | d< | | } t) j | | t d | | S ) aw
Saves the current states of the models, optimizers, scaler, and RNG generators to a given directory.
If `safe_serialization` is `True`, models will be saved with `safetensors` while the rest are saved using native
`pickle`.
Args:
output_dir (`str` or `os.PathLike`):
The name of the folder to save all relevant weights and states.
model_states (`List[torch.nn.Module]`):
A list of model states
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
dataloaders (`List[torch.utils.data.DataLoader]`):
A list of dataloader instances to save their sampler states
process_index (`int`):
The current process index in the Accelerator state
step (`int`):
The current step in the internal step tracker
scaler (`torch.amp.GradScaler`, *optional*):
An optional gradient scaler instance to save;
save_on_each_node (`bool`, *optional*):
Whether to save on every node, or only the main node.
safe_serialization (`bool`, *optional*, defaults to `True`):
Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
r ._)r r! zModel weights saved in .binFzOptimizer state saved in zScheduler state saved in r IterableDatasetShardSeedableRandomSampleruse_stateful_dataloaderdl_state_dict.bindl_state_dict_z Sampler state for dataloader z
saved in Nz Gradient scaler state saved in .pklr random_statenumpy_random_seedtorch_manual_seedtorch_xpu_manual_seedtorch_mlu_manual_seedtorch_cuda_manual_seedxm_seedzRandom states saved in )$r enumerater r replacejoinpathr loggerinfo
state_dictr r r
data_loaderr' r(
isinstancedataset get_samplergetattrtorchr r
randomgetstatenp get_state
get_rng_stater xpuget_rng_state_allr mlucudar xm)!r r r r r r r r r r! istate weights_nameoutput_model_fileoptoptimizer_nameoutput_optimizer_file schedulerscheduler_nameoutput_scheduler_file
dataloader sampler_nameoutput_sampler_filer' r( samplerdataloader_state_dict_name!output_dataloader_state_dict_filer9 output_scaler_filestates states_nameoutput_states_files! T/var/www/html/ai-engine/env/lib/python3.11/site-packages/accelerate/checkpointing.pysave_accelerator_stater_ 4 s X j!!J l++ C C5+=T||CT q55'//XXXX>>L & / / = = U%9J_qrrrr A.?AABBBBJ'' I I3 45FFN0000>@[@[TU@[@[@[ * 3 3N C C U)=Nchiiii G0EGGHHHH!*-- I I 9$$&&45FFN0000>@[@[TU@[@[@[ * 3 3N C C U)=Nchiiii G0EGGHHHH";// X X
:01Q,,,,,| ? ? r ,,..G'#899
rW 1EVkpqqqq : 8% @ @ F@AQ)<)"$)"5"5"7"7F "'"5"7"7F H*/)*E*E*G*G&' J*/)*E*E*G*G&''+0:+G+G+I+I'( / , . .y#,,[99 Jv)***
KK>*<>>??? c
t } |dvrt d |d}n|dk rt j }t | } t | D ]\ }
} |
dk rd|
nd} | t | d }
|
r t | |
fd
t | i| c| t | d }
t j
|
| } | j |fi | t d
t | D ]h\ }
}|
dk r
t" d n t" d|
d }| | }t j
|| }||
| it d t | D ]^\ }
}|
dk r
t$ d n t$ d|
d }| | }| t j
| _t d t | D ]\ }
}|
dk r
t& d n t& d|
d }| | }ddlm}m} t/ |j | rK| }t/ || r'| t j
| }t7 |dd r`|
dk rdnd|
d } | | } | r)t j
| }| | t d |[| t8 } | t j
| t d t j
| t: d| d } d| v r | d | d< t= j | d t@ j ! | d t j" | d tG r%t j$ % | d tM r&t j' % | d n%t j( % | d tS rtU j" | d t d! n*# tV $ r t d" Y nw xY w| S )#a^
Loads states of the models, optimizers, scaler, and RNG generators from a given directory.
Args:
input_dir (`str` or `os.PathLike`):
The name of the folder to load all relevant weights and states.
models (`List[torch.nn.Module]`):
A list of model instances
optimizers (`List[torch.optim.Optimizer]`):
A list of optimizer instances
schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`):
A list of learning rate schedulers
process_index (`int`):
The current process index in the Accelerator state
scaler (`torch.amp.GradScaler`, *optional*):
An optional *GradScaler* instance to load
map_location (`str`, *optional*):
What device to load the optimizer state onto. Should be one of either "cpu" or "on_device".
load_model_func_kwargs (`dict`, *optional*):
Additional arguments that can be passed to the model's `load_state_dict` method.
Returns:
`dict`: Contains the `Accelerator` attributes to override while loading the state.
)Ncpu on_devicezaUnsupported optimizer map location passed, please choose one of `None`, `'cpu'`, or `'on_device'`Nrb rc r r$ z .safetensorsdevicer% map_locationz%All model weights loaded successfullyz(All optimizer states loaded successfullyz(All scheduler states loaded successfullyr r& r) Fr* r+ z1All dataloader sampler states loaded successfullyz$GradScaler state loaded successfullyr, r r- r. r/ r0 r1 r2 r3 z%All random states loaded successfullyz Could not load random states),dict TypeErrorr re r r4 r6 r existsr strr r? loadload_state_dictr7 r8 r r r
r: r' r( r; r< r= set_samplerr> r r
r@ setstaterB set_state
set_rng_stater rE set_rng_state_allr rG rH r rI Exception) input_dirmodelsr r r r r rg load_model_func_kwargsoverride_attributesrJ modelendinginput_model_filer9 rN rO input_optimizer_fileoptimizer_staterQ rR input_scheduler_filerT rU input_sampler_filer' r( rW rX input_dataloader_state_dict_fileinput_scaler_filer[ s r^ load_accelerator_stater s H &&555 o
$ $#~~, YI f%% H H5 EEQr$-- .V&.V.V.VWW " " $ $ H u . c cs<7H7H cLb c c c c ) 1 1Z2M2M2M2M N N $4<PPPJ !E !* G G0F G G G G
KK7888 J'' 7 7345FFN0000>@[@[TU@[@[@[ ( 1 1. A A *%9 UUU1
%%o6666
KK:;;; "*-- D D 945FFN0000>@[@[TU@[@[@[ ( 1 1. A A !!%*-A"B"BCCCC
KK:;;;";// 7 7
:01Q,,,,,| ? ? Q ,,..G'#899
Q$00r s
( ( ( ( ( ( % % % % % % $ *)))))))))
H # #m mmt*m m m
m
m m
m m m m m mn w w w w tO O OD O O O OG G G G G G G Gr`