g^ddlZddlmZmZejdedGddZdS) N)AcceleratorState GradientStateignoreztorch.optim.lr_scheduler)categorymodulecHeZdZdZddedefdZdZdZd Zd Z d Z d Z d S)AcceleratedSchedulerau A wrapper around a learning rate scheduler that will only step when the optimizer(s) have a training step. Useful to avoid making a scheduler step too fast when gradients went overflow and there was no training step (in mixed precision training) When performing gradient accumulation scheduler lengths should not be changed accordingly, Accelerate will always step the scheduler to account for it. Args: scheduler (`torch.optim.lr_scheduler._LRScheduler`): The scheduler to wrap. optimizers (one or a list of `torch.optim.Optimizer`): The optimizers used. step_with_optimizer (`bool`, *optional*, defaults to `True`): Whether or not the scheduler should be stepped at each optimizer step. split_batches (`bool`, *optional*, defaults to `False`): Whether or not the dataloaders split one batch across the different processes (so batch size is the same regardless of the number of processes) or create batches on each process (so batch size is the original batch size multiplied by the number of processes). TFstep_with_optimizer split_batchesc||_t|ttfr|n|g|_||_||_t|_dSN) scheduler isinstancelisttuple optimizersr r rgradient_state)selfrrr r s P/var/www/html/ai-engine/env/lib/python3.11/site-packages/accelerate/scheduler.py__init__zAcceleratedScheduler.__init__/sM"(2:e}(M(M_**T^S_*#6 +ooc|js|jj|i|dS|jjs#|jjr|jxjdz c_dS|jD] }|jrdS |j r|jj|i|dStj }t|D]V}t|jdr-|jj|jjkr|jj|i|D|jj|i|WdS)Nr total_steps)r rsteprsync_gradientsadjust_scheduler _step_countrstep_was_skippedr r num_processesrangehasattrr)rargskwargsoptr _s rrzAcceleratedScheduler.step6sJ'  DN  0 0 0 0 F"1 "3 0**a/** F?  C#     9 DN  0 0 0 0 0 0-..=999~1T^5OOO++T,,,+++333'''88888rr )warningsstaterrfilterwarnings UserWarningr r7rrr<s"22222222;?YZZZZI8I8I8I8I8I8I8I8I8I8r