gx d Z ddlZddlZddlmZ ddlmZmZmZmZ ddl Z
ddl m Z ddl
mZmZ ddlmZ dd lmZ ej e Ze G d
d Z G d d
Ze G d de Z G d d Z G d de Z G d de Z G d de Z G d de Z G d dee Z dS ) zJ
Callbacks to use with the Trainer class and customize the training loop.
N) dataclass)DictListOptionalUnion)tqdm )IntervalStrategy
has_length)TrainingArguments)loggingc e Zd ZU dZdZee ed< dZe ed< dZ
e ed< dZ e ed< dZ e ed < dZ
e ed
< dZe ed < dZe ed < dZe ed
< dZeed< dZeeeef ed< dZee ed< dZee ed< dZeed< dZeed< dZeed< dZ eed< dZ eee eee ef f ed< dZ ed ed< d Z d efd Z!e"d efd Z#dS ) TrainerStatea
A class containing the [`Trainer`] inner state that will be saved along the model and optimizer when checkpointing
and passed to the [`TrainerCallback`].
In all this class, one step is to be understood as one update step. When using gradient accumulation, one update
step may require several forward and backward passes: if you use `gradient_accumulation_steps=n`, then one update
step requires going through *n* batches.
Args:
epoch (`float`, *optional*):
Only set during training, will represent the epoch the training is at (the decimal part being the
percentage of the current epoch completed).
global_step (`int`, *optional*, defaults to 0):
During training, represents the number of update steps completed.
max_steps (`int`, *optional*, defaults to 0):
The number of update steps to do during the current training.
logging_steps (`int`, *optional*, defaults to 500):
Log every X updates steps
eval_steps (`int`, *optional*):
Run an evaluation every X steps.
save_steps (`int`, *optional*, defaults to 500):
Save checkpoint every X updates steps.
train_batch_size (`int`, *optional*):
The batch size for the training dataloader. Only needed when
`auto_find_batch_size` has been used.
num_input_tokens_seen (`int`, *optional*, defaults to 0):
The number of tokens seen during training (number of input tokens, not the number of prediction tokens).
total_flos (`float`, *optional*, defaults to 0):
The total number of floating operations done by the model since the beginning of training (stored as floats
to avoid overflow).
log_history (`List[Dict[str, float]]`, *optional*):
The list of logs done since the beginning of training.
best_metric (`float`, *optional*):
When tracking the best model, the value of the best metric encountered so far.
best_model_checkpoint (`str`, *optional*):
When tracking the best model, the value of the name of the checkpoint for the best model encountered so
far.
is_local_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on
several machines) main process.
is_world_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the global main process (when training in a distributed fashion on several
machines, this is only going to be `True` for one process).
is_hyper_param_search (`bool`, *optional*, defaults to `False`):
Whether we are in the process of a hyper parameter search using Trainer.hyperparameter_search. This will
impact the way data will be logged in TensorBoard.
stateful_callbacks (`List[StatefulTrainerCallback]`, *optional*):
Callbacks attached to the `Trainer` that should have their states be saved or restored.
Relevent callbacks should implement a `state` and `from_state` function.
Nepochr global_step max_stepsi
logging_steps
eval_steps
save_stepstrain_batch_sizenum_train_epochsnum_input_tokens_seen
total_flos log_history best_metricbest_model_checkpointTis_local_process_zerois_world_process_zeroFis_hyper_param_search
trial_name trial_paramsTrainerCallbackstateful_callbacksc | j g | _ | j i | _ d S t | j t rd S i }| j D ]}t |t s t dt
| |j j }||v rUt || t s || g||< ||
| | ||< || _ d S )NzNAll callbacks passed to be saved must inherit `ExportableState`, but received ) r r#
isinstancedictExportableState TypeErrortype __class____name__listappendstate)selfr# callbacknames Y/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/trainer_callback.py
__post_init__zTrainerState.__post_init__p s, # !D " *&(D # # #
/
6
6 9 D "$ 3
@
@!(_>> #yimnviwiwyy )2--- &&8&>EE N4Ft4L3M*40&t,33HNN4D4DEEEE/7~~/?/?&t,,&8D # # # json_pathc t j t j | dd dz }t |dd 5 }| | ddd dS # 1 swxY w Y dS ) zDSave the content of this instance in JSON format inside `json_path`. T)indent sort_keys
wutf-8encodingN)jsondumps dataclassesasdictopenwrite)r/ r5 json_stringfs r2 save_to_jsonzTrainerState.save_to_json s j!3D!9!9!tTTTW[[
)S7
3
3
3 !q
GGK ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !s A""A&)A&c t |dd 5 }| }ddd n # 1 swxY w Y | di t j | S )z3Create an instance from the content of `json_path`.rr< r= N )rC readr? loads)clsr5 rF texts r2 load_from_jsonzTrainerState.load_from_json s )S7
3
3
3 q6688D s&&TZ%%&&&s 488)$r+
__module__ __qualname____doc__r r float__annotations__r intr r r r r r r r r r r strr r r boolr r r r! r r# r3 rG classmethodrO rJ r4 r2 r r # s 5 5n "E8E?!!!KIs M3 JJ c c !"3"""J*.Kd3:&'...#'K% '''+/8C=///"&4&&&"&4&&&"'4'''J<@L$sE#uc4"7 889@@@26 ./6669 9 96!c ! ! ! ! 's ' ' ' [' ' 'r4 r c 4 e Zd ZdZdefdZed ZdS )r' aj
A class for objects that include the ability to have its state
be saved during `Trainer._save_checkpoint` and loaded back in during
`Trainer._load_from_checkpoint`.
These must implement a `state` function that gets called during the respective
Trainer function call. It should only include parameters and attributes needed to
recreate the state at a particular time, to avoid utilizing pickle/maintain standard
file IO writing.
Example:
```python
class EarlyStoppingCallback(TrainerCallback, ExportableState):
def __init__(self, early_stopping_patience: int = 1, early_stopping_threshold: Optional[float] = 0.0):
self.early_stopping_patience = early_stopping_patience
self.early_stopping_threshold = early_stopping_threshold
# early_stopping_patience_counter denotes the number of times validation metrics failed to improve.
self.early_stopping_patience_counter = 0
def state(self) -> dict:
return {
"args": {
"early_stopping_patience": self.early_stopping_patience,
"early_stopping_threshold": self.early_stopping_threshold,
},
"attributes": {
"early_stopping_patience_counter": self.early_stopping_patience_counter,
}
}
```returnc t d )Nz e Zd ZdZdededefdZdededefdZdededefdZ dededefdZ
dededefd Z dededefd
Z dededefd Z
dededefd Zdededefd
ZdededefdZdededefdZdededefdZdededefdZdededefdZdededefdZdS )r" a
A class for objects that will inspect the state of the training loop at some events and take some decisions. At
each of those events the following arguments are available:
Args:
args ([`TrainingArguments`]):
The training arguments used to instantiate the [`Trainer`].
state ([`TrainerState`]):
The current state of the [`Trainer`].
control ([`TrainerControl`]):
The object that is returned to the [`Trainer`] and can be used to make some decisions.
model ([`PreTrainedModel`] or `torch.nn.Module`):
The model being trained.
tokenizer ([`PreTrainedTokenizer`]):
The tokenizer used for encoding the data. This is deprecated in favour of `processing_class`.
processing_class ([`PreTrainedTokenizer` or `BaseImageProcessor` or `ProcessorMixin` or `FeatureExtractionMixin`]):
The processing class used for encoding the data. Can be a tokenizer, a processor, an image processor or a feature extractor.
optimizer (`torch.optim.Optimizer`):
The optimizer used for the training steps.
lr_scheduler (`torch.optim.lr_scheduler.LambdaLR`):
The scheduler used for setting the learning rate.
train_dataloader (`torch.utils.data.DataLoader`, *optional*):
The current dataloader used for training.
eval_dataloader (`torch.utils.data.DataLoader`, *optional*):
The current dataloader used for evaluation.
metrics (`Dict[str, float]`):
The metrics computed by the last evaluation phase.
Those are only accessible in the event `on_evaluate`.
logs (`Dict[str, float]`):
The values to log.
Those are only accessible in the event `on_log`.
The `control` object is the only one that can be changed by the callback, in which case the event that changes it
should return the modified version.
The argument `args`, `state` and `control` are positionals for all events, all the others are grouped in `kwargs`.
You can unpack the ones you need in the signature of the event using them. As an example, see the code of the
simple [`~transformers.PrinterCallback`].
Example:
```python
class PrinterCallback(TrainerCallback):
def on_log(self, args, state, control, logs=None, **kwargs):
_ = logs.pop("total_flos", None)
if state.is_local_process_zero:
print(logs)
```r_ r. controlc dS )zS
Event called at the end of the initialization of the [`Trainer`].
NrJ r/ r_ r. rx kwargss r2 on_init_endzTrainerCallback.on_init_end8
r4 c dS )z<
Event called at the beginning of training.
NrJ rz s r2 on_train_beginz TrainerCallback.on_train_begin> r} r4 c dS )z6
Event called at the end of training.
NrJ rz s r2 on_train_endz TrainerCallback.on_train_endD r} r4 c dS )z<
Event called at the beginning of an epoch.
NrJ rz s r2 on_epoch_beginz TrainerCallback.on_epoch_beginJ r} r4 c dS )z6
Event called at the end of an epoch.
NrJ rz s r2 on_epoch_endz TrainerCallback.on_epoch_endP r} r4 c dS )z
Event called at the beginning of a training step. If using gradient accumulation, one training step might take
several inputs.
NrJ rz s r2
on_step_beginz TrainerCallback.on_step_beginV
r4 c dS )zv
Event called before the optimizer step but after gradient clipping. Useful for monitoring gradients.
NrJ rz s r2 on_pre_optimizer_stepz%TrainerCallback.on_pre_optimizer_step] r} r4 c dS )z}
Event called after the optimizer step but before gradients are zeroed out. Useful for monitoring gradients.
NrJ rz s r2 on_optimizer_stepz!TrainerCallback.on_optimizer_stepc r} r4 c dS )zU
Event called at the end of an substep during gradient accumulation.
NrJ rz s r2 on_substep_endz TrainerCallback.on_substep_endi r} r4 c dS )z
Event called at the end of a training step. If using gradient accumulation, one training step might take
several inputs.
NrJ rz s r2 on_step_endzTrainerCallback.on_step_endo r r4 c dS )z9
Event called after an evaluation phase.
NrJ rz s r2 on_evaluatezTrainerCallback.on_evaluatev r} r4 c dS )z=
Event called after a successful prediction.
NrJ )r/ r_ r. rx metricsr{ s r2
on_predictzTrainerCallback.on_predict| r} r4 c dS )z7
Event called after a checkpoint save.
NrJ rz s r2 on_savezTrainerCallback.on_save r} r4 c dS )z;
Event called after logging the last logs.
NrJ rz s r2 on_logzTrainerCallback.on_log r} r4 c dS )z7
Event called after a prediction step.
NrJ rz s r2 on_prediction_stepz"TrainerCallback.on_prediction_step r} r4 N)r+ rP rQ rR r r rh r| r r r r r r r r r r r r r r rJ r4 r2 r" r" s 1 1 f
1
,
Q_
#4
\
Tb
!2
<
R`
#4
\
Tb
!2
<
R`
"3
L
Sa
*;
L
[i
&7
We
#4
\
Tb
1
,
Q_
1
,
Q_
0
P^
-
l
^
,
\
N
'8
Xf
r4 r" c r e Zd ZdZd Zd Zd Zd Zed Z de
de d e fd
Z
de
de d e fd Zde
de d e fd Zde
de d e fd
Zde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZde
de d e fdZd Z dS )CallbackHandlerz>Internal class that just calls the list of callbacks in order.c g | _ |D ]}| | || _ || _ || _ || _ d | _ d | _ t d | j D s$t
d| j z d S d S )Nc 3 @ K | ]}t |t V d S N)r% DefaultFlowCallback.0cbs r2 z+CallbackHandler.__init__.. s- PP2:b"566PPPPPPr4 zThe Trainer will not work properly if you don't have a `DefaultFlowCallback` in its callbacks. You
should add one before training with `trainer.add_callback(DefaultFlowCallback). The current list ofcallbacks is
:) callbacks add_callbackmodelprocessing_class optimizer lr_schedulertrain_dataloadereval_dataloaderanyloggerwarning
callback_list)r/ r r r r r r s r2 __init__zCallbackHandler.__init__ s " "B b ! ! ! !
0 " ( $ # PP PPPPP NN$ $%
r4 c 0 t |t r
| n|}t |t r|n|j }|d | j D v r)t d| ddz | j z | j | d S )Nc g | ] }|j
S rJ )r* )r cs r2
z0CallbackHandler.add_callback.. s <<< <<>N))"--- III "
r4 c t |t r9| j D ]/}t || r | j | d S 0d S | j | d S r r r s r2 remove_callbackz CallbackHandler.remove_callback s h % % ,n
b(++ N))"---FF
N ! !( + + + + +r4 c J d d | j D S )Nr: c 3 . K | ]}|j j V d S r )r* r+ r s r2 r z0CallbackHandler.callback_list.. s' HH2 .HHHHHHr4 )joinr r] s r2 r z CallbackHandler.callback_list s% yyHH HHHHHHr4 r_ r. rx c 2 | d||| S )Nr|
call_eventr/ r_ r. rx s r2 r| zCallbackHandler.on_init_end }dE7CCCr4 c @ d|_ | d||| S )NFr )ri r r s r2 r z CallbackHandler.on_train_begin s# ',$ /ugFFFr4 c 2 | d||| S )Nr r r s r2 r z CallbackHandler.on_train_end ~tUGDDDr4 c @ d|_ | d||| S )NFr )rj r r s r2 r z CallbackHandler.on_epoch_begin s# $)! /ugFFFr4 c 2 | d||| S )Nr r r s r2 r z CallbackHandler.on_epoch_end r r4 c \ d|_ d|_ d|_ | d||| S )NFr )rm rl rk r r s r2 r z CallbackHandler.on_step_begin s2 ""' #eWEEEr4 c 2 | d||| S )Nr r r s r2 r z%CallbackHandler.on_pre_optimizer_step s 6eWMMMr4 c 2 | d||| S )Nr r r s r2 r z!CallbackHandler.on_optimizer_step s 2D%IIIr4 c 2 | d||| S )Nr r r s r2 r z CallbackHandler.on_substep_end s /ugFFFr4 c 2 | d||| S )Nr r r s r2 r zCallbackHandler.on_step_end r r4 c D d|_ | d|||| S )NFr r )rl r r/ r_ r. rx r s r2 r zCallbackHandler.on_evaluate s& "' }dE7GTTTr4 c 6 | d|||| S )Nr r r r s r2 r zCallbackHandler.on_predict s |T5'7SSSr4 c @ d|_ | d||| S )NFr )rk r r s r2 r zCallbackHandler.on_save s" #y$w???r4 c D d|_ | d|||| S )NFr )logs)rm r )r/ r_ r. rx r s r2 r zCallbackHandler.on_log s& "xugDIIIr4 c 2 | d||| S )Nr r r s r2 r z"CallbackHandler.on_prediction_step s 3T5'JJJr4 c | j D ]D} t || |||f| j | j | j | j | j | j d|}||}E|S )N)r r r r r r )r getattrr r r r r r )r/ eventr_ r. rx r{ r0 results r2 r zCallbackHandler.call_event s ! !H-WXu-- j!%!6 . ! .!%!6 $ 4 F ! r4 N) r+ rP rQ rR r r r r propertyr r r rh r| r r r r r r r r r r r r r r r rJ r4 r2 r r so HH & " " "
, , , I I XID 1 D, DQ_ D D D DG#4 G\ GTb G G G GE!2 E< ER` E E E EG#4 G\ GTb G G G GE!2 E< ER` E E E EF"3 FL FSa F F F F N*; NL N[i N N N NJ&7 J JWe J J J JG#4 G\ GTb G G G GD 1 D, DQ_ D D D DU 1 U, UQ_ U U U UT 0 T TP^ T T T T@ - @l @^ @ @ @ @J, J\ JN J J J JK'8 K KXf K K K K r4 r c : e Zd ZdZdededefdZdededefdZdS )r zx
A [`TrainerCallback`] that handles the default flow of the training loop for logs, evaluation and checkpoints.
r_ r. rx c |j dk r|j rd|_ |j t j k r|j |j z dk rd|_ |j t j k r*|j |j z dk r|j |j k rd|_
|j t j k r%|j dk r|j |j z dk rd|_
|j |j k r#d|_ |j t j k rd|_
|S )Nr Tr )r logging_first_steprm logging_strategyr
STEPSr
eval_strategyr
eval_delayrl
save_strategyr rk r ri NOrz s r2 r z DefaultFlowCallback.on_step_end s ! !d&= !!%G $4$: : :u?PSXSf?fjk?k?k!%G
"2"8 8 8!E$44995#444&*G #
"2"8 8 8 1$$!E$4499"&G / /+/G (!%5%888&*#r4 c |j t j k rd|_ |j t j k r|j |j k rd|_ |j t j k rd|_ |S )NT)
r r
EPOCHrm r r r rl r rk rz s r2 r z DefaultFlowCallback.on_epoch_end= sg $4$: : :!%G !1!7 7 7DOu{$(@%/0,,,r4 c |j r t j n t j }|j 1 |||j r)t ||j z
| j k r d| _ d S | xj dz
c_ d S )Nr r )greater_is_betternpgreaterlessr absr
r
)r/ r_ r. rx metric_valueoperators r2 check_metric_valuez(EarlyStoppingCallback.check_metric_value s} !%!7D2::RW $ H\5#4 5 5 %L5#44558UUU34D 0 0 0 0 0A 5 0 0 0 0r4 c |j s
J d |j
J d |j t j k s
J d d S )Nzr' s ! ! ! ! ! ! . . . . . . . . . . . . 7 7 7 7 7 7 7 7 , , , , , ,
H % % r' r' r' r' r' r' r' r'j) ) ) ) ) ) ) )X :
:
:
:
:
_ :
:
:
zO
O
O
O
O
O
O
O
d@ @ @ @ @o @ @ @F2 2 2 2 2/ 2 2 2j7% 7% 7% 7% 7% 7% 7% 7%t o E
E
E
E
E
O_ E
E
E
E
E
r4