gݘdZddlZddlZddlmZddlmZmZmZm Z m Z ddl Z ddl m Z ddl mZddlmZmZdd lmZmZdd lmZdd lmZdd lmZejeZd@d ZdAdedefdZ defdZ!dedefdZ"dAdededefdZ#dededefdZ$dAdZ%dededede&fdZ' dBdededede&def dZ(dedededefdZ) dCdededededef dZ*dededed e&d!e&d"ef d#Z+ dDd&Z,dd'deded(efd)Z- dEdeded(edefd*Z.d+d,dededede&d-e&f d.Z' dFdededede&ded/e&d-e&fd0Z/deded1ed2ede&d3e&f d4Z0 dGdeded1ed2ed3e&de&defd5Z1ej2e%ej3e(ej4e*ej5e,ej6e ej7e#ej8e.ej9e!ej:e/ej;e1i Z< dHd6e e=efdedeedeed7ee>f d8Z?Gd9d:eZ@Gd;deZBdId?ZCdS)Jz$PyTorch optimization for BERT model.N)partial)CallableIterableOptionalTupleUnion)nn) Optimizer)LambdaLRReduceLROnPlateau)LayerWiseDummyOptimizerLayerWiseDummyScheduler) SchedulerType)logging)require_versioncdSNr )_s U/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/optimization.py_get_constant_lambdar$s 1 optimizer last_epochc0t|t|S)a Create a schedule with a constant learning rate, using the learning rate set in optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. r)r r)rrs rget_constant_scheduler(s I3 K K KKrc t|fi|S)a Create a schedule with a constant learning rate that decreases when a metric has stopped improving. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. kwargs (`dict`, *optional*): Extra parameters to be passed to the scheduler. See `torch.optim.lr_scheduler.ReduceLROnPlateau` for possible parameters. Return: `torch.optim.lr_scheduler.ReduceLROnPlateau` with the appropriate schedule. )r )rkwargss rget_reduce_on_plateau_scheduler"9s Y 1 1& 1 11r current_stepnum_warmup_stepscl||kr-t|ttd|z SdS)N?floatmax)r#r$s r,_get_constant_schedule_with_warmup_lr_lambdar*Ks9&&&\""U3s4D+E+E%F%FFF 3rcRtt|}t|||S)ad Create a schedule with a constant learning rate preceded by a warmup period during which the learning rate increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. r$r)rr*r )rr$r lr_lambdas r!get_constant_schedule_with_warmupr.Qs-"DWghhhI IyZ @ @ @@rnum_training_stepsc ||kr-t|ttd|z Stdt||z ttd||z z S)Nr r')r#r$r/s r*_get_linear_schedule_with_warmup_lr_lambdar2fsp&&&\""U3q2B+C+C%D%DDD sE,|;<k>k8l8llH sC3$'E*4E4E*E*Kh*V!W!WWX Y YYrr8cTtt|||}t|||S)a Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to 0, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. num_cycles (`float`, *optional*, defaults to 0.5): The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a half-cosine). last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. r$r/r6)rr@r rr$r/r6rr-s rget_cosine_schedule_with_warmuprDs822)- I Iy* 5 55rc n||kr-t|ttd|z St||z ttd||z z }|dkrdStdddtjtjt||zdzzzzS)Nr r&r1r8r:r>s r=_get_cosine_with_hard_restarts_schedule_with_warmup_lr_lambdarFs&&&\""U3q2B+C+C%D%DDD\$4455c!EWZjEj>k>k8l8llH3s sC3$'eJ6G6G(6RVY5Y*Z![![[\ ] ]]rcTtt|||}t|||S)a Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to 0, with several hard restarts, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. num_cycles (`int`, *optional*, defaults to 1): The number of hard restarts to use. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. rB)rrFr rCs r2get_cosine_with_hard_restarts_schedule_with_warmuprHs80E)- I Iy* 5 55rlr_endpowerlr_initc||kr-t|ttd|z S||kr||z S||z }||z }d||z |z z }|||zz|z} | |z Srr') r#r$r/rIrJrKlr_range decay_steps pct_remainingdecays r4_get_polynomial_decay_schedule_with_warmup_lr_lambdarQs&&&\""U3q2B+C+C%D%DDD * * *V#(+;; \,<< KK =%//&8wrHz>r&c|jd}||kstd|d|dtt|||||}t |||S)a Create a schedule with a learning rate that decreases as a polynomial decay from the initial lr set in the optimizer to end lr defined by *lr_end*, after a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. lr_end (`float`, *optional*, defaults to 1e-7): The end LR. power (`float`, *optional*, defaults to 1.0): Power factor. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Note: *power* defaults to 1.0 as in the fairseq implementation, which in turn is based on the original BERT implementation at https://github.com/google-research/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/optimization.py#L37 Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. lrzlr_end (z#) must be smaller than initial lr ())r$r/rIrJrK)defaults ValueErrorrrQr )rr$r/rIrJrrKr-s r)get_polynomial_decay_schedule_with_warmuprXsx> &G f  YFYYwYYYZZZ<)- I Iy* 5 55r) timescalerYc||kr-t|ttd|z S||z }dtj||z|z z }|S)Nr r&)r(r)r;sqrt)r#r$rYshiftrPs r$_get_inverse_sqrt_schedule_lr_lambdar]!sb&&&\""U3q2B+C+C%D%DDD ( (E $)\E1Y>?? ?E Lrc`||pd}tt||}t|||S)a Create a schedule with an inverse square-root learning rate, from the initial lr set in the optimizer, after a warmup period which increases lr linearly from 0 to the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. timescale (`int`, *optional*, defaults to `num_warmup_steps`): Time scale. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. Ni')r$rYr)rr]r )rr$rYrr-s rget_inverse_sqrt_scheduler_)s@.$. k>k8l8llH C$(47U:->->#>#Dx#OPPP QF q; '+ 5F q&>>rmin_lrc||td|||jdz }n|tdtt||||}t |||S)a Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to min_lr, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. num_cycles (`float`, *optional*, defaults to 0.5): The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a half-cosine). last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. min_lr (`float`, *optional*): The minimum learning rate to reach after the cosine schedule. min_lr_rate (`float`, *optional*): The minimum learning rate as a ratio of the initial learning rate. If set, `min_lr` should not be set. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. Nz/Only one of min_lr or min_lr_rate should be setrTzLOne of min_lr or min_lr_rate should be set through the `lr_scheduler_kwargs`)r$r/r6r`)rWrVrr@r )rr$r/r6rrcr`r-s r+get_cosine_with_min_lr_schedule_with_warmupreRsFk5JKKK  y1$77  ghhh2)- I Iy* 5 55rnum_stable_stepsnum_decay_steps min_lr_ratioc ||kr-t|ttd|z S|||zkrdS|||z|zkrt||z |z ttd|z }tdddtjtjt|zdz|zzz}d|z |z|zS|S)Nr r&r1r8r9r:)r#r$rfrgr6rhr?values r_get_wsd_scheduler_lambdarks&&&\""U3q2B+C+C%D%DDD&)9999s&)99OKKK(88;KKLLuUXYZ\kUlUlOmOmmCdhtwz9J9J/JS/PS[/[&\&\ \]^^l"e+l:: rcXtt|||||}t|||S)a Create a schedule with a learning rate that has three stages: 1. linear increase from 0 to initial lr. 2. constant lr (equal to initial lr). 3. decrease following the values of the cosine function between the initial lr set in the optimizer to a fraction of initial lr. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_stable_steps (`int`): The number of steps for the stable phase. num_decay_steps (`int`): The number of steps for the cosine annealing phase. min_lr_ratio (`float`, *optional*, defaults to 0): The minimum learning rate as a ratio of the initial learning rate. num_cycles (`float`, *optional*, defaults to 0.5): The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a half-cosine). last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. )r$rfrgrhr6)rrkr )rr$rfrgrhr6rr-s rget_wsd_schedulerms?H!))'! I Iy* 5 55rnamescheduler_specific_kwargsc  t|}t|}|t|tr|j}i |D]}t ||||| |< fd}|D]}|jr||t||j dS|tj kr ||S|i}|tj kr ||fi|S|t|d|tjkr |||S|tjkr |||S|tjkr ||fd|i|S|t|d ||f||d |S) a Unified API to get any scheduler from its name. Args: name (`str` or `SchedulerType`): The name of the scheduler to use. optimizer (`torch.optim.Optimizer`): The optimizer that will be used during training. num_warmup_steps (`int`, *optional*): The number of warmup steps to do. This is not required by all schedulers (hence the argument being optional), the function will raise an error if it's unset and the scheduler type requires it. num_training_steps (`int``, *optional*): The number of training steps to do. This is not required by all schedulers (hence the argument being optional), the function will raise an error if it's unset and the scheduler type requires it. scheduler_specific_kwargs (`dict`, *optional*): Extra parameters for schedulers such as cosine with restarts. Mismatched scheduler types and scheduler parameters will cause the scheduler function to raise a TypeError. N)rr$r/c<|dSN)step)paramscheduler_dicts rscheduler_hookz%get_scheduler..scheduler_hooks" 5 ! & & ( ( ( ( (rrT)optimizer_dictrTz; requires `num_warmup_steps`, please provide that argument.r,r$z= requires `num_training_steps`, please provide that argument.r4)rTYPE_TO_SCHEDULER_FUNCTION isinstancerrwkeys get_scheduler requires_grad"register_post_accumulate_grad_hookrrVCONSTANTREDUCE_ON_PLATEAUrWCONSTANT_WITH_WARMUP INVERSE_SQRTWARMUP_STABLE_DECAY) rnrr$r/ro schedule_funcrwrtrvrus @rr{r{s32   D.t4MI7N!O!O"1#((**  E$1(/!1#5 %%%N5 ! ! ) ) ) ) ) $((** I IE" I88HHH&nI[\`Iabbbb }%%%}Y''' ($&! }...}YDD*CDDDD]]]^^^ }111}Y9IJJJJ })))}Y9IJJJJ }000}Ygg9IgMfggg!D___``` = )-   $   rceZdZdZ ddeejjd ed e eefd ed ed e de ffd Z e j ddefdZxZS)AdamWa; Implements Adam algorithm with weight decay fix as introduced in [Decoupled Weight Decay Regularization](https://arxiv.org/abs/1711.05101). Parameters: params (`Iterable[nn.parameter.Parameter]`): Iterable of parameters to optimize or dictionaries defining parameter groups. lr (`float`, *optional*, defaults to 0.001): The learning rate to use. betas (`Tuple[float,float]`, *optional*, defaults to `(0.9, 0.999)`): Adam's betas parameters (b1, b2). eps (`float`, *optional*, defaults to 1e-06): Adam's epsilon for numerical stability. weight_decay (`float`, *optional*, defaults to 0.0): Decoupled weight decay to apply. correct_bias (`bool`, *optional*, defaults to `True`): Whether or not to correct bias in Adam (for instance, in Bert TF repository they use `False`). no_deprecation_warning (`bool`, *optional*, defaults to `False`): A flag used to disable the deprecation warning (set to `True` to disable the warning). MbP?g?g+?ư>r1TFparamsrTbetaseps weight_decay correct_biasno_deprecation_warningc|stjdttd|dkrt d|dd|dcxkrdksnt d|dd d|d cxkrdksnt d|d d d|kst d |d|||||d }t ||dS) NzThis implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning torch>=1.5.0r1zInvalid learning rate: z - should be >= 0.0rr&zInvalid beta parameter: z - should be in [0.0, 1.0)r zInvalid epsilon value: )rTrrrr)warningswarn FutureWarningrrWsuper__init__) selfrrTrrrrrrV __class__s rrzAdamW.__init__Ds7&  M     ''' 88NrNNNOO OeAh$$$$$$$$\a\\\]] ]eAh$$$$$$$$\a\\\]] ]czzOsOOOPP PuS,htuu *****rNclosurecd}| |}|jD]}|dD]}|j |j}|jrtd|j|}t |dkr3d|d<t j||d<t j||d<|d|d}}|d\} } |dxxd z cc<||  |d | z ||  ||d | z | |d } |d} |dr6d | |dzz } d | |dzz }| tj |z| z } | || | |ddkr'| ||d |dz |S)z Performs a single optimization step. Arguments: closure (`Callable`, *optional*): A closure that reevaluates the model and returns the loss. NrzJAdam does not support sparse gradients, please consider SparseAdam insteadrrsexp_avg exp_avg_sqrr r&alpha)rjrrTrrr1) param_groupsgrad is_sparse RuntimeErrorstatelentorch zeros_likemul_add_addcmul_r[r;addcdiv_)rrlossgroupprrrrbeta1beta2denom step_sizebias_correction1bias_correction2s rrsz AdamW.stepas&  799D&. L. LE8_- L- L6>v>u&'sttt 1 u::??$%E&M','7':':E)$*/*:1*=*=E,'&+I&6l8K$W~ uf "  U##((cEk(CCC&&//d#+/NNN"))..uU|<<!$K (['*UeFm-C'C$'*UeFm-C'C$ )DI6F,G,G GJZ ZI 7E) <<<(3..FF1eDk\E.4I%IFKKK[- L^ r)rrrr1TFrr)__name__ __module__ __qualname____doc__rr parameter Parameterr(rboolrrno_gradrrs __classcell__rs@rrr.s0%1!!',++/0+ +UE\" +  +  ++!%++++++:U]__;;H;;;_;;;;;rrceZdZdZ dfd Zed Zed Zed Zed Z e j ddZ xZ S) Adafactora) AdaFactor pytorch implementation can be used as a drop in replacement for Adam original fairseq code: https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py Paper: *Adafactor: Adaptive Learning Rates with Sublinear Memory Cost* https://arxiv.org/abs/1804.04235 Note that this optimizer internally adjusts the learning rate depending on the `scale_parameter`, `relative_step` and `warmup_init` options. To use a manual (external) learning rate schedule you should set `scale_parameter=False` and `relative_step=False`. Arguments: params (`Iterable[nn.parameter.Parameter]`): Iterable of parameters to optimize or dictionaries defining parameter groups. lr (`float`, *optional*): The external learning rate. eps (`Tuple[float, float]`, *optional*, defaults to `(1e-30, 0.001)`): Regularization constants for square gradient and parameter scale respectively clip_threshold (`float`, *optional*, defaults to 1.0): Threshold of root mean square of final gradient update decay_rate (`float`, *optional*, defaults to -0.8): Coefficient used to compute running averages of square beta1 (`float`, *optional*): Coefficient used for computing running averages of gradient weight_decay (`float`, *optional*, defaults to 0.0): Weight decay (L2 penalty) scale_parameter (`bool`, *optional*, defaults to `True`): If True, learning rate is scaled by root mean square relative_step (`bool`, *optional*, defaults to `True`): If True, time-dependent learning rate is computed instead of external learning rate warmup_init (`bool`, *optional*, defaults to `False`): Time-dependent learning rate computation depends on whether warm-up initialization is being used This implementation handles low-precision (FP16, bfloat) values, but we have not thoroughly tested. Recommended T5 finetuning settings (https://discuss.huggingface.co/t/t5-finetuning-tips/684/3): - Training without LR warmup or clip_threshold is not recommended. - use scheduled LR warm-up to fixed LR - use clip_threshold=1.0 (https://arxiv.org/abs/1804.04235) - Disable relative updates - Use scale_parameter=False - Additional optimizer operations like gradient clipping should not be used alongside Adafactor Example: ```python Adafactor(model.parameters(), scale_parameter=False, relative_step=False, warmup_init=False, lr=1e-3) ``` Others reported the following combination to work well: ```python Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None) ``` When using `lr=None` with [`Trainer`] you will most likely need to use [`~optimization.AdafactorSchedule`] scheduler as following: ```python from transformers.optimization import Adafactor, AdafactorSchedule optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None) lr_scheduler = AdafactorSchedule(optimizer) trainer = Trainer(..., optimizers=(optimizer, lr_scheduler)) ``` Usage: ```python # replace AdamW with Adafactor optimizer = Adafactor( model.parameters(), lr=1e-3, eps=(1e-30, 1e-3), clip_threshold=1.0, decay_rate=-0.8, beta1=None, weight_decay=0.0, relative_step=False, scale_parameter=False, warmup_init=False, ) ```NgKH9rr&皙r1TFc td|| rtd| r| std|||||||| | d } t|| dS)Nrz;Cannot combine manual `lr` and `relative_step=True` optionsz0`warmup_init=True` requires `relative_step=True`) rTrclip_threshold decay_raterrscale_parameter relative_step warmup_init)rrWrr) rrrTrrrrrrrrrVrs rrzAdafactor.__init__s ''' >m>Z[[ [  Q} QOPP P,$(.*&    *****rc|d}|dr@|dr d|dznd}t|dtj|dz }d}|dr"t|d d |d }||zS) NrTrrrrsg{Gz?r&rrr RMS)minr;r[r)) param_group param_state rel_step_szmin_step param_scales r_get_lrzAdafactor._get_lrs!$'  ' N5@5OYtk&111UYHhdi F8K.L.L(LMMK ( ) Ik%03[5GHHK[((rcDt|dk}|ddu}||fS)Nr)r)r param_shapefactoreduse_first_moments r _get_optionszAdafactor._get_options s0{##q(&w/t;)))rc\|d|dzz S)Nrr8)normnumel)tensors r_rmszAdafactor._rms&s${{1~~3!677rc||ddz d}|d}t j||S)NrT)dimkeepdim)meanrsqrt_ unsqueezersqrtrmul)exp_avg_sq_rowexp_avg_sq_colr_factorc_factors r_approx_sq_gradzAdafactor._approx_sq_grad*sm#^%8%8R%8%N%NNVVXXbbceff!++B//5577y8,,,rc d}| |}|jD]}|dD]}|j |j}|jtjtjhvr|}|jrtd|j |}|j }| ||\}} t|dkrd|d<| rtj ||d<|rptj|dd||d<tj|dd |ddz||d <ntj ||d <d|d <n}| r|d||d<|r=|d||d<|d ||d <n|d ||d <|} |jtjtjhvr| } |dxxd z cc<|| |d <|||} dt%j|d|dz } |dz|ddz} |r|d}|d }|| | dd| z || | d d| z |||} | |n\|d }|| | d| z ||} | || |dz d| | | rC|d}||d| d |dz |} |ddkr!| | |d | z| | |jtjtjhvr|| |S)z Performs a single optimization step Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nrz,Adafactor does not support sparse gradients.rrsrrrrrrrr r&rrr)rrr)rrr)rrdtyperfloat16bfloat16r(rrrshaperrrzerostorrr;powrrrrrdiv_clamp_copy_)rrrrrrr grad_shaperr p_data_fp32rTbeta2tupdaterrrrs rrszAdafactor.step2s  799D&M )M )E8_L )L )6>v:%-!@@@::<W&'UVVV 1 !Z -1->->uj-Q-Q**u::??$%E&M'B+0+;D+A+Ai(E27+j"o2N2N2Q2QRV2W2W./27+j"oPZ[][^[^P_>_2`2`2c2cdh2i2i.//.3.>t.D.Dl+#$E%LL'E+0+;+>+>t+D+Di(K278H2I2L2LT2R2R./278H2I2L2LT2R2R.//.3L.A.D.DT.J.Jl+ 7u}en==="-"3"3"5"5Kf " #yy55e \\%//txf u\7JKKK'U5\!_4;%*+;%.lr_lambdas  rr)rrr)rrrr-rrs ` rrzAdafactorSchedule.__init__s     + - -E",E,   I...+ $ $El## $ $rcp|jfdjD}t|dkr|j}|S)Ncg|]B}|ddj|j|ddCS)rr)rrr).0ropts r z,AdafactorSchedule.get_lr..sS   Xq!&2 KKsyx);< = =222rr)rrrbase_lrs)rlrsrs @rget_lrzAdafactorSchedule.get_lrsRn    )    s88q==-C rr1)rrrrrrrrs@rrrsV$$$$$$       rrc"t||S)aX Get a proxy schedule for [`~optimization.Adafactor`] Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. initial_lr (`float`, *optional*, defaults to 0.0): Initial lr Return: [`~optimization.Adafactor`] proxy schedule object. )r)rrs rget_adafactor_schedulers Y 3 33rrr)r)r8r)r r)rRr&r)Nr)r8rNN)rr8r)NNNr)Drr;r functoolsrtypingrrrrrrr torch.optimr torch.optim.lr_schedulerr r trainer_pt_utilsrr trainer_utilsrutilsrutils.versionsr get_loggerrloggerrintrr"r*r.r2r5r(r@rDrFrHrQrXr]r_rerkrmLINEARCOSINECOSINE_WITH_RESTARTS POLYNOMIALr~rrrCOSINE_WITH_MIN_LRrrxstrdictr{rrrrrrrrs+* ============== !!!!!!@@@@@@@@NNNNNNNN((((((++++++  H % %    LLYLCLLLL"2i2222$sY\ AAAcA_bAAAA*uSuWZupsuuuu 66666ZZ,/ZEHZV[ZZZZvx666,/6EH6V[6or6666D^^,/^EH^VY^^^^rt666,/6EH6VY6kn6666B     ,Y[+6+6+6+6\hlsQTad[]AAA,/Asv,/EHV[jo1616161616 16  16  1616161616h     2,6,6,6,6,6 ,6  ,6  ,6,6,6,6,6`99&(ZG1&(I 9#%C$&Q%'7 "'+(,04 TT ]" #TTsmT! T (~ TTTTnoooooIooodnnnnn nnnb<444444r