gq ldZddlZddlmZdeedefdZdeefdZd ded ed ed efd ZdS)aV A collection of utilities for comparing `examples/complete_*_example.py` scripts with the capabilities inside of each `examples/by_feature` example. `compare_against_test` is the main function that should be used when testing, while the others are used to either get the code that matters, or to preprocess them (such as stripping comments) N)Listlinesnamec|dkr|dkrtd|dgd}}|D]V}|sd||vrd}||#|r1|dkrd|vr|cS|dkrd |vr|cS||Wd S) a. Extracts a function from `lines` of segmented source code with the name `name`. Args: lines (`List[str]`): Source code of a script seperated by line. name (`str`): The name of the function to extract. Should be either `training_function` or `main` training_functionmainz Incorrect function name passed: z-, choose either 'main' or 'training_function'Fzdef Tzdef mainz if __name__N) ValueErrorappend)rr good_lines found_startlines Z/var/www/html/ai-engine/env/lib/python3.11/site-packages/accelerate/test_utils/examples.pyget_function_contents_by_namers """tv~~oDoooppp % J $ $ }d}}44K   d # # #   $***zT/A/A!!!!v~~-4"7"7!!!!   d # # # $ $cd|DS)z Filters `lines` and removes any entries that start with a comment ('#') or is just a newline (' ') Args: lines (`List[str]`): Source code of a script seperated by line. clg|]1}|d)|dk/|2S)# )lstrip startswith).0r s r zclean_lines..<s9 Y Y YT$++--*B*B3*G*G YDTXLLDLLLr)rs r clean_linesr4s Z YU Y Y YYr base_filenamefeature_filename parser_onlysecondary_filenamect|5}|}dddn #1swxYwYttjtjdd5}|}dddn #1swxYwYt|5}|}dddn #1swxYwY|;t|5}|}dddn #1swxYwY|rwt t|d} t t|d} t t|d} |t t|dnvt t|d} t t|d} t t|d} |t t|dd} g} g}t| }tt| dz D]o}||vrit|}|| vrV| | kr>d|vr+| || |`t|}pgg}t| D]S\}}||vrJ|| vrF| | kr.d|vr* || |Tfd | D}|fd |Dfd |D}|S) a Tests whether the additional code inside of `feature_filename` was implemented in `base_filename`. This should be used when testing to see if `complete_*_.py` examples have all of the implementations from each of the `examples/by_feature/*` scripts. It utilizes `nlp_example.py` to extract out all of the repeated training code, so that only the new additional code is examined and checked. If something *other* than `nlp_example.py` should be used, such as `cv_example.py` for the `complete_cv_example.py` script, it should be passed in for the `secondary_filename` parameter. Args: base_filename (`str` or `os.PathLike`): The filepath of a single "complete" example script to test, such as `examples/complete_cv_example.py` feature_filename (`str` or `os.PathLike`): The filepath of a single feature example script. The contents of this script are checked to see if they exist in `base_filename` parser_only (`bool`): Whether to compare only the `main()` sections in both files, or to compare the contents of `training_loop()` secondary_filename (`str`, *optional*): A potential secondary filepath that should be included in the check. This function extracts the base functionalities off of "examples/nlp_example.py", so if `base_filename` is a script other than `complete_nlp_example.py`, the template script should be included here. Such as `examples/cv_example.py` Nexamplesznlp_example.pyrrzMtrain_dataloader, eval_dataloader = get_dataloaders(accelerator, batch_size) TESTING_MOCKED_DATALOADERScg|]}|v| Srr)rr new_full_example_partss rrz(compare_against_test..s$aaa$dJ`>`>`>`>`>`rcg|]}|v| Srr)rr secondary_file_funcs rrz(compare_against_test..s$```$L_@_@_@_@_@_rcg|]}|v| Srr)rr diff_from_twos rrz(compare_against_test..s#]]]d4}C\C\TC\C\C\r)open readlinesospathabspathjoinrriterrangelennextrr enumerate)rrrrfbase_file_contentsfull_file_contentsfeature_file_contentssecondary_file_contentsbase_file_funcfull_file_funcfeature_file_func_dl_linenew_feature_code passed_idxsitir _diff_from_exampler(r$r&s @@@rcompare_against_testrC?sw0 m  +[[]]+++++++++++++++ bgoobgll:7GHHII J J+a[[]]+++++++++++++++   .1 ! ...............% $ % % 4&'kkmm # 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 $%BCUW]%^%^__$%BCUW]%^%^__'(EF[]c(d(dee  )"-.KLcek.l.l"m"m $%BCUWj%k%kll$%BCUWj%k%kll'(EF[]p(q(qrr  )"--.EGZ[[## `HK  B 3())A- . . ! ! K  88DN**(1J1J/t;;$++D111&&q))))RA K^,,**4 K  N**(1J1J/t;;*11$777&&q)))baaa*:aaa%````*<``` ]]]].?]]] sB488 B**B.1B.C%%C),C)D""D&)D&)N) __doc__r+typingrstrrrboolrCrrrrHs   $c$#$$$$4ZtCyZZZZSSSsSQUSknSSSSSSr