NgTYXddlZddlZddlZddlmZddlmZmZmZddl Z ddl m Z m Z mZddlmZde deded ed e jf d Zd e deded e jfd ZdedefdZ d.de dededededed e jfdZdedeee jffdZdededed ededede d e de dededefdZdededededede d e de dededefdZdZ d/d ed!eed"eed#eed eee jee jee jff d$Z d/d%ed!eed"eed#eed eee jee jee jff d&Zd'Zd(ed)edededededed!eed"eed#eed*edededefd+Z d,Z!e"d-kr e!dSdS)0N)Path)DictOptionalTuple) ModelProto TensorProto numpy_helper) OnnxModel input_ids batch_sizesequence_lengthdictionary_sizereturnc|jjjtjtjtjfvsJtj |||ftj }|jjjtjkrtj |}n3|jjjtjkrtj |}|S)a`Create input tensor based on the graph input of input_ids Args: input_ids (TensorProto): graph input of the input_ids input tensor batch_size (int): batch size sequence_length (int): sequence length dictionary_size (int): vocabulary size of dictionary Returns: np.ndarray: the input tensor created )sizedtype) type tensor_type elem_typerFLOATINT32INT64nprandomrandintint32float32int64)r r r rdatas c/var/www/html/ai-engine/env/lib/python3.11/site-packages/onnxruntime/transformers/bert_test_data.pyfake_input_ids_datar!s > % /4     9  _J3PXZX`  a aD~!+{/@@@z$  # -1B B Bx~~ K segment_idsc|jjjtjtjtjfvsJtj||ftj }|jjjtjkrtj |}n3|jjjtjkrtj |}|S)a,Create input tensor based on the graph input of segment_ids Args: segment_ids (TensorProto): graph input of the token_type_ids input tensor batch_size (int): batch size sequence_length (int): sequence length Returns: np.ndarray: the input tensor created r) rrrrrrrrzerosrrr)r#r r rs r fake_segment_ids_datar'2s   ' 16     8Z1 B B BD#-1BBBz$   % /;3D D Dx~~ Kr"max_sequence_lengthaverage_sequence_lengthc|dkr||ksJd|z|krtjd|z|z |Stjdd|zdz S)N)rr)r(r)s r get_random_lengthr-Mso "a ' ',CGZ,Z,Z,Z,Z ""%888~a"99** f fQRUVQVQVtJNQ,>'?$q1u+'M'M\]Z!^$YZ]^Y^Y^tJNQ4F/G$qSTu+/U/UdeQ^a'!+,, f:&& 2 21Q:>** K K'(+B'BZ!^$/03J/JQ^a'!+,,", 0AAAz$  $ .+2C C Cx~~ Kr" directoryinputsc btj|sL tj|t d|dn6#t $rt d|dYnwxYwt d|dt |D]\}\}}tj ||}ttj |d|dd 5}| | d d d n #1swxYwYd S) zOutput input tensors of test data to a directory Args: directory (str): path of a directory inputs (Dict[str, np.ndarray]): map from input name to value z#Successfully created the directory  zCreation of the directory z failedzWarning: directory z$ existed. Files will be overwritten.input_.pbwbN)ospathexistsmkdirprintOSError enumerateitemsr from_arrayopenjoinwriteSerializeToString)r;r<indexnamertensorfiles r output_test_datarSs 7>>) $ $U F HY    D DDD E E E E C C C AyAAA B B B B B C SISSSTTT(8833|d(t44 "',,y*=5*=*=*=>> E E 3 JJv//11 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 333s#A A)(A)/(D##D' *D' test_casesverbose random_seedc |Jtj|tj|g} t|D]} t ||||}|j|i}|rt |||||j<|rt|||| | | ||j<|r#t| dkrtd|| || S)aCreate given number of input data for testing Args: batch_size (int): batch size sequence_length (int): sequence length test_cases (int): number of test cases dictionary_size (int): vocabulary size of dictionary for input_ids verbose (bool): print more information or not random_seed (int): random seed input_ids (TensorProto): graph input of input IDs segment_ids (TensorProto): graph input of token type IDs input_mask (TensorProto): graph input of attention mask average_sequence_length (int): average sequence length excluding paddings random_sequence_length (bool): whether use uniform random number for sequence length mask_type (int): mask type 1 is mask index; 2 is 2D mask; 3 is key len, cumulated lengths of query and key Returns: List[Dict[str,numpy.ndarray]]: list of test cases, where each test case is a dictionary with input name as key and a tensor as value NrzExample inputs) rrseedr4r!rPr'r:lenrFappend)r r rTrrUrVr r#r.r)r/r0 all_inputs _test_caseinput_1r<s r fake_test_datar^s D  INN; K JJ''"" %i_o^^.'*  g'<[*Ve'f'fF;# $  &:J9PRhjs''F:? #  ,s:!++ "F + + +&!!!! r"rXc d} t|||| ||||||| | } t| |krtd| S)aCreate given number of input data for testing Args: batch_size (int): batch size sequence_length (int): sequence length test_cases (int): number of test cases seed (int): random seed verbose (bool): print more information or not input_ids (TensorProto): graph input of input IDs segment_ids (TensorProto): graph input of token type IDs input_mask (TensorProto): graph input of attention mask average_sequence_length (int): average sequence length excluding paddings random_sequence_length (bool): whether use uniform random number for sequence length mask_type (int): mask type 1 is mask index; 2 is 2D mask; 3 is key len, cumulated lengths of query and key Returns: List[Dict[str,numpy.ndarray]]: list of test cases, where each test case is a dictionary with input name as key and a tensor as value i'z$Failed to create test data for test.)r^rYrF) r r rTrXrUr r#r.r)r/r0rr[s r generate_test_datar`se@O   J :*$$ 4555 r"c|t|jkrdS|j|}||}|C|||}|+|jdkr ||jd}|S)NCastr)rYinputfind_graph_input get_parentop_type) onnx_model embed_node input_indexrc graph_input parent_nodes r get_graph_input_from_embed_noderl%sc**++++t  [ )E--e44K ++J DD  "{':f'D'D$55k6G6JKKK r"rginput_ids_namesegment_ids_nameinput_mask_namec|}|||}|td|d}|r)||}|td|d}|r)||}|td|d|rdndz|rdndz}t||kr"td|dt||||fSt|dkrtdt||d } t| dkrw| d} t || d}t || d}t || d }|$|D]!} | j} d | vr| }"|td |||fSd}d}d}|D]/} | j} d | vr| }"d | vsd| vr| }-| }0|r |r|r|||fStd)a Find graph inputs for BERT model. First, we will deduce inputs from EmbedLayerNormalization node. If not found, we will guess the meaning of graph inputs based on naming. Args: onnx_model (OnnxModel): onnx model object input_ids_name (str, optional): Name of graph input for input IDs. Defaults to None. segment_ids_name (str, optional): Name of graph input for segment IDs. Defaults to None. input_mask_name (str, optional): Name of graph input for attention mask. Defaults to None. Raises: ValueError: Graph does not have input named of input_ids_name or segment_ids_name or input_mask_name ValueError: Expected graph input number does not match with specified input_ids_name, segment_ids_name and input_mask_name Returns: Tuple[Optional[np.ndarray], Optional[np.ndarray], Optional[np.ndarray]]: input tensors of input_ids, segment_ids and input_mask Nz Graph does not have input named r+rzExpect the graph to have z inputs. Got r2z'Expect the graph to have 3 inputs. Got EmbedLayerNormalizationmaskz#Failed to find attention mask inputtokensegmentz?Fail to assign 3 inputs. You might try rename the graph inputs.)'get_graph_inputs_excluding_initializersrd ValueErrorrYget_nodes_by_op_typerlrPlower) rgrmrnro graph_inputsr r#r.expected_inputs embed_nodesrhrcinput_name_lowers r find_bert_inputsr~2s4EEGGL!//??  PPPQQ Q  X$556FGGK" !VDT!V!VWWW  W#44_EEJ! !UO!U!UVVVK6qqQ7 ;Q11PQR |   / /jjjWZ[gWhWhjjkk k+z11 <AV3|CTCTVVWWW112KLLK ;1 ^ 3J ANN 5j*aPP 4ZQOO  % ' '#(:#3#3#5#5 ---!&J  BCC C+z11IKJ   :++-- % % %JJ ' ' '98H+H+HKKII2[2Z2+z11 V W WWr" onnx_filect}t|d5}||dddn #1swxYwYt |}t ||||S)aFind graph inputs for BERT model. First, we will deduce inputs from EmbedLayerNormalization node. If not found, we will guess the meaning of graph inputs based on naming. Args: onnx_file (str): onnx model path input_ids_name (str, optional): Name of graph input for input IDs. Defaults to None. segment_ids_name (str, optional): Name of graph input for segment IDs. Defaults to None. input_mask_name (str, optional): Name of graph input for attention mask. Defaults to None. Returns: Tuple[Optional[np.ndarray], Optional[np.ndarray], Optional[np.ndarray]]: input tensors of input_ids, segment_ids and input_mask rbN)rrKParseFromStringreadr r~)rrmrnromodelrRrgs r get_bert_inputsrs( LLE i  +$ diikk***+++++++++++++++5!!J J8H/ Z ZZs(AAActj}|ddtd|ddtdd|d dtd d |d dtd d|ddtdd|ddtdd|ddtdd|ddtd d|ddtdd|dddd|d|dddd |d!|d"d#d$td%&|d'd(ddd)|d*|d+dtd,d-|}|S).Nz--modelTzbert onnx model path.)requiredrhelpz --output_dirFz4output test data path. Default is current directory.)rrdefaultrz --batch_sizer+zbatch size of inputz--sequence_lengthz maximum sequence length of inputz--input_ids_namezinput name for input idsz--segment_ids_namezinput name for segment idsz--input_mask_namezinput name for attention maskz --samplesz$number of test cases to be generatedz--seedr2z random seedz --verbose store_truezprint verbose information)ractionr)rUz--only_input_tensorsz-only save input tensors and no output tensors)only_input_tensorsz-az--average_sequence_lengthz)average sequence length excluding padding)rrrz-rz--random_sequence_lengthz3use uniform random instead of fixed sequence length)r/z --mask_typer,z^mask type: (1: mask index, 2: raw 2D mask, 3: key lengths, cumulated lengths of query and key))argparseArgumentParser add_argumentstrint set_defaults parse_args)parserargss r parse_argumentsrs  $ & &F  DsAXYYY   C  S!Rghhh   /    '    )    ,    3  5sAMZZZ  (   &&&  <   5111  #  8   " B  u555   m      D Kr"r output_dirrc t|||| \}}}t||||||||| | | }t|D]E\}}tj|dt |z}t||F| rdSddl}d| vrddgndg}| ||}d| D}t|D]\}}tj|dt |z}| ||}t|D]\}}tjtj|||}t#tj|d|d d 5}||dddn #1swxYwYdS) aICreate test data for a model, and save test data to a directory. Args: model (str): path of ONNX bert model output_dir (str): output directory batch_size (int): batch size sequence_length (int): sequence length test_cases (int): number of test cases seed (int): random seed verbose (bool): whether print more information input_ids_name (str): graph input name of input_ids segment_ids_name (str): graph input name of segment_ids input_mask_name (str): graph input name of input_mask only_input_tensors (bool): only save input tensors, average_sequence_length (int): average sequence length excluding paddings random_sequence_length (bool): whether use uniform random number for sequence length mask_type(int): mask type test_data_set_NrCUDAExecutionProviderCPUExecutionProvider) providerscg|] }|j S)rP).0outputs r z-create_and_save_test_data..OsDDDFFKDDDr"output_r@rA)rr`rHrBrCrLrrS onnxruntimeget_available_providersInferenceSession get_outputsrunr rJrasarrayrKrMrN)rrr r rTrXrUrmrnrorr)r/r0r r#r.r[r6r<r;rrsession output_namesresult output_name tensor_resultrRs r create_and_save_test_datars\D*9P`bq)r)r&I{J#   Jz**,, 6GLL-=A-FGG F++++ #k&I&I&K&K K K !"899$ % **5I*FFGDDg.A.A.C.CDDDLz**>> 6GLL-=A-FGG \622' 55 > >NA{(3BJvay4I4I;WWMbgll9.>.>.>.>??FF >$ =::<<=== > > > > > > > > > > > > > > > >>>s,(G  G$ 'G$ c>t}|jdkr |j|_|j}|It |j}t j|j d|j d|j}|'t |}| ddntdt|j||j |j|j|j|j|j|j|j|j|j|j|jtd|dS)Nrbatch__seq_T)parentsexist_okz7Directory existed. test data files will be overwritten.z Test data is saved to directory:)rr)r rrrrBrCrLparentr rErFrsamplesrXrUrmrnrorr/r0)rrprCs r mainrZs    D #q(('+';$J   W\\!(,aT_,a,a4K_,a,abb J 4$ //// GHHH           $ # " ,j99999r"__main__)r,)NNN)#rrBrpathlibrtypingrrrnumpyronnxrrr rgr rndarrayr!r'r-boolr:rrSr^r`rlr~rrrr__name__rr"r rs ((((((((((6666666666 (+>ATWZ<{VY^`^h6B3BBBBB  FFFFF! F ! F  FZFFFFR33T#rz/-B3333.7777 7  7  7777!7!77777t1111  1  1  111!1!11111h   %)&*%) YXYXYXSMYXsmYXc] YX  8BJ "*!5x 7K KL YXYXYXYX|%)&*%) [[[SM[sm[c] [  8BJ "*!5x 7K KL [[[[8aaaHI> I>I>I> I>  I>  I>I>SMI>smI>c]I>I>!I>!I>I>I>I>I>X$:$:$:N zDFFFFFr"