gNddlZddlmZddlmZmZddlmZddlm Z m Z m Z m Z ddl mZmZddlmZmZddlmZdd lmZmZmZed Zgd ZGd d eZdededefdZdefdZdZdedede e ee ee effdZ d3dededede edef dZ!dede"dede#fdZ$dede"defd Z% d4dededede"de ede#defd#Z&d$edefd%Z'd$edefd&Z(d'efd(Z)e*d)krneZ+e+,Z-ee-j./e-_. e0d*e&e-j1e-j2e-j.e-j3e-j4e-j5e-je-j(rVeee-j1d+kr e0d,e0d-e'e-j.e-_6e(e-j6e-_7e-j8rUe0d.e)e-j.e9e-d/re)e-j6e9e-d0re)e-j7dSdSdS#e:$r$Z;e0d1e;e= than 2Gbz --quantizez/Quantize the neural network to be run with int8output)super__init__ add_argumentstrSUPPORTED_PIPELINESint)self __class__s ^/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/convert_graph_to_onnx.pyr'z$OnnxConverterArgumentParser.__init__3sm )***  '(     G    -c8rsss  4L2    )#r@STTT  7     #4     B    (#####)__name__ __module__ __qualname____doc__r' __classcell__)r-s@r.rr.sB&$&$&$&$&$&$&$&$&$r/rfilename identifierreturncv|j|j|z|jS)aE Append a string-identifier at the end (before the extension, if any) to the provided filepath Args: filename: pathlib.Path The actual path object we would like to add an identifier suffix identifier: The suffix to add Returns: String with concatenated identifier at the end of the filename )parentjoinpathstem with_suffixsuffix)r5r6s r.generate_identified_filenamer>\s1 ? # #HMJ$> ? ? K KHO \ \\r/minimum_versionc ddl}t|j}|tkrt d|jd|ddS#t$rt dwxYw)z Check onnxruntime is installed and if the installed version match is recent enough Raises: ImportError: If onnxruntime is not installed or too old version is found rNz*We found an older version of onnxruntime (z&) but we require onnxruntime to be >= zp to enable all the conversions options. Please update onnxruntime by running `pip install --upgrade onnxruntime`zonnxruntime doesn't seem to be currently installed. Please install the onnxruntime by running `pip install onnxruntime` and relaunch the conversion.) onnxruntimer __version__ORT_QUANTIZE_MINIMUM_VERSION ImportError)r?rA ort_versions r.check_onnxruntime_requirementsrFis K344  5 5 5[[=T[[7F[[[  6 5     ,    s >AAc8td|jjj}gg}}|ddD]I}||vr1|||||7t|dtd||t |fS)a Ensure inputs are presented in the correct order, without any Non Args: model: The model used to forward the input data tokens: BatchEncoding holding the input data input_names: The name of the inputs Returns: Tuple z$Ensuring inputs are in correct orderNz, is not present in the generated input list.zGenerated inputs order: )printforward__code__ co_varnamesappendtuple)modeltokens input_namesmodel_args_name model_argsordered_input_namesarg_names r.ensure_valid_inputrVs 0111m,8O&("#J#ABB' { " "  & &x 0 0 0   fX. / / / / XKKK L L L  :%8 : :;;; j 1 1 11r/nlp frameworkc: dtdtdtf fd |d|}|jjd |dkr |jd i|n||}t|tr| }t|ttfs|f}t| } fd | D}g}|D]I}t|ttfr||4||Jd t!t#|D} fd t%||D} t'|fi| } ||| |fS) a? Attempt to infer the static vs dynamic axes for each input and output tensors for a specific model Args: nlp: The pipeline object holding the model to be exported framework: The framework identifier to dispatch to the correct inference scheme (pt/tf) Returns: - List of the inferred input variable names - List of the inferred output variable names - Dictionary with input/output variables names as key and shape tensor as value - a BatchEncoding reference which was used to infer all the above information nameis_inputseq_lenct|ttfrfd|DSdt|jDddi}rCt |jdkrd|d<ndt dt |jd fd t|jD}|d |Dtd rd nddd||S)Nc,g|]}|Sr_).0tbuild_shape_dictr[rZr\s r. z:infer_shapes..build_shape_dict..s+QQQQ$$T1h@@QQQr/c$g|] \}}|dk |S)rHr_)r`axisnumels r.rcz:infer_shapes..build_shape_dict..s"SSSkdEQR T r/rbatchsequencerHzUnable to infer tensor axes ()c&g|] \}}|k |Sr_r_)r`dimshaper\s r.rcz:infer_shapes..build_shape_dict..s(^^^JCUV]M]M]CM]M]M]r/ci|]}|dS)rir_)r`rls r. z:infer_shapes..build_shape_dict..sAAAS*AAAr/zFound inputr% z with shape: ) isinstancerNlist enumeratermlen ValueErrorupdaterI)rZtensorr[r\axesseq_axesrbs` `` r.rbz&infer_shapes..build_shape_dicts2 fudm , , CQQQQQQQ&QQQ QTSYv|-D-DSSSTUVX_`D Cv|$$))(DGG$%YSEVEV%Y%Y%YZZZ^^^^)FL2I2I^^^ AAAAABBB T(8wwTT4TTdTTUUU r/zThis is a sample output)return_tensorsr c 4i|]\}}|||dS)Tr_r`kvrbr\s r.roz infer_shapes..s1^^^tq!!--aD'BB^^^r/cg|]}d|S)output_r_)r`is r.rcz infer_shapes..sDDDaMaMMDDDr/c 4i|]\}}|||dS)Fr_r~s r.roz infer_shapes..s1qqqA1..q!UGDDqqqr/r_)r)boolr+ tokenizer input_idsrmrOrrrto_tuplersrNkeysitemsextendrMrangeruzipdict) rWrXrPoutputs input_varsinput_dynamic_axes outputs_flatr% output_namesoutput_dynamic_axes dynamic_axesrbr\s @@r. infer_shapesrs sdS&]]4Y] O OF$R(G%.$%6%6ici!!&!!!CIIf 2Gb /!\rN)rBzUsing framework TensorFlow: z , tf2onnx: r!cNg|]!\}}j||"S))rZ) TensorSpec from_tensor)r`keyrxr!s r.rcz&convert_tensorflow..Cs3iii;3PV2=44V#4FFiiir/)r output_pathzCannot import z6 required to convert TF model to ONNX. Please install z first. )rrrI tensorflowtf2onnxrBversionVERSIONrrOpredictdatarconvert from_kerasrrDrZ)rWrr%rt2ovrQrrrPinput_signature model_proto_er!s @r.convert_tensorflowr&sk   hfggg RSSS ////// RRZ-?RRDRRSSS;GsD:Q:Q7 \< &+&&&iiiiZ`ZfZfZhZhiii 33 IeARAR4   QQQ     nQV n n[\[a n nkl n n    sB9C** D4"DDFrc Dtjdttd|t ||||fi|}|js>td|jt|jnatt|jdkr*td|jd|dkrt||||dSt|||dS) a Convert the pipeline object to the ONNX Intermediate Representation (IR) format Args: framework: The framework the pipeline is backed by ("pt" or "tf") model: The name of the model to load for the pipeline output: The path where the ONNX graph will be stored opset: The actual version of the ONNX operator set to use tokenizer: The name of the model to load for the pipeline, default to the model's name if not provided use_external_format: Split the model definition from its parameters to allow model bigger than 2GB (PyTorch only) pipeline_name: The kind of pipeline to instantiate (ner, question-answering, etc.) model_kwargs: Keyword arguments to be forwarded to the model constructor Returns: zoThe `transformers.convert_graph_to_onnx` package is deprecated and will be removed in version 5 of TransformerszONNX opset version set to: zCreating folder rzFolder z" is not empty, aborting conversionr N)warningswarn FutureWarningrIrr9existsrrrurrrr) rXrOr%rrrrrrWs r.rrNs66 M   / / /000 }i Z Z\ Z ZC =   ! !` 000111''))**** WV]++-- . . / /! 3 3^&-"8"8":":^^^___DUF,?@@@@@3v.....r/onnx_model_pathcddlm}m}t|d}|}||_|||}t d|dt d|S)a> Load the model at the specified path and let onnxruntime look at transformations on the graph to enable all the optimizations possible Args: onnx_model_path: filepath where the model binary description is stored Returns: Path where the optimized model binary description has been saved rInferenceSessionSessionOptionsz -optimizedz$Optimized model has been written at : ✔zY/!\ Optimized model contains hardware specific operators which might not be portable. /!\)rArrr>roptimized_model_filepathrI)rrropt_model_path sess_optionrs r.optimizers=<<<<<<<2/<PPN .""K+9+B+B+D+DK(1133[AAA W W W WXXX ghhh r/cddl}ddl}ddlm}ddlm}ddlm}ddlm }| | }t|j tdkrtd|}||t|j td kr)||d d |jd d d dddt#| } n(||d d |jd d d dddt#| } | t'|d} td| d|| jj| | S)z Quantize the weights of the model from float32 to in8 to allow very efficient inference on modern CPU Args: onnx_model_path: Path to location the exported ONNX model is stored Returns: The Path generated for the quantized rN) ModelProto)QuantizationMode) ONNXQuantizer)IntegerOpsRegistryz1.5.0zpModels larger than 2GB will fail to quantize due to protobuf constraint. Please upgrade to onnxruntime >= 1.5.0.z1.13.1FT) rO per_channel reduce_rangemodestatic weight_qType input_qType tensors_rangenodes_to_quantizenodes_to_excludeop_types_to_quantize) rOrrrrractivation_qTyperrrrz -quantizedz$Quantized model has been written at r)onnxrA onnx.onnx_pbronnxruntime.quantizationr'onnxruntime.quantization.onnx_quantizerr!onnxruntime.quantization.registryrloadrr rBrICopyFrom IntegerOpsrsquantize_modelr> save_modelrO) rrrArrrr onnx_model copy_model quantizerquantized_model_paths r.quantizersKKK''''''999999EEEEEEDDDDDD?335566J T w//  6   J ###  [ $%%h77!M!,"!!%&8!9!9    "M!,""!!%&8!9!9    8VV ]1E ] ] ]^^^OOIO)+?+H+H+J+JKKK r/pathcddlm}m}ddlm}t d|d |}|||dg}t d|d dS#|$r}t d |d Yd}~dSd}~wwxYw) Nrr)RuntimeExceptionz"Checking ONNX model loading from: z ...CPUExecutionProvider) providerszModel u correctly loaded: ✔zError while loading the model u: ✘)rArr+onnxruntime.capi.onnxruntime_pybind11_staterrIr)rrrr onnx_optionsrres r.verifyrs<<<<<<<<LLLLLL 9t 9 9 9:::I%~''  T]]__lG]F^ _ _ _ DtDDDEEEEE III GrGGGHHHHHHHHHIs>A##B(BB__main__z' ====== Converting model to ONNX ======r!aV Using TensorFlow might not provide the same optimization level compared to PyTorch. For TensorFlow users you can try optimizing the model directly through onnxruntime_tools. For more information, please refer to the onnxruntime documentation: https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers z$ ====== Optimizing ONNX model ======z+ ====== Check exported ONNX model(s) ======optimized_outputquantized_outputz"Error while converting the model: rH)N)NFr)=rargparserosrrpathlibrtypingrrr r packaging.versionr r transformers.pipelinesr rtransformers.tokenization_utilsrtransformers.utilsrrrrCr*rr)r>rFrVrrr+rrrrrrrr0parser parse_argsargsr%absoluterIrXrOrrrrr check_loadinghasattrrrexitr_r/r.rsY###### ............,,,,,,,,55555555999999OOOOOOOOOO %uW~~   +$+$+$+$+$.+$+$+$\ ]4 ]S ]T ] ] ] ] G    :2228=:h=:3=:5cDItUb9b3c=:=:=:=:BPTvvv#&v/2v?G}v vvvv>" " #" t" RV" " " " J% H% S% $% % % % Z $ %-////// // //  // } //  //////////ddt2H dH tH H H H V I I I I I z ( ( * *F     D$t{##,,..DK/ 8999 N J K J N  $ M    = D * *+G H H H~%%w E9 : : :%-HT[$9$9D !%-HT-B$C$CD !   . E@ A A A F4;   wt/00 .t,---wt/00 .t,----- . . . .  6166777 QisC6H<<I%I  I%