NgF,ddlmZddlZddlZddlZddlmZddlmZm Z m Z ddl Z ddl m Z mZejeZer.ddlmZ ddlmZn #e$rYnwxYw ddlmZmZn #e$rYnwxYw d(d)dZ d(d*dZ d+d,d!Z d-d.d'ZdS)/) annotationsN)Path) TYPE_CHECKINGCallableLiteral)disable_datasets_cachingis_datasets_availableSentenceTransformer)OVQuantizationConfig)OptimizationConfigQuantizationConfigFmodelr optimization_config4OptimizationConfig | Literal['O1', 'O2', 'O3', 'O4']model_name_or_pathstr push_to_hubbool create_pr file_suffix str | NonereturnNonec R ddlm}ddlm} ddlm}m} ddlm} n#t$rtdwxYwt||r@t|r1t|d|rt|dj |std|dj } | |  ttr4| jvrtdpt#| d t% fd d |||d d S)a Export an optimized ONNX model from a SentenceTransformer model. The O1-O4 optimization levels are defined by Optimum and are documented here: https://huggingface.co/docs/optimum/main/en/onnxruntime/usage_guides/optimization The optimization levels are: - O1: basic general optimizations. - O2: basic and extended general optimizations, transformers-specific fusions. - O3: same as O2 with GELU approximation. - O4: same as O3 with mixed precision (fp16, GPU-only) See https://sbert.net/docs/sentence_transformer/usage/efficiency.html for more information & benchmarks. Args: model (SentenceTransformer): The SentenceTransformer model to be optimized. Must be loaded with `backend="onnx"`. optimization_config (OptimizationConfig | Literal["O1", "O2", "O3", "O4"]): The optimization configuration or level. model_name_or_path (str): The path or Hugging Face Hub repository name where the optimized model will be saved. push_to_hub (bool, optional): Whether to push the optimized model to the Hugging Face Hub. Defaults to False. create_pr (bool, optional): Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False. file_suffix (str | None, optional): The suffix to add to the optimized model file name. Defaults to None. Raises: ImportError: If the required packages `optimum` and `onnxruntime` are not installed. ValueError: If the provided model is not a valid SentenceTransformer model loaded with `backend="onnx"`. ValueError: If the provided optimization_config is not valid. Returns: None rr  Transformer)ORTModelForFeatureExtraction ORTOptimizer)AutoOptimizationConfigPlease install Optimum and ONNX Runtime to use this function. You can install them with pip: `pip install optimum[onnxruntime]` or `pip install optimum[onnxruntime-gpu]`]The model must be a Transformer-based SentenceTransformer model loaded with `backend="onnx"`.z\optimization_config must be an OptimizationConfig instance or one of 'O1', 'O2', 'O3', 'O4'.N optimizedc4|SN)r)optimize)save_dirrr optimizers Y/var/www/html/ai-engine/env/lib/python3.11/site-packages/sentence_transformers/backend.pyz-export_optimized_onnx_model..j););QRRTT! ssssss:"-       :quantization_configFQuantizationConfig | Literal['arm64', 'avx2', 'avx512', 'avx512_vnni']c  ddlm}ddlm} ddlm}m} ddlm} n#t$rtdwxYwt||r@t|r1t|d|rt|dj |std|dj } | |  ttr\dvrtd d d } t!| d p"jjd | !jjdt) fdd|||dd S)a Export a quantized ONNX model from a SentenceTransformer model. This function applies dynamic quantization, i.e. without a calibration dataset. Each of the default quantization configurations quantize the model to int8, allowing for faster inference on CPUs, but are likely slower on GPUs. See https://sbert.net/docs/sentence_transformer/usage/efficiency.html for more information & benchmarks. Args: model (SentenceTransformer): The SentenceTransformer model to be quantized. Must be loaded with `backend="onnx"`. quantization_config (QuantizationConfig): The quantization configuration. model_name_or_path (str): The path or Hugging Face Hub repository name where the quantized model will be saved. push_to_hub (bool, optional): Whether to push the quantized model to the Hugging Face Hub. Defaults to False. create_pr (bool, optional): Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False. file_suffix (str | None, optional): The suffix to add to the quantized model file name. Defaults to None. Raises: ImportError: If the required packages `optimum` and `onnxruntime` are not installed. ValueError: If the provided model is not a valid SentenceTransformer model loaded with `backend="onnx"`. ValueError: If the provided quantization_config is not valid. Returns: None rr r)r ORTQuantizer)AutoQuantizationConfigr!r")arm64avx2avx512 avx512_vnnizqquantization_config must be an QuantizationConfig instance or one of 'arm64', 'avx2', 'avx512', or 'avx512_vnni'.NF) is_static_ _quantizedc4|Sr%quantize)r'rrC quantizers r)r*z5export_dynamic_quantized_onnx_model..r+r,#export_dynamic_quantized_onnx_modelr.r/)r4r r5rr6rrFr7rGr8r9r:r;r<r=rr? weights_dtypenamelowerr@)rrCrrrrr rrrFrGrAquantization_config_namerRs ` ` @r)rSrSus"B:99999DDDDDD RRRRRRRRLLLLLLL     8     u1 2 2 5zz %(K00 %(-/KLL   k   /4Ah.AI,,Y77I%s++t &P P PD $7qqq#9 Rg&<>QRR]bccc!s(;(I(N(T(T(V(V%s%sYq%s%s ,:?EEGGSSS ssssssB"-      rBqint8_quantized"OVQuantizationConfig | dict | None dataset_namedataset_config_name dataset_split column_namec ddlm} ddlm}  ddlm} m} m}m}n#t$rtdwxYwtstdt| r@tr1td| rtdj | std| |}dj }| | ||t!d |||fDr,t#d |||fDstd fd ||nd}||nd}||nd}ndt%5||fd||jnd|dddn #1swxYwYt+fdd||||| ddS)a Export a quantized OpenVINO model from a SentenceTransformer model. This function applies Post-Training Static Quantization (PTQ) using a calibration dataset, which calibrates quantization constants without requiring model retraining. Each default quantization configuration converts the model to int8 precision, enabling faster inference while maintaining accuracy. See https://sbert.net/docs/sentence_transformer/usage/efficiency.html for more information & benchmarks. Args: model (SentenceTransformer): The SentenceTransformer model to be quantized. Must be loaded with `backend="openvino"`. quantization_config (OVQuantizationConfig | dict | None): The quantization configuration. If None, default values are used. model_name_or_path (str): The path or Hugging Face Hub repository name where the quantized model will be saved. dataset_name(str, optional): The name of the dataset to load for calibration. If not specified, the `sst2` subset of the `glue` dataset will be used by default. dataset_config_name (str, optional): The specific configuration of the dataset to load. dataset_split (str, optional): The split of the dataset to load (e.g., 'train', 'test'). Defaults to None. column_name (str, optional): The column name in the dataset to use for calibration. Defaults to None. push_to_hub (bool, optional): Whether to push the quantized model to the Hugging Face Hub. Defaults to False. create_pr (bool, optional): Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False. file_suffix (str, optional): The suffix to add to the quantized model file name. Defaults to `qint8_quantized`. Raises: ImportError: If the required packages `optimum` and `openvino` are not installed. ValueError: If the provided model is not a valid SentenceTransformer model loaded with `backend="openvino"`. ValueError: If the provided quantization_config is not valid. Returns: None rr r)OVConfigOVModelForFeatureExtractionr OVQuantizerzPlease install datasets, optimum-intel and openvino to use this function. You can install them with pip: `pip install datasets optimum[openvino]`zaPlease install datasets to use this function. You can install it with pip: `pip install datasets`zaThe model must be a Transformer-based SentenceTransformer model loaded with `backend="openvino"`.N)rCc3K|]}|duV dSN.0params r) z9export_static_quantized_openvino_model..s' j j5  j j j j j jr,c3K|]}|duV dSrcrdres r)rhz9export_static_quantized_openvino_model..s@ww#Twwwwwwr,zEither specify all of `dataset_name`, `dataset_config_name`, `dataset_split`, and `column_name`, or leave them all unspecified.c6|dddS)N max_lengthiT)paddingrk truncation) tokenizer)examplesrs r)preprocess_functionzCexport_static_quantized_openvino_model..preprocess_functionsx#Z^___r,gluesst2trainsentencec&|Srcrd)ror]rps r)r*z8export_static_quantized_openvino_model..%s1D1DXkEZ1[1[r,i,)rZr[rp num_samplesr\c4|S)N)save_directory ov_configrP)r'calibration_datasetryrRs r)r*z8export_static_quantized_openvino_model..+s#);); I*<* * r,&export_static_quantized_openvino_modelopenvinor/)r4r r5r optimum.intelr_r`r rar8r r9r:r;r<r=anyallrget_calibration_datasetrvr@)rrCrrZr[r\r]rrrr rr_r`r raov_modelrzryrprRs` ` @@@@r)r{r{sJT:99999DDDDDD jjjjjjjjjjjjj     V    ! " "  o   u1 2 2 5zz %(K00 %(-/JKK   o   "2244,1!H,?H-@AAAI++H55I j j<9Lm]h*i j j jjj svww(46I=Ze'fwwwtt  N   `````$0#;<<L1D1P--V\%2%>MMGM!,!8++jK ! # #  '??% 3 [ [ [ [ [;N;Z+77`c' @                       F"-      s !;((FF #F r.r0rr1r3c|dkrd|d}n |dkrd|d}tj5} || |dkrot| |z } tj| dz | |z tj| dz | |z d | } |dkrit| |z } | d d t| |z } | |z } tj| | | } |rd } |r`t| d d dd dd}d|d|d|d|d|d|d } tj | ||dd|d|| |nt||z } | d d t| |z } | |z } tj | | |dkr_t| |z d }t| |z d }tj ||ddddS#1swxYwYdS)Nr.model_z.onnxr|openvino_model_z.xmlzopenvino_model.xmlzopenvino_model.binz.binT)parentsexist_ok(z( z, z, )z )zGHello! *This pull request has been automatically generated from the [`zT`](https://sbert.net/docs/package_reference/util.html#sentence_transformers.backend.zI) function from the Sentence Transformers library.* ## Config ```python a ``` ## Tip: Consider testing this pull request before merging by loading the model from this PR with the `revision` argument: ```python from sentence_transformers import SentenceTransformer # TODO: Fill in the PR number pr_number = 2 model = SentenceTransformer( "z5", revision=f"refs/pr/{pr_number}", backend="z#", model_kwargs={"file_name": "a "}, ) # Verify that everything works as expected embeddings = model.encode(["The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium."]) print(embeddings.shape) similarities = model.similarity(embeddings, embeddings) print(similarities) ``` rz Add exported z model ) folder_path path_in_reporepo_id repo_typecommit_messagecommit_descriptionr) tempfileTemporaryDirectoryrshutilmove with_suffixas_posixmkdirreprreplacehuggingface_hub upload_foldercopy)r0r1r2rrrrr3 file_namer'dst_dirsource destinationropt_config_string bin_sourcebin_destinations r)r@r@8sp&/[/// J  7k777  $ & &Q9(!!! j H~~/H K#77I9M N N N K#77(Y:N9[9[\b9c9c d d d((**H f  8nnw.G MM$M 6 6 6(^^i/F!I-K K , , ,''))H < 9!# ! $(LL$8$8g$F$F$N$NtU\$]$]$e$efikp$q$q!&AU&&k&&&&$%&&()&&*#,+&&&"B  )$$*!LwLLyLL#5#     -..8G MM$M 6 6 6(^^i/F!I-K K , , ,*$$"8nny8EEfMM #'==9#<"I"I&"Q"Q J888cQ9Q9Q9Q9Q9Q9Q9Q9Q9Q9Q9Q9Q9Q9Q9Q9Q9Q9sH2I,,I03I0)FFN)rr rrrrrrrrrrrr)rr rCrDrrrrrrrrrr)NNNNFFrX)rr rCrYrrrZrr[rr\rr]rrrrrrrrr)FFNr.)r0rr1rrrrrrrrrr3r) __future__rloggingrrpathlibrtypingrrrrsentence_transformers.utilrr getLogger__name__logger)sentence_transformers.SentenceTransformerr r}r r8r7r rr-rSr{r@rdr,r)rs"""""" 3333333333VVVVVVVV  8 $ $  MMMMMM 6666666       \\\\\\\\\      " VVVVVz" QQQQQp $&* $"(lllllh"`9`9`9`9`9`9`9s$AAAA$$A,+A,