Ng_%ddlmZddlZddlZddlmZddlZddlZddl m Z ddl m Z ddlmZddlmZddlmZdd lmZGd d ejZdS) ) annotationsN)Path) load_file) save_file) Tokenizer)nn)PreTrainedTokenizerFast)get_device_nameceZdZ d*d+fd Zd,dZd-dZd.dZed/dZd/dZ d0d1dZ d2dZ e d3d4d'Z e d5d)ZxZS)6StaticEmbeddingN tokenizer#Tokenizer | PreTrainedTokenizerFastembedding_weightsnp.array | torch.Tensor | None embedding_dim int | NonereturnNonec tt|tr|j}n$t|t st d|Ut|tjrtj |}tj |d|_n>|-tj |||_nt d|jj|_|jj|_||_|j|dd|_dS)u Initializes the StaticEmbedding model given a tokenizer. The model is a simple embedding bag model that takes the mean of trained per-token embeddings to compute text embeddings. Args: tokenizer (Tokenizer | PreTrainedTokenizerFast): The tokenizer to be used. Must be a fast tokenizer from ``transformers`` or ``tokenizers``. embedding_weights (np.array | torch.Tensor | None, optional): Pre-trained embedding weights. Defaults to None. embedding_dim (int | None, optional): Dimension of the embeddings. Required if embedding_weights is not provided. Defaults to None. Example:: from sentence_transformers import SentenceTransformer from sentence_transformers.models import StaticEmbedding from tokenizers import Tokenizer # Pre-distilled embeddings: static_embedding = StaticEmbedding.from_model2vec("minishlab/M2V_base_output") # or distill your own embeddings: static_embedding = StaticEmbedding.from_distillation("BAAI/bge-base-en-v1.5", device="cuda") # or start with randomized embeddings: tokenizer = Tokenizer.from_pretrained("FacebookAI/xlm-roberta-base") static_embedding = StaticEmbedding(tokenizer, embedding_dim=512) model = SentenceTransformer(modules=[static_embedding]) embeddings = model.encode(["What are Pandas?", "The giant panda (Ailuropoda melanoleuca; Chinese: 大熊猫; pinyin: dàxióngmāo), also known as the panda bear or simply the panda, is a bear native to south central China."]) similarity = model.similarity(embeddings[0], embeddings[1]) # tensor([[0.9177]]) (If you use the distilled bge-base) Raises: ValueError: If the tokenizer is not a fast tokenizer. ValueError: If neither `embedding_weights` nor `embedding_dim` is provided. zThe tokenizer must be fast (i.e. Rust-backed) to use this class. Use Tokenizer.from_pretrained() from `tokenizers` to load a fast tokenizer.NF)freezez?Either `embedding_weights` or `embedding_dim` must be provided. base_model)super__init__ isinstancer _tokenizerr ValueErrornpndarraytorch from_numpyr EmbeddingBagfrom_pretrained embeddingget_vocab_sizenum_embeddingsrr no_paddinggetr)selfr rrkwargs __class__s h/var/www/html/ai-engine/env/lib/python3.11/site-packages/sentence_transformers/models/StaticEmbedding.pyrzStaticEmbedding.__init__s9V  i!8 9 9 !,IIIy11 ^   (+RZ88 H$)$45F$G$G!_<<=NW\<]]DNN  &_Y-E-E-G-GWWDNN^__ _"n;!^9$- !!###!**\488texts list[str]dict[str, torch.Tensor]c (|j|d}d|D}tjt jdgd|ddDz}tjd|Dtj}||d S) NF)add_special_tokenscg|] }|j S)ids).0encodings r+ z,StaticEmbedding.tokenize..]s@@@(@@@r,rc,g|]}t|Sr3)len)r5 token_idss r+r7z,StaticEmbedding.tokenize.._s3g3g3gyC NN3g3g3gr,cg|] }|D]}| Sr3r3)r5r:token_ids r+r7z,StaticEmbedding.tokenize..`s&!d!d!dyZc!d!dh(!d!d!d!dr,)dtype) input_idsoffsets)r encode_batchrr rcumsumtensorlong)r(r-r) encodings encodings_idsr@r?s r+tokenizezStaticEmbedding.tokenize[sN//%/PP @@i@@@ "29aS3g3gTabecebeTf3g3g3g-g#h#hiiL!d!dM!d!d!dlqlvwww &7;;;r,featuresc P||d|d|d<|S)Nr?r@sentence_embedding)r#)r(rHr)s r+forwardzStaticEmbedding.forwardcs+)-8MxXaOb)c)c%&r,dict[str, float]ciSNr3r(s r+get_config_dictzStaticEmbedding.get_config_dictgs r,intctjSrN)mathinfrOs r+max_seq_lengthzStaticEmbedding.max_seq_lengthjs xr,c|jSrN)rrOs r+ get_sentence_embedding_dimensionz0StaticEmbedding.get_sentence_embedding_dimensionns !!r,Tsave_dirstrsafe_serializationboolc |rAt|tj|dnEt j|tj|d|jtt|dz dS)Nmodel.safetensorspytorch_model.bintokenizer.json) save_safetensors_file state_dictospathjoinrsaver rYr)r(rXrZr)s r+rezStaticEmbedding.saveqs  W !$//"3"3RW\\(L_5`5` a a a a Jt(("',,xAT*U*U V V V CX1A ABBCCCCCr,load_dirc tjtt|dz }tjtj|dr.ttj|d}nGtj tj|dtj dd}|d}t||S) Nr_r]r^cpuT) map_location weights_onlyzembedding.weight)r) r from_filerYrrbrcexistsrdload_safetensors_filerloaddevicer )rfr)r weightss r+rnzStaticEmbedding.loadxs'DNN=M,M(N(NOO 7>>"',,x1DEE F F +BGLLCV,W,WXXGGj X':;;%,W\J]J]lpG,-yGDDDDr, model_name vocabularylist[str] | Nonero str | Nonepca_dims apply_zipf use_subwordc> ddlm}n#t$rtdwxYwt}|||||||}t |jt jrtj |j} n |jj } |j } || | |S)a Creates a StaticEmbedding instance from a distillation process using the `model2vec` package. Args: model_name (str): The name of the model to distill. vocabulary (list[str] | None, optional): A list of vocabulary words to use. Defaults to None. device (str): The device to run the distillation on (e.g., 'cpu', 'cuda'). If not specified, the strongest device is automatically detected. Defaults to None. pca_dims (int | None, optional): The number of dimensions for PCA reduction. Defaults to 256. apply_zipf (bool): Whether to apply Zipf's law during distillation. Defaults to True. use_subword (bool): Whether to use subword tokenization. Defaults to True. Returns: StaticEmbedding: An instance of StaticEmbedding initialized with the distilled model's tokenizer and embedding weights. Raises: ImportError: If the `model2vec` package is not installed. r)distillz\To use this method, please install the `model2vec` package: `pip install model2vec[distill]`)rsrorvrwrxrr) model2vec.distillrz ImportErrorr rr#rrrr weightr ) clsrrrsrorvrwrxrz static_modelrr s r+from_distillationz!StaticEmbedding.from_distillations<  1 1 1 1 1 1 1   n   !""w !!#     l,bj 9 9 > % 01G H H   , 6 = +5 s90AjYYYY #model_id_or_pathc* ddlm}n#t$rtdwxYw||}t |jt jrtj |j}n |jj }|j }||||S)aH Create a StaticEmbedding instance from a model2vec model. This method loads a pre-trained model2vec model and extracts the embedding weights and tokenizer to create a StaticEmbedding instance. Args: model_id_or_path (str): The identifier or path to the pre-trained model2vec model. Returns: StaticEmbedding: An instance of StaticEmbedding initialized with the tokenizer and embedding weights the model2vec model. Raises: ImportError: If the `model2vec` package is not installed. r) StaticModelzSTo use this method, please install the `model2vec` package: `pip install model2vec`r{) model2vecrr}r"rr#rrrr r~r )rrrrrr s r+from_model2veczStaticEmbedding.from_model2vecs" u - - - - - - - u u ustt t u#223CDD l,bj 9 9 > % 01G H H   , 6 = +5 s90AN^____r)NN)r rrrrrrr)r-r.rr/)rHr/rr/)rrL)rrQ)T)rXrYrZr[rr)rfrYrr )NNrqTT)rrrYrsrtrorurvrrwr[rxr[rr )rrYrr )__name__ __module__ __qualname__rrGrKrPpropertyrUrWrern classmethodrr __classcell__)r*s@r+r r s]=A$( F9F9F9F9F9F9F9P<<<<X""""DDDDD E E E E(,!" 3Z3Z3Z3Z[3Zj```[`````r,r ) __future__rrSrbpathlibrnumpyrrsafetensors.torchrrmrr` tokenizersrr transformersr sentence_transformers.utilr Moduler r3r,r+rs""""""  @@@@@@@@@@@@ 000000666666D`D`D`D`D`biD`D`D`D`D`r,