Ng}/ddlmZddlZddlZddlmZddlmZddlZddlm Z m Z ddl m Z ddlmZddlmZdd lmZdd lmZGd d ZGd dZGdde jZdS)) annotationsN)Iterable)Any)Tensornn) functional)SentenceTransformer)CachedGISTEmbedLoss)"CachedMultipleNegativesRankingLoss) TransformercBeZdZdZddZddZdd Zdd Zdd ZddZ dS)TransformerDecoratorz Decorator that caches the embeddings of all layers of the transformer. When `layer_idx` is set, it returns the cached embeddings of that layer instead. This is meant to override the forward function of the Transformer. transformerr returnNonech||_||_g|_g|_g|_d|_d|_dSNr)roriginal_forward embeddingslast_embeddingsfeatures layer_idxcall_idx)selfrrs j/var/www/html/ai-engine/env/lib/python3.11/site-packages/sentence_transformers/losses/AdaptiveLayerLoss.py__init__zTransformerDecorator.__init__s:& 0/1-/13  c"||_d|_dSr)rr)rrs r set_layer_idxz"TransformerDecorator.set_layer_idx#s" rrcRtjfdjDdS)Nc*g|]}|jS)r).0 embeddingrs r z=TransformerDecorator.get_layer_embeddings..(s XXX9Yt~6XXXrdimtorchconcatrrs`rget_layer_embeddingsz)TransformerDecorator.get_layer_embeddings's0|XXXXXXX^_````rdict[str, Tensor]c|j||}n%||}|xjdz c_|S)Nr&)rcall_grow_cachecall_use_cacherrroutputs r__call__zTransformerDecorator.__call__*sG > !))(33FF((22F MMQ MM rrc|jjjj}d|jjj_||}t |ddz |_|j|ddd|j |d|j d| D||jjj_|r|d=|S)z Temporarily sets the output_hidden_states to True, runs the model, and then restores the original setting. Use the all_layer_embeddings to get the embeddings of all layers. Tall_layer_embeddingsr&token_embeddingsc"i|] \}}|dv || S))r6r8r")r#keyvalues r z8TransformerDecorator.call_grow_cache..As) t t tJCCGs???  t t&,,.. t t t   C`#*? ( /-. rchi|j|jd|j|j|jiS)Nr8)rrrr)rrs rr1z#TransformerDecorator.call_use_cacheLs5s$- .s0BDOTXTaDbcgcqDrsssrN)rr rrrrrr)rr.rr.rr.) __name__ __module__ __qualname____doc__rrr-r4r0r1r"rrrrsaaaa4ttttttrrc*eZdZdZd dZd dZd d Zd S)ForwardDecoratorz Decorator that caches the embeddings after all modules (e.g. pooling) of the model. Required to get the embeddings after all modules for the KL-divergence loss. This is meant to override the forward function of the SentenceTransformer. rrc"||_g|_dS)N)fnr)rrPs rrzForwardDecorator.__init__Xsrrr.cp||}|j|d|S)Nsentence_embedding)rPrrBr2s rr4zForwardDecorator.__call__\s4"" v&:;<<< rrcJtj|jd}g|_|S)Nrr'r))rrs rget_embeddingszForwardDecorator.get_embeddingsas%\$/q999 rNrFrHrG)rIrJrKrLrr4rTr"rrrNrNPsZ rrNcTeZdZ ddfd ZddZddZeddZxZS) AdaptiveLayerLossr&?333333?modelr loss nn.Modulen_layers_per_stepintlast_layer_weightfloatprior_layers_weight kl_div_weightkl_temperaturerrct||_||_||_||_||_||_||_t|jdtsJt|trtj ddt|trtj dddSdS)a4 The AdaptiveLayerLoss can be seen as a loss *modifier* that allows you to use other loss functions at non-final layers of the Sentence Transformer model. This is useful for when you want to train a model where users have the option to lower the number of layers used to improve their inference speed and memory usage. Args: model: SentenceTransformer model loss: The loss function to be used, e.g. :class:`MultipleNegativesRankingLoss`, :class:`CoSENTLoss`, etc. n_layers_per_step: The number of layers to use per step. If -1, then all layers are used. If > 0, then a random sample of `n_layers_per_step` layers are used per step, separate from the final layer, which is always used. The 2DMSE paper uses `n_layers_per_step=1`. The default value is 1. last_layer_weight: The weight to use for the loss of the final layer. Increase this to focus more on the performance when using all layers. The default value is 1.0. prior_layers_weight: The weight to use for the loss of the prior layers. Increase this to focus more on the performance when using fewer layers. The default value is 1.0. kl_div_weight: The weight to use for the KL-divergence loss that is used to make the prior layers match that of the last layer. Increase this to focus more on the performance when using fewer layers. The default value is 1.0. kl_temperature: The temperature to use for the KL-divergence loss. If 0, then the KL-divergence loss is not used. The default value is 1.0. References: - The concept was inspired by the 2DMSE paper: https://arxiv.org/abs/2402.14776 - `Adaptive Layers <../../examples/training/adaptive_layer/README.html>`_ Requirements: 1. The base loss cannot be :class:`CachedMultipleNegativesRankingLoss` or :class:`CachedGISTEmbedLoss`. Inputs: +---------------------------------------+--------+ | Texts | Labels | +=======================================+========+ | any | any | +---------------------------------------+--------+ Relations: - :class:`Matryoshka2dLoss` uses this loss in combination with :class:`MatryoshkaLoss` which allows for output dimensionality reduction for faster downstream tasks (e.g. retrieval). Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.MultipleNegativesRankingLoss(model=model) loss = losses.AdaptiveLayerLoss(model, loss) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() rzIMatryoshkaLoss is not compatible with CachedMultipleNegativesRankingLoss.) stacklevelz:MatryoshkaLoss is not compatible with CachedGISTEmbedLoss.N)superrrYrZr\r^r`rarb isinstancer r warningswarnr ) rrYrZr\r^r`rarb __class__s rrzAdaptiveLayerLoss.__init__hsd   !2!2#6 *,$*Q-55555 d> ? ? u Mers t t t t d/ 0 0 f MVcd e e e e e e f frsentence_featuresIterable[dict[str, Tensor]]labelsrc|jdj}t|jd|}||jd_|jj}t|}||j_||||jz}|jdkr2|}tj ||jz d}|j } t| dz } |j dkr(|j | dz krtj| |j } | D]} || |||} || d| zz t!| z |jzz}|jdkr\|} tjtj| |jz d|d}|||jz|jzz}||jd_||j_|S)Nrr7r'r& batchmean) reduction)rYforwardrrNrZr^rbrTFsoftmaxrAranger\randomsamplerr@r`kl_div log_softmaxra)rrkrmoriginal_transformer_forwardtransformer_decoratorrforward_decoratorrZfinal_embeddingsrA layer_indicesr layer_lossr kl_div_losss rrqzAdaptiveLayerLoss.forwards'+z!}'<$ 4TZ]D` a a 5 1  :-,-=>>.  yy*F33d6LL   " "0??AA  y)9D*Q*Q"M-9OPPM' U UI ! / / : : :#4f==J*I 6]9K9KKdNfffD"Q&&.==?? hM*t/B"BKKK$) kD,??$BTTT < 1 -  rdict[str, Any]cd|jjj|j|j|j|j|jdS)N)rZr\r^r`rarb)rZrjrIr\r^r`rarbr,s rget_config_dictz!AdaptiveLayerLoss.get_config_dicts:I'0!%!7!%!7#'#;!/"1    rstrcdS)Nz @misc{li20242d, title={2D Matryoshka Sentence Embeddings}, author={Xianming Li and Zongxi Li and Jing Li and Haoran Xie and Qing Li}, year={2024}, eprint={2402.14776}, archivePrefix={arXiv}, primaryClass={cs.CL} } r"r,s rcitationzAdaptiveLayerLoss.citations   r)r&rWrWrWrX)rYr rZr[r\r]r^r_r`r_rar_rbr_rr)rkrlrmrrr)rr)rr) rIrJrKrrqrpropertyr __classcell__)rjs@rrVrVgs "##&%(" #^f^f^f^f^f^f^f@----^       X     rrV) __future__rrurhcollections.abcrtypingrr*rrtorch.nnrrrsentence_transformersr 0sentence_transformers.losses.CachedGISTEmbedLossr ?sentence_transformers.losses.CachedMultipleNegativesRankingLossr sentence_transformers.modelsr rrNModulerVr"rrrsa"""""" $$$$$$ $$$$$$555555PPPPPPnnnnnn444444;t;t;t;t;t;t;t;t|.eeeee eeeeer