NgddlmZddlmZddlmZddlZddlmZmZddl m Z ddl m Z Gdd ej ZdS) ) annotations)Iterable)AnyN)Tensornn)util)SentenceTransformercXeZdZdejfdfd Zdd ZddZeddZ xZ S)MultipleNegativesRankingLossg4@modelr scalefloatreturnNonect||_||_||_t j|_dS)ad This loss expects as input a batch consisting of sentence pairs ``(a_1, p_1), (a_2, p_2)..., (a_n, p_n)`` where we assume that ``(a_i, p_i)`` are a positive pair and ``(a_i, p_j)`` for ``i != j`` a negative pair. For each ``a_i``, it uses all other ``p_j`` as negative samples, i.e., for ``a_i``, we have 1 positive example (``p_i``) and ``n-1`` negative examples (``p_j``). It then minimizes the negative log-likehood for softmax normalized scores. This loss function works great to train embeddings for retrieval setups where you have positive pairs (e.g. (query, relevant_doc)) as it will sample in each batch ``n-1`` negative docs randomly. The performance usually increases with increasing batch sizes. You can also provide one or multiple hard negatives per anchor-positive pair by structuring the data like this: ``(a_1, p_1, n_1), (a_2, p_2, n_2)``. Then, ``n_1`` is a hard negative for ``(a_1, p_1)``. The loss will use for the pair ``(a_i, p_i)`` all ``p_j`` for ``j != i`` and all ``n_j`` as negatives. Args: model: SentenceTransformer model scale: Output of similarity function is multiplied by scale value similarity_fct: similarity function between sentence embeddings. By default, cos_sim. Can also be set to dot product (and then set scale to 1) References: - Efficient Natural Language Response Suggestion for Smart Reply, Section 4.4: https://arxiv.org/pdf/1705.00652.pdf - `Training Examples > Natural Language Inference <../../examples/training/nli/README.html>`_ - `Training Examples > Paraphrase Data <../../examples/training/paraphrases/README.html>`_ - `Training Examples > Quora Duplicate Questions <../../examples/training/quora_duplicate_questions/README.html>`_ - `Training Examples > MS MARCO <../../examples/training/ms_marco/README.html>`_ - `Unsupervised Learning > SimCSE <../../examples/unsupervised_learning/SimCSE/README.html>`_ - `Unsupervised Learning > GenQ <../../examples/unsupervised_learning/query_generation/README.html>`_ Requirements: 1. (anchor, positive) pairs or (anchor, positive, negative) triplets Inputs: +-------------------------------------------------+--------+ | Texts | Labels | +=================================================+========+ | (anchor, positive) pairs | none | +-------------------------------------------------+--------+ | (anchor, positive, negative) triplets | none | +-------------------------------------------------+--------+ | (anchor, positive, negative_1, ..., negative_n) | none | +-------------------------------------------------+--------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. Relations: - :class:`CachedMultipleNegativesRankingLoss` is equivalent to this loss, but it uses caching that allows for much higher batch sizes (and thus better performance) without extra memory usage. However, it is slightly slower. - :class:`MultipleNegativesSymmetricRankingLoss` is equivalent to this loss, but with an additional loss term. - :class:`GISTEmbedLoss` is equivalent to this loss, but uses a guide model to guide the in-batch negative sample selection. `GISTEmbedLoss` yields a stronger training signal at the cost of some training overhead. Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.MultipleNegativesRankingLoss(model) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() N)super__init__r r similarity_fctrCrossEntropyLosscross_entropy_loss)selfr r r __class__s u/var/www/html/ai-engine/env/lib/python3.11/site-packages/sentence_transformers/losses/MultipleNegativesRankingLoss.pyrz%MultipleNegativesRankingLoss.__init__sIb   ,"$"5"7"7sentence_featuresIterable[dict[str, Tensor]]labelsrc.fd|D}|d}tj|dd}||jz}tjd|d|j}||S)NcFg|]}|dS)sentence_embedding)r ).0sentence_featurers r z8MultipleNegativesRankingLoss.forward..fs-mmmGW +,,-ABmmmrr)device)torchcatrr arangesizer%r)rrrreps embeddings_a embeddings_bscores range_labelss` rforwardz$MultipleNegativesRankingLoss.forwardesmmmm[lmmmAw yabb** $$\<@@4:M|Av{{1~~fmLLL &&v|<<rDs""""""$$$$$$ &&&&&&IIIIIIppppp29pppppr