Ng0ddlmZddlmZddlZddlmZmZddlmZddl m Z GddZ Gd d ej Z dS) ) annotations)IterableN)Tensornn)util)SentenceTransformercDeZdZdZed dZed d dZdS) $BatchHardTripletLossDistanceFunctionz`This class defines distance functions, that can be used with Batch[All/Hard/SemiHard]TripletLoss embeddingsrreturnc2dtj||z S)zWCompute the 2D matrix of cosine distances (1-cosine_similarity) between all embeddings.)rpytorch_cos_sim)r s m/var/www/html/ai-engine/env/lib/python3.11/site-packages/sentence_transformers/losses/BatchHardTripletLoss.pycosine_distancez4BatchHardTripletLossDistanceFunction.cosine_distances4' J????Fctj||}tj|}|dd|zz |dz}d||dk<|sI|d}||dzz}d|z tj|z}|S)a Compute the 2D matrix of eucledian distances between all the embeddings. Args: embeddings: tensor of shape (batch_size, embed_dim) squared: Boolean. If true, output is the pairwise squared euclidean distance matrix. If false, output is the pairwise euclidean distance matrix. Returns: pairwise_distances: tensor of shape (batch_size, batch_size) rg@rgؗҜ<?)torchmatmultdiag unsqueezeeqfloatsqrt)r squared dot_product square_norm distancesmasks reucledian_distancez7BatchHardTripletLossDistanceFunction.eucledian_distancesl:z||~~>> j--  ))!,,s[/@@;CXCXYZC[C[[ $% )a-  =<<??((**D!D5L0Ituz)'<'< BatchHardTripletLoss takes a batch with (sentence, label) pairs and computes the loss for all possible, valid triplets, i.e., anchor and positive must have the same label, anchor and negative a different label. It then looks for the hardest positive and the hardest negatives. The labels must be integers, with same label indicating sentences from the same class. Your train dataset must contain at least 2 examples per label class. Args: model: SentenceTransformer model distance_metric: Function that returns a distance between two embeddings. The class SiameseDistanceMetric contains pre-defined metrics that can be used margin: Negative samples should be at least margin further apart from the anchor than the positive. Definitions: :Easy triplets: Triplets which have a loss of 0 because ``distance(anchor, positive) + margin < distance(anchor, negative)``. :Hard triplets: Triplets where the negative is closer to the anchor than the positive, i.e., ``distance(anchor, negative) < distance(anchor, positive)``. :Semi-hard triplets: Triplets where the negative is not closer to the anchor than the positive, but which still have a positive loss, i.e., ``distance(anchor, positive) < distance(anchor, negative) + margin``. References: * Source: https://github.com/NegatioN/OnlineMiningTripletLoss/blob/master/online_triplet_loss/losses.py * Paper: In Defense of the Triplet Loss for Person Re-Identification, https://arxiv.org/abs/1703.07737 * Blog post: https://omoindrot.github.io/triplet-loss Requirements: 1. Each sentence must be labeled with a class. 2. Your dataset must contain at least 2 examples per labels class. 3. Your dataset should contain hard positives and negatives. Inputs: +------------------+--------+ | Texts | Labels | +==================+========+ | single sentences | class | +------------------+--------+ Recommendations: - Use ``BatchSamplers.GROUP_BY_LABEL`` (:class:`docs `) to ensure that each batch contains 2+ examples per label class. Relations: * :class:`BatchAllTripletLoss` uses all possible, valid triplets, rather than only the hardest positive and negative samples. * :class:`BatchSemiHardTripletLoss` uses only semi-hard triplets, valid triplets, rather than only the hardest positive and negative samples. * :class:`BatchHardSoftMarginTripletLoss` does not require setting a margin, while this loss does. Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") # E.g. 0: sports, 1: economy, 2: politics train_dataset = Dataset.from_dict({ "sentence": [ "He played a great game.", "The stock is up 20%", "They won 2-1.", "The last goal was amazing.", "They all voted against the bill.", ], "label": [0, 1, 0, 0, 2], }) loss = losses.BatchHardTripletLoss(model) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() N)super__init__sentence_embeddertriplet_margindistance_metric)selfr,r4r- __class__s rr1zBatchHardTripletLoss.__init__;s:d !&$.rsentence_featuresIterable[dict[str, Tensor]]labelsrcp||dd}|||S)Nrsentence_embedding)r2batch_hard_triplet_loss)r5r7r9reps rforwardzBatchHardTripletLoss.forwards7$$%6q%9::;OP++FC888rr c||}t|}||z}|dd\}}t|}|dd\} }|| d|z zz} | dd\} }|| z |jz} d| | dk<| } | S)abBuild the triplet loss over a batch of embeddings. For each anchor, we get the hardest positive and hardest negative to form a triplet. Args: labels: labels of the batch, of size (batch_size,) embeddings: tensor of shape (batch_size, embed_dim) margin: margin for triplet loss squared: Boolean. If true, output is the pairwise squared euclidean distance matrix. If false, output is the pairwise euclidean distance matrix. Returns: Label_Sentence_Triplet: scalar tensor containing the triplet loss rT)keepdimrr) r4r* get_anchor_positive_triplet_maskrmax get_anchor_negative_triplet_maskminr3mean)r5r9r pairwise_distmask_anchor_positiveanchor_positive_disthardest_positive_dist_mask_anchor_negativemax_anchor_negative_distanchor_negative_disthardest_negative_disttl triplet_losss rr<z,BatchHardTripletLoss.batch_hard_triplet_losss,,Z88  4TTU[\\bbdd 4mC$8#;#;At#;#L#L q 4TTU[\\bbdd'4&7&74&7&H&H# !,/G3QeKe/ff$8#;#;At#;#L#L q#%: :T=P P26 wwyy rctj|d|j}|}|d}|d}|d}||z|z}|d|dk}|d}|d} | |z} | |zS)a1Return a 3D mask where mask[a, p, n] is True iff the triplet (a, p, n) is valid. A triplet (i, j, k) is valid if: - i, j, k are distinct - labels[i] == labels[j] and labels[i] != labels[k] Args: labels: tf.int32 `Tensor` with shape [batch_size] rdevicerreyesizerSboolr) r9 indices_equalindices_not_equal i_not_equal_j i_not_equal_k j_not_equal_kdistinct_indices label_equal i_equal_j i_equal_k valid_labelss rget_triplet_maskz%BatchHardTripletLoss.get_triplet_masks &++a..GGGLLNN *N)33A66 )33A66 )33A66 )M9]J&&q))V-=-=a-@-@@ ))!,, ))!,, !zI- ...rctj|d|j}|}|d|dk}||zS)aReturn a 2D mask where mask[a, p] is True iff a and p are distinct and have same label. Args: labels: tf.int32 `Tensor` with shape [batch_size] Returns: mask: tf.bool `Tensor` with shape [batch_size, batch_size] rrRrrU)r9rYrZ labels_equals rrAz5BatchHardTripletLoss.get_anchor_positive_triplet_masksl &++a..GGGLLNN *N''**f.>.>q.A.AA ///rc\|d|dkS)zReturn a 2D mask where mask[a, n] is True iff a and n have distinct labels. Args: labels: tf.int32 `Tensor` with shape [batch_size] Returns: mask: tf.bool `Tensor` with shape [batch_size, batch_size] rr)r)r9s rrCz5BatchHardTripletLoss.get_anchor_negative_triplet_masks.!!!$$(8(8(;(;;<r<r'rcrArCpropertyri __classcell__)r6s@rr*r*:s=O U/U/U/U/U/U/U/n9999))))V///\/2000\0$ = = =\ =   X     rr*) __future__rcollections.abcrrrrsentence_transformersr)sentence_transformers.SentenceTransformerrr Moduler*r(rrrqs""""""$$$$$$ &&&&&&IIIIII++++++++\PPPPP29PPPPPr