Ng+ddlmZddlZddlZddlZddlmZddlZddl Z ddl m Z m Z ddl m Z mZddlmZGdde jZGd d e jZGd d ZdS) ) annotationsN)Iterable)Tensornn) InputExampleutil)SentenceTransformercDeZdZdZdfd Zdd Zedd ZxZS)ContrastiveTensionLossa This loss expects only single sentences, without any labels. Positive and negative pairs are automatically created via random sampling, such that a positive pair consists of two identical sentences and a negative pair consists of two different sentences. An independent copy of the encoder model is created, which is used for encoding the first sentence of each pair. The original encoder model encodes the second sentence. The embeddings are compared and scored using the generated labels (1 if positive, 0 if negative) using the binary cross entropy objective. Note that you must use the `ContrastiveTensionDataLoader` for this loss. The `pos_neg_ratio` of the ContrastiveTensionDataLoader can be used to determine the number of negative pairs per positive pair. Generally, :class:`ContrastiveTensionLossInBatchNegatives` is recommended over this loss, as it gives a stronger training signal. Args: model: SentenceTransformer model References: * Semantic Re-Tuning with Contrastive Tension: https://openreview.net/pdf?id=Ov_sMNau-PF * `Unsupervised Learning > CT <../../examples/unsupervised_learning/CT/README.html>`_ Inputs: +------------------+--------+ | Texts | Labels | +==================+========+ | single sentences | none | +------------------+--------+ Relations: * :class:`ContrastiveTensionLossInBatchNegatives` uses in-batch negative sampling, which gives a stronger training signal than this loss. Example: :: from sentence_transformers import SentenceTransformer, losses from sentence_transformers.losses import ContrastiveTensionDataLoader model = SentenceTransformer('all-MiniLM-L6-v2') train_examples = [ 'This is the 1st sentence', 'This is the 2nd sentence', 'This is the 3rd sentence', 'This is the 4th sentence', 'This is the 5th sentence', 'This is the 6th sentence', 'This is the 7th sentence', 'This is the 8th sentence', 'This is the 9th sentence', 'This is the final sentence', ] train_dataloader = ContrastiveTensionDataLoader(train_examples, batch_size=3, pos_neg_ratio=3) train_loss = losses.ContrastiveTensionLoss(model=model) model.fit( [(train_dataloader, train_loss)], epochs=10, ) modelr returnNonect||_tj||_t jd|_dS)Nsum) reduction) super__init__model2copydeepcopymodel1rBCEWithLogitsLoss criterion)selfr __class__s o/var/www/html/ai-engine/env/lib/python3.11/site-packages/sentence_transformers/losses/ContrastiveTensionLoss.pyrzContrastiveTensionLoss.__init__KsJ  mE** ->>>sentence_featuresIterable[dict[str, Tensor]]labelsrct|\}}||d}||d}tj|dddf|dddddfdd}||||}|S)Nsentence_embedding)tuplerrtorchmatmulsqueezertype_as) rrr sentence_features1sentence_features2reps_1reps_2 sim_scoreslosss rforwardzContrastiveTensionLoss.forwardQs167H1I1I../001EF/001EF L4&AAAt*< = = E Eb I I Q QRT U U ~~j&..*D*DEE rstrcdSNar @inproceedings{carlsson2021semantic, title={Semantic Re-tuning with Contrastive Tension}, author={Fredrik Carlsson and Amaru Cuba Gyllensten and Evangelia Gogoulou and Erik Ylip{"a}{"a} Hellqvist and Magnus Sahlgren}, booktitle={International Conference on Learning Representations}, year={2021}, url={https://openreview.net/forum?id=Ov_sMNau-PF} } rs rcitationzContrastiveTensionLoss.citation] r)r r r rrrr rr rr r0) __name__ __module__ __qualname____doc__rr/propertyr5 __classcell__rs@rr r s|88t??????       X     rr cPeZdZdejfdfd Zdd ZeddZxZ S)&ContrastiveTensionLossInBatchNegativesg4@r r scalefloatr rcFt||_tj||_||_tj|_ tj tj gtj|z|_dS)a\ This loss expects only single sentences, without any labels. Positive and negative pairs are automatically created via random sampling, such that a positive pair consists of two identical sentences and a negative pair consists of two different sentences. An independent copy of the encoder model is created, which is used for encoding the first sentence of each pair. The original encoder model encodes the second sentence. Unlike :class:`ContrastiveTensionLoss`, this loss uses the batch negative sampling strategy, i.e. the negative pairs are sampled from the batch. Using in-batch negative sampling gives a stronger training signal than the original :class:`ContrastiveTensionLoss`. The performance usually increases with increasing batch sizes. Note that you should not use the `ContrastiveTensionDataLoader` for this loss, but just a normal DataLoader with `InputExample` instances. The two texts of each `InputExample` instance should be identical. Args: model: SentenceTransformer model scale: Output of similarity function is multiplied by scale value similarity_fct: similarity function between sentence embeddings. By default, cos_sim. Can also be set to dot product (and then set scale to 1) References: - Semantic Re-Tuning with Contrastive Tension: https://openreview.net/pdf?id=Ov_sMNau-PF - `Unsupervised Learning > CT (In-Batch Negatives) <../../examples/unsupervised_learning/CT_In-Batch_Negatives/README.html>`_ Relations: * :class:`ContrastiveTensionLoss` does not select negative pairs in-batch, resulting in a weaker training signal than this loss. Inputs: +------------------------+--------+ | Texts | Labels | +========================+========+ | (anchor, anchor) pairs | none | +------------------------+--------+ Example: :: from sentence_transformers import SentenceTransformer, losses from torch.utils.data import DataLoader model = SentenceTransformer('all-MiniLM-L6-v2') train_examples = [ InputExample(texts=['This is a positive pair', 'Where the distance will be minimized'], label=1), InputExample(texts=['This is a negative pair', 'Their distance will be increased'], label=0), ] train_examples = [ InputExample(texts=['This is the 1st sentence', 'This is the 1st sentence']), InputExample(texts=['This is the 2nd sentence', 'This is the 2nd sentence']), InputExample(texts=['This is the 3rd sentence', 'This is the 3rd sentence']), InputExample(texts=['This is the 4th sentence', 'This is the 4th sentence']), InputExample(texts=['This is the 5th sentence', 'This is the 5th sentence']), ] train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=32) train_loss = losses.ContrastiveTensionLossInBatchNegatives(model=model) model.fit( [(train_dataloader, train_loss)], epochs=10, ) N)rrrrrrsimilarity_fctrCrossEntropyLosscross_entropy_loss Parameterr%onesnplog logit_scale)rr rBrErs rrz/ContrastiveTensionLossInBatchNegatives.__init__kszz  mE** ,"$"5"7"7< 2(FGGrrrr rct|\}}||d}||d}||||jz}t jtt|t j |j }| ||| | |zdz S)Nr")dtypedevice)r$rrrErLexpr%tensorrangelenlongrOrGt)rrr r)r* embeddings_a embeddings_bscoress rr/z.ContrastiveTensionLossInBatchNegatives.forwards167H1I1I..{{#5667KL {{#5667KL $$\<@@4CSCWCWCYCYYeCKK00 6=YYY''77$:Q:QRXRZRZR\R\^d:e:eeijjjrr0cdSr2r3r4s rr5z/ContrastiveTensionLossInBatchNegatives.citationr6r)r r rBrCr rr7r8) r9r:r;rcos_simrr/r=r5r>r?s@rrArAjsBFW[WcBHBHBHBHBHBHBHHkkkk   X     rrAc"eZdZddZdZdZdS)ContrastiveTensionDataLoaderc||_||_||_d|_|j|jzdkrt d|d|ddS)Nrz@ContrastiveTensionDataLoader was loaded with a pos_neg_ratio of z and a batch size of z7. The batch size must be divisible by the pos_neg_ratio) sentences batch_size pos_neg_ratio collate_fn ValueError)rr`rarbs rrz%ContrastiveTensionDataLoader.__init__s|"$* ?T/ /1 4 4{S`{{xB{{{  5 4rc#Ktj|jd}g}|dzt|jkr|j|}t||jzdkr|dz }|j|}d}n|j|}d}|dz }|t ||g|t||jkr"|j||n|Vg}|dzt|jkdSdS)Nr)textslabel) randomshuffler`rTrbappendrrarc)r sentence_idxbatchs1s2rhs r__iter__z%ContrastiveTensionDataLoader.__iter__s!t~&&& QT^!4!444 -B5zzD..22! ^L1^L1 A L LLRHEBBB C C C5zzT_,,040Kdooe,,,QVVVVQT^!4!4444444rcdtjt|jd|jzz S)NrP)mathfloorrTr`rar4s r__len__z$ContrastiveTensionDataLoader.__len__s(z#dn--T_1DEFFFrN)r^)r9r:r;rrprtr3rrr]r]sK    ,GGGGGrr]) __future__rrrrricollections.abcrnumpyrJr%rrsentence_transformersrr)sentence_transformers.SentenceTransformerr Moduler rAr]r3rrr{sA"""""" $$$$$$ 44444444IIIIIIWWWWWRYWWWtXXXXXRYXXXB#G#G#G#G#G#G#G#G#G#Gr