Ng:ddlmZddlmZmZddlmZddlmZddl m Z ddl Z ddl Z ddl m Z mZddlmZmZdd lmZmZdd lmZGd d ZddZGddejZdS)) annotations)IterableIterator) nullcontext)partial)AnyN)Tensornn)get_device_statesset_device_states)SentenceTransformerutil)StaticEmbeddingc*eZdZdZddZddZddZdS) RandContexta Random-state context manager class. Reference: https://github.com/luyug/GradCache. This class will back up the pytorch's random state during initialization. Then when the context is activated, the class will set up the random state with the backed-up one. returnNonecbtj|_t|\|_|_dSN)torch get_rng_state fwd_cpu_stater fwd_gpu_devicesfwd_gpu_states)selftensorss {/var/www/html/ai-engine/env/lib/python3.11/site-packages/sentence_transformers/losses/CachedMultipleNegativesRankingLoss.py__init__zRandContext.__init__s."0224Ew4O1d111ctj|jd|_|jtj|jt|j|j dS)NT)devicesenabled) rrandomfork_rngr_fork __enter__ set_rng_staterr rrs rr&zRandContext.__enter__sf\**43GQU*VV   D.///$.0CDDDDDrcL|j|||d|_dSr)r%__exit__)rexc_typeexc_valexc_tbs rr*zRandContext.__exit__#s' Hgv666 rN)rr)__name__ __module__ __qualname____doc__rr&r*rrrrsbPPPPEEEE rr grad_outputr sentence_featuresIterable[dict[str, Tensor]]loss_obj"CachedMultipleNegativesRankingLossrrc |jJ|jJtj5t ||j|jD]\}}}t ||dd||D]X\\}}}tj|||z} | Y ddddS#1swxYwYdS)zOA backward hook to backpropagate the cached gradients mini-batch by mini-batch.NTF)sentence_feature with_gradcopy_random_state random_states) cacher<r enable_gradzipembed_minibatch_iterdotflattenbackward) r3r4r6r9gradr<reps_mb_grad_mb surrogates r_backward_hookrI(sT > % % %  ! - - -     % %589JHN\d\r5s5s % % 1 dM),--%5"&+"/ . ** % %% !g"Igoo&7&79J9JKKkY ""$$$$ % % % % % % % % % % % % % % % % % % % %sB#CCCceZdZdejddfd-fd Z d.d/dZ d.d0dZd1d#Zd1d$Z d2d(Z d3d*Z e d4d,Z xZS)5r7g4@ Fmodelr scalefloatsimilarity_fct"callable[[Tensor, Tensor], Tensor]mini_batch_sizeintshow_progress_barboolrrc.tt|dtrt d||_||_||_tj |_ ||_ d|_ d|_ ||_dS)a\ Boosted version of MultipleNegativesRankingLoss (https://arxiv.org/pdf/1705.00652.pdf) by GradCache (https://arxiv.org/pdf/2101.06983.pdf). Constrastive learning (here our MNRL loss) with in-batch negatives is usually hard to work with large batch sizes due to (GPU) memory limitation. Even with batch-scaling methods like gradient-scaling, it cannot work either. This is because the in-batch negatives make the data points within the same batch non-independent and thus the batch cannot be broke down into mini-batches. GradCache is a smart way to solve this problem. It achieves the goal by dividing the computation into two stages of embedding and loss calculation, which both can be scaled by mini-batches. As a result, memory of constant size (e.g. that works with batch size = 32) can now process much larger batches (e.g. 65536). In detail: (1) It first does a quick embedding step without gradients/computation graphs to get all the embeddings; (2) Calculate the loss, backward up to the embeddings and cache the gradients wrt. to the embeddings; (3) A 2nd embedding step with gradients/computation graphs and connect the cached gradients into the backward chain. Notes: All steps are done with mini-batches. In the original implementation of GradCache, (2) is not done in mini-batches and requires a lot memory when batch size large. One drawback is about the speed. GradCache will sacrifice around 20% computation time according to the paper. Args: model: SentenceTransformer model scale: Output of similarity function is multiplied by scale value similarity_fct: similarity function between sentence embeddings. By default, cos_sim. Can also be set to dot product (and then set scale to 1) mini_batch_size: Mini-batch size for the forward pass, this denotes how much memory is actually used during training and evaluation. The larger the mini-batch size, the more memory efficient the training is, but the slower the training will be. It's recommended to set it as high as your GPU memory allows. The default value is 32. show_progress_bar: If True, a progress bar for the mini-batches is shown during training. The default is False. References: - Efficient Natural Language Response Suggestion for Smart Reply, Section 4.4: https://arxiv.org/pdf/1705.00652.pdf - Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup: https://arxiv.org/pdf/2101.06983.pdf Requirements: 1. (anchor, positive) pairs or (anchor, positive, negative pairs) 2. Should be used with large batch sizes for superior performance, but has slower training time than :class:`MultipleNegativesRankingLoss` Inputs: +-------------------------------------------------+--------+ | Texts | Labels | +=================================================+========+ | (anchor, positive) pairs | none | +-------------------------------------------------+--------+ | (anchor, positive, negative) triplets | none | +-------------------------------------------------+--------+ | (anchor, positive, negative_1, ..., negative_n) | none | +-------------------------------------------------+--------+ Recommendations: - Use ``BatchSamplers.NO_DUPLICATES`` (:class:`docs `) to ensure that no in-batch negatives are duplicates of the anchor or positive samples. Relations: - Equivalent to :class:`MultipleNegativesRankingLoss`, but with caching that allows for much higher batch sizes (and thus better performance) without extra memory usage. This loss also trains roughly 2x to 2.4x slower than :class:`MultipleNegativesRankingLoss`. Example: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, losses from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = losses.CachedGISTEmbedLoss(model, mini_batch_size=64) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() rzCachedMultipleNegativesRankingLoss is not compatible with a SentenceTransformer model based on a StaticEmbedding. Consider using MultipleNegativesRankingLoss instead.N)superr isinstancer ValueErrorrLrMrOr CrossEntropyLosscross_entropy_lossrQr=r<rS)rrLrMrOrQrS __class__s rrz+CachedMultipleNegativesRankingLoss.__init__@sj  eAh 0 0 G    ,"$"5"7"7.04 =A!2rNr9dict[str, Tensor]beginendr:r; random_stateRandContext | None!tuple[Tensor, RandContext | None]c|rtn tj}|tn|}fd|D} |5|5|rt | nd}|| d} dddn #1swxYwYdddn #1swxYwY| |fS)zYDo forward pass on a minibatch of the input features and return corresponding embeddings.Nc,i|]\}}||Sr2r2).0kvr]r^s r zFCachedMultipleNegativesRankingLoss.embed_minibatch..s'%[%[%[$!Qa59%[%[%[rsentence_embedding)rrno_graditemsrvaluesrL) rr9r]r^r:r;r_ grad_contextrandom_state_contextsentence_feature_minibatchrepss `` rembed_minibatchz2CachedMultipleNegativesRankingLoss.embed_minibatchsx'0B{{U] 0<0D{}}},%[%[%[%[%[BRBXBXBZBZ%[%[%[" ! T T T TTeo{,F,M,M,O,OPPko zz"<==>RS T T T T T T T T T T T T T T T T T T T T T T T T T T T T T T\!!s6 B6;B B6B# #B6&B# 'B66B:=B:r<list[RandContext] | None+Iterator[tuple[Tensor, RandContext | None]]c # K|d}|j\}}ttjd||jd|j D]=\}} | |jz} ||| | |||dn||\} } | | fV>dS)z`Do forward pass on all the minibatches of the input features and yield corresponding embeddings. input_idsrzEmbed mini-batchesdescdisableN)r9r]r^r:r;r_)shape enumeratetqdmtrangerQrSrp) rr9r:r;r<rtbszrFiberor_s rr@z7CachedMultipleNegativesRankingLoss.embed_minibatch_iters-[9 Q K$) 22       % %DAqD((A!%!5!5!1#"3%2%:TT a@P "6"" D, $ $ $ $ $% % %rrolist[list[Tensor]]r ctj|d}tjd|ddD}t|}tjt |tj|j}g}tjd||j d|j D]}||j z}| |||||j z} | | |||t| z|z } | || t#|} d|D|_| S) zMCalculate the cross-entropy loss and cache the gradients wrt. the embeddings.rc6g|]}tj|Sr2rcatrdrs r zYCachedMultipleNegativesRankingLoss.calculate_loss_and_cache_gradients.. !A!A!A1%)A,,!A!A!ArNdtypedevicePreparing cachesruc&g|]}d|DS)cg|] }|j Sr2)rDrs rrzdCachedMultipleNegativesRankingLoss.calculate_loss_and_cache_gradients...s***!qv***rr2)rdrss rrzYCachedMultipleNegativesRankingLoss.calculate_loss_and_cache_gradients..s':::r**r***:::r)rrlentensorrangelongrrzr{rQrSrOrMrZrCappenddetachsumrequires_grad_r= rro embeddings_a embeddings_b batch_sizelabelslossesr~rscores loss_mbatchlosss r"calculate_loss_and_cache_gradientszECachedMultipleNegativesRankingLoss.calculate_loss_and_cache_gradientss~ya)) y!A!AQRR!A!A!ABB &&  *  UZ 8K   &(   #..    0 0AD((A!00ac1BLQQTXT^^F(,(?(?qQRs (T(TWZ[aWbWb(beo(oK  " " " MM+,,.. / / / /6{{))++::T:::  rcftj|d}tjd|ddD}t|}tjt |tj|j}g}tjd||j d|j D]x}||j z}| |||||j z} | | |||t| z|z } || yt|} | S)zACalculate the cross-entropy loss. No need to cache the gradients.rc6g|]}tj|Sr2rrs rrzECachedMultipleNegativesRankingLoss.calculate_loss..rrrNrrru)rrrrrrrrzr{rQrSrOrMrZrrrs rcalculate_lossz1CachedMultipleNegativesRankingLoss.calculate_losss?ya)) y!A!AQRR!A!A!ABB &&  *  UZ 8K   &(   #..    ' 'AD((A!00ac1BLQQTXT^^F(,(?(?qQRs (T(TWZ[aWbWb(beo(oK MM+ & & & &6{{ rr4r5rc0g}g|_|D]}g}g}||ddD]S\}}||||T|||j|t jr@||} | tt||n| |} | S)NFT)r9r:r;)r4r6) r<r@rrrris_grad_enabledr register_hookrrIr) rr4rror9reps_mbsrandom_state_mbsrEr_rs rforwardz*CachedMultipleNegativesRankingLoss.forward s3 1 8 8 H! )-)B)B!1"&*C** 6 6%  0 0 ? ? A ABBB '' 5555 KK ! ! !   % %&6 7 7 7 7  " " -::4@@D   w~IZeijjj k k k k&&t,,D rdict[str, Any]c*|j|jjdS)N)rMrO)rMrOr.r(s rget_config_dictz2CachedMultipleNegativesRankingLoss.get_config_dict)st7J7STTTrstrcdS)Na @misc{gao2021scaling, title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup}, author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan}, year={2021}, eprint={2101.06983}, archivePrefix={arXiv}, primaryClass={cs.LG} } r2r(s rcitationz+CachedMultipleNegativesRankingLoss.citation,s   r) rLr rMrNrOrPrQrRrSrTrrr)r9r\r]rRr^rRr:rTr;rTr_r`rra) r9r\r:rTr;rTr<rqrrr)rorrr )r4r5rr rr )rr)rr)r.r/r0rcos_simrrpr@rrrrpropertyr __classcell__)r[s@rr7r7?s =A\!"' c3c3c3c3c3c3c3X,0"""""037 %%%%%<:2:UUUU   X     r)r3r r4r5r6r7rr) __future__rcollections.abcrr contextlibr functoolsrtypingrrrzr r torch.utils.checkpointr r sentence_transformersr rsentence_transformers.modelsrrrIModuler7r2rrrsM""""""........"""""" GGGGGGGG;;;;;;;;888888.%%%%.xxxxxxxxxxr