Ng!t.ddlmZddlZddlZddlmZddlmZmZm Z ddl Z ddl Z ddl m Z mZddlmZddlmZddlmZmZdd lmZmZmZmZdd lmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(ej)e*Z+GddeZ,dS)) annotationsN)wraps)CallableLiteraloverload)Tensornn) Optimizer) DataLoader)tqdmtrange) AutoConfig"AutoModelForSequenceClassification AutoTokenizeris_torch_npu_available) BatchEncoding)PushToHubMixin)SentenceEvaluator) InputExample)SentenceTransformer)fullnameget_device_nameimport_from_stringceZdZdZ djdkdZdldZdmdZdddejd d!e j j d"d#id$d%dd&dddd&fdnd;Z e dodpdIZe dqdrdMZe dodsdNZe dodtdQZ dudvdTZ dwdxd\Zdyd]Zd&d^dzdaZd&d^dzdbZeejddd&ddcd{fdiZxZS)| CrossEncodera2 A CrossEncoder takes exactly two sentences / texts as input and either predicts a score or label for this sentence pair. It can for example predict the similarity of the sentence pair on a scale of 0 ... 1. It does not yield a sentence embedding and does not work for individual sentences. Args: model_name (str): A model name from Hugging Face Hub that can be loaded with AutoModel, or a path to a local model. We provide several pre-trained CrossEncoder models that can be used for common tasks. num_labels (int, optional): Number of labels of the classifier. If 1, the CrossEncoder is a regression model that outputs a continuous score 0...1. If > 1, it output several scores that can be soft-maxed to get probability scores for the different classes. Defaults to None. max_length (int, optional): Max length for input sequences. Longer sequences will be truncated. If None, max length of the model will be used. Defaults to None. device (str, optional): Device that should be used for the model. If None, it will use CUDA if available. Defaults to None. automodel_args (Dict, optional): Arguments passed to AutoModelForSequenceClassification. Defaults to None. tokenizer_args (Dict, optional): Arguments passed to AutoTokenizer. Defaults to None. config_args (Dict, optional): Arguments passed to AutoConfig. Defaults to None. cache_dir (`str`, `Path`, optional): Path to the folder where cached files are stored. trust_remote_code (bool, optional): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to True for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. Defaults to False. revision (Optional[str], optional): The specific model version to use. It can be a branch name, a tag name, or a commit id, for a stored model on Hugging Face. Defaults to None. local_files_only (bool, optional): If `True`, avoid downloading the model. Defaults to False. default_activation_function (Callable, optional): Callable (like nn.Sigmoid) about the default activation function that should be used on-top of model.predict(). If None. nn.Sigmoid() will be used if num_labels=1, else nn.Identity(). Defaults to None. classifier_dropout (float, optional): The dropout ratio for the classification head. Defaults to None. NF model_namestr num_labelsint max_lengthdevice str | Noneautomodel_argsdicttokenizer_args config_args cache_dirtrust_remote_codeboolrevisionlocal_files_onlyclassifier_dropoutfloatreturnNonec |i}|i}|i}tj|f| | | |d||_d}|jj#t d|jjD}| | |j_||sd}| ||j_tj|f|j| | | |d||_tj|f| | | |d||_ ||_ |+t}td|tj||_| j| |_ t'|j|j_dS#t*$r5}tdt/|Yd}~dSd}~wwxYwt1|jd r4|jj(t3|jj|_dS|jjdkrt5jnt5j|_dS) N)r(r*r+r'Tc8g|]}|dS)ForSequenceClassification)endswith).0archs l/var/www/html/ai-engine/env/lib/python3.11/site-packages/sentence_transformers/cross_encoder/CrossEncoder.py z)CrossEncoder.__init__..]s%bbb:;;bbb)configr*r(r+r')r*r+r(r'zUse pytorch device: zEWas not able to update config about the default_activation_function: $sbert_ce_default_activation_function)rfrom_pretrainedr: architecturesanyr,rrmodelr tokenizerr rloggerinfotorchr!_target_devicedefault_activation_functionrr; Exceptionwarningrhasattrrr SigmoidIdentity)selfrrr r!r#r%r&r'r(r*r+rEr,classifier_trainedes r6__init__zCrossEncoder.__init__<s  !N  !N  K 0  /-       " ; $ 0!$bb Habbb""   )-?DK *  &8 J  !%/DK "7G  ;/-      '6  -/      % >$&&F KK7v77 8 8 8#l622 & 2/JD , qCKDLlCmCm @@@ q q qogjklgmgmooppppppppp q DK!G H H n @L/s/A$+Br/s/s/u/uD , , ,?C{?UYZ?Z?Zrz|||`b`k`m`mD , , ,sD11 E0;*E++E0batchlist[InputExample]tuple[BatchEncoding, Tensor]c^dtt|djD}g}|D]c}t|jD]2\}}|||3||jd|j|ddd|jd}tj ||j j dkr tj n tj|j}|D]%}|||j||<&||fS) Ncg|]}gSrTr4_s r6r7z7CrossEncoder.smart_batching_collate..s888888r8rT longest_firstptpadding truncationreturn_tensorsr r9)dtype)rangelentexts enumerateappendstriplabelr@r rCtensorr:rr-longtorD) rKrOr`labelsexampleidxtext tokenizednames r6smart_batching_collatez#CrossEncoder.smart_batching_collatesA88U3uQx~#6#677888 ) )G&w}55 0 0 Tc !!$**,,//// MM'- ( ( ( ("DN D_T^b^m   f4;;QUV;V;VEKK\a\fgggjj     F FD'o001DEEIdOO&  r8rcfdtt|dD}|D]D}t|D]2\}}|||3E|j|ddd|jd}|D]%}|||j||<&|S)Ncg|]}gSrTrTrUs r6r7zACrossEncoder.smart_batching_collate_text_only..s222222r8rTrWrXrY) r^r_rarbrcr@r rgrD)rKrOr`rirjrkrlrms r6 smart_batching_collate_text_onlyz-CrossEncoder.smart_batching_collate_text_onlys22U3uQx==11222 0 0G&w// 0 0 Tc !!$**,,//// 0#DN D_T^b^m    F FD'o001DEEIdOOr8r9 WarmupLineari'lrgh㈵>g{Gz?rTtrain_dataloaderr evaluatorrepochs scheduler warmup_stepsoptimizer_classtype[Optimizer]optimizer_paramsdict[str, object] weight_decayevaluation_steps output_pathsave_best_model max_grad_normuse_ampcallback!Callable[[float, int, int], None]show_progress_barc @ |j|_|rUtr$tjj}n#tjj}|j |j | tj | dd|_ tt||z}t!|j}gd fd|D| d fd|Dd dg}||fi| }t%|t&rt)j|||| }|6|jjd krt1jnt1j}d }t7|d | D] }d}|j|jt=|dd| D]\}}|rWtj|j j 5|jdi|ddi}||j!}|jjd kr|"d}|||}dddn #1swxYwY|#}|$|%|&|tjj'(|j)||*||+|#|k}n|jdi|ddi}||j!}|jjd kr|"d}|||}|%tjj'(|j)||*||s|*|d z }|[| dkrU|| zdkrL|,|| | ||||j|j||,|| | |d| dS)a Train the model with the given training objective Each training objective is sampled in turn for one batch. We sample only as many batches from each objective as there are in the smallest one to make sure of equal training with each dataset. Args: train_dataloader (DataLoader): DataLoader with training InputExamples evaluator (SentenceEvaluator, optional): An evaluator (sentence_transformers.evaluation) evaluates the model performance during training on held-out dev data. It is used to determine the best model that is saved to disc. Defaults to None. epochs (int, optional): Number of epochs for training. Defaults to 1. loss_fct: Which loss function to use for training. If None, will use nn.BCEWithLogitsLoss() if self.config.num_labels == 1 else nn.CrossEntropyLoss(). Defaults to None. activation_fct: Activation function applied on top of logits output of model. scheduler (str, optional): Learning rate scheduler. Available schedulers: constantlr, warmupconstant, warmuplinear, warmupcosine, warmupcosinewithhardrestarts. Defaults to "WarmupLinear". warmup_steps (int, optional): Behavior depends on the scheduler. For WarmupLinear (default), the learning rate is increased from o up to the maximal learning rate. After these many training steps, the learning rate is decreased linearly back to zero. Defaults to 10000. optimizer_class (Type[Optimizer], optional): Optimizer. Defaults to torch.optim.AdamW. optimizer_params (Dict[str, object], optional): Optimizer parameters. Defaults to {"lr": 2e-5}. weight_decay (float, optional): Weight decay for model parameters. Defaults to 0.01. evaluation_steps (int, optional): If > 0, evaluate the model using evaluator after each number of training steps. Defaults to 0. output_path (str, optional): Storage path for the model and evaluation files. Defaults to None. save_best_model (bool, optional): If true, the best model (according to evaluator) is stored at output_path. Defaults to True. max_grad_norm (float, optional): Used for gradient normalization. Defaults to 1. use_amp (bool, optional): Use Automatic Mixed Precision (AMP). Only for Pytorch >= 1.6.0. Defaults to False. callback (Callable[[float, int, int], None], optional): Callback function that is invoked after each evaluation. It must accept the following three parameters in this order: `score`, `epoch`, `steps`. Defaults to None. show_progress_bar (bool, optional): If True, output a tqdm progress bar. Defaults to True. NT)exist_okiig)biaszLayerNorm.biaszLayerNorm.weightcRg|]"\}tfdD |#S)c3 K|]}|vV dSNrTr4ndns r6 z.CrossEncoder.fit...s(C_C_PRB!GC_C_C_C_C_C_r8r>r4prno_decays @r6r7z$CrossEncoder.fit..sA```AC_C_C_C_V^C_C_C_@_@_`1```r8)paramsr}cRg|]"\}tfdD |#S)c3 K|]}|vV dSrrTrs r6rz.CrossEncoder.fit...s'.s@YYYdaO:OPPP2 i2 iEN J " " " J     $( {dPaLa%%%* '* ' &%D4G4LMMM>>,6DJ,T,T,T,Tt,T,T,T)!/0A0H!I!I;1Q66%+[[__F%-Xff%=%= >>>>>>>>>>>>>>>)/(8(8(:(:%LL,,55777OOI...HN224:3H3H3J3JMZZZKK ***MMOOO%+%5%5%7%7;L%LNN(2 (P(PX(P(P4(P(P(P%+^,=,DEEF{-22!'R!)&&!9!9J'')))HN224:3H3H3J3JMZZZNN$$$##%%%%%NN$$$!#(-=-A-AnWgFgklFlFl..!;X`J((***J$$&&&$**9k?TY[]_ghhhe2 i2 isAI##I' *I' . sentencestuple[str, str] | list[str] batch_size bool | None num_workersrCallable | None apply_softmaxconvert_to_numpyLiteral[False]convert_to_tensor torch.Tensorc dSrrT rKrrrrrrrrs r6predictzCrossEncoder.predict> sr8Elist[tuple[str, str]] | list[list[str]] | tuple[str, str] | list[str] Literal[True] np.ndarrayc dSrrTrs r6rzCrossEncoder.predictKs Sr8c dSrrTrs r6rzCrossEncoder.predictXrr8'list[tuple[str, str]] | list[list[str]]list[torch.Tensor]c dSrrTrs r6rzCrossEncoder.predictes !Sr8 .list[torch.Tensor] | np.ndarray | torch.Tensorc d} t|dtr|g}d} t|||j|d} |Ntt jkp&tt jk}| } |rt| d} ||j }g} |j |j |jtj5| D]y} |j d i| ddi}||j}|r?t%|dd kr&tjj|d }| |z dddn #1swxYwY|jjd kr d | D} |rtj| } n |rt5jd | D} | r| d} | S)a Performs predictions with the CrossEncoder on the given sentence pairs. Args: sentences (Union[List[Tuple[str, str]], Tuple[str, str]]): A list of sentence pairs [(Sent1, Sent2), (Sent3, Sent4)] or one sentence pair (Sent1, Sent2). batch_size (int, optional): Batch size for encoding. Defaults to 32. show_progress_bar (bool, optional): Output progress bar. Defaults to None. num_workers (int, optional): Number of workers for tokenization. Defaults to 0. activation_fct (callable, optional): Activation function applied on the logits output of the CrossEncoder. If None, nn.Sigmoid() will be used if num_labels=1, else nn.Identity. Defaults to None. convert_to_numpy (bool, optional): Convert the output to a numpy matrix. Defaults to True. apply_softmax (bool, optional): If there are more than 2 dimensions and apply_softmax=True, applies softmax on the logits output. Defaults to False. convert_to_tensor (bool, optional): Convert the output to a tensor. Defaults to False. Returns: Union[List[float], np.ndarray, torch.Tensor]: Predictions for the passed sentence pairs. The return type depends on the `convert_to_numpy` and `convert_to_tensor` parameters. If `convert_to_tensor` is True, the output will be a torch.Tensor. If `convert_to_numpy` is True, the output will be a numpy.ndarray. Otherwise, the output will be a list of float values. Examples: :: from sentence_transformers import CrossEncoder model = CrossEncoder("cross-encoder/stsb-roberta-base") sentences = [["I love cats", "Cats are amazing"], ["I prefer dogs", "Dogs are loyal"]] model.predict(sentences) # => array([0.6912767, 0.4303499], dtype=float32) FrT)rrrshuffleNBatches)rrr9)dimcg|] }|d S)rrTr4scores r6r7z(CrossEncoder.predict..s===58===r8cg|]L}|MSrT)cpudetachr-numpyrs r6r7z(CrossEncoder.predict..sD%d%d%dueiikk&8&8&:&:&@&@&B&B&H&H&J&J%d%d%dr8rT)rrr rqrAgetEffectiveLevelloggingINFODEBUGr rEr?evalrgrDrCno_gradrr_r functionalsoftmaxextendr:rstacknpasarray)rKrrrrrrrrinput_was_stringinp_dataloaderiterator pred_scoresrrrs r6rzCrossEncoder.predictrshX! ilC ( ( $" I# # !<#      $((**gl:if>V>V>X>X\c\i>i "  <N;;;H  !!=N   d)*** ]__ + +$ + +$.DJ$L$L$L$Lt$L$L$L!'(9(@AA HS^^a%7%7"X088Q8GGF""6****  + + + + + + + + + + + + + + + + ; !Q & &=====K  f+k22KK  f*%d%dXc%d%d%deeK  )%a.Ks7A=FFFquery documents list[str]top_k int | Nonereturn_documentsDlist[dict[Literal['corpus_id', 'score', 'text'], int | float | str]]c <fd|D} || ||||| | | } g}t| D]B\}}|||d|r#|dd||iCt |dd}|d |S) ab Performs ranking with the CrossEncoder on the given query and documents. Returns a sorted list with the document indices and scores. Args: query (str): A single query. documents (List[str]): A list of documents. top_k (Optional[int], optional): Return the top-k documents. If None, all documents are returned. Defaults to None. return_documents (bool, optional): If True, also returns the documents. If False, only returns the indices and scores. Defaults to False. batch_size (int, optional): Batch size for encoding. Defaults to 32. show_progress_bar (bool, optional): Output progress bar. Defaults to None. num_workers (int, optional): Number of workers for tokenization. Defaults to 0. activation_fct ([type], optional): Activation function applied on the logits output of the CrossEncoder. If None, nn.Sigmoid() will be used if num_labels=1, else nn.Identity. Defaults to None. convert_to_numpy (bool, optional): Convert the output to a numpy matrix. Defaults to True. apply_softmax (bool, optional): If there are more than 2 dimensions and apply_softmax=True, applies softmax on the logits output. Defaults to False. convert_to_tensor (bool, optional): Convert the output to a tensor. Defaults to False. Returns: List[Dict[Literal["corpus_id", "score", "text"], Union[int, float, str]]]: A sorted list with the "corpus_id", "score", and optionally "text" of the documents. Example: :: from sentence_transformers import CrossEncoder model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2") query = "Who wrote 'To Kill a Mockingbird'?" documents = [ "'To Kill a Mockingbird' is a novel by Harper Lee published in 1960. It was immediately successful, winning the Pulitzer Prize, and has become a classic of modern American literature.", "The novel 'Moby-Dick' was written by Herman Melville and first published in 1851. It is considered a masterpiece of American literature and deals with complex themes of obsession, revenge, and the conflict between good and evil.", "Harper Lee, an American novelist widely known for her novel 'To Kill a Mockingbird', was born in 1926 in Monroeville, Alabama. She received the Pulitzer Prize for Fiction in 1961.", "Jane Austen was an English novelist known primarily for her six major novels, which interpret, critique and comment upon the British landed gentry at the end of the 18th century.", "The 'Harry Potter' series, which consists of seven fantasy novels written by British author J.K. Rowling, is among the most popular and critically acclaimed books of the modern era.", "'The Great Gatsby', a novel written by American author F. Scott Fitzgerald, was published in 1925. The story is set in the Jazz Age and follows the life of millionaire Jay Gatsby and his pursuit of Daisy Buchanan." ] model.rank(query, documents, return_documents=True) :: [{'corpus_id': 0, 'score': 10.67858, 'text': "'To Kill a Mockingbird' is a novel by Harper Lee published in 1960. It was immediately successful, winning the Pulitzer Prize, and has become a classic of modern American literature."}, {'corpus_id': 2, 'score': 9.761677, 'text': "Harper Lee, an American novelist widely known for her novel 'To Kill a Mockingbird', was born in 1926 in Monroeville, Alabama. She received the Pulitzer Prize for Fiction in 1961."}, {'corpus_id': 1, 'score': -3.3099542, 'text': "The novel 'Moby-Dick' was written by Herman Melville and first published in 1851. It is considered a masterpiece of American literature and deals with complex themes of obsession, revenge, and the conflict between good and evil."}, {'corpus_id': 5, 'score': -4.8989105, 'text': "'The Great Gatsby', a novel written by American author F. Scott Fitzgerald, was published in 1925. The story is set in the Jazz Age and follows the life of millionaire Jay Gatsby and his pursuit of Daisy Buchanan."}, {'corpus_id': 4, 'score': -5.082967, 'text': "The 'Harry Potter' series, which consists of seven fantasy novels written by British author J.K. Rowling, is among the most popular and critically acclaimed books of the modern era."}] cg|]}|gSrTrT)r4docrs r6r7z%CrossEncoder.rank..s===CE3<===r8)rrrrrrrr) corpus_idrrrkc|dS)NrrT)xs r6z#CrossEncoder.rank..'s ' r8T)keyreverseN)rrarbrsorted)rKrrrrrrrrrrrquery_doc_pairsscoresresultsirs ` r6rankzCrossEncoder.ranksJ>===9===%!/#)'-/   !&)) ; ;HAu NNU;; < < < ; ""FIaL#9:::&:&:DIIIvvr8c|I|||||}| ||||||jkr"||_|r||dSdSdSdS)z#Runs evaluation during the trainingN)rrsteps)rsave)rKrurrrrrrs r6rz"CrossEncoder._eval_during_training*s  Id 5PUVVVE#u---t&&"'"+IIk***** ! '&++r8)safe_serializationpathrc |dStd||jj|fd|i||jj|fi|dS)zW Saves the model and tokenizer to path; identical to `save_pretrained` NzSave model to r)rArBr?save_pretrainedr@rKrrkwargss r6rzCrossEncoder.save5sp < F +T++,,," "4YY %&& &*##'!%       '&     r8r)- __future__rrr functoolsrtypingrrrrrrCrr torch.optimr torch.utils.datar tqdm.autonotebookr r transformersrrrr$transformers.tokenization_utils_basertransformers.utilsr2sentence_transformers.evaluation.SentenceEvaluatorrsentence_transformers.readersr)sentence_transformers.SentenceTransformerrsentence_transformers.utilrrr getLoggerr+rArrTr8r6r@s"""""" .......... !!!!!!''''''********nnnnnnnnnnnn>>>>>>------PPPPPP666666IIIIIITTTTTTTTTT  8 $ $D  D  D  D  D  >D  D  D  D  D  r8