Ngn1ddlmZddlZddlZddlZddlmZddlmZm Z ddl Z ddl Z ddl Z ddlmZmZddlmZddlmZerddlmZejeZGd d eZdS) ) annotationsN) nullcontext) TYPE_CHECKINGCallable)average_precision_score ndcg_score)SentenceEvaluator)cos_sim)SentenceTransformerc VeZdZdZdddedddddf d#fd Z d$d%dZd Zd!Zd"Z xZ S)&RerankingEvaluatora This class evaluates a SentenceTransformer model for the task of re-ranking. Given a query and a list of documents, it computes the score [query, doc_i] for all possible documents and sorts them in decreasing order. Then, MRR@10, NDCG@10 and MAP is compute to measure the quality of the ranking. Args: samples (list): A list of dictionaries, where each dictionary represents a sample and has the following keys: - 'query': The search query. - 'positive': A list of positive (relevant) documents. - 'negative': A list of negative (irrelevant) documents. at_k (int, optional): Only consider the top k most similar documents to each query for the evaluation. Defaults to 10. name (str, optional): Name of the evaluator. Defaults to "". write_csv (bool, optional): Write results to CSV file. Defaults to True. similarity_fct (Callable[[torch.Tensor, torch.Tensor], torch.Tensor], optional): Similarity function between sentence embeddings. By default, cosine similarity. Defaults to cos_sim. batch_size (int, optional): Batch size to compute sentence embeddings. Defaults to 64. show_progress_bar (bool, optional): Show progress bar when computing embeddings. Defaults to False. use_batched_encoding (bool, optional): Whether or not to encode queries and documents in batches for greater speed, or 1-by-1 to save memory. Defaults to True. truncate_dim (Optional[int], optional): The dimension to truncate sentence embeddings to. `None` uses the model's current truncation dimension. Defaults to None. mrr_at_k (Optional[int], optional): Deprecated parameter. Please use `at_k` instead. Defaults to None. T@FNat_kintnamestr write_csvboolsimilarity_fct4Callable[[torch.Tensor, torch.Tensor], torch.Tensor] batch_sizeshow_progress_baruse_batched_encoding truncate_dim int | Nonemrr_at_kc Lt||_||_| &td| d| |_n||_||_||_||_ ||_ | |_ t|jtr+t|j|_d|jD|_d|rd|zndzd|jdz|_d d d d |jd |jg|_||_d|_dS)Nz?The `mrr_at_k` parameter has been deprecated; please use `at_k=z ` instead.cvg|]6}t|ddkt|ddk4|7S)positivernegativelen.0samples o/var/www/html/ai-engine/env/lib/python3.11/site-packages/sentence_transformers/evaluation/RerankingEvaluator.py z/RerankingEvaluator.__init__..OsQ   VJ5G1H1H11L1LQTU[\fUgQhQhklQlQlFQlQlQlr _rz _results_@z.csvepochstepsMAPMRR@NDCG@map)super__init__samplesrloggerwarningrrrrrr isinstancedictlistvaluescsv_file csv_headersrprimary_metric) selfr4rrrrrrrrr __class__s r(r3zRerankingEvaluator.__init__.sL     NNq]eqqq r r r DIIDI,$!2$8!( dlD ) ) 7 3 3 5 566DL  !%    -d0Jd KNj[_[dNjNjNjj    49   DI    ##r*modelr output_pathr,r-returndict[str, float]c|dkr|dkrd|}n d|d|d}nd}|j|d|jd z }td |jd |d ||}|d }|d}|d} d|jD} d|jD} tdt |jdtj| ddtj | ddtj | ddtj| ddtj | ddtj | dtd|dzdtd|j d|dzdtd|j d| dzd||j rtj||j} tj| } t%| d| rdndd !5}t'j|}| s||j|||||| gdddn #1swxYwYd |d"|j |d#|j | i}|||j}||||S)$a Evaluates the model on the dataset and returns the evaluation metrics. Args: model (SentenceTransformer): The SentenceTransformer model to evaluate. output_path (str, optional): The output path to write the results. Defaults to None. epoch (int, optional): The current epoch number. Defaults to -1. steps (int, optional): The current step number. Defaults to -1. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. r@z after epoch z in epoch z after z stepsrNz (truncated to )z0RerankingEvaluator: Evaluating the model on the z dataset:r1mrrndcgc8g|]}t|dS)r!r#r%s r(r)z/RerankingEvaluator.__call__..%LLLVVJ/00LLLr*c8g|]}t|dS)r"r#r%s r(r)z/RerankingEvaluator.__call__..rKr*z Queries: z Positives: Min z.1fz, Mean z, Max z Negatives: Min zMAP: dz.2fr/z: r0awzutf-8)newlinemodeencodingzmrr@zndcg@)rr5inforcompute_metricesr4r$npminmeanmaxrrospathjoinr;isfileopencsvwriterwriterowr<prefix_name_to_metrics store_metrics_in_model_card_data)r>rArBr,r-out_txtscoresmean_apmean_mrr mean_ndcg num_positives num_negativescsv_pathoutput_file_existsfr_metricss r(__call__zRerankingEvaluator.__call__^s B;;{{1%11BuBBUBBBG   ( =):=== =G dtyddZadddeee&&u---%=6N MLt|LLL LLt|LLL   PDL)) P Pbf]>S>S P P P`b`ghu`v`v P P PCECIJWCXCX P P PqsqwxEqFqF P P PSUSZ[hSiSi P P Puwu{|IuJuJ P P P     /GcM///000 <49<<3<<<=== >DI>>S>>>???  "t~ "w||K??H!#!9!9 h8J1SPS^efff NjkA)6OOD$4555w) LMMM  N N N N N N N N N N N N N N N 7 49   DI    --gtyAA --eW===s3A J  JJcd|jr||n||S)a Computes the evaluation metrics for the given model. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. )rcompute_metrices_batchedcompute_metrices_individual)r>rAs r(rTz#RerankingEvaluator.compute_metricess8( 9D ) )% 0 0 011%88 r*cXg}g}g}|jtn||j5|d|jDd|j|j}g}|jD]8}||d||d9||d|j|j}dddn #1swxYwYd\} } |jD]w} || } | dz } t| d} t| d}|| | | z|z}| | |zz } | d ks|d kr_| | |}t|j dkr|d }tj | }| }dg| zd g|zz}d }t|d |jD]\}}||r d|dzz }n|||t%|g|g|j |t'||yt)j|}t)j|}t)j|}|||d S) aE Computes the evaluation metrics in a batched way, by batching all queries and all documents together. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. Ncg|] }|d S)queryr%s r(r)z?RerankingEvaluator.compute_metrices_batched..s<<rAall_mrr_scoresall_ndcg_scores all_ap_scoresall_query_embsall_docsr' all_docs_embs query_idxdocs_idxinstance query_embnum_posnum_negdocs_emb pred_scorespred_scores_argsort is_relevant mrr_scorerankindexrerfrgs r(rpz+RerankingEvaluator.compute_metrices_batcheds6 "/7[]]]U=_=_`d`q=r=r  "\\<**GO,, xCCCsBCC!Ccg}g}g}tj|j|j dD]}|d}t|d}t|d}t |dkst |dkr\||z} dgt |zdgt |zz} |jt n||j5||gd |j d } || d |j d } dddn #1swxYwY| | | } t | j dkr| d} tj | }| } d}t!|d|jD]\}}| |r d|dzz }n|||t'| g| g|j |t)| | t+j|}t+j|}t+j|}|||d S)aO Computes the evaluation metrics individually by embedding every (query, positive, negative) tuple individually. Args: model (SentenceTransformer): The SentenceTransformer model to compute metrics for. Returns: Dict[str, float]: A dictionary containing the evaluation metrics. Samples)disabledescrtr!r"rrxNTFrvryr{)tqdmr4rr9r$rrr|r}rrrrrrrrrrrrrUrW)r>rArrrrrtr!r"docsrrrrrrrrrerfrgs r(rqz.RerankingEvaluator.compute_metrices_individuals  $,D**GO,, xCCCs >D**D. 1D. )rrrrrrrrrrrrrrrrrr)Nr@r@) rAr rBrr,rr-rrCrD) __name__ __module__ __qualname____doc__r r3rnrTrprq __classcell__)r?s@r(r r s2OV"'%)#'#.$.$.$.$.$.$.$bbd=====~   HDHDHDT:D:D:D:D:D:D:Dr*r ) __future__rr^loggingrY contextlibrtypingrrnumpyrUrrsklearn.metricsrr2sentence_transformers.evaluation.SentenceEvaluatorr sentence_transformers.utilr )sentence_transformers.SentenceTransformerr getLoggerrr5r rur*r(rs,""""""  """"""******** ????????PPPPPP......NMMMMMM  8 $ $ZDZDZDZDZD*ZDZDZDZDZDr*