§ ‡ìNgGãó`—dZddlmZddlZddlmZmZddlmZddl m Z m Z m Z m Z mZddlmZddlmZdd lmZdd lmZdd lmZeje¦«ZGd „d ee¦«ZGd„de¦«ZGd„d¦«ZGd„dee¦«Z Gd„dee¦«Z!Gd„dee¦«Z"dS)z3Interfaces to be implemented by general evaluators.é)Ú annotationsN)ÚABCÚabstractmethod)ÚEnum)ÚAnyÚOptionalÚSequenceÚTupleÚUnion)Úwarn)Ú AgentAction)ÚBaseLanguageModel)Úrun_in_executor)ÚChaincóˆ—eZdZdZdZ dZ dZ dZ dZ dZ dZ d Z d Z d Z d Z d Z dZ dZ dZ dZ dZ dZ dZ dZdS)Ú EvaluatorTypezThe types of the evaluators.ÚqaÚcot_qaÚ context_qaÚpairwise_stringÚ score_stringÚlabeled_pairwise_stringÚlabeled_score_stringÚ trajectoryÚcriteriaÚlabeled_criteriaÚstring_distanceÚ exact_matchÚ regex_matchÚpairwise_string_distanceÚembedding_distanceÚpairwise_embedding_distanceÚ json_validityÚ json_equalityÚjson_edit_distanceÚjson_schema_validationN)Ú__name__Ú __module__Ú __qualname__Ú__doc__ÚQAÚCOT_QAÚ CONTEXT_QAÚPAIRWISE_STRINGÚ SCORE_STRINGÚLABELED_PAIRWISE_STRINGÚLABELED_SCORE_STRINGÚAGENT_TRAJECTORYÚCRITERIAÚLABELED_CRITERIAÚSTRING_DISTANCEÚ EXACT_MATCHÚ REGEX_MATCHÚPAIRWISE_STRING_DISTANCEÚEMBEDDING_DISTANCEÚPAIRWISE_EMBEDDING_DISTANCEÚ JSON_VALIDITYÚ JSON_EQUALITYÚJSON_EDIT_DISTANCEÚJSON_SCHEMA_VALIDATION©óúW/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain/evaluation/schema.pyrrsЀ€€€€Ø&Ð&à €Bðà €Fð%ð€JØSØ'€Oðà!€Lðà7ÐðHà1Ðð@à#ÐØVØ€Hð<à)Ðð7à'€OØPØ€KØIØ€KØNØ9ÐØ=Ø-ÐØMØ"?ÐØ;Ø#€MØ.Ø#€MØ=Ø-ÐØTØ5ÐØIÐIr@rcó:—eZdZdZeed d„¦«¦«ZdS) Ú LLMEvalChainz,A base class for evaluators that use an LLM.ÚllmrÚkwargsrÚreturnc ó—dS)z#Create a new evaluator from an LLM.Nr?)ÚclsrDrEs rAÚfrom_llmzLLMEvalChain.from_llmMó€€€r@N)rDrrErrFrC)r'r(r)r*Ú classmethodrrIr?r@rArCrCJsB€€€€€Ø6Ð6àØð2ð2ð2ñ„^ñ„[ð2ð2ð2r@rCcó€—eZdZdZedd„¦«Zedd„¦«Zedd„¦«Zedd„¦«Z ddd„Z d S)Ú_EvalArgsMixinz(Mixin for checking evaluation arguments.rFÚboolcó—dS©z2Whether this evaluator requires a reference label.Fr?©Úselfs rAÚrequires_referencez!_EvalArgsMixin.requires_referenceVó €ðˆur@có—dS)ú0Whether this evaluator requires an input string.Fr?rQs rAÚrequires_inputz_EvalArgsMixin.requires_input[rTr@Ústrcó"—d|jj›dS)z&Warning to show when input is ignored.zIgnoring input in ú, as it is not expected.©Ú __class__r'rQs rAÚ_skip_input_warningz"_EvalArgsMixin._skip_input_warning`s€ðV D¤NÔ$;ÐUÐUÐUÐUr@có"—d|jj›dS)z*Warning to show when reference is ignored.zIgnoring reference in rZr[rQs rAÚ_skip_reference_warningz&_EvalArgsMixin._skip_reference_warninges€ð W T¤^Ô%<Ð VÐ VÐ Vð r@NÚ referenceú Optional[str]ÚinputÚNonecó—|jr|€t|jj›d¦«‚||jst |j¦«|jr|€t|jj›d¦«‚||jst |j¦«dSdSdS)a†Check if the evaluation arguments are valid. Args: reference (Optional[str], optional): The reference label. input (Optional[str], optional): The input string. Raises: ValueError: If the evaluator requires an input string but none is provided, or if the evaluator requires a reference label but none is provided. Nz requires an input string.z requires a reference string.)rWÚ ValueErrorr\r'r r]rSr_)rRr`rbs rAÚ_check_evaluation_argsz%_EvalArgsMixin._check_evaluation_argsls­€ð Ô ð + 5 =Ý ¤Ô 7ÐSÐSÐSÑTÔTÐ TØ Ð  tÔ':Ð Ý Ô)Ñ *Ô *Ð *Ø Ô "ð / yÐ'8Ý ¤Ô 7ÐVÐVÐVÑWÔWÐ WØ Ð "¨4Ô+BÐ "Ý Ô-Ñ .Ô .Ð .Ð .Ð .ð#Ð "Ð "Ð "r@©rFrN©rFrX)NN)r`rarbrarFrc) r'r(r)r*ÚpropertyrSrWr]r_rfr?r@rArMrMSs½€€€€€Ø2Ð2à ðððñ„Xðððððñ„XðððVðVðVñ„XðVðð ð ð ñ„Xð ð$(Ø#ð/ð/ð/ð/ð/ð/ð/r@rMcó’—eZdZdZedd„¦«Zedd„¦«Zedddœdd„¦«Zdddœdd„Z dddœdd„Z dddœdd„Z dS)ÚStringEvaluatorzcGrade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels.rFrXcó—|jjS)zThe name of the evaluation.r[rQs rAÚevaluation_namezStringEvaluator.evaluation_nameˆs€ðŒ~Ô&Ð&r@rNcó—dSrPr?rQs rArSz"StringEvaluator.requires_referencerTr@N©r`rbÚ predictionúUnion[str, Any]r`úOptional[Union[str, Any]]rbrErÚdictc ó—dS)a:Evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nr?©rRrpr`rbrEs rAÚ_evaluate_stringsz!StringEvaluator._evaluate_strings’rJr@c‹ó@K—td|jf|||dœ|¤Žƒd{V—†S)aIAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. N©rpr`rb)rrvrus rAÚ_aevaluate_stringsz"StringEvaluator._aevaluate_stringsªs`èè€õ,%Ø Ø Ô "ð ð"ØØð  ð ð ð  ð ð ð ð ð ð ð ð r@rac óR—| ||¬¦«|jd|||dœ|¤ŽS)aúEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. rorxr?)rfrvrus rAÚevaluate_stringsz StringEvaluator.evaluate_stringsÉsM€ð$ ×#Ò#¨i¸uÐ#ÑEÔEÐEØ%ˆtÔ%ð Ø!¨Y¸eð ð ØGMð ð ð r@c‹óbK—| ||¬¦«|jd|||dœ|¤Žƒd{V—†S)a Asynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. rorxNr?)rfryrus rAÚaevaluate_stringsz!StringEvaluator.aevaluate_stringsàsoèè€ð$ ×#Ò#¨i¸uÐ#ÑEÔEÐEØ,TÔ,ð Ø!¨Y¸eð ð ØGMð ð ð ð ð ð ð ð ð r@rhrg) rprqr`rrrbrrrErrFrs) rprXr`rarbrarErrFrs) r'r(r)r*rirmrSrrvryr{r}r?r@rArkrk„s€€€€€ð ð ðð'ð'ð'ñ„Xð'ððððñ„Xððð 04Ø+/ð  ð ð ð ð ñ„^ð ð604Ø+/ð  ð ð ð ð ð ðF$(Ø#ð  ð ð ð ð ð ð6$(Ø#ð  ð ð ð ð ð ð ð r@rkcób—eZdZdZedddœdd„¦«Zdddœdd„Zdddœdd„Zdddœdd„ZdS)ÚPairwiseStringEvaluatorzDCompare the output of two models (or two outputs of the same model).NrorprXÚ prediction_br`rarbrErrFrsc ó—dS)á1Evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nr?©rRrpr€r`rbrEs rAÚ_evaluate_string_pairsz.PairwiseStringEvaluator._evaluate_string_pairsûrJr@c‹óBK—td|jf||||dœ|¤Žƒd{V—†S)á@Asynchronously evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. N©rpr€r`rb)rr„rƒs rAÚ_aevaluate_string_pairsz/PairwiseStringEvaluator._aevaluate_string_pairsscèè€õ(%Ø Ø Ô 'ð ð"Ø%ØØð  ð ðð ð ð ð ð ð ð ð ð r@c óT—| ||¬¦«|jd||||dœ|¤ŽS)r‚ror‡r?)rfr„rƒs rAÚevaluate_string_pairsz-PairwiseStringEvaluator.evaluate_string_pairs/sT€ð( ×#Ò#¨i¸uÐ#ÑEÔEÐEØ*ˆtÔ*ð Ø!Ø%ØØð  ð ð ð  ð ð r@c‹ódK—| ||¬¦«|jd||||dœ|¤Žƒd{V—†S)r†ror‡Nr?)rfrˆrƒs rAÚaevaluate_string_pairsz.PairwiseStringEvaluator.aevaluate_string_pairsLsvèè€ð( ×#Ò#¨i¸uÐ#ÑEÔEÐEØ1TÔ1ð Ø!Ø%ØØð  ð ð ð  ð ð ð ð ð ð ð ð r@) rprXr€rXr`rarbrarErrFrs) r'r(r)r*rr„rˆrŠrŒr?r@rArrøs¾€€€€€ØNÐNàð $(Ø#ð  ð ð ð ð ñ„^ð ð4$(Ø#ð  ð ð ð ð ð ðF$(Ø#ð  ð ð ð ð ð ðD$(Ø#ð  ð ð ð ð ð ð ð r@rcór—eZdZdZedd„¦«Zeddœdd„¦«Zddœdd„Zddœdd„Z ddœdd„Z dS)ÚAgentTrajectoryEvaluatorz,Interface for evaluating agent trajectories.rFrNcó—dS)rVTr?rQs rArWz'AgentTrajectoryEvaluator.requires_inputms €ðˆtr@N)r`rprXÚagent_trajectoryú!Sequence[Tuple[AgentAction, str]]rbr`rarErrsc ó—dS)á–Evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. Nr?©rRrprrbr`rEs rAÚ_evaluate_agent_trajectoryz3AgentTrajectoryEvaluator._evaluate_agent_trajectoryrrJr@c‹óBK—td|jf||||dœ|¤Žƒd{V—†S)á¥Asynchronously evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. N)rprr`rb)rr•r”s rAÚ_aevaluate_agent_trajectoryz4AgentTrajectoryEvaluator._aevaluate_agent_trajectory‰scèè€õ*%Ø Ø Ô +ð ð"Ø-ØØð  ð ðð ð ð ð ð ð ð ð ð r@c óT—| ||¬¦«|jd||||dœ|¤ŽS)r“ro©rprbrr`r?)rfr•r”s rAÚevaluate_agent_trajectoryz2AgentTrajectoryEvaluator.evaluate_agent_trajectory¨sT€ð* ×#Ò#¨i¸uÐ#ÑEÔEÐEØ.ˆtÔ.ð Ø!ØØ-Øð  ð ð ð  ð ð r@c‹ódK—| ||¬¦«|jd||||dœ|¤Žƒd{V—†S)r—roršNr?)rfr˜r”s rAÚaevaluate_agent_trajectoryz3AgentTrajectoryEvaluator.aevaluate_agent_trajectoryÆsvèè€ð* ×#Ò#¨i¸uÐ#ÑEÔEÐEØ5TÔ5ð Ø!ØØ-Øð  ð ð ð  ð ð ð ð ð ð ð ð r@rg) rprXrr‘rbrXr`rarErrFrs) r'r(r)r*rirWrr•r˜r›rr?r@rArŽrŽjsÒ€€€€€Ø6Ð6à ðððñ„Xððð$(ð  ð ð ð ð ñ„^ð ð8$(ð  ð ð ð ð ð ðJ$(ð  ð ð ð ð ð ðH$(ð  ð ð ð ð ð ð ð r@rŽ)#r*Ú __future__rÚloggingÚabcrrÚenumrÚtypingrrr r r Úwarningsr Úlangchain_core.agentsr Úlangchain_core.language_modelsrÚlangchain_core.runnables.configrÚlangchain.chains.baserÚ getLoggerr'ÚloggerrXrrCrMrkrrŽr?r@rAúrªsðØ9Ð9à"Ð"Ð"Ð"Ð"Ð"à€€€Ø#Ð#Ð#Ð#Ð#Ð#Ð#Ð#ØÐÐÐÐÐØ8Ð8Ð8Ð8Ð8Ð8Ð8Ð8Ð8Ð8Ð8Ð8Ð8Ð8ØÐÐÐÐÐà-Ð-Ð-Ð-Ð-Ð-Ø<Ð<Ð<Ð<Ð<Ð<Ø;Ð;Ð;Ð;Ð;Ð;à'Ð'Ð'Ð'Ð'Ð'à ˆÔ ˜8Ñ $Ô $€ð3Jð3Jð3Jð3Jð3JC˜ñ3Jô3Jð3Jðl2ð2ð2ð2ð25ñ2ô2ð2ð./ð./ð./ð./ð./ñ./ô./ð./ðbq ðq ðq ðq ðq n cñq ôq ðq ðho ðo ðo ðo ðo ˜n¨cño ôo ðo ðdx ðx ðx ðx ðx ˜~¨sñx ôx ðx ðx ðx r@