§ †ìNgW.ãóâ—dZddlmZmZmZmZmZmZmZm Z ddl m Z ddl m Z ddlmZmZmZddlmZmZGd„de ¦«ZGd „d e ¦«Zd eeefd efd „ZGd„de¦«ZdS)zGContains the LLMEvaluator class for building LLM-as-a-judge evaluators.é)ÚAnyÚCallableÚDictÚListÚOptionalÚTupleÚUnionÚcast)Ú BaseModel)Ú warn_beta)ÚEvaluationResultÚEvaluationResultsÚ RunEvaluator)ÚExampleÚRuncóf—eZdZUdZeed<eeed<eed<dZeed<dZ e eed<dS) ÚCategoricalScoreConfigz&Configuration for a categorical score.ÚkeyÚchoicesÚ descriptionFÚinclude_explanationNÚexplanation_description) Ú__name__Ú __module__Ú __qualname__Ú__doc__ÚstrÚ__annotations__rrÚboolrr©óú^/var/www/html/ai-engine/env/lib/python3.11/site-packages/langsmith/evaluation/llm_evaluator.pyrr sc€€€€€€Ø0Ð0à €H€HHØ #ŒYÐÐÑØÐÐÑØ %ИÐ%Ð%Ñ%Ø-1ИX cœ]Ð1Ð1Ñ1Ð1Ð1r!rcól—eZdZUdZeed<dZeed<dZeed<eed<dZ e ed <d Z e eed <d S) ÚContinuousScoreConfigz%Configuration for a continuous score.rrÚminéÚmaxrFrNr) rrrrrrr%Úfloatr'rrrrr r!r"r$r$sm€€€€€€Ø/Ð/à €H€HHØ€Cˆ€N€NNØ€Cˆ€N€NNØÐÐÑØ %ИÐ%Ð%Ñ%Ø-1ИX cœ]Ð1Ð1Ñ1Ð1Ð1r!r$Ú score_configÚreturncó—i}t|t¦«r+d|jdd |j¦«›ddœ|d<nJt|t¦«r&d|j|jd|j›d |j›d d œ|d<ntd ¦«‚|jrd|j €d n|j dœ|d<|j |j d||jrddgndgdœS)NÚstringz%The score for the evaluation, one of z, ú.)ÚtypeÚenumrÚscoreÚnumberz&The score for the evaluation, between z and z , inclusive.)r.ÚminimumÚmaximumrz9Invalid score type. Must be 'categorical' or 'continuous'zThe explanation for the score.)r.rÚ explanationÚobject)Útitlerr.Ú propertiesÚrequired) Ú isinstancerrÚjoinr$r%r'Ú ValueErrorrrrr)r)r7s r"Ú_create_score_json_schemar<!sL€ð"$€JÝ,Õ 6Ñ7Ô7ðVàØ Ô(ð2ØyŠy˜Ô-Ñ.Ô.ð2ð2ð2ð ð ˆ 7ÑÐõ LÕ"7Ñ 8Ô 8ð VàØ#Ô'Ø#Ô'ðEØÔðEðEØ&2Ô&6ðEðEðEð  ð ˆ 7ÑÐõÐTÑUÔUÐUàÔ'ð àð Ô7Ð?ð1Ð0à!Ô9ð % ð% ˆ =Ñ!ðÔ!Ø#Ô/ØØ à(4Ô(HÐ WˆWmÐ $Ð $ÈwÈið  ð ðr!có~—eZdZdZddddœdeeeeeeffdeee fde e e e e gefd ed ef d „Zedd œd edeeeeeeffdeee fde e e e e geffd„¦«Zdeeeeeeffdeee fde e e e e gefdefd„Ze dde de e deeeffd„¦«Ze dde de e deeeffd„¦«Zde de e defd„Zdedeeeffd„ZdS)Ú LLMEvaluatorz/A class for building LLM-as-a-judge evaluators.Nzgpt-4oÚopenai)Ú map_variablesÚ model_nameÚmodel_providerÚprompt_templater)r@rArBc ó — ddlm}n"#t$r}td¦«|‚d}~wwxYw|d||dœ|¤Ž} | |||| ¦«dS)aæInitialize the LLMEvaluator. Args: prompt_template (Union[str, List[Tuple[str, str]]): The prompt template to use for the evaluation. If a string is provided, it is assumed to be a human / user message. score_config (Union[CategoricalScoreConfig, ContinuousScoreConfig]): The configuration for the score, either categorical or continuous. map_variables (Optional[Callable[[Run, Example], dict]], optional): A function that maps the run and example to the variables in the prompt. Defaults to None. If None, it is assumed that the prompt only requires 'input', 'output', and 'expected'. model_name (Optional[str], optional): The model to use for the evaluation. Defaults to "gpt-4o". model_provider (Optional[str], optional): The model provider to use for the evaluation. Defaults to "openai". r)Úinit_chat_modelzmLLMEvaluator requires langchain to be installed. Please install langchain by running `pip install langchain`.N)ÚmodelrBr )Úlangchain.chat_modelsrEÚ ImportErrorÚ _initialize) ÚselfrCr)r@rArBÚkwargsrEÚeÚ chat_models r"Ú__init__zLLMEvaluator.__init__Os¤€ð6 Ø =Ð =Ð =Ð =Ð =Ð =Ð =øÝð ð ð ÝðOñôðð øøøøð øøøð %_ð ب^ð ð Ø?Eð ð ˆ ð ×Ò˜¨,¸ ÀzÑRÔRÐRÐRÐRs‚ ‰ (“#£()r@rFcó`—| |¦«}| ||||¦«|S)a›Create an LLMEvaluator instance from a BaseChatModel instance. Args: model (BaseChatModel): The chat model instance to use for the evaluation. prompt_template (Union[str, List[Tuple[str, str]]): The prompt template to use for the evaluation. If a string is provided, it is assumed to be a system message. score_config (Union[CategoricalScoreConfig, ContinuousScoreConfig]): The configuration for the score, either categorical or continuous. map_variables (Optional[Callable[[Run, Example]], dict]], optional): A function that maps the run and example to the variables in the prompt. Defaults to None. If None, it is assumed that the prompt only requires 'input', 'output', and 'expected'. Returns: LLMEvaluator: An instance of LLMEvaluator. )Ú__new__rI)ÚclsrFrCr)r@Úinstances r"Ú from_modelzLLMEvaluator.from_modelxs4€ð4—;’;˜sÑ#Ô#ˆØ×Ò˜_¨l¸MÈ5ÑQÔQÐQ؈r!rMcó\— ddlm}ddlm}n"#t$r}t d¦«|‚d}~wwxYwt ||¦«rt |d¦«std¦«‚t |t¦«r|  d|fg¦«|_ n|  |¦«|_ t|j j ¦«hd £z r|std ¦«‚||_ ||_t|j¦«|_| |j¦«}|j |z|_dS) aÑShared initialization code for __init__ and from_model. Args: prompt_template (Union[str, List[Tuple[str, str]]): The prompt template. score_config (Union[CategoricalScoreConfig, ContinuousScoreConfig]): The score configuration. map_variables (Optional[Callable[[Run, Example]], dict]]): Function to map variables. chat_model (BaseChatModel): The chat model instance. r)Ú BaseChatModel)ÚChatPromptTemplatez|LLMEvaluator requires langchain-core to be installed. Please install langchain-core by running `pip install langchain-core`.NÚwith_structured_outputzRchat_model must be an instance of BaseLanguageModel and support structured output.Úhuman>ÚinputÚoutputÚexpectedzrmap_inputs must be provided if the prompt template contains variables other than 'input', 'output', and 'expected')Ú*langchain_core.language_models.chat_modelsrUÚlangchain_core.promptsrVrHr9Úhasattrr;rÚ from_messagesÚpromptÚsetÚinput_variablesr@r)r<Ú score_schemarWÚrunnable)rJrCr)r@rMrUrVrLs r"rIzLLMEvaluator._initialize–s‚€ð" Ø PÐ PÐ PÐ PÐ PÐ PØ AÐ AÐ AÐ AÐ AÐ AÐ AøÝð ð ð ÝðYñôðð øøøøð øøøõ z =Ñ 1Ô 1ð å˜ Ð$<Ñ=Ô=ð õðCñôð õ o¥sÑ +Ô +ð LØ,×:Ò:¸WÀoÐr>LsÈ€€€€€Ø9Ð9ðMQØ"Ø&ð'Sð'Sð'Sð˜s D¨¨s°C¨x¬Ô$9Ð9Ô:ð'SðÐ2Ð4IÐIÔJð 'Sð   ¨#¨x¸Ô/@Ð)AÀ4Ð)GÔ HÔIð 'Sð ð 'Sðð'Sð'Sð'Sð'SðRðMQð ððàðð˜s D¨¨s°C¨x¬Ô$9Ð9Ô:ð ð Ð2Ð4IÐIÔJð ð   ¨#¨x¸Ô/@Ð)AÀ4Ð)GÔ HÔIð ððñ„[ðð:41à˜s D¨¨s°C¨x¬Ô$9Ð9Ô:ð41ðÐ2Ð4IÐIÔJð41ð  ¨#¨x¸Ô/@Ð)AÀ4Ð)GÔ HÔIð 41ð ð 41ð41ð41ð41ðlà59ð*ð*Øð*Ø!)¨'Ô!2ð*à ÐÐ!2Ð2Ô 3ð*ð*ð*ñ„Yð*ðà59ð*ð*Øð*Ø!)¨'Ô!2ð*à ÐÐ!2Ð2Ô 3ð*ð*ð*ñ„Yð*ð7 cð7°H¸WÔ4Eð7È$ð7ð7ð7ð7ðr  Dð ¨UÐ3CÐEVÐ3VÔ-Wð ð ð ð ð ð r!r>N)rÚtypingrrrrrrr r Úpydanticr Ú#langsmith._internal._beta_decoratorr Úlangsmith.evaluationr rrÚlangsmith.schemasrrrr$rir<r>r r!r"úr‚sdðØMÐMàJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJàÐÐÐÐÐà9Ð9Ð9Ð9Ð9Ð9ØRÐRÐRÐRÐRÐRÐRÐRÐRÐRØ*Ð*Ð*Ð*Ð*Ð*Ð*Ð*ð2ð2ð2ð2ð2˜Yñ2ô2ð2ð2ð2ð2ð2ð2˜Iñ2ô2ð2ð(ØÐ.Ð0EÐEÔFð(à ð(ð(ð(ð(ðVXðXðXðXðX<ñXôXðXðXðXr!