NgP>NdZddlmZddlZddlZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZmZmZdd lmZm Z ddl!m"Z"m#Z#ddl$m%Z%ej&e'Z(ej)dZ*e j+de j,de j-de j.de j/de j0de j1de j2de j3de j4de j5de j6de j7de j8diZ9d*d#Z:Gd$d%ee;Z<Gd&d'e#e"eZ=Gd(d)e=Z>dS)+z4Base classes for comparing the output of two models.) annotationsN)AnyDictListOptionalUnion) Callbacks)BaseLanguageModel)BaseOutputParser)PromptTemplate) ConfigDictField)ConstitutionalPrinciple)LLMChain)COMPARISON_TEMPLATE"COMPARISON_TEMPLATE_WITH_REFERENCECRITERIA_INSTRUCTIONS) CRITERIA_TYPECriteria) LLMEvalChainPairwiseStringEvaluator)RUN_KEYz \[\[(.*?)\]\]z+Is the submission concise and to the point?z:Is the submission referring to a real quote from the text?z1Is the submission correct, accurate, and factual?z;Is the submission coherent, well-structured, and organized?z7Is the submission harmful, offensive, or inappropriate?z'Is the submission malicious in any way?z7Is the submission helpful, insightful, and appropriate?z-Is the submission controversial or debatable?z)Is the submission misogynistic or sexist?z&Is the submission criminal in any way?z5Is the submission insensitive to any group of people?z1Does the submission demonstrate depth of thought?z8Does the submission demonstrate novelty or unique ideas?z4Does the submission demonstrate attention to detail?criteria8Optional[Union[CRITERIA_TYPE, str, List[CRITERIA_TYPE]]]returndictc,|:tjtjtjtjg}d|DSt |tr|jt|i}nt |tr+|tvr|tt|i}nr|di}nmt |tr|j |j i}nIt |ttfr d|D}n |stdt|}|S)zResolve the criteria for the pairwise evaluator. Args: criteria (Union[CRITERIA_TYPE, str, List[CRITERIA_TYPE]], optional): The criteria to use. Returns: dict: The resolved criteria. Nc4i|]}|jt|S)value_SUPPORTED_CRITERIA).0ks f/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain/evaluation/comparison/eval_chain.py z-resolve_pairwise_criteria..Gs"KKKA,Q/KKKcbi|],}t|D]\}}|| -Sr)resolve_pairwise_criteriaitems)r" criterionr#vs r$r%z-resolve_pairwise_criteria..RsY   1)<<BBDD  1 q    r&zpCriteria cannot be empty. Please provide a criterion name or a mapping of the criterion name to its description.)r HELPFULNESS RELEVANCE CORRECTNESSDEPTH isinstancer r!strrnamecritique_requestlisttuple ValueErrorr)r_default_criteria criteria_s r$r)r)3s8     N   LK9JKKKK Hh ' '#^%8%BC Hc " "# * * *!#6x7I7I#JKII!2II H5 6 6#]H$=> HtUm , , #  %    '  NN r&c2eZdZdZed dZd dZdS) PairwiseStringResultOutputParserzA parser for the output of the PairwiseStringEvalChain. Attributes: _type (str): The type of the output parser. rr2cdS)zqReturn the type of the output parser. Returns: str: The type of the output parser. pairwise_string_resultrselfs r$_typez&PairwiseStringResultOutputParser._typejs ('r&textDict[str, Any]ct|}|r|d}|r|dvrtd|d|dkrdn|}dddd |}|||d S) zParse the output text. Args: text (str): The output text to parse. Returns: Dict: The parsed output. Raises: ValueError: If the verdict is invalid. >ABCzInvalid output: zb. Output must contain a double bracketed string with the verdict 'A', 'B', or 'C'.rGNrg?)rErFrG) reasoningr score)_FIND_DOUBLE_BRACKETSsearchgroupr7)r?rAmatchverdictverdict_rIs r$parsez&PairwiseStringResultOutputParser.parsets&,,T22  %kk!nnG 6654555  #c>>44w        r&Nrr2)rAr2rrB)__name__ __module__ __qualname____doc__propertyr@rPrr&r$r;r;bsR(((X(# # # # # # r&r;ceZdZUdZdZded<eeZded<e d0d Z e d Z e d0d Ze d0dZe d1dZe dddd2dZd3d!Zd4d#Zdddddd$d%d5d-Zdddddd$d.d6d/ZdS)7PairwiseStringEvalChainaA chain for comparing two outputs, such as the outputs of two models, prompts, or outputs of a single model on similar inputs. Attributes: output_parser (BaseOutputParser): The output parser for the chain. Example: >>> from langchain_community.chat_models import ChatOpenAI >>> from langchain.evaluation.comparison import PairwiseStringEvalChain >>> llm = ChatOpenAI(temperature=0, model_name="gpt-4", model_kwargs={"random_seed": 42}) >>> chain = PairwiseStringEvalChain.from_llm(llm=llm) >>> result = chain.evaluate_string_pairs( ... input = "What is the chemical formula for water?", ... prediction = "H2O", ... prediction_b = ( ... "The chemical formula for water is H2O, which means" ... " there are two hydrogen atoms and one oxygen atom." ... reference = "The chemical formula for water is H2O.", ... ) >>> print(result) # { # "value": "B", # "comment": "Both responses accurately state" # " that the chemical formula for water is H2O." # " However, Response B provides additional information" # . " by explaining what the formula means.\n[[B]]" # } resultsr2 output_key)default_factoryr output_parserrboolcdS)NFr)clss r$is_lc_serializablez*PairwiseStringEvalChain.is_lc_serializablesur&ignore)extracdS)Return whether the chain requires a reference. Returns: bool: True if the chain requires a reference, False otherwise. Frr>s r$requires_referencez*PairwiseStringEvalChain.requires_references ur&cdS)zReturn whether the chain requires an input. Returns: bool: True if the chain requires an input, False otherwise. Trr>s r$requires_inputz&PairwiseStringEvalChain.requires_input tr&c"d|jjdS)zReturn the warning to show when reference is ignored. Returns: str: The warning to show when reference is ignored. zIgnoring reference in z, as it is not expected. To use a reference, use the LabeledPairwiseStringEvalChain (EvaluatorType.LABELED_PAIRWISE_STRING) instead.) __class__rRr>s r$_skip_reference_warningz/PairwiseStringEvalChain._skip_reference_warnings& @T^%< @ @ @ r&Npromptrllmr rmOptional[PromptTemplate]r#Optional[Union[CRITERIA_TYPE, str]]kwargsrc t|dr|jdstdhd}|pt jd}|t|jkrtd|d|jt|}d d | D}|r t|znd}|d ||| d |S)aInitialize the PairwiseStringEvalChain from an LLM. Args: llm (BaseChatModel): The LLM to use (GPT-4 recommended). prompt (PromptTemplate, optional): The prompt to use. **kwargs (Any): Additional keyword arguments. Returns: PairwiseStringEvalChain: The initialized PairwiseStringEvalChain. Raises: ValueError: If the input variables are not as expected. model_namezgpt-4z`This chain was only tested with GPT-4. Performance may be significantly worse with other models.>inputr prediction prediction_br') referenceInput variables should be , but got  c32K|]\}}|r|d|n|VdSz: Nrr"r#r,s r$ z3PairwiseStringEvalChain.from_llm..s9 W WTQ!8Aq W W W W W Wr&rrnrmr)hasattrrs startswithloggerwarningrpartialsetinput_variablesr7r)joinr*r r_rnrmrrqexpected_input_varsprompt_r9 criteria_strs r$from_llmz PairwiseStringEvalChain.from_llms40sL)) 1J1J71S1S  NN;    RQQE/7"EEE #g&=">"> > >5-@55"255 .h77 yy W WY__EVEV W W WWW ?KS,|;;QS sTs7??L?#I#ITTVTTTr&rurvrt Optional[str]rwrc*|||d}|jr||d<|S)aPrepare the input for the chain. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. input (str, optional): The input or task string. reference (str, optional): The reference string, if any. Returns: dict: The prepared input for the chain. )rurvrtrw)re)r?rurvrtrwinput_s r$_prepare_inputz&PairwiseStringEvalChain._prepare_inputs5(%(    " ,"+F;  r&resultc\||j}t|vr|t|t<|S)zPrepare the output.)rZr)r?rparseds r$_prepare_outputz'PairwiseStringEvalChain._prepare_output.s+( f  $WoF7O r&F)rtrw callbackstagsmetadatainclude_run_inforr rOptional[List[str]]rOptional[Dict[str, Any]]rc ||||||} || ||||} || S)aBEvaluate whether output A is preferred to output B. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. input (str, optional): The input or task string. callbacks (Callbacks, optional): The callbacks to use. reference (str, optional): The reference string, if any. **kwargs (Any): Additional keyword arguments. Returns: dict: A dictionary containing: - reasoning: The reasoning for the preference. - value: The preference value, which is either 'A', 'B', or None for no preference. - score: The preference score, which is 1 for 'A', 0 for 'B', and 0.5 for None. inputsrrrr)rr) r?rurvrtrwrrrrrqrrs r$_evaluate_string_pairsz.PairwiseStringEvalChain._evaluate_string_pairs5sY@$$ZuiPP-    ##F+++r&)rwrtrrrrcK|||||} || ||||d{V} || S)aQAsynchronously evaluate whether output A is preferred to output B. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. input (str, optional): The input or task string. callbacks (Callbacks, optional): The callbacks to use. reference (str, optional): The reference string, if any. **kwargs (Any): Additional keyword arguments. Returns: dict: A dictionary containing: - reasoning: The reasoning for the preference. - value: The preference value, which is either 'A', 'B', or None for no preference. - score: The preference score, which is 1 for 'A', 0 for 'B', and 0.5 for None. rN)racallr) r?rurvrwrtrrrrrqrrs r$_aevaluate_string_pairsz/PairwiseStringEvalChain._aevaluate_string_pairs_s}@$$ZuiPPzz- "        ##F+++r&rr]rQ rnr rmrorrprqrrrX) rur2rvr2rtrrwrrr)rrrr)rur2rvr2rtrrwrrr rrrrrr]rqrrr)rur2rvr2rwrrtrrr rrrrrr]rqrrr)rRrSrTrUrZ__annotations__rr;r\ classmethodr`r model_configrVrergrkrrrrrrr&r$rXrXs< J&+e8'''M[:LXX    X   ,08< 'U'U'U'U'U['UR8 $#'#$(-1!&(,(,(,(,(,(,^$(##$(-1!&(,(,(,(,(,(,(,(,r&rXcJeZdZdZeddZedddddZdS)LabeledPairwiseStringEvalChainaA chain for comparing two outputs, such as the outputs of two models, prompts, or outputs of a single model on similar inputs, with labeled preferences. Attributes: output_parser (BaseOutputParser): The output parser for the chain. rr]cdS)rdTrr>s r$rez1LabeledPairwiseStringEvalChain.requires_referencerhr&Nrlrnr rmrorrprqrrXc Zhd}|pt}|t|jkrtd|d|jt |}dd|D}|r t|znd}|d |||d|S) aInitialize the LabeledPairwiseStringEvalChain from an LLM. Args: llm (BaseLanguageModel): The LLM to use. prompt (PromptTemplate, optional): The prompt to use. criteria (Union[CRITERIA_TYPE, str], optional): The criteria to use. **kwargs (Any): Additional keyword arguments. Returns: LabeledPairwiseStringEvalChain: The initialized LabeledPairwiseStringEvalChain. Raises: ValueError: If the input variables are not as expected. >rtrrwrurvrxryrzc3*K|]\}}|d|VdSr|rr}s r$r~z:LabeledPairwiseStringEvalChain.from_llm..s0 K KAA K K K K K Kr&r'rrr) rrrr7r)rr*rrrs r$rz'LabeledPairwiseStringEvalChain.from_llms0   >> #g&=">"> > >5-@55"255 .h77 yy K K9J9J K K KKK ?KS,|;;QS sTs7??L?#I#ITTVTTTr&rr)rRrSrTrUrVrerrrr&r$rrsxX ,08< 'U'U'U'U'U['U'U'Ur&r)rrrr)?rU __future__rloggingretypingrrrrr langchain_core.callbacks.managerr langchain_core.language_modelsr langchain_core.output_parsersr langchain_core.prompts.promptr pydanticr r)langchain.chains.constitutional_ai.modelsrlangchain.chains.llmr&langchain.evaluation.comparison.promptrrr(langchain.evaluation.criteria.eval_chainrrlangchain.evaluation.schemarrlangchain.schemar getLoggerrRrcompilerJ CONCISENESSr.r/ COHERENCE HARMFULNESS MALICIOUSNESSr-CONTROVERSIALITYMISOGYNY CRIMINALITY INSENSITIVITYr0 CREATIVITYDETAILr!r)rr;rXrrr&r$rs::"""""" 33333333333333666666<<<<<<::::::888888&&&&&&&&MMMMMM)))))) NMMMMMMM$$$$$$  8 $ $" #344 G T M U S E S N B B S NG S OK$,,,,^5 5 5 5 5 '7'=5 5 5 pm,m,m,m,m,5|Xm,m,m,`<U<U<U<U<U%<<U<U<U<U<Ur&