NgddlmZddlZddlmZmZmZddlmZddl m Z m Z erddl m Z ejeZ dddZGddeZdS)) annotationsN) TYPE_CHECKINGAnyOptional)BaseCallbackHandler) get_from_env guard_import)LoggerF sentimentbooltoxicitythemesreturnrctd}tdtd|rtd|rtd|rtd|S)aImport the langkit python package and raise an error if it is not installed. Args: sentiment: Whether to import the langkit.sentiment module. Defaults to False. toxicity: Whether to import the langkit.toxicity module. Defaults to False. themes: Whether to import the langkit.themes module. Defaults to False. Returns: The imported langkit module. langkitzlangkit.regexeszlangkit.textstatzlangkit.sentimentzlangkit.toxicityzlangkit.themes)r )r r rrs j/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/callbacks/whylabs_callback.pyimport_langkitrsy9%%G"####$$$*())))'((( '%&&& Nc neZdZdZdfd Zdd Zdd Zdd Zd dZe ddddddddd!dZ xZ S)"WhyLabsCallbackHandlera Callback Handler for logging to WhyLabs. This callback handler utilizes `langkit` to extract features from the prompts & responses when interacting with an LLM. These features can be used to guardrail, evaluate, and observe interactions over time to detect issues relating to hallucinations, prompt engineering, or output validation. LangKit is an LLM monitoring toolkit developed by WhyLabs. Here are some examples of what can be monitored with LangKit: * Text Quality - readability score - complexity and grade scores * Text Relevance - Similarity scores between prompt/responses - Similarity scores against user-defined themes - Topic classification * Security and Privacy - patterns - count of strings matching a user-defined regex pattern group - jailbreaks - similarity scores with respect to known jailbreak attempts - prompt injection - similarity scores with respect to known prompt attacks - refusals - similarity scores with respect to known LLM refusal responses * Sentiment and Toxicity - sentiment analysis - toxicity analysis For more information, see https://docs.whylabs.ai/docs/language-model-monitoring or check out the LangKit repo here: https://github.com/whylabs/langkit --- Args: api_key (Optional[str]): WhyLabs API key. Optional because the preferred way to specify the API key is with environment variable WHYLABS_API_KEY. org_id (Optional[str]): WhyLabs organization id to write profiles to. Optional because the preferred way to specify the organization id is with environment variable WHYLABS_DEFAULT_ORG_ID. dataset_id (Optional[str]): WhyLabs dataset id to write profiles to. Optional because the preferred way to specify the dataset id is with environment variable WHYLABS_DEFAULT_DATASET_ID. sentiment (bool): Whether to enable sentiment analysis. Defaults to False. toxicity (bool): Whether to enable toxicity analysis. Defaults to False. themes (bool): Whether to enable theme analysis. Defaults to False. loggerr handlerrcNtt|dr||t|dr||_n-t |_td||_ dS)zInitiate the rolling logger.init_get_callbacksz&initialized handler without callbacks.N) super__init__hasattrrr _callbacksdictdiagnostic_loggerwarning_logger)selfrr __class__s rrzWhyLabsCallbackHandler.__init__Vs  7F # #  LL    7, - - P%4466DOO"ffDO  % %&N O O O rrNonec|jrJt|jdr7|jtddSdSdS)z;Explicitly write current profile if using a rolling logger. _do_rolloverz+Flushing WhyLabs logger, writing profile...N)r#rr(r!infor$s rflushzWhyLabsCallbackHandler.flushbsh < RGDL.AA R L % % ' ' '  " "#P Q Q Q Q Q R R R Rrc|jrJt|jdr7|jtddSdSdS)zFClose any loggers to allow writing out of any profiles before exiting.closez*Closing WhyLabs logger, see you next time!N)r#rr-r!r)r*s rr-zWhyLabsCallbackHandler.closehsh < QGDL':: Q L    " "#O P P P P P Q Q Q Qrc|SNr*s r __enter__z WhyLabsCallbackHandler.__enter__ns rexception_typeexception_value tracebackc.|dSr/)r-)r$r2r3r4s r__exit__zWhyLabsCallbackHandler.__exit__qs rNF)api_keyorg_id dataset_idr r rrr7 Optional[str]r8r9r r r rOptional[Logger]cBt|||td}tdj} tdj} tdj} |||pt dd}|pt d d }|pt d d }| ||| } |ddd| } | | nt d|} | | |}t d|S)anInstantiate whylogs Logger from params. Args: api_key (Optional[str]): WhyLabs API key. Optional because the preferred way to specify the API key is with environment variable WHYLABS_API_KEY. org_id (Optional[str]): WhyLabs organization id to write profiles to. If not set must be specified in environment variable WHYLABS_DEFAULT_ORG_ID. dataset_id (Optional[str]): The model or dataset this callback is gathering telemetry for. If not set must be specified in environment variable WHYLABS_DEFAULT_DATASET_ID. sentiment (bool): If True will initialize a model to perform sentiment analysis compound score. Defaults to False and will not gather this metric. toxicity (bool): If True will initialize a model to score toxicity. Defaults to False and will not gather this metric. themes (bool): If True will initialize a model to calculate distance to configured themes. Defaults to None and will not gather this metric. logger (Optional[Logger]): If specified will bind the configured logger as the telemetry gathering agent. Defaults to LangKit schema with periodic WhyLabs writer. )r r rwhylogszlangkit.callback_handlerzwhylogs.api.writer.whylabsz$whylogs.experimental.core.udf_schemaNr7WHYLABS_API_KEYr8WHYLABS_DEFAULT_ORG_IDr9WHYLABS_DEFAULT_DATASET_ID)r7r8r9rollingM)modeintervalwhenschema)writerz'Using passed in whylogs logger {logger})rimpluGStarted whylogs Logger with WhyLabsWriter and initialized LangKit. 📝) rr get_callback_instance WhyLabsWriter udf_schemarr append_writerr!r))clsr7r8r9r r rrwhyrJrKrLwhylabs_writerwhylabs_loggercallback_handler_clss r from_paramsz"WhyLabsCallbackHandler.from_paramsvsfJ XfMMMM9%% , &! !  %%ABBP !"HIIT >Ki9J!K!KGO|H6NOOF#|:((J+]:N!ZZZZ\\(N  ( ( ( ? ? ? ?  " "#L M M M#N44NQTUUU U   $#r)rr rr)rr&)rr)r2rr3rr4rrr&)r7r:r8r:r9r:r r r r rr rr;rr) __name__ __module__ __qualname____doc__rr+r-r1r6 classmethodrS __classcell__)r%s@rrr*s))V      RRRR QQQQ  "& $$(#'D$D$D$D$D$[D$D$D$D$D$rr)FFF)r r r r rr rr) __future__rloggingtypingrrrlangchain_core.callbacksrlangchain_core.utilsrr whylogs.api.logger.loggerr getLoggerrTr!rrr0rrras""""""//////////888888;;;;;;;;1000000%G%h//6Q$Q$Q$Q$Q$0Q$Q$Q$Q$Q$r