Ng'ddlmZmZmZmZmZmZmZmZddl m Z ddl m Z erddl mZmZddlmZGdde ZdS) ) TYPE_CHECKINGAnyDict GeneratorListMappingOptionalUnion)CallbackManagerForLLMRun)LLM)RESTfulChatModelHandleRESTfulGenerateModelHandle)LlamaCppGenerateConfigceZdZUdZeed<eeed< eeed< eeefed< ddeedeedeffd Z e defd Z e de eeffd Z dd ed eeed eededef dZ ddedd ed eededdeeddff dZxZS) Xinferencea`Xinference` large-scale model inference service. To use, you should have the xinference library installed: .. code-block:: bash pip install "xinference[all]" If you're simply using the services provided by Xinference, you can utilize the xinference_client package: .. code-block:: bash pip install xinference_client Check out: https://github.com/xorbitsai/inference To run, you need to start a Xinference supervisor on one server and Xinference workers on the other servers Example: To start a local instance of Xinference, run .. code-block:: bash $ xinference You can also deploy Xinference in a distributed cluster. Here are the steps: Starting the supervisor: .. code-block:: bash $ xinference-supervisor Starting the worker: .. code-block:: bash $ xinference-worker Then, launch a model using command line interface (CLI). Example: .. code-block:: bash $ xinference launch -n orca -s 3 -q q4_0 It will return a model UID. Then, you can use Xinference with LangChain. Example: .. code-block:: python from langchain_community.llms import Xinference llm = Xinference( server_url="http://0.0.0.0:9997", model_uid = {model_uid} # replace model_uid with the model UID return from launching the model ) llm.invoke( prompt="Q: where can we visit in the capital of France? A:", generate_config={"max_tokens": 1024, "stream": True}, ) To view all the supported builtin models, run: .. code-block:: bash $ xinference list --all client server_url model_uid model_kwargsNc F ddlm}n:#t$r- ddlm}n"#t$r}td|d}~wwxYwYnwxYw|pi}t jdi|||d|jtd|jtd|||_ dS)Nr) RESTfulClientzCould not import RESTfulClient from xinference. Please install it with `pip install xinference` or `pip install xinference_client`.rrrzPlease provide server URLzPlease provide the model UID) xinference.clientr ImportErrorxinference_clientsuper__init__r ValueErrorrr)selfrrrre __class__s _/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/llms/xinference.pyrzXinference.__init__\s  7 7 7 7 7 7 7    ;;;;;;;   !Y <; $)r   (& ,    ? "899 9 > !;<< <#mJ// s) AA ;6;AAreturncdS)zReturn type of llm. xinferencerr s r# _llm_typezXinference._llm_types |c<id|jid|jid|jiS)zGet the identifying parameters.rrrrr's r#_identifying_paramszXinference._identifying_paramss< T_- DN+ t01  r)promptstop run_managerkwargsc ^|j|j}|di}i|j|}|r||d<|r:|dr%d}|||||D]}||z }|S|||} | ddd S) aqCall the xinference model and return the output. Args: prompt: The prompt to use for generation. stop: Optional list of stop words to use when generating. generate_config: Optional dictionary for the configuration used for generation. Returns: The generated string by the model. generate_configr-stream)modelr,r.r1r,r1choicesrtext)r get_modelrgetr_stream_generategenerate) r r,r-r.r/r4r1combined_text_outputtoken completions r#_callzXinference._calls$ %%dn554:JJ?PRT4U4UBT.B/B  +&*OF #  4228<< 4#% ..' / / . . %-$$' 'vWWJi(+F3 3r)r4)rr r1rc#rK|||}|D]}t|tr|dg}|rk|d}t|trN|dd} |d} |r|| |j| | VdS) a^ Args: prompt: The prompt to use for generation. model: The model used for generation. stop: Optional list of stop words to use when generating. generate_config: Optional dictionary for the configuration used for generation. Yields: A string token. r5r6rr7r3logprobs)r=verbose log_probsN)r; isinstancedictr9on_llm_new_tokenrB) r r4r,r.r1streaming_responsechunkr6choicer=rCs r#r:zXinference._stream_generates$#^^?,  ( $ $E%&& $))Ir22 $$QZF!&$//$ & 62 6 6$*JJz$:$: &'88&+T\Y9$  $ $r))NN)__name__ __module__ __qualname____doc__r__annotations__r strrrpropertyr(rr+rr r?r rr: __classcell__)r"s@r#rr sFFP KKK &}#sCx.   :%)#'!0!0SM!0C=!0 !0!0!0!0!0!0F3X WS#X%6   X %):> (4(4(4tCy!(467 (4  (4 (4(4(4(4\;?>B !$!$KL!$!$67 !$ "":; !$ 3d? # !$!$!$!$!$!$!$!$r)rN)typingrrrrrrr r langchain_core.callbacksr #langchain_core.language_models.llmsr rr rxinference.model.llm.corerrrr)r#rVsVVVVVVVVVVVVVVVVVVVV======333333ATTTTTTTT@@@@@@M$M$M$M$M$M$M$M$M$M$r)