Ng(ddlmZddlZddlmZddlmZmZmZm Z m Z ddl Z ddl m Z ddlmZddlmZddlmZdd lmZGd d eZdS) ) annotationsN)StringIO)AnyDictIteratorListOptional)CallbackManagerForLLMRun)LLM)GenerationChunk)get_pydantic_field_names) ConfigDictceZdZUdZdZded< dZded< dZd ed < d Zd ed < dZ ded< dZ d ed< dZ ded< dZ ded< d Z d ed< dZd ed< dZded< dZded< dZded< dZd ed < d!Zd ed"< d#Zded$< d#Zded%< dZd ed&< d'Zded(< d)Zded*< ed+,Zed@d.ZedAd0ZedBd2Z dCdDd7Z dEdFd;Z dEdGd=Z!dHd?Z"dS)I LlamafileaOLlamafile lets you distribute and run large language models with a single file. To get started, see: https://github.com/Mozilla-Ocho/llamafile To use this class, you will need to first: 1. Download a llamafile. 2. Make the downloaded file executable: `chmod +x path/to/model.llamafile` 3. Start the llamafile in server mode: `./path/to/model.llamafile --server --nobrowser` Example: .. code-block:: python from langchain_community.llms import Llamafile llm = Llamafile() llm.invoke("Tell me a joke.") zhttp://localhost:8080strbase_urlNz Optional[int]request_timeoutFbool streamingintseedg?float temperature(top_kgffffff?top_pg?min_p n_predictrn_keepg?tfs_z typical_pg?repeat_penalty@ repeat_last_nT penalize_nlgpresence_penaltyfrequency_penaltymirostatg@ mirostat_taug? mirostat_etaforbid)extrareturncdS)N llamafile)selfs ^/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/llms/llamafile.py _llm_typezLlamafile._llm_typexs{ List[str]cPgdfdt|jD}|S)N) rcachecallback_manager callbacksmetadatanamerrtagsverbosecustom_get_token_idscg|]}|v| Sr1r1).0k ignore_keyss r3 z/Llamafile._param_fieldnames..s*   1KCWCWACWCWCWr5)r __class__)r2attrsrCs @r3_param_fieldnameszLlamafile._param_fieldnames|sM        /??    r5Dict[str, Any]cDi}|jD]}t||||<|SN)rGgetattr)r2params fieldnames r3_default_paramszLlamafile._default_paramss5/ 9 9I 'i 8 8F9   r5stopOptional[List[str]]kwargsrc |j}|D]\}}||vr|||<|t|dkr||d<|jrd|d<|S)NrrOTstream)rNitemslenr)r2rOrQrLrBvs r3_get_parameterszLlamafile._get_parameterssr% LLNN  DAqF{{q  D A !F6N > $#F8  r5prompt run_manager"Optional[CallbackManagerForLLMRun]c P|jrjt5}|j|f||d|D]}||j|}dddn #1swxYwY|S|jdd|i|}d|i|} tj|j dddi| d|j } n@#tj j $r)tj d |j d wxYw| d | _| d }|S)aRequest prompt completion from the llamafile server and return the output. Args: prompt: The prompt to use for generation. stop: A list of strings to stop generation when encountered. run_manager: **kwargs: Any additional options to pass as part of the generation request. Returns: The string generated by the model. )rOrYNrOrX /completion Content-Typeapplication/jsonFurlheadersjsonrStimeoutTCould not connect to Llamafile server. Please make sure that a server is running at .zutf-8contentr1)rr_streamwritetextgetvaluerWrequestspostrr exceptionsConnectionErrorraise_for_statusencodingrb) r2rXrOrYrQbuffchunkrirLpayloadresponses r3_callzLlamafile._calls, >$  't)T\!%;BH++EJJuz****}}  ' ' ' ' ' ' ' ' ' ' ' ' ' ' 'K*T)>>t>v>>F262G #==555&(:!  0&6   )99D37=DDD   % % ' ' ' 'H ==??9-DKsAA%%A),A)(B11=C.Iterator[GenerationChunk]c+K|jdd|i|}d|vrd|d<d|i|} tj|jdddi|d|j}n@#tjj$r)tjd |jd wxYwd |_|d D]H}| |} t| } |r| | j | VIdS)a"Yields results objects as they are generated in real time. It also calls the callback manager's on_llm_new_token event with similar parameters to the OpenAI LLM class method of the same name. Args: prompt: The prompts to pass into the model. stop: Optional list of stop words to use when generating. run_manager: **kwargs: Any additional options to pass as part of the generation request. Returns: A generator representing the stream of tokens being generated. Yields: Dictionary-like objects each containing a token Example: .. code-block:: python from langchain_community.llms import Llamafile llm = Llamafile( temperature = 0.0 ) for chunk in llm.stream("Ask 'Hi, how are you?' like a pirate:'", stop=["'"," "]): result = chunk["choices"][0] print(result["text"], end='', flush=True) rOrSTrXr\r]r^r_rdreutf8)decode_unicode)ri)tokenNr1) rWrkrlrrrmrnrp iter_lines_get_chunk_contentr on_llm_new_tokenri) r2rXrOrYrQrLrsrt raw_chunkrfrrs r3rgzLlamafile._streamslL&%::4:6:: 6 ! !#F8 V.v. }}111"$6,HH"2   %55@/3}@@@   #!,,D,AA  I--i88G#111E ?,,5:,>>>KKKK   s (A =Brrc|dr1|d}tj|}|dS|S)zWhen streaming is turned on, llamafile server returns lines like: 'data: {"content":" They","multimodal":true,"slot_id":0,"stop":false}' Here, we convert this to a dict and return the value of the 'content' field zdata:zdata: rf) startswithlstriprbloads)r2rrcleaneddatas r3r|zLlamafile._get_chunk_content1sI   G $ $ ll8,,G:g&&D ? "Lr5)r.r)r.r6)r.rHrJ)rOrPrQrr.rH)NN) rXrrOrPrYrZrQrr.r) rXrrOrPrYrZrQrr.rv)rrrr.r)#__name__ __module__ __qualname____doc__r__annotations__rrrrrrrrr r!r"r#r%r&r'r(r)r*r+r model_configpropertyr4rGrNrWrurgr|r1r5r3rrs*,H++++;%)O))))%IJ DNNNN#K#EOOOOE7E@I< FOOOO ENI  NM$K"!!!!A"""""AHLGLF:LXX.X+/.%):> :::::~%):> DDDDDLr5r) __future__rrbiortypingrrrrr rk langchain_core.callbacks.managerr #langchain_core.language_models.llmsr langchain_core.outputsr langchain_core.utilsr pydanticrrr1r5r3rs"""""" 66666666666666EEEEEE333333222222999999ppppppppppr5