Ngg$ddlmZddlmZmZmZmZddlmZddl m Z ddl m Z ddl mZmZddlmZGdd eeZGd d eZGd d e ZdS))Enum)AnyIteratorListOptional)CallbackManagerForLLMRun)LLM)GenerationChunk) BaseModel ConfigDict)enforce_stop_tokensceZdZdZdZdZdS)Devicez,The device to use for inference, cuda or cpucudacpuN)__name__ __module__ __qualname____doc__rrb/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/llms/titan_takeoff.pyrr s66 D CCCrrceZdZUdZedZeed< ej Z eed< dZ eed< dZ e eed < d Zeed < d Zeed <dS) ReaderConfigzAConfiguration for the reader to be deployed in Titan Takeoff API.r)protected_namespaces model_namedeviceprimaryconsumer_groupNtensor_parallelimax_seq_lengthmax_batch_size)rrrrr model_configstr__annotations__rrrrr rintr!r#rrrrrsKK:LOOO&F   6#NC###5%)OXc])))INCKNC@@rrc VeZdZUdZdZeed< dZeed< dZ eed< dZ e ed < d Z e ed < ddddgfdededed e d eef fd ZedefdZ ddedeeedeede def dZ ddedeeedeede deef dZxZS) TitanTakeoffaTitan Takeoff API LLMs. Titan Takeoff is a wrapper to interface with Takeoff Inference API for generative text to text language models. You can use this wrapper to send requests to a generative language model and to deploy readers with Takeoff. Examples: This is an example how to deploy a generative language model and send requests. .. code-block:: python # Import the TitanTakeoff class from community package import time from langchain_community.llms import TitanTakeoff # Specify the embedding reader you'd like to deploy reader_1 = { "model_name": "TheBloke/Llama-2-7b-Chat-AWQ", "device": "cuda", "tensor_parallel": 1, "consumer_group": "llama" } # For every reader you pass into models arg Takeoff will spin # up a reader according to the specs you provide. If you don't # specify the arg no models are spun up and it assumes you have # already done this separately. llm = TitanTakeoff(models=[reader_1]) # Wait for the reader to be deployed, time needed depends on the # model size and your internet speed time.sleep(60) # Returns the query, ie a List[float], sent to `llama` consumer group # where we just spun up the Llama 7B model print(embed.invoke( "Where can I see football?", consumer_group="llama" )) # You can also send generation parameters to the model, any of the # following can be passed in as kwargs: # https://docs.titanml.co/docs/next/apis/Takeoff%20inference_REST_API/generate#request # for instance: print(embed.invoke( "Where can I see football?", consumer_group="llama", max_new_tokens=100 )) zhttp://localhostbase_urli porti mgmt_portF streamingNclientmodelsct|||| ddlm}n#t$rt dwxYw||j|j|j|_|D]}|j |dS)aInitialize the Titan Takeoff language wrapper. Args: base_url (str, optional): The base URL where the Takeoff Inference Server is listening. Defaults to `http://localhost`. port (int, optional): What port is Takeoff Inference API listening on. Defaults to 3000. mgmt_port (int, optional): What port is Takeoff Management API listening on. Defaults to 3001. streaming (bool, optional): Whether you want to by default use the generate_stream endpoint over generate to stream responses. Defaults to False. In reality, this is not significantly different as the streamed response is buffered and returned similar to the non-streamed response, but the run manager is applied per token generated. models (List[ReaderConfig], optional): Any readers you'd like to spin up on. Defaults to []. Raises: ImportError: If you haven't installed takeoff-client, you will get an ImportError. To remedy run `pip install 'takeoff-client==0.4.0'` )r*r+r,r-r) TakeoffClientzjtakeoff-client is required for TitanTakeoff. Please install it with `pip install 'takeoff-client>=0.4.0'`.)r+r,N) super__init__takeoff_clientr1 ImportErrorr*r+r,r. create_reader) selfr*r+r,r-r/r1model __class__s rr3zTitanTakeoff.__init__os< DI      4 4 4 4 4 4 4   P   $m M T^     - -E K % %e , , , , - -s /A returncdS)zReturn type of llm. titan_takeoffr)r7s r _llm_typezTitanTakeoff._llm_types rpromptstop run_managerkwargsc |jr)d}||||D] }||jz } |S|jj|fi|}|d}|t ||}|S)aCall out to Titan Takeoff (Pro) generate endpoint. Args: prompt: The prompt to pass into the model. stop: Optional list of stop words to use when generating. run_manager: Optional callback manager to use when streaming. Returns: The string generated by the model. Example: .. code-block:: python model = TitanTakeoff() prompt = "What is the capital of the United Kingdom?" # Use of model(prompt), ie `__call__` was deprecated in LangChain 0.1.7, # use model.invoke(prompt) instead. response = model.invoke(prompt) )r>r?r@text)r-_streamrDr.generater ) r7r>r?r@rA text_outputchunkresponserDs r_callzTitanTakeoff._calls: > K'& * * uz)  '4;'99&99  &tT22D rc+.K|jj|fi|}d}|D]}||jz }d|vr|drd}t |dddkr.|dd\}} |d}|r3t|} d}|r|| j | V|rGt| dd} |r|| j | Vd Sd S) aCall out to Titan Takeoff (Pro) stream endpoint. Args: prompt: The prompt to pass into the model. stop: Optional list of stop words to use when generating. run_manager: Optional callback manager to use when streaming. Yields: A dictionary like object containing a string token. Example: .. code-block:: python model = TitanTakeoff() prompt = "What is the capital of the United Kingdom?" response = model.stream(prompt) # OR model = TitanTakeoff(streaming=True) response = model.invoke(prompt) rCzdata: )rD)tokenzN) r.generate_streamdata startswithlensplitrstripr on_llm_new_tokenrDreplace) r7r>r?r@rArIbufferrDcontent_rHs rrEzTitanTakeoff._streams]@/4;.v@@@@  D di F&  $$W-- Fv||GQ//00A55!'gq!9!9JGQ$^^D11F +888EF"G#445:4FFFKKK  #)C)CDDDE ?,,5:,>>>KKKKK   r)NN)rrrrr*r%r&r+r'r,r-boolr.rrrr3propertyr=rrrJrr rE __classcell__)r9s@rr)r)-s00d'Hc&&&WD#EIsPIt8FCE+%' ,-,-,-,- ,-  ,- \" ,-,-,-,-,-,-\3X%):> ,,,tCy!,67 ,  , ,,,,b%):> 888tCy!867 8  8 / " 88888888rr)N)enumrtypingrrrrlangchain_core.callbacksr#langchain_core.language_models.llmsr langchain_core.outputsr pydanticr r langchain_community.llms.utilsr r%rrr)rrrres7000000000000======333333222222********>>>>>>S$AAAAA9AAA4[[[[[3[[[[[r