Ng"~ddlZddlmZmZmZmZddlmZddlm Z ddl m Z ddl m Z dZdZd ZGd d e ZdS) N)AnyListMappingOptional)CallbackManagerForLLMRun)LLM) ConfigDict)enforce_stop_tokenszgoogle/flan-t5-largetext2text-generation)r text-generation summarizationceZdZUdZdZeed<eZe ed< dZ e e ed< dZ e e ed< edZe dde d e d e ed e e de e de e de ede ede ededefdZedee effdZede fdZ dde de ee de edede f dZdS)WeightOnlyQuantPipelineaWeight only quantized model. To use, you should have the `intel-extension-for-transformers` packabge and `transformers` package installed. intel-extension-for-transformers: https://github.com/intel/intel-extension-for-transformers Example using from_model_id: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline from intel_extension_for_transformers.transformers import ( WeightOnlyQuantConfig ) config = WeightOnlyQuantConfig hf = WeightOnlyQuantPipeline.from_model_id( model_id="google/flan-t5-large", task="text2text-generation" pipeline_kwargs={"max_new_tokens": 10}, quantization_config=config, ) Example passing pipeline in directly: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline from intel_extension_for_transformers.transformers import ( AutoModelForSeq2SeqLM ) from intel_extension_for_transformers.transformers import ( WeightOnlyQuantConfig ) from transformers import AutoTokenizer, pipeline model_id = "google/flan-t5-large" tokenizer = AutoTokenizer.from_pretrained(model_id) config = WeightOnlyQuantConfig model = AutoModelForSeq2SeqLM.from_pretrained( model_id, quantization_config=config, ) pipe = pipeline( "text-generation", model=model, tokenizer=tokenizer, max_new_tokens=10, ) hf = WeightOnlyQuantPipeline(pipeline=pipe) Npipelinemodel_id model_kwargspipeline_kwargsallow)extraFtaskdevice device_map load_in_4bit load_in_8bitquantization_configkwargsreturnc |*t|tr|dkrtdtjdtd ddlm} m} ddl m } dd l m }dd l m }n#t$rtd wxYwt|tr2|dkr,| std d t|z}nt|tr|dkrd}||d}|pi}|j|fi|} |dkr| j|f||| d|d|}n4|dvr| j|f||| d|d|}ntd|dt"dn&#t$r}td|d|d}~wwxYwd|vrd|D}|pi}|d|||||d|}|jt"vr td|jdt"d|d||||d| S)z5Construct the pipeline object from model_id and task.Nrz7`Device` and `device_map` cannot be set simultaneously!torchz;Weight only quantization pipeline only support PyTorch now!r)AutoModelForCausalLMAutoModelForSeq2SeqLM)is_ipex_available) AutoTokenizer)rzCould not import transformers python package. Please install it with `pip install transformers` and `pip install intel-extension-for-transformers`.z)Don't find out Intel GPU on this machine!zxpu:cpur F)rrruse_llm_runtimer)r r Got invalid task , currently only  are supportedzCould not load the z# model due to missing dependencies.trust_remote_codec&i|]\}}|dk ||S)r*).0kvs m/var/www/html/ai-engine/env/lib/python3.11/site-packages/langchain_community/llms/weight_only_quantization.py z9WeightOnlyQuantPipeline.from_model_id..s0A!?R:R:R1:R:R:R)rmodel tokenizerrr)rrrrr,) isinstanceint ValueError importlibutil find_spec-intel_extension_for_transformers.transformersr!r",intel_extension_for_transformers.utils.utilsr# transformersr$r ImportErrorstrfrom_pretrained VALID_TASKSitemsr)clsrrrrrrrrrrr!r"r#r$ hf_pipeline _model_kwargsr4r3e_pipeline_kwargsrs r0 from_model_idz%WeightOnlyQuantPipeline.from_model_idOs  !z&#'>'> !6B;;VWW W > # #G , , 4M           W V V V V V 2 2 2 2 2 2 < < < < < < <   F   fc " " v{{$$&& N !LMMM#f++-JJ  $ $ !F >!" $* 1M1(LLmLL  (((<,<!-!-(;$)) $BBB=-=!-!-(;$)) $!BBB&1BBB   OdOOO   - / /!.!4!4!6!6M+0b; &       = + +>HM>>"->>> s &,       s%A77BAE!! F+E??Fc,|j|j|jdS)zGet the identifying parameters.rrrrJselfs r0_identifying_paramsz+WeightOnlyQuantPipeline._identifying_paramss$  -#3   r2cdS)zReturn type of llm.weight_only_quantizationr,rKs r0 _llm_typez!WeightOnlyQuantPipeline._llm_types *)r2promptstop run_managerc ||}|jjdkr$|ddt|d}nc|jjdkr|dd}nD|jjdkr|dd}n%td|jjd td |rt ||}|S) abCall the HuggingFace model and return the output. Args: prompt: The prompt to use for generation. stop: A list of strings to stop generation when encountered. Returns: The generated text. Example: .. code-block:: python from langchain_community.llms import WeightOnlyQuantPipeline llm = WeightOnlyQuantPipeline.from_model_id( model_id="google/flan-t5-large", task="text2text-generation", ) llm.invoke("This is a prompt.") r rgenerated_textNr r summary_textr'r(r))rrlenr7rAr )rLrQrRrSrresponsetexts r0_callzWeightOnlyQuantPipeline._calls4==(( = !2 2 2A;/0V?DD ] #9 9 9A;/0DD ] ? 2 2A;~.DD>DM$6>>"->>>   3'tT22D r2)rNNNFFN)NN)__name__ __module__ __qualname____doc__rr__annotations__DEFAULT_MODEL_IDrr?rrdictrr model_config classmethodr6boolrrHpropertyrrMrPrrrZr,r2r0rrs//bHc$Hc$$$*#'L(4.'''1&*OXd^***4:L !#$('+*.',',-1h h h h  h SM h tn h "$h tnh tnh &c]h h  h h h [h T WS#X%6   X *3***X*%):> +++tCy!+67 +  + ++++++r2r)r8typingrrrr langchain_core.callbacks.managerr#langchain_core.language_models.llmsrpydanticr langchain_community.llms.utilsr r` DEFAULT_TASKrArr,r2r0rls////////////EEEEEE333333>>>>>>)% J dddddcdddddr2