g$ddlZddlmZerddlmZmZddlmZGddejj Z d ded ej d ej fd Z dS) N)is_torch_available)PreTrainedModel StaticCache)"is_torch_greater_or_equal_than_2_3c eZdZdZdeffd ZdejdejfdZe dej j dejd e d ejfd Z xZS) $TorchExportableModuleWithStaticCachea A wrapper module designed to make a `PreTrainedModel` exportable with `torch.export`, specifically for use with static caching. This module ensures that the exported model is compatible with further lowering and execution in `ExecuTorch`. Note: This class is specifically designed to support export process using `torch.export` in a way that ensures the model can be further lowered and run efficiently in `ExecuTorch`. modelct|jtd|jjstd|jjdkrtd||_t|jj|jjj j |jjj j |jj |_ td|jjjD|_|jrat#jt#j|j j |j j t"j}|d |d dSdS) a Initializes the wrapper module with the pretrained model. Args: model (`PreTrainedModel`): The pretrained model to wrap. The model must have caching enabled and use a 'static' caching implementation. Raises: AssertionError: If the pretrained model does not have caching enabled or if it does not use a 'static' caching implementation in `model.generation_config`. NzkThe model must have a generation config to be exported with static caching. Please set `generation_config`.zvThe model must have caching enabled to be exported with static caching. Please set `generation_config.use_cache=True`.staticzThe model must use a 'static' caching implementation to be exported with static caching. Please set `generation_config.cache_implementation='static'`.)config batch_size max_cache_lendtypec3K|]}d|vV dS)CausalLMN).0archs `/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/integrations/executorch.py z@TorchExportableModuleWithStaticCache.__init__..Ms'\\DZ4/\\\\\\rmaskF) persistent)super__init__generation_configAssertionError use_cachecache_implementationr rr cache_configrrr static_cacheany architectures is_causaltorchtrilonesboolregister_buffer)selfr causal_mask __class__s rrz-TorchExportableModuleWithStaticCache.__init__%sz   " * 2  &0  A   " 78 C C P   ':$z3@K*6CQ*"     \\DJ H* %3%3*K   G G G G G H Hr input_idscache_positionc|j\}}|jr|j|d|fnd}||||d||jd}|jS)a Forward pass of the module, which is compatible with the ExecuTorch runtime. Args: input_ids (`torch.Tensor`): Tensor representing current input token id to the module. cache_position (`torch.Tensor`): Tensor representing current input position in the cache. Returns: torch.Tensor: Logits output from the model. This forward adapter serves two primary purposes: 1. **Making the Model `torch.export`-Compatible**: The adapter hides unsupported objects, such as the `Cache`, from the graph inputs and outputs, enabling the model to be exportable using `torch.export` without encountering issues. 2. **Ensuring Compatibility with `ExecuTorch` runtime**: The adapter matches the model's forward signature with that in `executorch/extension/llm/runner`, ensuring that the exported model can be executed in `ExecuTorch` out-of-the-box. NrT)r/attention_mask position_idsr0past_key_valuesr )shaper&rr unsqueezer#logits)r,r/r0_seqlen attn_maskoutss rforwardz,TorchExportableModuleWithStaticCache.forwardXsu*O 6:>.RDIngvg566d zz$'11!44) -   {rexported_programprompt_token_idsmax_new_tokensreturnc|jd}||z}|D]9\}}|dr|jd}t||}n:g}t t||D]} ||dd| | dzftj| gtj } | |d|  tj | dddddfd  } | | t||kr|tj| ggtj tjt|gtj } tj | dddddfd  } | | t||ktj|gtj S) a Generate a sequence of tokens using an exported program. This util function is designed to test exported models by simulating the generation process. It processes the input prompt tokens sequentially (no parallel prefill). This generate function is not intended to replace the original `generate` method, and the support for leveraging the original `generate` is potentially planed! Args: exported_program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. prompt_token_ids (`torch.Tensor`): Tensor representing the input prompt token IDs. max_new_tokens (`int`): Maximum number of new tokens to generate. Note that the total generation length is limited by both `max_new_tokens` and the model's cache size. Returns: torch.Tensor: A tensor containing the generated sequence of token IDs, including the original prompt tokens. zstatic_cache.key_cacherNr)r/r0r)dim)r5 named_buffers startswithminrangemoduler<r'tensorlongappenditemargmaxlen) r=r>r?prompt_token_lenmax_generation_length buffer_namebufferrresponse_tokens input_posresult current_tokens rgeneratez-TorchExportableModuleWithStaticCache.generateys`*,1"5 0> A#3#A#A#C#C   K%%&>??  & Q (+,A=(Q(Q%  s#8:JKKLL J JI%,,..66*111i)a-.G+GH$|YKuzJJJ7F  " "#3A#6y#A#F#F#H#H I I I I VAAAr111H%52>>>CCEE }---/""%:::%,,..66,'8 KKK$|S-A-A,B%*UUU7F"L2qqq)9rBBBGGIIM  " "= 1 1 1 /""%:::|_-UZ@@@@r)__name__ __module__ __qualname____doc__rrr'Tensorr< staticmethodexportExportedProgramintrX __classcell__)r.s@rr r s1Ho1H1H1H1H1H1Hfu|B/A,6/AJO,/Ahk/A /A/A/A\/A/A/A/A/Arr r example_input_idsexample_cache_positioncftstdddl}|j5||n|jdgg|j}||n|jdg|j}|jjt||fd|idd }|cdddS#1swxYwYdS) a6 Convert a `PreTrainedModel` into an exportable module and export it using `torch.export`, ensuring the exported model is compatible with `ExecuTorch`. Args: model (`PreTrainedModel`): The pretrained model to be exported. example_input_ids (`torch.Tensor`): Example input token id used by `torch.export`. example_cache_position (`torch.Tensor`): Example current cache position used by `torch.export`. Returns: Exported program (`torch.export.ExportedProgram`): The exported program generated via `torch.export`. ztorch >= 2.3 is required.rNrCrr0FT)argskwargs pre_dispatchstrict) r ImportErrortorch.export._traceno_gradrJrKr__trace_exportr )r rcrdr'r=s rconvert_and_export_with_cacheros?$ .75666   "3!>  LELSTRUQV^c^hDiDiDi '=&H " "lel\][^fkfpNqNqNq  !<.66 0 7 7#%$&<= 7   %                  sA/B&&B*-B*)NN) r'utils.import_utilsr transformersrrtransformers.pytorch_utilsrnnModuler r]rorrrrus 333333NNMMMMMOAOAOAOAOA58?OAOAOAh'++/) ) ) |) "L) ) ) ) ) ) r