gm< dZddlmZmZmZddlZddlmZddl m Z ddl m Z m Z mZmZddlmZmZd d lmZGd d e d ZGdde d ZdeededeeefdZdeeeedeeedededejf dZdedededefdZGddeZdS) zProcessor class for Mllama.)ListOptionalUnionN) BatchFeature) ImageInput) ImagesKwargsProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)make_list_of_imagesc&eZdZUeeed<dS)MllamaImagesKwargsmax_image_tilesN)__name__ __module__ __qualname__rint__annotations__h/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/mllama/processing_mllama.pyrr"s"c]"""""rrF)totalc&eZdZUeed<dddiiZdS)MllamaProcessorKwargs images_kwargs image_kwargsrN)rrrrr _defaultsrrrrr&s2%%%%  q IIIrr input_idsimage_token_idreturncfdt|D}t|dkrgSt|dkr |ddggSdt|dd|ddD}||dt|g|dd}|dddD]$}|d|ddz kr||d<|d}%|S)a Generate a cross-attention token mask for image tokens in the input sequence. This function identifies the positions of image tokens in the input sequence and creates a mask that defines which subsequent tokens each image token should attend to. Args: input_ids (List[int]): A list of token ids representing the input sequence. image_token_id (int): The id of the token used to represent images in the sequence. Returns: List[List[int]]: A list of [start, end] pairs, where each pair represents the range of tokens an image token should attend to. Notes: - If no image tokens are present, an empty list is returned. - For a single image token, it attends to all subsequent tokens until the end of the sequence. - For multiple image tokens, each attends to tokens up to the next image token or the end of the sequence. - Consecutive image tokens are treated as a group and attend to all subsequent tokens together. c&g|] \}}|k |Srr).0itokenr$s r z2get_cross_attention_token_mask..Fs(___81euP^G^G^QG^G^G^rrrcg|] \}}||g Srr)r(loc1loc2s rr+z2get_cross_attention_token_mask..Os nnnZT4T4LnnnrN) enumeratelenzipappend)r#r$image_token_locations vision_masks last_mask_end vision_masks ` rget_cross_attention_token_maskr80s",`___y/C/C___ !!Q&&  !!Q&&&q)2.//nn37LSbS7QShijikikSl3m3mnnnL.r2C NNCDDD !$Q'M#DDbD)'' q>[^a/ / /*KN#A rcross_attention_token_mask num_tiles max_num_tileslengthc t|}td|D}tj||||ftj}t t ||D]k\}\}} t t || D]E\} \} } t| dkr*| \} }t||}|dkr|}d||| || d| f<Fl|S)a Convert the cross attention mask indices to a cross attention mask 4D array. This function takes a sparse representation of cross attention masks and converts it to a dense 4D numpy array. The sparse representation is a nested list structure that defines attention ranges for each image in each batch item. Args: cross_attention_token_mask (List[List[List[int]]]): A nested list structure where: - The outer list represents the batch dimension. - The middle list represents different images within each batch item. - The inner list contains pairs of integers [start, end] representing token ranges for each image. num_tiles (List[List[int]]): A nested list structure specifying the number of tiles for each image in each batch item. max_num_tiles (int): The maximum possible number of tiles. length (int): The total sequence length of the input. Returns: np.ndarray: A 4D numpy array of shape (batch_size, length, max_num_images, max_num_tiles) The array contains `1` where attention is allowed and `0` where it is not. Note: - Special handling is done for cases where the end token is -1, which is interpreted as attending to the end of the sequence. c,g|]}t|Srr1)r(maskss rr+z@convert_sparse_cross_attention_mask_to_dense..~sMMM#e**MMMr)shapedtyper,rN)r1maxnpzerosint64r0r2min)r9r:r;r< batch_sizemax_num_imagescross_attention_mask sample_idx sample_maskssample_num_tilesmask_idx locationsmask_num_tilesstartends r,convert_sparse_cross_attention_mask_to_denserT`s:/00JMM2LMMMNNN86>=Ah 9B#F`bkBlBl8m8m[[4 4\#35>s>> build_string_from_input("Hello world", "", "<|image|>") 'Hello world' >>> build_string_from_input("<|image|>Hello world", "", "<|image|>") '<|image|>Hello world' >>> build_string_from_input("Hello world", "", "<|image|>") 'Hello world' rNr) startswithr1)rUrVrWnum_image_tokens_on_starts rbuild_string_from_inputr[s4F !   K ( ('K((**+!Q&!   K ( ('55 Jy J& J JJrc eZdZdZddgZdZdZfdZ ddee d ee e e e e e e fd eed efd Zd ZdZedZxZS)MllamaProcessoraw Constructs a Mllama processor which wraps [`MllamaImageProcessor`] and [`PretrainedTokenizerFast`] into a single processor that inherits both the image processor and tokenizer functionalities. See the [`~MllamaProcessor.__call__`] and [`~OwlViTProcessor.decode`] for more information. The preferred way of passing kwargs is as a dictionary per modality, see usage example below. ```python from transformers import MllamaProcessor from PIL import Image processor = MllamaProcessor.from_pretrained("meta-llama/Llama-3.2-11B-Vision") processor( images=your_pil_image, text=["<|image|>If I had to write a haiku for this one"], images_kwargs = {"size": {"height": 448, "width": 448}}, text_kwargs = {"padding": "right"}, common_kwargs = {"return_tensors": "pt"}, ) ``` Args: image_processor ([`MllamaImageProcessor`]): The image processor is a required input. tokenizer ([`PreTrainedTokenizer`, `PreTrainedTokenizerFast`]): The tokenizer is a required input. image_processor tokenizerMllamaImageProcessorPreTrainedTokenizerFastcd|_||j|_d|_||j|_|j|_|j|_t||dS)Nz <|image|>z<|python_tag|>) rWconvert_tokens_to_idsr$ python_tokenpython_token_idrV chat_templatesuper__init__)selfr^r_ __class__s rrhzMllamaProcessor.__init__sz&'==d>NOO,(>>t?PQQ",&4 )44444rNimagestextkwargsr%c ||tdjtfdjji|}|d}|d}|d} i} |t |t r|g}nDt |ttfrtd|Dstdfd |D} fd |D}| d d} j|fi|} | | d g}|t|}d |D}|td| Dr(td| Dstdt|t| krA|tdtdt| dt|d|8j|fi|}| d}| ||U|Sfd| dD}t!||jjt%d| dD}|| d<| dd}t'| |}|S)a& Main method to prepare text(s) and image(s) to be fed as input to the model. This method forwards the `text` arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizerFast.__call__`] if `text` is not `None` to encode the text. To prepare the image(s), this method forwards the `images` arguments to MllamaImageProcessor's [`~MllamaImageProcessor.__call__`] if `images` is not `None`. Please refer to the docstring of the above two methods for more information. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. Both channels-first and channels-last formats are supported. text (`str`, `List[str]`, `List[List[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. TODO: add aspect_ratio_ids and aspect_ratio_mask and cross_attention_mask Nz'You must specify either text or images.tokenizer_init_kwargs text_kwargsr common_kwargsc3@K|]}t|tVdSN) isinstancestr)r(ts r z+MllamaProcessor.__call__..s-=_=_UVjC>P>P=_=_=_=_=_=_rzAInvalid input text. Please provide a string, or a list of stringscDg|]}|jSr)countrW)r(rvris rr+z,MllamaProcessor.__call__..s(HHHa(8 9 9HHHrcFg|]}t|jjSr)r[rVrW)r( text_itemris rr+z,MllamaProcessor.__call__..s,ooo]f+It~tGWXXooor padding_siderc,g|]}t|Srr?)r(samples rr+z,MllamaProcessor.__call__..#s!C!C!C&#f++!C!C!Crc3"K|] }|dkV dSrNrr( batch_imgs rrwz+MllamaProcessor.__call__..&s&DDi9>DDDDDDrc3"K|] }|dkV dSrrrs rrwz+MllamaProcessor.__call__..&s?QQ#, QQQQQQQrzaIf a batch of text is provided, there should be either no images or at least one image per samplez@No image were provided, but there are image tokens in the promptzThe number of image token (z:) should be the same as in the number of provided images ()r:c:g|]}t|jSr)r8r$)r( token_idsris rr+z,MllamaProcessor.__call__..;s4***S\.y$:MNN***rr#c34K|]}t|VdSrsr?)r(r#s rrwz+MllamaProcessor.__call__..Bs(QQi3y>>QQQQQQr)r:r;r<rKreturn_tensors)data tensor_type) ValueError _merge_kwargsrr_ init_kwargsrtrulisttupleallpopupdateranysumr^rTrrDr)rirkrlaudiovideosrm output_kwargsrprrqrn_images_in_text_encodingn_images_in_imagesimage_featuresr:r9rKr batch_features` r__call__zMllamaProcessor.__call__sN *?*???>$%ghhh$bc:J6K6KbbHKL^H_H_bbb  1T1&JJMJJN&**;77I KK ' ' '  $"2****`hit`u*** &$P*#"2BQQ8K;PQQQQQ $$$ ,@D' (&**+;TBB$$NKKK rc&|jj|i|S)z This method forwards all its arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizer.batch_decode`]. Please refer to the docstring of this method for more information. )r_ batch_decoderiargsrms rrzMllamaProcessor.batch_decodeKs +t~*D;F;;;rc&|jj|i|S)z This method forwards all its arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizer.decode`]. Please refer to the docstring of this method for more information. )r_decoders rrzMllamaProcessor.decodeRs %t~$d5f555rc^|jj}|jj}t||zdgzS)NrK)r_model_input_namesr^r)ritokenizer_input_namesimage_processor_input_namess rrz!MllamaProcessor.model_input_namesYs7 $ @&*&:&L#),GGKaJbbcccr)NNNN)rrr__doc__ attributesimage_processor_classtokenizer_classrhrrrrr rr rrrrrpropertyr __classcell__)rjs@rr]r]s:$[1J2/O55555(,hl ii$iuY(94 ?DQbLccdei ./ i iiiiV<<<666ddXdddddrr])rtypingrrrnumpyrEfeature_extraction_utilsr image_utilsrprocessing_utilsr r r r tokenization_utils_baser rimage_processing_mllamarrrrr8ndarrayrTrur[r]rrrrs4 "!((((((((((444444%%%%%%VVVVVVVVVVVV 988888#####U####,E-d3i--QUVZ[^V_Q`----`- $T$s)_ 5- DI- -   - Z - - - - `"KC"KC"Kc"Kc"K"K"K"KJhdhdhdhdhdnhdhdhdhdhdr