g<dZddlmZmZmZddlmZddlmZm Z ddl m Z m Z m Z mZmZmZddlmZmZmZddlmZejeZd Zd ed Dd ed DzZGddeZGdde ZGdde dZde fdZ!dZ"dZ#dZ$deeefdZ%Gdde Z&dS)z Processor class for PaliGemma. )ListOptionalUnion) BatchFeature) ImageInputis_valid_image) ImagesKwargsProcessingKwargsProcessorMixin TextKwargsUnpack!_validate_images_text_input_order) AddedTokenPreTokenizedInput TextInput)loggingzcg|] }d|dd S)z4>.0is n/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/paligemma/processing_paligemma.py r*s"555Aq555icg|] }d|dd S)z3rrrs rrr*s"8]8]8]Q8]8]8]rcReZdZUeeeeeeeefed<dS)PaliGemmaTextKwargssuffixN) __name__ __module__ __qualname__rrrrr__annotations__rrrr r -s< U9&7i$O`Jaab ccccccrr c&eZdZUeeed<dS)PaliGemmaImagesKwargsdo_convert_rgbN)r"r#r$rboolr%rrrr'r'1s"TN"""""rr'c6eZdZUeed<eed<ddiddidZdS) PaliGemmaProcessorKwargs text_kwargs images_kwargspaddingF data_formatchannels_first)r,r-N)r"r#r$r r%r' _defaultsrrrr+r+5sM$$$$(((( u  + IIIrr+F)totalreturncVt|to|dS)Nhttp) isinstancestr startswith)vals ris_urlr:Cs# c3   :CNN6$:$::rc>t|pt|SN)r:r elems ris_image_or_image_urlr?Hs $<< />$///rcJt|tpt|Sr<)r6r7r?r=s r_is_str_or_imagerALs dS " " A&;D&A&AArc ||z|z||dS)aZ Builds a string from the input prompt and image tokens. For example, for the call: build_string_from_input( prompt="Prefix str" bos_token="", image_seq_len=3, image_token="", ) The output will be: "Initial str" Args: prompt (`List[Union[str, ImageInput]]`): The input prompt. bos_token (`str`): The beginning of sentence token. image_seq_len (`int`): The length of the image sequence. image_token (`str`): The image token. num_images (`int`): Number of images in the prompt.  rprompt bos_token image_seq_len image_token num_imagess rbuild_string_from_inputrJPs'&M)J6 M M6 M M MMrczt|ttfrIt|dttfr't|ddr d|DSt|ttfrt|dr|St|r|gSt d|)a  Accepts images in list or nested list format, and makes a list of images for preprocessing. Args: images (`Union[List[List[ImageInput]], List[ImageInput], ImageInput]`): The input image. Returns: list: A list of images. rcg|] }|D]}| Srr)rimg_listimgs rrz'make_batched_images..ss%???h??s????rz"Could not make batched video from )r6listtupler ValueError)imagess rmake_batched_imagesrSgs&4-((Zq D%=-Q-QVdeklmenopeqVrVr??F???? FT5M * *~fQi/H/H   x B&BB C CCrc eZdZdZddgZdgZdZdZ dfd Z dd e d e e e e e e e fd eed efd ZdZdZedZxZS)PaliGemmaProcessora Constructs a PaliGemma processor which wraps a PaliGemma image processor and a PaliGemma tokenizer into a single processor. [`PaliGemmaProcessor`] offers all the functionalities of [`SiglipImageProcessor`] and [`LlamaTokenizerFast`]. See the [`~PaliGemmaProcessor.__call__`] and [`~PaliGemmaProcessor.decode`] for more information. Args: image_processor ([`SiglipImageProcessor`], *optional*): The image processor is a required input. tokenizer ([`LlamaTokenizerFast`], *optional*): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. image_processor tokenizer chat_templateSiglipImageProcessor)GemmaTokenizerGemmaTokenizerFastNc |td|tdt|dstd|j|_ttdd}d|gi}|||t|t|_ d|_ d|_ t ||| dS) Nz)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.image_seq_lengthz;Image processor is missing an `image_seq_length` attribute.FT) normalizedspecialadditional_special_tokens)rX)rQhasattrr]r IMAGE_TOKENadd_special_tokens add_tokens EXTRA_TOKENSconvert_tokens_to_idsimage_token_id add_bos_token add_eos_tokensuper__init__)selfrVrWrXkwargsrH tokens_to_add __class__s rrkzPaliGemmaProcessor.__init__s  "HII I  ABB B(:;; \Z[[ [ / @ MMM 4{mD $$]333\***'==kJJ"' "'  )=QQQQQrrRtextrmr3c t||\}}jtfdjji|}|ddd}|dnd}|t d|tdd }t|r|g}n+t|trt|d r ||td |Dst d t|tret|trPt|t|kr0t d t|dt|dt!|r|gg}nt|tr"t!|d r d|D}nZt|tr6t|d trt!|d d st d|t|r|g}|fd|D}fdt#||D} t%|}nfd|D}d|D} j|fi|dd} |ddd|ddxxjz cc<j| f||d|d} i| d| i} |r=| d| dd kd} | d| it1| S)ah Main method to prepare for the model one or several sequences(s) and image(s). This method forwards the `text` and `kwargs` arguments to LlamaTokenizerFast's [`~LlamaTokenizerFast.__call__`] if `text` is not `None` to encode the text. To prepare the image(s), this method forwards the `images` and `kwrags` arguments to SiglipImageProcessor's [`~SiglipImageProcessor.__call__`] if `images` is not `None`. Please refer to the doctsring of the above two methods for more information. The usage for PaliGemma fine-tuning preparation is slightly different than usual. suffix passed are suffixes to the prompt in `text`, and will be placed after the prompt. This is because attention is handled differently for the prefix and the suffix. For instance, ```python image = PIL_cow_image prompt = "answer en Where is the cow standing?" suffix = "on the beach" inputs = processor(text=prompt, images=image, suffix=suffix) ``` Here `inputs` will contain the `input_ids` and `token_type_ids` that follow ```python inputs["input_ids"][:, 256:] # tensor([[ 2, 6006, 603, 573, 13910, 9980, 235336, 108, 477, 573, 8318]]) inputs["token_type_ids"][:, 256:] tensor([[0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]]) ``` Meaning the last three tokens are of "label" ("suffix") type while the other ones are of "prefix" type. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. text (`str`, `List[str]`, `List[List[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. suffix (`str`, `List[str]`, `List[List[str]]`): The suffixes or batch of suffixes to be encoded. Only necessary for finetuning. See https://github.com/google-research/big_vision/blob/main/big_vision/configs/proj/paligemma/README.md for more information. If your prompt is " What is on the image", the suffix corresponds to the expected prediction "a cow sitting on a bench". Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. If `suffix` is provided, the `input_ids` will also contain the suffix input ids. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. - **labels** -- Labels compatible with training if `suffix` is not None tokenizer_init_kwargsr,r!NTFzF`images` are expected as arguments to a `PaliGemmaProcessor` instance.z]You are using PaliGemma without a text prefix. It will perform as a picture-captioning model.rc3(K|] }t|vVdSr<)rbrsamples r z.PaliGemmaProcessor.__call__..s'@@{f,@@@@@@raiYou are passing both `text` and `images` to `PaliGemmaProcessor`. The processor expects special image tokens in the text, as many tokens as there are images per each text. It is recommended to add `` tokens in the very beginning of your text and `` token after that. For this call, we will infer how many images each text has and add special tokens.z Received z images for zK prompts. Each prompt should be associated with an image or list of images.cg|]}|gSrr)rimages rrz/PaliGemmaProcessor.__call__..s:::%ug:::rzAimages must be an image, list of images or list of list of imagesc.g|]}|jjzSr)rW eos_token)rsfxrls rrz/PaliGemmaProcessor.__call__..!s#OOOcDN$<<OOOrc g|]Q\}}t|jjjtt |t rt|ndRS)rD)rJrWrFr]rbr6rOlen)rrE image_listrls rrz/PaliGemmaProcessor.__call__..#so ! ! !+ ,%"&.":&*&;$/6@T6R6R#Y3z???XY  ! ! !rc`g|]*}|ttjz+Sr)replacerbr])rrvrls rrz/PaliGemmaProcessor.__call__../s0lll]c{K$BW4WXXlllrcg|]}|dS)rCrrus rrz/PaliGemmaProcessor.__call__..0s B B B6F B B Brr- pixel_values max_length) text_pairreturn_token_type_ids input_idstoken_type_idsilabels)data)r _merge_kwargsr+rW init_kwargspoprQlogger warning_oncerAr6rOanywarningrrr ziprSrVgetr] masked_fillupdater)rlrRrpaudiovideosrm output_kwargsr!r input_stringsrinputs return_datars` r__call__zPaliGemmaProcessor.__call__s"D9FF ** $  "&."<   }-11(DAA(.(: >eff f <   o   D D ! ! 6DD d # # (8a(A(A     2@@4@@@@@( C<dD))j.F.F6{{c$ii//(XF XXTXXX "&))j%hZFF--j.2K2Kj::6:::FF$VT22jz&)T7R7RjWeflmnfopqfrWsWsj$%hiii%*:6*B*B%$XF%OOOOOOOF ! ! ! !/2$.?.? ! ! ! -V44llllgklll B BT B B B +t+FUUmO6TUUVde   ' + +L$ ? ? K - ( 6 6 6$:O O 6 6 6  "7  M*   ?>>>  3K(44VO RRRRRR8"^b W.W.W.I0$y/4HYCZZ[W. 12 W. W.W.W.W.t<<<666XXXXXXXXrrUN)'rtypingrrrfeature_extraction_utilsr image_utilsrr processing_utilsr r r r rrtokenization_utils_baserrrutilsr get_loggerr"rrbrangerer r'r+r)r:r?rArJrSrUrrrrs)(((((((((44444455555555   H % % 55t5558]8]RWRWX[R\R\8]8]8]] ddddd*ddd#####L###     /u    ;4;;;; 000BBBNNN.D4Z(8#9DDDD.]X]X]X]X]X]X]X]X]X]Xr