g DdZddlmZddlmZmZmZmZddlZddl Zddlm Z ddl m Z m Z mZddlmZdd lmZmZdd lmZmZmZdd lmZdd lmZdd lmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(m)Z)e!rddl*m+Z+e#j,e-Z.dZ/dZ0dZ1dZ2dej3dej3fdZ4dej3dej3fdZ5dej3dej3fdZ6eGddeZ7eGddeZ8eGdd eZ9Gd!d"e j:Z;Gd#d$e j:Z<Gd%d&e j:Z=Gd'd(e=Z>Gd)d*e=Z?e=e?e>d+Z@Gd,d-e j:ZAGd.d/e j:ZBGd0d1eZCd2ZDd3ZEd4ZFd5ZGGd6d7e j:ZHGd8d9e j:ZIed:eDGd;de j:ZKed?eDGd@dAeCZLeeDGdBdCeCZMedDeDGdEdFeCZNedGeDGdHdIeCZOedJeDGdKdLeCZPdS)MzPyTorch CLIP model.) dataclass)AnyOptionalTupleUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)PreTrainedModel)"is_torch_greater_or_equal_than_2_2) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardis_flash_attn_2_available#is_flash_attn_greater_or_equal_2_10loggingreplace_return_docstrings torch_int) CLIPConfigCLIPTextConfigCLIPVisionConfig)_flash_attention_forwardrzopenai/clip-vit-base-patch32LABEL_0logitsreturnctj|tjt ||jS)Ndevice)r functional cross_entropytorcharangelenr()r$s b/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.pycontrastive_lossr/=s3 = & &vu|CKKPVP]/^/^/^ _ __ similaritycrt|}t|}||zdz S)Ng@)r/t)r1 caption_loss image_losss r. clip_lossr6As4#J//L!*,,..11J : % ,,r0tensorctj|d}tj|dd}tj|d}|S)z This method is equivalent to tensor.norm(p=2, dim=-1, keepdim=True) and used to make model `executorch` exportable. See issue https://github.com/pytorch/executorch/issues/3566 T)dimkeepdim?)r+powsum)r7 square_tensor sum_tensor normed_tensors r._get_vector_normrCGsB Ifa((M=b$???JIj#..M r0ceZdZUdZdZeejed<dZ ejed<dZ ee ejdfed<dZ ee ejdfed<dS)CLIPVisionModelOutputa Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__rFrr+ FloatTensor__annotations__rGrHrrIr0r.rErERs*15L(5,-444+/u(///=AM8E%"3S"89:AAA:>Ju0#567>>>>>r0rEceZdZUdZdZeejed<dZ ejed<dZ ee ejdfed<dZ ee ejdfed<dS)CLIPTextModelOutputa Base class for text model's outputs that also contains a pooling of the last hidden states. Args: text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N text_embedsrG.rHrI) rJrKrLrMrSrr+rNrOrGrHrrIrPr0r.rRrRos*04K%+,333+/u(///=AM8E%"3S"89:AAA:>Ju0#567>>>>>r0rRceZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ eed<dZeed <d eefd ZdS) CLIPOutputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPVisionModel`]. Nlosslogits_per_imagelogits_per_textrSrFtext_model_outputvision_model_outputr%c^tfdDS)Nc3tK|]2}|dvr|n!t|V3dS))rYrZN)getattrto_tuple).0kselfs r. z&CLIPOutput.to_tuple..sc   LLLDGGRYZ^`aRbRbRkRkRmRm      r0)tuplekeysras`r.r^zCLIPOutput.to_tuplesC    YY[[      r0)rJrKrLrMrVrr+rNrOrWrXrSrFrYrrZrrr^rPr0r.rUrUs()-D(5$ %,,,*.e'...)-OU&---%)K")))&*L%#***4818886:3::: %*      r0rUcveZdZdeffd ZdejdededejfdZd d ej dejfd Z xZ S) CLIPVisionEmbeddingsconfigczt||_|j|_|j|_|j|_tjtj |j|_ tj |j |j|j|jd|_|j|jzdz|_|jdz|_tj|j|j|_|dtj|jdddS)NF) in_channels out_channels kernel_sizestridebiasr9r position_idsrr: persistent)super__init__rh hidden_size embed_dim image_size patch_sizer Parameterr+randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr,expandrarh __class__s r.rtzCLIPVisionEmbeddings.__init__s   + + +!|EK,G,GHH!y+?    !Ot>1D!-1"$,t/A4>"R"R ^U\$:L-M-M-T-TU\-]-]jopppppr0 embeddingsheightwidthr%c|jddz }|jjd}|jddz }tjs&||kr ||kr||jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} | d| | | }| dddd}tj || | fdd }| dddddd| }t j||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr:r=r r9bicubicF)sizemode align_cornersr;)shaperweight unsqueezer+jit is_tracingrorxrreshapepermuterr) interpolateviewcat) rarrrrrrclass_pos_embedpatch_pos_embedr; new_height new_widthsqrt_num_positionss r.interpolate_pos_encodingz-CLIPVisionEmbeddings.interpolate_pos_encodings!&q)A- !4;EEaHH*03a7 y##%% >+*F*F6UZ??**4+<== =,QQQU3,QQQU3r"t. T_, &}c'9::)11!5GI[]`aa)11!Q1==-33 i( 4  *11!Q1==BB1b#NNy/?;CCCCr0F pixel_valuesc <|j\}}}}|s<||jks ||jkr&td|d|d|jd|jd |jjj}|||}|ddd}|j |dd} tj | |gd } |r| | | ||z} n| ||jz} | S) NzInput image size (*z) doesn't match model ().)dtyper9rr:r)rrw ValueErrorr~rrtoflatten transposer{rr+rrrro) rarr batch_size_rr target_dtype patch_embeds class_embedsrs r.forwardzCLIPVisionEmbeddings.forwardsF'3'9$ Avu' Vt-F-F%SWSbJbJbuVuueuuPTP_uubfbquuu +28 ++LOO,O,O,OPP #++A..88A>> +22:q"EE Y l;CCC # Q#d&C&CJPVX]&^&^^JJ#d&=&=d>O&P&PPJr0F) rJrKrLr!rtr+TensorintrrNr __classcell__rs@r.rgrgsq/qqqqqq,'D5<'D'DUX'D]b]i'D'D'D'DRE$5Z_Zfr0rgc eZdZdeffd Z d deejdeejdeejdej fdZ xZ S) CLIPTextEmbeddingsrhcVt|j}tj|j||_tj|j||_| dtj |j dddS)NrorpFrq) rsrtrurr vocab_sizetoken_embeddingmax_position_embeddingsrrr+r,rrarhrvrs r.rtzCLIPTextEmbeddings.__init__s & !|F,=yII"$,v/My"Y"Y  EL)GHHOOPWXXej      r0N input_idsro inputs_embedsr%c| |jdn |jd}||jddd|f}|||}||}||z}|S)Nr:)rrorr)rarror seq_lengthposition_embeddingsrs r.rzCLIPTextEmbeddings.forwards -6,AY_R((}GZ[]G^  ,QQQ  ^ )T^ ; ;'dn''N''' ]D( / i?? i?? i??  $.$.AA r0r7seq_lenbszc||||j|jddS)Nrr9)rrrr contiguous)rar7rrs r._shapezCLIPAttention._shape;s<{{3GGQQRSUVWWbbdddr0NFrHattention_maskcausal_attention_maskoutput_attentionsr%c|\}}}|||jz}|||d|} |||d|} ||jzd|jf} ||||j| }| j| } | j| } | d} tj || dd} | ||jz|| fkr2td||jz|| fd| |||d|| fkr+td|d|| fd|| ||j|| |z} | ||jz|| } |||d|| fkr+td|d|| fd|| ||j|| |z} | ||jz|| } tj| d} |r=| ||j|| }|||jz|| } nd}tj| |j|j }tj || }|||jz||jfkr5td ||j||jfd||||j||j}| dd}||||}||}||fS) z#Input shape: Batch x Time x Channelr:rr9z$Attention weights should be of size z , but is Nz!Attention mask should be of size r)ptrainingz `attn_output` should be of size )rrrrrrrrrr+bmmrrrr)softmaxrrrr)rarHrrrrtgt_lenrv query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r.rzCLIPAttention.forward>s0#0"4"4"6"6Wi{{=11DJ> [[]!;!;REE {{4;;}#=#=r3GG DN*B > Ct{{<#>>CZP $Z_j1 (|(*5 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**  ! ,$))++Q/III 7a'8R77-224477(,,S$.'7SSVkkL',,S4>-A7GTTL  %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL',,S4>-A7GTTL},,\r,BB  ) %1$5$5c4>7T[$\$\ !055cDN6JGU\]]LL$( !]**<4 L2 u|Xel33 4 L2L2L2L2L2L2L2L2r0rceZdZdZfdZ d dejdeejdeejdeed e ejeejff d Z xZ S) CLIPFlashAttention2aM CLIPAttention flash attention module. This module inherits from `CLIPAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cbtj|i|t |_dSN)rsrtr_flash_attn_uses_top_left_mask)raargskwargsrs r.rtzCLIPFlashAttention2.__init__s9$)&))) 3V2W2W.W+++r0NFrHrrrr%c d}|\}}}||}||} ||} ||||j|j}| |||j|j} | |||j|j} |jr|jnd} |j } | tj krtj rtj } n3t|jdr |jj} n|jjj } t$d| d|| }| | } | | } t+|| | ||| |du|j}||||j}||}|sd}||fS)NF_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .)r is_causaluse_top_left_mask)rrrrrrrrrrr+float32is_autocast_enabledget_autocast_gpu_dtypehasattrrhrrlogger warning_oncerr"rrrvrr)rarHrrrrq_lenrrrr dropout_rate input_dtyperrrs r.rzCLIPFlashAttention2.forwards ",1133 E1{{=11 [[// {{=11 $((UDNDMZZ __Z VV #((UDNDMZZ '+}=t||# #( %- ' '(** 8$;== &?@@ 8#{B #{17   $ $$$    (??<88L#|44J'??<88L.      +47"A    "))*eT^LLWWYY mmK00   LL((r0r) rJrKrLrMrtr+rrrrrrrs@r.rrsXXXXX268<,1 C)C)|C)!.C) ( 5 C) $D> C) u|Xel33 4 C)C)C)C)C)C)C)C)r0rceZdZdZ d dejdeejdeejdeedeejeejff fd Z xZ S) CLIPSdpaAttentionz SDPA attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `CLIPAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. NFrHrrrr%cJ|r?tdt||||S||||z}n||}n|}|\}}}||} ||} ||} | |d|j |j  dd} | |d|j |j  dd} | |d|j |j  dd} tsN| j jdkr>|<| } | } | } t jj| | | ||jr|jnd|j} | dd} | |||} || } | dfS) NaCLIPModel is using CLIPSdpaAttention, but `torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to the manual attention implementation, but specifying the manual implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rHrrrr:rr9cudar) attn_mask dropout_pr)rrrsrrrrrrrrrrr(typerr+rr)scaled_dot_product_attentionrrrrr)rarHrrrrrrrvrrrrrs r.rzCLIPSdpaAttention.forwards<     f    77??+-&;"3 #   %*?*K&)>>II " .-II&I"/"4"4"6"6Wi{{=11 [[// {{=11 #((b$.$-PPZZ[\^_`` __S"dndmLLVVWXZ[\\ #((b$.$-PPZZ[\^_`` 2 5l6I6NRX6X6X]f]r'2244L#..00J'2244Lh)FF   &*m!>!|>!!.>! ( 5 >! $D> >! u|Xel33 4 >!>!>!>!>!>!>!>!>!>!r0r)eagersdpaflash_attention_2cBeZdZfdZdejdejfdZxZS)CLIPMLPct||_t|j|_t j|j|j |_ t j|j |j|_ dSr) rsrtrhr hidden_act activation_fnrrruintermediate_sizefc1fc2rs r.rtzCLIPMLP.__init__5sf  #F$569V/1IJJ9V5v7IJJr0rHr%c||}||}||}|Sr)rrr)rarHs r.rzCLIPMLP.forward<s=// **=99 // r0)rJrKrLrtr+rrrrs@r.r r 4scKKKKKU\elr0r c eZdZdeffd Z d dejdejdejdeede ej f d Z xZ S) CLIPEncoderLayerrhcZt|j|_t |j||_tj|j|j |_ t||_ tj|j|j |_ dSNeps)rsrtrurvCLIP_ATTENTION_CLASSES_attn_implementation self_attnr LayerNormlayer_norm_eps layer_norm1r mlp layer_norm2rs r.rtzCLIPEncoderLayer.__init__Ds +/0KLVTT<F & u ! &&&&&&&&r0rc,eZdZdZeZdZdZdZdZ dZ dS)CLIPPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. clipTcd |jj}t|trT|jjjd|dz|jjjd|dzn t|tr|jj}tj |j d|j dz|ztj |jj|jj|ztj |jj|jj|zn[t|t r|jj}|j dzd|jjzdzz|z}|j dz|z}tj |jj|tj |jj|tj |jj|tj |jj|n_t|t,r|jj}|jjdzd|jjzdzz|z}d|jjzdz|z}tj |jj|tj |jj|nt|t4rtj |jj|jdz|jjztj |jj|jdz|jjznt|t>rFtj |jj|jjdz|jjznt|t@rFtj |jj|jjdz|jjzn_t|tBrJtj |j"j|jj#jdz|jjzt|tj$r=|j%j&|jj'dt|tj(r'|j%"|j%j&dSdSdS) zInitialize the weightsrg{Gz?)meanstdr)r+r9g?N))rhinitializer_factor isinstancerrrdatanormal_rrgrinitr{rvr~initializer_rangernum_hidden_layersrrrrr rurr CLIPModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimCLIPVisionModelWithProjectionCLIPTextModelWithProjectionCLIPForImageClassification classifier vision_configrrnzero_fill_r)ramodulefactor in_proj_std out_proj_stdfc_stds r. _init_weightsz!CLIPPreTrainedModel._init_weightss/ f0 1 1-   " ) . 6 6CVd] 6 S S S  % , 1 9 9sQU 9 V V V V  4 5 5* [3F GOOF2&BRTXBX[aBaO b b b GOOF29v}?^ag?gO h h h GOOF5<&-BadjBjO k k k k  . .% [3F!+T1q6=;Z7Z_c6cdgmmK",d2f >  GOO(/K+T1DK4RR      ; < <  GOO&-K+T1DK4RR      : ; ;  GOO!(K-94?$+B``     fbl + + * K  " " $ $ $ M  $ $S ) ) ) fbi ( ( %V[-D K  " " $ $ $ $ $ % %-D-Dr0N) rJrKrLrMr config_classbase_model_prefixsupports_gradient_checkpointing_supports_sdpa_supports_flash_attn_2rDrPr0r.r'r'usK L&*#N!6%6%6%6%6%r0r'a= This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`CLIPConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*, defaults `False`): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`CLIPImageProcessor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*, defaults `False`): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. ceZdZdZdeffd Z d deejdeejdee dee d ee d e e e ff d Z xZS) CLIPEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPEncoderLayer`]. Args: config: CLIPConfig rhct|_tjfdt jD|_d|_dS)Nc.g|]}tSrP)r)r_rrhs r. z(CLIPEncoder.__init__...s"$g$g$g!%5f%=%=$g$g$gr0F) rsrtrhr ModuleListranger2layersgradient_checkpointingrs `r.rtzCLIPEncoder.__init__+s`  m$g$g$g$guVMeGfGf$g$g$ghh &+###r0Nrrroutput_hidden_states return_dictr%c||n |jj}||n |jj}||n |jj}|rdnd}|rdnd}|} t |jD]_\} } |r|| fz}|jr&|jr|| j | |||} n| | |||} | d} |r || dfz}`|r|| fz}|std| ||fDSt| ||S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrP)rrrc3K|]}||V dSrrP)r_vs r.rbz&CLIPEncoder.forward..}s(eeqWXWdWdWdWdWdeer0)rGrHrI) rhrrSuse_return_dict enumeraterQrRr_gradient_checkpointing_func__call__rcr) rarrrrrSrTencoder_statesall_attentionsrHidx encoder_layer layer_outputss r.rzCLIPEncoder.forward1sL2C1N--TXT_Tq$8$D $+Jj &1%>N fee]NN$Seeeee e+>Vd    r0NNNNN)rJrKrLrMrrtrr+rrrrrrrrs@r.rKrK"s,z,,,,,,268<,0/3&*O O !.O  ( 5 O $D> O 'tn O d^O  uo% &O O O O O O O O r0rKceZdZdeffd Zeeeee d de e j de e j de e j de e d e e d e e d e eeffd ZxZS)CLIPTextTransformerrhc,t||_|j}t ||_t ||_tj ||j |_ |j |_ |j dk|_dS)Nrr )rsrtrhrurrrKencoderrrrfinal_layer_norm eos_token_idr_use_flash_attention_2rs r.rtzCLIPTextTransformer.__init__s  & ,V44"6** " YF P 'tn P d^P  u00 1P P P dcGFP P P P P r0rcz?The text model from CLIP without any head or projection on top.c<eZdZeZddgZdeffd ZdejfdZ dZ e e e ee dd eejd eejd eejd eedeedeedeeeffdZxZS) CLIPTextModelrrrhct|t||_|dSr)rsrtrc text_model post_initrs r.rtzCLIPTextModel.__init__s@    -f55 r0r%c$|jjjSrrxrrres r.get_input_embeddingsz"CLIPTextModel.get_input_embeddings)99r0c(||jj_dSrr{ravalues r.set_input_embeddingsz"CLIPTextModel.set_input_embeddings5:"222r0riNrrrorrSrTcX||n |jj}|||||||S)aK Returns: Examples: ```python >>> from transformers import AutoTokenizer, CLIPTextModel >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```NrrrorrSrT)rhrXrx)rarrrorrSrTs r.rzCLIPTextModel.forwardsD8&1% # 'tn # d^#  u00 1# # # dcGF# # # # # r0rvceZdZdeffd Zeeeee d de e j de e de e d e e d e e d e eeff d ZxZS)CLIPVisionTransformerrhc4t||_|j}t ||_t j||j|_ t||_ t j||j|_ dSr) rsrtrhrurgrrrr pre_layrnormrKrepost_layernormrs r.rtzCLIPVisionTransformer.__init__%s  & .v66L8MNNN"6**  l9&:OPPPr0riNFrrrSrTrr%c||n |jj}||n |jj}||n |jj}|t d|||}||}|||||}|d}|dddddf} || } |s|| f|ddzSt|| |j |j S)rlNz You have to specify pixel_values)r)rrrSrTrrrm) rhrrSrXrrrrerrrHrI) rarrrSrTrrHrqrGrrs r.rzCLIPVisionTransformer.forward/s/2C1N--TXT_Tq$8$D $+Jj &1%+ 'tn + d^ + #+4. +  u00 1+ + + feIH+ + + + + r0rzAThe vision model from CLIP without any head or projection on top.ceZdZeZdZdgZdeffd Zdej fdZ e e e ee ddeejd eed eed ed eedeeeff dZxZS)CLIPVisionModelrrrhct|t||_|dSr)rsrtr vision_modelryrs r.rtzCLIPVisionModel.__init__hsA    1&99 r0r%c$|jjjSrrrr~res r.r|z$CLIPVisionModel.get_input_embeddingsn +;;r0riNFrrSrrTcV||n |jj}||||||S)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPVisionModel >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```N)rrrSrTr)rhrXr)rarrrSrrTs r.rzCLIPVisionModel.forwardqsD@&1%& 'tn & #' & d^ &  u00 1& & & feIH& & & & & r0rceZdZeZgdZdeffd Zee dde e j de e j de e j de e d e e d e e d e j fd Zee dde e j de e d e e de d e e d e j f dZeeeee dde e jde e j de e j de e jde e de e d e e de d e e d eeeffdZxZS)r3)rrrgrhclt|t|jts%t dt |jdt|jts%t dt |jd|j}|j}|j |_ |j |_ |j |_ t|}|j|_t |}|j|_t%j|j |j d|_t%j|j |j d|_t%jt/j|jj|_|dS)NzKconfig.text_config is expected to be of type CLIPTextConfig but is of type rzOconfig.vision_config is expected to be of type CLIPVisionConfig but is of type Frn)rsrtr- text_configr TypeErrorrr<r!projection_dimrur5r7rv _from_configrxrrrrr6r4ryr+r7rhlogit_scale_init_value logit_scalery)rarhrr<rxrrs r.rtzCLIPModel.__init__s    &,n== 0+,,000  &.0@AA 2-..222  ( , $3)5 - 9"// << $/&33MBB (5!#4+@$BU\a!b!b!b!y)QX]^^^< T[5W(X(XYY r0NrrrorrSrTr%c||n |jj}||n |jj}||n |jj}|||||||}|d}||} | S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```Nrr)rhrrSrXrxr4) rarrrorrSrT text_outputsrr text_featuress r.get_text_featureszCLIPModel.get_text_featuress62C1N--TXT_Tq$8$D $+Jj &1%>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```NrrrSrrTr)rhrrSrXrr6) rarrrSrrTvision_outputsrrimage_featuress r.get_image_featureszCLIPModel.get_image_featuress@2C1N--TXT_Tq$8$D $+Jj &1%>r0ri return_lossc ||n |jj}||n |jj}| | n |jj} |||||| } ||||||| } | d} || } | d} || } | t| z } | t| z } |j }tj | | | j|| jz}| }d}|rt!|}| s||| | | | f}||f|zn|St#|||| | | | S)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```Nrrr)rVrWrXrSrFrYrZ)rhrrSrXrrxr6r4rCrexpr+matmulr3rr(r6rU)rarrrrorrrSrrTrrrFrSrrXrWrVoutputs r.rzCLIPModel.forward&sN2C1N--TXT_Tq$8$D $+Jj &1% , 'tn ,d^,  ,,,GF,\+*+GHH59,0/3).&* 00u010$D>0'tn 0 #' 0 d^ 0  000IH0d+*+@AA:JOOO15481537&*,0/3).&*] ] E,-] u01] !. ] u/0 ] d^ ] $D>] 'tn] #'] d^]  uj !] ] ] POBA] ] ] ] ] r0r3zf CLIP Text Model with a projection layer on top (a linear layer on top of the pooled output). c<eZdZeZddgZdeffd ZdejfdZ dZ e e e ee dd eejd eejd eejd eedeedeedeeeffdZxZS)r9rrrhct|t||j}|j|_t j|j|j d|_ | dSN)attn_implementationFr) rsrtrvrrrxrrrurr4ry)rarhrxrs r.rtz$CLIPTextModelWithProjection.__init__sx    "//FLg/hh $/!y);V=RY^___ r0r%c$|jjjSrr{res r.r|z0CLIPTextModelWithProjection.get_input_embeddingsr}r0c(||jj_dSrr{rs r.rz0CLIPTextModelWithProjection.set_input_embeddingsrr0riNrrrorrSrTc6||n |jj}|||||||}|d}||} |s.| |df|ddz} t d| DSt | |j|j|jS)a  Returns: Examples: ```python >>> from transformers import AutoTokenizer, CLIPTextModelWithProjection >>> model = CLIPTextModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> text_embeds = outputs.text_embeds ```Nrrrr9c3K|]}||V dSrrPr_rs r.rbz6CLIPTextModelWithProjection.forward..'LLF9K9K9K9K9KLLr0)rSrGrHrI) rhrXrxr4rcrRrGrHrI) rarrrorrSrTrrrrSr%s r.rz#CLIPTextModelWithProjection.forwards6&1%^\\\-115/3,0/3&*1 1 EL)1 !.1 u|, 1 $D> 1 'tn 1 d^1  u)) *1 1 1 ]\GF1 1 1 1 1 r0r9zh CLIP Vision Model with a projection layer on top (a linear layer on top of the pooled output). ceZdZeZdZdeffd ZdejfdZ e e e e e ddeejd eed eed ed eedeee ff d ZxZS)r8rrhct|t||j}|j|_t j|j|j d|_ | dSr) rsrtrrrrrrrurr6ryrarhrrs r.rtz&CLIPVisionModelWithProjection.__init__sz    &33FPVPk3ll (5!#6+=v?T[`!a!a!a r0r%c$|jjjSrrres r.r|z2CLIPVisionModelWithProjection.get_input_embeddingsrr0riNFrrSrrTc4||n |jj}||||||}|d}||}|s.||df|ddz} t d| DSt ||j|j|jS)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, CLIPVisionModelWithProjection >>> model = CLIPVisionModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> image_embeds = outputs.image_embeds ```Nrrrr9c3K|]}||V dSrrPrs r.rbz8CLIPVisionModelWithProjection.forward..!rr0)rFrGrHrI) rhrXrr6rcrErGrHrI) rarrrSrrTrrrrFr%s r.rz%CLIPVisionModelWithProjection.forwards>&1%(6%0     r0r)rJrKrLr!rErrtrrr|rrrrErr+rNrrrrrrs@r.r8r8s)$L$O /      4 'tn 4 #' 4 d^ 4  u++ ,4 4 4 a`IH4 4 4 4 4 r0r8z CLIP vision encoder with an image classification head on top (a linear layer on top of the pooled final hidden states of the patch tokens) e.g. for ImageNet. ceZdZdZdeddffd Zeeee e e e  d de ejde ejde ed e ed e edeee ff d ZxZS) r:rrhr%Ncnt||j|_t|j}|j|_|jdkr$tj|jj |jntj |_ | dS)Nr) rsrt num_labelsrrr<rrrruIdentityr;ryrs r.rtz#CLIPForImageClassification.__init__5s     +&33F4HII (5OUN_bcNcNcBIf*68I J J Jikitiviv  r0) checkpointrjrEexpected_outputlabelsrrSrTcx||n |jj}||n |jj}||n |jj}|||||}|d}t j|ddddddfd}||}d} |t||j }|jj f|j dkr d|j_ nN|j dkr7|j t j ks|j t jkr d|j_ n d|j_ |jj dkrWt} |j dkr1| ||} n| ||} n|jj dkrGt#} | |d |j |d } n*|jj dkrt'} | ||} |s|f|d dz} | | f| zn| St)| ||j|j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrSrTrrr regressionsingle_label_classificationmulti_label_classificationr:r9)rVr$rHrI)rhrrSrXrr+r*r;rr( problem_typerrlongrr squeezer rr rrHrI) rarrrrSrTr%sequence_outputr$rVloss_fctrs r.rz"CLIPForImageClassification.forwardDsq*2C1N--TXT_Tq$8$D $+Jj &1%AFFF11  YYv}--F{'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x// FY,F)-)9TGf$$v E$!/)     r0ra)rJrKrLrrrtrrr_IMAGE_CLASS_CHECKPOINTr_CONFIG_FOR_DOC_IMAGE_CLASS_EXPECTED_OUTPUTrr+rrrrcrrrs@r.r:r:+s/%O z d      +*+@AA*)$4 04)-,0/3&* D D u|,D &D $D> D 'tn D d^ D  u++ ,D D D BAD D D D D r0r:)QrM dataclassesrtypingrrrrr+torch.utils.checkpointrtorch.nnr r r activationsr modeling_attn_mask_utilsrrmodeling_outputsrrrmodeling_utilsr pytorch_utilsrutilsrrrrrrrrrconfiguration_cliprr r!modeling_flash_attention_utilsr" get_loggerrJrr_CHECKPOINT_FOR_DOCrrrr/r6rCrErRrUrrgrrrrrr rr'CLIP_START_DOCSTRINGrtrrrKrcrvrrr3r9r8r:rPr0r.rsO!!!!!!............ AAAAAAAAAA!!!!!!ddddddddbbbbbbbbbb------??????                      MLLLLLLLLLKJJJJJJ  H % %49( `U\`el````-%,-5<---- U\el ?????K?? ?8 ?????+?? ?8 ! ! ! ! ! ! !  ! HPPPPP29PPPfBe2e2e2e2e2BIe2e2e2PT)T)T)T)T)-T)T)T)nF!F!F!F!F! F!F!F!T ,     bi   /////ry///dB%B%B%B%B%/B%B%B%J @ "%P^ ^ ^ ^ ^ ")^ ^ ^ Ba a a a a ")a a a HI6 6 6 6 6 '6 6 6 r8 8 8 8 8 BI8 8 8 vK6 6 6 6 6 )6 6 6 r*++h h h h h #h h ,+h V  I I I I I "5I I  I X  H H H H H $7H H  H V ] ] ] ] ] !4] ] ] ] ] r0