gdZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl mcmZddlZ ddl mZmZddlmZddlmZdd lmZdd lmZmZmZmZd d lmZm Z m!Z!m"Z"ej#e$Z%d Z&dZ'eGddeZ(eGddeZ)Gddej*Z+Gddej*Z,Gddej*Z-Gddej*Z.Gddej*Z/Gddej*Z0Gdd ej*Z1Gd!d"ej*Z2Gd#d$ej*Z3Gd%d&ej*Z4Gd'd(ej*Z5Gd)d*ej*Z6Gd+d,ej*Z7Gd-d.ej*Z8Gd/d0ej*Z9Gd1d2eZ:d3Z;d4Z<ed5d6e;Gd7d8e:Z=dS)9zPyTorch SAM model.N) dataclass)DictListOptionalTupleUnion)Tensornn)ACT2FN)BaseModelOutput)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) SamConfigSamMaskDecoderConfigSamPromptEncoderConfigSamVisionConfigrzfacebook/sam-vit-hugeceZdZUdZdZeejed<dZ ejed<dZ ee ejdfed<dZ ee ejdfed<dS)SamVisionEncoderOutputa Base class for sam vision model's outputs that also contains image embeddings obtained by applying the projection layer to the pooler_output. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrr`/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/sam/modeling_sam.pyrr(s,15L(5,-444+/u(///=AM8E%"3S"89:AAA:>Ju0#567>>>>>r&rceZdZUdZdZejed<dZejed<dZ e e ejdfed<dZ e e ejdfed<dZ e e ejdfed<dS) SamImageSegmentationOutputa Base class for Segment-Anything model's output Args: iou_scores (`torch.FloatTensor` of shape `(batch_size, num_masks)`): The iou scores of the predicted masks. pred_masks (`torch.FloatTensor` of shape `(batch_size, num_masks, height, width)`): The predicted low resolutions masks. Needs to be post-processed by the processor vision_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the vision model at the output of each layer plus the optional initial embedding outputs. vision_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. mask_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N iou_scores pred_masks.vision_hidden_statesvision_attentionsmask_decoder_attentions)rrr r!r*r"r#r$r+r,rrr-r.r%r&r'r)r)Fs6%)J!((($(J!(((DH(5):C)?#@AHHHAExe&7&< =>EEEGKXeE,=s,B&CDKKKKKr&r)c(eZdZdZfdZdZxZS)SamPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct|j|j}}|j|j}}t |tjj r|n||f}t |tjj r|n||f}|d|dz|d|dzz}||_||_||_||_ tj |||||_ dS)Nrr kernel_sizestride)super__init__ image_size patch_size num_channels hidden_size isinstance collectionsabcIterable num_patchesr Conv2d projection)selfconfigr7r8r9r:r? __class__s r'r6zSamPatchEmbeddings.__init__qs !'!2F4EJ $*$79Kk #-j+/:R#S#SqZZZdfpYq #-j+/:R#S#SqZZZdfpYq !!} 1 5*Q-:VW=:XY $$(&)L+:^hiiir&c P|j\}}}}||jkrtd||jdks||jdkr2td|d|d|jdd|jdd ||ddd d}|S) NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrzInput image size (*z) doesn't match model (z).r )shaper9 ValueErrorr7rApermute)rB pixel_values batch_sizer9heightwidth embeddingss r'forwardzSamPatchEmbeddings.forwards2>2D/ L&% 4, , ,w  T_Q' ' '5DOA4F+F+FwVwwewwDO\]L^wwaeapqraswww __\22::1aAFF r&)rrr r!r6rP __classcell__rDs@r'r0r0jsV j j j j j       r&r0cBeZdZfdZdejdejfdZxZS) SamMLPBlockcttj|j|j|_tj|j|j|_t|j |_ dSN) r5r6r Linearr:mlp_dimlin1lin2r hidden_actactrBrCrDs r'r6zSamMLPBlock.__init__s\ If0&.AA Ifnf.@AA &+,r&rreturnc||}||}||}|SrV)rYr\rZrBrs r'rPzSamMLPBlock.forwards; -00 //  -00 r&)rrr r6r"r rPrQrRs@r'rTrTs^----- U\elr&rTcHeZdZdZdfd ZdejdejfdZxZS) SamLayerNormaALayerNorm that supports two data formats: channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch_size, height, width, channels) while channels_first corresponds to inputs with shape (batch_size, channels, height, width). ư> channels_lastc`ttjt j||_tjt j||_||_ ||_ |j dvrtd|j |f|_ dS)N)rdchannels_firstzUnsupported data format: ) r5r6r Parameterr"onesweightzerosbiaseps data_formatNotImplementedErrornormalized_shape)rBrorlrmrDs r'r6zSamLayerNorm.__init__s l5:.>#?#?@@ L-=!>!>?? &  #F F F%&T$BR&T&TUU U!1 3r&xr^c*|jdkr=tjj||j|j|j|j}n|jdkr|j }| }| dd}||z d dd}||z tj ||jzz }||}|jddddf|z|jddddfz}|S)NrdrfrT)keepdimrG)dtype)rmr"r functional layer_normrorirkrlrsfloatmeanpowsqrtto)rBrp input_dtypeuss r'rPzSamLayerNorm.forwards   . .#..q$2GVZV_aeaijjAA  !1 1 1'K Aq$''AQ A##At#44AQ%*Q\222A;''A AAAtTM*Q.111dD=1IIAr&)rcrd) rrr r!r6r"r rPrQrRs@r'rbrbsm 444444  %,        r&rbc jeZdZdZdfd ZdededefdZdededefd Zdd ed ed ed edef dZ xZ S) SamAttentionz SAM's attention layer that allows for downscaling the size of the embedding after projection to queries, keys, and values. Nct|j|_||jn|}|j|z|_|j|_|j|jzdkrt dtj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|j|_ dS)Nrz,num_attention_heads must divide hidden_size.) r5r6r:attention_downsample_rate internal_dimnum_attention_headsrIr rWq_projk_projv_projout_proj)rBrCdownsample_raterDs r'r6zSamAttention.__init__s !->M>U&::[j"./A#)#=  v9 9Q > >KLL Li 0$2CDD i 0$2CDD i 0$2CDD  $"3T5EFF r&rrr^c|j\}}}}||z}|||z|||}|ddSNrrG)rHreshape transpose)rBrrbatchpoint_batch_sizen_tokenschannel c_per_heads r'_separate_headszSamAttention._separate_headssV5B5H27 33 %--e6F.FRegqrr &&q!,,,r&rc|j\}}}}|dd}|||z||||zSr)rHrr)rBrrrn_headsrrs r'_recombine_headszSamAttention._recombine_headssQ/%>@PRZ\cfp\pqqqr&querykeyvalueattention_similaritycd||}||}||}|jd}|||j}|||j}|||j}|j\}}}}||ddddz}||dzz }tj|d}|||z}tj|d}||z} | | |} | | } | S)Nrrr rG?dim) rrrrHrrrJr"softmaxrr) rBrrrrr_rattnouts r'rPzSamAttention.forwards5 E""kk# E"" ;q>$$UD,DEE""3(@AA$$UD,DEE$k1as{{1aA...z3'}Tr***  +..D=2...DUl##C)9::mmC   r&rV) rrr r!r6r intrrrPrQrRs@r'rrs GGGGGG -V-#-RX---- rfrrPVrrrr V&W]ior&rc NeZdZd dedeffd Z ddededed ed ed ef d ZxZS)SamTwoWayAttentionBlockrGFrskip_first_layer_pecZt|j|_|j|_t |d|_t j|j|j|_t |||_ t j|j|j|_ t||_ t j|j|j|_ t j|j|j|_t |||_||_dS)a A transformer block with four layers: (1) self-attention of sparse inputs (2) cross attention of sparse inputs -> dense inputs (3) mlp block on sparse inputs (4) cross attention of dense inputs -> sparse inputs Arguments: config (`SamMaskDecoderConfig`): The configuration file used to instantiate the block attention_downsample_rate (*optionalk*, int, defaults to 2): The downsample ratio of the block used to reduce the inner dim of the attention. skip_first_layer_pe (*optional*, bool, defaults to `False`): Whether or not to skip the addition of the query_point_embedding on the first layer. r)rrlN)r5r6r:layer_norm_epsr self_attnr LayerNorm layer_norm1cross_attn_token_to_image layer_norm2rTmlp layer_norm3 layer_norm4cross_attn_image_to_tokenr)rBrCrrrDs r'r6z SamTwoWayAttentionBlock.__init__s !-$3%fa@@@<(8d>QRRR)5fNg)h)h)h&<(8d>QRRRv&&<(8d>QRRR<(8d>QRRR)5fNg)h)h)h   r&querieskeysquery_point_embeddingkey_point_embeddingroutput_attentionsc(|jr||||}n"||z}||||}||z}||}||z}||z} ||| ||}||z}||}||} || z}||}||z}||z} || ||}||z}||}||f} |r| |fz} n| dz} | S)Nrrr)rrrrrV) rrrrrrrrr) rBrrrrrrrattn_outrmlp_outoutputss r'rPzSamTwoWayAttentionBlock.forwardsn  # )nn7wnOOGG33E~~EuG~LLH(G""7++//((11SCW2  H$""7++((7##G#""7++//((11g1VVh%%D/  ( +GG'Gr&)rGFF) rrr rboolr6r rPrQrRs@r'rrs77#7X\777777P#(2222 & 2 $ 2 % 2 22222222r&rceZdZdeffd Z d dededededeed eed eed ee e ffd Z xZ S)SamTwoWayTransformerrCct||_|j|_t j|_t|jD]/}|jt||dk0t||_ t j |j |_dS)Nr)r)r5r6rCnum_hidden_layersr ModuleListlayersrangeappendrrfinal_attn_token_to_imagerr:layer_norm_final_attn)rBrCirDs r'r6zSamTwoWayTransformer.__init__Qs  !'!9moo t-.. ^ ^A K  6vTUYZTZ\\\ ] ] ] ])5f)=)=&%'\&2D%E%E"""r&Npoint_embeddingsimage_embeddingsimage_positional_embeddingsrroutput_hidden_states return_dictr^c b||n |jj}||n |jj}||n |jj}d} |t d|ddddd}|ddddd}|} |} |jD]&} || |z } | | | ||||\} } } |r| | fz} '| |z}| |z}| ||| }| |z} | | } | | | fS)Nr%z&You have to specify an image_embeddingrGrr)rrrrrrr) rCrruse_return_dictrIflattenrJ unsqueezerrr)rBrrrrtarget_embeddingrrrall_attentionsrrlayerattention_outputsrrrs r'rPzSamTwoWayTransformer.forward^s2C1N--TXT_Tq$8$D $+Jj &1%>q!QGGQQRSTT&A&I&I!&L&L&T&TUVXY[\&]&]&g&ghi&j&j##[ G GE+++/4u&6$?%9"3 000 ,GT,! G!/3D2F!F**00113d1SSH$,,W55n,,r&NNNN) rrr rr6r rrrrr rPrQrRs@r'rrPs F3 F F F F F F&,0/3&*6-6- 6-!6-&, 6- % 6-$D>6-'tn6-d^6- uo% &6-6-6-6-6-6-6-6-r&rc >eZdZ d dededededef fd ZdZxZS) SamFeedForwardF input_dim hidden_dim output_dim num_layerssigmoid_outputcht||_tj|_tj||_tj||_tj fdt|dz D|_ ||_ dS)Nc:g|]}tjSr%)r rW).0rrs r' z+SamFeedForward.__init__..s%$f$f$f1RYz:%F%F$f$f$fr&rG) r5r6rr ReLU activationrWproj_inproj_outrrrr)rBrrrrrrDs ` r'r6zSamFeedForward.__init__s $'))yJ77  *j99 m$f$f$f$fPUV`cdVdPePe$f$f$fgg ,r&c ||}||}|jD] }|||}!||}|jrt j|}|SrV)rrrrrFsigmoid)rBrrs r'rPzSamFeedForward.forwards ]33  66 [ B BE OOEE-,@,@AAMM m44   5Im44Mr&r)rrr rrr6rPrQrRs@r'rrs|hm - - -*- -;> -LO -ae - - - - - -       r&rceZdZdeffd Z ddejdejdejdejded eed ejd ejd e ejejffd Z xZ S)SamMaskDecoderrCcht|j|_|j|_|jdz|_t jd|j|_t j|j|j|_t||_ t j |j|jdzdd|_ t j |jdz|jdzdd|_ t|jdzd|_t j|_g}t%|jD]*}|t'|j|j|jdzdgz }+t j||_t'|j|j|j|j|_dS) NrrGr2rfrmr )r5r6r:num_multimask_outputsnum_mask_tokensr Embedding iou_token mask_tokensr transformerConvTranspose2d upscale_conv1 upscale_conv2rbupscale_layer_normGELUrrrroutput_hypernetworks_mlpsiou_head_hidden_dimiou_head_depthiou_prediction_head)rBrC mlps_listrrDs r'r6zSamMaskDecoder.__init__s !-%+%A"%;a?a)9::<(NOO/77 /0@$BRVWBWefopqqq/0@A0EtGW[\G\jktuvvv".t/?1/DRb"c"c"c')) t+,, h hA .)94;KTM]abMbdeffg gII)+y)A)A  f8$:NPVPe$ $    r&Nrrsparse_prompt_embeddingsdense_prompt_embeddingsmultimask_outputrrrr^c |j\} } } } |jd} tj|jj|jjgd}|| | dd}|dkrtj||fd}n|}| |jjj }||z}| | d}| | d}| ||||||\}}}|dddddddf}|dddddd|j zddf}|dd| | z| | | }||}|||}|||}g}t)|j D].}|j|}|||dddd|ddfgz }/tj|d}|j\}} } } || | | | | z}||z| | d| | }||}|rt1dd}nt1dd}|dddd|ddddf}|dddd|f}||f}|r||fz}n|d z}|S) a Predict masks given image and prompt embeddings. Args: image_embeddings (`torch.Tensor`): the embeddings from the image encoder image_positional_embedding (`torch.Tensor`): positional encoding with the shape of image_embeddings sparse_prompt_embeddings (`torch.Tensor`): The embeddings of the points and boxes dense_prompt_embeddings (`torch.Tensor`): the embeddings of the mask inputs multimask_output (bool): Whether to return multiple masks or a single mask. output_attentions (bool, *optional*): Whether or not to return the attentions tensors of all attention layers. rrrrG)rrrrrrNr rrV)rHr"catrrirrepeatsumitemrzrsrepeat_interleaverrrrrrrrrrstackrslice)rBrrrrrrrrrLr9rMrNr output_tokenstokensrpoint_embeddingr iou_token_outmask_tokens_outupscaled_embedding hyper_in_listr current_mlphyper_inrmasksiou_pred mask_slicers r'rPzSamMaskDecoder.forwards83C2H/ L&%39!< 4>#8$:J:Q"RXYZZZ %,,Z9I1aPP # ' ' ) ) . . 0 0A 5 5Y /GHaPPPFF"F!99T^%:%@AA,.EE+==>NPQRR&A&S&STdfg&h&h#9=8H8H--(C!5-/ 9I9 9 5):(111a 3 )!!!QQQQ9M5M0NPQPQPQ*QR,55a;;CC ) )<  "//0@AA!__T-D-DEW-X-XYY!__T-?-?@R-S-STT t+,, H HA8;K kk/!!!QQQ111**EFFG GMM;}!444);)A&</77 DTVbdjmrdrss..77 DTVXZ`bghh++M::  %q$JJq!JaaaJ111,-AAAqqq*,-(#  ( -GG'Gr&NNN) rrr rr6r"r rrrrPrQrRs@r'rrs 3      F-1-1)-^^,^&+\^#(, ^ "' ^  ^$D>^$l^ ,^ u|U\) *^^^^^^^^r&rc&eZdZfdZddZxZS)SamPositionalEmbeddingct|jdz|_|d|jt jd|jfzdS)NrGpositional_embedding)r5r6r:scaleregister_bufferr"randn num_pos_featsr]s r'r6zSamPositionalEmbedding.__init__.s_ '1,  3TZ%+qRXRfNgBhBh5hiiiiir&Nc|}|P|dddddddf|dz |dddddddf<|dddddddf|dz |dddddddf<d|zdz }||jj}||jz}dtjz|z}t jt j|t j |gdS)z8Positionally encode points that are normalized to [0,1].NrrrGrr) clonerzrrsnppir"rsincos)rB input_coords input_shape coordinatess r'rPzSamPositionalEmbedding.forward3s "((**  "&1!!!QQQ1*&= A&NK111aaa #&1!!!QQQ1*&= A&NK111aaa #+o) !nnT%>%DEE !D$== "%i+- y%)K00%)K2H2HIrRRRRr&rV)rrr r6rPrQrRs@r'rr-sYjjjjj SSSSSSSSr&rc*eZdZdeffd ZdZxZS)SamMaskEmbeddingrCct|jdz|_t|j|_t jd|jdd|_t j|j|jdd|_ t j|j|j d|_ t|j|j d|_t|jdz|j d|_dS)NrrrGr2)r3rf)rlrm)r5r6mask_input_channelsr r[rr r@conv1conv2r:conv3rbrrrr]s r'r6zSamMaskEmbedding.__init__Es #)#=#B  !23Yq$":RSTTT Yt79Sabklmmm Yv96;M[\]]] '  $&*?M]   (  $q (f.CQa   r&c,||}||}||}||}||}||}||}|SrV)r-rrr.rr/)rBrrdense_embeddingss r'rPzSamMaskEmbedding.forwardSs 5)) ((77  66  =11 ((77  66 ::m44r&)rrr rr6rPrQrRs@r'r*r*DsT  5               r&r*c 0eZdZdeffd ZdejdejdedejfdZdejdejfd Z d e e ejejfd e ejd e ejd e ejde ejejff dZ xZ S)SamPromptEncoderrCct||_t|_t jdj|_j j f|_ j |_ t j fdtjD|_j|_t jdj|_dS)NrcDg|]}tjdjS)r)r rr:)rrrCs r'rz-SamPromptEncoder.__init__..js( ] ] ]QR\!V/ 0 0 ] ] ]r&)r5r6shared_embeddingr* mask_embedr rr: no_mask_embedimage_embedding_sizer7input_image_sizerrnum_point_embeddings point_embednot_a_point_embed)rBrCshared_patch_embeddingrDs ` r'r6zSamPromptEncoder.__init__`s  6*622\!V-?@@%+%@&B]$^! & 1= ] ] ] ]%@[:\:\ ] ] ]  "-!#a1C!D!Dr&pointslabelspadr^c |dz}|r|jd|jdd|jdf}|jd|jddf}tj||j}tj||j }tj||gd}tj||gd}|j|jf}|||} tj|ddk|j j | } tj|dd k| tj d | j | j } tj|dkd d d d d d d f| |j dj d d d d d d fz| } tj|dkd d d d d d d f| |j dj d d d d d d fz| } | S) zEmbeds point prompts.rrrrdevicerGr).NirsrDN)rHr"rjrDrhrr:r6wherer=ritensorrsr<) rBr?r@rAtarget_point_shapetarget_labels_shape padding_point padding_labelr'r s r' _embed_pointszSamPromptEncoder._embed_pointsos#  ?"(,q/6<?Av|TVGW!X #)<?FLOQ"G !K(:6=QQQM"Z(;FMRRRRMY 6A>>>FY 6A>>>F,d.CD // DD +fY&72&=t?U?\^mnn + 9  $  LO$9/BX Y Y Y    + q[!!!QQQ4- ( d.q18tQQQ9IJ J     + q[!!!QQQ4- ( d.q18tQQQ9IJ J    r&boxescT|dz}|jdd\}}|||dd}|j|jf}|||}|dddddddfxx|jdjz cc<|dddddddfxx|jdjz cc<|S)zEmbeds box prompts.rNrGrrr )rHrr:r6r<ri)rBrNrLnb_boxescoordsr'corner_embeddings r' _embed_boxeszSamPromptEncoder._embed_boxess ${2A2 Hz8Q::,d.CD 00EEAAAq!!!$$$(8(;(BB$$$AAAq!!!$$$(8(;(BB$$$r& input_points input_labels input_boxes input_maskscFd}d}|jjj}|?|jdd\}}|t d||||du} | }|?|jd}||} || }ntj|| gd}|| |} nN|j j dddd |d|jd|jd} |tj|dd|jf| }|| fS) au Embeds different types of prompts, returning both sparse and dense embeddings. Args: points (`torch.Tensor`, *optional*): point coordinates and labels to embed. boxes (`torch.Tensor`, *optional*): boxes to embed masks (`torch.Tensor`, *optional*): masks to embed NrrGz5If points are provided, labels must also be provided.)rArrrrC)r6rrDrHrIrMrSr"rr7r8rirexpandr9rjr:) rBrTrUrVrWsparse_embeddingsrL target_devicerrbox_embeddingsr1s r'rPzSamPromptEncoder.forwardsc$! -BI  #+7+=bqb+A (J(# !XYYY#11, S^bfSf1hh  0   "$*1-J!..{;;N ($2!!$)I/@..QWX$Y$Y$Y!  "#{;;  #18@@B1MMTTB 9! WXY>Z     $ % ZAt?O,PYf g g g  "222r&)rrr rr6r"r rrMrSrrrPrQrRs@r'r3r3_s% E5 E E E E E E$EL$%,$T$V[Vb$$$$L  %,  5<     ,3uU\5<%?@A,3u|,,3el+ ,3 el+ ,3 u|U\) * ,3,3,3,3,3,3,3,3r&r3ceZdZdZfdZdededejdejfdZdejd ejd ejd ejde eefde eefdejfd Z ddejdejfdZ xZ S)SamVisionAttentionz=Multi-head Attention block with relative position embeddings.ct|dkr|j|jz|j|jzfn||f}|j|_|j|jz}|dz|_|j|_tj |j|jdz|j |_ tj |j|j|_ |j|_|jr|tdtjt#jd|dzdz ||_tjt#jd|dzdz ||_dSdS)Nrgr )rkzBInput size must be provided if using relative positional encoding.rGr)r5r6r7r8rr:rattention_dropoutdropoutr rWqkv_biasqkvproj use_rel_posrIrgr"rj rel_pos_h rel_pos_w)rBrC window_size input_sizehead_dimrDs r'r6zSamVisionAttention.__init__sZ a &"3 3V5F&J[5[ \ \{+  $*#= %)CCt^ / 9V/1Ca1Gfo^^^If0&2DEE !-   X! !efff \%+a*Q-6G!6KX*V*VWWDN\%+a*Q-6G!6KX*V*VWWDNNN  X Xr&q_sizek_sizerel_posr^cntdt||zdz }tj|d|jddddd|d}|d|dd}tj|dddft||z dz}tj|dddft||z dz}||z |dz t||z dzz}|| S) a Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (int): size of the query. k_size (int): size of key k. rel_pos (`torch.Tensor`): relative position embeddings (L, channel). Returns: Extracted positional embeddings according to relative positions. rGrrrlinear)sizemodeNg?) rmaxr interpolaterrHrJr"arangelong) rBrkrlrm max_rel_distrel_pos_resizedq_coordsk_coordsrelative_coordss r' get_rel_poszSamVisionAttention.get_rel_poss2 1s66222Q677 - OOAw}Q/ 4 4 < bhwkzbhwc,wkc->bhwkN)r{rHrr"einsum)rBrrrfrgrkrl query_height query_width key_height key_widthrelative_position_heightrelative_position_widthrLrrreshaped_queryrel_hrel_ws r'add_decomposed_rel_posz)SamVisionAttention.add_decomposed_rel_pos s$>%+! k & I#'#3#3L*i#X#X "&"2"2; 9"U"U"[ Asz<cRR -~?WXX -~?VWW||J k:yYYeAAAqqq!!!QQQ,--aaaAAAtQQQ6F0GG||J {(BJQZDZ[[ r&Frc |j\}}}}|||||zd|jdddddd}|d||jz||zdd\}} } ||jz| ddz} |jr(| | ||j |j ||f||f} tj j| tjd|j} tj| |j|j } | | z||j||d} | ddddd|||d} || } |r| | f}n| df}|S) Nr rrGrrr)rsr)ptraining)rHrcrrrJunbindrrrerrfrgr"r rtrfloat32rzrsrarrd)rBrrrLrMrNrrcrrr attn_weights attn_probs attn_outputrs r'rPzSamVisionAttention.forward8s'4':$ FE1 HH] # # WZ%D4Lb Q Q WQ1a # #  KK:8P+PRX[`R`bdeellmnoosE *cmmB.C.CC   66eT^T^fe_W]_dVeLx*22 > @ @ H HU_ajln o o &aaa&&5&!!!&;<GGII r&Frc|}||}|jdkr8|jd|jd}}|||j\}}|||\}}|jdkr|||j|||f}||z}||}|||z}|f} |r| |fz } | S)NrrrG)rr)rrhrHrrrrr) rBrrresidualrMrNrrlayernorm_outputrs r'rPzSamVisionLayer.forwards !((77  a  )/2M4G4JEF+/+@+@PTP`+a+a (M=&*ii'/'0' ' # |  a   33M4CSUbekmrdsttM =0 ++M::%1A(B(BB  "  '   &Gr&r)rrr r6r"r rrrrrrr#rPrQrRs@r'rr[s'''''0el00QVW\WcejknpsksetWtQu00000|25FKCQTHoglmprumugv B-2|$D> u ! r&rc*eZdZdeffd ZdZxZS) SamVisionNeckrCc`t||_tj|j|jdd|_t|jd|_ tj|j|jddd|_ t|jd|_ dS)NrF)r3rkrfrr )r3paddingrk) r5r6rCr r@r:output_channelsr-rbrr.rr]s r'r6zSamVisionNeck.__init__s  Yv163IWX_deee '(>L\]]]Yv5v7M[\fgnsttt '(>L\]]]r&c|dddd}||}||}||}||}|S)Nrr rrG)rJr-rr.rr`s r'rPzSamVisionNeck.forwardsi%--aAq99  =11 ((77  =11 ((77 r&)rrr rr6rPrQrRs@r'rrsZ^^^^^^^r&rc eZdZdeffd ZdZ d deejdee dee dee d e e e ff d Z xZS) SamVisionEncoderrCcVt||_|j|_t ||_d|_|jrMtj tj d|j|j z|j|j z|j |_tj|_t!|jD]=}t%|||jvr|jnd}|j|>t-||_d|_dS)Nrr)rhF)r5r6rCr7r0 patch_embed pos_embed use_abs_posr rgr"rjr8r:rrrrrglobal_attn_indexesrhrrneckgradient_checkpointing)rBrCrrrDs r'r6zSamVisionEncoder.__init__s$   +-f55   \ %)::%)::& DNmoo v/00 & &A"236;U2U2UF..[\E K  u % % % %!&)) &+###r&c|jSrV)rrBs r'get_input_embeddingsz%SamVisionEncoder.get_input_embeddingss r&NrKrrrr^cR||n |jj}||n |jj}||n |jj}|t d||}|j ||jz}|rdnd}|rdnd}t|jD]Z\}} |r||fz}|j r#|j r| | j |} n | ||} | d}|r || dfz}[|r||fz}| |}|s|f} |r| |fz} |r| |fz} | St|||S)Nz You have to specify pixel_valuesr%)rrr)rrr)rCrrrrIrr enumeraterrr_gradient_checkpointing_func__call__rr) rBrKrrrrall_hidden_statesall_self_attentionsr layer_module layer_outputsrs r'rPzSamVisionEncoder.forwards2C1N--TXT_Tq$8$D $+Jj &1% %)DN:M"6@BBD$5?bb4(55 P POA|# I$58H$H!* at} a $ A A )!!! !- ]N_ ` ` ` )!,M  P&9]1=M   59,0/3&* 9 9 u019 $D>9 'tn 9 d^ 9 u,, - 9 9 9 9 9 9 9 9 r&rc&eZdZeZdZdZdgZdZdS)SamPreTrainedModelsamrKr^c|jj}t|tjtjtjfrJ|jj d||j |j j dSdSt|tj rS|jj d||j -|jj|j  dSdSdS)NrE)rwstd)rCinitializer_ranger;r rWr@rridatanormal_rkzero_r padding_idx)rBmodulers r' _init_weightsz SamPreTrainedModel._init_weights3sk+ fry")R5GH I I ? M  & &CS & 9 9 9{&  &&((((('&  - - ? M  & &CS & 9 9 9!- "6#56<<>>>>> ? ?--r&N) rrr r config_classbase_model_prefixmain_input_name_no_split_modulesrr%r&r'rr-s<L$O-. ? ? ? ? ?r&ra< This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SamConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`SamProcessor`]. See [`SamProcessor.__call__`] for details. input_points (`torch.FloatTensor` of shape `(batch_size, num_points, 2)`): Input 2D spatial points, this is used by the prompt encoder to encode the prompt. Generally yields to much better results. The points can be obtained by passing a list of list of list to the processor that will create corresponding `torch` tensors of dimension 4. The first dimension is the image batch size, the second dimension is the point batch size (i.e. how many segmentation masks do we want the model to predict per input point), the third dimension is the number of points per segmentation mask (it is possible to pass multiple points for a single mask), and the last dimension is the x (vertical) and y (horizontal) coordinates of the point. If a different number of points is passed either for each image, or for each mask, the processor will create "PAD" points that will correspond to the (0, 0) coordinate, and the computation of the embedding will be skipped for these points using the labels. input_labels (`torch.LongTensor` of shape `(batch_size, point_batch_size, num_points)`): Input labels for the points, this is used by the prompt encoder to encode the prompt. According to the official implementation, there are 3 types of labels - `1`: the point is a point that contains the object of interest - `0`: the point is a point that does not contain the object of interest - `-1`: the point corresponds to the background We added the label: - `-10`: the point is a padding point, thus should be ignored by the prompt encoder The padding labels should be automatically done by the processor. input_boxes (`torch.FloatTensor` of shape `(batch_size, num_boxes, 4)`): Input boxes for the points, this is used by the prompt encoder to encode the prompt. Generally yields to much better generated masks. The boxes can be obtained by passing a list of list of list to the processor, that will generate a `torch` tensor, with each dimension corresponding respectively to the image batch size, the number of boxes per image and the coordinates of the top left and botton right point of the box. In the order (`x1`, `y1`, `x2`, `y2`): - `x1`: the x coordinate of the top left point of the input box - `y1`: the y coordinate of the top left point of the input box - `x2`: the x coordinate of the bottom right point of the input box - `y2`: the y coordinate of the bottom right point of the input box input_masks (`torch.FloatTensor` of shape `(batch_size, image_size, image_size)`): SAM model also accepts segmentation masks as input. The mask will be embedded by the prompt encoder to generate a corresponding embedding, that will be fed later on to the mask decoder. These masks needs to be manually fed by the user, and they need to be of shape (`batch_size`, `image_size`, `image_size`). image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_channels, window_size, window_size)`): Image embeddings, this is used by the mask decder to generate masks and iou scores. For more memory efficient computation, users can first retrieve the image embeddings using the `get_image_embeddings` method, and then feed them to the `forward` method instead of feeding the `pixel_values`. multimask_output (`bool`, *optional*): In the original implementation and paper, the model always outputs 3 masks per image (or per point / per bounding box if relevant). However, it is possible to just output a single mask, that corresponds to the "best" mask, by specifying `multimask_output=False`. attention_similarity (`torch.FloatTensor`, *optional*): Attention similarity tensor, to be provided to the mask decoder for target-guided attention in case the model is used for personalization as introduced in [PerSAM](https://arxiv.org/abs/2305.03048). target_embedding (`torch.FloatTensor`, *optional*): Embedding of the target concept, to be provided to the mask decoder for target-semantic prompting in case the model is used for personalization as introduced in [PerSAM](https://arxiv.org/abs/2305.03048). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zYSegment Anything Model (SAM) for generating segmentation masks, given an input image and z) optional 2D location and bounding boxes.ceZdZdgZfdZdZdZej dde e de e de e fd Z ej dd e ej d e ej d e ej d e ej fdZee dde ej d e ej d e ej d e ej d e ej de ej de de ej de ej de e de e de e deeeejffdZxZS)SamModelz4prompt_encoder.shared_embedding.positional_embeddingcFt|t|j|_t |j|_t|j|j|_ t|j |_ | dSrV)r5r6r vision_configshared_image_embeddingrvision_encoderr3prompt_encoder_configprompt_encoderrmask_decoder_config mask_decoder post_initr]s r'r6zSamModel.__init__s    &>qAAAr&NrrrcH|||||}|d}|S)a Returns the image embeddings by passing the pixel values through the vision encoder. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Input pixel values output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. rrrr)r)rBrKrrr vision_outputrs r'get_image_embeddingszSamModel.get_image_embeddingss=,++ /!5# ,  )+r&rTrUrVrWc8|||||}|S)a Returns the prompt embeddings by passing the input points, labels, boxes and masks through the prompt encoder. Args: input_points (`torch.FloatTensor` of shape `(batch_size, point_batch_size, num_points_per_image, 2)`): Optional input points for the prompt encoder. The padding of the point is automatically done by the processor. `point_batch_size` refers to the number of masks that we want the model to predict per point. The model will output `point_batch_size` times 3 masks in total. input_labels (`torch.LongTensor` of shape `(batch_size, point_batch_size, num_points_per_image)`): Optional input labels for the prompt encoder. The padding of the labels is automatically done by the processor, or can be fed by the user. input_boxes (`torch.FloatTensor` of shape `(batch_size, num_boxes_per_image, 4)`): Optional input boxes for the prompt encoder. The padding of the boxes is automatically done by the processor. users can also pass manually the input boxes. input_masks (`torch.LongTensor` of shape `(batch_size, image_size, image_size)`): Optional input masks for the prompt encoder. rTrUrVrW)r)rBrTrUrVrW prompt_outputs r'get_prompt_embeddingszSamModel.get_prompt_embeddingss22++%%## ,  r&TrKrrrrr^c H| | n |jj} | | n |jj} | | n |jj} ||t d||t d|@t |jdkr(t dd|j|@t |jdkr(t dd|j|E|C|jd }|jd }||kr#t d |||}| |jd n |jd }| |d d d }d}d}|5| || | | }|d }| r|d }| r|d }|8|6tj |ddddddd ftj |j}|Y|jd |jd kr=t dd|jd |jd ddd|||||\}}|||||||| | \}}}| s||f}| r||fz}| r|||fz}|St#|||||S)a Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoModel, AutoProcessor >>> model = AutoModel.from_pretrained("facebook/sam-vit-base") >>> processor = AutoProcessor.from_pretrained("facebook/sam-vit-base") >>> img_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/sam-car.png" >>> raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB") >>> input_points = [[[400, 650]]] # 2D location of a window on the car >>> inputs = processor(images=raw_image, input_points=input_points, return_tensors="pt") >>> # Get segmentation mask >>> outputs = model(**inputs) >>> # Postprocess masks >>> masks = processor.post_process_masks( ... outputs.pred_masks, inputs["original_sizes"], inputs["reshaped_input_sizes"] ... ) ``` Nz9Either pixel_values or image_embeddings must be provided.z>Only one of pixel_values and image_embeddings can be provided.rzlThe input_points must be a 4D tensor. Of shape `batch_size`, `point_batch_size`, `nb_points_per_image`, `2`.z got {}.r zMThe input_points must be a 3D tensor. Of shape `batch_size`, `nb_boxes`, `4`.rzQYou should provide as many bounding boxes as input points per box. Got {} and {}.rrrrFzNThe batch size of the image embeddings and the input points must be the same. zGot {} and {} respectively.zS if you want to pass multiple points for the same image, make sure that you passed zS input_points of shape (batch_size, point_batch_size, num_points_per_image, 3) and zK input_labels of shape (batch_size, point_batch_size, num_points_per_image)r)rrrrrrrr)r*r+r,r-r.)rCrrrrIlenrHformatrrrr" ones_likerrDrrr))rBrKrTrUrVrWrrrrrrrkwargsrbox_batch_sizerrLr-r,vision_outputsrZr1 low_res_masksiou_predictionsr.outputs r'rPzSamModel.forwardsT2C1N--TXT_Tq$8$D $+Jj &1%(?(?1(D(D~!!,"455   "s;+<'='='B'B_!!+"344   # (?+1!4 (.q1N>11 gnn(. '+&O&O&Q&Q#.:.F\'**L\LbcdLe &A&H&HUVXY[\&]&]# #  #!00"3%9' 1N .a0 # 9'5a'8$  7$22$6!  # (< ?<111aaa +C59]i]pqqqL  #(8(>q(A\EWXYEZ(Z(Z`-445E5KA5NP\PbcdPeffee]  /3.A.A%%## /B/ / ++CGBSBS-(C%6$4-!5-/CT C C ? (? %}5F# :#7"99  O#46M"NNM)&$!5/$;     r&rr) NNNNNNTNNNNN)rrr _tied_weights_keysr6rrr"no_gradrrrr# LongTensorrrSAM_INPUTS_DOCSTRINGrrstrr rPrQrRs@r'rrs QQ::: B B BU]__-1/3&*   $D> 'tn  d^    _ <U]__59373726 u01u/0e/0  e./ _@+*+?@@59483737268L 'tnL d^L  d3 $% &L L L A@L L L L L r&r)>r!r< dataclassesrtypingrrrrrnumpyr"r"torch.nn.functionalr rtrtorch.utils.checkpointr activationsr modeling_outputsr modeling_utilsrutilsrrrrconfiguration_samrrrr get_loggerrlogger_CONFIG_FOR_DOC_CHECKPOINT_FOR_DOCrr)Moduler0rTrbrrrrrrr*r3r^rrrrSAM_START_DOCSTRINGrrr%r&r'rs8!!!!!!55555555555555 !!!!!!//////------ffffffffffffgggggggggggg  H % %- ?????[?? ?:  L L L L L L L  LF        F     ")   29<<<<<<29<<<~TTTTTbiTTTnD-D-D-D-D-29D-D-D-NRY0{{{{{RY{{{|SSSSSRYSSS.     ry   6m3m3m3m3m3rym3m3m3`IIIIIIIIX\\\\\RY\\\~BI(\ \ \ \ \ ry\ \ \ ~????????$  CL_/ j j j j j !j j  j j j r&