gDz dZddlZddlmZddlmZmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZdd lmZmZmZmZmZmZd d lmZejeZ d Z!dZ"gdZ#eGddeZ$eGddeZ%Gdde j&Z'Gdde j&Z(Gdde j&Z)Gdde j&Z*dAde j+de,d e-d!e j+fd"Z.Gd#d$e j&Z/Gd%d&e j&Z0Gd'd(e j&Z1Gd)d*e j&Z2Gd+d,e j&Z3Gd-d.e j&Z4Gd/d0eZ5d1Z6d2Z7ed3e6Gd4d5e5Z8d6e j+d7e9d!e j+fd8Z:d6e j+d9e9d:e9d!e j+fd;Z;Gd<d=e j&Z<ed>e6Gd?d@e5Z=dS)BzPyTorch SegGpt model.N) dataclass)DictListOptionalTupleUnion)nn) functional)ACT2FN)PreTrainedModel) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings torch_int) SegGptConfigrzBAAI/seggpt-vit-large)r iiceZdZUdZejed<dZee ejed<dZ ee ejed<dZ ee ejed<dS)SegGptEncoderOutputa Output type of [`SegGptEncoderOutput`]. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, patch_height, patch_width, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`Tuple[torch.FloatTensor]`, `optional`, returned when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, patch_height, patch_width, hidden_size)`. attentions (`Tuple[torch.FloatTensor]`, `optional`, returned when `config.output_attentions=True`): Tuple of *torch.FloatTensor* (one for each layer) of shape `(batch_size, num_heads, seq_len, seq_len)`. intermediate_hidden_states (`Tuple[torch.FloatTensor]`, *optional*, returned when `config.intermediate_hidden_state_indices` is set): Tuple of `torch.FloatTensor` of shape `(batch_size, patch_height, patch_width, hidden_size)`. Each element in the Tuple corresponds to the output of the layer specified in `config.intermediate_hidden_state_indices`. Additionaly, each feature passes through a LayerNorm. last_hidden_stateN hidden_states attentionsintermediate_hidden_states) __name__ __module__ __qualname____doc__torch FloatTensor__annotations__rrrrrf/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/seggpt/modeling_seggpt.pyrr1s"((((82D/ L&% 4, , ,w  T_Q' ' '5DOA4F+F+FwVwwewwDO\]L^wwaeapqraswww __\22::1aAFF r$)rrrrr0rK __classcell__r>s@r%r+r+csV j j j j j       r$r+c eZdZdZdeddffd ZdededejfdZ dd ejd ejd e ej d e e dejf d Z xZS)SegGptEmbeddingszX Construct the embeddings from patch, position embeddings for input and prompt. r=returnNc8ttjt jddd|j|_tjt jddd|j|_tjt jddd|j|_ tjt jddd|j|_ tjt jddd|j|_ t||_ |j|jzdzdz}tjt jd||j|_tj|j|_dS)NrrB)r/r0r Parameterr zerosr4 mask_tokensegment_token_inputsegment_token_prompttype_token_semantictype_token_instancer+patch_embeddingspretrain_image_sizer2randnposition_embeddingsDropouthidden_dropout_probdropout)r<r= num_positionsr>s r%r0zSegGptEmbeddings.__init__s7 ,u{1aF@SUWXX``abdeghjklle_# O#**1aA66 6"**1feR@@ @r$rFprompt_pixel_valuesbool_masked_posembedding_typecX||}||}|j\}}} } |j||| d} |d| d|| d} |d| z z| | zz}||nd}||| } ||jz}||j z}|| z}|| z}|dkr|j }n |dkr|j }ntd|||z}||z}tj||fd}|S)NrcrinstancesemanticzBEmbedding type should be either 'semantic' or 'instance', but got rdim)rYrCrTexpand unsqueezetype_asrlrorUrVrWrXrDr cat)r<rFrprqrrinput_embeddingsprompt_embeddingsrG patch_height patch_width_rTw pos_embedtype_embeddingrJs r%rKzSegGptEmbeddings.forwards 00>> 112EFF3C3I0 L+q_++J kSUVV  % %b ) ) 1 1* = = E Eb,Xcef g g-Q7*q.H+9+E:11, LL ,d.FF-0II,i7- 9 Z ' '!5NN z ) )!5NNrbprrss s+n<->Y 02CD!LLL r$)NN)rrrrrr0intr Tensorror BoolTensorstrrKrLrMs@r%rOrOs>|>>>>>>> AsA3A5<AAAA,7;(, ++l+#\+"%"23 + ! +  ++++++++r$rOceZdZdZfdZdededejdejfdZdejd ejd ejd ejde eefde eefdejfd Z ddejdejfdZ xZ S)SegGptAttentionz=Multi-head Attention block with relative position embeddings.cRt|j|j}}t |t jjr|n||f}t |t jjr|n||f}|d|jz|d|jzf}|j|j z}|j |_ |dz|_ tj |j|jdz|j |_tj |j|j|_|j|_|jr|t#dtjt'jd|dzdz ||_tjt'jd|dzdz ||_dSdS)Nrrgr biaszBInput size must be provided if using relative positional encoding.rB)r/r0r1r2r5r6r7r8r4num_attention_headsscaler Linearqkv_biasqkvproj use_relative_position_embeddingsrDrRr rS rel_pos_h rel_pos_w)r<r=r1r2 input_sizehead_dimr>s r%r0zSegGptAttention.__init__s !'!2F4EJ #-j+/:R#S#SqZZZdfpYq #-j+/:R#S#SqZZZdfpYq  mv'88*Q-6K\:\] %)CC#)#= t^ 9V/1Ca1Gfo^^^If0&2DEE 060W-  0 X! !efff \%+a*Q-6G!6KX*V*VWWDN\%+a*Q-6G!6KX*V*VWWDNNN  X Xr$q_sizek_sizerel_posrPcntdt||zdz }tj|d|jddddd|d}|d|dd}tj|dddft||z dz}tj|dddft||z dz}||z |dz t||z dzz}|| S) a Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (int): size of the query. k_size (int): size of key k. rel_pos (`torch.Tensor`): relative position embeddings (L, channel). Returns: Extracted positional embeddings according to relative positions. rBrrrclinear)rerfN?) rmaxrjrkrlrCrEr arangelong) r<rrr max_rel_distrel_pos_resizedq_coordsk_coordsrelative_coordss r% get_rel_poszSegGptAttention.get_rel_poss2 1s66222Q677 - OOAw}Q/ 4 4 < bhwkzbhwc,wkc->bhwkN)rrCrlr einsum)r<rrrrrr query_height query_width key_height key_widthrelative_position_heightrelative_position_widthrGrrwreshaped_queryrel_hrel_ws r%add_decomposed_rel_posz&SegGptAttention.add_decomposed_rel_poss$>%+! k & I#'#3#3L*i#X#X "&"2"2; 9"U"U"[ Asz<cRR -~?WXX -~?VWW||J k:yYYeAAAqqq!!!QQQ,--aaaAAAtQQQ6F0GG||J {(BJQZDZ[[ r$Frc |j\}}}}|||||zd|jdddddd}|d||jz||zdd\}} } ||jz| ddz} |jr(| | ||j |j ||f||f} tj j| tjd|j} |rC| ||j||zd} | ||jz||zd} nd} | | z||j||d} | ddddd|||d} || } | | fS) Nr rcrBrr)dtyperw)rCrrlrrEunbindr transposerrrrr r r softmaxfloat32torviewr)r<routput_attentionsrGrHrIrrrkeyvalue attn_weightsattn_weights_reshaped attn_outputs r%rKzSegGptAttention.forwardCs'4':$ FE1 HH] # # WZ%D4Lb Q Q WQ1a # #  KK:8P+PRX[`R`bdeellmnoosE *cmmB.C.CC  0 66eT^T^fe_W]_dVeLx*22s r%r0zSegGptMlp.__init__ks\ If0&.AA Ifnf.@AA &+,r$rrPc||}||}||}|Sr)rrrr<rs r%rKzSegGptMlp.forwardqs; -00 //  -00 r$)rrrr0r rrKrLrMs@r%rrjs^----- U\elr$rFinput drop_probtrainingrPc|dks|s|Sd|z }|jdfd|jdz zz}|tj||j|jz}||||z}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)r)rdevice)rCndimr randrrfloor_div)rrr keep_probrC random_tensoroutputs r% drop_pathrysCx I I [^  Q 7 7E 5 EL Y Y YYM YYy ! !M 1F Mr$cjeZdZdZd deeddffd ZdejdejfdZ de fdZ xZ S) SegGptDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).NrrPcVt||_dSr)r/r0r)r<rr>s r%r0zSegGptDropPath.__init__s$ "r$rc8t||j|jSr)rrrrs r%rKzSegGptDropPath.forwards FFFr$c6d|jS)Nzp={})formatrr<s r% extra_reprzSegGptDropPath.extra_reprs}}T^,,,r$r) rrrrrfloatr0r rrKrrrLrMs@r%rrsbb##(5/#T######GU\GelGGGG-C--------r$rceZdZdededdffd Z d dejded e d e de e ejejfe ejff d Z xZ S) SegGptLayerr=drop_path_raterPNctt||_t ||_|dkrt |ntj|_ tj |j |j |_ tj |j |j |_dS)Nreps)r/r0r attentionrmlprr Identityr LayerNormr4layer_norm_epslayernorm_beforelayernorm_after)r<r=rr>s r%r0zSegGptLayer.__init__s (00V$$;IC;O;O777UWU`UbUb " V-?VEZ [ [ [!|F,>FDYZZZr$Fr ensemble_condfeature_ensemblerc||||}|d}|dd}|r|jddz|kr||jddzd\}} |dkra|jddz} | d| d} | dd| } | j|j} n*| dd| } tj|| gd}| ||z}|} | |}| |}| | |z}|f|z}|S) N)rrrrBrvrcT)rwkeepdim) rrrCsplitrlmean expand_asr r{rrr) r<rrrrself_attention_outputsattention_outputoutputspromptinputs num_promptsresiduals r%rKzSegGptLayer.forwards"&  ! !- 0 0/"0" " 2!4(,  B 0 6q 9Q >- O O-334D4J14MQR4RXY3ZZNFF!!.4Q71< ;;;D99CCFKK'6D99CCFKK$y&&)9qAAA '788=H  ,,];; //  4>>-#@#@@  "W,r$)FF)rrrrrr0r rrboolrrrKrLrMs@r%rrs[|[U[t[[[[[["'"' ##|## #  # uU\5</0% 2EE F ########r$rcleZdZdeddffd Z d dejded ed ed edee e ff d Z xZ S) SegGptEncoderr=rPNcvt|_dtjdjjDtjfdtjD|_ tj j j |_d|_dS)Nc6g|]}|Sr#)item).0xs r% z*SegGptEncoder.__init__..s dddAqvvxxdddr$rc<g|]}t|Sr#)r)r ir=dprs r%r z*SegGptEncoder.__init__..s'$j$j$jQ[Q%@%@$j$j$jr$rF)r/r0r=r linspacernum_hidden_layersr ModuleListrangelayersrr4r layernormgradient_checkpointing)r<r=rr>s `@r%r0zSegGptEncoder.__init__s  dd63H&Jb!c!cdddm$j$j$j$j$j%PVPhJiJi$j$j$jkk f&8f>STTT&+###r$FTrrroutput_hidden_states return_dictc|rdnd}|rdnd}g}t|jD]\} } |r||fz}|jj| krdnd} |jr&|jr|| j|| ||} n| || ||} | d}| |jjkr4|d|jddz||jddzdzdz}| |jj vr(| | ||r || dfz}|r||fz}|std||||fDSt||||S)Nr#rBrrrbc3K|]}||V dSrr#)r vs r% z(SegGptEncoder.forward..s0= ===r$)rrrr) enumeraterr= merge_indexrr_gradient_checkpointing_func__call__rC!intermediate_hidden_state_indicesappendrtupler) r<rrrrrall_hidden_statesall_self_attentionsrr  layer_moduler layer_outputss r%rKzSegGptEncoder.forwards#7@BBD$5?bb4%'"(55 P POA|# I$58H$H!"&!81! channels_lastc`ttjt j||_tjt j||_||_ ||_ |j dvrtd|j |f|_ dS)N)r*channels_firstzUnsupported data format: ) r/r0r rRr onesweightrSrr data_formatNotImplementedErrornormalized_shape)r<r1rr/r>s r%r0zSegGptLayerNorm.__init__s l5:.>#?#?@@ L-=!>!>?? &  #F F F%&T$BR&T&TUU U!1 3r$r rPc*|jdkr=tjj||j|j|j|j}n|jdkr|j }| }| dd}||z d dd}||z tj ||jzz }||}|jddddf|z|jddddfz}|S)Nr*r,rT)rrBr)r/r r r layer_normr1r.rrrrrpowsqrtr)r<r  input_dtypeuss r%rKzSegGptLayerNorm.forward!s   . .#..q$2GVZV_aeaijjAA  !1 1 1'K Aq$''AQ A##At#44AQ%*Q\222A;''A AAAtTM*Q.111dD=1IIAr$)r)r*) rrrrr0r rrKrLrMs@r%r(r(sm 444444  %,        r$r(c4eZdZfdZdejfdZxZS)SegGptDecoderHeadcJttj|j|jdd|_t |j|jd|_t|j |_ tj|jddd|_ dS)Nr r)r-paddingr,)r1rr/T)r-r) r/r0r r:decoder_hidden_sizeconvr(rrr ract_fctheadrs r%r0zSegGptDecoderHead.__init__0s I  &  &     )#7V=R`p   f/0 If8!QUVVV r$rc||}||}||}||}|Sr)r?rr@rArs r%rKzSegGptDecoderHead.forward>sL -00 }55  ]33  -00 r$)rrrr0r r!rKrLrMs@r%r;r;/s[ W W W W WU%6r$r;cXeZdZfdZdejdejfdZdejfdZxZS) SegGptDecoderc:ttj|jt |jz|jdz|jzd|_ t||_ |j|_|j|_||_ dS)NrBTr) r/r0r rr4lenr r2r> decoder_embedr; decoder_predr=rs r%r0zSegGptDecoder.__init__Hs Y  V%M!N!N N  q 6#= =    .f55 +#)#=  r$rrPc|j\}}}}|||||j|j|j}|dddddd}||d||jz||jzf}|S) Nrrr rBrrcrC)rCrlr2r>rE)r<rrGr~rrs r%_reshape_hidden_statesz$SegGptDecoder._reshape_hidden_statesTs3@3F0 L+q%--  k4?DOUYUm  &--aAq!Q?? %--r<$/#A;QUQ`C`a.  r$c||}||}||}|Sr)rGrLrHrs r%rKzSegGptDecoder.forward`sA**=99 33MBB ))-88 r$) rrrr0r r!rLrKrLrMs@r%rDrDGs~      E4E %J[    U%6r$rDcjeZdZdZeZdZdZdZddgZ de e j e j e jfdd fd Zd S) SegGptPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. modelrFTrOrmodulerPNc|jj}t|tjtjfrtj|jj tj d| |jj |j_ |j |jj dSdSt|tjr?|jj |jj ddSt|t$rtj|jj tj d| |jj |j_ tj|jj tj d| |jj |j_ dSt|t*rItj|jj tj d| |jj |j_ tjj|j|tjj|j|tjj|j|tjj|j|tjj|j|dSdS)zInitialize the weightsr)rstdNr)rS)r=initializer_ranger5r rr:init trunc_normal_r.datarr rrrzero_rfill_rrrrOr\normal_rTrUrVrWrX)r<rQrSs r% _init_weightsz#SegGptPreTrainedModel._init_weightstsk+ fry")4 5 5# G"$!6!6v}7I7L7LU]7[7[bekn!6!o!o!r!r #""FM {&  &&((((('&  - - G K  " " $ $ $ M  $ $S ) ) ) ) )  0 0 G$&G$9$9 %((77%:%%b!'((   ! %'G$9$9 %((77%:%%b!'((   ! ! !  0 1 1 G.0g.C.C*/225=AA/D//b+122  & + HM ! !&"3 ! = = = HM ! !&"<# ! F F F HM ! !&"=3 ! G G G HM ! !&"<# ! F F F HM ! !&"<# ! F F F F F G Gr$)rrrrr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modulesrr rr:rr[r#r$r%rOrOhs~  L$O&*#+];&GE")RY *L$M&GRV&G&G&G&G&G&Gr$rOaH This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SegGptConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a  Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`SegGptImageProcessor.__call__`] for details. prompt_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Prompt pixel values. Prompt pixel values can be obtained using [`AutoImageProcessor`]. See [`SegGptImageProcessor.__call__`] for details. prompt_masks (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Prompt mask. Prompt mask can be obtained using [`AutoImageProcessor`]. See [`SegGptImageProcessor.__call__`] for details. bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). feature_ensemble (`bool`, *optional*): Boolean indicating whether to use feature ensemble or not. If `True`, the model will use feature ensemble if we have at least two prompts. If `False`, the model will not use feature ensemble. This argument should be considered when doing few-shot inference on an input image i.e. more than one prompt for the same image. embedding_type (`str`, *optional*): Embedding type. Indicates whether the prompt is a semantic or instance embedding. Can be either instance or semantic. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare SegGpt Model transformer outputting raw hidden-states without any specific head on top.c~eZdZdeffd ZdefdZdeee efddfdZ e e e ee dd ejd ejd ejd eejd eedeedeejdeedeedeedeeeffdZxZS) SegGptModelr=ct|||_t||_t ||_|dSr)r/r0r=rOrJrencoder post_initrs r%r0zSegGptModel.__init__sX     *622$V,,  r$rPc|jjSr)rJrYrs r%get_input_embeddingsz SegGptModel.get_input_embeddingss //r$heads_to_pruneNc|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrdlayerr prune_heads)r<rhrkheadss r% _prune_headszSegGptModel._prune_headssU +0022 C CLE5 L u % / ; ;E B B B B C Cr$ output_typer\rFrp prompt_masksrqrrrlabelsrrrc ||n |jj}| | n |jj} | | n |jj} ||nd}|jjjjj} | | }| | }tj ||fd}|tj ||fdntj ||fd}||t d|h|jjj} tj| tj |j}d|| dzd<|d}||||| } || ||| | }|S) a labels (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`, `optional`): Ground truth mask for input images. Returns: Examples: ```python >>> from transformers import SegGptImageProcessor, SegGptModel >>> from PIL import Image >>> import requests >>> image_input_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_2.jpg" >>> image_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1.jpg" >>> mask_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1_target.png" >>> image_input = Image.open(requests.get(image_input_url, stream=True).raw) >>> image_prompt = Image.open(requests.get(image_prompt_url, stream=True).raw) >>> mask_prompt = Image.open(requests.get(mask_prompt_url, stream=True).raw).convert("L") >>> checkpoint = "BAAI/seggpt-vit-large" >>> model = SegGptModel.from_pretrained(checkpoint) >>> image_processor = SegGptImageProcessor.from_pretrained(checkpoint) >>> inputs = image_processor(images=image_input, prompt_images=image_prompt, prompt_masks=mask_prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> list(outputs.last_hidden_state.shape) [1, 56, 28, 1024] ``` NFrBrvzLabels were provided, but bool_masked_pos were not. It will be set to default value. If you're training the model, make sure to provide a bool_masked_pos.r3rr)rrrq)rrrr)r=rruse_return_dictrJrYr;r.rrr r{logger warning_oncer9rSrrryrd)r<rFrprqrqrrrrrrrrexpected_dtyper9embedding_outputencoder_outputss r%rKzSegGptModel.forwards^2C1N--TXT_Tq$8$D $+Jj &1%_]]] 7;+/(,.2,0/3&*[[l[#\[l [ "%"23 [ #4. [! [*+[$D>['tn[d^[ u)) *[[[^]DC[[[[[r$rbtensorr2c|j\}}}}||z}||z}|||||||f}|dddddd}||||z|dzdzf}|S)NrKrrBrr rJr)rCrlrE)r}r2rGr3rHrIr~rs r%patchifyrGs.4l+J feZ'L:%K ^^:|\:Wbdn"o^ p pF ^^Aq!Q1 - -F ^^:|k/I:WX=[\K\"]^ ^ ^F Mr$r~rc ||jd}t|jddz dz}||z|jdkr$td|jdd|d|d ||||||df }|dd ddd d }||d||z||zf }|S)Nrrcr rbrzNumber of patches z does not match patch height (z ) and width (rArKrJrBr)rCrrDrlrE)r}r~rrGr2s r% unpatchifyrSsaJfl2&*s233Jk!V\!_44 za z zP\ z zkv z z z   ^^:|[*V`bc"d^ e eF ^^Aq!Q1 - -F ^^:q,2K[[eMe"f^ g gF Mr$c^eZdZfdZdejdejdejdejfdZxZS) SegGptLosscxt|j|_|j|_dSr)r/r0betar2rs r%r0zSegGptLoss.__init__cs0 K  +r$rqr)rrrqctj||fd}|dddddfdd|jdzdz}t ||jd|jz|jd|jz}t j||d|j}||z | z }|S)aNComputes the L1 loss between the predicted masks and the ground truth masks. Args: prompt_masks (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values from mask prompt. pred_masks (`torch.FloatTensor` of shape `(batch_size, num_channels, 2*height, width)`): Predicted masks. labels (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Ground truth mask for input images. bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: `torch.FloatTensor`: The mean L1 loss between the predicted masks and the ground truth masks. rBrvNrr none) reductionr) r r{repeatr2rrCrjsmooth_l1_lossrsum)r<rqr)rrrq ground_truthmaskr(s r%rKzSegGptLoss.forwardhs2y,!7Q??? qqq!!!Tz*11!Q8JQ8NOO$ 21 5 H,J\]^J_cgcrJrss LFQUQZ[[[t   ""TXXZZ/ r$) rrrr0r r!rrKrLrMs@r%rrbs~,,,,, !'!%!! ! ) !!!!!!!!r$rzCSegGpt model with a decoder on top for one-shot image segmentation.cFeZdZdeffd Zeeeee  dde j de j de j de e j d e ed e ed e e jd e ed e ede edeeeffdZxZS)SegGptForImageSegmentationr=ct|||_t||_t ||_|dSr)r/r0r=rbrPrDdecoderrers r%r0z#SegGptForImageSegmentation.__init__sX      (( $V,,  r$roNrFrprqrqrrrrrrrrrPc ||n |jj}| | n |jj} | | n |jj} |m|jjjj} tj | tj  |j }d|| dzd<| d}|||||||||| |  } | r| jn| d} tj| d} || }d}|"t#|j}|||||}| s/|f}| r || dfz}|r| rdnd}|| |fz}||f|z}|St%||| j| j S) aM labels (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`, `optional`): Ground truth mask for input images. Returns: Examples: ```python >>> from transformers import SegGptImageProcessor, SegGptForImageSegmentation >>> from PIL import Image >>> import requests >>> image_input_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_2.jpg" >>> image_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1.jpg" >>> mask_prompt_url = "https://raw.githubusercontent.com/baaivision/Painter/main/SegGPT/SegGPT_inference/examples/hmbb_1_target.png" >>> image_input = Image.open(requests.get(image_input_url, stream=True).raw) >>> image_prompt = Image.open(requests.get(image_prompt_url, stream=True).raw) >>> mask_prompt = Image.open(requests.get(mask_prompt_url, stream=True).raw).convert("L") >>> checkpoint = "BAAI/seggpt-vit-large" >>> model = SegGptForImageSegmentation.from_pretrained(checkpoint) >>> image_processor = SegGptImageProcessor.from_pretrained(checkpoint) >>> inputs = image_processor(images=image_input, prompt_images=image_prompt, prompt_masks=mask_prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> result = image_processor.post_process_semantic_segmentation(outputs, target_sizes=[image_input.size[::-1]])[0] >>> print(list(result.shape)) [170, 297] ``` Nr3rrBr) rFrprqrqrrrrrrrrrcrv)r(r)rr)r=rrrtrPrJrYr9r rSrrrryrr{rrr'rr)r<rFrprqrqrrrrrrrrr9rrr)r(loss_fnridxs r%rKz"SegGptForImageSegmentation.forwards^2C1N--TXT_Tq$8$D $+Jj &1%a 'tna d^a  u33 4a a a hgDCa a a a a r$r)rF)>rcollections.abcr6 dataclassesrtypingrrrrrr torch.utils.checkpointr torch.nnr rj activationsr modeling_utilsr utilsrrrrrrconfiguration_seggptr get_loggerrrur|_CHECKPOINT_FOR_DOC_EXPECTED_OUTPUT_SHAPErr'Moduler+rOrrrrrrrrrr(r;rDrOSEGGPT_START_DOCSTRINGr{rbrrrrrr#r$r%rsQ!!!!!!55555555555555 $$$$$$!!!!!!------/.....  H % %!.& JJJJJ+JJ J0 :::::K:: :0     BI   FRRRRRryRRRjK4K4K4K4K4biK4K4K4^        U\eTV[Vb* - - - - -RY - - -,,,,,"),,,^B B B B B BIB B B Lbi< 0BIB2G2G2G2G2GO2G2G2Gj "Jfsssss'ss sl U\ s u|     u| 3 S U\    ''''''''TIn n n n n !6n n n n n r$