gSdZddlmZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZd d lmZd d lmZmZmZmZejeZd Zd Zdejdejdedededeejejff dZdejdejdedeejejffdZdejdedejfdZeGddeZ Gddej!Z"Gddej!Z#Gdd ej!Z$Gd!d"ej!Z%Gd#d$e Z&d%Z'd&Z(ed'e'Gd(d)e&Z)dS)*zPyTorch SuperPoint model.) dataclass)OptionalTupleUnionN)nn)PreTrainedModel)BaseModelOutputWithNoAttention)SuperPointConfig)#is_torch_greater_or_equal_than_1_13) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingr zmagic-leap-community/superpoint keypointsscoresborderheightwidthreturnc|dddf|k|dddf||z kz}|dddf|k|dddf||z kz}||z}||||fS)zPRemoves keypoints (and their associated scores) that are too close to the borderNr)rrrrrmask_hmask_wmasks n/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/superpoint/modeling_superpoint.pyremove_keypoints_from_bordersr,s1o'IaaadOv,O PF1o'IaaadOuv~,N OF F?D T?F4L ((kcx|t|kr||fStj||d\}}|||fS)z(Keeps the k keypoints with highest scorer)dim)lentorchtopk)rrr indicess rtop_k_keypointsr'6sHC NN&  j222OFG W v %%r nms_radiuscldkrtdfd}tj|}|||k}tdD]Q}||dk}tj|||}|||k}|||zz}Rtj|||S)z)Applies non-maximum suppression on scoresrz'Expected positive values for nms_radiuscVtj|dzdzdS)Nr kernel_sizestridepadding)r functional max_pool2d)xr(s rmax_poolzsimple_nms..max_poolCs-}''zA~7IRS]g'hhhrr+) ValueErrorr$ zeros_likerangefloatwhere) rr(r3zerosmax_mask_ supp_mask supp_scores new_max_masks ` r simple_nmsr?>sA~~BCCCiiiii  V $ $E&)))H 1XX<<HX^^--..2 k)UF;; "hh{&;&;; | z:; ;x / //rceZdZUdZdZeejed<dZ eej ed<dZ eejed<dZ eejed<dZ eejed<dZeeejed<dS) #SuperPointKeypointDescriptionOutputa Base class for outputs of image point description models. Due to the nature of keypoint detection, the number of keypoints is not fixed and can vary from image to image, which makes batching non-trivial. In the batch of images, the maximum number of keypoints is set as the dimension of the keypoints, scores and descriptors tensors. The mask tensor is used to indicate which values in the keypoints, scores and descriptors tensors are keypoint information and which are padding. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*): Loss computed during training. keypoints (`torch.FloatTensor` of shape `(batch_size, num_keypoints, 2)`): Relative (x, y) coordinates of predicted keypoints in a given image. scores (`torch.FloatTensor` of shape `(batch_size, num_keypoints)`): Scores of predicted keypoints. descriptors (`torch.FloatTensor` of shape `(batch_size, num_keypoints, descriptor_size)`): Descriptors of predicted keypoints. mask (`torch.BoolTensor` of shape `(batch_size, num_keypoints)`): Mask indicating which values in keypoints, scores and descriptors are keypoint information. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states (also called feature maps) of the model at the output of each stage. Nlossrr descriptorsr hidden_states)__name__ __module__ __qualname____doc__rBrr$ FloatTensor__annotations__r IntTensorrrCr BoolTensorrDrrrrrArAPs2)-D(5$ %,,,+/Ix(///*.FHU& '.../3K%+,333'+D(5# $+++8td|_g}|t ||j|jddt dt|jdz D]A}|t ||j|dz |j|dB|t ||jd|jddtj ||_ dS)NrrT)rRF) rWrX input_dimappendrNencoder_hidden_sizesr6r#r ModuleList conv_blocks)r`rOrsiras rrXzSuperPointEncoder.__init__s7   8STU8Vdh i i i   q#f9::Q>??  A   #F7A>@[\]@^lp      3B79TUW9Xfk      =55rFToutput_hidden_states return_dictc|rdnd}|jD]}||}|r||fz}|}|std||fDSt||S)Nrc3K|]}||V dSrcr.0vs r z,SuperPointEncoder.forward..s"QQq1=====QQr)last_hidden_staterD)rstupler )r`inputrurvall_hidden_states conv_blockoutputs rrdzSuperPointEncoder.forwards #7@BBD* A AJJu%%E# A$5$@! RQQV->$?QQQQQ Q-$+    r)FT) rErFrGrHr rXrrfrrr rdrhris@rrkrks 6/6D666666205&*   'tn d^  u44 5         rrkceZdZdZdeddffd ZdejdeejejffdZ dejdejfdZ d ejdeejejffd Z xZ S) SuperPointInterestPointDecodera The SuperPointInterestPointDecoder uses the output of the SuperPointEncoder to compute the keypoint with scores. The scores are first computed by a convolutional layer, then a softmax is applied to get a probability distribution over the 65 possible keypoint classes. The keypoints are then extracted from the scores by thresholding and non-maximum suppression. Post-processing is then applied to remove keypoints too close to the image borders as well as to keep only the k keypoints with highest score. rOrNct|j|_|j|_|j|_|j|_t jd|_t j dd|_ t j |j d|j ddd|_t j |j |jddd |_dS NTrTr+rVrnr rr,r)rWrXkeypoint_threshold max_keypointsr(border_removal_distancerr\r]r^r_rYrqdecoder_hidden_size conv_score_akeypoint_decoder_dim conv_score_br`rOras rrXz'SuperPointInterestPointDecoder.__init__s "(";#1 +'-'E$GD))) LQq999 I  ' +  &    I  &(CQR[\fg   rencodedcd||}||\}}||fSrc)_get_pixel_scores_extract_keypoints)r`rrrs rrdz&SuperPointInterestPointDecoder.forwards8''00 33F;; 6&  rc|||}||}tj|dddddf}|j\}}}}|dddd|||dd}|ddddd||dz|dz}t||j }|S) zKBased on the encoder output, compute the scores for each pixel of the imagerNrnrr+r ) r]rrrr0softmaxshapepermutereshaper?r()r`rr batch_sizer;rrs rrz0SuperPointInterestPointDecoder._get_pixel_scoress4,,W5566""6**&&vq11!!!SbS&9'-|$ Avu1a++33JqRSTT1aA..66z6A:uWXyYYFDO44 rrc|j\}}}tj|d|jk}|dt |}t |||j|dz|dz\}}|jdkrt|||j\}}tj |dg }||fS)ztBased on their scores, extract the pixels that represent the keypoints that will be used for descriptors computationrrr) rr$nonzerorr~trrrr'flipr7)r`rr;rrrs rrz1SuperPointInterestPointDecoder._extract_keypointss!<65M&)d.E"EFF 5//0: vt;VaZQR   6   " " / 64CU V V IvJy1#..4466 &  r) rErFrGrHr rXr$rgrrdrrrhris@rrrs / D      (!u|!elEL6P0Q!!!!  %,    !!% el@Z:[!!!!!!!!rrceZdZdZdeddffd ZdejdejdejfdZe d d e dejfd Z xZ S) SuperPointDescriptorDecoderag The SuperPointDescriptorDecoder uses the outputs of both the SuperPointEncoder and the SuperPointInterestPointDecoder to compute the descriptors at the keypoints locations. The descriptors are first computed by a convolutional layer, then normalized to have a norm of 1. The descriptors are then interpolated at the keypoints locations. rOrNc^ttjd|_tjdd|_tj|jd|j ddd|_ tj|j |j ddd |_ dSr) rWrXrr\r]r^r_rYrqrconv_descriptor_adescriptor_decoder_dimconv_descriptor_brs rrXz$SuperPointDescriptorDecoder.__init__s GD))) LQq999 !#  ' +  & " " " "$  &  ) " " " rrrcJ||||}tj|dd}||d|dddd}tj|dd}|S)zXBased on the encoder output and the keypoints, compute the descriptors for each keypointr+rpr"Nrr) rr]rrr0 normalize_sample_descriptorsr$ transpose)r`rrrCs rrdz#SuperPointDescriptorDecoder.forward%s,,TYYt7M7Mg7V7V-W-WXX m--kQA-FF ..y At@TVWXXYZ[ ok1a88 rrscalec|j\}}}}||dz z dz}tj||z|dz z dz ||z|dz z dz gg}||}||z}|dzdz }trddini}||ddd}t jj||fddi|}| ||d}t j |dd }|S) z-Interpolate descriptors at keypoint locationsr+g?r align_cornersTrnmodebilinearr) rr$tensortor viewrr0 grid_samplerr) rrCrr num_channelsrrdivisorkwargss rrz/SuperPointDescriptorDecoder._sample_descriptors1s3>2C/ L&% )C/ ,%%-%!)";c"AVe^V[^_V_E_beEe ghii**Y''W MA% ,OW/4((UWNN:q"a88 m/ YbbZb[abb !))*lBGG m--kQA-FF r)r) rErFrGrHr rXr$rgrd staticmethodrerrhris@rrrs / D      ( u|      3u|\rrceZdZdZeZdZdZdZde e j e j e j fddfdZdejdejfd ZdS) SuperPointPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. superpoint pixel_valuesFmodulerNct|tjtjfrT|jjd|jj|j |j j dSdSt|tj r?|j j |jj ddSdS)zInitialize the weightsg)meanstdNg?) isinstancerLinearrYweightdatanormal_rOinitializer_rangebiaszero_ LayerNormfill_)r`rs r _init_weightsz'SuperPointPreTrainedModel._init_weightsOs fry")4 5 5 * M  & &CT[5R & S S S{&  &&((((('&  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *rcF|dddddddfdddddddfS)aG Assuming pixel_values has shape (batch_size, 3, height, width), and that all channels values are the same, extract the first channel value to get a tensor of shape (batch_size, 1, height, width) for SuperPoint. This is a workaround for the issue discussed in : https://github.com/huggingface/transformers/pull/25786#issuecomment-1730176446 Args: pixel_values: torch.FloatTensor of shape (batch_size, 3, height, width) Returns: pixel_values: torch.FloatTensor of shape (batch_size, 1, height, width) Nrr)r`rs r extract_one_channel_pixel_valuesz:SuperPointPreTrainedModel.extract_one_channel_pixel_values[s<AAAq!!!QQQJ'4AAA 66r)rErFrGrHr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingrrrrYrrr$rIrrrrrrDs $L$$O&+# *E")RY *L$M *RV * * * *7U=N7SXSd777777rraP This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SuperPointConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a- Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`SuperPointImageProcessor`]. See [`SuperPointImageProcessor.__call__`] for details. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z6SuperPoint model outputting keypoints and descriptors.ceZdZdZdeddffd Zee d dej de ej de e d e e de eeff d ZxZS) SuperPointForKeypointDetectiona SuperPoint model. It consists of a SuperPointEncoder, a SuperPointInterestPointDecoder and a SuperPointDescriptorDecoder. SuperPoint was proposed in `SuperPoint: Self-Supervised Interest Point Detection and Description `__ by Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. It is a fully convolutional neural network that extracts keypoints and descriptors from an image. It is trained in a self-supervised manner, using a combination of a photometric loss and a loss based on the homographic adaptation of keypoints. It is made of a convolutional encoder and two decoders: one for keypoints and one for descriptors. rOrNct|||_t||_t ||_t||_| dSrc) rWrXrOrkencoderrkeypoint_decoderrdescriptor_decoder post_initrs rrXz'SuperPointForKeypointDetection.__init__sg     (00 >v F F"=f"E"E rrlabelsrurvc .d}|td||n jj}||n jj}|}|jd}|||}|d}fd|D} d| D} d| D} fdt|| D} td | D} tj || d f|j }tj || f|j }tj || jj f|j }tj || f|j tj }tt| | | D]Y\}\}}}|||d|jdf<|||d|jdf<|||d|jdf<d ||d|jdf<Z|r|d nd}|std||||||fDSt!||||||S)a Examples: ```python >>> from transformers import AutoImageProcessor, SuperPointForKeypointDetection >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("magic-leap-community/superpoint") >>> model = SuperPointForKeypointDetection.from_pretrained("magic-leap-community/superpoint") >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) ```Nz-SuperPoint does not support training for now.r)rurvcFg|]}|dS)N.)r)rzr}r`s r z:SuperPointForKeypointDetection.forward..s:! ! ! DUD ! !"3I"> ? ?! ! ! rcg|] }|d S)rrrzkeypoints_scoress rrz:SuperPointForKeypointDetection.forward..s\\\2B*1-\\\rcg|] }|d S)rrrs rrz:SuperPointForKeypointDetection.forward..sYYY/?'*YYYrcZg|]'\}}|d|d(Sr)r)rzr}rr`s rrz:SuperPointForKeypointDetection.forward..sI   ,!9  # #$5i$@)IBV W W   rc30K|]}|jdVdS)rN)r)rzrs rr|z9SuperPointForKeypointDetection.forward..s)#W#W9IOA$6#W#W#W#W#W#Wrr+)device)rdtyperc3K|]}||V dSrcrrys rr|z9SuperPointForKeypointDetection.forward..s(qqqcdcpcpcpcpcpqqr)rBrrrCrrD)r4rOruuse_return_dictrrrzipmaxr$r9rrre enumerater~rA)r`rrrurvrBrencoder_outputsr}list_keypoints_scoreslist_keypoints list_scoreslist_descriptorsmaximum_num_keypointsrrrCrrt _keypoints_scores _descriptorsrDs` rrdz&SuperPointForKeypointDetection.forwards4  LMM M%9$D $+Jj &1%|GZbgbklll6?NT_aq@r@r6s6s , , 2A2 G\2rsa !!!!!!)))))))))) ((((((UTTTTT@@@@@@  H % %$7)|)%*\);>)HK)TW) 5< %&))))&u|&U\&c&eTYT`bgbnTnNo&&&&0u|000000$ =====+== =D")k k k k k r