gz dZddlZddlmZmZmZmZddlZddlZddlm Z ddl m Z ddl m Z mZddlmZdd lmZmZdd lmZmZmZmZmZd d lmZejeZd ZdZ gdZ!d=dej"de#de$dej"fdZ%Gdde j&Z'Gdde j&Z(Gdde j&Z)Gdde j&Z*Gdd e j&Z+Gd!d"e j&Z,Gd#d$e j&Z-Gd%d&e j&Z.Gd'd(e j&Z/Gd)d*eZ0d+Z1d,Z2ed-e1Gd.d/e0Z3Gd0d1e j&Z4Gd2d3e j&Z5Gd4d5e j&Z6Gd6d7e j&Z7Gd8d9e j&Z8ed:e1Gd;dzPyTorch GLPN model.N)ListOptionalTupleUnion)nn)ACT2FN)BaseModelOutputDepthEstimatorOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) GLPNConfigrzvinvino02/glpn-kitti)riFinput drop_probtrainingreturnc|dks|s|Sd|z }|jdfd|jdz zz}|tj||j|jz}||||z}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)r)dtypedevice)shapendimtorchrandrrfloor_div)rrr keep_probr random_tensoroutputs b/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/glpn/modeling_glpn.py drop_pathr*2sCx I I [^  Q 7 7E 5 EL Y Y YYM YYy ! !M 1F McjeZdZdZd deeddffd ZdejdejfdZ de fdZ xZ S) GLPNDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).NrrcVt||_dSN)super__init__r)selfr __class__s r)r1zGLPNDropPath.__init__Js$ "r+ hidden_statesc8t||j|jSr/)r*rr)r2r4s r)forwardzGLPNDropPath.forwardNs FFFr+c6d|jS)Nzp={})formatr)r2s r) extra_reprzGLPNDropPath.extra_reprQs}}T^,,,r+r/) __name__ __module__ __qualname____doc__rfloatr1r"Tensorr6strr9 __classcell__r3s@r)r-r-Gsbb##(5/#T######GU\GelGGGG-C--------r+r-c(eZdZdZfdZdZxZS)GLPNOverlapPatchEmbeddingsz+Construct the overlapping patch embeddings.cttj|||||dz|_tj||_dS)N kernel_sizestridepadding)r0r1rConv2dproj LayerNorm layer_norm)r2 patch_sizerI num_channels hidden_sizer3s r)r1z#GLPNOverlapPatchEmbeddings.__init__Ys[ I  "!O     ,{33r+c||}|j\}}}}|ddd}||}|||fS)NrFr)rLr flatten transposerN)r2 pixel_values embeddings_heightwidths r)r6z"GLPNOverlapPatchEmbeddings.forwardesgYY|,, (.1fe ''**44Q:: __Z00 65((r+r:r;r<r=r1r6rArBs@r)rDrDVsM55 4 4 4 4 4)))))))r+rDc2eZdZdZfdZdZ ddZxZS)GLPNEfficientSelfAttentionzSegFormer's efficient self-attention mechanism. Employs the sequence reduction process introduced in the [PvT paper](https://arxiv.org/abs/2102.12122).ct||_||_|j|jzdkr t d|jd|jdt |j|jz |_|j|jz|_tj |j|j|_ tj |j|j|_ tj |j|j|_ tj |j|_||_|dkr8tj|||||_tj||_dSdS)NrzThe hidden size (z6) is not a multiple of the number of attention heads ()r)rHrI)r0r1rQnum_attention_heads ValueErrorintattention_head_size all_head_sizerLinearquerykeyvalueDropoutattention_probs_dropout_probdropoutsr_ratiorKsrrMrNr2configrQr_sequence_reduction_ratior3s r)r1z#GLPNEfficientSelfAttention.__init__ts] &#6  d6 6! ; ;6D$4662666  $'t'7$:R'R#S#S !58PPYt/1CDD 9T-t/ABBYt/1CDD z&"EFF 0 #a ' 'iWoDG!l;77DOOO ( 'r+c|dd|j|jfz}||}|ddddS)NrrFrr)sizer_rbviewpermute)r2r4 new_shapes r)transpose_for_scoresz/GLPNEfficientSelfAttention.transpose_for_scoressY!&&(("-1I4Kc0dd %**955 $$Q1a000r+Fc"|||}|jdkr|j\}}}|ddd||||}||}|||dddd}||}|||} || |} tj || dd} | tj|jz } t j| d} || } tj | | } | dddd} | dd|jfz}| |} |r| | fn| f}|S)NrrrFrqdimr)rvrerkr rtreshaperlrNrfrgr"matmulrTmathsqrtrbr functionalsoftmaxrj contiguousrrrcrs)r2r4rXrYoutput_attentions query_layer batch_sizeseq_lenrP key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputss r)r6z"GLPNEfficientSelfAttention.forwards// =0I0IJJ =1  0=0C -J)11!Q::BB:|]cejkkM GGM22M)11*lBOOWWXY[\^_``M OOM::M--dhh}.E.EFF // =0I0IJJ !< Y5H5HR5P5PQQ+di8P.Q.QQ-//0@b/II,,77 _kBB %--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S%**+BCC 6G]=/22mM]r+F)r:r;r<r=r1rvr6rArBs@r)r\r\psj11888886111 ********r+r\c$eZdZfdZdZxZS)GLPNSelfOutputcttj|||_tj|j|_dSr/)r0r1rrddenserhhidden_dropout_probrj)r2rnrQr3s r)r1zGLPNSelfOutput.__init__sD Y{K88 z&"<== r+cZ||}||}|Sr/)rrj)r2r4 input_tensors r)r6zGLPNSelfOutput.forwards* =11  ]33 r+r:r;r<r1r6rArBs@r)rrsG>>>>> r+rc,eZdZfdZdZddZxZS) GLPNAttentionctt|||||_t |||_t |_dS)N)rnrQr_ro)rQ)r0r1r\r2rr(set pruned_headsrms r)r1zGLPNAttention.__init__sc .# 3%=     %VEEE EEr+ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrrry)lenr r2r_rbrrrerfrgr(rrcunion)r2headsindexs r) prune_headszGLPNAttention.prune_headss u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::r+Fc|||||}||d|}|f|ddz}|S)Nrr)r2r()r2r4rXrYr self_outputsattention_outputrs r)r6zGLPNAttention.forwardsMyy?PQQ ;;|A FF#% QRR(88r+r)r:r;r<r1rr6rArBs@r)rrs[ " " " " ";;;$r+rc&eZdZdfd ZdZxZS) GLPNDWConvc ttj||dddd||_dS)NrrT)biasgroups)r0r1rrKdwconv)r2rzr3s r)r1zGLPNDWConv.__init__s= iS!QSIII r+c|j\}}}|dd||||}||}|ddd}|S)NrrF)r rTrsrrS)r2r4rXrYrrrPs r)r6zGLPNDWConv.forwardsv,9,?) G\%//155:::|U[]bcc  M22 %--a00::1a@@ r+)rrrBs@r)rrsRJJJJJJr+rc&eZdZdfd ZdZxZS) GLPNMixFFNNct|p|}tj|||_t ||_t|jtrt|j|_ n |j|_ tj|||_ tj |j|_dSr/)r0r1rrddense1rr isinstance hidden_actr@r intermediate_act_fndense2rhrrj)r2rn in_featureshidden_features out_featuresr3s r)r1zGLPNMixFFN.__init__s #2{ i _==  11 f' - - 9'-f.?'@D $ $'-'8D $i>> z&"<== r+c||}||||}||}||}||}||}|Sr/)rrrrjr)r2r4rXrYs r)r6zGLPNMixFFN.forwardst M22  M65AA 00??  ]33  M22  ]33 r+)NNrrBs@r)rrsL > > > > > >r+rc*eZdZdZfdZddZxZS) GLPNLayerzCThis corresponds to the Block class in the original implementation.cttj||_t |||||_|dkrt|ntj|_ tj||_ t||z}t||||_ dS)N)rQr_ror)rr)r0r1rrM layer_norm_1r attentionr-Identityr* layer_norm_2rarmlp) r2rnrQr_r*ro mlp_ratiomlp_hidden_sizer3s r)r1zGLPNLayer.__init__ s L55& # 3%=     5>OOi000L55kI566f+___r+FcJ||||||}|d}|dd}||}||z}|||||}||}||z} | f|z}|S)N)rrr)rrr*rr) r2r4rXrYrself_attention_outputsrr mlp_output layer_outputs r)r6zGLPNLayer.forward.s!%   m , ,  / "0" " 2!4(, >>*:;;(=8 XXd// >>NN ^^J// !M1 /G+r+rrZrBs@r)rrsWMM ` ` ` ` `r+rc,eZdZfdZ ddZxZS) GLPNEncoderc <t|_dtjdjt jD}g}tj D]d}| tj |j ||dkrjnj|dz j|et!j||_g}d}tj D]}g}|dkr|j|dz z }tj|D]_}| t'j|j||||zj|j|`| t!j|t!j||_t!jfdtj D|_dS)Nc6g|]}|S)item).0xs r) z(GLPNEncoder.__init__..Ns ^^^Aqvvxx^^^r+rr)rOrIrPrQ)rQr_r*rorcNg|]!}tjj|"Sr)rrM hidden_sizes)rirns r)rz(GLPNEncoder.__init__..vs+ \ \ \aR\&-a0 1 1 \ \ \r+)r0r1rnr"linspacedrop_path_ratesumdepthsrangenum_encoder_blocksappendrD patch_sizesstridesrPrr ModuleListpatch_embeddingsrr_ sr_ratios mlp_ratiosblockrN) r2rndprrVrblockscurlayersjr3s ` r)r1zGLPNEncoder.__init__Is3  _^63H#fmJ\J\!]!]^^^ v011  A   *%1!4!>!,89Q!4!4FDWXY\]X]D^ & 3A 6     !# j 9 9v011 1 1AFAvvv}QU++6=+,,   $*$7$:,2,Fq,I"%cAg,171A!1D"("3A"6      MM"-// 0 0 0 0]6** - \ \ \ \5AZ;[;[ \ \ \  r+FTc@|rdnd}|rdnd}|jd}|}tt|j|j|jD]\} } | \} } } | |\}}}t| D])\}}|||||}|d}|r ||dfz}*| |}||||ddddd}|r||fz}|std|||fDSt|||S) NrrrrqrrFc3K|]}||V dSr/r)rvs r) z&GLPNEncoder.forward..s(mmq_`_l_l_l_l_lmmr+last_hidden_stater4 attentions) r enumerateziprrrNr{rtrtupler )r2rUroutput_hidden_states return_dictall_hidden_statesall_self_attentionsrr4idxrembedding_layer block_layer norm_layerrXrYrblk layer_outputss r)r6zGLPNEncoder.forwardys#7@BBD$5?bb4!'* $ D$94:t W WXX I IFC78 4O[*+:?=+I+I (M65#K00 T T3 #M65BS T T -a 0 $T*=qAQ@S*S'&J}55M)11*feRPPXXYZ\]_`bcddooqqM# I$58H$H! nmm]4EGZ$[mmmmm m++*    r+)FFTrrBs@r)rrHsX. . . . . f " $ $ $ $ $ $ $ $ r+rc(eZdZdZeZdZdZgZdZ dS)GLPNPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. glpnrUct|tjtjfrT|jjd|jj|j |j j dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|j j |jjddSdS)zInitialize the weightsr)meanstdNg?)rrrdrKweightdatanormal_rninitializer_rangerzero_ Embedding padding_idxrMfill_)r2modules r) _init_weightsz!GLPNPreTrainedModel._init_weightss0 fry")4 5 5 * M  & &CT[5R & S S S{&  &&((((('&  - - * M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r+N) r:r;r<r=r config_classbase_model_prefixmain_input_name_no_split_modulesrrr+r)rrsE L$O*****r+raG This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`GLPNConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aG Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`GLPNImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zfThe bare GLPN encoder (Mix-Transformer) outputting raw hidden-states without any specific head on top.ceZdZfdZdZeedee e e de  d de jdeed eed eed eee ff d ZxZS) GLPNModelct|||_t||_|dSr/)r0r1rnrencoder post_initr2rnr3s r)r1zGLPNModel.__init__sK     #6**  r+c|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrlayerrr)r2heads_to_prunerrs r) _prune_headszGLPNModel._prune_headssU +0022 C CLE5 L u % / ; ;E B B B B C Cr+z(batch_size, sequence_length)vision) checkpoint output_typer modalityexpected_outputNrUrrrrc||n |jj}||n |jj}||n |jj}|||||}|d}|s|f|ddzSt ||j|jS)Nrrrrrr)rnrruse_return_dictrr r4r)r2rUrrrencoder_outputssequence_outputs r)r6zGLPNModel.forwards 2C1N--TXT_Tq$8$D $+Jj &1% 'tn  d^  uo% &    ji     r+rc*eZdZdZdfd ZdZxZS)GLPNSelectiveFeatureFusionz Selective Feature Fusion module, as explained in the [paper](https://arxiv.org/abs/2201.07436) (section 3.4). This module adaptively selects and integrates local and global features by attaining an attention map for each feature. @c ttjtjt |dz|dddtj|tj|_tjtj|t |dz dddtjt |dz tj|_ tjt |dz dddd|_ tj |_ dS)NrFrr) in_channels out_channelsrHrIrJ) r0r1r SequentialrKra BatchNorm2dReLUconvolutional_layer1convolutional_layer2convolutional_layer3Sigmoidsigmoid)r2 in_channelr3s r)r1z#GLPNSelectiveFeatureFusion.__init__ s $&M I#j1n"5"5J\]fgqr s s s N: & & GII% % ! %'M I*3zA~;N;N\]fgqr s s s N3zA~.. / / GII% % ! %'IJN++!ST^_% % % !z|| r+ctj||fd}||}||}||}||}||dddddddfdz||dddddddfdzz}|S)Nrryr)r"catr1r2r3r5 unsqueeze)r2local_featuresglobal_featuresfeaturesattnhybrid_featuress r)r6z"GLPNSelectiveFeatureFusion.forward5s9no>AFFF,,X66,,X66,,X66||H%%(41aaa +;+E+Ea+H+HH?]a AAq!!!QQQJ^ )A,,Lr+)r*rZrBs@r)r)r)sV $$$$$$*r+r)c&eZdZfdZddZxZS)GLPNDecoderStagect||k}|stj||dntj|_t ||_tjddd|_ dS)Nr)rHrFbilinearF scale_factormode align_corners) r0r1rrKr convolutionr)fusionUpsampleupsample)r2r,r- should_skipr3s r)r1zGLPNDecoderStage.__init__Gs| !\1 Vat29[,ANNNNgigrgtgt0>>  SXYYY r+Nc||}||||}||}|Sr/)rGrHrJ)r2 hidden_stateresiduals r)r6zGLPNDecoderStage.forwardNsE'' 55  ;;|X>>L}}\22 r+r/rrBs@r)r@r@FsQZZZZZ        r+r@cZeZdZfdZdeejdeejfdZxZS) GLPNDecoderct|jddd}|jt jfd|D|_d|jd_t jddd|_ dS)Nrqc0g|]}t|Sr)r@)rrQr-s r)rz(GLPNDecoder.__init__..bs$ b b b[ k< 8 8 b b br+rrFrBFrC) r0r1rdecoder_hidden_sizerrstagesrHrIfinal_upsample)r2rnreserved_hidden_sizesr-r3s @r)r1zGLPNDecoder.__init__[s  & 3DDbD 91 m b b b bLa b b b   !% A kqzY^___r+r4rcg}d}t|ddd|jD]&\}}|||}||'|||d<|S)Nrq)rrTrrU)r2r4stage_hidden_statesstage_hidden_staterMstages r)r6zGLPNDecoder.forwardis !#&}TTrT':DK#H#H ; ; L%!&|5G!H!H   & &'9 : : : :"&"5"56H"I"IB""r+ r:r;r<r1rr"r?r6rArBs@r)rPrPZsm ` ` ` ` ` #T%,%7 #D>> from transformers import AutoImageProcessor, GLPNForDepthEstimation >>> import torch >>> import numpy as np >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("vinvino02/glpn-kitti") >>> model = GLPNForDepthEstimation.from_pretrained("vinvino02/glpn-kitti") >>> # prepare image for the model >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... predicted_depth = outputs.predicted_depth >>> # interpolate to original size >>> prediction = torch.nn.functional.interpolate( ... predicted_depth.unsqueeze(1), ... size=image.size[::-1], ... mode="bicubic", ... align_corners=False, ... ) >>> # visualize the prediction >>> output = prediction.squeeze().cpu().numpy() >>> formatted = (output * 255 / np.max(output)).astype("uint8") >>> depth = Image.fromarray(formatted) ```NTrrrF)rirtr4r) rnrrrr4rxrnr]r r) r2rUryrrrrr4outrtriloss_fctr(s r)r6zGLPNForDepthEstimation.forwards:f&1% S 'tn S d^ S  uU\"$88 9S S S _^hgS S S S S r+rv)rF):r=r}typingrrrrr"torch.utils.checkpointr activationsr modeling_outputsr r modeling_utilsr pytorch_utilsr rutilsrrrrrconfiguration_glpnr get_loggerr:loggerr$r#r%r?r>r'r*Moduler-rDr\rrrrrrrGLPN_START_DOCSTRINGr"rr)r@rPr]rkrvrr+r)rs //////////// !!!!!!EEEEEEEE------QQQQQQQQ+*****  H % %-)))U\eTV[Vb* - - - - -29 - - -))))))))4NNNNNNNNd     RY   #####BI###N        0((((( (((VU U U U U ")U U U p*****/***: $l9 9 9 9 9 #9 9 9 x))))))))Xry(#####")###6 *bi2g` ` ` ` ` 0` ` ` ` ` r+