gqvdZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZdd lmZmZdd lmZd dlmZejeZdZ eGddeZ!Gdde j"Z#Gdde j"Z$Gdde j"Z%Gdde j"Z&Gdde j"Z'Gdde j"Z(Gdde j"Z)dFd!Z*Gd"d#e j"Z+Gd$d%e j"Z,Gd&d'e j"Z-e j.j/dGd*e0d+e1fd,Z2Gd-d.e j"Z3Gd/d0e j"Z4Gd1d2e j"Z5Gd3d4e j"Z6Gd5d6e j"Z7Gd7d8e j"Z8Gd9d:e j"Z9Gd;de j"Z;Gd?d@eZedCe=GdDdEe<Z?dS)HzPyTorch ZoeDepth model.N) dataclass)ListOptionalTupleUnion)nn)ACT2FN)add_start_docstrings%add_start_docstrings_to_model_forwardreplace_return_docstrings)DepthEstimatorOutput)PreTrainedModel) ModelOutputlogging) load_backbone)ZoeDepthConfigrceZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ee ejdfed<dZ ee ejdfed<dS) ZoeDepthDepthEstimatorOutputa Extension of `DepthEstimatorOutput` to include domain logits (ZoeDepth specific). Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. predicted_depth (`torch.FloatTensor` of shape `(batch_size, height, width)`): Predicted depth for each pixel. domain_logits (`torch.FloatTensor` of shape `(batch_size, num_domains)`): Logits for each domain (e.g. NYU and KITTI) in case multiple metric heads are used. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, num_channels, height, width)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, patch_size, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlosspredicted_depth domain_logits. hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrrj/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/zoedepth/modeling_zoedepth.pyrr,s2)-D(5$ %,,,)-OU&---'+M5$+++=AM8E%"3S"89:AAA:>Ju0#567>>>>>r$rc^eZdZdZfdZdeejdeejfdZxZ S)ZoeDepthReassembleStageaE This class reassembles the hidden states of the backbone into image-like feature representations at various resolutions. This happens in 3 stages: 1. Map the N + 1 tokens to a set of N tokens, by taking into account the readout ([CLS]) token according to `config.readout_type`. 2. Project the channel dimension of the hidden states according to `config.neck_hidden_sizes`. 3. Resizing the spatial dimensions (height, width). Args: config (`[ZoeDepthConfig]`): Model configuration class defining the model architecture. c 6t|j|_tj|_t |j|jD]/\}}|j t|||0|jdkr|tj|_ |j }|jD]W}|j tj tjd|z|t|jVdSdS)N)channelsfactorproject)super__init__ readout_typer ModuleListlayerszipneck_hidden_sizesreassemble_factorsappendZoeDepthReassembleLayerreadout_projectsbackbone_hidden_size SequentialLinearr hidden_act)selfconfigneck_hidden_sizer* hidden_size_ __class__s r%r.z ZoeDepthReassembleStage.__init__^s "/moo (+F,DfF_(`(` j j $ f K  6vHXaghhh i i i i  ) + +$&MOOD ! 5K-  %,,M")A O["I"I6RXRcKdee , +  r$rreturnc|djd}tj|d}|dddf|ddddf}}|j\}}}|||||}|dddd}|jdkri|dd}|d |} tj|| fd }n#|jd kr||d z}g} t| |dD]~\} } |jdkr|j | | } | ddd|d ||} |j | | } | | | S) z Args: hidden_states (`List[torch.FloatTensor]`, each of shape `(batch_size, sequence_length + 1, hidden_size)`): List of hidden states from the backbone. rdimNrr r,r+)rr,radd)shaper catreshapepermute contiguousr/flatten unsqueeze expand_as enumeratesplitr7r1r5) r<r patch_height patch_width batch_size cls_tokentotal_batch_sizesequence_length num_channelsreadoutout stage_idx hidden_states r%forwardzZoeDepthReassembleStage.forwardos #1%+A.  -Q777 #0A#6 aaae8L= :G:M7/<%--.> k[ghh %--aAq99DDFF   ) ))11!44<CCMM  % ' ')I,?,?,C,CCM'01D1DZUV1D1W1W'X'X % % #I| I--?t4Y? MM (//1a88@@RQ]_jkkL14;y1,??L JJ| $ $ $ $ r$ rrrrr.rr Tensorr] __classcell__rAs@r%r'r'Nsu  "&T%,%7&W[\a\hWi&&&&&&&&r$r'c$eZdZfdZdZxZS)r6c t|j}tj||d|_|dkr tj||||d|_dS|dkrtj|_dS|dkr0tj||dtd|z d|_dSdS)Nr) in_channels out_channels kernel_sizerrfstridepaddingr ) r-r.r8rConv2d projectionConvTranspose2dresizeIdentityint)r<r=r)r*r?rAs r%r.z ZoeDepthReassembleLayer.__init__s 1 ) (`abbb A::,XxV\blmnnnDKKK q[[+--DKKK aZZ)HhAcRSV\R\ooghiiiDKKKZr$cZ||}||}|SN)rkrmr<r\s r%r]zZoeDepthReassembleLayer.forwards*|44 {{<00 r$rrrr.r]r`ras@r%r6r6sL j j j j j r$r6c$eZdZfdZdZxZS)ZoeDepthFeatureFusionStagecttj|_t t |jD])}|jt|*dSrq) r-r.rr0r1rangelenr3r5ZoeDepthFeatureFusionLayer)r<r=r@rAs r%r.z#ZoeDepthFeatureFusionStage.__init__sx moo s634455 C CA K  9&AA B B B B C Cr$c|ddd}g}|jd|d}||t|dd|jddD]&\}}|||}||'|S)NrFrr)r1r5r2)r<rfused_hidden_statesfused_hidden_stater\layers r%r]z"ZoeDepthFeatureFusionStage.forwards%ddd+  +T[^M!,<==""#5666#&}QRR'8$+abb/#J#J ; ; L%!&'9*>x*H*HHL++L99 }00 qzI[1  |44 r$)Trqrrrrr.r]r`ras@r%ryrysaCCCCCCr$ryc^eZdZdZfdZdeejdeejfdZxZ S) ZoeDepthNeckaO ZoeDepthNeck. A neck is a module that is normally used between the backbone and the head. It takes a list of tensors as input and produces another list of tensors as output. For ZoeDepth, it includes 2 stages: * ZoeDepthReassembleStage * ZoeDepthFeatureFusionStage. Args: config (dict): config dict. c t||_|j|jjdvrd|_nt ||_tj|_ |j D]8}|j tj ||j ddd9t||_dS)N)swinv2r rF)rfrir)r-r.r=backbone_config model_typereassemble_stager'rr0convsr3r5rjrru fusion_stage)r<r=channelrAs r%r.zZoeDepthNeck.__init__9s    ! -&2H2SWa2a2a$(D ! !$;F$C$CD !]__ / s sG J  bi1JXYcdkpqqq r r r r7v>>r$rrBc|t|ttfstdt |t jjkrtdj|||}fdt|D} |}||dfS)z Args: hidden_states (`List[torch.FloatTensor]`, each of shape `(batch_size, sequence_length, hidden_size)` or `(batch_size, hidden_size, height, width)`): List of hidden states from the backbone. z2hidden_states should be a tuple or list of tensorszOThe number of hidden states should be equal to the number of neck hidden sizes.NcBg|]\}}j||Sr#)r).0ifeaturer<s r% z(ZoeDepthNeck.forward..Zs-VVVzq'MDJqM'**VVVr$rF) isinstancetuplelist TypeErrorrxr=r3 ValueErrorrrPr)r<rrRrSfeaturesoutputs` r%r]zZoeDepthNeck.forwardJs -%77 RPQQ Q }  T[%B!C!C C Cnoo o  , 11-{[[MVVVVY}=U=UVVV""8,,x|##r$r^ras@r%rr,su  ?????"$T%,%7$W[\a\hWi$$$$$$$$r$rcReZdZdZfdZdeejdejfdZxZ S)#ZoeDepthRelativeDepthEstimationHeada Relative depth estimation head consisting of 3 convolutional layers. It progressively halves the feature dimension and upsamples the predictions to the input resolution after the first convolutional layer (details can be found in DPT's paper's supplementary material). ct|j|_d|_|jrt jddddd|_|j}t j||dzddd|_t j ddd |_ t j|dz|j ddd|_ t j|j dddd |_ dS) N)r r )rrrgr,r rrTrr)r-r. head_in_indexrkadd_projectionrrjrconv1Upsampleupsamplenum_relative_featuresconv2conv3)r<r=rrAs r%r.z,ZoeDepthRelativeDepthEstimationHead.__init__is #1   e iSfV]cdddDO,YxQAaYZ[[[  SWXXX Yx1}f.JXYbcmnooo Yv;QAVWabccc r$rrBc||j}|j1||}tj|}||}||}||}tj|}|}||}tj|}|d}||fS)NrrD) rrkrrrrrrsqueeze)r<rrrs r%r]z+ZoeDepthRelativeDepthEstimationHead.forwardxs%d&89 ? & OOM::M%BGIIm44M =11  m44  =11 ! -00   =11 ! -00 '//A/66((r$r^ras@r%rrbsr d d d d d)T%,%7)EL))))))))r$rHz>c||z}||z}|tj|z|tj|zz ||z tj||z |zzz S)z%log(nCk) using stirling approximation)r log)nkrs r% log_binomrsY CA CA uy|| a%)A,,. .!a%59QUS[;Q;Q1Q QQr$c6eZdZdejffd ZddZxZS)LogBinomialSoftmaxrc xt||_||_|dt jd|ddddd|dt j|jdz gddddddS) a7Compute log binomial distribution for n_classes Args: n_classes (`int`, *optional*, defaults to 256): Number of output classes. act (`torch.nn.Module`, *optional*, defaults to `torch.softmax`): Activation function to apply to the output. k_idxrrrFF) persistent k_minus_1N) r-r.ractregister_bufferr arangeviewr_)r< n_classesrrAs r%r.zLogBinomialSoftmax.__init__s  Wel1i&@&@&E&EaQPQ&R&R_deee [%, |*D*D*I*I!RQRTU*V*Vchiiiiir$?-C6?c|jdkr|d}tjd|z |d}tj||d}t |j|j|jtj|zz|j|jz tj|zz}|||z dS)aCompute the log binomial distribution for probabilities. Args: probabilities (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Tensor containing probabilities of each class. temperature (`float` or `torch.Tensor` of shape `(batch_size, num_channels, height, width)`, *optional*, defaults to 1): Temperature of distribution. eps (`float`, *optional*, defaults to 1e-4): Small number for numerical stability. Returns: `torch.Tensor` of shape `(batch_size, num_channels, height, width)`: Log binomial distribution logbinomial(p;t). r rrD) ndimrNr clamprrrrr)r< probabilities temperaturerone_minus_probabilitiesys r%r]zLogBinomialSoftmax.forwards   " ")33A66M"'+a-.?a"H"H M3:: dndj 1 1j59]333 4~ *ei8O.P.PP Q xxKQx///r$)rr)rrrr softmaxr.r]r`ras@r%rrs^!$%- j j j j j j00000000r$rc*eZdZ dfd ZdZxZS)%ZoeDepthConditionalLogBinomialSoftmaxrr,c t||z|z}tjtj||z|dddtjtj|ddddtj|_d|_|j |_ |j |_ t|tj |_dS)aPer-pixel MLP followed by a Conditional Log Binomial softmax. Args: in_features (`int`): Number of input channels in the main feature. condition_dim (`int`): Number of input channels in the condition feature. n_classes (`int`, *optional*, defaults to 256): Number of classes. bottleneck_factor (`int`, *optional*, defaults to 2): Hidden dim factor. rrrgr)rN)r-r.rr9rjGELUSoftplusmlpp_epsmax_tempmin_temprr rlog_binomial_transform)r<r= in_features condition_dimrbottleneck_factor bottleneckrAs r%r.z.ZoeDepthConditionalLogBinomialSoftmax.__init__s* !M16GG = IkM1:1UV`a b b b GII Ij%Qq! L L L KMM      &8 &V&V&V###r$c|tj||fd}|dddddf|dddddf}}||jz}|ddddf|ddddf|ddddfzz }||jz}|ddddf|ddddf|ddddfzz }|d}|j|jz |z|jz}|||S)az Args: main_feature (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Main feature. condition_feature (torch.Tensor of shape `(batch_size, num_channels, height, width)`): Condition feature. Returns: `torch.Tensor`: Output log binomial distribution rrDNr,.r)rr concatrrNrrr)r< main_featurecondition_featureprobabilities_and_temperaturerrs r%r]z-ZoeDepthConditionalLogBinomialSoftmax.forwards@)-|M^>_ef1g1g1g(h(h% )!!!RaR* 5 )!!!QRR* 5# & 2 %aaaCi0M!!!Q)4L}]^]^]^`acf]fOg4gh !DJ. !!!!Q), AAAq#I0FUVUVUVXY[^U^I_0_` !++A.. }t}4 CdmS **=+FFFr$)rr,rsras@r%rrsb  #W#W#W#W#W#WJGGGGGGGr$rc&eZdZdfd ZdZxZS)ZoeDepthSeedBinRegressorrMbP? ct|j|_|j|_||_||_tj|j|ddd|_ tj d|_ tj||ddd|_ |jdkrtj dntj |_dS)adBin center regressor network. Can be "normed" or "unnormed". If "normed", bin centers are bounded on the (min_depth, max_depth) interval. Args: config (`int`): Model configuration. n_bins (`int`, *optional*, defaults to 16): Number of bin centers. mlp_dim (`int`, *optional*, defaults to 256): Hidden dimension. min_depth (`float`, *optional*, defaults to 1e-3): Min depth value. max_depth (`float`, *optional*, defaults to 10): Max depth value. rrTinplacenormedN)r-r.bottleneck_featuresrbin_centers_type min_depth max_depthrrjrract1rract2)r<r=n_binsmlp_dimrrrAs r%r.z!ZoeDepthSeedBinRegressor.__init__s" !5 & 7""Yt/!QBB GD))) Yw1a88 -1-Bh-N-NBGD))))TVT_TaTa r$c||}||}||}||}|jdkr|dz}||ddz }|j|jz |z}tj |dd|j}tj |d }d |d d d d d f|d d dd d fzz}||fS||fS)z] Returns tensor of bin_width vectors (centers). One vector b for every pixel rrrTrEkeepdim)rrrrrrconstant)rvaluerDg?NrF.) rrrrrsumrrrrpadr cumsum)r<x bin_centersbin_widths_normed bin_widths bin_edgess r%r]z ZoeDepthSeedBinRegressor.forward"s  JJqMM IIaLL JJqMMiill  H , ,%,K +koo!To.R.R R .4>9=NNJ**:7IPZbfbp*qqJ ZQ777I111crc3;!7)AAAqrr3J:O!OPK$k1 1 + +r$)rrrrrsras@r%rrsRbbbbbb:,,,,,,,r$r,r,alphagammac^|d|||zzS)a:Inverse attractor: dc = dx / (1 + alpha*dx^gamma), where dx = a - c, a = attractor point, c = bin center, dc = shift in bin center This is the default one according to the accompanying paper. Args: dx (`torch.Tensor`): The difference tensor dx = Ai - Cj, where Ai is the attractor point and Cj is the bin center. alpha (`float`, *optional*, defaults to 300): Proportional Attractor strength. Determines the absolute strength. Lower alpha = greater attraction. gamma (`int`, *optional*, defaults to 2): Exponential Attractor strength. Determines the "region of influence" and indirectly number of bin centers affected. Lower gamma = farther reach. Returns: torch.Tensor: Delta shifts - dc; New bin centers = Old bin centers + dc r)divpow)dxrrs r% inv_attractorr<s*" 66!ebffUmm++ , ,,r$c0eZdZ d fd Zd dZxZS) ZoeDepthAttractorLayerrrrFct|j|_|j|_|j|_||_||_ ||_ ||_ ||_ |j x}}tj||ddd|_tjd|_tj||dzddd|_tjd|_dS)zq Attractor layer for bin centers. Bin centers are bounded on the interval (min_depth, max_depth) rrTrr,N)r-r.attractor_alpharattractor_gammagemmaattractor_kindkind n_attractorsrrrmemory_efficientbin_embedding_dimrrjrrrrr r<r=rr$rrr%rrrAs r%r.zZoeDepthAttractorLayer.__init__Qs + + ) ( "" 0!' 88 gY{GQ1== GD))) Yw q(8!QBB GD))) r$NTcn|7|r0tj||jdddd}||z}||}||}||}||}|dz}|j\}}}} |||j d|| }|dddddd f} tj||| fdd} |j shtj tj d |j} | t| d| d z d } nztj| | j } t'|j D]5}| t| dd|d fd | z z } 6|jdkr | |j z } | | z}|j|jz |z|jz} tj| d \} }tj| |j|j} || fS)ao The forward pass of the attractor layer. This layer predicts the new bin centers based on the previous bin centers and the attractor points (the latter are predicted by the MLP). Args: x (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): Feature block. prev_bin (`torch.Tensor` of shape `(batch_size, prev_number_of_bins, height, width)`): Previous bin centers normed. prev_bin_embedding (`torch.Tensor`, *optional*): Optional previous bin embeddings. interpolate (`bool`, *optional*, defaults to `True`): Whether to interpolate the previous bin embeddings to the size of the input features. Returns: `Tuple[`torch.Tensor`, `torch.Tensor`]: New bin centers normed and scaled. NrTrrrr,r.meanr rrDdevicer,)rrrrHrrrrrr$r%r r,r r#rrN zeros_liker.rwrrsortclip)r<rprev_binprev_bin_embeddingr attractorsrTr@heightwidthattractors_normedrfuncdelta_crbin_new_centerss r%r]zZoeDepthAttractorLayer.forwardpsP&  ) %']%>%>& :UY&?&&"&&A JJqMM IIaLL JJqMMYYq\\ $& '1'7$ Avu__Z1BAvuUU 'qqq!!!Q|4m//65/PZjn/oo $ 6!Juy99$)DDd=):)D)DQ)G)G+J_J_`aJbJb)bccijkkkGG&{;;MNNNG4,-- b b=):111a9)E)O)OPQ)R)RU`)`aaayF""!D$55%/~6/IDNZ KQ777 QjdndnMM  ++r$)rrrFNTrsras@r%rrPs_ ******><,<,<,<,<,<,<,<,r$rc0eZdZ dfd Zd dZxZS) ZoeDepthAttractorLayerUnnormedrrrTct||_||_||_||_|j|_|j|_|j |_ ||_ |j x}}tj||ddd|_tjd|_tj||ddd|_tj|_dS)zL Attractor layer for bin centers. Bin centers are unbounded rrTrN)r-r.r$rrrrrrr"r#r%r&rrjrrrrrrr's r%r.z'ZoeDepthAttractorLayerUnnormed.__init__s ( ""+ + ) 0 & 88 gY{GQ1== GD))) Yw aA>> KMM r$Ncr|7|r0tj||jdddd}||z}||}||}||}||}|jdd\}}tj|||fdd}|jshtj tj d|j } | t|d|dz d } nztj||j } t#|jD]5} | t|dd| d fd|z z } 6|j d kr | |jz } || z} | }| |fS) a The forward pass of the attractor layer. This layer predicts the new bin centers based on the previous bin centers and the attractor points (the latter are predicted by the MLP). Args: x (`torch.Tensor` of shape (batch_size, num_channels, height, width)`): Feature block. prev_bin (`torch.Tensor` of shape (batch_size, prev_num_bins, height, width)`): Previous bin centers normed. prev_bin_embedding (`torch.Tensor`, *optional*): Optional previous bin embeddings. interpolate (`bool`, *optional*, defaults to `True`): Whether to interpolate the previous bin embeddings to the size of the input features. Returns: `Tuple[`torch.Tensor`, `torch.Tensor`]: New bin centers unbounded. Two outputs just to keep the API consistent with the normed version. Nr)rTr*r+r,rrDr-.r,)rrrrHrrrrr%r r,r r#rrNr/r.rwr$) r<rr2r3rr4r5r6rr8r9rr:s r%r]z&ZoeDepthAttractorLayerUnnormed.forwards&  ) %']%>%>& :UY&?&&"&&A JJqMM IIaLL JJqMMYYq\\ "(- m//65/PZjn/oo $ 6!Juy99$)DDd=)=)=a)@)@;CXCXYZC[C[)[\\bcdddGG&{;;MNNNG4,-- [ [=AAAq#I)>)H)H)K)Kk)YZZZyF""!D$55%/%  ++r$)rrrTr;rsras@r%r=r=s_ """""":3,3,3,3,3,3,3,3,r$r=cDeZdZdfd ZdejdejfdZxZS)ZoeDepthProjectorcttj||ddd|_tjd|_tj||ddd|_dS)aProjector MLP. Args: in_features (`int`): Number of input channels. out_features (`int`): Number of output channels. mlp_dim (`int`, *optional*, defaults to 128): Hidden dimension. rrTrN)r-r.rrjrrrr)r<r out_featuresrrAs r%r.zZoeDepthProjector.__init__se Y{GQ1== 74(((Yw aA>> r$r\rBc||}||}||}|Srq)rrrrrs r%r]zZoeDepthProjector.forwards;zz,// xx -- zz,// r$)rB)rrrr.r r_r]r`ras@r%rArAsc??????"ELU\r$rAceZdZdZfdZdejdejfdZ ddejd ejd ejd eej d ee de ejf d Z xZ S)ZoeDepthMultiheadAttentionzKEquivalent implementation of nn.MultiheadAttention with `batch_first=True`.c t||zdkrtd|d|d||_t ||z |_|j|jz|_tj||j|_ tj||j|_ tj||j|_ tj|||_ tj ||_dS)NrzThe hidden size (z6) is not a multiple of the number of attention heads ())r-r.rnum_attention_headsroattention_head_size all_head_sizerr:querykeyr out_projDropoutdropout)r<r?rJrQrAs r%r.z#ZoeDepthMultiheadAttention.__init__"s  , , 1 11K11-111  $7 #&{5H'H#I#I !58PPY{D,>?? 9[$*<==Y{D,>??  +{;; z'** r$rrBc|dd|j|jfz}||}|ddddS)NrFrr,rr )rrJrKrrK)r<r new_x_shapes r%transpose_for_scoresz/ZoeDepthMultiheadAttention.transpose_for_scores6sPffhhssmt'?AY&ZZ FF;  yyAq!$$$r$NFquerieskeysvaluesattention_maskoutput_attentionsc(|||}|||}|||}t j||dd} | tj|j z } || |z} tj | d} | | } t j| |} | dddd} | dd|jfz} | | } || } |r| | fn| f} | S)NrFr)rDrr,rr )rTrMrNr r matmul transposemathsqrtrKrrrrQrKrLrrLrrO)r<rUrVrWrXrY query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputss r%r]z"ZoeDepthMultiheadAttention.forward;sy// 70C0CDD --dhhtnn== // 60B0BCC !< Y5H5HR5P5PQQ+di8P.Q.QQ  %/.@ -//0@b/II,,77 _kBB %--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S%**+BCC  m44 6G]=/22mM]r$)NF)rrrrr.r r_rTrr!boolrr]r`ras@r%rGrGsUU+++++(%el%u|%%%%7;,1 %%%l% % !!23 % $D> % u| %%%%%%%%r$rGcFeZdZdfd Z ddeejfdZxZS) ZoeDepthTransformerEncoderLayer皙?reluc&t|j}|j}|j}t ||||_tj|||_ tj ||_ tj|||_ tj ||_tj ||_tj ||_tj ||_t$||_dS)N)rQ)r-r.patch_transformer_hidden_size#patch_transformer_intermediate_size%patch_transformer_num_attention_headsrG self_attnrr:linear1rPrQlinear2 LayerNormnorm1norm2dropout1dropout2r activation)r<r=rQrxr?intermediate_sizerJrAs r%r.z(ZoeDepthTransformerEncoderLayer.__init__ds : "F$J3KAT^efffy.?@@ z'** y!2K@@ \+.. \+..  7++  7++  ,r$Nsrc_maskc |x}}|||||d}|||z}||}|||||}|||z}||}|S)N)rUrVrWrXr) rprvrtrrrQrxrqrwru)r<srcrzrUrVsrc2s r%r]z'ZoeDepthTransformerEncoderLayer.forwardxs $~~gDU]~^^_`aDMM$'''jjoo||DLLc9J9J)K)KLLMMDMM$'''jjoo r$)rjrkrq) rrrr.rr r_r]r`ras@r%riricsl------.,0  5<(        r$ric:eZdZfdZdejfdZdZxZS)ZoeDepthPatchTransformerEncoderc tj}tjfdt jD|_tj|j ddd|_ dS)zViT-like transformer block Args: config (`ZoeDepthConfig`): Model configuration class defining the model architecture. c.g|]}tSr#)ri)rr@r=s r%rz.s" i i i ,V 4 4 i i ir$rrrgN) r-r.rrr0rwnum_patch_transformer_layerstransformer_encoderrjrmembedding_convPxP)r<r=rdrAs ` r%r.z(ZoeDepthPatchTransformerEncoder.__init__s 0 #%= i i i ieFDg>h>h i i i$ $  "$ =1UV`a" " " r$cpuc tjd|||d}tjd|d||d}tj|tjtjd| |z z}||z} tjtj| tj| gd} | d |dd} | S)zGenerate positional encodings Args: sequence_length (int): Sequence length embedding_dim (int): Embedding dimension Returns: torch.Tensor: Positional encodings. r)dtyper.rr,g@r-rD) r rrNexprtensorrIsincosrepeat) r<rTrW embedding_dimr.rpositionindexdiv_term pos_encodings r%positional_encoding_1dz6ZoeDepthPatchTransformerEncoder.positional_encoding_1ds<?%OOOYYZ[\\ Q qfMMMWWXYZZ9Uuygf1U1U1U'V'V&VYf&fghh(* y%)L"9"959\;R;R!SYZ[[[ #--!-44;;J1MM r$cx||d}tj|d}|ddd}|j\}}}||||||j|j z}tdD]}|j ||}|S)zForward pass Args: x (torch.Tensor - NCHW): Input feature tensor Returns: torch.Tensor - Transformer output embeddings of shape (batch_size, sequence_length, embedding_dim) r,)rrrr)r.rr) rrMrrr rKrHrr.rrwr)r<r embeddingsrTrWrrs r%r]z'ZoeDepthPatchTransformerEncoder.forwards++A..66q99 ]&&z6:: ''1a00 5?5E2 O]$"="= z?PXbXh#># #  q A AA41!4Z@@JJr$) rrrr.r float32rr]r`ras@r%rrsh     &Y^ejer$r$rc&eZdZdfd ZdZxZS)ZoeDepthMLPClassifierrBNct|}tj|||_tj|_tj|||_dSrq)r-r.rr:rqrrxrr)r<rrDhidden_featuresrAs r%r.zZoeDepthMLPClassifier.__init__sU %yo>> '))y,?? r$c||}||}||}|Srq)rqrxrr)r<r\rs r%r]zZoeDepthMLPClassifier.forwards;||L11 |44  \22 r$)rBNrsras@r%rrsR@@@@@@r$rc(eZdZdZfdZdZxZS)*ZoeDepthMultipleMetricDepthEstimationHeadszn Multiple metric depth estimation heads. A MLP classifier is used to route between 2 different heads. c4tjjj|_j|_j}tj||ddd|_ t|_ tdd|_ |jdkrtn|jdkrttjfd jD|_t%|dz |_tjfd t+d D|_tjfd jD|_jtjfdjD|_dS)NrrrgrBr,rrDrsoftplusc pi|]2}|dt|ddz|d|d3S)namerr,rr)rrrr)r)rconfr&r=s r% zGZoeDepthMultipleMetricDepthEstimationHeads.__init__..sc   V 6>-2";/";/    r$rrDrcDg|]}tjdzS)r,rrArrr@r&r=s r%rzGZoeDepthMultipleMetricDepthEstimationHeads.__init__..sK     " & 9!2-2   r$rc i|]FdtjfdttDGS)rc Rg|]#}|dd$S)rrrrrr#)rr Attractorr= configurationr$s r%rzRZoeDepthMultipleMetricDepthEstimationHeads.__init__...sV " "#/?&3K&@&3K&@ r$)rr0rwrx)rrrr=r$s @r%rzGZoeDepthMultipleMetricDepthEstimationHeads.__init__..s   "f%r}"'s<'8'8!9!9 ( (   r$c Ri|]#}|dt|dd$S)rrr)r)r)rrr&r=last_ins r%rzGZoeDepthMultipleMetricDepthEstimationHeads.__init__..$sV   "f%'L%!(+&' (((   r$)r-r.r&num_attractorsbin_configurationsrrrrjrrpatch_transformerrmlp_classifierrr= ModuleDictseed_bin_regressorsrAseed_projectorr0rw projectorsr4rconditional_log_binomial)r<r=rrr&rr$rAs ` @@@@r%r.z3ZoeDepthMultipleMetricDepthEstimationHeads.__init__s "4, "("; & 7%8Y24GUV_`jklll "A!H!H3RSTTT  H , ,.II  "j 0 06I$&=     #5    $ $  0+:KUfjkUk   -      q       -      &,%>     ".(*       &,%>    ) ) %%%r$c||}||dddddf}||}tj|ddd}d|jD} | tj|d  fd|jDd} n!#t$rtdd wxYw| d } | d } |j } | |\}}|j d vr || z | | z z }n|}||}|j}t!|j||D]'\}}}||}||||d \}}|}|}(|}t$j||jdddd}t$j||jdddd}|j}|||}tj||zdd}||fS)NrTrrFrDcg|] }|d Srr#)rrs r%rzFZoeDepthMultipleMetricDepthEstimationHeads.forward..<sTTT=v&TTTr$c,g|]}|dk|Srr#)rr=bin_configurations_names r%rzFZoeDepthMultipleMetricDepthEstimationHeads.forward..@s(nnnvF6NVmDmDmFDmDmDmr$zbin_configurations_name z! not found in bin_configurationssrrrhybrid2rr)rr*r)rrrr rr rargmaxritem IndexErrorrrrrr4r2rrrrrHr)r<outconv_activationrfeature_blocksrelative_depthr embeddingr domain_votenamesrrrseed_bin_regressorr@seed_bin_centersr2r3r4 projector attractorr bin_embeddingbinrlastrrZrs @r%r]z2ZoeDepthMultipleMetricDepthEstimationHeads.forward0s JJz " "**1--aaaAAAg6 ++I66 mM$5$5!T$5$J$JPRSSS UTDZoeDepthMetricDepthEstimationHead.__init__..s:   "f.GVghhh   r$rc <g|]}|S))rr$rrr#)rrrr=rrr$rs r%rz>ZoeDepthMetricDepthEstimationHead.__init__..sO    !!-a''    r$)r)r-r.rr&rrrrrrrjrrr=rrrArr0rwrr4rrr) r<r=bin_configurationrrrrr&rrr$rrAs ` @@@@@@r%r.z*ZoeDepthMetricDepthEstimationHead.__init__es "5a8"8,%k2 %k2 "4, !2"" 0%8Y24GUV_`jklll   H , ,.II  "j 0 06I": 6Y)# # # 09dnt~>]^HH'H!0033.1$/Sa-b-b 7 7 )Iy'%Ig..M(yBTbfggg Cyy{{H!.!4!4!6!6  !!/ 8 8 ; ; " 9 9 ! 122ZW[!:! ! y$ 56A>>> 11-BCCWaqu1vv  ) )$ > >m// QWRSS\PZjn/oo iKQ===Dyr$rsras@r%rrdsH9 9 9 9 9 v"""""""r$rc(eZdZdZeZdZdZdZdZ dS)ZoeDepthPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. zoedepth pixel_valuesTct|tjtjtjfrT|jjd|jj |j |j j dSdSt|tj r?|j j |jj ddSdS)zInitialize the weightsg)r,stdNr)rrr:rjrlweightdatanormal_r=initializer_rangerzero_rsfill_)r<modules r% _init_weightsz%ZoeDepthPreTrainedModel._init_weightss fry")R5GH I I * M  & &CT[5R & S S S{&  &&((((('&  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r$N) rrrrr config_classbase_model_prefixmain_input_namesupports_gradient_checkpointingrr#r$r%rrsE "L"$O&*# * * * * *r$raE This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ViTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`DPTImageProcessor.__call__`] for details. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. zU ZoeDepth model with one or multiple metric depth estimation head(s) on top. ceZdZfdZeeeee d de j de e j de e de e de e d eee jeff d ZxZS) ZoeDepthForDepthEstimationc@t|t||_t |jjdrGt |jjdr-|jjj|_|jjj|_ntdt||_ t||_ t|jdkrt!|nt#||_|dS)Nr? patch_sizezXZoeDepth assumes the backbone's config to have `hidden_size` and `patch_size` attributesr)r-r.rbackbonehasattrr=r?r8rrrneckr relative_headrxrrr metric_head post_init)r<r=rAs r%r.z#ZoeDepthForDepthEstimation.__init__s    %f-- 4=' 7 7 GDMDXZf*JF '"m2=DOOj !(( @HH6,--11 7v > > >26::  r$) output_typerNrlabelsrYoutput_hidden_states return_dictrBcd}|td||n |jj}||n |jj}||n |jj}|j|||}|j}|j\} } } } |j } | | z} | | z}| || |\}}|g|z}| |\}}|g|z}| |d|d|dd|\}}| d}|s*|||f|ddz}n|f|ddz}||f|zn|St||||j|j S) a labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth depth estimation maps for computing the loss. Returns: Examples: ```python >>> from transformers import AutoImageProcessor, ZoeDepthForDepthEstimation >>> import torch >>> import numpy as np >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("Intel/zoedepth-nyu-kitti") >>> model = ZoeDepthForDepthEstimation.from_pretrained("Intel/zoedepth-nyu-kitti") >>> # prepare image for the model >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> # interpolate to original size >>> post_processed_output = image_processor.post_process_depth_estimation( ... outputs, ... source_sizes=[(image.height, image.width)], ... ) >>> # visualize the prediction >>> predicted_depth = post_processed_output[0]["predicted_depth"] >>> depth = predicted_depth * 255 / predicted_depth.max() >>> depth = depth.detach().cpu().numpy() >>> depth = Image.fromarray(depth.astype("uint8")) ```NzTraining is not implemented yet)rrYrrr,)rrrrrD)rrrrr)NotImplementedErrorr=use_return_dictrrYrforward_with_filtered_kwargs feature_mapsrHrrrrrrrr)r<rrrYrrrrfrr@r5r6rrRrSrrZr metric_depthrrs r%r]z"ZoeDepthForDepthEstimation.forwards`  %&GHH H%0% ] 'tn ] d^ ]  uU\"$88 9] ] ] _^FE] ] ] ] ] r$r)r)rr,)@rr] dataclassesrtypingrrrrr torch.utils.checkpointr activationsr file_utilsr r r modeling_outputsrmodeling_utilsrutilsrrutils.backbone_utilsrconfiguration_zoedepthr get_loggerrloggerrrModuler'r6rurryrrrrrrjitscriptfloatrorrr=rArGrirrrrrZOEDEPTH_START_DOCSTRINGrrr#r$r%rs !!!!!!//////////// !!!!!! 544444------))))))))111111222222  H % %# ?????;?? ?BGGGGGbiGGGTbi0########0;';';';';'");';';'~""""""""J3$3$3$3$3$293$3$3$l))))))))))")))))))XRRRR)0)0)0)0)0)0)0)0X@G@G@G@G@GBI@G@G@GF5,5,5,5,5,ry5,5,5,p--U-----&\,\,\,\,\,RY\,\,\,~Q,Q,Q,Q,Q,RYQ,Q,Q,h 6BBBBBBBBJ!!!!!bi!!!H<<<<<bi<<<~BI"J"J"J"J"J"J"J"J"Z^^^^^ ^^^D*****o***0 "  y y y y y !8y y  y y y r$