g'dZddlmZmZddlZddlmZddlm Z Gdd ej j j Z Gd d ej j j ZGd d ej j j ZdS)a Generic interface to various configurations of the Perceiver Resampler, that simply takes in a series of (potentially time-indexed) contextual embeddings, and "resamples" (compresses) them down to a pre-specified number of latents! Note that the Perceiver in general resamples based solely off the *long-range* context; there's a nice opportunity here to prime the Perceiver Resampler with say a single layer's worth of language embeddings (the target domain), and use that to softly "retrieve & compress" what we need --> this would be a novel contribution we should explore. References: - DeepMind's Flamingo: https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model - Code borrowed w/ love from: https://github.com/lucidrains/flamingo-pytorch )OptionalTupleN) shape_list) IdeficsConfigcjeZdZdededededededdffd Zfd Zd ejdejfd Z xZ S) TFIdeficsPerceiverResamplerconfig embed_dimdepthn_headshead_dim n_latentsreturnNc Ftjd i|||||f\|_|_|_|_|jj|_t|j ds |jdzn|j jdz|_ g|_ t|D]a}|j t|j|j|j|jd|dt!|j |d|dgbt"jjdd |_d S) ao Instantiates a Perceiver Resampler that operates over a sequence of embeddings (say from a ResNet or ViT or MAE) of a given dimension, performs `depth` blocks of cross-attention with a fixed `n_latents` inputs, then returns a Tensor of shape [bsz, n_latents, embed_dim]. :param embed_dim: Dimensionality of embeddings being fed to the Perceiver Resampler (also dimensionality of latent embeddings *returned* by the Perceiver Resampler. Could be e.g., VIT embed_dim, ResNet pool dim, and so on. Args: config (`IdeficsConfig`): config object embed_dim (`int`): The size of each embedding vector depth (`int`): Depth of the Perceiver Resampler (Transformer w/ cross attention). Should be shallow (< 3). n_heads (`int`): Number of heads in each Transformer block (for multi-headed self-attention). head_dim (`int`): Dimensionality of each head projection in the Transformer block. n_latents (`int`): Number of latent embeddings to resample ("compress") the input sequence to (usually < 128). r zblocks.z.0namez.1h㈵> layer_normepsilonrN)super__init__r rrrperceiver_configqk_layer_norms_perceiverqk_layer_normshasattr vision_configintermediate_dimblocksrangeappendTFIdeficsPerceiverAttention TFIdeficsMLPtfkeraslayersLayerNormalizationr) selfr r r rrrkwargsi __class__s d/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/idefics/perceiver_tf.pyrz$TFIdeficsPerceiverResampler.__init__1sD( ""6"""FOQXZbdmFmC dmT^$5N6/== 4DNQ  %/!3   u  A K  / dmTEX_nij_n_n_n!!6_q___UUU     (/<4>2[_fo'     k"""""r1contextctj|jd}tj|tj|dddg}|jD]"\}}||||z}|||z}#||S)zWResample arbitrary length context & *compress* down to self.n_latents latent embeddingsraxisr)r( expand_dimsr4tiler5r#r)r,r;r4attnffs r0callz TFIdeficsPerceiverResampler.callcs.A666''BHW$5$5a$8!Q#?@@  , ,HD"d7G,,w6GbkkG+GGw'''r1) __name__ __module__ __qualname__rintrr9r(TensorrC __classcell__r/s@r0r r 0s)^#)^03)^ let long-form inputs be `context`, resampled embeddings be `latents`rcontext_layer_normrlatents_layer_norm q_layer_norm k_layer_normgFq_projuse_biasrk_projv_proj output_projNr)rrr rrrr(r)r*r+rMrNrOrPqk_scaleDenserQrTrUrV)r,r rrrr-r/s r0rz$TFIdeficsPerceiverAttention.__init__ps""6"""6?(3 dm,"$(/"D"DTXl"D"m"m"$(/"D"DTXl"D"m"m   f " B B4Vd B e eD  " B B4Vd B e eD  t+ ho++DL4=,HSX_g+hh ho++DL4=,HSX_g+hh ho++DL4=,HSX_g+hh 8?00UQ^0__r1r;r4c ^ |}|}t|\ }}|}t j||gd}t j||gd} fd|||fD\}}}jr* |} |}t j d|j z|}|t j |ddz } t j| d} t j d| |} t jt j| gd   djjzfS) a= Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: context (`tf.Tensor`): Tensor of shape `[bsz, seq, embed_dim]` representing long-form context to resample. latents (`tf.Tensor`): Tensor of shape `[bsz, n_latents, embed_dim]` representing fixed length latents to compress to. Returns: `tf.Tensor`: Tensor of shape `[bsz, n_latents, embed_dim]` representing attention over latents w/ cross from context. r=c g|]F}tjtj||jdjjfgdGS)rrrrperm)r( transposereshaper5rr).0x batch_sizer,s r0 z4TFIdeficsPerceiverAttention.call..s^    LA AGAJ dm'\]]dpdpdp q q q   r1z... i d, ... j d -> ... i jT)r>keepdimsz... i j, ... j d -> ... i dr\r^)rMrNrrQrTr(concatrUrrOrPeinsumrW reduce_maxnnsoftmaxrVrar`rr) r,r;r4 seq_lengthr qkvscoresstabilized_scoresrA resampledrds ` @r0rCz TFIdeficsPerceiverAttention.calls))'22))'22,6w,?,?) J  KK  KK 7G"42>>> ? ? KK 7G"42>>> ? ?     AY   1a   %!!!$$A!!!$$A8!dm:KQOO"R]6T%R%R%RRu}}.R}88I;T1EE  Jr|ILLLAAAJPRTXT`cgcpTpCq r r   r1) rDrErFrGboolrr(rHrCrIrJs@r0r&r&os`#``s`TX`gk``````*+ BI+  + bi+ + + + + + + + r1r&c`eZdZdeffd ZdeeejdejfdZ xZ S)r'r c tjd i||jj|_tjjdd|_tjj |dd|_ tjj d|_ tjj |jdd |_ d S) z:Simple MLP block with intermediate_size and embedding sizerlnrFfcrRactrc_projNr)rrr!r r(r)r*r+rwrXrxReLUryrz)r,intermediate_sizer r-r/s r0rzTFIdeficsMLP.__init__s""6"""-7(/44T4MM(/''(9EPT'UU8?''U'33ho++DNUQY+ZZ r1 hidden_statesrc||}||}||}||}|S)N)rwrxryrz)r,r}s r0rCzTFIdeficsMLP.callsL ..  .. //  M22 r1) rDrErFrrrrr(rHrCrIrJs@r0r'r's{[-[[[[[[(5+;"<r1r')__doc__typingrr tensorflowr(modeling_tf_utilsrconfiguration_ideficsrr)r*Layerr r&r'rr1r0rs4  #"""""""++++++000000<(<(<(<(<("(/"7<(<(<(~A A A A A "(/"7A A A H28?(r1