g$dZddlmZmZddlZddlmZddlmZGddej Z Gdd ej Z Gd d ej Z dS) a Generic interface to various configurations of the Perceiver Resampler, that simply takes in a series of (potentially time-indexed) contextual embeddings, and "resamples" (compresses) them down to a pre-specified number of latents! Note that the Perceiver in general resamples based solely off the *long-range* context; there's a nice opportunity here to prime the Perceiver Resampler with say a single layer's worth of language embeddings (the target domain), and use that to softly "retrieve & compress" what we need --> this would be a novel contribution we should explore. References: - DeepMind's Flamingo: https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model - Code borrowed w/ love from: https://github.com/lucidrains/flamingo-pytorch )OptionalTupleN) IdeficsConfigc`eZdZdededededededdffd Zd ejdejfd ZxZ S) IdeficsPerceiverResamplerconfig embed_dimdepthn_headshead_dim n_latentsreturnNc$t||||f\____jj_tj tj jjd_ tjds jdznjjdz_tjfdt%|D_tjj_dS)ao Instantiates a Perceiver Resampler that operates over a sequence of embeddings (say from a ResNet or ViT or MAE) of a given dimension, performs `depth` blocks of cross-attention with a fixed `n_latents` inputs, then returns a Tensor of shape [bsz, n_latents, embed_dim]. :param embed_dim: Dimensionality of embeddings being fed to the Perceiver Resampler (also dimensionality of latent embeddings *returned* by the Perceiver Resampler. Could be e.g., VIT embed_dim, ResNet pool dim, and so on. Args: config (`IdeficsConfig`): config object embed_dim (`int`): The size of each embedding vector depth (`int`): Depth of the Perceiver Resampler (Transformer w/ cross attention). Should be shallow (< 3). n_heads (`int`): Number of heads in each Transformer block (for multi-headed self-attention). head_dim (`int`): Dimensionality of each head projection in the Transformer block. n_latents (`int`): Number of latent embeddings to resample ("compress") the input sequence to (usually < 128). T) requires_gradr c g|]O}tjtjjjjtjgPS) nn ModuleListIdeficsPerceiverAttentionr r r qk_layer_norms IdeficsMLPintermediate_dim).0_r selfs a/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/idefics/perceiver.py z6IdeficsPerceiverResampler.__init__..Ssi     1$.$,PTP]_c_rss"4#8&AA   N)super__init__r r r rperceiver_configqk_layer_norms_perceiverrr Parametertorchrandnlatentshasattr vision_configrrrangeblocks LayerNorm layer_norm)rr r r r r r __class__s`` rr"z"IdeficsPerceiverResampler.__init__1s( FOQXZbdmFmC dmT^$5N|EK$O$O_cddd 6/== 4DNQ  %/!3  m     u       ,t~66r contextc|j|jddd}|jD]"\}}||||z}|||z}#||S)zWResample arbitrary length context & *compress* down to self.n_latents latent embeddingsrr)r(repeatshaper,r.)rr0r(attnffs rforwardz!IdeficsPerceiverResampler.forward_sw,%%gmA&61==  , ,HD"d7G,,w6GbkkG+GGw'''r ) __name__ __module__ __qualname__rintr"r&Tensorr6 __classcell__r/s@rrr0s,7#,703,7 let long-form inputs be `context`, resampled embeddings be `latents`gFbiasN)r!r"r r r rrr-context_layer_normlatents_layer_norm q_layer_norm k_layer_normqk_scaleLinearq_projk_projv_proj output_proj)rr r r rr/s rr"z"IdeficsPerceiverAttention.__init__ms7 6?(3 dm,"$,t~">">"$,t~">">   < " T] ; ;D  " T] ; ;D  t+ i t}0LSXYYY i t}0LSXYYY i t}0LSXYYY 9T\DM%A9SXYYYr r0r(cH |}|}|jdd\ }}|}t j||gd}t j||gd} fd|||fD\}}}jr* |} |}t j d|j z|}|| dd z } | d} t j d | |} | d d dS) aF Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: context (`torch.Tensor`): Tensor of shape `[bsz, seq, embed_dim]` representing long-form context to resample. latents (`torch.Tensor`): Tensor of shape `[bsz, n_latents, embed_dim]` representing fixed length latents to compress to. Returns: `torch.Tensor`: Tensor of shape `[bsz, n_latents, embed_dim]` representing attention over latents w/ cross from context. N)dimcg|]C}||jdjjddDS)r)reshaper3r r transpose)rx batch_sizers rrz5IdeficsPerceiverAttention.forward..sHuuufg199ZT\4=QQ[[\]_`aauuur z... i d, ... j d -> ... i jT)rOkeepdimz... i j, ... j d -> ... i drrQ)rBrCr3rHrIr&catrJrrDrEeinsumrFamaxdetachsoftmaxrKrSflatten) rr0r( seq_lengthr qkvscoresstabilized_scoresr4 resampledrUs ` @rr6z!IdeficsPerceiverAttention.forwards))'22))'22,3M"1",=) J  KK  KK 7G"4"=== > > KK 7G"4"=== > > vuuuulmoprsktuuu1a   %!!!$$A!!!$$A;Q=NPQRR"fkkb$k&G&G&N&N&P&PQ ((R(00L!>aHH  3 3Aq 9 9 A A" E EFFFr ) r7r8r9r:boolr"r&r;r6r<r=s@rrrlsZ#ZZsZTXZ]aZZZZZZ*(Gu|(Gel(Gu|(G(G(G(G(G(G(G(Gr rc`eZdZdeffd ZdeeejdejfdZ xZ S)rr cZt|jj|_t j|j|_t j|j|d|_t j |_ t j||jd|_ dS)z:Simple MLP block with intermediate_size and embedding sizeFr@N) r!r"r*r rr-lnrGfcReLUactc_proj)rintermediate_sizer r/s rr"zIdeficsMLP.__init__s -7,t~..)DN,=EJJJ799i 14>NNN r hidden_statesrc||}||}||}||}|S)N)rhrirkrl)rrns rr6zIdeficsMLP.forwardsL ..  .. //  M22 r ) r7r8r9rr"rrr& FloatTensorr6r<r=s@rrrs}O-OOOOOOXeE4E.F%GEL]r r) __doc__typingrrr&torch.nnrconfiguration_ideficsrModulerrrrr rrvs4  #""""""" 0000009(9(9(9(9( 9(9(9(x>G>G>G>G>G >G>G>GBr