gY~dZddlZddlmZmZmZmZmZddlZddl Zddlm Z ddl m Z ddl mZmZmZmZddlmZdd lmZdd lmZmZmZmZmZdd lmZdd lmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,e'j-e.Z/dZ0Gdde j1Z2 ddl3m4Z4e4Z2e/5dn&#e6$rYne7$re/8dYnwxYwej9e2Gdde j1Z:Gdde j1Z;Gdde j1Z<Gdde j1Z=Gdde j1Z>Gd d!eZ?d"Z@d#ZAe#d$e@Gd%d&e?ZBGd'd(e j1ZCGd)d*e j1ZDGd+d,e j1ZEGd-d.e j1ZFGd/d0e j1ZGGd1d2e j1ZHd3ZId4ZJd5ZKe#d6eIGd7d8e?ZLe#d9eIGd:d;e?eZMdS)cttjt j||_||_dS)zc Construct a layernorm module in the T5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s n/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/pix2struct/modeling_pix2struct.pyr)zPix2StructLayerNorm.__init__:sD l5:k#:#:;; #ch|tjddd}|tj||jzz}|jjtj tj fvr||jj}|j|zS)NT)keepdim) tor+float32powmeanrsqrtr.r-dtypefloat16bfloat16)r/ hidden_statesvariances r3forwardzPix2StructLayerNorm.forwardBs !##EM2266q99>>r4>PP% Ht?T4T(U(UU  ;  ? ? ?),,T[->??M{]**r4)r&__name__ __module__ __qualname__r)rC __classcell__r2s@r3r%r%9sL$$$$$$ + + + + + + +r4r%) FusedRMSNormzWDiscovered apex.normalization.FusedRMSNorm - will use it instead of Pix2StructLayerNormzJDiscovered apex but it failed to load, falling back to Pix2StructLayerNormcPeZdZdZdeddffd ZdejdejfdZxZ S)Pix2StructVisionEmbeddingsa- Construct the embeddings from patch. In `Pix2Struct` the input is different from classic Vision-transformer models. Here the input is a sequence of `seq_len` flattened patches that also combines padding patches (tokens). Each patch is represented by a vector of `hidden_size` values. configreturnNc\ttj|j|j|_tj|j|j|_ tj|j|j|_ tj |j |_ dSN)r(r)rLinearpatch_embed_hidden_sizer0patch_projection Embeddingseq_len row_embeddercolumn_embedderDropout dropout_ratedropoutr/rMr2s r3r)z#Pix2StructVisionEmbeddings.__init__is}  " &*H&J\ ] ]L9KLL!|FNFA \\*-- r4) rErFrG__doc__r!r)r+TensorrCrHrIs@r3rLrLbs| 7/7D777777%,r4rLc.eZdZfdZ ddZxZS)Pix2StructVisionAttentionct|j|_|j|_|j|_|j|_|j|jz|_ tj |j|j d|_ tj |j|j d|_ tj |j|j d|_tj |j |jd|_d|_dSNFbias)r(r)r0d_kvkey_value_proj_dimnum_attention_headsn_headsattention_dropoutrZ inner_dimrrQquerykeyvalueoutputgradient_checkpointingr[s r3r)z"Pix2StructVisionAttention.__init__s !-"(+1 / (??Yt/eLLL 9T-t~EJJJYt/eLLL i0@uMMM &+###r4NFc|jdd\}fd}||}||} ||} t j|| dd} |t jdj||f| j | j }j rj rd|_ |dkr,||ddddddf|j z}nn||||j z}nNt!s@t j|f|j |j }|||j z}d|z }||dkt j| j j} | | z } t j| t jt j| j j} t.j| dtj | } t.j| jj } || |z} t j| | }|dddj} |}|f|fz}|r|| fz}|S) z& Self-attention block Nr6c|djjddS) projectionr7r r6) contiguousviewrorm transpose)states batch_sizer/s r3to_projection_shapez>Pix2StructVisionAttention.forward..to_projection_shapes@$$&&++JDL$Jabbllmnpqrr rr4r r devicer>Tr7)dimr>ptraining)!shaperrrsrtr+matmulr|zerosrorr>rvr requires_gradrr9rr, masked_fillfinfominmaxtensorr functionalsoftmaxr:type_asrZrzr{rqru)r/rAattention_mask position_biaslayer_head_maskoutput_attentions seq_lengthr query_states key_states value_statesscoresposition_bias_masked attn_weights attn_outputoutputsr~s` @r3rCz!Pix2StructVisionAttention.forwards$"/!4RaR!8 J s s s s s s +*4::m+D+DEE )(-)@)@AA **4::m+D+DEE lJ,@,@A,F,FGG  !KDL*j9&-W]WcM* 3t} 3.2 +!!##q(( -qqq$aaa?O0P0S0STaTh0i0i i + -0A0A-BV0W0W W -// X!&,]5IQ^Qd"""!.0A0A-BV0W0W W  -M,88!9KU[Y_YeMfMfMjkk&&65< FL0I0I0M#N#NOO},,V5=,QQYYZ`aa },,\T\TXTa,bb   &'/9Ll<>> "++Aq11<<>>CCJPRTXTbcc kk+.. .M#33  0/Gr4)NNNFrDrIs@r3rgrgsb,,,,,& MMMMMMMMr4rgc*eZdZdeffd ZdZxZS)Pix2StructVisionMlprMcttj|j|jd|_tj|j|jd|_tj|j|jd|_tj |j |_ t|j |_dSrir(r)rrQr0d_ffwi_0wi_1worXrYrZr dense_act_fnactr[s r3r)zPix2StructVisionMlp.__init__ If0&+EJJJ If0&+EJJJ )FK);%HHHz&"566 &-.r4c|||}||}||z}||}t |jjtjr]|j |jjj krC|jjj tj kr$| |jjj }||}|SrP rrrrZ isinstancerr-r+rer>int8r9r/rA hidden_gelu hidden_linears r3rCzPix2StructVisionMlp.forwardhhtyy7788  -00 #m3  ]33 tw~u| 4 4 C#tw~';;;$ 22),,TW^-ABBM .. r4)rErFrGr#r)rCrHrIs@r3rrsT/5//////r4rceZdZdeddffd Z d dejdeejdeejd ede e ejejfe ejff d Z xZ S) Pix2StructVisionLayerrMrNNc>t|j|_d|_t ||_t ||_t|j |j |_ t|j |j |_ dS)Nr r1) r(r)chunk_size_feed_forward seq_len_dimrg attentionrmlpr%r0layer_norm_epspre_mlp_layer_normpre_attention_layer_normr[s r3r)zPix2StructVisionLayer.__init__s '-'E$26::&v.."5f6HfNc"d"d"d(;Fz4Pix2StructVisionEncoder.__init__..4s"#k#k#ka$9&$A$A#k#k#kr4F) r(r)rMr ModuleListrangenum_hidden_layerslayerrvr[s `r3r)z Pix2StructVisionEncoder.__init__1sa  ]#k#k#k#k5QWQiKjKj#k#k#kll &+###r4FTrArrroutput_hidden_states return_dictc|rdnd}|rdnd}t|jD]j\} } |r||fz}||| nd} |jr&|jr|| j||| |} n| ||| |} | d}|r || dfz}k|r||fz}|st d|||fDSt|||S)Nrrr c3K|]}||V dSrPrrvs r3 z2Pix2StructVisionEncoder.forward..]s(mmq_`_l_l_l_l_lmmr4last_hidden_staterA attentions) enumeraterrvr_gradient_checkpointing_func__call__tupler) r/rArrrrrall_hidden_statesall_self_attentionsi layer_moduler layer_outputss r3rCzPix2StructVisionEncoder.forward7sN#7@BBD$5?bb4(44 P POA|# I$58H$H!.7.CillO* pt} p $ A A )!"#% !! !- ]NO]n o o )!,M  P&9]1=Mret|jtr|jj j n |jj }|j jj d||dzzdSt|t8j!t8j"frt8j#$|jj%tLj'd|jj(%|jj)|j_|j |jjdSdSt|tr*|j!|jjddSdSt|t8jr]|jj d|jj(|j-|jj|jdSdSdS)zInitialize the weights?g)r<stdrkN)*rMinitializer_factorrr%r-datafill_ Pix2StructTextDenseGatedActDenser! text_configr0rrnormal_hasattrrkzero_rrPix2StructTextAttentionrl num_headsrrrsrtruhas_relative_attention_biasrelative_attention_biasrrT padding_idxPix2StructTextModellm_headrQConv2dinit trunc_normal_r9r+r:initializer_ranger>)r/modulefactorr0rrmros r3 _init_weightsz'Pix2StructPreTrainedModel._init_weightszs/ f1 2 2J ? M  $ $Vc\ 2 2 2 2 2  @ A AH ?dk+;<<- '33[,  4>dkK[3\3\r4;*//bfbmbrD K  # + +&[UYDY:Z + [ [ [v{F++ . 0@0L  %++--- K  # + +&[UYDY:Z + [ [ [v{F++ . 0@0L  %++--- I  ! ) )s4D.8Q ) R R Rvy&)) ,fin.H #))+++++ , ,.H.H  7 8 87 ? dk+;<<- '33[,  1;4;HX0Y0Yv ',,_c_j_v  dk+;<<+ '11[*  L  $ , ,#6kTfFfkoEo;p , q q q J  " * *;PTCT9U * V V V L  $ , ,#6[RVEV;W , X X X M % - -3FwQcGchlFl>>>>.-  3 4 4 ?dk+;<<- '33[,  N ! & . .CV X\G\=] . ^ ^ ^ ^ ^ BI 6 7 7 ?"$!6!6 "%%em443DKDa"7""b$%% M {&  &&((((('&  3 4 4 ?}( "((-----)(  - - ? M  & &CT[5R & S S S!- "6#56<<>>>>> ? ?--r4c|jj}|jj}|tdt |rHt j|jdddz|}t j||dddfgd}nD| |j}|dddf |dddf<||d<|td| |d k||S) Nzself.model.config.decoder_start_token_id has to be defined. In Pix2Struct it is usually set to the pad_token_id. See Pix2Struct docs for more information.r7)r .rr ).rz1self.model.config.pad_token_id has to be defined.) rMdecoder_start_token_id pad_token_id ValueErrorrr+fullrcat new_zerosclone masked_fill_)r/rrr shifted_input_idss r3 _shift_rightz&Pix2StructPreTrainedModel._shift_rights!%!C{/ ! )<  Y ' ' ? % 9?3B3+?$+FH^ _ _  % +Q*RXZ [ [ [   ) 3 3IO D D )238)<)B)B)D)D c122g &(> f %  PQQ Q&&'8D'@,OOO  r4N) rErFrGrdr! config_class_supports_cache_class_supports_static_cachepropertyrrrrr4r3rresl $L " XM?M?M?`!!!!!r4raL This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Pix2StructConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: flattened_patches (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_channels x patch_height x patch_width)`): Flattened and padded pixel values. These values can be obtained using [`AutoImageProcessor`]. See [`Pix2StructVisionImageProcessor.__call__`] for details. Check the [original paper](https://arxiv.org/abs/2210.03347) (figure 5) for more details. attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zjThe bare Pix2StructVision Model transformer outputting raw hidden-states without any specific head on top.cXeZdZeZdZdZdgZdeffd Z dZ de e e e fdd fd Zeeeee  ddeejd eejd eejdeedeedeedeeeffdZxZS)Pix2StructVisionModelr\TrrMct|||_t||_t ||_t|j|j |_ | dSNr) r(r)rMrLrarencoderr%r0r layernorm post_initr[s r3r)zPix2StructVisionModel.__init__sr     4V<<.v66 ,V-?VEZ[[[ r4c|jjSrP)rarSr/s r3get_input_embeddingsz*Pix2StructVisionModel.get_input_embeddings"s //r4heads_to_prunerNNc|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrr prune_heads)r/r rheadss r3 _prune_headsz"Pix2StructVisionModel._prune_heads%sU +0022 C CLE5 L u % / ; ;E B B B B C Cr4 output_typerrrrrrc||n |jj}||n |jj}||n |jj}|t d|,|ddk}|||jj}| |}| ||||||}|d} | | } |s| f} | |ddzSt| |j |jS) a. Returns: Example: ```python >>> import requests >>> from PIL import Image >>> from transformers import AutoProcessor, Pix2StructVisionModel >>> image_processor = AutoProcessor.from_pretrained("google/pix2struct-textcaps-base") >>> model = Pix2StructVisionModel.from_pretrained("google/pix2struct-textcaps-base") >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 2048, 768] ``` Nz%You have to specify flattened_patchesr7rr)rrrrrr r)rMrruse_return_dictr sumfloat get_head_maskrrarrrrAr) r/r\rrrrrembedding_outputencoder_outputssequence_output head_outputss r3rCzPix2StructVisionModel.forward-sMH2C1N--TXT_Tq$8$D $+Jj &1% K 'tn K d^K  u00 1K K K edONK K K K K r4rc*eZdZdeffd ZdZxZS)rrMcttj|j|jd|_tj|j|jd|_tj|j|jd|_tj |j |_ t|j |_dSrirr[s r3r)z)Pix2StructTextDenseGatedActDense.__init__rr4c|||}||}||z}||}t |jjtjr]|j |jjj krC|jjj tj kr$| |jjj }||}|SrPrrs r3rCz(Pix2StructTextDenseGatedActDense.forwardrr4rErFrGr"r)rCrHrIs@r3rr~sT/3//////r4rc*eZdZdeffd ZdZxZS)Pix2StructTextLayerFFrMctt||_t |j|j|_tj |j |_ dSr) r(r)rDenseReluDenser%r0layer_norm_epsilon layer_normrrXrYrZr[s r3r)zPix2StructTextLayerFF.__init__s[ >vFF-f.@fF_```z&"566 r4c||}||}|||z}|SrP)r@r>rZ)r/rAforwarded_statess r3rCzPix2StructTextLayerFF.forwardsF??=99../?@@% 5E(F(FF r4r:rIs@r3r<r<sT73777777r4r<creZdZ d dedeeffd Zed d Zdd Z dd Z xZ S)rFNrM layer_idxc*t||_|j|_|j|_|j|_|j|_|j|_ |j |_ |j |jz|_ ||_ |(td|jjdt%j|j|jd|_t%j|j|jd|_t%j|j|jd|_t%j|j|jd|_|jr$t%j|j|j |_t5|_d|_dS)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Frj)r(r)rrelative_attention_num_bucketsrelative_attention_max_distancer0rlrmrrorYrZrqrDlogger warning_oncer2rErrQrrrsrtrurTrset pruned_headsrvr/rMrrDr2s r3r)z Pix2StructTextAttention.__init__sn +F(.4.S+/5/U,!-"(+' * (??"     ,4>+B,,,   Yt/1ANNN 9T-t/?eLLLYt/1ANNN i 0$2BOOO  + k+-<8[]a]i+j+jD (EE&+###r4T cPd}|rC|dz}||dktj|zz }tj|}n(tj|tj| }|dz}||k}|tj||z tj||z z ||z ztjz}tj|tj ||dz }|tj |||z }|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr6r ) r9r+r^absr zeros_likelogr+math full_likewhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges r3_relative_position_bucketz1Pix2StructTextAttention._relative_position_buckets>.  c A K !2Q!6 : :5: F F T T  % *; < <  !&+NO`>a>a!b!b b  1$ $y0&/ I'--//); < <h|i/00 1Y& ( "UZ.. &" &+Y &8RT_bcTc(d(d& & " EK2CE_```r4c||jjj}tj|tj|dddf}tj|tj|dddf}||z }||d|j|j}||}| gd d}|S)z%Compute binned relative position biasN)r>rF)rWrXrY)r6rr r) rr-rr+aranger^r^rFrGpermute unsqueeze) r/ query_length key_lengthrcontext_positionmemory_positionrVrelative_position_bucketvaluess r3 compute_biasz$Pix2StructTextAttention.compute_biass >18?F < EJvVVVWXWXWXZ^W^_,zFSSSTXZ[Z[Z[T[\+.>>#'#A#A ;= $B$ $  --.FGG **44Q77 r4c |jdd\} } |du} ||}|| d|j|jdd}|0|j|j }| r|j }n|j }| r|n|}| r)|r'|r%|j |j }|j |j }n||}||}|| d|j|jdd}|| d|j|jdd}|9| s| nd} ||||j d| i\}}| rd|j|j <t#j||dd}|| | ddzn|}||n |jd}|js@t#jd|j||f|j|j}|jr|jrd|_n||||j }|||z}|jrUt#j|jd}d |t;|j<|dd|f}n|}||z }t>j !|"d #|}t>j $||j$|j }|||z}t#j||}|dd| d|j%}|&|}|f|fz|fz}| r||fz}|S) z Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). Nr6r7r cache_positionTr rrrrr)'rrrrzr{rormr| is_updatedgetrDcross_attention_cacheself_attention_cache key_cache value_cachersrtupdater+rrrrr>rvrrrirKr,listrrrrr+rrZrqru)r/rAmaskkey_value_statesrpast_key_valuerrc use_cacherrkr~ris_cross_attentionrrmcurrent_statesrrrreal_seq_lengthrdrrrrs r3rCzPix2StructTextAttention.forwards#$"/!4RaR!8 J.T9zz-00;;== #((RtG^__iijkmnoo  %'266t~FFJ! E!/!E!/!D.@R))]  E. EZ E'1$.AJ)5dnELL.11<<>>J::n55@@BBL#RtG^__iijkmnooJ',,ZT\4KbccmmnoqrssL)7I!St+9+@+@ dn?OQ_>`,,( L&E@DN-dn=lJ,@,@A,F,FGG  8D8LnR0144R^O,<,DJZJ`abJcJ3 e %  ozB6=`f`l!!! .74=726M/ $ 1 1/:V\Vc 1 d d  - 4   1:m1!455D,-Dd'(( )#0DIIKK#@ #0 &&},,V\\^^,DDLLVTT },,\T\TXTa,bb   &'/9Ll<>> !++Aq11<<>>CCJPRTXTbcc kk+.. .N#44 7GG  0/Gr4FN)TrMrNrP) NNNNNNFFN) rErFrGr"rr4r) staticmethodr^rirCrHrIs@r3rrsjn,,*,ZbcfZg,,,,,,>- - - \- `(eeeeeeeer4rcHeZdZddeeffd Z ddZxZS) Pix2StructTextLayerSelfAttentionFNrDctt||||_t |j|j|_tj |j |_ dS)NrrDr r(r)rrr%r0r?r@rrXrYrZrLs r3r)z)Pix2StructTextLayerSelfAttention.__init__xsl 0 0KW`   .f.@fF_```z&"566 r4c ||} || |||||||} ||| dz}|f| ddz} | S)N)rurrrwrxrrkrr r@rrZ) r/rArrrrwrxrrknormed_hidden_statesrrs r3rCz(Pix2StructTextLayerSelfAttention.forwards $}==>> '+)/)*   & 5Ea5H(I(II  "%5abb%99r4r|)NNNNFFNrErFrGrr4r)rCrHrIs@r3rrwsy77XVY]777777r4rcJeZdZddeeffd Z ddZxZS)!Pix2StructTextLayerCrossAttentionNrDctt|d||_t |j|j|_tj |j |_ dS)NFrrr)r/rMrDr2s r3r)z*Pix2StructTextLayerCrossAttention.__init__sc 0UZfoppp-f.@fF_```z&"566 r4Fc ||} || |||||||| |  } ||| dz} | f| ddz}|S)N) rurvrrrwrxrcrrkrr r)r/rArvrrrrwrxrcrrkrrrrs r3rCz)Pix2StructTextLayerCrossAttention.forwards $}==>> -'+)%/)*   %t||4DQ4G'H'HH /$4QRR$88r4rP)NNNNFNFNrrIs@r3rrs{77(3-777777r4rcReZdZddeeffd Z ddZxZS) Pix2StructTextBlockFNrDctt||||_t ||_t ||_dS)Nr)r(r)rself_attentionrencoder_decoder_attentionr<rrLs r3r)zPix2StructTextBlock.__init__s_ > (C    *K6)R)R&(00r4Tc |||||| | | | }|dd\}} |dd}|jtjkr_tj|r9tj|jjdz }tj|| |}|du}|r| |||||| | ddz| | |  }|dd\}} |jtjkr_tj|r9tj|jjdz }tj|| |}||ddz}| |}|jtjkr_tj|r9tj|jjdz }tj|| |}|f}| r || fz|z}n||z}|S)N)rrrrwrxrrkr6i)rrr7r ) rvrrrrwrcrxrrk) rr>r+r?isinfanyrrclamprr)r/rArrencoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasrcross_attn_layer_head_maskrwrxrrrkrattention_outputs clamp_valuedo_cross_attentioncross_attention_outputsrs r3rCzPix2StructTextBlock.forwardsF "&!4!4 )'+)/)"5 " " )?rr(B% ~21226  %- / /EK 4N4N4R4R4T4T /+m&9::>EK!K K<[YYYM2$>  P&*&D&D!65; :-+B/!3#"3-'E ' ' #-DBQB,G )M>"em33 M8R8R8V8V8X8X3#k-*=>>BTI % M |Q\ ] ] ] !24KABB4O O //   %- / /EK 4N4N4R4R4T4T /+m&9::>EK!K K<[YYYM "  2 114EEGG 11Gr4r|) NNNNNNNNFFTNrrIs@r3rrs 1 1XVY] 1 1 1 1 1 1 "#&*#'IIIIIIIIr4ra The Pix2Struct model was proposed in [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://arxiv.org/abs/2210.03347) by Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova. It's an encoder decoder transformer pre-trained in a image-to-text setting. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (Union[`Pix2StructConfig`, `Pix2StructTextConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Pix2StructText is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [Pix2StructText Training](./t5#training). attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pix2StructText uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [Pix2StructText Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, `optional`: *hidden_states*, `optional`: *attentions*) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)` is a sequence of hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention layers. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. a Args: flattened_patches (`torch.FloatTensor` of shape `(batch_size, seq_length, hidden_size)`): Flattened pixel patches. the `hidden_size` is obtained by the following formula: `hidden_size` = `num_channels` * `patch_size` * `patch_size` The process of flattening the pixel patches is done by `Pix2StructProcessor`. attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pix2StructText uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [Pix2StructText Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, `optional`: *hidden_states*, `optional`: *attentions*) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)` is a sequence of hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention layers. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss for the decoder. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z)The standalone text decoder of Pix2Structc$eZdZeZdgZdgZdZfdZdZ dZ dZ dZ d Z eeeee  d%d eejd eejdeejdeejdeejdeejdeejdeeeejdeedeedeedeejdeedeejdeeejdfeffdZd ejdejdejdedef dZed ejded ed!ej d"ej!dejd#efd$Z"xZ#S)&rrzlm_head.weightTcttjjj|_tjfdtj D|_ tjj |_ tjj|_tjjjd|_|d|_dS)Nc Vg|]%}tt|dk|&S)rr)rr)rrrMs r3rz0Pix2StructTextModel.__init__..sD   $FQRSV `abbb   r4rFrj)r(r)rrT vocab_sizer0 embed_tokensrr num_layersrr%r?final_layer_normrXrYrZrQrrrvr[s `r3r)zPix2StructTextModel.__init__s    L):F\%\ " "%%r4c|jSrPrrs r3rz(Pix2StructTextModel.get_input_embeddings)s   r4c||_dSrPrr/new_embeddingss r3set_input_embeddingsz(Pix2StructTextModel.set_input_embeddings,s*r4c|jSrPrrs r3get_output_embeddingsz)Pix2StructTextModel.get_output_embeddings/ |r4c||_dSrPrrs r3set_output_embeddingsz)Pix2StructTextModel.set_output_embeddings2s % r4r&Nrrrr inputs_embedsrcross_attn_head_maskrrxrrlabelsrrkrN.c & | | n |jj} | | n |jj} | | n |jj} | | n |jj} ||t d|1|}|d|d}n.||dd}nt d|&|j Jd||}|\}}d}d}| s|t|tr4t|tsd}t|t}npt|ts1d}tdtj|}n*|(ttt}d }| |d }n||}|t#j|||z|j }|7|||zn|}t#j|||j }|jjr#||||||jnd| }nO|ddddddf}||j }d |z t#j|jjz}|O|\}}}||f}|t#j||j }||}nd}|||jj}|||jj}| rd nd}| rd nd}| rd nd}d} d}!||}"tA|j!D]\}#}$||#}%||#}&| r||"fz}|j"rL|j#rE| rt$dd} |%|$j&|"|| |||!|%|&d| | | }'n|$|"|| |||!|%|&|| | | }'| dur|'dddz|'ddz}'|'dd\}"}(|'d} | |'| rdnd}!| r||'dfz}| ||'dfz}|'|"}"||"}"|(|"})| r||"fz}d}*| | |)j} tSj*dd}+|+|)+d|)d| +d}*| r|(nd},|r|j},|r|,},| st[d|*|)|,|||fDSt]|*|)|,|||S)a Returns: Example: ```python >>> from transformers import AutoProcessor, Pix2StructTextModel >>> processor = AutoProcessor.from_pretrained("google/pix2struct-textcaps-base") >>> model = Pix2StructTextModel.from_pretrained("google/pix2struct-textcaps-base") >>> inputs = processor(text="Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ``` NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer7zEYou have to specify either decoder_input_ids or decoder_inputs_embedszrrzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...) rrrrrrrrwrxrrkr rPr6r rr<) ignore_index reductionc3K|]}||V dSrPrrs r3rz.Pix2StructTextModel.forward.. s4  =!===  r4)losslogitsrrArcross_attentions)/rMrxrrr)r sizer{rrr r r rHrIfrom_legacy_cacheget_seq_lengthr+r`rr, is_decoder_update_causal_maskrpr9r>rrinvert_attention_maskr,rrZrrrvrrrrCrrrCrossEntropyLossrzto_legacy_cacherr)-r/rrrrrrrrrxrrrrrkkwargs input_shaper~rreturn_legacy_cachereturn_self_attention_cachepast_key_values_lengthmask_seq_length causal_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskrall_attentionsall_cross_attentionsrrrArrrrrnext_decoder_cacherrloss_fct next_caches- r3rCzPix2StructTextModel.forward5s'H"+!6IIDKstt t  "#..**K!r;r?;;II  &',,..ss3KKdee e  $002p000 --i88M!, J$&+#  V3/511 V*_Vi:j:j V.2+"5o|~~"V"V1DEE V&*###` #6"G"X"X ("5lnnlnn"U"U!"  %%3A%6 " "  (%4%C%C%E%E "  !"\&(>(KTaThN  !BQA\..00:==bl #Z OML`aaaN ; ! U228G8S44Y]! KK)D$)9:K%..}/B.CCK, MO O 0=bqb0A -M- *!,M$00=CT>[aaZ[0\-  V!/=3C2E!E(4+?=QRCSBU+U(--m<<  ]33 m,,  E 1]4D D   YYv}--F*OOOH8F--//44RRII6K\K\K^K^KcKcdfKgKghhD+4>''$ & >(=J  ;(88::J   %"(       1&+%1     r4 input_tensorc |jjdkr |d|vr|SdS||nd}t|t}|jjdkr#|s!|st j||||jrdS|j|j } }|j d} |r| } n/t|tj r |j dn|| zdz} ||| | || ||j d} |jjdkrB|@|j jd kr0|s.tj|j} t j| | } | S) Nflash_attention_2rrsdpa)rr is_trainingr r7)sequence_length target_lengthr>rrkr~cuda)rM_attn_implementationrrrr_ignore_causal_mask_sdparr>rrget_max_cache_shaper+re5_prepare_4d_causal_attention_mask_with_cache_positiontyperr_unmask_unattended)r/rrrkrrpast_seen_tokensusing_static_cacher>rrrr min_dtypes r3rz'Pix2StructTextModel._update_causal_mask#s ; +/B B B)c^.C.C%%4 @O?Z?99;;;`a'EE ; +v 5 5>P 5Yj 5%>*'7 M   t$*L,?v&,Q/  +??AAMMnel;;<$R((%7!; PP +')#)!,Q   K , 6 6*%*f44%5  E**.I0CKQZ[[Kr4rrr>rr~c ||dkr|}n+tj|j} tj||f| ||}|dkrtj|d}|tj|||ddkz}|ddddddf|ddd}|| }|j d} |ddddddd| f|ddddddfz} | dk} |ddddddd| f | | |ddddddd| f<|S) a Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. device (`torch.device`): The device to plcae the 4D attention mask on. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr) fill_valuer>rr )diagonalrlr7r) rr+rrr triur`reshapeexpandrrr) rrrr>rrkr~rrr mask_length padding_masks r3rzIPix2StructTextModel._prepare_4d_causal_attention_mask_with_cache_positiondsD  %.*<*<*>*>!*C*C(KK E**.I* -0Ye\bK!###jqAAA 5< fEEEH^H^_acdHeHee eK%dD!!!QQQ&67>>z1bRTUUK))//11 ,226 *111aaaL[L+@ANSTSTSTVZ\`bcbcbcScDdd +q0 5@AAAqqq,;,AV5W5c5c )66 AAAqqq!!!\k\12r4)NNNNNNNNNNNNNN)$rErFrGr"rr3_tied_weights_keysr2r)rrrrrr PIX2STRUCT_TEXT_INPUTS_DOCSTRINGrrr6rr+ LongTensor FloatTensorrerrrrCr rr}r4r>rrrHrIs@r3rrs (L./*+&*#,,,,,(&&&<!!!+++&&&+*+KLL+L[jkkk156:=A>B48157;EI$(,0/3-1&*59i i E,-i !!23i  ((9: i !)): ; i   01 i E-.i 'u|4i "%e.?(@"ABi D>i $D>i 'tni )*i d^i !!12i " uU&+,.OO P#i i i lkMLi X? ?l? ?  ?  ????B5 555{ 5  5  5555\55555r4rzhA conditional generation model with a language modeling head. Can be used for sequence generation tasks.c'eZdZeZdZdgZdeffd ZdZdZ de j fdZ d Z d d eede jfd Zd ZdZeeeee d!deejdeejdeejdeejdeejdeejdeejdeeeejdeeeejdeejdeejdeedeedeedeedeejde eejeff"dZ!xZ"S)""Pix2StructForConditionalGenerationr\zdecoder.lm_head.weightrMct|t|j|_t |j|_|j|_| dSrP) r(r)r vision_configrrrdecoderis_vqarr[s r3r)z+Pix2StructForConditionalGeneration.__init__s`    ,V-ABB *6+=>> m  r4c4|jSrP)rrrs r3rz7Pix2StructForConditionalGeneration.get_input_embeddingss|00222r4c:|j|dSrP)rrrs r3rz7Pix2StructForConditionalGeneration.set_input_embeddingss )).99999r4rNc4|jSrP)rrrs r3rz8Pix2StructForConditionalGeneration.get_output_embeddingss|11333r4c:|j|dSrP)rrrs r3rz8Pix2StructForConditionalGeneration.set_output_embeddingss **>:::::r4Nnew_num_tokensc\|j|}||jj_|SrP)rresize_token_embeddingsrMrr)r/r model_embedss r3rz:Pix2StructForConditionalGeneration.resize_token_embeddingss,|;;NKK .< *r4c|jSrP)rrs r3 get_decoderz.Pix2StructForConditionalGeneration.get_decoderrr4c|jSrP)rrs r3 get_encoderz.Pix2StructForConditionalGeneration.get_encoderrr4r&rrrrdecoder_head_maskrr.rrdecoder_inputs_embedsrxrrrrkc| | n|jjj} ||n |jj}|||||| ||}ne|rct |t sNt |dt|dkr|dndt|dkr|dnd}|d}| W|U| S|| }||n0| |jj  }d|dddf<| ||| | ||||| | || ||}|s||zSt|j|j|j|j|j|j|j|j|j S) a Returns: Example: Inference: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Pix2StructForConditionalGeneration >>> processor = AutoProcessor.from_pretrained("google/pix2struct-textcaps-base") >>> model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-textcaps-base") >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> # autoregressive generation >>> generated_ids = model.generate(**inputs, max_new_tokens=50) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_text) A stop sign is on a street corner. >>> # conditional generation >>> text = "A picture of" >>> inputs = processor(text=text, images=image, return_tensors="pt", add_special_tokens=False) >>> generated_ids = model.generate(**inputs, max_new_tokens=50) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_text) A picture of a stop sign with a red stop sign ``` Training: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Pix2StructForConditionalGeneration >>> processor = AutoProcessor.from_pretrained("google/pix2struct-base") >>> model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-base") >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "A stop sign is on the street corner." >>> inputs = processor(images=image, return_tensors="pt") >>> labels = processor(text=text, return_tensors="pt").input_ids >>> # forward pass >>> outputs = model(**inputs, labels=labels) >>> loss = outputs.loss >>> print(f"{loss.item():.5f}") 5.94282 ```N)r\rrrrrrr r6r)rrrrrrrrrxrrrrrk) rrrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)rMrrxr)rrrrrner r+rrrrrrArrr)r/r\rrrrr rr.rrrrxrrrrkrAdecoder_outputss r3rCz*Pix2StructForConditionalGeneration.forwards`"+!6IIDK6:8<=A159=7;EIEI-18<$(,0/3&*59#N N #E$56N !!23N $E$45 N !))9 : N E-. N $E$56N 'u|4N "%e.?(@"ABN "%e.?(@"ABN )*N  ( 5N D>N $D>N 'tnN d^!N "!!12#N $ uU&');; <%N N N ]\HGN N N N N r4r)NrdrStypingrrrrrr+torch.utils.checkpointr activationsr cache_utilsr r r r generationrmodeling_attn_mask_utilsrmodeling_outputsrrrrrmodeling_utilsr pytorch_utilsrutilsrrrrrrrrconfiguration_pix2structr!r"r# get_loggerrErHr6rr%apex.normalizationrJinfo ImportError ExceptionrappendrLrgrrrr!PIX2STRUCT_VISION_START_DOCSTRINGr5rrr<rrrrPIX2STRUCT_START_DOCSTRINGrrrrrr4r3r,s 55555555555555 !!!!!!PPPPPPPPPPPP))))))>>>>>>.-----111111                    eddddddddd  H % %%+++++")+++2  //////& KKijjjj   D    NN_```D /000!!!!!!!!H^^^^^ ^^^D"):(((((BI(((V2 2 2 2 2 bi2 2 2 j~!~!~!~!~!~!~!~!B %!&"6p%j j j j j 5j j j \ry:     BI    HHHHHbiHHHX!!!!!ry!!!J##### ###LWWWWW")WWWt,`$ DVr/jjjjj3jj jZ nz z z z z )BOz z z z z sB==C C C