g}6dZddlZddlZddlmZmZmZmZddlZ ddl Z ddl Z ddl m Z ddl mZddlmZddlmZdd lmZmZdd lmZmZmZmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#d dl$m%Z%e"j&e'Z(dZ)dZ*de j+de,de,fdZ-Gdde j.Z/Gdde j0Z1Gdde j0Z2de1iZ3Gdde j0Z4GddeZ5d Z6d!Z7d"Z8Gd#d$e5Z9Gd%d&e5Z:e d'e6Gd(d)e5Z;e d*e6Gd+d,e5eZ<Gd-d.e5Z=Gd/d0e5eZ>dS)1z=PyTorch MarianMTModel model, ported from the Marian C++ repo.N)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)GenerationMixin)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)add_end_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) MarianConfigrzHelsinki-NLP/opus-mt-en-de input_ids pad_token_iddecoder_start_token_idc||j}|ddddf|ddddf<||dddf<|td||dk||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrrshifted_input_idss f/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/marian/modeling_marian.pyshift_tokens_rightr'7s"++IO<<(CRC06688aaae4aaadLMMM""#4#zOMarianSinusoidalPositionalEmbedding._init_weight...Vs7 L L LAcBHUAaL3$6777 L L Lr()range)r>rAr@s @r&rBzDMarianSinusoidalPositionalEmbedding._init_weight..Vs9 e e eQT L L L L Ls L L L e e er(Fr;rrN) r!r<arrayrC requires_gradtorch FloatTensorsincosdetach_)r7n_pos position_encsentinelr@s @r&r3z0MarianSinusoidalPositionalEmbedding._init_weightNs Y sx e e e eX]^cXdXd e e e  ""Qw!||3!88#(a".rvl111add76K/L/LMMAAAqzM!-bf\!!!QTT'5J.K.KLLAAAxyyL  r(rinput_ids_shapepast_key_values_lengthc|dd\}}tj|||ztj|jj}t |S)z3`input_ids_shape` is expected to be [bsz x seqlen].Nr;)dtypedevice)rFarangelongr4rRr1forward)r5rNrObszseq_len positionsr6s r&rUz+MarianSinusoidalPositionalEmbedding.forward_s]'rr* WL "$:W$DEJ_c_j_q   wwy)))r(r0)r)__name__ __module__ __qualname____doc__intrr2 staticmethodr Parameterr3rFno_gradSizeTensorrU __classcell__r6s@r&r*r*GsNN55c5#5HUXM5ei555555",2<\ U]__**uz*3*W\Wc*****_*****r(r*cheZdZdZ ddededed ed ed ed eeffd Z de j dedefdZ dde j dee j dee e j dee j dee j dede e j ee j ee e j ffdZxZS)MarianAttentionz=Multi-headed attention from 'Attention Is All You Need' paperFTN embed_dim num_headsdropout is_decoderbias is_causalconfigc t||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ tj ||||_ tj ||||_tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rl)r1r2rhrirjhead_dimrnr#scalingrkrmrLineark_projv_projq_projout_proj) r5rhrirjrkrlrmrnr6s r&r2zMarianAttention.__init__ms "" !Y.  MI %$. 8 83dn33%.333 }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r(tensorrWrVc||||j|jddS)Nrr;)viewrirq transpose contiguous)r5rxrWrVs r&_shapezMarianAttention._shapes<{{3GGQQRSUVWWbbdddr( hidden_stateskey_value_statespast_key_valueattention_masklayer_head_maskoutput_attentionsr.cZ |du}|\}} } |||jz} |r6|4|djd|jdkr|d} |d} n>|rU|||d|} |||d|} n||||d|} |||d|} tj|d| gd} tj|d| gd} nT|||d|} |||d|} |j r| | f}||j zd|j f}|| | |j |} | j |} | j |} | d}tj| | dd}|||j z| |fkr2t!d||j z| |fd||||d| |fkr+t!d |d| |fd|| ||j | ||z}| ||j z| |}t"j|d}|||j fkr-t!d |j fd|| dddd| ||j | |z}| ||j z| |}|r=| ||j | |}| ||j z| |}nd}t"j||j|j }tj|| }|||j z| |j fkr7t!d ||j z| |j fd|| ||j | |j }|dd}| || |j}||}|||fS) z#Input shape: Batch x Time x ChannelNrr;rrr@z$Attention weights should be of size z , but is z!Attention mask should be of size z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )sizervrrr!r}rtrurFcatrkrirqrzreshapebmmr{r#r functionalsoftmaxrjrrhrw)r5r~rrrrris_cross_attentionrVtgt_len_ query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r&rUzMarianAttention.forwards.T9',,..Wa{{=11DL@  L*q!'*.>.DQ.GGG(*J)!,LL  LT[[1A%B%BBLLJ;;t{{3C'D'Db#NNLL  'T[[%?%?SIIJ;;t{{='A'A2sKKLN1$5z#BJJJJ 9nQ&7%FANNNLLT[[%?%?SIIJ;;t{{='A'A2sKKL ? 8),7NDN*B > Ct{{<#>>CZP 'Z'4 +|+Z8 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**   %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL',,S4>-A7GTTL},,\r,BB  &##%%$.)::: 1t~FW11',,..11+//2q!<<|?P?PQTVZVdfmov?w?wwL',,S4>-A7GTTL  ) %1$5$5c4>7T[$\$\ !055cDN6JGU\]]LL$( !]**<4AAr()rgFTFN)NNNNF)rYrZr[r\r]floatboolrrr2rFrbr}rrUrcrds@r&rfrfjsGG  )-CCCC C  C  CC&CCCCCC>eU\eCeceeee 488<1526"'vBvB|vB#5<0vB!u|!45 vB !. vB "%,/ vB vB u|Xel3XeEL>Q5RR SvBvBvBvBvBvBvBvBr(rfceZdZdeffd Z d dejdejdejdeede ejeejff d Z xZ S) MarianEncoderLayerrnc*t|j|_t |j|j|j|j||_tj |j|_ |j |_ t|j|_|j|_tj|j|j|_tj|j|j|_tj |j|_dS)N)rhrirjrn)r1r2d_modelrhMARIAN_ATTENTION_CLASSES_attn_implementationencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrjr activation_function activation_fnactivation_dropoutrsencoder_ffn_dimfc1fc2final_layer_normr5rnr6s r&r2zMarianEncoderLayer.__init__ s 1&2MNn4,     %'L$@$@!~ #F$>?"(";9T^V-CDD9V3T^DD " T^ < <r(Fr~rrrr.c||}|||||\}}}tj||j|j}||z}||}|}|||}tj||j|j}| |}tj||j|j}||z}| |}|j tj krtj|s&tj|r9tj|j jdz }tj|| |}|f} |r| |fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r~rrrri)minmax)rrrrjrrrrrrrrQrFfloat16isinfanyisnanfinforclamp) r5r~rrrresidualrr clamp_valueoutputss r&rUzMarianEncoderLayer.forwards$!)-')+/ *8* * & |Q  --mt|VZVc-dd  =0 11-@@  **488M+B+BCC  --mt?Vaean-oo //  --mt|VZVc-dd  =0 --m<<  %- / / K & & * * , , 005 M0J0J0N0N0P0P 0 +m&9::>EK!K K<[YYYM "  '   &Gr()F) rYrZr[rr2rFrGrrrrUrcrds@r&rr s=|======.-2 00(0)0* 0 $D> 0 u (5+<"== > 00000000r(reagercbeZdZdeffd Z ddejdeejdeejd eejd eejd eejd eeejd ee dee deej eeej ej fffdZ xZ S)MarianDecoderLayerrnct|j|_t |j|j|j|jdd||_|j |_ t|j |_ |j |_ tj|j|_t |j|j|j|jd||_tj|j|_tj|j|j|_tj|j|j|_tj|j|_dS)NT)rhrirjrkrmrn)rjrkrn)r1r2rrhrrdecoder_attention_headsrrrjr rrrrrr encoder_attnencoder_attn_layer_normrsdecoder_ffn_dimrrrrs r&r2zMarianDecoderLayer.__init__Ts* 1&2MNn4,    ~ #F$>?"(";$&L$@$@!4V5PQ N  *,    (*|DN'C'C$9T^V-CDD9V3T^DD " T^ < <r(NFTr~rencoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskrr use_cacher.c x|} | |ddnd} ||| |||\}} } tj||j|j}| |z}||}d}d}|z|} | |ddnd}|||||||\}}}tj||j|j}| |z}||}| |z} |} || |}tj||j |j}| |}tj||j|j}| |z}| |}|f}|r|| |fz }| r|| fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. Nr;)r~rrrrr)r~rrrrr) rrrrjrrrrrrrrr)r5r~rrrrrrrrrself_attn_past_key_valueself_attn_weightspresent_key_valuecross_attn_present_key_valuecross_attn_weightscross_attn_past_key_valuers r&rUzMarianDecoderLayer.forwardqs,<!:H9S>"1"#5#5Y] >Bnn'3)+/ ?M? ? ; (*; --mt|VZVc-dd  =0 11-@@ (,$! ,$H@N?Yrss(;(;_c %NRN_N_+!65 :8"3 O`OO KM-/KM11-4 >G  , )+ +Gr()NNNNNNFT) rYrZr[rr2rFrbrrrrGrUrcrds@r&rrSsH=|======@268<9=26=A8<,1$(WW|W!.W ( 5 W !) 6 W "%,/ W%-U\$:W!u|!45W$D>WD>W u (51BEDU1U+V"WW XWWWWWWWWr(rcbeZdZeZdZdZdeej ej e ffdZ e dZdS)MarianPreTrainedModelmodelTmodulec|jj}t|tjrJ|jjd||j |jj dSdSt|trdSt|tj rS|jjd||j -|jj|j  dSdSdS)Nrg)meanstd) rninit_std isinstancerrsr4datanormal_rlzero_r* Embeddingr-)r5rrs r& _init_weightsz#MarianPreTrainedModel._init_weightssk" fbi ( ( ? M  & &CS & 9 9 9{&  &&((((('&  C D D ? D  - - ? M  & &CS & 9 9 9!- "6#56<<>>>>> ? ?--r(c|jj}tjgddddd|gg|j}||||d}|S)N)r r;r r;rR)rrdecoder_input_ids)rnrrFrxrRne)r5 pad_tokenr dummy_inputss r&rz"MarianPreTrainedModel.dummy_inputssdK, L"2"2"2Q2q)4L!MVZVabbb 'll955"!*  r(N)rYrZr[r config_classbase_model_prefixsupports_gradient_checkpointingrrrsrr*rpropertyrr(r&rrslL&*# ?E")R\Cf*f$g ? ? ? ?Xr(raK This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MarianConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. u4 Pytorch version of marian-nmt's transformer.h (c++). Designed for the OPUS-NMT translation checkpoints. Available models are listed [here](https://huggingface.co/models?search=Helsinki-NLP). Examples: ```python >>> from transformers import AutoTokenizer, MarianMTModel >>> src = "fr" # source language >>> trg = "en" # target language >>> model_name = f"Helsinki-NLP/opus-mt-{src}-{trg}" >>> model = MarianMTModel.from_pretrained(model_name) >>> tokenizer = AutoTokenizer.from_pretrained(model_name) >>> sample_text = "où est l'arrêt de bus ?" >>> batch = tokenizer([sample_text], return_tensors="pt") >>> generated_ids = model.generate(**batch) >>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] "Where's the bus stop?" ``` ah Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Marian uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, *optional*: `hidden_states`, *optional*: `attentions`) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. ceZdZdZddedeejffd ZdZ dZ dde j d ee j d ee j d ee jd eed eedeedeee j effdZxZS) MarianEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`MarianEncoderLayer`]. Args: config: MarianConfig embed_tokens (nn.Embedding): output embedding Nrn embed_tokensc@tj|_j|_j}j|_j|_ j rtj |nd|_ |||_n%tjj||j|_t%j||j|_tjfdt+jD|_d|_|dS)N?c.g|]}tSr)rr>rrns r&rBz*MarianEncoder.__init__.."$f$f$fA%7%?%?$f$f$fr(F)r1r2rjencoder_layerdrop layerdroprrr-max_position_embeddingsmax_source_positionsscale_embeddingmathsqrt embed_scalerrr vocab_sizer*embed_positions ModuleListrCencoder_layerslayersgradient_checkpointing post_init)r5rnrrhr6s ` r&r2zMarianEncoder.__init__ws    ~ 1N !.$*$B!393IR49Y///s  # ,D   " V-> 4K[ \ \D B  *It7G  m$f$f$f$fvOdIeIe$f$f$fgg &+# r(c|jSr0rr5s r&get_input_embeddingsz"MarianEncoder.get_input_embeddings   r(c||_dSr0rr5values r&set_input_embeddingsz"MarianEncoder.set_input_embeddings!r(rr head_mask inputs_embedsroutput_hidden_states return_dictr.c>||n |jj}||n |jj}||n |jj}||t d|G||||}|d|d}n.||dd}nt d||||j z}| |} || z} tj | |j |j} |t||j}|rdnd} |rdnd} |k|dt#|jks;Jdt#|jd |dd t'|jD]\} }|r| | fz} d }|jr!t)jg}||jkrd }|rd }nX|jr0|jr)||j| |||| nd|}n|| |||| nd|}|d} |r | |dfz} |r| | fz} |st5d| | | fDSt7| | | S)a~ Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrrrz&The head_mask should be specified for  layers, but it is for .FT)NN)rrrc3K|]}||V dSr0rr>vs r& z(MarianEncoder.forward..s(eeqWXWdWdWdWdWdeer(last_hidden_stater~ attentions)rnrruse_return_dictr#%warn_if_padding_and_no_attention_maskrrzrrrrrrjrr rQlenr enumeraterFrandrr_gradient_checkpointing_func__call__tupler)r5rrrrrrr input_shape embed_posr~encoder_statesall_attentionsidx encoder_layerto_dropdropout_probability layer_outputss r&rUzMarianEncoder.forwardse\2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**K!r;r?;;II  &',,..ss3KKTUU U   --i884;KKM((55 % 1  --mt|VZVc-dd   %7 H[\\N3=0:d  >>##A&DK  wDK8H8Hwwajaoaoaqaqrsatwww#,DK"8"8 F F C# C!/=2B!BG} #&+jnn#&77"G 1 , .4=$($E$E%.%&+4+@3d) %%MM%2M%&;D;P3VZ*; %%%M!.a 0   F!/=3C2E!E  ?+}.>>N fee]NN$Seeeee e+>Vd    r(r0)NNNNNNN)rYrZr[r\rrrrr2rr rF LongTensorrbrGrrrrrUrcrds@r&rrmsE|8BL;Q2!!!""" '+59,059,0/3&*~ ~ #~ !!12~ EL) ~   12 ~ $D> ~ 'tn~ d^~  uU\"O3 4~ ~ ~ ~ ~ ~ ~ ~ r(rceZdZdZddedeejffd ZdZ dZ dde j d ee j d ee jd ee j d ee j d ee j deeee jdee jdeedeedeedeedeee j effdZxZS) MarianDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MarianDecoderLayer`] Args: config: MarianConfig embed_tokens (nn.Embedding): output embedding NrnrcPtj|_j|_j|_j|_j rtj j nd|_ |||_n*tjjj |j|_t%jj |j|_tjfdt+jD|_d|_|dS)Nrc.g|]}tSr)rrs r&rBz*MarianDecoder.__init__..0rr(F)r1r2rjdecoder_layerdroprrr-rmax_target_positionsrrrrrrrrdecoder_vocab_sizer*rrrCdecoder_layersrrr)r5rnrr6s ` r&r2zMarianDecoder.__init__ s     ~ 1!.$*$B!8>8NW49V^444TW  # ,D   " V-FX\Xh i iD B  *FNDJd|dt/|jd|ddvt3|jD]\}}| r||fz }|jr t5jg}||jkr4|||nd}|jr?|jr8||j|||||||nd|||ndd| | }n(||||||||nd|||nd|| |  }|d}| r||| rdndfz }| r||dfz }| ||dfz }| r||fz }| r|nd}| st?d|||||fDStA|||||S)a  Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timerzEYou have to specify either decoder_input_ids or decoder_inputs_embedsrr;)rrzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Frrr9zThe `z` should be specified for rr)rrrrrrrrr rc3K|]}||V dSr0rrs r&rz(MarianDecoder.forward..s0= ===r()rr:r~rcross_attentions)!rnrrrrr#rrzr!rrr r rQrrrrjrrlogger warning_onceziprrr rFr!rr"r#r$r)r5rrrrrr9r:rrrrrr%rOrXr~all_hidden_statesall_self_attnsall_cross_attentionsnext_decoder_cache attn_mask mask_namer) decoder_layerr,rr- next_caches r&rUzMarianDecoder.forward<s `2C1N--TXT_Tq$8$D $+Jj "+!6IIDKstt t  "#..**K!r;r?;;II  &',,..ss3KKdee eDSC^!3A!6!''$  '5FXlm  9+&+%1     r(r0) NNNNNNNNNNNN)rYrZr[r\rrrrr2rr rFr.rbrGrrrrrUrcrds@r&r0r0s|8BL;Q,!!!""" '+15=A=A,07;EI59$(,0/3&*P P #P !.P  ((9: P !))9 : P EL) P 'u|4P "%e.?(@"ABP   12P D>P $D>P 'tnP d^P  uU\"$MM NP P P P P P P P r(r0zTThe bare Marian Model outputting raw hidden-states without any specific head on top.c#deZdZddgZdeffd ZdZdZdZdZ d Z d Z d e d e jfd Zeeeee d dejdeejdeejdeejdeejdeejdeejdeeeejefdeeeejdeejdeejdeedeedeedeed ef dZxZS)! MarianModelzencoder.embed_tokens.weightzdecoder.embed_tokens.weightrnct||j|j}}t j||j||_|jj r |jx}}n9tj |j}tj |j}d|_t|||_ t|||_|dSr0)r1r2rrrrrsharedrn share_encoder_decoder_embeddingscopydeepcopyrencoderr0decoderr)r5rnr-rencoder_embed_tokensdecoder_embed_tokensr6s r&r2zMarianModel.__init__s    "("5v7HZ l:v~{KK ; 7 :>+ E #7#7$(=#=#= #'=#=#= DK$V-ABB $V-ABB  r(cN|Sr0) get_encoderrrs r&rz MarianModel.get_input_embeddings+s !!66888r(c|jjr+||_|j|j_|j|j_dS||j_dSr0)rnrMrLrPrrQr s r&r z MarianModel.set_input_embeddings/sD ; 7 .DK(, DL %(, DL % % %(-DL % % %r(c|jjrtd|S)Nz`get_decoder_input_embeddings` should not be called if `config.share_encoder_decoder_embeddings` is `True`. Please use `get_input_embeddings` instead.)rnrMr# get_decoderrrs r&get_decoder_input_embeddingsz(MarianModel.get_decoder_input_embeddings7sG ; 7 H !!66888r(cT|jjrtd||j_dS)Na `config.share_encoder_decoder_embeddings` is set to `True` meaning the decoder input embeddings are shared with the encoder. In order to set the decoder input embeddings, you should simply set the encoder input embeddings by calling `set_input_embeddings` with the appropriate embeddings.)rnrMr#rQrr s r&set_decoder_input_embeddingsz(MarianModel.set_decoder_input_embeddings?s9 ; 7 r  %* !!!r(c|jSr0)rPrs r&rUzMarianModel.get_encoderH |r(c|jSr0rQrs r&rXzMarianModel.get_decoderKr]r(new_num_tokensr.c*|jjrtd|}|||}|||}||S||j_||SNz`resize_decoder_token_embeddings` should not be called if `config.share_encoder_decoder_embeddings` is `True`. Please use `resize_token_embeddings` instead.)rnrMr#rY_get_resized_embeddingsr[r5 tie_weights)r5r`old_embeddingsnew_embeddings model_embedss r&resize_decoder_token_embeddingsz+MarianModel.resize_decoder_token_embeddingsNs ; 7 K  ::<<55nnUU )).99988::  ! *8 & r( output_typerNrrrdecoder_attention_maskrdecoder_head_maskr9encoder_outputsr:rdecoder_inputs_embedsrrrrcd| | n |jj} ||n |jj}| | n |jj} ||n |jj}|||||| | ||}ne|rct |tsNt|dt|dkr|dndt|dkr|dnd}| |||d|||| | | | || }|s||zSt|j |j |j |j|j|j |j |jS) aX Returns: Example: ```python >>> from transformers import AutoTokenizer, MarianModel >>> tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> model = MarianModel.from_pretrained("Helsinki-NLP/opus-mt-en-de") >>> inputs = tokenizer("Studies have been shown that owning a dog is good for you", return_tensors="pt") >>> decoder_inputs = tokenizer( ... " Studien haben gezeigt dass es hilfreich ist einen Hund zu besitzen", ... return_tensors="pt", ... add_special_tokens=False, ... ) >>> outputs = model(input_ids=inputs.input_ids, decoder_input_ids=decoder_inputs.input_ids) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 26, 512] ```N)rrrrrrrrrr;r rrrrrr9r:rrrrr)rr:decoder_hidden_statesdecoder_attentionsr=encoder_last_hidden_staterencoder_attentions)rnrrrrrPrrrrQrrr:r~rr=)r5rrrrkrrlr9rmr:rrnrrrrdecoder_outputss r&rUzMarianModel.forwardfsV2C1N--TXT_Tq$8$D $+Jj "+!6IIDK^ $D>^ 'tn^ d^!^ " #^ ^ ^ ]\DC^ ^ ^ ^ ^ r(rJzNThe Marian Model with a language modeling head. Can be used for summarization.c&4eZdZdZgdZddgZgdZdeffd ZdZ d Z d*d e d e e d e jffd Zd*d e d e jfdZdZd e d d fdZdZde jfdZdZeeeeeee d+dejde ejde ejde ejde ejde ejde ejde e e!eje"fde e!e!ej#d e ej#d!e ej#d"e ejd#e e$d$e e$d%e e$d&e e$d ef"d'Z%d"ejfd(Z&e'd)Z(xZ)S), MarianMTModelr)final_logits_biaszencoder.embed_positions.weightzdecoder.embed_positions.weightz$model.encoder.embed_positions.weightz$model.decoder.embed_positions.weight)z!model.encoder.embed_tokens.weightz!model.decoder.embed_tokens.weightlm_head.weightrncZt|t||_|jr|jn|j}|dtj d|ftj |j |d|_ |dS)Nr{rFrp)r1r2rJrrMrr5register_bufferrFzerosrrsrlm_headr)r5rntarget_vocab_sizer6s r&r2zMarianMTModel.__init__s     (( 171XwF--^d^w 0%+qBS>T2U2UVVVy1BOOO  r(c4|jSr0)rrUrs r&rUzMarianMTModel.get_encoderz%%'''r(c4|jSr0)rrXrs r&rXzMarianMTModel.get_decoderrr(Nr`pad_to_multiple_ofr.ct||}|jjr|||Sr0)r1resize_token_embeddingsrnrM_resize_final_logits_bias)r5r`rrfr6s r&rz%MarianMTModel.resize_token_embeddingssD88I[\\ ; 7 ;  * *> : : :r(c|}||||}|||jjd}|jjr ||j_|jjr_|K|jj s?|}| ||}| ||S)Nr) rrcr r4r!rnrMr5get_output_embeddingstie_word_embeddings_get_resized_lm_headset_output_embeddings)r5r`rargsrerf old_lm_head new_lm_heads r&_resize_token_embeddingsz&MarianMTModel._resize_token_embeddingss224455nnVhii !!.111'.4Q7 ; 7 <-;DK * K 8 4**,,8K394466K33KPPK  & &{ 3 3 3((***r(c0|jjrtd|j}|||}|j||K|jjs?|}| ||}| ||j}||S||j_ | | ||Srb)rnrMr#rrYrcr[rrrrr5rdr)r5r`rerfrrrgs r&rhz-MarianMTModel.resize_decoder_token_embeddingss ; 7 K  @@BB55nnUU //???  % % ' ' 3DK<[ 34466K33KPPK  & &{ 3 3 3z>>@@  ! *8 &  &&~666r(c|jjd}||kr|jddd|f}nBtjd||z f|jj}tj|j|gd}|d|dS)Nrrrrr{)r{r!rFrrRrr~)r5r`old_num_tokensnew_bias extra_biass r&rz'MarianMTModel._resize_final_logits_bias$s/5b9 ^ + +-aaa..@AHHa.)H%IRVRhRopppJy$"8*!E1MMMH 0(;;;;;r(c|jSr0rrs r&rz#MarianMTModel.get_output_embeddings-r]r(rfc||_dSr0rr5rfs r&rz#MarianMTModel.set_output_embeddings0 % r(cT|}|Rt|jddr<|}|||t|jddrnt|jddrXt ||jrt||j}||j |j |jd}||_ | D]&}t |dr| 'dS) z Tie the weights between the input embeddings and the output embeddings. If the `torchscript` flag is set in the configuration, can't handle parameter sharing so we are cloning the weights instead. NrTis_encoder_decoderFtie_encoder_decoderrP _tie_weights)rgetattrrnrXr_tie_or_clone_weightshasattrr_tie_encoder_decoder_weightsrPrQ_dynamic_tied_weights_keysmodulesr)r5output_embeddingsword_embeddings tied_weightsrs r&rdzMarianMTModel.tie_weights3s/!6688  (WT[BWY]-^-^ ("..00EEGGO  & &'8/ J J J 4; 4e < < ;VkmrAsAs ;tT344 =tT%;<<<< dlD,BIL /;D +llnn & &Fv~.. &##%%% & &r(rirrrrkrrlr9rmr:rrnlabelsrrrrc||n |jj}| G| rtdd} |'| %t | |jj|jj}|||||||||| | | | |||}||d|j z}d}| Kt}|| d|jj | d}|s|f|ddz}||f|zn|St|||j|j|j|j|j|j|j S) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Returns: NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F)rrrmrkrrlr9r:rrnrrrrrrr) losslogitsr:rqrrr=rsrrt)rnrr>warningr'rrrrr{rrzr5rr:rqrrr=rsrrt)r5rrrrkrrlr9rmr:rrnrrrrrr lm_logitsmasked_lm_lossloss_fctoutputs r&rUzMarianMTModel.forwardOs>&1% past_statebeam_idxs r&rz/MarianMTModel._reorder_cache..sCrrU_j--aZ=N1O1OPPrrrrrrr(r;r$r:rreordered_past layer_pasts ` r&_reorder_cachezMarianMTModel._reorder_cachesm)  J rrrrcmnpopnpcqrrrrrQRR.! NNr(r0)NNNNNNNNNNNNNNNN)*rYrZr[r_keys_to_ignore_on_load_missing_keys_to_ignore_on_savervrr2rUrXr]rrrrrrhrrrrdrrwrrrxrMARIAN_GENERATION_EXAMPLErFr.rbrrrrGrrUrr^rrcrds@r&rzrzs '''# FGmnuuu |      ((((((cxX[}hjht++s+_a_k++++,@<<<<<<&BL&&&&&&&8+*+BCC?YYY122'+158<9=,0487;QUEI59=A-1$(,0/3&*#M M #M !.M $E$45 M !) 6 M EL) M $EL1M 'u|4M "%el(;_(L"MNM "%e.?(@"ABM   12M  ((9:M )*M D>M $D>M 'tn!M "d^#M $ %M M M 32ZYDCM ^hELhhhh\r(rzc(eZdZdZfdZdZxZS)MarianDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. crt|t||_dSr0)r1r2r0rQrs r&r2zMarianDecoderWrapper.__init__s.    $V,, r(c|j|i|Sr0r_)r5rkwargss r&rUzMarianDecoderWrapper.forwardst|T,V,,,r()rYrZr[r\r2rUrcrds@r&rrsQ ------------r(rc eZdZdgZfdZdZdZdZdZdZ dZ e e e  dd ejd eejd eejdeejdeejdeejdeeejdeejdeejdeedeedeedeedeee ffdZedZxZS)MarianForCausalLMr|c*tj|}d|_d|_t |t ||_tj |j |j d|_ | dS)NTFrp)rNrOrkrr1r2rrrrs hidden_sizerrrrs r&r2zMarianForCausalLM.__init__sv&& $)!    )&11 y!3V5FUSSS  r(c$|jjjSr0rrQrrs r&rz&MarianForCausalLM.get_input_embeddingssz!..r(c(||jj_dSr0rr s r&r z&MarianForCausalLM.set_input_embeddingss*/ '''r(c|jSr0rrs r&rz'MarianForCausalLM.get_output_embeddingsr]r(c||_dSr0rrs r&rz'MarianForCausalLM.set_output_embeddingsrr(c||j_dSr0rrQ)r5rQs r& set_decoderzMarianForCausalLM.set_decoders$ r(c|jjSr0rrs r&rXzMarianForCausalLM.get_decoders z!!r(riNrrrrrr9r:rrrrrrr.c<| | n |jj} | | n |jj} | | n |jj} |j||||||||| | | |  }||d}d}| e| |j} t}|| d|jj | d}| s|f|ddz}||f|zn|St|||j |j|j|jS)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. The two additional tensors are only required when the model is used as a decoder in a Sequence to Sequence model. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Returns: Example: ```python >>> from transformers import AutoTokenizer, MarianForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-fr-en") >>> model = MarianForCausalLM.from_pretrained("Helsinki-NLP/opus-mt-fr-en", add_cross_attention=False) >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nrprrr)rrr:r~rr=)rnrrrrrQrrrRrrzrrr:r~rr=)r5rrrrrr9r:rrrrrrrrrrrs r&rUzMarianForCausalLM.forwardseL2C1N--TXT_Tq$8$D $+Jj &1%.wsCnnU_j--aZ=N1O1OPPnnnnnnr(rrs ` r&rz MarianForCausalLM._reorder_cachersQ)  J nnnncmnnnnn NNr() NNNNNNNNNNNNN)rYrZr[rvr2rr rrrrXrrrxrFr.rrbrGrrrrrUr^rrcrds@r&rrs*+     ///000&&&%%%"""+L[jkkk'+15=A>B,07;=A59-1$(,0/3&*N N #N !.N  ((9: N !)): ; N EL) N 'u|4N "$u'8"9:N   12N )*N D>N $D>N 'tnN d^N  u77 8N N N lkN `\r(r)?r\rNrtypingrrrrnumpyr<rFtorch.utils.checkpointrtorch.nnr activationsr generationr modeling_attn_mask_utilsr r modeling_outputsrrrrrmodeling_utilsrutilsrrrrrconfiguration_marianr get_loggerrYr>rx_CHECKPOINT_FOR_DOCrbr]r'rr*ModulerfrrrrMARIAN_START_DOCSTRINGrrwrr0rJrzrrrr(r&rskDC //////////// %%%%%%!!!!!!))))))eeeeeeee.-----/.....  H % % 2 %, c [^     *****",***F[B[B[B[B[Bbi[B[B[B~CCCCCCCCL$_5uuuuuuuupO< 2Xvg g g g g )g g g Tu u u u u )u u u pZ\rt t t t t 't t t nTVlaaaaa)?aaaJ - - - - -0 - - -yyyyy-yyyyyr(