gddlZddlZddlZddlmZddlmZmZmZddl Z ddl m Z ddl m Z m Z mZddlmZddlmZmZmZmZmZmZmZmZdd lmZdd lmZmZdd lm Z m!Z!m"Z"m#Z#m$Z$m%Z%d d l&m'Z'e$j(e)Z*dZ+dZ,dZ-dZ.dZ/dZ0dZ1dZ2dZ3dZ4dZ5dZ6dZ7dZ8Gdde j9Z:e j;e:dZ<Gdd e j9Z=Gd!d"e j9Z>Gd#d$e j9Z?Gd%d&e j9Z@Gd'd(e j9ZAGd)d*e j9ZBGd+d,e j9ZCGd-d.e j9ZDGd/d0e j9ZEGd1d2e j9ZFGd3d4e j9ZGGd5d6e j9ZHGd7d8e j9ZIGd9d:e j9ZJGd;de j9ZLGd?d@e j9ZMGdAdBe j9ZNGdCdDeZOeGdEdFe ZPdGZQdHZRe"dIeQGdJdKeOZSe"dLeQGdMdNeOZTe"dOeQGdPdQeOZUGdRdSe j9ZVe"dTeQGdUdVeOZWe"dWeQGdXdYeOZXe"dZeQGd[d\eOZYe"d]eQGd^d_eOZZe"d`eQGdadbeOZ[dS)cN) dataclass)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)MobileBertConfigzgoogle/mobilebert-uncasedrz mrm8488/mobilebert-finetuned-nerzK['I-ORG', 'I-ORG', 'O', 'O', 'O', 'O', 'O', 'I-LOC', 'O', 'I-LOC', 'I-LOC']gQ?z#csarron/mobilebert-uncased-squad-v2z'a nice puppet'gףp= @ zlordtt13/emo-mobilebertz'others'z4.72c  ddl}ddl}ddl}n)#t$rtdwxYwt j|}t d||j |}g}g} |D]j\} } t d| d| |j || } | | | | kt|| D]\} } | dd} | d d } | d d } | d d} | d} t#d| Dr1t dd| |} | D]H}|d|r|d|}n|g}|ddks |ddkrt)| d} n|ddks |ddkrt)| d} n|ddkrt)| d} nv|ddkrt)| d} nY t)| |d} nA#t*$r4t dd| YwxYwt-|dkrt/|d}| |} J|ddd krt)| d} n|dkr|| } | j| jksJd!| jd"| jd#n/#t4$r"}|xj| j| jfz c_d}~wwxYwt d$| t9j| | _|S)%z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape ffn_layerffn FakeLayerNorm LayerNormextra_output_weightsz dense/kernelbert mobilebert/c3K|]}|dvV dS))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0ns n/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/mobilebert/modeling_mobilebert.py z0load_tf_weights_in_mobilebert..qs<   n n      z Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifierri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendzipreplacesplitanyjoin fullmatchgetattrAttributeErrorlenint transposeshapeAssertionErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathrBnptftf_path init_varsnamesarraysnamer[arraypointerm_name scope_namesnumes r4load_tf_weights_in_mobilebertrqPsa     Q      goo011G KKBBBCCC''00I E F  e BBB5BBCCC&&w55 T e5&))1/1/ e||K//||O[99||2NCC||FL11zz#          KK4CHHTNN44 5 5 5  ' 'F||,f55 ' hhy&99 %h 1~))[^w-F-F!'844Q=00KNf4L4L!'622Q#333!'844Q7**!'<88%g{1~>>GG%KK J J"%88;PP ^^J// \\*-- r6)NNNN) rrr__doc__rwrr^ LongTensor FloatTensorrrrrs@r4rrsQQ     015593759 00E,-0!!120u/0 0   12 0  00000000r6rceZdZfdZdZ d dejdejdejdeejdeejd ee d e ejfd Z xZ S) MobileBertSelfAttentionct|j|_t|j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j r|jn|j |j|_ tj|j|_dSru)rvrwnum_attention_headsrYtrue_hidden_sizeattention_head_size all_head_sizerrquerykeyuse_bottleneck_attentionrrrattention_probs_dropout_probrr{rbr~s r4rwz MobileBertSelfAttention.__init__s #)#= #&v'>A['[#\#\ !58PPYv68JKK 9V4d6HIIY'-'F ^F # #FL^`d`r   z&"EFF r6c|dd|j|jfz}||}|ddddS)Nrrr@rr )rrrviewpermute)r{x new_x_shapes r4transpose_for_scoresz,MobileBertSelfAttention.transpose_for_scoressPffhhssmt'?AY&ZZ FF;  yyAq!$$$r6N query_tensor key_tensor value_tensorattention_mask head_maskoutput_attentionsrc||}||}||} ||} ||} || } t j| | dd} | tj|j z } || |z} tj | d}| |}|||z}t j|| }|dddd}|dd|jfz}||}|r||fn|f}|S)Nrrrr@rr )rrrrr^matmulrZmathsqrtrrrsoftmaxrr contiguousrrr)r{rrrrrrmixed_query_layermixed_key_layermixed_value_layer query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputss r4rzMobileBertSelfAttention.forward s!JJ|44((:.. JJ|44//0ABB --o>> //0ABB !< Y5H5HR5P5PQQ+di8P.Q.QQ  %/.@ -//0@b/II,,77  - 9O _kBB %--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S%**+BCC 6G]=/22mM]r6NNN) rrrrwrr^rrrboolrrrrs@r4rrs G G G G G%%%7;15,0$$l$L$l $ !!23 $ E-. $$D>$ u| $$$$$$$$r6rcPeZdZfdZdejdejdejfdZxZS)MobileBertSelfOutputcLt|j|_tj|j|j|_t|j|j|j |_ |js tj |j |_ dSdSNr})rvrwuse_bottleneckrrrdenserrlayer_norm_epsr&rrrrs r4rwzMobileBertSelfOutput.__init__2s $3Yv68OPP  !:;F u| r6rcBeZdZfdZdejdejfdZxZS)MobileBertIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSru) rvrwrrrintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrs r4rwzMobileBertIntermediate.__init__usn Yv68PQQ f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r6rrcZ||}||}|Sru)rrr{rs r4rzMobileBertIntermediate.forward}s, =11 00?? r6rrs@r4r r ts^99999U\elr6r cPeZdZfdZdejdejdejfdZxZS)OutputBottleneckc"ttj|j|j|_t|j|j|j |_ tj |j |_ dSr)rvrwrrrrrrrrr&rrrrs r4rwzOutputBottleneck.__init__sm Yv68JKK  !:;F>>>> U\ELUZUar6rc^eZdZfdZdejdejdejdejfdZxZS)MobileBertOutputcft|j|_tj|j|j|_t|j |j|_ |js tj |j |_ dSt||_dSru)rvrwrrrr rrrrr&rrrr bottleneckrs r4rwzMobileBertOutput.__init__s $3Yv79PQQ  !:;F>,9J*JKKLL>>,9J*JKKL??<9JKKLr6rrs@r4rrsu77777 #(< DIL ejeq          r6rcBeZdZfdZdejdejfdZxZS)BottleneckLayercttj|j|j|_t|j|j|j |_ dSr) rvrwrrrintra_bottleneck_sizerrrrr&rs r4rwzBottleneckLayer.__init__sY Yv163OPP  !:;F%>"  ( ].014 4  - ]%)^^M%B%B "*,BMSmn n!=-A[\ \r6 rrrrwr^rrrrrs@r4r&r&sm55555]U\]eEL6I]]]]]]]]r6r&cPeZdZfdZdejdejdejfdZxZS) FFNOutputcttj|j|j|_t|j|j|j |_ dSr) rvrwrrr rrrrrr&rs r4rwzFFNOutput.__init__sY Yv79PQQ  !:;F> $7GG r6rrs@r4r5r5s^((((( U\elr6r5c eZdZfdZ d dejdeejdeejdeede ejf dZ xZ S) MobileBertLayerctj|_j|_t |_t |_t|_ |jrt|_ jdkrz,MobileBertLayer.__init__..s!%k%k%k1hv&6&6%k%k%kr6)rvrwrnum_feedforward_networksrr*r r7rrr&rr ModuleListranger$rs `r4rwzMobileBertLayer.__init__s $3(.(G%,V4426::&v..   1(00DO  *Q . .}%k%k%k%kfFehiFi@j@j%k%k%kllDHHH / .r6Nrrrrrc |jr||\}}}}n |gdz\}}}}||||||||} | d} | f} | dd} |jdkr+t |jD]\} }|| } | | fz } || }||| |}|f| ztj d||||| |fz| z} | S)Nr,)rrri) rrr*rA enumerater$r7rr^tensor)r{rrrrrrrrself_attention_outputsrsri ffn_moduler9rs r4rzMobileBertLayer.forwardsZ   VBF//R_B`B` ?L*lKKCP/TUBU ?L*lK!%      /"0" " 2!4  (,  (A - -!*48!4!4 ) ) :#-:.>#?#? &(("//0@AA{{#68H-XX O  T"" #   r6r) rrrrwr^rrrrrrrrs@r4r;r;s m m m m m 7;15,0 ..|.!!23.E-. . $D> . u|  ........r6r;ceZdZfdZ d dejdeejdeejdeed eed eed e e e ffd Z xZ S)MobileBertEncodercttjfdt jD|_dS)Nc.g|]}tSr1)r;r>s r4r@z.MobileBertEncoder.__init__..3s!#e#e#eOF$;$;#e#e#er6)rvrwrrBrCnum_hidden_layerslayerrs `r4rwzMobileBertEncoder.__init__1sO ]#e#e#e#eU6KcEdEd#e#e#eff r6NFTrrrroutput_hidden_states return_dictrc"|rdnd}|rdnd}t|jD]7\} } |r||fz}| |||| |} | d}|r || dfz}8|r||fz}|std|||fDSt|||S)Nr1rrc3K|]}||V dSrur1)r2vs r4r5z,MobileBertEncoder.forward..Ts(hhqZ[ZgZgZgZgZghhr6)last_hidden_stater attentions)rErPtupler ) r{rrrrrQrRall_hidden_statesall_attentionsrI layer_modulers r4rzMobileBertEncoder.forward5s#7@BBD0:d(44 F FOA|# I$58H$H!(L! ! M *!,M  F!/=3C2E!E  E 1]4D D  ihh]4E~$Vhhhhh h+;LYg    r6)NNFFT)rrrrwr^rrrrrrr rrrs@r4rLrL0sggggg7;15,1/4&*" " |" !!23" E-. " $D> " 'tn " d^"  uo% &" " " " " " " " r6rLcBeZdZfdZdejdejfdZxZS)MobileBertPoolerct|j|_|jr&t j|j|j|_dSdSru)rvrwclassifier_activation do_activaterrrrrs r4rwzMobileBertPooler.__init__[sY !7   K6#5v7IJJDJJJ K Kr6rrc|dddf}|js|S||}tj|}|S)Nr)r`rr^tanh)r{rfirst_token_tensor pooled_outputs r4rzMobileBertPooler.forwardasO+111a40 !% % JJ'9::M!J}55M r6rrs@r4r]r]ZscKKKKK !U\ !el ! ! ! ! ! ! ! !r6r]cBeZdZfdZdejdejfdZxZS)!MobileBertPredictionHeadTransformcXttj|j|j|_t |jtrt|j|_ n |j|_ td|j|j |_ dS)Nrr)rvrwrrrrr r rr transform_act_fnrrr&rs r4rwz*MobileBertPredictionHeadTransform.__init__ns Yv163EFF f' - - 6$*6+<$=D ! !$*$5D ! .v/AvG\]]]r6rrc||}||}||}|Sru)rrhr&rs r4rz)MobileBertPredictionHeadTransform.forwardws= =11 --m<< }55 r6rrs@r4rfrfmsc^^^^^U\elr6rfcJeZdZfdZddZdejdejfdZxZS)MobileBertLMPredictionHeadctt||_t j|j|j|jz d|_ t j|j|jd|_ t j tj |j|_|j|j _dS)NF)r<)rvrwrf transformrrrrrrdecoderrxr^ryr<rs r4rwz#MobileBertLMPredictionHead.__init__s :6BBYv0&2DvG\2\chiii y!68IPUVVV LV->!?!?@@  I r6rNc(|j|j_dSru)r<rnr{s r4 _tie_weightsz'MobileBertLMPredictionHead._tie_weightss I r6rc||}|tj|jj|jjgd}||jjz }|S)Nrr) rmrr^rrnr9trr<rs r4rz"MobileBertLMPredictionHead.forwardsh}55 %,,UY 8K8M8M8O8OQUQ[Qb7cij-k-k-kll ** r6)rN) rrrrwrqr^rrrrs@r4rkrk~sr & & & & &&&&&U\elr6rkcBeZdZfdZdejdejfdZxZS)MobileBertOnlyMLMHeadcptt||_dSru)rvrwrk predictionsrs r4rwzMobileBertOnlyMLMHead.__init__s/ 5f==r6sequence_outputrc0||}|Sru)rw)r{rxprediction_scoress r4rzMobileBertOnlyMLMHead.forwards ,,_==  r6rrs@r4rurus^>>>>>!u|! !!!!!!!!r6ruc\eZdZfdZdejdejdeejfdZxZS)MobileBertPreTrainingHeadsctt||_t j|jd|_dSNr@)rvrwrkrwrrrseq_relationshiprs r4rwz#MobileBertPreTrainingHeads.__init__sF 5f== " &*>>>>.- v 6 7 7 * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r6N) rrrrr config_classrqload_tf_weightsbase_model_prefixrr1r6r4rrs? $L3O$*****r6rceZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ee ejed<dZ ee ejed<dS)MobileBertForPreTrainingOutputab Output type of [`MobileBertForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossprediction_logitsseq_relationship_logitsrrW)rrrrrrr^r__annotations__rrrrrWr1r6r4rrs2)-D(5$ %,,,+/u(///15U.5558>> from transformers import AutoTokenizer, MobileBertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForNextSentencePrediction.from_pretrained("google/mobilebert-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> loss = outputs.loss >>> logits = outputs.logits ```rzoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nrrrr@r) warningswarn FutureWarningpoprbrr)rr rrrrW)r{rrrrrrrrrQrRkwargsrrdrrrrs r4rz+MobileBertForNextSentencePrediction.forward{sIX !F * * M%    ZZ 566F%0%Q 'tnQ d^Q  u11 2Q Q Q fenmQ Q Q Q Q r6rz MobileBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZeedeee e e e  dde ejde ejde ejde ejd e ejd e ejd e ejd e ed e ede edeeeje ffdZxZS)#MobileBertForSequenceClassificationcdt||j|_||_t ||_|j|jn|j}tj ||_ tj |j |j|_ |dSru)rvrw num_labelsrbrr)classifier_dropoutrrrrrrr?rr{rbrr~s r4rwz,MobileBertForSequenceClassification.__init__s     + )&11)/)B)NF % %TZTn z"455 )F$68IJJ r6rrNrrrrrrrrrQrRrc | | n |jj} ||||||||| |  } | d} || } || } d}|Z|jjf|jdkr d|j_nN|jdkr7|jtj ks|jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n|| |}n|jjdkrGt}|| d|j|d}n*|jjdkrt}|| |}| s| f| ddz}||f|zn|St!|| | j| j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr regressionsingle_label_classificationmulti_label_classificationrr@r)rbrr)rr? problem_typerrr^rrYr squeezer rrrrrW)r{rrrrrrrrrQrRrrdrrrrs r4rz+MobileBertForSequenceClassification.forwards6&1%E 'tnE d^E  uU\"$<< =E E E nmE E E E E r6rz MobileBert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZfdZeedeee e e e e e ddeejdeejdeejdeejd eejd eejd eejd eejd eedeedeedeeeje ffdZxZS)MobileBertForQuestionAnsweringct||j|_t|d|_t j|j|j|_| dSr) rvrwrrr)rrr qa_outputsrrs r4rwz'MobileBertForQuestionAnswering.__init__Bsj     +)&EJJJ)F$68IJJ r6r)rrrqa_target_start_indexqa_target_end_indexrrNrrrrrrstart_positions end_positionsrrQrRrc | | n |jj} |||||||| | |  } | d} || }|dd\}}|d}|d}d}||t|dkr|d}t|dkr|d}|d}| d|}| d|}t|}|||}|||}||zdz }| s||f| ddz}||f|zn|St|||| j | j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrrr) ignore_indexr@)r start_logits end_logitsrrW)rbrr)rrRrrrXrclampr rrrW)r{rrrrrrr r rrQrRrrxrrrr ignored_indexr start_lossend_lossrs r4rz&MobileBertForQuestionAnswering.forwardLsD&1%H 'tnH d^H  uU\"$@@ AH H H nmH H H H H r6rz MobileBert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. ceZdZfdZeedeee e  dde e j de e j de e j de e j d e e j d e e j d e e j d e ed e ede edeee j e ffdZxZS)MobileBertForMultipleChoicec4t|t||_|j|jn|j}t j||_t j |j d|_ | dS)Nr) rvrwrr)rrrrrrrr?rrs r4rwz$MobileBertForMultipleChoice.__init__s    )&11)/)B)NF % %TZTn z"455 )F$6:: r6z(batch_size, num_choices, sequence_lengthrNrrrrrrrrrQrRrc | | n |jj} | |jdn |jd} |)|d|dnd}|)|d|dnd}|)|d|dnd}|)|d|dnd}|=|d|d|dnd}||||||||| |  } | d} || } || }|d| }d}|t}|||}| s|f| ddz}||f|zn|St||| j | j S)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrrrrr@r) rbrr[rrr)rr?r rrrW)r{rrrrrrrrrQrR num_choicesrrdrreshaped_logitsrrrs r4rz#MobileBertForMultipleChoice.forwards*6&1%G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqM[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  // ))%'/!5#"      ]33 // ++b+66  '))H8OV44D F%''!""+5F)-)9TGf$$v E("!/)     r6r)rrrrwrrrrrrrrr^rrrrrrrs@r4rrs     +*#**+UVV &-$-11515/3,004)-,0/3&*@ @ EL)@ !.@ !. @ u|, @ EL) @   -@ &@ $D>@ 'tn@ d^@  uU\"$== >@ @ @ @ @ @ @ @ r6rz MobileBert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. ceZdZfdZeedeee e e e  dde ejde ejde ejde ejd e ejd e ejd e ejd e ed e ede edeeeje ffdZxZS) MobileBertForTokenClassificationcZt||j|_t|d|_|j|jn|j}tj||_ tj |j |j|_ | dSr)rvrwrrr)rrrrrrrr?rrs r4rwz)MobileBertForTokenClassification.__init__ s     +)&EJJJ)/)B)NF % %TZTn z"455 )F$68IJJ r6rrNrrrrrrrrrQrRrc | | n |jj} ||||||||| |  } | d} || } || } d}|Ft }|| d|j|d}| s| f| ddz}||f|zn|St|| | j | j S)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrr@r) rbrr)rr?r rrrrrW)r{rrrrrrrrrQrRrrxrrrrs r4rz(MobileBertForTokenClassification.forwards2&1%2 'tn2 d^2  uU\"$99 :2 2 2 nm2 2 2 2 2 r6r!)\rrHr dataclassesrtypingrrrr^rtorch.nnrr r activationsr modeling_outputsr rrrrrrrmodeling_utilsr pytorch_utilsrrutilsrrrrrrconfiguration_mobilebertr get_loggerrrFrrr$r%r&rrrrrrrrrqModulersr&rrrrrr rrr r&r1r5r;rLr]rfrkrur|rrMOBILEBERT_START_DOCSTRINGrrrrrrrrrr!r1r6r4r3s. !!!!!!)))))))))) AAAAAAAAAA!!!!!!                    .-----QQQQQQQQ766666  H % %1$(J$l!;'+D''!KKK\66666RY666& 9 9IIIII29IIIX77777bi777t29"/////")///d     RY        ry   ry0     bi   !]!]!]!]!]!]!]!]H             ry   <<<<<bi<<<~' ' ' ' '  ' ' ' T!!!!!ry!!!& ",!!!!!BI!!! 9 9 9 9 9 9 9 9********6 :::::[:: :B /djh h h h h /h h h V i i i i i 8i i i XTVpqqT T T T T 5T T rqT n&&&&&BI&&&Z] ] ] ] ] *C] ] ] @ ] ] ] ] ] *C] ] ] @ ] ] ] ] ] %>] ] ] @ V V V V V ";V V V r I I I I I '@I I I I I r6