gdZddlZddlZddlmZddlmZmZmZddl Z ddl Z ddl m Z ddl m Z mZmZddlmZmZdd lmZmZmZmZmZmZdd lmZmZdd lmZmZm Z dd l!m"Z"m#Z#m$Z$m%Z%d dl&m'Z'e%j(e)Z*dZ+dZ,dZ-Gdde j.Z/GddeZ0Gdde j.Z1Gdde j.Z2Gdde j.Z3Gdde j.Z4Gdde j.Z5Gd d!e j.Z6Gd"d#e j.Z7Gd$d%e j.Z8Gd&d'e j.Z9Gd(d)e j.Z:d*Z;d+Z<e#d,e;Gd-d.e0Z=Gd/d0e j.Z>e#d1e;Gd2d3e0Z?Gd4d5e j.Z@e#d6e;Gd7d8e0ZAe#d9e;Gd:d;e0ZBe#de0ZCe#d?e;Gd@dAe0ZDdS)BzPyTorch ConvBERT model.N) attrgetter)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNget_activation)"BaseModelOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModelSequenceSummary)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging)ConvBertConfigzYituTech/conv-bert-baserc ddl}n)#t$rtdwxYwtj|}td||j |}i}|D]E\}}td|d||j ||} | ||<Fddd d d d d d} |j dkrd} nd} t|j D]:} d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d | d| d!<d| d"| d| d#<d| d$| d| d%<d| d&| d| d'<d| d(| d| d)<d| d*| d| d+<d| d,| d| d-<d| d.| d| d/<d| d0| d| d1<d| d2| d| d3<d| d4| d| d5<d| d6| d7| d| d8<d| d6| d9| d| d:<d| d;| d7| d| d<<d| d;| d9| d| d=<d| d>| d| d?<d| d@| d| dA<<|D]7} | d}t|}||}| |}t!j||}tdB|dC|dD|d7r1|dEs|dFs|j}|dGr|ddHd}|dIr|dHdd}|dJr|dK}||_9|S)Lz'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape z"electra/embeddings/word_embeddingsz&electra/embeddings/position_embeddingsz(electra/embeddings/token_type_embeddingsz"electra/embeddings/LayerNorm/gammaz!electra/embeddings/LayerNorm/betaz!electra/embeddings_project/kernelzelectra/embeddings_project/bias)z!embeddings.word_embeddings.weightz%embeddings.position_embeddings.weightz'embeddings.token_type_embeddings.weightzembeddings.LayerNorm.weightzembeddings.LayerNorm.biaszembeddings_project.weightzembeddings_project.biasrg_densedensezelectra/encoder/layer_z/attention/self/query/kernelzencoder.layer.z.attention.self.query.weightz/attention/self/query/biasz.attention.self.query.biasz/attention/self/key/kernelz.attention.self.key.weightz/attention/self/key/biasz.attention.self.key.biasz/attention/self/value/kernelz.attention.self.value.weightz/attention/self/value/biasz.attention.self.value.biasz./attention/self/conv_attn_key/depthwise_kernelz4.attention.self.key_conv_attn_layer.depthwise.weightz./attention/self/conv_attn_key/pointwise_kernelz4.attention.self.key_conv_attn_layer.pointwise.weightz"/attention/self/conv_attn_key/biasz(.attention.self.key_conv_attn_layer.biasz'/attention/self/conv_attn_kernel/kernelz(.attention.self.conv_kernel_layer.weightz%/attention/self/conv_attn_kernel/biasz&.attention.self.conv_kernel_layer.biasz&/attention/self/conv_attn_point/kernelz%.attention.self.conv_out_layer.weightz$/attention/self/conv_attn_point/biasz#.attention.self.conv_out_layer.biasz/attention/output/dense/kernelz.attention.output.dense.weightz!/attention/output/LayerNorm/gammaz".attention.output.LayerNorm.weightz/attention/output/dense/biasz.attention.output.dense.biasz /attention/output/LayerNorm/betaz .attention.output.LayerNorm.biasz/intermediate/z/kernelz.intermediate.dense.weightz/biasz.intermediate.dense.biasz/output/z.output.dense.weightz.output.dense.biasz/output/LayerNorm/gammaz.output.LayerNorm.weightz/output/LayerNorm/betaz.output.LayerNorm.biaszTF: z, PT:  z/intermediate/g_dense/kernelz/output/g_dense/kernelz/depthwise_kernelz/pointwise_kernelz/conv_attn_key/bias) tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variable num_groupsrangenum_hidden_layersnamed_parametersrtorch from_numpyendswithTpermute unsqueezedata)modelconfigtf_checkpoint_pathtftf_path init_varstf_datanameshapearray param_mappinggroup_dense_namejparam param_name retrieverresulttf_namevalues j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/convbert/modeling_convbert.pyload_tf_weights_in_convbertrO0sX   Q      goo011G KKBBBCCC''00IG  e BBB5BBCCC&&w55 .R1Y3]'K%H%H#DM1$" 6+ , ,CwCw DQ D D D FqFFFG CQ B B B DqDDDE CQ B B B DqDDDE AQ @ @ @ BqBBBC EQ D D D FqFFFG CQ B B B DqDDDE WQ V V V ^q^^^_ WQ V V V ^q^^^_ KQ J J J RqRRRS PQ O O O RqRRRS NQ M M M PqPPPQ OQ N N N OqOOOP MQ L L L MqMMMN GQ F F F HqHHHI JQ I I I LqLLLM EQ D D D FqFFFG IQ H H H JqJJJK PQ O O6F O O O DqDDDE NQ M M6F M M M BqBBBC JQ I I0@ I I I >q>>>? HQ G G0@ G G G  J J"%88;PP ^^J// \\*-- rs)NNNN) __name__ __module__ __qualname____doc__r[rr4 LongTensor FloatTensorr __classcell__rrs@rNrQrQsQQ     (15593759 $$E,-$!!12$u/0 $   12 $   $$$$$$$$rsrQc(eZdZdZeZeZdZdZ dZ dS)ConvBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. convbertTct|tjrT|jjd|jj|j |jj dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|jj |jj ddSdS)zInitialize the weightsmeanstdNg?) isinstancerLinearweightr:normal_r<initializer_rangebiaszero_r\rSrefill_)rqmodules rN _init_weightsz%ConvBertPreTrainedModel._init_weightss) fbi ( ( * M  & &CT[5R & S S S{&  &&((((('&  - - * M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *rsN) rrrrr config_classrOload_tf_weightsbase_model_prefixsupports_gradient_checkpointingrrsrNrrsE "L1O"&*#*****rsrcFeZdZdZfdZdejdejfdZxZS)SeparableConv1DzSThis class implements separable convolution, i.e. a depthwise and a pointwise layerc ttj|||||dzd|_tj||dd|_tjtj|d|_ |jj j d|j |jj j d|j dS)Nr#F) kernel_sizegroupspaddingrr)rrrr)rZr[rConv1d depthwise pointwise Parameterr4rmrrr:rr)rqr< input_filtersoutput_filtersrkwargsrrs rNr[zSeparableConv1D.__init__s   # 1$    =.aV[\\\L^Q!?!?@@  "**9Q*RRR "**9Q*RRRRRrs hidden_statesrvcn||}||}||jz }|SN)rrr)rqrxs rNrzSeparableConv1D.forwards4 NN= ) ) NN1   TYrs rrrrr[r4Tensorrrrs@rNrrsi]]SSSSS U\elrsrceZdZfdZdZ d dejdeejdeejdeejd ee d e ejeejff d Z xZ S) ConvBertSelfAttentioncpt|j|jzdkr0t |ds t d|jd|jd|j|jz}|dkr|j|_d|_n||_|j|_|j|_|j|jzdkrt d|j|jzdz|_|j|jz|_ tj |j|j |_ tj |j|j |_ tj |j|j |_t||j|j |j|_tj |j |j|jz|_tj |j|j |_tj|jdgt)|jdz dz dg |_tj|j|_dS) Nrr^zThe hidden size (z6) is not a multiple of the number of attention heads ()rz6hidden_size should be divisible by num_attention_headsr#)rr)rZr[ hidden_sizenum_attention_headsrz ValueError head_ratioconv_kernel_sizeattention_head_size all_head_sizerrquerykeyrMrkey_conv_attn_layerconv_kernel_layerconv_out_layerUnfoldintunfoldrgattention_probs_dropout_probri)rqr<new_num_attention_headsrrs rNr[zConvBertSelfAttention.__init__s&    : :a ? ?PVXhHiHi ?8F$688 4888  #)"<@Q"Q "Q & &$8DO'(D $ $'>D $$/DO & 7   8 8A = =UVV V$*$6$:R$RWX#X !58PPYv143EFF 9V/1CDDYv143EFF #2 F&(:D> //0ABB .)BDUVV 22?CC!M*;b$BWYZ=[\\!M*;CCC,,];;~ BHZ7[\\'11!Q77BBDDNNrRR-- .2+a/A5q9 .  (11!Q77?? D.0E  ~D P u|Xel33 4PPPPPPPPrsrcPeZdZfdZdejdejdejfdZxZS)ConvBertSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNrT) rZr[rrrr!rerfrgrhrirps rNr[zConvBertSelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== rsr input_tensorrvc||}||}|||z}|Srr!rirerqrrs rNrzConvBertSelfOutput.forward@ =11  ]33 }|'CDD rsrrrr[r4rrrrs@rNrrsi>>>>> U\RWR^rsrceZdZfdZdZ d dejdeejdeejdeejd ee d e ejeejff d Z xZ S) ConvBertAttentionctt||_t ||_t |_dSr)rZr[rrqroutputset pruned_headsrps rNr[zConvBertAttention.__init__sI )&11 (00 EErsct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrrr)lenrrqrrrrrrrMrr!runion)rqheadsindexs rN prune_headszConvBertAttention.prune_headss u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::rsNFrrrrrrvc||||||}||d|}|f|ddz}|S)Nrr)rqr) rqrrrrr self_outputsattention_outputrs rNrzConvBertAttention.forwards[yy    !     ;;|A FF#% QRR(88rsr) rrrr[rr4rrrrrrrrs@rNrrs""""" ;;;*7;158<,1 |!!23E-.  ( 5  $D>  u|Xe&788 9rsrcBeZdZfdZdejdejfdZxZS)GroupedLinearLayerct||_||_||_|j|jz|_|j|jz|_tjtj |j|j|j|_ tjtj ||_ dSr) rZr[ input_size output_sizer0 group_in_dim group_out_dimrrr4emptyrr)rqr rr0rrs rNr[zGroupedLinearLayer.__init__s $&$ Ot>!-@l5;t@QSWSe#f#fgg L[!9!9:: rsrrvcvt|d}tj|d|j|jg}|ddd}tj||j}|ddd}tj||d|j g}||j z}|S)Nrr$rr#) listrnr4rr0rr8rrrr)rqrrrs rNrzGroupedLinearLayer.forwards-,,..//2 M-"dot?P)Q R R IIaA   LDK ( ( IIaA   M!j"d.>? @ @  Mrsrrs@rNr r s^;;;;;U\elrsr cBeZdZfdZdejdejfdZxZS)ConvBertIntermediatecrt|jdkr%tj|j|j|_n&t|j|j|j|_t|j trt|j |_ dS|j |_ dS)Nrr rr0)rZr[r0rrrintermediate_sizer!r r hidden_actstrr intermediate_act_fnrps rNr[zConvBertIntermediate.__init__s    ! !6#5v7OPPDJJ+!-6;S`f`qDJ f' - - 9'-f.?'@D $ $ $'-'8D $ $ $rsrrvcZ||}||}|Sr)r!rrqrs rNrzConvBertIntermediate.forwards, =11 00?? rsrrs@rNrrs^ 9 9 9 9 9U\elrsrcPeZdZfdZdejdejdejfdZxZS)ConvBertOutputczt|jdkr%tj|j|j|_n&t|j|j|j|_tj |j|j |_ tj |j |_ dS)NrrrT)rZr[r0rrrrr!r rerfrgrhrirps rNr[zConvBertOutput.__init__s    ! !6#;V=OPPDJJ+!3AS`f`qDJf&8f>STTTz&"<== rsrrrvc||}||}|||z}|Srrrs rNrzConvBertOutput.forwardrrsrrs@rNrrsi > > > > >U\RWR^rsrceZdZfdZ d dejdeejdeejdeejdeejd eed e ejeejffd Z d Z xZ S) ConvBertLayerc~t|j|_d|_t ||_|j|_|j|_|jr-|jst|dt ||_ t||_ t||_ dS)Nrz> should be used as a decoder model if cross attention is added)rZr[chunk_size_feed_forward seq_len_dimr attention is_decoderadd_cross_attention TypeErrorcrossattentionr intermediaterrrps rNr[zConvBertLayer.__init__s '-'E$*622 +#)#=  # <? i4 g g ghhh"3F";";D 088$V,, rsNFrrrrencoder_attention_maskrrvc^|||||}|d}|dd} |jrS|Qt|dstd|d||||||} | d}| | ddz} t |j|j|j|} | f| z} | S)N)rrrr+z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) r'r(rzAttributeErrorr+rfeed_forward_chunkr%r&) rqrrrrr-rself_attention_outputsr rcross_attention_outputs layer_outputs rNrzConvBertLayer.forward%s"&   / "0" "  2!4(, ? <4@4!122 $DdDDD'+&9&9 &%! '' # 7q9  7 ;;G0  #T%A4CSUe   /G+rsc\||}|||}|Sr)r,r)rqr intermediate_outputr3s rNr0z ConvBertLayer.feed_forward_chunkMs2"//0@AA{{#68HII rs)NNNNF) rrrr[r4rrrrrrr0rrs@rNr#r#s - - - - -"7;158<9=,1&&|&!!23&E-. & ( 5 & !) 6 &$D>& u|Xe&788 9&&&&Prsr#ceZdZfdZ ddejdeejdeejdeejd eejd eed eed eed e e e ffdZ xZ S)ConvBertEncoderct|_tjfdt jD|_d|_dS)Nc.g|]}tSr)r#).0_r<s rN z,ConvBertEncoder.__init__..Ws!#c#c#caM&$9$9#c#c#crsF) rZr[r<r ModuleListr1r2layergradient_checkpointingrps `rNr[zConvBertEncoder.__init__Ts`  ]#c#c#c#c5IaCbCb#c#c#cdd &+###rsNFTrrrrr-routput_hidden_states return_dictrvc |rdnd} |rdnd} |r|jjrdnd} t|jD]\} } |r| |fz} ||| nd}|jr(|jr!|| j||||||}n| ||||||}|d}|r$| |dfz} |jjr | |dfz} |r| |fz} |std|| | | fDSt|| | | S)Nrrrr#c3K|]}||V dSrr)r:vs rN z*ConvBertEncoder.forward..s0= ===rs)last_hidden_stater attentionscross_attentions) r<r) enumerater>r?training_gradient_checkpointing_func__call__tupler)rqrrrrr-rr@rAall_hidden_statesall_self_attentionsall_cross_attentionsi layer_modulelayer_head_mask layer_outputss rNrzConvBertEncoder.forwardZs#7@BBD$5?bb4%6d4;;Zdrr`d(44 V VOA|# I$58H$H!.7.CillO* t}  $ A A )!"#)*%!! !- !"#)*% !! *!,M  V&9]1=M; 'tn; d^;  u88 9; ; ; ; ; ; ; ; rsr7cBeZdZfdZdejdejfdZxZS)ConvBertPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSr) rZr[rrrr!rrrr transform_act_fnrerfrps rNr[z(ConvBertPredictionHeadTransform.__init__s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTrsrrvc||}||}||}|Sr)r!rXrers rNrz'ConvBertPredictionHeadTransform.forwards= =11 --m<< }55 rsrrs@rNrVrVscUUUUUU\elrsrVaK This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ConvBertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a8 Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare ConvBERT Model transformer outputting raw hidden-states without any specific head on top.ceZdZfdZdZdZdZee de e e e  ddeejd eejd eejd eejd eejd eejdeedeedeedeee ffdZxZS) ConvBertModelc8t|t||_|j|jkr$t j|j|j|_t||_ ||_ | dSr) rZr[rQrr^rrrembeddings_projectr7encoderr< post_initrps rNr[zConvBertModel.__init__s    ,V44  F$6 6 6&(i0EvGY&Z&ZD #&v..   rsc|jjSrrr`rqs rNget_input_embeddingsz"ConvBertModel.get_input_embeddingss ..rsc||j_dSrra)rqrMs rNset_input_embeddingsz"ConvBertModel.set_input_embeddingss*/'''rsc|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr^r>r'r)rqheads_to_pruner>rs rN _prune_headszConvBertModel._prune_headssU +0022 C CLE5 L u % / ; ;E B B B B C Crsbatch_size, sequence_length checkpoint output_typerNrtrrXrVrrurr@rArvc ||n |jj}||n |jj}| | n |jj} ||t d|+||||} n.||dd} nt d| \} } ||jn|j} |tj | | }|gt|j dr1|j j ddd| f}| | | }|}n!tj| tj| }||| }|||jj}| ||||}t|dr||}||||||| }|S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer$z5You have to specify either input_ids or inputs_embeds)ryrXrx)rtrVrXrur])rrrr@rA)r<rr@use_return_dictr%warn_if_padding_and_no_attention_maskrnryr4onesrzrrXrlrmroget_extended_attention_mask get_head_maskr2r]r^)rqrtrrXrVrrurr@rAr{rr|ryr}r~extended_attention_maskrs rNrzConvBertModel.forwards$2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U!, J%.%:!! @T  !"Z FCCCN  !t(899 [*./*HKZK*X'3J3Q3QR\^h3i3i0!A!&[ SY!Z!Z!Z"&"B"B>S^"_"_&&y$+2OPP l>iv(   4- . . C 33MBBM 2/!5# %  rs) NNNNNNNNN)rrrr[rcrerirCONVBERT_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOCrr4rrrrrrrrs@rNr[r[s     ///000CCC+*+D+K+KLi+j+jkk&6$156:59371559,0/3&*<<E,-<!!23<!!12 < u/0 < E-. <  12<$D><'tn<d^< u88 9<<< lk <<<<<rsr[cFeZdZdZfdZdejdejfdZxZS)ConvBertGeneratorPredictionszAPrediction module for the generator, made up of two dense layers.cttd|_t j|j|j|_t j|j |j|_ dS)NgelurT) rZr[r activationrrer^rfrrr!rps rNr[z%ConvBertGeneratorPredictions.__init__Psa (00f&;AVWWWYv163HII rsgenerator_hidden_statesrvc||}||}||}|Sr)r!r}re)rqr~rs rNrz$ConvBertGeneratorPredictions.forwardWs< #:;;  66 }55 rs) rrrrr[r4rrrrs@rNrzrzMskKKJJJJJu/@UEVrsrzz6ConvBERT Model with a `language modeling` head on top.ceZdZdgZfdZdZdZee de e e e  ddeejd eejd eejd eejd eejd eejdeejdeedeedeedeee ffdZxZS)ConvBertForMaskedLMzgenerator.lm_head.weightc t|t||_t ||_t j|j|j |_ | dSr) rZr[r[rrzgenerator_predictionsrrr^r]generator_lm_headr_rps rNr[zConvBertForMaskedLM.__init__csj    %f-- %A&%I%I"!#6+@&BS!T!T rsc|jSrrrbs rNget_output_embeddingsz)ConvBertForMaskedLM.get_output_embeddingsms %%rsc||_dSrr)rqr`s rNset_output_embeddingsz)ConvBertForMaskedLM.set_output_embeddingsps!0rsrjrkNrtrrXrVrrulabelsrr@rArvc | | n |jj} ||||||||| | } | d} || } || } d}|Pt j}|| d|jj|d}| s| f| ddz}||f|zn|St|| | j | j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrr$rlosslogitsrrG) r<rorrrrr rr]rrrG)rqrtrrXrVrrurrr@rAr~generator_sequence_outputprediction_scoresrloss_fctrs rNrzConvBertForMaskedLM.forwardss-2&1%! 667PQQ 223DEE  *,,H8-222t{7MNNPVP[P[\^P_P_``D F'),CABB,GGF)-)9TGf$$v E$1?.9     rs NNNNNNNNNN)rrr_tied_weights_keysr[rrrrurvrrwrrxrr4rrrrrrrrs@rNrr_s45&&&111+*+D+K+KLi+j+jkk&"$156:59371559-1,0/3&*4 4 E,-4 !!234 !!12 4 u/0 4 E-. 4   124 )*4 $D>4 'tn4 d^4  un$ %4 4 4 lk 4 4 4 4 4 rsrcFeZdZdZfdZdejdejfdZxZS)ConvBertClassificationHeadz-Head for sentence-level classification tasks.cBttj|j|j|_|j|jn|j}tj||_ tj|j|j |_ ||_ dSr) rZr[rrrr!classifier_dropoutrhrgri num_labelsout_projr<rqr<rrrs rNr[z#ConvBertClassificationHead.__init__s Yv163EFF )/)B)NF % %TZTn z"455  &"4f6GHH  rsrrvc |dddddf}||}||}t|jj|}||}||}|S)Nr)rir!r r<rr)rqrrrs rNrz"ConvBertClassificationHead.forwardst !!!Q' " LLOO JJqMM 4;) *1 - - LLOO MM!  rsrrs@rNrrsd77     U\ rsrz ConvBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZeedeee e  dde e j de e jde e j de e j d e e jd e e jd e e j d e ed e ede edeee ffdZxZS)!ConvBertForSequenceClassificationct||j|_||_t ||_t ||_|dSr) rZr[rr<r[rr classifierr_rps rNr[z*ConvBertForSequenceClassification.__init__sb     + %f-- 4V<< rsrjrkNrtrrXrVrrurrr@rArvc | | n |jj} ||||||||| |  } | d} || } d}|Z|jjf|jdkr d|j_nN|jdkr7|jtjks|jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n|| |}n|jjdkrGt}|| d|j| d}n*|jjdkrt}|| |}| s| f| ddz}||f|zn|St|| | j| j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrXrVrrurr@rArr regressionsingle_label_classificationmulti_label_classificationr$r)r<rorr problem_typerrYr4rorr squeezer rrrrrGrqrtrrXrVrrurrr@rArsequence_outputrrrrs rNrz)ConvBertForSequenceClassification.forwards 2&1%D 'tnD d^D  u.. /D D D lk D D D D D rsrz ConvBERT Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. ceZdZfdZeedeee e  dde e j de e jde e j de e j d e e jd e e jd e e j d e ed e ede edeee ffdZxZS)ConvBertForMultipleChoicect|t||_t ||_t j|jd|_ | dS)Nr) rZr[r[rrsequence_summaryrrrrr_rps rNr[z"ConvBertForMultipleChoice.__init__/sh    %f-- / 7 7)F$6:: rsz(batch_size, num_choices, sequence_lengthrkNrtrrXrVrrurrr@rArvc | | n |jj} | |jdn |jd} |)|d|dnd}|)|d|dnd}|)|d|dnd}|)|d|dnd}|=|d|d|dnd}||||||||| |  } | d} || }||}|d| }d}|t}|||}| s|f| ddz}||f|zn|St||| j | j S)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr$rrrr) r<rorCrrnrrrr rrrG)rqrtrrXrVrrurrr@rA num_choicesrr pooled_outputrreshaped_logitsrrrs rNrz!ConvBertForMultipleChoice.forward9s,6&1%G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqM[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  -- ))%'/!5#   "!*--o>> // ++b+66  '))H8OV44D F%''!""+5F)-)9TGf$$v E("!/)     rsr)rrrr[rrurvrrwrrxrr4rrrrrrrrs@rNrr's+*!(()STT &-$156:59371559-1,0/3&*@ @ E,-@ !!23@ !!12 @ u/0 @ E-. @   12@ )*@ $D>@ 'tn@ d^@  u// 0@ @ @ @ @ @ @ @ rsrz ConvBERT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. ceZdZfdZeedeee e  dde e j de e jde e j de e j d e e jd e e jd e e j d e ed e ede edeee ffdZxZS)ConvBertForTokenClassificationcVt||j|_t||_|j|jn|j}tj||_ tj |j |j|_ | dSr)rZr[rr[rrrhrrgrirrrr_rs rNr[z'ConvBertForTokenClassification.__init__s     +%f-- )/)B)NF % %TZTn z"455 )F$68IJJ rsrjrkNrtrrXrVrrurrr@rArvc | | n |jj} ||||||||| |  } | d} || } || } d}|Ft }|| d|j|d}| s| f| ddz}||f|zn|St|| | j | j S)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr$rr) r<rorrirr rrrrrGrs rNrz&ConvBertForTokenClassification.forwards.&1%2 'tn2 d^2  u++ ,2 2 2 lk 2 2 2 2 2 rsrz ConvBERT Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). ceZdZfdZeedeee e  dde e j de e jde e j de e j d e e jd e e jd e e j d e e j d e ede ede edeee ffdZxZS)ConvBertForQuestionAnsweringct||j|_t||_t j|j|j|_| dSr) rZr[rr[rrrr qa_outputsr_rps rNr[z%ConvBertForQuestionAnswering.__init__se     +%f-- )F$68IJJ rsrjrkNrtrrXrVrrustart_positions end_positionsrr@rArvc | | n |jj} |||||||| | |  } | d} || }|dd\}}|d}|d}d}||t|dkr|d}t|dkr|d}|d}| d|}| d|}t|}|||}|||}||zdz }| s||f| ddz}||f|zn|St|||| j | j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrr$r) ignore_indexr#)r start_logits end_logitsrrG)r<rorrsplitrrrrnclampr rrrG)rqrtrrXrVrrurrrr@rArrrrr total_loss ignored_indexr start_lossend_lossrs rNrz$ConvBertForQuestionAnswering.forwards<&1%H 'tnH d^H  u22 3H H H lk H H H H H rsr)Errr)operatorrtypingrrrr4torch.utils.checkpointrtorch.nnrr r activationsr r modeling_outputsrrrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrrconfiguration_convbertr get_loggerrr'rwrxrOModulerQrrrrrr rrr#r7rVCONVBERT_START_DOCSTRINGrur[rzrrrrrrrrsrNrsB )))))))))) AAAAAAAAAA11111111?>>>>>>>lllllllllluuuuuuuuuuuu222222  H % %/"yyyx99999999x*****o***8bi4}}}}}BI}}}@        ***** ***Z,29(RY&:::::BI:::zB B B B B biB B B Jbi" 2jh]]]]]+]] ]@29$RTlmmM M M M M 1M M nmM `0 U U U U U (?U U U p S S S S S 7S S S l G G G G G %<G G G T Y Y Y Y Y #:Y Y Y Y Y rs