g'dZddlZddlZddlmZmZmZddlZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZmZdd lmZmZmZmZmZmZdd lmZmZmZmZdd l m!Z!d d l"m#Z#e!j$e%Z&dZ'dZ(Gdde j)Z*ded9e:Gd:d;e9Z?dS)=zPyTorch MarkupLM model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)add_start_docstrings%add_start_docstrings_to_model_forwardreplace_return_docstrings))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModelapply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)logging)MarkupLMConfigzmicrosoft/markuplm-baserc*eZdZdZfdZddZxZS)XPathEmbeddingszConstruct the embeddings from xpath tags and subscripts. We drop tree-id in this version, as its info can be covered by xpath. ctt|j|_t jj|jzj|_t j j |_ t j |_ t jj|jzdjz|_t jdjzj|_t jfdt#|jD|_t jfdt#|jD|_dS)NcNg|]!}tjjj"S)r Embeddingmax_xpath_tag_unit_embeddingsxpath_unit_hidden_size.0_configs j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/markuplm/modeling_markuplm.py z,XPathEmbeddings.__init__..Ks;    VA6C`aa   cNg|]!}tjjj"Sr!)rr"max_xpath_subs_unit_embeddingsr$r%s r)r*z,XPathEmbeddings.__init__..Rs;    VBFDabb   r+)superr__init__ max_depthrLinearr$ hidden_sizexpath_unitseq2_embeddingsDropouthidden_dropout_probdropoutReLU activationxpath_unitseq2_inner inner2emb ModuleListrangexpath_tag_sub_embeddingsxpath_subs_sub_embeddingsselfr( __class__s `r)r/zXPathEmbeddings.__init__>sE ot$$--///))+63PSWSa3acicu)v)v&z&"<== '))$&If.Kdn.\^_bhbt^t$u$u!1v'9#96;MNN(*     t~..   ) ) %*,    t~..   * * &&&r+Nc g}g}t|jD]n}||j||dddd|f||j||dddd|fot j|d}t j|d}||z}||| | |}|S)Ndim) r<r0appendr=r>torchcatr:r6r8r9)r@xpath_tags_seqxpath_subs_seqxpath_tags_embeddingsxpath_subs_embeddingsixpath_embeddingss r)forwardzXPathEmbeddings.forwardXs' " "t~&& e eA ! ( ()I)Fq)I.YZYZYZ\]\]\]_`Y`Ja)b)b c c c ! ( ()J)G)J>Z[Z[Z[]^]^]^`aZaKb)c)c d d d d % *?R H H H % *?R H H H03HH>>$,,ttG`G`aqGrGr7s7s*t*tuur+)NN)__name__ __module__ __qualname____doc__r/rO __classcell__rAs@r)rr8sV      4        r+rc||}tj|d||z|z}||zS)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rrD)neintrGcumsumtype_aslong) input_ids padding_idxpast_key_values_lengthmaskincremental_indicess r)"create_position_ids_from_input_idsraksg << $ $ ( ( * *D <!444<eZdZdZfdZdZ ddZxZS)MarkupLMEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ctt|||_t j|j|j|j|_ t j|j |j|_ |j |_ t||_t j|j|j|_t j|j|j|_t j|j|_|dt/j|j dd|j|_t j|j |j|j|_ dS)N)r]eps position_ids)rrCF) persistent)r.rcr/r(rr" vocab_sizer2 pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsr0rrNtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsr4r5r6register_bufferrGarangeexpandr]r?s r)r/zMarkupLMEmbeddings.__init__~sF  $''00222 !|F,=v?Q_e_rsss#%<0NPVPb#c#c ) / 7 7%'\&2H&J\%]%]"f&8f>STTTz&"<==   EL)GHHOOPWXXej    ".#%<  *F,>DL\$ $ $    r+c|dd}|d}tj|jdz||jzdztj|j}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrCrdtypedevicer)sizerGrsr]r[rx unsqueezert)r@ inputs_embeds input_shapesequence_lengthrgs r)&create_position_ids_from_inputs_embedsz9MarkupLMEmbeddings.create_position_ids_from_inputs_embedss$((**3B3/ %a.|  q /D4D"Dq"HPUPZcpcw   %%a((// <<*::;;5:V\777N  !![4uzd;''4>*::;;5:V\888N)"66|DD $ : :> J J00PP%(;;>SSVff ^^J// \\*-- r+)NNNNNNr)rPrQrRrSr/r~rOrTrUs@r)rcrc{sxQQ     2===& 22222222r+rccPeZdZfdZdejdejdejfdZxZS)MarkupLMSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNre) r.r/rr1r2denserprqr4r5r6r?s r)r/zMarkupLMSelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== r+ hidden_states input_tensorreturnc||}||}|||z}|SNrr6rpr@rrs r)rOzMarkupLMSelfOutput.forward@ =11  ]33 }|'CDD r+rPrQrRr/rGTensorrOrTrUs@r)rri>>>>> U\RWR^r+rcBeZdZfdZdejdejfdZxZS)MarkupLMIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r.r/rr1r2intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr?s r)r/zMarkupLMIntermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r+rrcZ||}||}|Sr)rrr@rs r)rOzMarkupLMIntermediate.forwards, =11 00?? r+rrUs@r)rrs^99999U\elr+rcPeZdZfdZdejdejdejfdZxZS)MarkupLMOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r.r/rr1rr2rrprqr4r5r6r?s r)r/zMarkupLMOutput.__init__sf Yv79KLL f&8f>STTTz&"<== r+rrrc||}||}|||z}|Srrrs r)rOzMarkupLMOutput.forwardrr+rrUs@r)rrrr+rcBeZdZfdZdejdejfdZxZS)MarkupLMPoolercttj|j|j|_tj|_dSr)r.r/rr1r2rTanhr8r?s r)r/zMarkupLMPooler.__init__ sC Yv163EFF '))r+rrcr|dddf}||}||}|S)Nr)rr8)r@rfirst_token_tensor pooled_outputs r)rOzMarkupLMPooler.forwards@+111a40 #566  66 r+rrUs@r)rr s^$$$$$ U\elr+rcBeZdZfdZdejdejfdZxZS)MarkupLMPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSr) r.r/rr1r2rrrrr transform_act_fnrprqr?s r)r/z(MarkupLMPredictionHeadTransform.__init__s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTr+rrc||}||}||}|Sr)rrrprs r)rOz'MarkupLMPredictionHeadTransform.forward&s= =11 --m<< }55 r+rrUs@r)rrscUUUUUU\elr+rc*eZdZfdZdZdZxZS)MarkupLMLMPredictionHeadc>tt||_t j|j|jd|_t j tj |j|_ |j |j_ dS)NF)bias) r.r/r transformrr1r2ridecoder ParameterrGrrr?s r)r/z!MarkupLMLMPredictionHead.__init__/sz 8@@y!3V5FUSSS LV->!?!?@@ !I r+c(|j|j_dSr)rrr@s r) _tie_weightsz%MarkupLMLMPredictionHead._tie_weights<s I r+cZ||}||}|Sr)rrrs r)rOz MarkupLMLMPredictionHead.forward?s*}55  ]33 r+)rPrQrRr/rrOrTrUs@r)rr.sV & & & & &&&&r+rcBeZdZfdZdejdejfdZxZS)MarkupLMOnlyMLMHeadcptt||_dSr)r.r/r predictionsr?s r)r/zMarkupLMOnlyMLMHead.__init__Gs/ 3F;;r+sequence_outputrc0||}|Sr)r)r@rprediction_scoress r)rOzMarkupLMOnlyMLMHead.forwardKs ,,_==  r+rrUs@r)rrFs^<<<<<!u|! !!!!!!!!r+rc,eZdZdfd ZdejdejfdZ ddejdeejd eejd eejd eejd ee e ejd ee de ejfdZ xZ S)MarkupLMSelfAttentionNcDt|j|jzdkr0t |ds t d|jd|jd|j|_t |j|jz |_|j|jz|_tj |j|j|_ tj |j|j|_ tj |j|j|_ tj|j|_|pt#|dd|_|jdks |jd kr6|j|_tjd |jzd z |j|_|j|_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()position_embedding_typeabsolute relative_keyrelative_key_queryr)r.r/r2num_attention_headshasattr ValueErrorrXattention_head_size all_head_sizerr1querykeyvaluer4attention_probs_dropout_probr6getattrrrlr"distance_embedding is_decoderr@r(rrAs r)r/zMarkupLMSelfAttention.__init__Rs    : :a ? ?PVXhHiHi ?8F$688 4888  $*#= #&v'9F( ' -zC C $  '> 9 9T=Y]q=q=q+1+ID (&(l1v7U3UXY3Y[_[s&t&tD # +r+xrc|dd|j|jfz}||}|ddddS)NrCrrrr )ryrrviewpermute)r@r new_x_shapes r)transpose_for_scoresz*MarkupLMSelfAttention.transpose_for_scoreslsPffhhssmt'?AY&ZZ FF;  yyAq!$$$r+Frattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsc^||}|du} | r||d} |d} |}n4| rS|||} |||} |}n||||} |||} t j|d| gd} t j|d| gd} nP|||} |||} ||} |du} |jr| | f}t j| | dd}|j dks |j dkrt| j d| j d}}| r>t j |dz tj |j dd}n:t j|tj |j dd}t j|tj |j dd}||z }|||jzdz }|| j }|j dkrt jd | |}||z}n?|j dkr4t jd | |}t jd | |}||z|z}|t+j|jz }|||z}t0j|d}||}|||z}t j|| }|dddd }|dd|jfz}||}|r||fn|f}|jr||fz}|S)NrrrrDrCrrrvrwzbhld,lrd->bhlrzbhrd,lrd->bhlrr ) rrrrrGrHrmatmul transposershapetensorr[rxrrsrrltorweinsummathsqrtrr functionalsoftmaxr6r contiguousryr)r@rrrrrrrmixed_query_layeris_cross_attention key_layer value_layer query_layer use_cacheattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss r)rOzMarkupLMSelfAttention.forwardqsZ!JJ}55 3$>  O."<&q)I(+K3NN  O11$((;P2Q2QRRI33DJJ?T4U4UVVK3NN  '11$((=2I2IJJI33DJJ}4M4MNNK >!#4i"@aHHHI)^A%6 $D!LLLKK11$((=2I2IJJI33DJJ}4M4MNNK//0ABB "$. ? 6(5N!< Y5H5HR5P5PQQ  '> 9 9T=Y]q=q=q'2'8';Y_Q=O*L w!&j1nEJWdWk!l!l!l!q!q"""'l%*UbUi!j!j!j!o!oprtu!v!v"\*EJ}OcdddiijkmoppN%6H#'#:#:8dFb;bef;f#g#g #7#:#:AR#:#S#S +~==+0<8H+Wk+l+l(#36N#N  -1EEE16>NP[]q1r1r./4|S"S%**+BCC 6G]=/22mM] ? 2 11Gr+rNNNNNF) rPrQrRr/rGrrr FloatTensorrboolrOrTrUs@r)rrQs.,,,,,,4%el%u|%%%%7;15=A>BDH,1cc|c!!23cE-. c ((9: c !)): ; c!uU->'?!@Ac$D>c u| ccccccccr+reagerceZdZdfd ZdZ ddejdeejdeejdeejd eejd ee e ejd ee d e ejfd Z xZ S)MarkupLMAttentionNctt|j|||_t ||_t|_dS)Nr) r.r/MARKUPLM_SELF_ATTENTION_CLASSES_attn_implementationr@routputset pruned_headsrs r)r/zMarkupLMAttention.__init__s` 3F4OP ,C    )00 EEr+ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)NrrrD)lenrr@rrrrrrrrrrunion)r@headsindexs r) prune_headszMarkupLMAttention.prune_headss u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::r+Frrrrrrrrc ||||||||}||d|} | f|ddz} | S)Nrr)r@r) r@rrrrrrr self_outputsattention_outputr s r)rOzMarkupLMAttention.forwardsayy    ! "     ;;|A FF#% QRR(88r+rr ) rPrQrRr/rrGrrr rr rOrTrUs@r)rrs"""""";;;*7;15=A>BDH,1|!!23E-.  ((9:  !)): ; !uU->'?!@A$D> u| r+rc eZdZfdZ ddejdeejdeejdeejdeejd eeeejd ee d eejfd Z d Z xZ S) MarkupLMLayerct|j|_d|_t ||_|j|_|j|_|jr/|jst|dt |d|_ t||_ t||_ dS)Nrz> should be used as a decoder model if cross attention is addedrr)r.r/chunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionrcrossattentionr intermediaterrr?s r)r/zMarkupLMLayer.__init__s '-'E$*622 +#)#=  # `? j D!h!h!hiii"3FT^"_"_"_D 088$V,, r+NFrrrrrrrrc | |ddnd}||||||} | d} |jr| dd} | d} n | dd} d} |jrp|nt|dstd|d| |d dnd}|| ||||||}|d} | |ddz} |d} | | z} t |j|j|j| }|f| z} |jr| | fz} | S) Nr)rrrrrCr'z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) r%rrrr'rfeed_forward_chunkr#r$)r@rrrrrrrself_attn_past_key_valueself_attention_outputsrr present_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputs layer_outputs r)rOzMarkupLMLayer.forward s:H9S>"1"#5#5Y] !%   /3 "0" " 2!4 ? 1,QrT2G 6r :  ,QRR0G'+$ ? Q4@4!122  DdDDD @N?Yrss(;(;_c %&*&9&9 %&)!'' # 7q9  7" ==G,C2+F ( 14P P 0  #T%A4CSUe   /G+ ? 5!2 44Gr+c\||}|||}|Sr)r(r)r@rintermediate_outputr1s r)r*z MarkupLMLayer.feed_forward_chunkas2"//0@AA{{#68HII r+r ) rPrQrRr/rGrrr rr rOr*rTrUs@r)r!r!s - - - - -"7;15=A>BDH,1??|?!!23?E-. ? ((9: ? !)): ; ?!uU->'?!@A?$D>? u| ????Br+r!cLeZdZfdZ ddejdeejdeejdeejd eejd eeeejd ee d ee d ee dee de eeje ffdZ xZ S)MarkupLMEncoderct|_tjfdt jD|_d|_dS)Nc.g|]}tSr!)r!r%s r)r*z,MarkupLMEncoder.__init__..ls!#c#c#caM&$9$9#c#c#cr+F) r.r/r(rr;r<num_hidden_layerslayergradient_checkpointingr?s `r)r/zMarkupLMEncoder.__init__is`  ]#c#c#c#c5IaCbCb#c#c#cdd &+###r+NFTrrrrrpast_key_valuesrroutput_hidden_states return_dictrc | rdnd} |rdnd} |r|jjrdnd} |jr%|jr|rtdd}|rdnd}t |jD]\}}| r| |fz} |||nd}|||nd}|jr)|jr"||j |||||||}n||||||||}|d}|r ||dfz }|r$| |dfz} |jjr | |dfz} | r| |fz} | std||| | | fDSt||| | | S) Nr!zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FrrCrrc3K|]}||V dSrr!)r&vs r) z*MarkupLMEncoder.forward..s4  =!===  r+)last_hidden_stater;r attentionscross_attentions) r(r&r:traininglogger warning_once enumerater9_gradient_checkpointing_func__call__rr)r@rrrrrr;rrr<r=all_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cacherM layer_modulelayer_head_maskr layer_outputss r)rOzMarkupLMEncoder.forwardosB#7@BBD$5?bb4%6d4;;Zdrr`d  & "4= " "##p" #,6RR$(44# V# VOA|# I$58H$H!.7.CillO3B3N_Q//TXN* t}  $ A A )!"#)*"% ! ! !- !"#)*"%!! *!,M ;"}R'8&::"  V&9]1=MBEI$(,1/4&*S S |S !!23S E-. S ((9: S !)): ; S "%e.?(@"ABS D>S $D>S 'tnS d^S  uU\"$MM NS S S S S S S S r+r5cleZdZdZeZdZdZede e e e j fffd ZxZS)MarkupLMPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. markuplmct|tjrT|jjd|jj|j |jj dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|jj |jj ddSdS)zInitialize the weightsg)meanstdN?)rrr1weightdatanormal_r(initializer_rangerzero_r"r]rpfill_)r@modules r) _init_weightsz%MarkupLMPreTrainedModel._init_weightss) fbi ( ( * M  & &CT[5R & S S S{&  &&((((('&  - - * M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r+pretrained_model_name_or_pathcLtt|j|g|Ri|Sr)r.rSfrom_pretrained)clsra model_argskwargsrAs r)rcz'MarkupLMPreTrainedModel.from_pretrainedsBBu,c22B ) ,6   :@   r+)rPrQrRrSr config_classbase_model_prefixr` classmethodrrrosPathLikercrTrUs@r)rSrSs "L"***  HU3PRP[K[E\<]     [     r+rSaK This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MarkupLMConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) xpath_tags_seq (`torch.LongTensor` of shape `({0}, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `({0}, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: `1` for tokens that are NOT MASKED, `0` for MASKED tokens. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: `0` corresponds to a *sentence A* token, `1` corresponds to a *sentence B* token [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: `1` indicates the head is **not masked**, `0` indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): If set to `True`, the attentions tensors of all attention layers are returned. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): If set to `True`, the hidden states of all layers are returned. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): If set to `True`, the model will return a [`~file_utils.ModelOutput`] instead of a plain tuple. zbThe bare MarkupLM Model transformer outputting raw hidden-states without any specific head on top.ceZdZdfd ZdZdZdZee de e e  dd e ejd e ejd e ejd e ejd e ejde ejde ejde ejde ede ede edeee ffdZdZxZS) MarkupLMModelTct|||_t||_t ||_|rt|nd|_| dSr) r.r/r(rcrr5encoderrpooler post_init)r@r(add_pooling_layerrAs r)r/zMarkupLMModel.__init__(sq     ,V44&v.. 0AKnV,,,t  r+c|jjSrrrkrs r)get_input_embeddingsz"MarkupLMModel.get_input_embeddings4s ..r+c||j_dSrrt)r@rs r)set_input_embeddingsz"MarkupLMModel.set_input_embeddings7s*/'''r+c|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsror9r%r)r@heads_to_pruner9rs r) _prune_headszMarkupLMModel._prune_heads:sU +0022 C CLE5 L u % / ; ;E B B B B C Cr+batch_size, sequence_length output_typergNr\rIrJrrrgrr{rr<r=rc | | n |jj} | | n |jj} | | n |jj} ||t d|+||||} n.||dd} nt d||jn|j} |tj | | }|!tj | tj | }| d d}| |j }d |z d z}||dkrr| d  d  d d}||jjdddd}nS|dkr;| d d d}| t%|j }ndg|jjz}||||||| }||||| | | }|d }|j||nd}| s||f|ddzSt/|||j|j|jS)a` Returns: Examples: ```python >>> from transformers import AutoProcessor, MarkupLMModel >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = MarkupLMModel.from_pretrained("microsoft/markuplm-base") >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 4, 768] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerCz5You have to specify either input_ids or inputs_embeds)rxrvrrrrXgr)r\rIrJrgrr{)rrr<r=)rB pooler_outputrrCrD)r(rr<use_return_dictr%warn_if_padding_and_no_attention_maskryrxrGrrr[rzrrwrErtr8next parametersrrorprrrCrD)r@r\rIrJrrrgrr{rr<r=r|rxextended_attention_maskembedding_outputencoder_outputsrrs r)rOzMarkupLMModel.forwardBs'H2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U%.%:!! @T  !"Z FCCCN  !"[EJvVVVN"0":":1"="="G"G"J"J"9"<"<4:"<"N"N#&)@#@H"L  }}!##%//22<>I??))%)' +  ,,  #/!5# '  *!,8< 8O O444UY  J#]3oabb6II I;-')7&1,=     r+cTd}|D]!}|tfd|Dfz }"|S)Nr!c3tK|]2}|d|jV3dS)rN) index_selectrrx)r& past_statebeam_idxs r)rAz/MarkupLMModel._reorder_cache..sCnnU_j--aZ=N1O1OPPnnnnnnr+)r)r@r;rreordered_past layer_pasts ` r)_reorder_cachezMarkupLMModel._reorder_cachesQ)  J nnnncmnnnnn NNr+)T) NNNNNNNNNNN)rPrQrRr/rurwr{r MARKUPLM_INPUTS_DOCSTRINGformatrr_CONFIG_FOR_DOCrrG LongTensorr r rrrOrrTrUs@r)rmrm"s      ///000CCC+*+D+K+KLi+j+jkk+Wfuvvv1559596:59371559,0/3&*e e E,-e !!12e !!12 e !!23 e !!12 e u/0e E-.e   12e $D>e 'tne d^e  uBB Ce e e wvlke Pr+rmz MarkupLM Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). c!eZdZfdZeedeee  dde e j de e j de e j de e j d e e j d e e j d e e j d e e j d e e j de e j de e de e de e deee j effdZxZS)MarkupLMForQuestionAnsweringct||j|_t|d|_t j|j|j|_| dSNF)rr) r.r/ num_labelsrmrTrr1r2 qa_outputsrqr?s r)r/z%MarkupLMForQuestionAnswering.__init__sj     +%fFFF )F$68IJJ r+r|r}Nr\rIrJrrrgrr{start_positions end_positionsrr<r=rc | | n |jj} |||||||||| | |  }|d}||}|dd\}}|d}|d}d}| | t| dkr| d} t| dkr| d} |d}| d|| d|t|}||| }||| }||zdz }| s||f|ddz}||f|zn|St||||j |j S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Returns: Examples: ```python >>> from transformers import AutoProcessor, MarkupLMForQuestionAnswering >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> model = MarkupLMForQuestionAnswering.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> html_string = " My name is Niels " >>> question = "What's his name?" >>> encoding = processor(html_string, questions=question, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> processor.decode(predict_answer_tokens).strip() 'Niels' ```N rIrJrrrgrr{rr<r=rrrCrD) ignore_indexr)loss start_logits end_logitsrrC)r(rrTrsplitsqueezerrryclamp_rrrrC)r@r\rIrJrrrgrr{rrrr<r=r rlogitsrr total_loss ignored_indexloss_fct start_lossend_lossrs r)rOz$MarkupLMForQuestionAnswering.forwardsl&1%f 'tnf d^f  uU\"$@@ Af f f gflkf f f f f r+rz9MarkupLM Model with a `token_classification` head on top.ceZdZfdZeedeee  dde e j de e j de e j de e j d e e j d e e j d e e j d e e j d e e j de e de e de e deee j effdZxZS)MarkupLMForTokenClassificationcZt||j|_t|d|_|j|jn|j}tj||_ tj |j |j|_ | dSr)r.r/rrmrTclassifier_dropoutr5rr4r6r1r2 classifierrqr@r(rrAs r)r/z'MarkupLMForTokenClassification.__init__6s     +%fFFF )/)B)NF % %TZTn z"455 )F$68IJJ r+r|r}Nr\rIrJrrrgrr{labelsrr<r=rc | | n |jj} |||||||||| | |  } | d}||}d}| Kt }||d|jj| d}| s|f| ddz}||f|zn|St||| j| j S)a  labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Returns: Examples: ```python >>> from transformers import AutoProcessor, AutoModelForTokenClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> processor.parse_html = False >>> model = AutoModelForTokenClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> nodes = ["hello", "world"] >>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span"] >>> node_labels = [1, 2] >>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```NrrrCrrrrrC) r(rrTrrrrrrrC)r@r\rIrJrrrgrr{rrr<r=r rrrrrs r)rOz&MarkupLMForTokenClassification.forwardDsX&1%XXX-115151515/3,004)-,0/3&*N N EL)N !.N !. N !. N !. N u|,N EL)N   -N &N $D>N 'tnN d^N  uU\"N2 3N N N YXlkN N N N N r+rz MarkupLM Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZeedeee  dde e j de e j de e j de e j d e e j d e e j d e e j d e e j d e e j de e de e de e deee j effdZxZS)!MarkupLMForSequenceClassificationcdt||j|_||_t ||_|j|jn|j}tj ||_ tj |j |j|_ |dSr)r.r/rr(rmrTrr5rr4r6r1r2rrqrs r)r/z*MarkupLMForSequenceClassification.__init__s     + %f-- )/)B)NF % %TZTn z"455 )F$68IJJ r+r|r}Nr\rIrJrrrgrr{rrr<r=rc | | n |jj} |||||||||| | |  } | d}||}||}d}| Z|jjf|jdkr d|j_nN|jdkr7| jtj ks| jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n||| }n|jjdkrGt}||d|j| d}n*|jjdkrt}||| }| s|f| ddz}||f|zn|St!||| j| j S) a& labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoProcessor, AutoModelForSequenceClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = AutoModelForSequenceClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```Nrr regressionsingle_label_classificationmulti_label_classificationrCrr)r(rrTr6r problem_typerrwrGr[rXr rrrrrrrC)r@r\rIrJrrrgrr{rrr<r=r rrrrrs r)rOz)MarkupLMForSequenceClassification.forwards#V&1%] 'tn] d^]  uU\"$<< =] ] ] cblk] ] ] ] ] r+r)r)@rSrrjtypingrrrrGtorch.utils.checkpointrtorch.nnrrr activationsr file_utilsr r rmodeling_outputsrrrrrrmodeling_utilsrrrrutilsrconfiguration_markuplmr get_loggerrPrF_CHECKPOINT_FOR_DOCrModulerrarcrrrrrrrrrrr!r5rSMARKUPLM_START_DOCSTRINGrrmrrrr!r+r)rs )))))))))) AAAAAAAAAA!!!!!!  222222  H % %/"/ / / / / bi/ / / f 4 4 4 4 ________F             29        RY        RY    bi$ry0!!!!!")!!!CCCCCBICCCN "# 00000 000hSSSSSBISSSnZ Z Z Z Z biZ Z Z z     o   B .bhLLLLL+LL L^ t t t t t #:t t t nUWopp` ` ` ` ` %<` ` qp` F p p p p p (?p p p p p r+