gDx
d Z ddlZddlZddlZddlmZmZmZmZ ddl m
Z
ddl m Z m
Z
ddlmZmZmZmZmZ ddlmZ dd lmZmZ eeeeef eeeeeef eeeeef eeeeeef f Z G d
d ed
Z G d ded
Z G d ded
Z G d de Z deeeeef dedeeef fdZ dededefdZ!d Z"d Z#d Z$d d Z%dS )!z Processor class for KOSMOS-2. N)ListOptionalTupleUnion ) BatchFeature)
ImageInput
is_batched) ImagesKwargsProcessingKwargsProcessorMixin
TextKwargsUnpack)
AddedToken)
BatchEncoding TextInputc ^ e Zd ZU eee ed< ee ed< ee ed< dS )Kosmos2ImagesKwargsbboxesnum_image_tokensfirst_image_token_idN)__name__
__module__ __qualname__r r float__annotations__int j/var/www/html/ai-engine/env/lib/python3.11/site-packages/transformers/models/kosmos2/processing_kosmos2.pyr r % sG T%[ !!!! sm###"3-'''''r r F)totalc & e Zd ZU ee ed< dS )Kosmos2TextKwargs
add_eos_tokenN)r r r r boolr r r r r# r# + s" D>!!!!!r r# c
F e Zd ZU eed< eed< dddddddddd ddid Zd
S ) Kosmos2ProcessorKwargs text_kwargs
images_kwargsTFr ) add_special_tokenspaddingstridereturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mappingreturn_token_type_idsverboser$ r @ )r( r) N)r r r r# r r _defaultsr r r r' r' / sg """"&&&& #' ).*/&+%* "
IIIr r' c e Zd ZdZddgZdgZdZdZd fd Z d d
e d e
e e e f d e
e d
efdZd Zd Z d de
e e e f d
e dedee d
e
ee e f f
dZd Zd Zd!dZed Z d ede
e e e e e e f d
efdZ d e
e eef e e e e e f f d
e eef fd Z xZ!S )"Kosmos2Processora,
Constructs an KOSMOS-2 processor which wraps a KOSMOS-2 image processor and a KOSMOS-2 tokenizer into a single
processor.
[`Kosmos2Processor`] offers all the functionalities of [`CLIPImageProcessor`] and some functionalities of
[`XLMRobertaTokenizerFast`]. See the docstring of [`~Kosmos2Processor.__call__`] and [`~Kosmos2Processor.decode`]
for more information.
Args:
image_processor (`CLIPImageProcessor`):
An instance of [`CLIPImageProcessor`]. The image processor is a required input.
tokenizer (`XLMRobertaTokenizerFast`):
An instance of ['XLMRobertaTokenizerFast`]. The tokenizer is a required input.
num_patch_index_tokens (`int`, *optional*, defaults to 1024):
The number of tokens that represent patch indices.
image_processor tokenizernum_patch_index_tokensCLIPImageProcessor
AutoTokenizer c d d|_ d| _ d| _ d| _ d| _ d| _ d| _ d| _ d | _ d
| _ d | _
d | _ | j | j | j | j | j | j | j | j | j | j
| j g | _ || _
d
t | j
D }g }| j |z D ](}| t! |ddd )| | t% || d S )NFzzzzzzz z z c Z g | ](}d t | d d)S )
)strzfill.0xs r
z-Kosmos2Processor.__init__.. s4 m m mQ @c!ffll1oo @ @ @ m m mr T)lstriprstrip
normalized)r0 eod_token boi_token eoi_token eoc_token eol_token bop_token eop_token boo_token eoo_token dom_token grd_token
tag_tokensr8 rangeappendr
add_tokenssuper__init__) selfr6 r7 r8 kwargspatch_index_tokens
tokens_to_addtoken __class__s r r[ zKosmos2Processor.__init__[ s? */ '!"##"#$#$9&
N N N N N N N N N N N
'=# m m%PTPkJlJl m m m
_'99 a aE E$uY^!_!_!_ ` ` ` ` ]+++
)44444r Nimagestextr] returnc ||t d j t fd j j i|}|d dd }|d dd }|d dd } |d d
d }
|d d } |d d
} |d dd }
t }|) j |fi |d }| | |lj
|||| }| rK|
sIt |t r j j
| }n#t |t r fd|D }|d d o|
|d d < || nd |d d
< ||
nd|d d< j d$d|i|d }| | | |d d < | |d d
< |
|d d< ||| j j dz } | }t! | dz }t t# | | |z }dgdg|z z dgz }g }g }|d }t |t r|g}|d g|d< |D ]}|d| |z |||z d z }| | t' j | }|rdg|z }|dgt) | t) | z
z z
}| | t |t r-t+ d t- |j D d }|d \ }} |d \ } }|d d o|
|d d < d|d d< j d$d|| gi|d }t) |j d | k r j j dk r5 fd|D } fd |D } fd |d D |d< nD j j d k r4 fd |D } fd |D } fd!|d D |d< t |t r#|
!|d }|d d |d< |d }| t3 ||d |d"|
# |S )%a
This method uses [`CLIPImageProcessor.__call__`] method to prepare image(s) for the model, and
[`XLMRobertaTokenizerFast.__call__`] to prepare text for the model.
Please refer to the docstring of the above two methods for more information.
The rest of this documentation shows the arguments specific to `Kosmos2Processor`.
Args:
bboxes (`Union[List[Tuple[int]], List[Tuple[float]], List[List[Tuple[int]]], List[List[Tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional* defaults to 64):
The number of (consecutive) places that are used to mark the placeholders to store image information.
This should be the same as `latent_query_num` in the instance of `Kosmos2Config` you are using.
first_image_token_id (`int`, *optional*):
The token id that will be used for the first place of the subsequence that is reserved to store image
information. If unset, will default to `self.tokenizer.unk_token_id + 1`.
add_eos_token (`bool`, defaults to `False`):
Whether or not to include `EOS` token id in the encoding when `add_special_tokens=True`.
Nz*You have to specify either images or text.tokenizer_init_kwargsr) r r r2 r r( r$ Fr* r+ return_tensors)r c 0 g | ]}j j | S r )r7 bos_token)rE sr\ s r rG z-Kosmos2Processor.__call__.. s) KKKt~ 7 < < <KKKr rc r input_idsattention_maskc 6 g | ]\ }}|t | fS r len)rE idxrF s r rG z-Kosmos2Processor.__call__.. s& TTTvsAc3q66]TTTr c | d S Nr )rF s r z+Kosmos2Processor.__call__.. s
defhdi r )keyrt rightc V g | ]%}|j j gt | z
z z &S r r7 pad_token_idrp rE rF max_len_paddedr\ s r rG z-Kosmos2Processor.__call__.. s; $v$v$vghQ$.*E)F.[^_`[a[aJa)b%b$v$v$vr c B g | ]}|d gt | z
z z S r ro rE rF r| s r rG z-Kosmos2Processor.__call__.. = 6 6 6DEA~A'> ? ?6 6 6r c B g | ]}|d gt | z
z z S r~ ro r s r rG z-Kosmos2Processor.__call__..
r r leftc V g | ]%}j j gt | z
z |z &S r ry r{ s r rG z-Kosmos2Processor.__call__.. s< $v$v$vghdn&A%BnWZ[\W]W]F]%^ab%b$v$v$vr c B g | ]}d gt | z
z |z S r~ ro r s r rG z-Kosmos2Processor.__call__.. = 6 6 6DEQC>CFF#: ;a ?6 6 6r c B g | ]}d gt | z
z |z S r~ ro r s r rG z-Kosmos2Processor.__call__.. r r )rl rm image_embeds_position_mask)data tensor_typer )
ValueError
_merge_kwargsr' r7 init_kwargspop
setdefaultr r6 updatepreprocess_examples
isinstancerB ri list unk_token_idr rW rX copyrp sorted enumeraterl padding_sider ) r\ rb rc audiovideosr]
output_kwargsr r r r$ r* r+ rg encodingimage_encoding
text_encodingwith_bos start_indeximage_token_ids base_image_embeds_position_maskrl r
all_input_idstext_idsmask
sorted_length_min_len_not_paddedrq r| s ` @r __call__zKosmos2Processor.__call__ sx 8 >dl IJJ J** "
"&."<
/33HdCC( 9==>PRTUU ,_ = A ABXZ^ _ _ %m488 %PP
*= 9:N O
.y9&}5@@AQSWXX >> 1T 1& [ [M/>>DD d++ LKKKKdKKKD m,-ABT}
- ()= > BH ggUZM- ( 3OU~^^cgM- ()9 : *DN U U U
m8T U UM OOM * * *=O
m$%9:29
m$Y/9G
m$%56 2#+'+~'BQ'F$ *H h--!+K #5)=?SVf?f#g#g h hO/0cQC:J4J.JaS.P + I)+ & $[ 1M$$$
J!.
.67G.H-I)* )
8
8#L[L1OCh{]mOmOoOoFpp ***y!@AA & 3:Ds8}}s4yy 899*11$7777$%%
&TT=;R1S1STTTZiZi! ! !
)6a(8%%&r*Q!-01EFX= m,-AB BF
m,-=> . ` `T#YK `=Q^C_ ` `
!$]%>$v$v$v$v$vlu$v$v$v 6 6 6 6Ic6 6 626 6 6 6IQRbIc6 6 6!12
$$$
K )? %aL -56F-G-J)*-G-J*
OO %.*23C*D6P
!/
r c |dS t |t st d |D ]}|t |t s|g}|D ]~}t |t rXt | dk rt d |D s;t | dk rt d |D st d dS )a
Check `bboxes` for a single text example. It could be
- `None`: no bounding box associated to a text.
- A list with each element being the bounding boxes associated to one ` ... ` pair found
in a text. This could be:
- `None`: no bounding box associated to a ` ... ` pair.
- A tuple of 2 integers: A single bounding box specified by patch indices.
- A tuple of 4 float point number: A single bounding box specified by (normalized) coordinates.
- A list containing the above 2 tuple types: Multiple bounding boxes for a
` ... ` pair.
Nz@`bboxes` (for a single text example) should be `None` or a list. c 3 @ K | ]}t |t V d S N)r r rD s r zAKosmos2Processor._check_bboxes_for_single_text..C s, .S.Saz!S/A/A.S.S.S.S.S.Sr r@ c 3 @ K | ]}t |t V d S r )r r rD s r r zAKosmos2Processor._check_bboxes_for_single_text..D s, 1X1X1*Q2F2F1X1X1X1X1X1Xr a' Each element in `bboxes` (for a single text example) should be either `None`, a tuple containing 2 integers or 4 float point numbers, or a list containing such tuples. Also make sure the arguments `texts` and `bboxes` passed to `preprocess_text` are both in batches or both for a single example.)r r r tuplerp all)r\ r bboxelements r _check_bboxes_for_single_textz.Kosmos2Processor._check_bboxes_for_single_text* s > FFD)) a _`` ` D| d++
v
!'511 \\Q&&3.S.S7.S.S.S+S+S&G ))c1X1XPW1X1X1X.X.X)$@
r c l | }|| d| }| || }|S )N )strip_insert_patch_index_tokens)r\ rc imager img_info_tokenss r _preprocess_single_examplez+Kosmos2Processor._preprocess_single_exampleM sD zz|| %....D ..tV<< r r2 textsr r c d j g|z }d j g|z j gz d}t |t rd}|g}|dgt | z }nt
| s|g}t | t | k r0t dt | dt | d |s | |g}nT|?t |t st d |D ]} | ndgt | z }t | t | k r0t d t | dt | d fd
t ||| D }|s|d }|S ) a- Add image and bounding box information to `texts` as image and patch index tokens.
Args:
texts (`Union[TextInput, List[TextInput]]`): The texts to be processed.
images (`ImageInput`, *optional*): The images associated to `texts`.
bboxes (`Union[List[Tuple[int]], List[Tuple[float]], List[List[Tuple[int]]], List[List[Tuple[float]]]]`, *optional*):
The bounding bboxes associated to `texts`.
num_image_tokens (`int`, *optional*, defaults to 64):
The number of image tokens (used as latent queries). This should corresponds to the `latent_query_num`
attribute in `Kosmos2Config`.
Returns:
`Union[TextInput, List[TextInput]]`: The processed texts with image and patch index tokens.
r TFNzGThe number of examples in `texts` and `images` should be the same. Got v.s. instead.zS`bboxes` should be `None` or a list (as a batch) when `texts` is passed as a batch.zGThe number of examples in `texts` and `bboxes` should be the same. Got c H g | ] \ }}} ||| S r )r )rE rc r r r r\ s r rG z8Kosmos2Processor.preprocess_examples.. sC
!eT
+ +D% O O
r r ) rL joinrM r rB rp r
r r r zip)
r\ r rb r r
img_tokensbatchedrF resultr s
` @r r z$Kosmos2Processor.preprocess_examplesW s9 , n%(88
((DN#3j#@DNCS#STT eS ! ! GGE >Vc%jj(FFF## XF u::V $ $ CZ]^cZdZd C Clopvlwlw C C C
) . .v 6 6 6 XFF
fd++
x !vwww
6
62215555
6 Vc%jj(F v;;#e** $ $ CZ]^cZdZd C Clopvlwlw C C C
%(%?%?
AYF
r c & | j j |i |S )z
This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please
refer to the docstring of this method for more information.
)r7 batch_decoder\ argsr] s r r z Kosmos2Processor.batch_decode s
+t~*D;F;;;r c & | j j |i |S )z
This method forwards all its arguments to PreTrainedTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to
the docstring of this method for more information.
)r7 decoder s r r zKosmos2Processor.decode s
%t~$d5f555r Tc h | | j d }|rt | S |S rs )splitrM +clean_text_and_extract_entities_with_bboxes)r\ rc cleanup_and_extractcaptions r post_process_generationz(Kosmos2Processor.post_process_generation s6 **T^,,R0 H>wGG Gr c | j j }| j j }t t ||z S r )r7 model_input_namesr6 r dictfromkeys)r\ tokenizer_input_namesimage_processor_input_namess r r z"Kosmos2Processor.model_input_names s< !% @ &*&:&L#DMM"7:U"UVVWWWr c |t | dk r|S t t j d| }t | t | k r0t dt | dt | d d}g }t || D ]\ }}| \ }} | ||| | }|>t |t r|g}g }
t d |D st d |D ]4} | | \ } }
|
| d |
5t |
dk rd
|
}| d | d |t | k r | ||d d
| }|S )Nr z.+?)stringzuThe number of elements in `bboxes` should be the same as the number of ` ... ` pairs in `text`. Got r r c 3 K | ]}|d uV d S r r )rE boxs r r z>Kosmos2Processor._insert_patch_index_tokens.. s& 773s$777777r zTThe multiple bounding boxes for a single phrase should not contain any `None` value.r z z )
rp r refinditerr r spanrX r r r #_convert_bbox_to_patch_index_tokensr )r\ rc r matched_phrasescurr_posbuffermatchedr r endpatch_index_stringsr
patch_index_1
patch_index_2 position_strs r r z+Kosmos2Processor._insert_patch_index_tokens sS >S[[A - -K r{+B4 P P PQQ 3v;; . . { IL M\ I] I] { { eh io ep ep { { {
& 9 9 @ @MGT \\^^FAs MM$x|, - - -H|$&&
v"$ 77$77777
j
O
O/3/W/WX[/\/\,
}#**m+M+Mm+M+MNNNN&''1,,=BBCVWWL MM>l>>> ? ? ? ? c$ii MM$xyy/ * * *wwv r r c F t | dk r|\ }}n9t t j | j }t || \ }}dt
| d d}dt
| d d}||fS )Nr r? r@ rA )rp r mathsqrtr8 coordinate_to_patch_indexrB rC )r\ r idx_1idx_2num_patches_per_sidetoken_1token_2s r r z4Kosmos2Processor._convert_bbox_to_patch_index_tokens s t99>> LE55 $'ty1L'M'M#N#N 4T;OPPLE58#e**"2"21"5"58888#e**"2"21"5"5888 r )r; )NNNN)NNr2 )T)"r r r __doc__
attributes valid_kwargsimage_processor_classtokenizer_classr[ r r r r r r' r r r r BboxInputr r rB r r r r propertyr r r r r
__classcell__)ra s @r r5 r5 D sV " $[1J,-L 0%O+5 +5 +5 +5 +5 +5^ "26 ` `` ItI./` /0
`
` ` ` `D! ! !F " *, @ @YY/0@ @ @
#3- @
sDI~
@ @ @ @F< < <6 6 6 X X XX
+s +E$uSzBRTXY^_dYeTfBf x1 and y2 > y1.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`Tuple[int, int]`: A pair of patch indices representing the upper-left patch and lower-right patch.
zTThe coordinates in `bbox` should be `(x1, y1, x2, y2)` with `x2 > x1` and `y2 > y1`.rk )r r floorceil) r r x1y1x2y2ul_xul_ylr_xlr_yul_idxlr_idxs r r r s RR GGRoppp :b// 0 0D :b// 0 0D 9R..2 3 3D 9R..2 3 3D
(
(4
/F
(
(4
/F 6>r r r c ( d|z }| |z }| |z }||z }||z }| |k r||z }||z } ||z |z }
||z |z } nS||k s||k r||z }||z } ||z |z }
||z |z } n,||z |dz z }||z |dz z } ||z |dz z }
||z |dz z } || |
| fS )a
Given a grid of length `num_patches_per_side` and the indices of the upper-left and lower-right corners of a
bounding box, returns the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
Args:
ul_idx (`int`): the index of the grid cell that corresponds to the upper-left corner of the bounding box.
lr_idx (`int`): the index of the grid cell that corresponds to the lower-right corner of the bounding box.
num_patches_per_side (`int`): the number of patches along each side.
Returns:
`Tuple[float]`: the normalized coordinates of the bounding box, in the form (x1, y1, x2, y2).
g ?r r ) r r r cell_sizer r r r r r r r s r patch_index_to_coordinater s **I ( (D ) )D ( (D ) )D
I
I
I
)
I
)
I
I
I
)
I
)
I
A
-
I
A
-
I
A
-
I
A
-
r2r>r c
d}t j || }g }|D ]}| d }| \ }}}|s8d}| d d | d d f}| d } g }
| D ]} t j d| } t j d| dd }
| r|
r|rX|
t | d t |
d f |
t | d t |
d f |r| |||
f |
D ]0}d|d d |d d
}| |||gf 1|S ) a Extract entities contained in `text`. The bounding bboxes is given in the form of patch indices.
This functioin is only intended to be used within `clean_text_and_extract_entities_with_bboxes` where further
processing happens, including converting to normalized coordinates and whitespace character cleaning up.
Examples:
```python
>>> text = " An image of a snowman warming himself by a fire."
>>> entities = extract_entities_with_patch_indices(text)
>>> entities
[(' a snowman', (31, 41), [(44, 863)]), (' a fire', (130, 137), [(5, 911)])]
```z(?:(([^<]+)))?r Nr r= zrk r? z>r N)rp r sub)r rc entity_namestartr adjusted_start adjusted_endadjusted_entitys r adjust_entity_positionsr# { sk & K %T&5&\::;;Nrvgr4:6677L"^\$BCO r c | }t | t | z
}g }|D ]\ }\ }}}t | t | z
} t | t | z
}
||z
| z }||z
|
z
}| }| |||f|f ||fS )z9Remove the spaces around the text and the entities in it.)r rp rH rI rX ) rc entitiesnew_textleading_spaces new_entitiesr r r r entity_name_leading_spacesentity_name_trailing_spacess r _cleanup_spacesr+ s zz||HYYT[[]]!3!33NL-5 A A) \eS6%(%5%5K>> text = " An image of a snowman warming himself by a fire."
>>> clean_text, entities = clean_text_and_extract_entities_with_bboxes(text)
>>> clean_text
'An image of a snowman warming himself by a fire.'
>>> entities
[('a snowman', (12, 21), [(0.390625, 0.046875, 0.984375, 0.828125)]), ('a fire', (41, 47), [(0.171875, 0.015625, 0.484375, 0.890625)])]
```r r r r c J g | ] }t |d |d S )r rk )r )rE r r s r rG z?clean_text_and_extract_entities_with_bboxes.. s0 qqqbf 5d1gtAwH\ ] ]qqqr )r r r r# rX r+ )
rc r processed_textr r% itemr r r" bboxes_in_coordss
` r r r s VGR..N"Ed"K"K H+ ? ? acDG1&$??qqqqjpqqq+;*==>>>> >8 4 44r )r, )&r r r r typingr r r r image_processing_utilsr image_utilsr r
processing_utilsr r r
r r tokenization_utilsr tokenization_utils_baser r r r r r r# r' r5 r r r r# r+ r r r r r8 s $ # / / / / / / / / / / / / 2 2 2 2 2 2 1 1 1 1 1 1 1 1 b b b b b b b b b b b b b b , , , , , , ? ? ? ? ? ? ? ?
sCx ueUE) *+eCHo eE5%'( )*,
( ( ( ( (,e ( ( ( ( " " " " "
% " " " " -U *n n n n n ~ n n n b
E%u*D$E ]` ejknpskset >(c (3 (c ( ( ( (Z7' 7' 7't " " "*5 5 5 5 5 5r