Ng%ddlmZddlmZmZmZddlmZddlm Z ddddZ dd Z d d Z d!dZ d dZd"dZd#dZd$dZ d%d&dZd'dZd(dZdS))) annotations)AnyDictList) BeautifulSoup) cells_to_html row_index col_indexcontentcellsList[Dict[str, Any]]cb t|d}t}|D]}|ddks |ddkr}t|d|d|dzD]W}t|d|d|dzD]/}||f|d|dfkr|||f0X|d|df|vr^|d|dc  fd|D}|D]}|dxxdz cc<| f|d|df|v^ |S) aMove cells to the right if spanned cells have an influence on the rendering. Args: cells: List of cells in the table in Deckerd format. Returns: List of cells in the table in Deckerd format with cells moved to the right if spanned. c"|d|dfS)Nyx)rs g/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/metrics/table/table_extraction.pyz/_move_cells_for_spanned_cells..s##/?keywhrrcDg|]}|dk|dk|S)rrr).0ccell_xcell_ys r z1_move_cells_for_spanned_cells.."s9!c!c!cQsVv=M=MRSTWRX\bRbRb!RbRbRbr)sortedsetrangeaddremove) r sorted_cellscells_occupied_by_spannedcellijcells_to_the_right cell_to_mover r!s @@r_move_cells_for_spanned_cellsr/s%%?%?@@@L # ? ? 9q==DIMM49d3i$s)&;<< > >tCy$s)d3i*?@@>>A1v$s)T#Y!777155q!f===>Cy$s)$(AAA!#YS NFF!c!c!c!c!c\!c!c!c  2 ' ' S!!!Q&!!!! % , ,ff-= > > > Cy$s)$(AAA rr strreturnc t|d}|d}|dg}g}t|D]\}}|ddg}t|D]s\}} ||t | jddt | jdd| jd } || tt|S) zConvert html format to Deckerd table structure. Args: content: The html content with a table to extract. Returns: A list of dictionaries where each dictionary represents a cell in the table. z html.parsertabletrthtdcolspanrrowspan)rrrrr ) rfindfindAll enumerateintattrsgettextappendr/) r soupr3rows table_datar+rowrr, cell_datar*s rhtml_table_to_deckerdrF)s - 0 0D IIg  E ==$ DJD// $ $3 T4L))%e,, $ $LAy,,Y::;;,,Y::;;$> D   d # # # # $ ) 4 44rc g}d|D}td|D}tt|}|D]}tt|d|d|dztt|d|d|dz|d|d|d|d|vd}||t |}|S) zConvert Deckerd table structure to html format. Args: cells: List of dictionaries where each dictionary represents a cell in the table. Returns: A string with the html content of the table. c*g|]}|ddk|S)rrrrr*s rr"z)deckerd_table_to_html..Rs!@@@catrc3&K|] }|dV dS)rNrrIs r z(deckerd_table_to_html..Ss&>>dS >>>>>>rrrrrr )row_nums column_numsrrz cell textz column header)maxr$r%listr@r)rtransformer_cellsfirst_row_cells header_length header_rowsr*rEr3s rdeckerd_table_to_htmlrTFsA@@@@O>>o>>>>>MeM**++K , ,U49d3i$s).CDDEEd3icT#Y1F G GHHcci!#Y+5      ++++ + , ,E Lrc>t|}t|S)a7Convert html format to table structure. As a middle step it converts html to the Deckerd format as it's more convenient to work with. Args: content: The html content with a table to extract. Returns: A list of dictionaries where each dictionary represents a cell in the table. )rF_convert_table_from_deckerd)r deckerd_cellss r_convert_table_from_htmlrXds*'22M &} 5 55rcg}|D]Z} |d|d|dd}n)#t$r t}Ynt$r t}YnwxYw||[|S)zConvert deckerd format to table structure. Args: content: The deckerd formatted content with a table to extract. Returns: A list of dictionaries where each dictionary represents a cell in the table. rrr r )KeyError EMPTY_CELL TypeErrorr@)r rCr3rEs rrVrVrsJ % % #"3Z"3Z +II  # # #"III # # #"III #)$$$$ s!AAArCList[List[Dict[str, Any]]]c&t|dS)Nc"|d|dfS)Nr r r)r*s rrz#_sort_table_cells..s[0A4 CT/Urr)r#)rCs r_sort_table_cellsr`s *"U"U V V VVr file_elementsc*g}|D]}d|vr|ddkr{d|vrw t|d}|t|O#t$r1}t d||iYd}~d}~wwxYw|S)aExtracts and converts tables data to a structured format based on the specified table type. Args: file_elements: List of elements from the ground truth file. Returns: A list of tables with each table represented as a list of cell data dictionaries. typeTabler?z$Error converting ground truth data: N)rVr@r` Exceptionprint)raground_truth_table_dataelementconverted_dataes r,extract_and_convert_tables_from_ground_truthrks! 3 3 W  G!;!;'@Q@Q 3!<FO""(../@/P/PQQQQ 3 3 3@Q@@AAA'..r22222222 3 #"s7A B'B  Bhtml source_typecFttd}||vrtd|d||}|dkrtnt}g}|D]Y}|ddkr>||}|s ||}|r$t |}||Z|S)apExtracts and converts table data to a structured format Args: file_elements: List of elements from the file. source_type: 'cells' or 'html'. 'cells' refers to reading 'table_as_cells' field while 'html' is extracted from 'text_as_html' Returns: A list of tables with each table represented as a list of cell data dictionaries. )rlrz source_type z: is not valid. Allowed source_types are "html" and "cells"rrcrd)extract_cells_from_text_as_html!extract_cells_from_table_as_cells ValueErrorr>r`r@) rarm$source_type_to_extraction_strategiesextract_cells_fnfallback_extract_cells_fnpredicted_table_datarhextracted_cellsr(s r*extract_and_convert_tables_from_predictionrws02,,(>>> b; b b b    Month A. 22< " } } Returns: List of extracted cells in a format: [ { "row_index": 0, "col_index": 0, "content": "Month A.", }, ..., ] metadata text_as_htmlzNz*Error converting Unstructured table data: )r>rXrerf)rhvalpredicted_cellsrjs rrorosD *  ! !. 1 1C )3&&tO@2377 @@@ >1>>????????@ s7 AAAcb|dd}d}|rt|}|S)ajExtracts and parse cells from "table_as_cells" field in Element structure Args: element: Example element: { "type": "Table", "metadata": { "table_as_cells": [{"x": 0, "y": 0, "w": 1, "h": 1, "content": "Month A."}, {"x": 0, "y": 1, "w": 1, "h": 1, "content": "22"}] } } Returns: List of extracted cells in a format: [ { "row_index": 0, "col_index": 0, "content": "Month A.", }, ..., ] r{table_as_cellsN)r>rV)rhr~converted_cellss rrprps>0j)--.>??OOG5oFF rN)rr)r r0r1r)rrr1r0)r rr1r)rCr]r1r])rarr1r])rl)rarrmr0r1r])rhrxr1ry)rhrxr1r) __future__rtypingrrrbs4r$unstructured_inference.models.tablesrr[r/rFrTrXrVr`rkrwrorprrrrsS"""""""""""""""">>>>>> 45555:< 6 6 6 62WWWW####8=C( ( ( ( ( V,,,,^r