Ng0 RdZddlZddlZddlmZddlmZddlmZm Z m Z m Z ddl Z ddl ZddlmZddlmZmZeGdd Zd Zd e ed efd Zd eeded eeeeffdZGddZe je jdde jde jdde jde jdeddddedede efd Z e!d!kr e dSdS)"ak The purpose of this script is to create a comprehensive metric for table evaluation 1. Verify table identification. a. Concatenate all text in the table and ground truth. b. Calculate the difference to find the closest matches. c. If contents are too different, mark as a failure. 2. For each identified table: a. Align elements at the level of individual elements. b. Match elements by text. c. Determine indexes for both predicted and actual data. d. Compare index tuples at column and row levels to assess content shifts. e. Compare the token orders by flattened along column and row levels f. Note: Imperfect HTML is acceptable unless it impedes parsing, in which case the table is considered failed. Example python table_eval.py --prediction_file "model_output.pdf.json" --ground_truth_file "ground_truth.pdf.json" N) dataclass)Path)AnyDictListOptional)TableAlignment),extract_and_convert_tables_from_ground_truth*extract_and_convert_tables_from_predictionceZdZUdZeed<eed<eed<eed<eed<eed<eed<eed <eed <eed <ed efd ZdS)TableEvaluationz,Class representing a gathered table metrics. total_tablestotal_predicted_tablestable_level_acctable_detection_recalltable_detection_precisiontable_detection_f1element_col_level_index_accelement_row_level_index_accelement_col_level_content_accelement_row_level_content_accreturncL|j|jz|j|jzdz zdz S)N)rrrr)selfs a/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/metrics/table/table_eval.pycomposite_structure_accz'TableEvaluation.composite_structure_acc6s>  ,. /1D4VVZ[[ \   N) __name__ __module__ __qualname____doc__int__annotations__floatpropertyrrrr r 's66!!!!$$$$!&&&&!&&&&#((((#(((( Xrr cNtjt|f}tj|}t |D]^\}}||}|dkrt jdtj|gd||||<_|S)zcomputes for each predicted table its accurary compared to ground truth. The accuracy is defined as the SequenceMatcher.ratio() between those two strings. If a prediction does not have a matched ground truth its accuracy is 0 Nr) npzeroslenr get_content_in_tables enumeratedifflibSequenceMatcherratio)predicted_table_dataground_truth_table_datamatched_indicesscoreground_truth_textidx predicted matched_idxs rrr?s Hc/**, - -E&<=TUU#$899  Y%c* "   ,   0) = =a @ k *   %'' c Lrr5rc4td|DS)aCounts the number of predicted tables that have a corresponding match in the ground truth. Args: matched_indices: List of indices indicating matches between predicted and ground truth tables. Returns: The count of matched predicted tables. c3&K|] }|dkdV dS)rNr().0r8s r z*_count_predicted_tables.._s&88Ssaxxqxxxx88r)sum)r5s r_count_predicted_tablesrATs! 88O888 8 88rground_truth_tables_numberct|}t|}d|vr|dt|}||z }|}|dkr||z nd}||zdkr|||zz nd}||zdkrd|z|z||zz nd} ||| fS)a Calculate the table detection metrics: recall, precision, and f1 score. Args: matched_indices: List of indices indicating matches between predicted and ground truth tables For example: matched_indices[i] = j means that the i-th predicted table is matched with the j-th ground truth table. ground_truth_tables_number: the number of ground truth tables. Returns: Tuple of recall, precision, and f1 scores r*rr)r-setremove) r5rBpredicted_tables_number matched_set true_positivefalse_positivepositiverecall precisionf1s r!calculate_table_detection_metricsrNbs"/22o&&K [2 $$M,} )A - - 788  ;Df:Lq:P:PY 9v#5 6 6VWB 9b  rc eZdZ ddeeeefdeeeefdedefdZe dd e d e de eded df d Z d e fd ZdS)TableEvalProcessor皙?html prediction ground_truthcutoff source_typec>||_||_||_||_dS)ai Initializes the TableEvalProcessor prediction and ground truth. Args: ground_truth: Ground truth table data. The tables text should be in the deckerd format. prediction: Predicted table data. cutoff: The cutoff value for the element level alignment. Default is 0.8. Examples: ground_truth: [ { "type": "Table", "text": [ { "id": "f4c35dae-105b-46f5-a77a-7fbc199d6aca", "x": 0, "y": 0, "w": 1, "h": 1, "content": "Cell text" }, ... } ] prediction: [ { "element_id": , ... "metadata": { ... "text_as_html": "
June....
", "table_as_cells": [ { "x": 0, "y": 0, "w": 1, "h": 2, "content": "June" }, ... ] } }, ] NrSrTrUrV)rrSrTrUrVs r__init__zTableEvalProcessor.__init__s)n%( &rNprediction_fileground_truth_filerc,t|5}tj|}dddn #1swxYwYt|5}tj|}dddn #1swxYwY||||||S||||S)aFactory classmethod to initialize the object with path to json files instead of dicts Args: prediction_file: Path to the json file containing the predicted table data. ground_truth_file: Path to the json file containing the ground truth table data. source_type: 'cells' or 'html'. 'cells' refers to reading 'table_as_cells' field while 'html' is extracted from 'text_as_html' cutoff: The cutoff value for the element level alignment. If not set, class default value is used (=0.8). Returns: TableEvalProcessor: An instance of the class initialized with the provided data. NrX)rSrTrV)openjsonload)clsrZr[rUrVfrSrTs rfrom_json_filesz"TableEvalProcessor.from_json_filess$*/ " " &a1J & & & & & & & & & & & & & & & # $ $ (9Q<*:' :'  :':':':'x #'! b b b  b b  b  b b b[ bDPoPPPPPPrrPz--prediction_filez&Path to the model prediction JSON fileT)exists)helptypez--ground_truth_filez"Path to the ground truth JSON filez--cutoffrQz]The cutoff value for the element level alignment. If not set, a default value is used)r{ show_defaultdefaultrzrZr[rUctt|t||}|}t |dS)zBRuns the table evaluation process and prints the computed metrics.reN)rPrbrrvprint)rZr[rU processorreports rrunr9sZ"#22 _ 3I  # # % %F &MMMMMr__main__)"r#r0r^ dataclassesrpathlibrtypingrrrrclicknumpyr+*unstructured.metrics.table.table_alignmentr +unstructured.metrics.table.table_extractionr r r rr$rAlisttupler&rNrPcommandoptionrwrr r(rrrs, !!!!!!,,,,,,,,,,,, EEEEEE   .* 9T#Y 93 9 9 9 9!!#Y!!