Ng *ddlZddlZddlZddlZddlmZddlmZmZm Z ddl Z ddl Z ddl mZmZejdZdededefd Zd edeefd Zd Zd ZdZ d"dedede jdedef dZdZdefdZdefdZd#de e jeefdee de edffdZ!d#deeedee de edffdZ"d#deeedee de edffdZ#deeedefd Z$d!Z%dS)$N)Path)ListOptionalUnion)elements_from_jsonelements_to_textzunstructured.evaldocpath output_typereturnc |dkrtt|}n)|dkrt|}ntd|dn1#t$r$}td||d}~wwxYw|S)z Convert given input document (path) into cct-ready. The function only support conversion from `json` or `txt` file. jsontxtzZFile type not supported. Expects one of `json` or `txt`, but received z instead.zCould not read the file N)rr_read_text_file ValueErrorloggererror)r r output_cctes V/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/metrics/utils.py_prepare_output_cctrs  & )*|tj||r{|S)z Recursively lists all files in the given directory and its subdirectories. Returns a list of all files found, with each file's path relative to the initial directory. .)oswalkpathrelpathappendjoin)rlistdirdirpath_ filenamesfilename relative_paths r_listdir_recursiver&%s G!#FFI! F FHGOOGS99M##x((((rw||M8DDEEEE  F Nc<ddddd}||S)a Renames aggregated columns in a DataFrame based on a predefined mapping. Parameters: df (pandas.DataFrame): The DataFrame with aggregated columns to rename. Returns: pandas.DataFrame: A new DataFrame with renamed aggregated columns. meanstdevpstdevcount)_mean_stdev_pstdev_count)columns)rename)df rename_maps r_rename_aggregated_columnsr57s*"WU\]]J 99Z9 ( ((r'cRtj|dS)zs Concatenates multiple pandas DataFrame objects along the columns (side-by-side) and resets the index. )axis)pdconcat reset_index)r3s r_format_grouping_outputr<Es% 9Ra , , . ..r'c tdkrdSj}fd|Dtjdfdt |Dtjdtzdt|dz zzD]\}}g|D]Q}t|tr |d/ t|Rtjdfd ttDdS) zD Displays the evaluation metrics in a formatted text table. rNc g|]<}tt|td|D=S)c3NK|] }tt|V!dSN)lenstr).0items r z&_display...Us.CCST^^CCCCCCr')maxrA)rCheaderr3s r z_display..TsQIOCKKCC6 CCCCCDDr' c3TK|]"\}}||V#dSr@ljust)rCirG col_widthss rrEz_display..Ws7XX 6 Z]33XXXXXXr'-r7z.3fc3ZK|]%}||V&dSr@rK)rCrMrN formatted_rows rrEz_display..as:^^q]1%++JqM::^^^^^^r')rAr1tolistclickechor enumeratesumiterrows isinstancefloatrrBrange)r3headersr"rowrDrNrQs` @@r_displayr]Ms 2ww!||j!!GSZJ JsxxXXXXYwEWEWXXXXXYYY JsS__$sc'llQ.>'??@@@++--   3  0 0D$&& 0$$]]3333$$SYY//// HH^^^^^E#mJ\J\D]D]^^^ ^ ^       r'wT directoryr$r3mode overwritec|dvrtd|r#t|dd|jvr#|dt |d<d|jvr"d|jvr|ddgd|st||}|tj ||d |d |d k d S)z Save the metrics report to tsv file. The function allows an option 1) to choose `mode` as `w` (write) or `a` (append) and 2) to `overwrite` the file if filename existed or not. )r^az/Mode not supported. Mode must be one of [w, a].T)exist_okr,r$ connector)byinplace Fr^)sepr`indexrGN) rrmkdirr1astypeint sort_values_get_non_duplicated_filenameto_csvrrr)r_r$r3r`ras r_write_to_filerqes :JKKK- Yt,,,"*k((--7 RZK2:$=$= ; 3TBBB E/ 8DDII  Y))t$eUY]`U`r'c^tjd|}|rt|dSdS)a A function that defines the sorting method for duplicated file names. For example, with filename.ext filename (1).ext filename (2).ext filename (10).ext - this function extracts the integer in the bracket and sort those numbers ascendingly. z(\d+)r)refindallrm)r$numberss r _sorting_keyrw{s6j8,,G72;qr'c |dd\}}dtj|dtj|d t fd|Dt}g}|D]N}tjd|}|r5|t|dO| d}|D]} | |kr|dz }|d zt|zd z|zS) z Checks the duplicity of the file name from the list and run the numerical check of the minimum number needed as extension to not overwrite the exising file. Returns a string of file name in the format of `filename ().ext`. rr7^z(?: \((\d+)\))?\.$c>g|]}tj||S)rtmatch)rCfpatterns rrHz"_uniquity_file..s*LLLQrx7K7KLqLLLr')keyz \((\d+)\)z (z).) rsplitrtescapesortedrwsearchrrmgroupsortrB) file_listtarget_filenameoriginal_filename extensionduplicated_filesrvfiler}counternumberrs @r_uniquity_filers" $3#9#9#q#A#A yY29.//YY")IBVBVYYYGLLLL)LLLR^___G 00 ,--  0 NN3u{{1~~.. / / / LLNNNG W   qLGG  t #c'll 2T 9I EEr'cJttj||}|S)zs Helper function to calls the `_uniquity_file` function. Takes in directory and file name to check on. )rrr )rr$s rroros bjoox88H Or'scoresroundingc|t|dkrdStj|}|s|St||S)z Find mean from the list. Returns None if no element in the list. Args: rounding (int): optional argument that allows user to define decimal points. Default at 3. rN)rA statisticsr)round)rrr)s rr-r-sF 6{{at ?6 " "D  x  r'cd|D}t|dkrdS|stj|Sttj||S)z Find standard deviation from the list. Returns None if only 0 or 1 element in the list. Args: rounding (int): optional argument that allows user to define decimal points. Default at 3. cg|]}||Sr@r|rCscores rrHz_stdev.. = = =5+ = = = =F 6{{at (''' !&))8 4 44r'cd|D}t|dkrdS|stj|Sttj||S)z Find population standard deviation from the list. Returns None if only 0 or 1 element in the list. Args: rounding (int): optional argument that allows user to define decimal points. Default at 3. cg|]}||Sr@r|rs rrHz_pstdev..rr'r7N)rArr+rrs rr/r/sa> = = = =F 6{{at ) ((( "6**H 5 55r'c t|S)z, Returns the row count of the list. )rA)rs rr0r0s v;;r'c4tj|std|d t |d5}|}dddn #1swxYwY|S#t $r}td|d|d}~wwxYw)zG Reads the contents of a text file and returns it as a string. z The file at z does not exist.ignore)errorsNz+An error occurred when reading the file at z: )rrexistsFileNotFoundErroropenreadOSErrorIOError)rr~textrs rrrs 7>>$  G Et E E EFFFQ $x ( ( ( A6688D                QQQODOOAOOPPPQs;A3A& A3&A**A3-A*.A33 B=BB)r^T)r)&loggingrrtrpathlibrtypingrrrrSpandasr9unstructured.staging.baserr getLoggerrrBrr&r5r<r] DataFrameboolrqrwrroSeriesrYrmr-r.r/r0rr|r'rrs (((((((((( JJJJJJJJ  . / /33*CDI$ ) ) )///   2Y]!')|;>QU,    F#FFFF83 ! !% 4;./ !8C= !QVW\^bWbQc ! ! ! !554(5HSM5%PUW[P[J\5555$ 6 6D%) 6Xc] 65QVX\Q\K] 6 6 6 6 4(UQQQQQr'