Ng/A$ddlmZddlZddlmZmZmZddlZddlm Z m Z m Z ddl m Z ddlmZddlmZmZgdZid d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*id+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;d<d=d>d?d@dAdBdCdDdEdFdGdHdIdJdKdLidMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\d]d^d_d`dadbdcdddedfdgdhdidjdkdldmdndodpdqdrZd!gfddwZddyZdd~ZddZddZddZddZdgfddZ dddZddZdS)) annotationsN)IterableIteratorOptional)DetectorFactory detect_langslang_detect_exception)Element)logger)TESSERACT_LANGUAGES_AND_CODESTESSERACT_LANGUAGES_SPLITTER)~aframharaasmazeaze_cyrlbelbenbodbosbrebulcatcebceschi_sim chi_sim_vertchi_tra chi_tra_vertchrcoscymdandeudivdzoellengenmepoequesteusfaofasfilfinfrafrkfrmfryglagleglggrcgujhathebhinhrvhunhyeikuindislitaita_oldjavjpnjpn_vertkankatkat_oldkazkhmkirkmrkorkor_vertlaolatlavlitltzmalmarmkdmltmonmrimsamyanepnldnorocioriosdpanpolporpusqueronrussansinslkslvsndsnumspaspa_oldsqisrpsrp_latnsunswaswesyrtamtatteltgkthatirtonturuigukrurduzbuzb_cyrlvieyidyorrafrarrazrberbsrbgrcsrchr chinese_chtr#cyr$dar%germanr)enr-etr0far3frr8gar>hir?hrr@hurCidrDisrEitrHjapanrQkoreanrPkurTrs_latinrUlvrVltrYmrr[mtr^msr`neranlrbnorgplrhptrkrorlruroskrpslrsesrusqrv rs_cyrillicryswrzsvr|tar~tertrrugrukuruzvi)rrr languagesOptional[list[str]]returnstrc |tddd|DD}tt|}t |dkrt jd|ddStj|S) zf Entry point: convert languages (list of strings) into tesseract ocr langcode format (uses +) Nz`languages` can not be `None`cg|]}||Sr).0 lang_codes ^/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/partition/common/lang.py z3prepare_languages_for_tesseract..s0   c34K|]}t|VdS)N)/_convert_language_code_to_pytesseract_lang_coderlangs r z2prepare_languages_for_tesseract..s<  FJ ;D A A      rrz@Failed to find any valid standard language code from languages: z, proceed with `eng` instead.r)) ValueErrorlistdictfromkeyslenr warningr join)rconverted_languagess rprepare_languages_for_tesseractrs8999  NW   t}}-@AABB 1$$ C# C C C   u ' ,-@ A AArtesseract_languagect|}|stj|ddS|S)z Convert TesseractOCR language code to PaddleOCR language code. :param tesseract_language: str, language code used in TesseractOCR :return: str, corresponding language code for PaddleOCR or None if not found zM is not a language code supported by PaddleOCR, proceeding with `en` instead.r)#PYTESSERACT_TO_PADDLE_LANG_CODE_MAPgetlowerr r)rrs rtesseract_to_paddle_languagers^ / 2 23E3K3K3M3M N ND ! - - -   t Kr list[str] ocr_languagesstr | list[str] | Nonelist[str] | Nonecd|r#t|}tjd|t|tsJ|rd|vrt dt|t std|r>|dgks |dgks|s.|t}tjd|rvd|vrmt|D]/\}}tj | |||<0t|}|sdS|t}ndg}|SdS)aHandle users defining both `ocr_languages` and `languages`. Give preference to `languages` and convert `ocr_languages` if needed, but default to `None`. `ocr_languages` is only a parameter for `auto.partition`, `partition_image`, & `partition_pdf`. `ocr_languages` should not be defined as 'auto' since 'auto' is intended for language detection which is not supported by `partition_image` or `partition_pdf`. zmThe ocr_languages kwarg will be deprecated in a future version of unstructured. Please use languages instead.Nautoa`ocr_languages` is deprecated but was used to extract text from pdfs and images. The 'auto' argument is only for language *detection* when it is assigned to `languages` and partitioning documents other than pdfs or images. Language detection is not currently supported in pdfs or images.zOThe language parameter must be a list of language codes as strings, ex. ['eng']z}Only one of languages and ocr_languages should be specified. languages is preferred. ocr_languages is marked for deprecation.)_clean_ocr_languages_argr r isinstancerrr TypeErrorsplitr enumerater rr)rrir str_languagess rcheck_language_argsrs 0??  ,     J}c$B$B  =00 P    i & &  ]    )x//93D3DI3D!''(DEE  O     " "$Y// U U4<@tTT ! 4Y??M  t%++,HIIII  I 4rc6|tS)z Convert ocr_languages parameter to list of langcode strings. Assumption: ocr_languages is in tesseract plus sign format )rr rs r&convert_old_ocr_languages_to_languagesr<s   ; < <zB_convert_language_code_to_pytesseract_lang_code..Ts GGG48GGGrz* is not a language supported by Tesseract.r) PYTESSERACT_LANG_CODES_get_iso639_language_objectpart3(_get_all_tesseract_langcodes_with_prefixr rpart2bpart2tr r)r lang_iso639pytesseract_langs_3matched_langcodess rrrEs %%% -d33KHG0FGGG   3 3 3 HIZ [ [ /45FGG G #6 6 6 HI[ \ \ /45FGG G #6 6 6 HI[ \ \ /45FGG G NdNNN O O O2 NdFFFGGG 2rOptional[iso639.Language]c tj|S#tj$rt j|dYdSwxYw)Nz' is not a valid standard language code.)iso639LanguagematchrLanguageNotFoundErrorr r)rs rrrmsc$$TZZ\\222  '$GGGHHHtts03&AAprefixc*fdtDS)zb Get all matching tesseract langcodes with this prefix (may be one or multiple variants). c>g|]}||Sr) startswith)rlangcoders rrz<_get_all_tesseract_langcodes_with_prefix..ys- [ [ [x?R?RSY?Z?Z [H [ [ [r)r)rs`rrrus! \ [ [ [%; [ [ [[rrtextc t|tstd|ddks|dkrdSt jd|r(t |dkrdgSdt_ g}|red|vra|D]\}tj | |}t|dd }|r||j]nt |d krt!jd |d  t%|}n1#t&j$r}t!j|Yd}~dSd}~wwxYwg}|D]w} t+| jd r|d?t| jdd }|r||jx|D]}||vr|||S)z Detects the list of languages present in the text (in the default "auto" mode), or formats and passes through the user inputted document languages if provided. zOThe language parameter must be a list of language codes as strings, ex. ["eng"]rrNz^[\x00-\x7F]+$r)rrz9Since "auto" is present in the input languages provided (z]), the language will be auto detected and the rest of the inputted languages will be ignored.zhzho)rrrstriprerrrrseedr rrrappendrr rrr LangDetectExceptionrrr ) r r doc_languagesrstr_langlanguagelangdetect_resultelangdetect_langslangobjs rdetect_languagesr|s` i & &  ]   |rTZZ\\R//t x!4((S->->-B-BwO!M (+V9,, 5 5D48tLLH28BQB<@@H 5$$X^444  5 y>>A   N-I---      ,T 2 2  $8    N1   44444 ') ) < .s3HHAWQ5G5GHHHHHHHr)r rrFr ) rrrrrmetadatarr'r )rrr! full_textdetected_languagesrs rapply_lang_metadatar+s<H RD h % %">>HHHHHHHI)yINNN&  NNa   '5 0 0  A#5AJ GGGG    Aq&!! '7'?'? $   rlist[str] | strct|trd|}tjdd|}tjdd|}|S)zFix common incorrect definitions for ocr_languages: defining it as a list, adding extra quotation marks, adding brackets. Returns a single string of ocr_languages+z[\"']rz[\[\]])rrrrsubrs rrrsU -&&0// F8R77MF9b-88M r)rrrr)rrrr)rrrrrr)rrrr)rrrr)rrrr)rrrr)r rrrrr)F)rr rrr!r"rr#)rr,rr) __future__rrtypingrrrr langdetectrrr unstructured.documents.elementsr unstructured.loggerr &unstructured.partition.utils.constantsr r rrrrrrrrrrr+rrrrr6s"""""" //////////  433333&&&&&&B7' 47' 47' 47' 4 7'  4 7'  4 7' 47't7'}7' 47' 47' 87' 47' 47' 47'  4!7'" 4#7'7'$ 4%7'& 4'7'( 4)7'* 4+7', 4-7'. 4/7'0 717'2 837'4 457'6 :77'8 497': 4;7'< 4=7'> 4?7'@ 4A7'B 4C7'D 4E7'7'7'F 4G7'H 4I7'J 4K7'L 4M7'N 4O7'P 4Q7'R 4S7'T 4U7'V 4W7'X =Y7'Z 4[7'\ 4]7'^ 4_7'` 4a7'b 4c7'd 4e7'f 4g7'7'h   m7'7'7'#tGLWBBBBB2&<<<<~====%%%%P\\\\'-XLLLLLd).-----`      r