ї МNgw ЦС.≈UddlmZddlZddlmZddlmZmZmZddl Z ddl m Zddl mZ ddl mZdZd ed <d└Zdd└Zeі╚d└і╚Zee╛і╚dd└і╚Zee╛і╚dd└і╚Zee╛і╚dd└і╚Z dS)И)зannotationsN)з lru_cache)зFinalзListзTuple)зpos_tag)з sent_tokenize)з word_tokenizeИ─z Final[int]зCACHE_MAX_SIZEcС^≈tjdd╛і╚tjdd╛і╚dS)Nзaveraged_perceptron_tagger_engT)зquietз punkt_tab)зnltkзdownload╘СЗU/var/www/html/ai-engine/env/lib/python3.11/site-packages/unstructured/nlp/tokenize.pyзdownload_nltk_packagesrs3─щ└Mп2╦$п?я?т?п?щ└M░+═Tп*я*т*п*п*п*rзpackage_nameзstrзpackage_categoryзreturnзboolcС.≈g}tjjD]L}|═dі╚s tj═|dі╚}|═|і╚▄M tj|⌡d|⌡²|╛і╚dS#ttf$rYdSwxYw)zEChecks to see if the specified NLTK package exists on the file systemз nltk_dataЗ/)зpathsTF) rзdataзpathзendswithзosзjoinзappendзfindзLookupErrorзOSError)rrrr!s rзcheck_for_nltk_packager)sґ─Ю─Eщ■ ■ПП┬ь▐}┼}≤[я)т)П 3щ■7≈<▓<══kя2т2┬Dь ▐┼░TятппПщ▄ п%п6п6╗п6п6╦eпDяDтDпDь┬tЬщ²п!ПППь┬u┬uПЬЬЬsа"A?а?BбBcСr≈tdd╛і╚}tdd╛і╚}|r|stі╚dSdS)z;If required NLTK packages are not available, download them.зtaggersr)rrз tokenizersrN)r)r)зtagger_availableзtokenizer_availables rз&_download_nltk_packages_if_not_presentr/%si─У.ь"ь5ПЯТпУ1ь%╟KПЯТпП П!п)9П!щя т п п п П!П!r)зmaxsizeзtextЗ List[str]cС<≈tі╚t|і╚S)zFA wrapper around the NLTK sentence tokenizer with LRU caching enabled.)r/з_sent_tokenize╘r1s rr r 5С─У+я,т,п,щ≤$ятпrcС<≈tі╚t|і╚S)zBA wrapper around the NLTK word tokenizer with LRU caching enabled.)r/з_word_tokenizer5s rr r <r6rЗList[Tuple[str, str]]cС╟≈tі╚t|і╚}g}|D]3}t|і╚}|═t |і╚і╚▄4|S)z>A wrapper around the NLTK POS tagger with LRU caching enabled.)r/r4r8зextendз_pos_tag)r1з sentencesзparts_of_speechзsentenceзtokenss rrrCse─У+я,т,п,У≤tя$т$─Iь-/─OьП1П1┬щ═я)т)┬ьвр²x╗я/т/я0т0п0п0ьпr)rrrrrr)r1rrr2)r1rrr9)з __future__rr#з functoolsrзtypingrrrrrr<r r4r r8rз__annotations__rr)r/rrrЗrEs{Пь"п"п"п"п"п"п"Ю ─ ─ ─ ьпппппь%п%п%п%п%п%п%п%п%п%Ю───ь$п$п$п$п$п$ь0п0п0п0п0п0ь0п0п0п0п0п0Ю ─п п п я П+П+П+П ПППП"─│└П!П!Я └П!П─░>п"я"т"П П П Я#т"П П─░>п"я"т"П П П Я#т"П П─░>п"я"т"ПППЯ#т"ПППr