Ng2 ddlZddlmZddlmZddlmZeddgZejeej d Z e d ej d d d dej dd dddej ddddej ddddej ddddd Z dS)!N)tqdm) word_tokenize)parallelize_preprocessz-hz--help)help_option_names)context_settingscdS)Nr D/var/www/html/ai-engine/env/lib/python3.11/site-packages/nltk/cli.pyclir s  Dr tokenizez --languagez-lenz1The language for the Punkt sentence tokenization.)defaulthelpz--preserve-lineTzIAn option to keep the preserve the sentence and not sentence tokenize it.)ris_flagrz --processesz-jzNo. of processes.z --encodingz-eutf8zSpecify encoding of file.z --delimiterz-d z%Specify delimiter to join the tokens.c tjd|5}tjd|5}|dkrWt|D]4}t |t |d|5nRtt ||dD]'}t ||d|(d d d n #1swxYwYd d d d S#1swxYwYd S) z;This command tokenizes text stream using nltk.word_tokenizestdin)encodingstdoutr )endfileT) progress_barN)clickget_text_streamr readlinesprintjoinrr) language preserve_line processesr delimiterfinfoutlineoutlines r tokenize_filer*s*  w : : : Hc  "8h ? ? ? H4A~~ 11TTD)..t)<)<==4dSSSSST 6!3==??ID   HHG)..11t$GGGGG H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H Hs5DB0C* D*C. .D1C. 2DD D)rrnltkr nltk.utilrdictCONTEXT_SETTINGSgroupversion_optionr commandoptionr*r r r r3s ,,,,,,44*:;;; .///  0/ Z   <     T mT13FGGG lD&7RSSS 4+R H HTSHG & H H Hr