NgF:HddlZddlZddlZddlZddlmZddlmZmZm Z ddl m Z m Z ddl mZmZdZdZdZd Zd Zd Zd Zd ZdZdZdZdZ d dZdZd!dZeddgZegdZ dZ!e"dkr edSdS)"N)treebank)BrillTaggerTrainer RegexpTagger UnigramTagger)PosWord)Template error_listc"tdS)z Run a demo with defaults. See source comments for details, or docstrings of any of the more specific demo_* functions. NpostagI/var/www/html/ai-engine/env/lib/python3.11/site-packages/nltk/tbl/demo.pydemors  HHHHHrc&tddS)N Exemplify repr(Rule) (see also str(Rule) and Rule.format("verbose")) repr ruleformatNr rrrdemo_repr_rule_formatrs frc&tddS)rstrrNr rrrdemo_str_rule_formatr$s erc&tddS)z* Exemplify Rule.format("verbose") verboserNr rrrdemo_verbose_rule_formatr+s i      rc`tttgdgdS)a The feature/s of a template takes a list of positions relative to the current word where the feature should be looked for, conceptually joined by logical OR. For instance, Pos([-1, 1]), given a value V, will hold whenever V is found one step to the left and/or one step to the right. For contiguous ranges, a 2-arg form giving inclusive end points can also be used: Pos(-3, -1) is the same as the arg below. ) templatesN)r r rrrrdemo_multiposition_featurer$2s2 hs<<<00112333333rc ~tttdgtddggdS)z8 Templates can have more than a single feature. rr r!r"N)r r rrrrrdemo_multifeature_templater&As: htQCyy#r2h--889::::::rc(tdddS)ah Show aggregate statistics per template. Little used templates are candidates for deletion, much used templates may possibly be refined. Deleting unused templates is mostly about saving time and/or space: training is basically O(T) in the number of templates T (also in terms of memory usage, which often will be the limiting factor). T)incremental_statstemplate_statsNr rrrdemo_template_statisticsr*Hs T$777777rc>tjgdddgd}tjgdddgd}tt j||gd }t d t|t|dd d S) a  Template.expand and Feature.expand are class methods facilitating generating large amounts of templates. See their documentation for details. Note: training with 500 templates can easily fill all available even on relatively small corpora )r!rr,F) excludezero)r r!rr,T)r,) combinationsz8Generated {} templates for transformation-based learning)r#r(r)N) rexpandrlistr printformatlenr )wordtplstagtplsr#s rdemo_generated_templatesr8Ts{:::1v5AAAHj!QTBBBGX_h%8vNNNOOI BII  NN    Y$tLLLLLLrc*tddddS)z Plot a learning curve -- the contribution on tagging accuracy of the individual rules. Note: requires matplotlib Tzlearningcurve.png)r(separate_baseline_datalearning_curve_outputNr rrrdemo_learning_curver<hs.  #1rc&tddS)zW Writes a file with context for each erroneous word after tagging testing data z errors.txt) error_outputNr rrrdemo_error_analysisr?us %%%%%%rc&tddS)zm Serializes the learned tagger to a file in pickle format; reloads it and validates the process. z tagger.pcl)serialize_outputNr rrrdemo_serialize_taggerrB|s  L))))))rc*tddddS)z Discard rules with low accuracy. This may hurt performance a bit, but will often produce rules which are more interesting read to a human. i gQ? ) num_sentsmin_acc min_scoreNr rrrdemo_high_accuracy_rulesrHs  T42666666r,r/皙?Frc  |pt}|ddlm}m}|}t |||||\}}}}|rt j|spt||}t|d5}tj ||dddn #1swxYwYtd |t|5}tj|}td|dddn #1swxYwYn t||}td|r5td ||t!j}t#|||| }td |||||}td t!j|z d d|r%td||z|dkr`tdt'|dD].\}}t|dd| | d/| rtd|||\} }!td|std|}"| r||!|r%t1||!|"|td|n:td||} | r|| t| d5}#|#d| z|#dt9|| ddzdddn #1swxYwYtd| | ||} t| d5}tj ||dddn #1swxYwYtd| t| 5}tj|}$dddn #1swxYwYtd| ||}%| |%krtd dStd!dSdS)"a Brill Tagger Demonstration :param templates: how many sentences of training and testing data to use :type templates: list of Template :param tagged_data: maximum number of rule instances to create :type tagged_data: C{int} :param num_sents: how many sentences of training and testing data to use :type num_sents: C{int} :param max_rules: maximum number of rule instances to create :type max_rules: C{int} :param min_score: the minimum score for a rule in order for it to be considered :type min_score: C{int} :param min_acc: the minimum score for a rule in order for it to be considered :type min_acc: C{float} :param train: the fraction of the the corpus to be used for training (1=all) :type train: C{float} :param trace: the level of diagnostic tracing output to produce (0-4) :type trace: C{int} :param randomize: whether the training data should be a random subset of the corpus :type randomize: C{bool} :param ruleformat: rule output format, one of "str", "repr", "verbose" :type ruleformat: C{str} :param incremental_stats: if true, will tag incrementally and collect stats for each rule (rather slow) :type incremental_stats: C{bool} :param template_stats: if true, will print per-template statistics collected in training and (optionally) testing :type template_stats: C{bool} :param error_output: the file where errors will be saved :type error_output: C{string} :param serialize_output: the file where the learned tbl tagger will be saved :type serialize_output: C{string} :param learning_curve_output: filename of plot of learning curve(s) (train and also test, if available) :type learning_curve_output: C{string} :param learning_curve_take: how many rules plotted :type learning_curve_take: C{int} :param baseline_backoff_tagger: the file where rules will be saved :type baseline_backoff_tagger: tagger :param separate_baseline_data: use a fraction of the training data exclusively for training baseline :type separate_baseline_data: C{bool} :param cache_baseline_tagger: cache baseline tagger to this file (only interesting as a temporary workaround to get deterministic output from the baseline unigram tagger between python versions) :type cache_baseline_tagger: C{string} Note on separate_baseline_data: if True, reuse training data both for baseline and rule learner. This is fast and fine for a demo, but is likely to generalize worse on unseen data. Also cannot be sensibly used for learning curves on training data (the baseline will be artificially high). Nr)brill24describe_template_sets)backoffwz)Trained baseline tagger, pickled it to {}zReloaded pickled tagger from zTrained baseline taggerz! Accuracy on test set: {:0.4f}rzTraining tbl tagger...zTrained tbl tagger in z0.2fz secondsz Accuracy on test set: %.4fr,z Learned rules: 4d szJIncrementally tagging the test data, collecting individual rule statisticsz Rule statistics collectedzbWARNING: train_stats asked for separate_baseline_data=True; the baseline will be artificially high)takez Wrote plot of learning curve to zTagging the test datazErrors for Brill Tagger %r  zutf-8z)Wrote tagger errors including context to zWrote pickled tagger to z4Reloaded tagger tried on test set, results identicalz;PROBLEM: Reloaded tagger gave different results on test set) REGEXP_TAGGERnltk.tag.brillrMrN_demo_prepare_dataospathexistsropenpickledumpr3r4loadaccuracytimertrain enumeraterulesbatch_tag_incremental train_statsprint_template_statistics _demo_plot tag_sentswritejoinr encode)&r# tagged_datarE max_rulesrGrFrbtrace randomizerr(r)r>rAr;learning_curve_takebaseline_backoff_taggerr:cache_baseline_taggerrMrN training_data baseline_data gold_data testing_databaseline_tagger print_rulestbrilltrainer brill_taggerrulenorule taggedtest teststats trainstatsfbrill_tagger_reloadedtaggedtest_reloadeds& rr r sp6FBBBBBBBB GII >PUIy2H??;]M9l)w~~344 +'>O+S11 :[ O[999 : : : : : : : : : : : : : : : ;BB)    ' ( ( KK$k+66O I2GII J J J K K K K K K K K K K K K K K K( ?VWWW '(((  / 6 6((33     Y[[F EjG "###== 9gNNL F49;;#7 F F F FGGGS .1F1Fy1Q1QQRRR zz !"""%l&8&8&:&:A>> > >LFD V<<<Z!8!8<<< = = = =5 X   #/"D"D )# # Y -...%  ,   "--//  >  2 29 = = =  N %y*CV     L5JLL M M M %&&&!++L99  5  2 2 4 4 4 , $ $ Y GG47GG H H H GGDIIjJ??@@GGPPSWW X X X Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y H,HHIII#!++L99 "C ( ( 3K K k 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 ;)9;;<<< " # # ={$*K $<$< ! = = = = = = = = = = = = = = = @.>@@AAA*44\BB , , , H I I I I I O P P P P P$#s[8BB!B'D  D D A%OO O  P..P25P2Q;;Q?Q?c|"tdtj}|t||krt|}|r5t jt|t j|t||z}|d|}|||}d|D}|s|} n&t|dz} |d| || d}} t|\} } t|\} }t| \}}td| dd|ddtd| dd| ddtd |||rd nd || ||fS) Nz%Loading tagged data from treebank... c&g|]}d|DS)cg|] }|d S)rr).0ts r z1_demo_prepare_data...as(((aQqT(((rr)rsents rrz&_demo_prepare_data..as'???T((4(((???rr/zRead testing data (dz sents/z wds)zRead training data (z-Read baseline data ({:d} sents/{:d} wds) {:s}z[reused the training set]) r3r tagged_sentsr5randomseedshuffleint corpus_sizer4)rmrbrErpr:cutoffrtrvrwru bl_cutoff trainseqs traintokenstestseqs testtokens bltrainseqs bltraintokenss rrXrXQs  5666+-- C ,, 99 $$ $ C $$%%%{### U" # #F(MF9,-I??Y???L ! %  &&!+ *9* % )** %& +=99Y (66Xz#.}#=#= [- F F F F: F F F FGGG I I I I[ I I I IJJJ 7>>  ( IBB.I   =)\ BBrcdg}dD] }||d|z !fd|d|D}dg}dD] }||d|z !fd|d|D}ddlm}tt t |}||||||gd||dS)N initialerrors rulescoresr!c,g|]}d|dz z Sr, tokencountr)rxrs rrz_demo_plot..}s(KKKQQ<000KKKrc,g|]}d|dz z Srr)rrrs rrz_demo_plot..s(NNNq!a*\222NNNrr)NNNg?) appendmatplotlib.pyplotpyplotr2ranger5plotaxissavefig) r;rrrT testcurve rulescore traincurvepltrs `` rrhrhysA?+,I|,44 223333KKKK)ETE:JKKKI_-.J -66 *R.945555NNNNJuu> " "##AHHQ 1j)))HH $ $ $%%%KK%&&&&&rz^-?[0-9]+(\.[0-9]+)?$CDz.*NN) r)z(The|the|A|a|An|an)$AT)z.*able$JJ)z.*ness$r)z.*ly$RB)z.*s$NNS)z.*ing$VBG)z.*ed$VBDrcRt|td|DfS)Nc34K|]}t|VdS)N)r5)rrs r zcorpus_size..s(00a3q66000000r)r5sum)seqss rrrs* IIs00400000 11r__main__)NNrIrJr/NrKr/FrFFNNNrJNFN)NN)#rYr]rra nltk.corpusrnltk.tagrrrrWrrnltk.tblr r rrrrr$r&r*r8r<r?rBrHr rXrh NN_CD_TAGGERrVr__name__rrrrs1 DDDDDDDDDD$$$$$$$$))))))))   !!! 4 4 4;;; 8 8 8MMM(   &&&***777    'BQBQBQBQJ%C%C%CP''''&|=}MNN        222 zr