NgdZ ddlZn #e$rYnwxYwddlZddlZddlmZddlmZddl m Z m Z m Z ddl mZmZmZddlmZmZmZddlmZdd lmZdd lmZd ZGd d eZeZGddZGddeZGddeZ Gdde Z!Gdde Z"GddeZ# d*dZ$dZ%dZ& d*dZ'dZ(d Z) d+d!Z*Gd"d#eZ+d$Z,d,d&Z-d'Z.d(Z/e0d)kr e/dSdS)-a A classifier model based on maximum entropy modeling framework. This framework considers all of the probability distributions that are empirically consistent with the training data; and chooses the distribution with the highest entropy. A probability distribution is "empirically consistent" with a set of training data if its estimated frequency with which a class and a feature vector value co-occur is equal to the actual frequency in the data. Terminology: 'feature' ====================== The term *feature* is usually used to refer to some property of an unlabeled token. For example, when performing word sense disambiguation, we might define a ``'prevword'`` feature whose value is the word preceding the target word. However, in the context of maxent modeling, the term *feature* is typically used to refer to a property of a "labeled" token. In order to prevent confusion, we will introduce two distinct terms to disambiguate these two different concepts: - An "input-feature" is a property of an unlabeled token. - A "joint-feature" is a property of a labeled token. In the rest of the ``nltk.classify`` module, the term "features" is used to refer to what we will call "input-features" in this module. In literature that describes and discusses maximum entropy models, input-features are typically called "contexts", and joint-features are simply referred to as "features". Converting Input-Features to Joint-Features ------------------------------------------- In maximum entropy models, joint-features are required to have numeric values. Typically, each input-feature ``input_feat`` is mapped to a set of joint-features of the form: | joint_feat(token, label) = { 1 if input_feat(token) == feat_val | { and label == some_label | { | { 0 otherwise For all values of ``feat_val`` and ``some_label``. This mapping is performed by classes that implement the ``MaxentFeatureEncodingI`` interface. N) defaultdict) ClassifierI) call_megamparse_megam_weightswrite_megam_file) call_tadmparse_tadm_weightswrite_tadm_file) CutoffCheckeraccuracylog_likelihood)gzip_open_unicode)DictionaryProbDist) OrderedDictz epytext enceZdZdZddZdZdZdZdZdZ dd Z dd Z ddZ dZ gdZe ddZdS)MaxentClassifiera A maximum entropy classifier (also known as a "conditional exponential classifier"). This classifier is parameterized by a set of "weights", which are used to combine the joint-features that are generated from a featureset by an "encoding". In particular, the encoding maps each ``(featureset, label)`` pair to a vector. The probability of each label is then computed using the following equation:: dotprod(weights, encode(fs,label)) prob(fs|label) = --------------------------------------------------- sum(dotprod(weights, encode(fs,l)) for l in labels) Where ``dotprod`` is the dot product:: dotprod(a,b) = sum(x*y for (x,y) in zip(a,b)) Tc~||_||_||_|t |ksJdS)a{ Construct a new maxent classifier model. Typically, new classifier models are created using the ``train()`` method. :type encoding: MaxentFeatureEncodingI :param encoding: An encoding that is used to convert the featuresets that are given to the ``classify`` method into joint-feature vectors, which are used by the maxent classifier model. :type weights: list of float :param weights: The feature weight vector for this classifier. :type logarithmic: bool :param logarithmic: If false, then use non-logarithmic weights. N) _encoding_weights _logarithmiclengthlen)selfencodingweights logarithmics P/var/www/html/ai-engine/env/lib/python3.11/site-packages/nltk/classify/maxent.py__init__zMaxentClassifier.__init__asA"" '  CLL000000c4|jSN)rlabelsrs rr"zMaxentClassifier.labelsxs~$$&&&rcl||_|jt|ksJdS)z Set the feature weight vector for this classifier. :param new_weights: The new feature weight vector. :type new_weights: list of float N)rrrr)r new_weightss r set_weightszMaxentClassifier.set_weights{s8 $ ~$$&&#k*:*:::::::rc|jS)zg :return: The feature weight vector for this classifier. :rtype: list of float rr#s rrzMaxentClassifier.weightss }rcP||Sr!) prob_classifymax)r featuresets rclassifyzMaxentClassifier.classifys"!!*--11333rc:i}|jD]i}|j||}|jr#d}|D]\}}||j||zz }|||<Gd}|D]\}}||j||zz}|||<jt ||jdS)Ng?T)log normalize)rr"encoderrr) rr, prob_dictlabelfeature_vectortotalf_idf_valprods rr*zMaxentClassifier.prob_classifys ^**,, ( (E!^22:uEEN (#199KD%T]40588EE#( %  #199KD%DM$/588DD#' %  ")1BdSSSSrc < d}dt|dz zdz}| t  jd}|d|}t d|d d |Dzt d d |dz d t|zzzzttt|D]\}}j ||}|fdd|D]\} } jrj| | z} nj| | z} j | } | dd} | d| zz } t| dkr | dddz} t || |d zdz| fz|xx| z cc<t d d |dz d t|zzzzt d|d fd|Dzt d|d  fd|DzdS)z Print a table showing the effect of each of the features in the given feature set, and how they combine to determine the probabilities of each label for that featureset. 2z %-zs%s%8.3fTkeyreverseNz Featurec36K|]}dd|zddzVdS)z%8s%sN).0ls r z+MaxentClassifier.explain..s3??1eq"1"~.??????rz -cDtj|dS)Nrabsr)fid__rs rz*MaxentClassifier.explain..s#dmE!H&=">">r and label is rz (%s)/,z... z TOTAL:c3.K|]}d|zVdSz%8.3fNrE)rFrGsumss rrHz+MaxentClassifier.explain..s,3V3V!Gd1g4E3V3V3V3V3V3Vrz PROBS:c3HK|]}d|zVdSrV)prob)rFrGpdists rrHz+MaxentClassifier.explain..s2>>!g 1 ->>>>>>r)strr*sortedsamplesrYprintljustjoinrrint enumeraterr2sortrrdescribesplit)rr,columns descr_widthTEMPLATEr"ir4r5r7r8scoredescrrZrWs` @@rexplainzMaxentClassifier.explains  C a000:="":.. UZFFF!   k * *gg??????? @    dSK!Oa#f++o=>>???3!&)) % %HAu!^22:uEEN   >>>>     . % % e$9 M$/%7EE M$/58E//55 $455a85(u::??!#2#J.Eh%Qe!<<===U u$  % dSK!Oa#f++o=>>???   [ ) )BGG3V3V3V3Vv3V3V3V,V,V V       [ ) )gg>>>>v>>>>> ?     r c tdrjd|Sttt t jfdd_jd|S)zW Generates the ranked list of informative features from most to least. _most_informative_featuresNc8tj|Sr!rL)fidrs rrOz.sDM#$6 7 7rTr>)hasattrror\listrangerr)rns` rmost_informative_featuresz*MaxentClassifier.most_informative_featuress 45 6 6 722A26 6.4U3t}--..//7777///D + 22A26 6rallcd}|dkrfd|D}n|dkrfd|D}|d|D]:}tj|ddj|;dS)z :param show: all, neg, or pos (for negative-only or positive-only) :type show: str :param n: The no. of top features :type n: int Nposc6g|]}j|dk|Srr(rFrqrs r zCMaxentClassifier.show_most_informative_features..*BBBC4=+=+A+AC+A+A+Arnegc6g|]}j|dk|Sr{r(r|s rr}zCMaxentClassifier.show_most_informative_features..r~rz8.3frS)rvr^rrrd)rrushowfidsrqs` rshow_most_informative_featuresz/MaxentClassifier.show_most_informative_featuress--d33 5==BBBB4BBBDD U]]BBBB4BBBD8 O OC T]3'MMMt~/F/Fs/K/KMM N N N N O Orcdt|j|jfzS)Nz:)rrr"rr#s r__repr__zMaxentClassifier.__repr__s?K %%'' ( ( N ! ! # #O   r)GISIISMEGAMTADMNrc n|d}|D]}|dvrtd|z|}|dkrt||||fi|S|dkrt||||fi|S|dkrt |||||fi|S|dkr(|} || d<|| d <|| d <|| d <t j|fi| Std |z) a Train a new maxent classifier based on the given corpus of training samples. This classifier will have its weights chosen to maximize entropy while remaining empirically consistent with the training corpus. :rtype: MaxentClassifier :return: The new maxent classifier :type train_toks: list :param train_toks: Training data, represented as a list of pairs, the first member of which is a featureset, and the second of which is a classification label. :type algorithm: str :param algorithm: A case-insensitive string, specifying which algorithm should be used to train the classifier. The following algorithms are currently available. - Iterative Scaling Methods: Generalized Iterative Scaling (``'GIS'``), Improved Iterative Scaling (``'IIS'``) - External Libraries (requiring megam): LM-BFGS algorithm, with training performed by Megam (``'megam'``) The default algorithm is ``'IIS'``. :type trace: int :param trace: The level of diagnostic tracing output to produce. Higher values produce more verbose output. :type encoding: MaxentFeatureEncodingI :param encoding: A feature encoding, used to convert featuresets into feature vectors. If none is specified, then a ``BinaryMaxentFeatureEncoding`` will be built based on the features that are attested in the training corpus. :type labels: list(str) :param labels: The set of possible labels. If none is given, then the set of all labels attested in the training data will be used instead. :param gaussian_prior_sigma: The sigma value for a gaussian prior on model weights. Currently, this is supported by ``megam``. For other algorithms, its value is ignored. :param cutoffs: Arguments specifying various conditions under which the training should be halted. (Some of the cutoff conditions are not supported by some algorithms.) - ``max_iter=v``: Terminate after ``v`` iterations. - ``min_ll=v``: Terminate after the negative average log-likelihood drops under ``v``. - ``min_lldelta=v``: Terminate if a single iteration improves log likelihood by less than ``v``. Niis) max_itermin_ll min_lldeltamax_acc min_accdelta count_cutoffnormexplicit bernoullizUnexpected keyword arg %rgismegamtadmtracerr"gaussian_prior_sigmazUnknown algorithm %s) TypeErrorlower train_maxent_classifier_with_iis train_maxent_classifier_with_gis"train_maxent_classifier_with_megamTadmMaxentClassifiertrain ValueError) cls train_toks algorithmrrr"rcutoffsr?kwargss rrzMaxentClassifier.trainsf|  I C CC    ;c ABBB OO%%   3E8V7> %  3E8V7> ' ! !5E8V5IMT & F#F7O!)F: %F8 -AF) *'-jCCFCC C3i?@@ @r)T)r:)rm)rmrw)NrNNr)__name__ __module__ __qualname____doc__rr"r&rr-r*rlrvrr ALGORITHMS classmethodrrErrrrNs$1111.''';;;444TTT(* * * * X 7 7 7 7OOOO    100JaAaAaA[aAaAaArrc0eZdZdZdZdZdZdZdZdS)MaxentFeatureEncodingIa A mapping that converts a set of input-feature values to a vector of joint-feature values, given a label. This conversion is necessary to translate featuresets into a format that can be used by maximum entropy models. The set of joint-features used by a given encoding is fixed, and each index in the generated joint-feature vectors corresponds to a single joint-feature. The length of the generated joint-feature vectors is therefore constant (for a given encoding). Because the joint-feature vectors generated by ``MaxentFeatureEncodingI`` are typically very sparse, they are represented as a list of ``(index, value)`` tuples, specifying the value of each non-zero joint-feature. Feature encodings are generally created using the ``train()`` method, which generates an appropriate encoding based on the input-feature values and labels that are present in a given corpus. ct)aC Given a (featureset, label) pair, return the corresponding vector of joint-feature values. This vector is represented as a list of ``(index, value)`` tuples, specifying the value of each non-zero joint-feature. :type featureset: dict :rtype: list(tuple(int, int)) NotImplementedErrorrr,r4s rr2zMaxentFeatureEncodingI.encode{"###rct)z :return: The size of the fixed-length joint-feature vectors that are generated by this encoding. :rtype: int rr#s rrzMaxentFeatureEncodingI.length "###rct)z :return: A list of the "known labels" -- i.e., all labels ``l`` such that ``self.encode(fs,l)`` can be a nonzero joint-feature vector for some value of ``fs``. :rtype: list rr#s rr"zMaxentFeatureEncodingI.labelss"###rct)z :return: A string describing the value of the joint-feature whose index in the generated feature vectors is ``fid``. :rtype: str rrrqs rrdzMaxentFeatureEncodingI.describerrct)ao Construct and return new feature encoding, based on a given training corpus ``train_toks``. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. r)rrs rrzMaxentFeatureEncodingI.trainrrN) rrrrr2rr"rdrrErrrrdsi, $ $ $$$$$$$$$$ $ $ $ $ $rrc0eZdZdZdZdZdZdZdZdS)#FunctionBackedMaxentFeatureEncodingz A feature encoding that calls a user-supplied function to map a given featureset/label pair to a sparse joint-feature vector. c0||_||_||_dS)ag Construct a new feature encoding based on the given function. :type func: (callable) :param func: A function that takes two arguments, a featureset and a label, and returns the sparse joint feature vector that encodes them:: func(featureset, label) -> feature_vector This sparse joint feature vector (``feature_vector``) is a list of ``(index,value)`` tuples. :type length: int :param length: The size of the fixed-length joint-feature vectors that are generated by this encoding. :type labels: list :param labels: A list of the "known labels" for this encoding -- i.e., all labels ``l`` such that ``self.encode(fs,l)`` can be a nonzero joint-feature vector for some value of ``fs``. N)_length_func_labels)rfuncrr"s rrz,FunctionBackedMaxentFeatureEncoding.__init__s0   rc.|||Sr!)rrs rr2z*FunctionBackedMaxentFeatureEncoding.encodeszz*e,,,rc|jSr!rr#s rrz*FunctionBackedMaxentFeatureEncoding.length |rc|jSr!rr#s rr"z*FunctionBackedMaxentFeatureEncoding.labelsrrcdS)Nzno description availablerErs rrdz,FunctionBackedMaxentFeatureEncoding.describes))rN) rrrrrr2rr"rdrErrrrsi 8---*****rrcJeZdZdZd dZdZdZdZdZe d d Z d S) BinaryMaxentFeatureEncodinga A feature encoding that generates vectors containing a binary joint-features of the form: | joint_feat(fs, l) = { 1 if (fs[fname] == fval) and (l == label) | { | { 0 otherwise Where ``fname`` is the name of an input-feature, ``fval`` is a value for that input-feature, and ``label`` is a label. Typically, these features are constructed based on a training corpus, using the ``train()`` method. This method will create one feature for each combination of ``fname``, ``fval``, and ``label`` that occurs at least once in the training corpus. The ``unseen_features`` parameter can be used to add "unseen-value features", which are used whenever an input feature has a value that was not encountered in the training corpus. These features have the form: | joint_feat(fs, l) = { 1 if is_unseen(fname, fs[fname]) | { and l == label | { | { 0 otherwise Where ``is_unseen(fname, fval)`` is true if the encoding does not contain any joint features that are true when ``fs[fname]==fval``. The ``alwayson_features`` parameter can be used to add "always-on features", which have the form:: | joint_feat(fs, l) = { 1 if (l == label) | { | { 0 otherwise These always-on features allow the maxent model to directly model the prior probabilities of each label. Fcdt|ttt|krt dt |_ |_ t|_ d_ d_ |rBfdt|D_ xjtj z c_|rKd|D}fdt|D_ xjt|z c_dSdS)a :param labels: A list of the "known labels" for this encoding. :param mapping: A dictionary mapping from ``(fname,fval,label)`` tuples to corresponding joint-feature indexes. These indexes must be the set of integers from 0...len(mapping). If ``mapping[fname,fval,label]=id``, then ``self.encode(..., fname:fval, ..., label)[id]`` is 1; otherwise, it is 0. :param unseen_features: If true, then include unseen value features in the generated joint-feature vectors. :param alwayson_features: If true, then include always-on features in the generated joint-feature vectors. HMapping values must be exactly the set of integers from 0...len(mapping)Nc,i|]\}}||jzSrErrFrir4rs r z8BinaryMaxentFeatureEncoding.__init__..,3,6Qq4<'rch|]\}}}| SrErErFfnamefvalr4s r z7BinaryMaxentFeatureEncoding.__init__..2@@@ 4ee@@@rc,i|]\}}||jzSrErrFrirrs rrz8BinaryMaxentFeatureEncoding.__init__..3&XXX EE1t|#3XXXr setvaluesrtrrrsr_mappingr _alwayson_unseenrbrr"mappingunseen_featuresalwayson_featuresfnamess` rrz$BinaryMaxentFeatureEncoding.__init__H" w~~ Cc'll(;(;$<$< < <8  F|| ( 97|| <, ,  0:CF:K:KDN LLC// /LL  (@@@@@FXXXXiPVFWFWXXXDL LLCKK 'LLLL ( (rcg}|D]\}}|||f|jvr&||j|||fdf7|jrC|jD]}|||f|jvrn,||jvr"||j|df|jr+||jvr"||j|df|SNrT)itemsrappendrrrrr,r4rrrlabel2s rr2z"BinaryMaxentFeatureEncoding.encode6s &++-- B BKE4tU#t}44udE/A!BA FGGGG B"lBBFtV, ==> ,, e) 8et~55 OOT^E2A6 7 7 7rct|tstd |jnV#t$rIdgt |jz|_|jD]\}}||j|<YnwxYw|t |jkr|j|\}}}|d|d|S|jrI||j vr.|jD]\}}||krd|zcSdS|j rI||j vr.|j D]\}}||krd|zcSdStdNzdescribe() expected an intz==rPz label is %rz %s is unseenzBad feature id isinstancerar _inv_mappingAttributeErrorrrrrrrrrr7inforirrr4f_id2s rrdz$BinaryMaxentFeatureEncoding.describeQ$$$ :899 9 ,     , , ,!#s4='9'9 9D =..00 , ,a'+!!$$ , , , #dm$$ $ $#'#4T#: UD%>>t>>U>> > ^ /(=(=(?(? ? ? $ 4 4 6 6 1 1 u5==(50000! 1 1\ /ddl&9&9&;&;;; $ 2 2 4 4 2 2 u5==)E1111! 2 2-.. ..ABBc|jSr!rr#s rr"z"BinaryMaxentFeatureEncoding.labelsj |rc|jSr!rr#s rrz"BinaryMaxentFeatureEncoding.lengthnrrrNc |i}t}tt}|D]\}} |r| |vrtd| z|| |D]A\} } || | fxxdz cc<|| | f|kr| | | f|vrt ||| | | f<B||}|||fi|S)a Construct and return new feature encoding, based on a given training corpus ``train_toks``. See the class description ``BinaryMaxentFeatureEncoding`` for a description of the joint-features that will be included in this encoding. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. :type count_cutoff: int :param count_cutoff: A cutoff value that is used to discard rare joint-features. If a joint-feature's value is 1 fewer than ``count_cutoff`` times in the training corpus, then that joint-feature is not included in the generated encoding. :type labels: list :param labels: A list of labels that should be used by the classifier. If not specified, then the set of labels attested in ``train_toks`` will be used. :param options: Extra parameters for the constructor, such as ``unseen_features`` and ``alwayson_features``. Unexpected label %srT)rrraraddrr rrrr"optionsr seen_labelscounttokr4rrs rrz!BinaryMaxentFeatureEncoding.trainrs8ee C  $ C CJC @%v-- !6!>??? OOE " " " #yy{{ C C teTk"""a'"""%55tU+7::69'lltU 23 C > Fs67..g...rFFrN rrrrrr2rdr"rrrrErrrrs&&P/(/(/(/(b6///20/0/0/[0/0/0/rrcDeZdZdZ d dZedZdZdZdZ dS) GISEncodinga A binary feature encoding which adds one new joint-feature to the joint-features defined by ``BinaryMaxentFeatureEncoding``: a correction feature, whose value is chosen to ensure that the sparse vector always sums to a constant non-negative number. This new feature is used to ensure two preconditions for the GIS training algorithm: - At least one feature vector index must be nonzero for every token. - The feature vector must sum to a constant non-negative number for every token. FNct||||||td|Ddz}||_dS)a  :param C: The correction constant. The value of the correction feature is based on this value. In particular, its value is ``C - sum([v for (f,v) in encoding])``. :seealso: ``BinaryMaxentFeatureEncoding.__init__`` Nch|]\}}}| SrErErs rrz'GISEncoding.__init__..s???3tUU???rrT)rrr_C)rr"rrrCs rrzGISEncoding.__init__s[ $,, &'?4E    9??w???@@1DArc|jS)zOThe non-negative constant that all encoded feature vectors will sum to.)r r#s rr z GISEncoding.Cs wrct|||}t|}td|D}||jkrt d|||j|z f|S)Nc3 K|] \}}|V dSr!rE)rFfvs rrHz%GISEncoding.encode..s&--&1aA------rz&Correction feature is not high enough!)rr2rsumr rr)rr,r4r base_lengthr6s rr2zGISEncoding.encodes.55dJNN188>> --H----- DG  EFF Fdgo6777rc<t|dzSr)rrr#s rrzGISEncoding.lengths*11$77!;;rc|t|kr d|jzSt||S)NzCorrection feature (%s))rrr rd)rr7s rrdzGISEncoding.describes? .55d;; ; ;,tw6 6.77dCC Cr)FFN) rrrrrpropertyr r2rrdrErrrrs  RV X   <<<DDDDDrrcFeZdZd dZdZdZdZdZed d Z dS) TadmEventMaxentFeatureEncodingFct||_t|_t|||j||dSr!)rr_label_mappingrr)rr"rrrs rrz'TadmEventMaxentFeatureEncoding.__init__sM#G,, )mm#,, &$-:K     rcxg}|D]\}}||f|jvrt|j|j||f<||jvr.{rrch|]\}}}| SrErErs rrz6TypedMaxentFeatureEncoding.__init__..rrc,i|]\}}||jzSrErrs rrz7TypedMaxentFeatureEncoding.__init__..rrrrs` rrz#TypedMaxentFeatureEncoding.__init__Trrcjg}|D]\}}t|ttfrL|t ||f|jvr2||j|t ||f|fm|||f|jvr&||j|||fdf|jrC|jD]}|||f|jvrn,||jvr"||j|df|j r+||j vr"||j |df|Sr) rrrafloattyperrrrrrs rr2z!TypedMaxentFeatureEncoding.encodesm&++-- F FKE4$e -- F4::u->>OOT]5$t**e3K%Ld$STTT4'4=88OOT]5$3E%F$JKKKK\F"&,FF!40DMAA!EB!DL00$OOT\%-@!,DEEE > 8et~55 OOT^E2A6 7 7 7rct|tstd |jnV#t$rIdgt |jz|_|jD]\}}||j|<YnwxYw|t |jkr|j|\}}}|d|d|S|jrI||j vr.|jD]\}}||krd|zcSdS|j rI||j vr.|j D]\}}||krd|zcSdStdrrrs rrdz#TypedMaxentFeatureEncoding.describerrc|jSr!rr#s rr"z!TypedMaxentFeatureEncoding.labelsrrc|jSr!rr#s rrz!TypedMaxentFeatureEncoding.lengthrrrNc i}t}tt}|D]\}} |r| |vrtd| z|| |D]m\} } t | ttfvrt | } || | fxxdz cc<|| | f|kr| | | f|vrt||| | | f<n||}|||fi|S)a) Construct and return new feature encoding, based on a given training corpus ``train_toks``. See the class description ``TypedMaxentFeatureEncoding`` for a description of the joint-features that will be included in this encoding. Note: recognized feature values types are (int, float), over types are interpreted as regular binary features. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. :type count_cutoff: int :param count_cutoff: A cutoff value that is used to discard rare joint-features. If a joint-feature's value is 1 fewer than ``count_cutoff`` times in the training corpus, then that joint-feature is not included in the generated encoding. :type labels: list :param labels: A list of labels that should be used by the classifier. If not specified, then the set of labels attested in ``train_toks`` will be used. :param options: Extra parameters for the constructor, such as ``unseen_features`` and ``alwayson_features``. rrT) rrrarrrr)r(rrs rrz TypedMaxentFeatureEncoding.trains8>ee C  $ C CJC @%v-- !6!>??? OOE " " " #yy{{ C C t::#u--::DeTk"""a'"""%55tU+7::69'lltU 23 C > Fs67..g...rrrrrErrr"r"s66p/(/(/(/(b@///25/5/5/[5/5/5/rr"rc |ddt|}|t||}t |dst dd|jz }t||}ttj |dkd}tj t|d } |D]} tj | | <t|| } tj|} ~|dkrt!d |dz|d kr,t!t!d t!d  |d krJ|jpt%| |} |jpt)| |}|j}t!d|| |fzt-| ||}|D]} || xxdz cc<tj|}~| } | | |z |zz } | | || |rnn #t4$rt!dYnxYw|d kr7t%| |} t)| |}t!d| dd|d| S)a Train a new ``ConditionalExponentialClassifier``, using the given training samples, using the Generalized Iterative Scaling algorithm. This ``ConditionalExponentialClassifier`` will encode the model that maximizes entropy from all the models that are empirically consistent with ``train_toks``. :see: ``train_maxent_classifier()`` for parameter descriptions. rdNr"r zJThe GIS algorithm requires an encoding that defines C (e.g., GISEncoding).r/rd ==> Training (%d iterations)r=- Iteration Log Likelihood Accuracy- ---------------------------------------T %9d %14.5f %9.3frT* Training stopped: keyboard interrupt Final 14.5f 9.3f) setdefaultr rrrrrr calculate_empirical_fcountrnumpynonzerozerosrNINF ConditionalExponentialClassifierlog2r^llr accr itercalculate_estimated_fcountrr&checkKeyboardInterrupt)rrrr"r cutoffcheckerCinvempirical_fcount unattestedrrq classifierlog_empirical_fcountrCrDiternumestimated_fcountlog_estimated_fcounts rrrs z3'''!'**M$$Z$?? 8S ! !  -     D2*hGGU]#3q#899!<==Jk#.//55G""z 1(GDDJ!:&677 qyy .1DDEEE qyy  =>>> =>>>  qyy"%O J)O)O#'K8J +K+K',3wC6HHIII :J    " + + %%%*%%%%#(:.>#?#?  !((**G ,/CCtK KG  " "7 + + +"":z:: 5 4 <<< :;;;;;  qyy J 3 3z:.. ;2;;;;;;<<< sCH--I I ctj|d}|D]1\}}|||D]\}}||xx|z cc<2|SNr1)r=r?rr2)rrfcountrr4indexvals rr<r<fsu [**C 0 0F !! U"//#u55 ! !JE3 5MMMS MMMM ! MrcDtj|d}|D]u\}}||}|D]F}||}|||D]\}} ||xx|| zz cc<Gv|SrS)r=r?rr*r]rYr2) rMrrrTrr4rZrYrqrs rrFrFps [**C 0 0F ++ U((--]]__ + +E::e$$D%__S%88 + + Ts td{*  + + Mrc |ddt|}|t||}t ||t |z }t ||}tjt||j d}tj |t |df} ttj |dkd} tjt |d} | D]} tj| | <t!|| } |dkrt#d |dz|d kr,t#t#d t#d  |d krJ|jpt'| |}|jpt+| |}|j}t#d|||fzt/|| | |||| |}| } | |z } | | || |rnn #t6$rt#dYnxYw|d kr7t'| |}t+| |}t#d|dd|d| S)a Train a new ``ConditionalExponentialClassifier``, using the given training samples, using the Improved Iterative Scaling algorithm. This ``ConditionalExponentialClassifier`` will encode the model that maximizes entropy from all the models that are empirically consistent with ``train_toks``. :see: ``train_maxent_classifier()`` for parameter descriptions. rr/Nr0)r?r1rTrr2r=r3r4Tr5r6r7r8r9r:)r;r rrr<rcalculate_nfmapr=arrayr\ __getitem__reshaperr>r?r@rAr^rCr rDr rEcalculate_deltasrr&rGrH)rrrr"rrIempirical_ffreqnfmapnfarray nftransposerLrrqrMrCrDrOdeltass rrrs z3'''!'**M.44Z4OO1XFFZXO J 1 1Ek&E,=>>>DDG-#g,,):;;KU]?a#788;<>> =>>>  qyy"%O J)O)O#'K8J +K+K',3wC6HHIII&  F!((**G v G  " "7 + + +"":z:: 5 4 <<< :;;;;;  qyy J 3 3z:.. ;2;;;;;;<<< sB-H11I Ic t}|D]\\}}|D]B}|td|||DC]dt |DS)a Construct a map that can be used to compress ``nf`` (which is typically sparse). *nf(feature_vector)* is the sum of the feature values for *feature_vector*. This represents the number of features that are active for a given labeled text. This method finds all values of *nf(t)* that are attested for at least one token in the given list of training tokens; and constructs a dictionary mapping these attested values to a continuous range *0...N*. For example, if the only values of *nf()* that were attested were 3, 5, and 7, then ``_nfmap`` might return the dictionary ``{3:0, 5:1, 7:2}``. :return: A map that can be used to compress ``nf`` to a dense vector. :rtype: dict(int -> int) c3 K|] \}}|V dSr!rErFidrVs rrHz"calculate_nfmap..s&KK)2s#KKKKKKrci|]\}}|| SrErE)rFrinfs rrz#calculate_nfmap..s 2 2 2gq"B 2 2 2r)rr"rrr2rb)rrnfsetr_r4s rrYrYs* EEEMMQ__&& M ME IIcKKxsE/J/JKKKKK L L L L M 2 25!1!1 2 2 22rc d}d} tj|d} tjt ||fd} |D]\} } || }|D]g} || | }td|D}|D]3\}}| |||fxx| | |zz cc<4h| t |z} t| D]}tj || }d|z}||z}tj|| zd}tj|| zd}|D]}||xxdz cc<| ||z | z z} tjt||z tjt| z }||kr| cS| S) a Calculate the update values for the classifier weights for this iteration of IIS. These update weights are the value of ``delta`` that solves the equation:: ffreq_empirical[i] = SUM[fs,l] (classifier.prob_classify(fs).prob(l) * feature_vector(fs,l)[i] * exp(delta[i] * nf(feature_vector(fs,l)))) Where: - *(fs,l)* is a (featureset, label) tuple from ``train_toks`` - *feature_vector(fs,l)* = ``encoding.encode(fs,l)`` - *nf(vector)* = ``sum([val for (id,val) in vector])`` This method uses Newton's method to solve this equation for *delta[i]*. In particular, it starts with a guess of ``delta[i]`` = 1; and iteratively updates ``delta`` with: | delta[i] -= (ffreq_empirical[i] - sum1[i])/(-sum2[i]) until convergence, where *sum1* and *sum2* are defined as: | sum1[i](delta) = SUM[fs,l] f[i](fs,l,delta) | sum2[i](delta) = SUM[fs,l] (f[i](fs,l,delta).nf(feature_vector(fs,l))) | f[i](fs,l,delta) = (classifier.prob_classify(fs).prob(l) . | feature_vector(fs,l)[i] . | exp(delta[i] . nf(feature_vector(fs,l)))) Note that *sum1* and *sum2* depend on ``delta``; so they need to be re-computed each iteration. The variables ``nfmap``, ``nfarray``, and ``nftranspose`` are used to generate a dense encoding for *nf(ltext)*. This allows ``_deltas`` to calculate *sum1* and *sum2* using matrices, which yields a significant performance improvement. :param train_toks: The set of training tokens. :type train_toks: list(tuple(dict, str)) :param classifier: The current classifier. :type classifier: ClassifierI :param ffreq_empirical: An array containing the empirical frequency for each feature. The *i*\ th element of this array is the empirical frequency for feature *i*. :type ffreq_empirical: sequence of float :param unattested: An array that is 1 for features that are not attested in the training data; and 0 for features that are attested. In other words, ``unattested[i]==0`` iff ``ffreq_empirical[i]==0``. :type unattested: sequence of int :param nfmap: A map that can be used to compress ``nf`` to a dense vector. :type nfmap: dict(int -> int) :param nfarray: An array that can be used to uncompress ``nf`` from a dense vector. :type nfarray: array(float) :param nftranspose: The transpose of ``nfarray`` :type nftranspose: array(float) g-q=i,r1c3 K|] \}}|V dSr!rEres rrHz#calculate_deltas..Ts&99Yb#S999999rr=r)axisrT) r=onesrr?rr*r"r2rrYrtouterrM)rrMrLffreq_empiricalr_r`rarNEWTON_CONVERGE MAX_NEWTONrbArr4distr5rhrfrVrangenumnf_delta exp_nf_deltanf_exp_nf_deltasum1sum2rqn_errors rr]r]s)ROJ Z))3 / /F  SZZ!2!23S99A ; ; U'',,__&& ; ;E%__S%88N99.99999B) ; ;C%)R-   DIIe$4$4s$::     ;  ;ZA*%%;w//({ % 4y)222y1,1555  C IIINIIII ?T)dU22)C$ 6778859S[[;Q;QQ _ $ $MMM % Mrc ^d}d}d|vr|d}d|vr|d}|5|dd}t|||d}n|td t jd \} } t | d 5} t||| || dddn #1swxYwYtj | n,#ttf$r} td | z| d} ~ wwxYwg} | gdz } |r| dgz } |s| dgz } |r d|dzz }nd}| dd|zdgz } |dkr| dgz } d|vr| dd|dzgz } d|vr| ddt|dzgz } t|dr| dgz } | d| gz } t| } tj| n,#t$r} td | d!| Yd} ~ nd} ~ wwxYwt!|||}|t%jt$jz}t+||S)"a Train a new ``ConditionalExponentialClassifier``, using the given training samples, using the external ``megam`` library. This ``ConditionalExponentialClassifier`` will encode the model that maximizes entropy from all the models that are empirically consistent with ``train_toks``. :see: ``train_maxent_classifier()`` for parameter descriptions. :see: ``nltk.classify.megam`` TrrNrr)r"rz$Specify encoding or labels, not bothznltk-prefixw)rrz,Error while creating megam training file: %s)z-nobiasz-repeat10z -explicitz-fvalsr/r=z-lambdaz%.2fz-tunerz-quietrz-maxirCll_deltaz-dppcostz -multilabel multiclasszWarning: unable to delete z: )getrrrtempfilemkstempopenroscloseOSErrorrMrrrremover^rrr=rBer)rrrr"rrrrrfdtrainfile_name trainfilerr inv_variancestdoutrs rrrsLHIV*%f;' zz.!44 .44  Vt5    ?@@@T%-W===N .# & & ) Hi(i                     Z TTTG!KLLRSSTG ++++G!K=  H:1144   6L0'::G qyyH:VGTF:$6677V FD3vj'9#:#::;;x  #M?" n--G  FB .!!!! BBB @>@@Q@@AAAAAAAAB"&(//*;*;XFFG uz%'"""G Hg . ..sT&(CB/# C/B33C6B37CC8 C33C8F// G9GGc$eZdZedZdS)rc |dd}|dd}|dd}|dd}|dd}|d d}|d } |d } |st||| }tjd d\} } tjd\} }t | d}t ||||g}|dg|d|g|r|dd|dzzg| r|dd| zg| r'|ddt| zg|d| g|d|g|dkr|dgn|dgt|t|5}t|}dddn #1swxYwYtj| tj||tjtjz}|||S)Nrtao_lmvmrrrr"rrrrrr0znltk-tadm-events-z.gz)r~suffixznltk-tadm-weights-r}rz-monitorz-methodz-l2z%.6fr=z-max_itz%dz-fatolz -events_inz -params_outz2>&1z-summary)rrrrrrr rextendrMrrr rrr=rBr)rrrrrrr"sigmarrr trainfile_fdr weightfile_fdweightfile_namerr weightfilers rrzTadmMaxentClassifier.trainsJJ{J77  7A&&::j$//Hd++ 1155zz.!44 ::j))::m,, 5;;L<H(0'7&u( ( ( $ n*2)9AU)V)V)V& %nc::  Hi888 |$$$ 9-...  7 NNE6E1H#45 6 6 6  9 NNIth7 8 8 8  ? NNHfs8}}&<= > > > n5666 7888 199 NNF8 $ $ $ $ NNJ< ( ( (' / " " 5j(44G 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 .!!! /""" 5:eg&&&s8W%%%sI""I&)I&N)rrrrrrErrrrs-5&5&[5&5&5&rrc ddl}ddlm}|}t|d5}|jt t |j||}dddn #1swxYwYt|d5}| |}dddn #1swxYwYt|d5}||}dddn #1swxYwYt|d5}| |}dddn #1swxYwY||||fS)Nr) MaxentDecoder /weights.txt /mapping.tab /labels.txt /alwayson.tab) r= nltk.tabdatarrrZrsmapfloat64txt2list tupkey2dict tab2ivdict) tab_dirr=rmdecrwgtmpglabaons rload_maxent_paramsrsYLLL****** =??D &&& ' 'F1ek$s5=$--2B2BCCDDEEFFFFFFFFFFFFFFF &&& ' '"1q!!""""""""""""""" %%% & &!mmA ''' ( (!Aooa  !!!!!!!!!!!!!!! S# sHAA77A;>A;B66B:=B:C55C9<C9D44D8;D8/tmpc zddlm}ddlm}ddlm}|}||s ||t d|t|dd5} | | tt| dddn #1swxYwYt|dd5} | | |dddn #1swxYwYt|d d5} | | |dddn #1swxYwYt|d d5} | ||ddddS#1swxYwYdS) Nr)mkdir)isdir) MaxentEncoderzSaving Maxent parameters in rrrrr)rros.pathrrrr^rwritelist2txtrreprtolist tupdict2tab ivdict2tab) rrrrrrrrmencrs rsave_maxent_paramsr2s****** =??D 5>> g 2 2 2333 &&& , ,= 4==T3::<rs,,Z LLLL   D  ######))))))QQQQQQQQQQMMMMMMMMMMFFFFFFFFFF''''''//////!!!!!! JAJAJAJAJA{JAJAJA\$4 F$F$F$F$F$F$F$F$R,*,*,*,*,**@,*,*,*^G/G/G/G/G/"8G/G/G/T:D:D:D:D:D-:D:D:Dz5/5/5/5/5/%@5/5/5/pa/a/a/a/a/!7a/a/a/T04____D   &04YYYYx3338{{{PKLT/T/T/T/x7&7&7&7&7&+7&7&7&~,++++. 3 3 3444  zDFFFFFs