Ng dZddlZddlZddlmZmZddlmZddlmZddl m Z ddl m Z ddl mZGd d e Zd Zd ZddZGdde ZdS)zLanguage Model Interface.N)ABCMetaabstractmethod)bisect) accumulate) NgramCounter) log_base2) VocabularycDeZdZdZdZedZedZdS) SmoothingzNgram Smoothing Interface Implements Chen & Goodman 1995's idea that all smoothing algorithms have certain features in common. This should ideally allow smoothing algorithms to work both with Backoff and Interpolation. c"||_||_dS)z :param vocabulary: The Ngram vocabulary object. :type vocabulary: nltk.lm.vocab.Vocabulary :param counter: The counts of the vocabulary items. :type counter: nltk.lm.counter.NgramCounter N)vocabcounts)self vocabularycounters G/var/www/html/ai-engine/env/lib/python3.11/site-packages/nltk/lm/api.py__init__zSmoothing.__init__s   ctNNotImplementedError)rwords r unigram_scorezSmoothing.unigram_score&!###rctrrrrcontexts r alpha_gammazSmoothing.alpha_gamma*rrN)__name__ __module__ __qualname____doc__rrrrrrr r sc$$^$$$^$$$rr ) metaclassc@t|t|z S)z0Return average (aka mean) for sequence of items.)sumlen)itemss r_meanr*/s u::E ""rcbt|tjr|Stj|Sr) isinstancerandomRandom)seed_or_generators r_random_generatorr04s.#V]33!  =* + ++rc$|stdt|t|krtdtt|}|d}|}|t |||zS)z`Like random.choice, but with weights. Heavily inspired by python 3.6 `random.choices`. z"Can't choose from empty populationz3The number of weights does not match the population) ValueErrorr(listrr-r) populationweightsrandom_generator cum_weightstotal thresholds r_weighted_choicer;:s ?=>>> :#g,,&&NOOOz'**++K OE ''))I f[%)*;<< ==rcdeZdZdZd dZddZddZeddZddZ dZ d Z d Z dd Z dS) LanguageModelzKABC for Language Models. Cannot be directly instantiated itself. Nc||_|r9t|ts$tjd|jjdd|tn||_|tn||_ dS)apCreates new LanguageModel. :param vocabulary: If provided, this vocabulary will be used instead of creating a new one when training. :type vocabulary: `nltk.lm.Vocabulary` or None :param counter: If provided, use this object to count ngrams. :type counter: `nltk.lm.NgramCounter` or None :param ngrams_fn: If given, defines how sentences in training text are turned to ngram sequences. :type ngrams_fn: function or None :param pad_fn: If given, defines how sentences in training text are padded. :type pad_fn: function or None z$The `vocabulary` argument passed to z- must be an instance of `nltk.lm.Vocabulary`.) stacklevelN) orderr,r warningswarn __class__r r rr)rrArrs rrzLanguageModel.__init__Ps  jZ@@  M?t~7N???     &0%7Z\\\Z (/lnnnW rcjs+|tdj|jfd|DdS)zeTrains the model on a text. :param text: Training text as a sequence of sentences. Nz:Cannot fit without a vocabulary or text to create it from.c3LK|]}j|VdSr)r lookup).0sentrs r z$LanguageModel.fit..ts3DDt4:,,T22DDDDDDr)r r3updater)rtextvocabulary_texts` rfitzLanguageModel.fithss z /& P J  o . . . DDDDtDDDDDDDDrc||j||r|j|ndS)zMasks out of vocab (OOV) words and computes their model score. For model-specific logic of calculating scores, see the `unmasked_score` method. N)unmasked_scorer rGrs rscorezLanguageModel.scorevsL "" J  d # #7%TTZ%6%6w%?%?%?PT   rct)aScore a word given some optional context. Concrete models are expected to provide an implementation. Note that this method does not mask its arguments with the OOV label. Use the `score` method for that. :param str word: Word for which we want the score :param tuple(str) context: Context the word is in. If `None`, compute unigram score. :param context: tuple(str) or None :rtype: float rrs rrPzLanguageModel.unmasked_scores"###rcHt|||S)zEvaluate the log score of this word in this context. The arguments are the same as for `score` and `unmasked_score`. )rrQrs rlogscorezLanguageModel.logscores D'22333rcd|r#|jt|dz|n |jjS)zHelper method for retrieving counts for a given context. Assumes context has been checked and oov words in it masked. :type context: tuple(str) or None )rr(unigrams)rrs rcontext_countszLanguageModel.context_countss27> WDKG q( )' 2 24;CW rc@dtfd|DzS)a?Calculate cross-entropy of model for given evaluation text. This implementation is based on the Shannon-McMillan-Breiman theorem, as used and referenced by Dan Jurafsky and Jordan Boyd-Graber. :param Iterable(tuple(str)) text_ngrams: A sequence of ngram tuples. :rtype: float r2cXg|]&}|d|dd'S)r2N)rT)rHngramrs r z)LanguageModel.entropy..s3 K K KeT]]59eCRCj 1 1 K K Kr)r*r text_ngramss` rentropyzLanguageModel.entropys5E K K K K{ K K K    rcHtd||S)zCalculates the perplexity of the given text. This is simply 2 ** cross-entropy for the text, so the arguments are the same. g@)powr_r]s r perplexityzLanguageModel.perplexitys 3 [11222rrVc|gnt|}t|}|dkrt|jkr|j dzdn|j}rR|sPtdkr ddngj}r|Pt|}t|tfd|D|Sg}t|D]0}| d||z|1|S)aGenerate words from the model. :param int num_words: How many words to generate. By default 1. :param text_seed: Generation can be conditioned on preceding context. :param random_seed: A random seed or an instance of `random.Random`. If provided, makes the random sampling part of generation reproducible. :return: One (str) word or a list of words generated from model. Examples: >>> from nltk.lm import MLE >>> lm = MLE(2) >>> lm.fit([[("a", "b"), ("b", "c")]], vocabulary_text=['a', 'b', 'c']) >>> lm.fit([[("a",), ("b",), ("c",)]]) >>> lm.generate(random_seed=3) 'a' >>> lm.generate(text_seed=['a']) 'b' NrVc3DK|]}|VdSr)rQ)rHwrrs rrJz)LanguageModel.generate..s1>>djjG,,>>>>>>r) num_words text_seed random_seed) r4r0r(rArXr rGsortedr;tuplerangeappendgenerate) rrfrgrhr7samples generated_rs ` @rrmzLanguageModel.generates*$+BBi ,[99 >>y>>TZ//4:+/++,,  ))$**;*;G*D*DEEG J' J),W)9)9'!""++r--dj.?.?.H.HII J' J WooG#>>>>>g>>>>>    y!!  A    ')3 0    r)NNr)rVNN)r r!r"r#rrNrQrrPrTrXr_rbrmr$rrr=r=Is EEEE0 E E E E     $ $ $^ $4444        333555555rr=r)r#r-rBabcrrr itertoolsrnltk.lm.counterr nltk.lm.utilrnltk.lm.vocabularyr r r*r0r;r=r$rrrvs6  '''''''' ((((((""""""))))))$$$$$'$$$$6### ,,, > > > >eeeeegeeeeeer