then we can condition the next word on previous words
and sample the new frequencies from it
In other words, a language model is general-purpose random text generator, like continually selecting the predicted text on your phone’s keyboard. A language model is designed to identify patterns in text and utilize them to make predictions about the next word. One common pattern is the repetition of a previous word within the same text. By accurately predicting the next word, we can generate complete sentences by iterating this process.
Markov Models
A Markov model is a basic language model that operates on the assumption that the future events are independent of the past events, given the present event. In other words, the probability of a future event is not influenced by the sequence of events that occurred before it.
This simple model calculates the frequencies of -grams (sequences of words) in a large corpus, with longer -grams providing more accurate predictions.
Example
Take in consideration the following passage:
For the Duchess. An invitation from the Queen to play "croquet". The Frog-Footman repeated, in the same solemn tone, only changing the order of the words a little, "From the Queen. An invitation for the Duchess to play ... "
What is the probability of the word “croquet” given the previous words “An invitation from the Queen to play…“?
Important operations on language models are:
Smoothing: add small constant to all counts before estimating probabilities to avoid zero probabilities.
where is the size of the vocabulary and is the smoothing parameter.
Backoff: use trigram count (if you have it), otherwise use bigram count or unigram count. So, don’t invent value for unseen -gram just but backoff to ()-gram. When estimating , if , must use instead, and if , must use instead.
Interpolation: interpolate trigram, bigram, unigram estimates to get more robust estimates.
where . The are chosen to maximize the probability on held-out development data.
Generating text from a Markov model
If we have the ability to estimate the probability of the next word given the context, we can utilize this probability estimate in various ways to generate text. Here are some techniques:
Greedy: At each step, choose the most probable term based on the context:
Random sampling: Sample a term according to the probability distribution:
Top- sampling: Limit the sampling to the most probable terms:
Temperature sampling: Limit the sampling to likely terms by raising probabilities to the power of :
where is the temperature parameter (higher means a more uniform distribution).
Beam search: Search forward one step at a time for the most likely sequence while limiting the search space to a maximum set of candidate sequences.
The greedy technique always produces the same text, while sampling techniques produce different text each time they are run.
These techniques provide flexibility in generating text and can be used in various natural language processing tasks.
Evaluation
There are two primary methods for evaluating language models.
Extrinsic evaluation involves using the model in a downstream task, such as a spelling corrector, and assessing its performance in that specific task.
Intrinsic evaluation, on the other hand, entails training the model’s parameters on a training set and then evaluating its performance on new data, typically a test set.
In this case, the likelihood that the model would generate the new data is used as a measure of how well the model is performing. These evaluation techniques provide valuable insights into the effectiveness and accuracy of language models.
Definition
Perplexity is a metric used to measure the level of surprise or confusion when encountering new text. It indicates how unlikely the observed data is according to the model. A lower perplexity value is considered better.
To calculate perplexity, follow these steps:
Determine the probability of the observed sequence under the model. For a bigram model, this can be computed as the product of individual word probabilities:
Normalize the probability by dividing it by the length of the text sequence. This gives the average per-word probability:
Invert the probability to calculate the uncertainty of the model. Lower perplexity indicates better performance:
Perplexity is a metric commonly used to train and evaluate predictive models. It represents the inverse of the average probability of correctly predicting the next word in a sequence.
where is the sequence of words .
The negative log likelihood (nLL) is the negative logarithm of the sequence’s probability, and dividing it by the sequence length gives the per-word nLL. Perplexity can be calculated as .
Cross Entropy (CE) measures the expected log surprise under the model, indicating the number of bits required to quantify the model’s surprise.
Problems with language models
Generally, the primary issue with Markov models is that as the value of (the number of words in a sequence) increases, the likelihood of finding that specific sequence in a corpus decreases exponentially. This means that there is never enough training data to accurately capture all possible sequences. As a result, relying solely on longer -grams severely limits the effectiveness of the model.
To generate coherent and meaningful language, it is crucial to model long-distance dependencies. However, as the length of the observable dependency increases, the memory and data requirements grow exponentially. This poses a significant challenge for Markov models, as they struggle to scale effectively. Despite this limitation, Markov models were considered state-of-the-art not too long ago.
To overcome these limitations, we need alternative methods that can generalize from limited data and handle longer dependencies more efficiently. These methods should strike a balance between capturing complex patterns in language and managing the computational demands associated with longer dependencies. By developing such techniques, we can enhance the performance and applicability of language models in various natural language processing tasks. Examples of these advanced models include:
