Word embeddings appeared around 2013 and improved performance on just about every NLP task.
Definition
Word embeddings are compact representations of words in a multi-dimensional space, usually ranging from 100 to 1000 dimensions. They are much smaller in size compared to one-hot encoding, which is commonly used for large vocabularies of 100 thousand to 1 million terms.
Topic models, such as Latent Dirichlet Allocation, also generate dense vectors. However, in this case, we are learning word representations rather than document representations. Similar to one-hot encodings, word embeddings can be combined to represent sentences or documents.
If your task is to fill in the blank in the sentence “Sure Sally, let’s have a skype call at 3pm ___ the 3rd of June.” you can use word embeddings to find the most likely word to fill in the blank, like a preposition, a day of the week or a particular time-zone.
Note that, very few words can fit the context and those that do come in groups that are semantically related to one other.
Supervised machine learning models are responsible for generating embeddings. These models are trained to predict the missing word by considering the surrounding context, which can include both previous (causal model) and future words (non-causal model).
Predicting the missing word involves considering features such as the words in the current context and a target word. This is a multi-class problem where the goal is to estimate the probability for each word in the vocabulary. However, this approach requires a large number of parameters. For instance, using a bag-of-words feature vector and a multi-class linear classifier like Logistic Regression, the number of parameters needed would be quadratic in the size of the vocabulary. For a vocabulary of thousand words, this would amount to billion parameters.
Neural Networks
The brain is made up of numerous neurons, with each neuron being connected to multiple other neurons. Neurons act as basic processing units, receiving electrochemical signals from other neurons. If the sum of these inputs surpasses a certain threshold, the neuron generates an output signal. Through the formation of specific patterns of connections between neurons, the brain learns to carry out complex tasks. These connections between neurons are referred to as synapses.
Definition
An artificial neural network is a computational model inspired by the brain, composed of a large number of simple processing units (neurons) that are connected to each other. Each neurons is modeled as a simple step function that takes the weighted inputs and the threshold as parameters to produce an output value using a non-linear activation function (like sigmoind, tanh or ReLU).
Configuration
It’s possible to create different types of models by varying topology (shape) of the network, activation function used in the neurons, and the loss function optimized. Neural networks made up of neurons arranged in layers, each layer receives inputs from the previous layer and, if activated, emits outputs to the next layer. The internal layers (neither input or output) are called hidden layers and they allow the model to find a non-linear decision boundary.
Parameters of the model are the weights of the connections between neurons, and the bias term that is added to the weighted sum of the inputs. The weights are learned by the model during the training process, where the model is presented with a set of inputs and the corresponding outputs, and the weights are adjusted to minimize the difference between the predicted and the actual outputs.
Non-linear classification problem
Consider the game of tic-tac-toe, where the goal is to classify the board as a x-win, o-win, or nobody-win.
X | O | X | O | | O | X
where is the first player, is the second player, and is an empty cell. In this case we have a 9 dimensional feature vector
A non-linear classification problem is a problem where no linear decision boundary can be found to separate the classes. In this case, a simple two layer neural network can:
the first layer of neurons can learn to identify simple winning patterns (like 3 vertical crosses or 3 diagonal crosses)
the second layer can identify the winner based on the output of the first layer and perform disjunction
Training
The process of learning parameters involves backpropagation, which consists of the following steps:
Random initialization: We start by randomly assigning weights and biases.
Training loop: For each sample in the training set, we perform the following:
Forward pass: We feed the input through the network to calculate the prediction.
Backward pass: We update the parameters by rewarding the connections that produced a correct prediction and penalizing the ones that didn’t.
This process can be seen as a gradient descent routine, where we optimize a nested function and use the chain rule from calculus to compute the derivative of the loss function with respect to each layer of parameters.
Properties
Neural networks possess immense learning capabilities, as even a single hidden layer network can learn any function, given that the layer is sufficiently large. There are numerous possibilities for network structures, and selecting the appropriate structure is a hyperparameter that requires tuning. Neural networks are utilized for both supervised learning and unsupervised learning tasks.
There are various parameters that can be adjusted in neural networks, including the network architecture and the choice of activation function for the neurons. Additionally, there are numerous hardware and software optimizations available for training neural networks, although these optimizations can make neural networks more computationally intensive compared to other learning algorithms.
Word embeddings can be viewed as a type of matrix decomposition, where a square count matrix () is utilized to capture word co-occurrences within a fixed-size context window. By factorizing this matrix, the information within these windows is generalized and word embedding vectors are generated.
Word2Vec
Definition
Word2Vec is a model that learns dense vector representations of words, using a shallow neural network with a single hidden layer. It was developed by Tomas Mikolov at Google in 2013.
Word2Vec addresses the issue of parameter space by utilizing a bag-of-words representation, a neural network with a single linear hidden layer, and a training approach that incorporates negative examples. There are two variations of Word2Vec:
The Continuous Bag of Words (CBOW) model, which is trained to predict an observed word based on the surrounding context of words. The context is defined as all the terms within a symmetric window around the target word.
The Skip-gram model, which is trained to predict a single context word given an observed word.
In summary, the skip-gram model is a one-to-one prediction model, while the CBOW is a many-to-one prediction model. The skip-gram model predict the context word given observed word using of the dot product of the embedding representations of the words
Trying to directly optimize it is headache because we need to sum over all the possible context words
so Milkov introduce negative examples: for each observed word and context word , we sample words from the vocabulary and train the model to predict that these words are not the context word. This turn the problem into a binary classification task, trying to find if a pair of words is observed or not, so objective contains no nasty sums.
In CBOW model, the continuous bag-of-words model, estimated exactly the same way except that the context is the sum of the word vectors in the context window.
GloVe
Definition
GloVe is a model that learns dense vector representations of words, using a matrix factorization technique on the word co-occurrence matrix.
The aim is to give probabilistic interpretation to translation in embedding space.
Example
Translation in space from “steam” to “ice” should increase the chance of seeing the word “solid”:
So, projection of “ice” minus “steam” onto vector of “solid” should be function of conditional probability of “solid” given “ice” and “steam”.
Solving the problem is equivalent to finding the best fit for the objective function, which is represented by the equation . In this equation, and are embedding vectors, and are bias terms, and represents the co-occurrence count. The objective function is approximated by minimizing a weighted least squares objective.
Comparison between Word2Vec(CBOW), Word2Vec(Skip-gram) and GloVe
According to the original paper, Skip-gram model beats CBOW model in terms of quality of embeddings (except for fastText). According to Levy paper, Word2Vec is faster to train, has lower memory requirements and is more accurate than GloVe.
Properties of Word Embeddings
Word embeddings possess a multitude of intriguing and somewhat unexpected characteristics. One such characteristic is semantic clustering, where words that are neighbors in the embedding space tend to be semantically related. This means that words with similar meanings or contexts are grouped together in the embedding space.
Another notable property of word embeddings is their dense distributed representation. Unlike traditional vector representations, where each dimension has a clear meaning, individual dimensions in word embeddings do not have specific interpretations. However, the translation of words in the embedding space holds meaning. This allows for the encoding of analogies and the additive nature of semantics within the embedding space.
Furthermore, word embeddings have the ability to uncover various interesting relationships between words. By analyzing the distances and similarities between word vectors, we can discover connections and associations that may not be immediately apparent. This opens up possibilities for exploring semantic relationships, identifying synonyms, and even detecting patterns and trends within language.
In summary, word embeddings offer a unique perspective on language representation. Through semantic clustering, dense distributed representations, and the discovery of relationships, word embeddings provide valuable insights into the structure and meaning of words in a computational context.
Applications of Word Embeddings
Word embeddings are utilized in various NLP tasks, which can be categorized into two groups:
Causal models: These models restrict the context words to appear before the missing word. They are used for predicting the next word in a sequence and can handle longer dependencies compared to -gram models, such as language modeling.
Non-causal models: These models can be used as additional feature vectors to represent words. They improve performance on most tasks, such as sentiment analysis or language translation, by leveraging additional domain knowledge, specifically the semantics of words.
The low-dimensional representation of word embeddings allows similar terms to have similar descriptions. This enables the model to generalize from semantically related examples. For example, in sentiment analysis, the model can learn that “good” and “great” are similar, while “bad” and “terrible” are similar. Additionally, embeddings implicitly encode part-of-speech and hypernym relationships in an additive manner.
By placing similar concepts close together in the embedding space, embeddings can be used to discover implied properties of those concepts.
Sub-word Embeddings
Word embedding works well if the vocabulary is fixed, so cannot deal with new words in the test set. If we see a new word, we don’t have to embedding for it and we must ignore it or use a special token to represent it.
Definition
FastText is a model that split words into character sequences and learn embeddings for character -grams combined with the word embeddings.
The main advantage of FastText is that it can deals nicely with morphologically related terms, like “believe” and “believing”, that share the same root.
