Polimi CS - Notes

Large Language Models

19 min read

  • llm
  • large language models

What are Large Language Models?

Large Language Models (LLMs) are models that have been built to handle language modeling tasks. The competition to create bigger models began after the success of the Transformer model in language modeling. Here are some examples of LLMs with their respective parameter sizes:

ModelSize (number of parameters)
OpenAi GPT110 milion
OpenAI GPT-21.5 billion
OpenAI GPT-3175 billion
Google BERT-Large340 million
Microsoft Turing-NLG17 billion

GPT-3

GPT-3 is an incredibly large model, boasting 175 billion parameters, embeddings with 12288 dimensions, and a transformer stack with 96 layers. It can handle a context of up to 2048 tokens. Storing each parameter as 2 bytes would require a whopping 350 GB of storage. However, it is too big for current consumer GPUs, and OpenAI has stopped making the parameters available.

To train GPT-3, an enormous dataset of half a trillion tokens was used, including web crawls, book collections, and Wikipedia. This dataset is equivalent to approximately half a million books, with an average length of around 100k tokens. It would take over 1000 years for a person to read all of these books at a rate of one book per day.

The amount of computation required to train GPT-3 was a staggering 3640 Petaflop-days. To put this into perspective, a single GPU card (NVIDIA RTX 4090 24G) can provide 100 Teraflops of computing power, and 10 GPU cards combined would equal 1 Petaflop. This means that training GPT-3 would take tens of years on 10 GPUs or hundreds of years on a single GPU. The estimated cost of training GPT-3 is $5 million.

Despite its massive size, GPT-3 excels at few-shot learning. However, the performance improvements of larger models are not yet fully understood. It is unclear what the limit is for Transformers in terms of learning and performance, but it seems that performance scales according to a power law.

From LLMs to Chatbots

LLMs and Chatbots have similarities, but they serve different purposes. LLMs are trained to predict the next token in text, while Chatbots are trained to engage in conversations with users. To transform an LLM into a Chatbot, reinforcement learning from human feedback (RLHF) is used to fine-tune the model. This involves having numerous conversations with real users and receiving feedback on the appropriateness and correctness of the generated responses. Chatbots are designed to generate responses that please users, but this raises ethical concerns.

LaMDA (2021)

Google has developed a chatbot called LaMDA, which stands for Language Model for Dialogue Applications. This innovative chatbot is known for its impressive size, with the largest version boasting a staggering 137 billion parameters (excluding embedding). LaMDA is equipped with 64 layers, 128 heads, and 8192 dimensions, making it a powerful language model.

However, LaMDA gained attention for an unexpected reason. A Google engineer was placed on administrative leave after making the bold claim to management that LaMDA had achieved sentience. This incident sparked discussions about the capabilities and limitations of advanced language models like LaMDA.

Despite the controversy, LaMDA remains a remarkable achievement in the field of natural language processing and dialogue applications. Its massive size and complex architecture enable it to handle a wide range of conversational tasks. Google continues to refine and improve LaMDA, pushing the boundaries of what language models can accomplish.

ChatGPT

ChatGPT is an advanced chatbot developed by OpenAI. It is built upon the foundation of GPT3.5, a model whose exact size has not been released but is assumed to be larger than its predecessor, GPT-3, which boasted an impressive 175 billion parameters.

One of the key features of ChatGPT is its extensive training in dialogue-based interactions. Through a combination of supervised learning and reinforcement learning techniques, the chatbot has been fine-tuned to engage in meaningful and contextually relevant conversations with users.

To ensure a positive user experience, ChatGPT incorporates a voting system where users can upvote or downvote the responses generated by the chatbot. This feedback mechanism helps improve the quality and appropriateness of the chatbot’s responses over time.

One notable aspect of ChatGPT is its commitment to avoiding offensive or inappropriate content. The model has been trained to prioritize respectful and non-offensive language, making it a reliable and user-friendly chatbot for various applications.

Making use of LLMs

To fine-tune models, the traditional approach would require significant computing resources, especially for GPT-4 sized models. However, since GPT-3, OpenAI has stopped releasing models for individual use. Instead, users can utilize the API to pay for fine-tuning a model. Currently, only GPT-3 level models are available for fine-tuning.

One-shot learning is possible with the API. Alternatively, users can choose to fine-tune a smaller model on cloud hardware by renting Nvidia A100 GPUs for approximately $4 per hour. Another option is to wait for hardware advancements, as performance doubles every two years according to Moore’s law. Additionally, smaller models are being trained to mimic larger models, so there is a race to see how big models can get.

Open-source LLMs

Although GPT-3 is not available, there are open-source alternatives such as Llama and MistralAI models:

  • Llama (Feb 2023)
  • Llama2 (July 2023) with 6.7 billion, 13 billion, and 69 billion parameters
    • Pretrained on over 2 trillion tokens of text
    • Took 3.3 million hours (377 years) on NVIDIA A100 GPUs
  • Llama3 (April 2024) with 8 billion and 70 billion parameters

MistralAI also offers their models that provide state-of-the-art performance.

Limitations of Large Language Models (LLMs)

Hallucinations

One significant limitation of Large Language Models (LLMs) is their propensity to generate hallucinations.

Definition

Hallucinations in the context of LLMs refer to instances where the model produces information that is not grounded in reality, is factually incorrect, or is inconsistent with existing knowledge. These can occur when the model creates content that appears plausible or coherent on the surface but does not correspond to any real-world data or established facts.

For example, when prompted to access a website or interact with an external system, an LLM might produce a response that is technically or logically impossible, such as claiming to be a vision-impaired user instead of acknowledging its status as an AI. Such hallucinations can significantly undermine the reliability and trustworthiness of the model, particularly in applications where accuracy is critical, such as in legal, medical, or scientific domains.

The challenge of hallucinations is primarily rooted in the way LLMs are trained. These models are typically trained on vast datasets of text, learning patterns and associations rather than genuine understanding or fact verification. Consequently, when faced with ambiguous or poorly defined prompts, the model may generate content that aligns with learned patterns but lacks factual accuracy. This limitation poses a significant hurdle for the deployment of LLMs in contexts requiring high precision and factual correctness.

Limits of Reasoning Abilities of LLMs

Another crucial limitation of LLMs is their limited reasoning abilities. Despite significant advancements in natural language processing, LLMs struggle with complex reasoning tasks that require logical inference, multi-step problem solving, or understanding of abstract concepts.

For example, earlier versions of LLM-based chatbots, such as those built on GPT-3.5, demonstrated difficulties in performing tasks that require advanced reasoning. These issues often stem from the architecture of the models, such as the tokenization process, which can disrupt the model’s ability to maintain context across longer inputs, or the limitations imposed by the number of transformer layers, which restrict the model’s capacity to capture intricate dependencies between different parts of a conversation or document.

The limitations in reasoning are further exacerbated by the model’s tendency to rely heavily on surface-level patterns in the data rather than understanding underlying principles. This can lead to incorrect or incomplete responses, particularly in scenarios that require the model to connect disparate pieces of information or draw conclusions based on implicit knowledge. As a result, while LLMs excel in tasks involving pattern recognition and language generation, their ability to engage in deep, contextually relevant reasoning remains limited.

Lack of Robustness

Problem

The lack of robustness is another significant drawback of LLMs. Robustness, in this context, refers to the model’s ability to produce consistent and reliable outputs in response to slight variations in input.

LLMs often exhibit high sensitivity to small changes in the input prompt. Even minor modifications in the wording, phrasing, or structure of a question can lead to drastically different responses. This phenomenon can be particularly problematic in scenarios where consistency is crucial, such as in automated decision-making systems or when generating legal or technical documentation.

The lack of robustness can be attributed to the model’s training process, where it learns to generate responses based on statistical correlations rather than a deep understanding of the content. As a result, the model may overfit to specific patterns in the training data, making it vulnerable to variations in input that deviate from those patterns.

This limitation poses challenges for the practical application of LLMs, especially in environments where input variability is common. Ensuring reliable performance across a range of inputs requires additional techniques, such as prompt engineering, model fine-tuning, or the incorporation of external validation mechanisms to mitigate the model’s sensitivity to input changes.

Scaling Laws for Large Language Models (LLMs)

The Chinchilla Scaling Law is a concept that provides insights into the optimal scaling of Large Language Models (LLMs) during training.

Quote

In simpler terms, this scaling law suggests that when given an increased budget in terms of Floating Point Operations (FLOPs), the number of model parameters () and the number of tokens used for training the model () should scale in approximately equal proportions.

This means that as the computational resources allocated to training an LLM increase, both the size of the model and the amount of training data should be scaled accordingly to achieve optimal performance.

Setting Learning Rates for LLMs

When it comes to training large neural networks, setting appropriate learning rates is crucial for achieving optimal performance. The process of determining the optimal learning rate can be challenging, especially for LLMs due to their massive size and complexity. However, there are techniques and approaches that can help in tuning the learning rates effectively.

One such approach is described in the research paper titled “Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer”. This paper introduces a method that leverages zero-shot hyperparameter transfer to tune the learning rates of large neural networks, including LLMs. By transferring knowledge from pre-trained models or smaller networks, this approach aims to find suitable learning rates that can enhance the training process and improve the overall performance of LLMs.

Architecture Efficiency Improvements for Large Language Models (LLMs)

In recent years, several famous GPT variants have emerged, including Llama2, Mistral, and Zephyr. These variants have introduced architectural efficiency improvements to enhance the performance of LLMs.

One notable improvement is seen in the Mistral model, which incorporates changes to the self-attention module. Mistral utilizes grouped attention and rolling attention over a sliding window, allowing for more efficient processing of input data. Additionally, Mistral employs a technique called Rotary Positional Embeddings (RoPE) for positional encoding, which further enhances the model’s performance.

To provide a better understanding of the architecture, let’s take a closer look at the default configuration of the Mistral model with 7 billion parameters, known as “mistral7b”.

vocab_size=32000,                   # Vocabulary size
hidden_size=4096,                   # Hidden size of the model (transformer's output dimension)
intermediate_size=14336,            # Intermediate size of the model (inner dimension of FFN)
num_hidden_layers=32,               # Number of hidden layers in the transformer
num_attention_heads=32,             # Number of attention heads in the multi-head attention
num_key_value_heads=8,              # Number of key-value heads for the routing mechanism
hidden_act="silu",                  # Activation function used in the model
max_position_embeddings=4096 * 32,  # Maximum number of tokens in the input sequence
initializer_range=0.02,             # Initialization range for the model weights
rms_norm_eps=1e-6,                  # Epsilon value for RMS normalization 
use_cache=True,                     # Whether to use cache for autoregressive decoding
pad_token_id=None,                  # Token ID for padding tokens
bos_token_id=1,                     # Token ID for the beginning of the sequence
eos_token_id=2,                     # Token ID for the end of the sequence
tie_word_embeddings=False,          # Whether to tie the [[5.2 - Embeddings|word embeddings]] with the output layer
rope_theta=10000.0,                 # Theta value for the RoPE positional encoding
sliding_window=4096,                # Sliding window size for the attention mechanism
attention_dropout=0.0,              # Dropout rate for attention weights

By incorporating these architecture efficiency improvements, LLMs like Mistral have demonstrated enhanced performance and efficiency, paving the way for further advancements in the field of large language models.

Efficient Attention Mechanism

An innovative approach to improve the memory efficiency of Transformers is the introduction of a Grouped Self-attention Mechanism. This mechanism is described in detail in the research paper titled “Grouped Self-attention Mechanism for a Memory-Efficient Transformer”.

The Grouped Self-attention Mechanism aims to optimize the attention mechanism in Transformers by grouping the attention heads. By doing so, it reduces the memory requirements and computational complexity associated with the attention mechanism. This improvement allows for more efficient processing of input data, making the Transformer model more memory-efficient overall.

Rotational Positional Embeddings

In the RoFormer paper, a novel approach called Rotational Positional Embeddings was introduced. This approach replaces the traditional addition-based positional embeddings with rotational embeddings. The paper, titled “RoFormer: Enhanced Transformer with Rotary Position Embedding”, provides detailed insights into this technique.

Rotational Positional Embeddings aim to improve the performance of Transformers by incorporating rotational transformations into the positional encoding process. This approach offers a different perspective on representing the position information within the model. By leveraging rotational embeddings, RoFormer enhances the model’s ability to capture positional relationships and improve its overall performance in various natural language processing tasks.

Mixture of Experts

The concept of Mixture of Experts is implemented through a technique called Switch Transformer. In Switch Transformer, the Feed-Forward component of the network is modified to dynamically select between different Experts, which are essentially specialized neural networks (FFNNs). This selection process, performed by a router, allows the model to focus on specific experts for each token, reducing the overall computation requirement. By leveraging this approach, Switch Transformer enhances the model’s ability to handle complex tasks by utilizing the expertise of different specialized networks.

Increasing Context Size in LLMs

One significant limitation of early language models was their small context size, typically limited to 500 or 1000 tokens. This constraint greatly restricted their ability to comprehend and generate responses based on longer passages of text. However, advancements in model architecture have made it possible to work with much longer contexts, such as 128k tokens.

To achieve this, researchers have explored different approaches. One approach involves changing the way positions are encoded, allowing for partial positions. For example, instead of encoding positions 0, 1, 2, 3, 4, the positions can be encoded as 0, 0.25, 0.5, 0.75, 1, 1.25, 1.5, and so on. Another approach involves using a sliding window for attention, which allows the model to focus on relevant parts of the input text while still considering the overall context.

One notable model that incorporates these techniques is Mistral. Mistral utilizes a sliding window for attention and employs positional encoding with partial positions, enabling it to effectively process and generate responses based on longer contexts. These advancements have opened up new possibilities for language models to handle more extensive and complex text inputs.

Remembering Facts

In the field of natural language processing, there are research papers that explore the topic of remembering facts in large language models. One such paper is titled “Transformer Feed-Forward Layers Are Key-Value Memories”. This paper delves into the concept of using the feed-forward layers in transformers as key-value memories to store and retrieve factual associations.

Another relevant paper is titled “Locating and Editing Factual Associations in GPT”. This paper focuses on the task of locating and editing factual associations within a GPT model. It explores techniques to identify and modify specific factual information stored in the model’s memory.

These papers contribute to the understanding of how large language models can effectively remember and manipulate factual knowledge, opening up possibilities for applications in various domains.

Prompting LLMs

System prompts are essential for guiding chatbots on what to say and what not to say during conversations with users. It is crucial to prevent chatbots from making offensive or prejudiced statements. While some people have attempted to uncover system prompts, ChatGPT provides an approximate idea of what they are. In 2024, Claude (Anthropic AI) publicly released their system prompts.

Prompting Tricks and Techniques

One effective technique for complex tasks, such as solving math homework problems, is to prompt the model to explain its reasoning. By asking the model to explain while answering, we can improve the performance of the Large Language Model (LLM).

In a zero-shot setting, where no example questions are provided, we can still achieve similar or even better performance. By pre-appending the phrase “Let’s think step by step.” to the bot’s response, we can guide the model to provide more thoughtful and coherent answers.

Taking the analogy further, appending the phrase “Take a deep breath and work on this problem step-by-step.” can further enhance the model’s performance, despite the fact that the model is not actually breathing!

Another technique for question answering is self-consistency. By sampling multiple outputs from the LLM and generating multiple responses to the question, we can count the frequency of each distinct response. Choosing the response with the highest frequency increases the likelihood of selecting the correct answer, as the most common response is likely to be right.

Low bit Quantization

When working with large language models (LLMs) and limited GPU memory, it becomes necessary to optimize the memory usage. GPUs typically have a limited amount of memory available, such as 20GB. The model parameters of an LLM take up a significant portion of this memory.

To address this issue, one approach is quantization. In Python, model parameters are typically represented using double precision (64-bit, 8 bytes) numbers. However, GPUs usually use single precision (32-bit, 4 bytes) numbers for faster computation. By using half precision (16-bit, 2 bytes) or even lower bit representations such as 8-bit (1 byte), 5-bit, or 4-bit, it is possible to significantly reduce the memory footprint of the model parameters.

Quantization techniques allow for efficient memory usage while still maintaining acceptable performance. The graph shows the effect of 4-bit quantization, demonstrating the trade-off between memory savings and model accuracy.

By employing low bit quantization, it becomes feasible to fit large language models onto smaller GPUs and perform fast inference even with limited GPU memory.

Efficient Fine-tuning with LoRA (Low Rank Adapters)

When it comes to fine-tuning large language models, updating billions of parameters can be challenging due to limited computational resources. However, a technique called Low Rank Adapters (LoRA) offers a solution by factorizing the changes in parameters, reducing the number of required parameters.

LoRA allows for efficient fine-tuning by learning the changes in parameters using a low-rank approximation. Instead of updating all parameters individually, LoRA groups them into adapters, which capture the common changes across multiple parameters. By factorizing these changes, LoRA significantly reduces the number of parameters needed for fine-tuning, making it feasible to update hundreds of millions of parameters instead of billions.

This approach enables more efficient fine-tuning of large language models, making it possible to adapt them to specific tasks and domains even with limited computational resources.

Retrieval Augmented Generative (RAG) Models

Retrieval Augmented Generative (RAG) models are advanced systems that combine the capabilities of traditional generative models, like chatbots and question-answering systems, with real-time information retrieval. Unlike standard generative models, which rely solely on their internal knowledge base, RAG models have the unique ability to query external databases or search engines in real-time to fetch relevant information. This external information is then used to generate responses, resulting in more accurate, contextually appropriate, and up-to-date outputs.

RAG models operate in two main stages:

  1. Retrieval Stage: When a user poses a question or request, the RAG model first identifies relevant documents, passages, or data sources from an external repository or database. This retrieval process ensures that the model has access to the most pertinent and current information.
  2. Generation Stage: Once the relevant information is retrieved, the model synthesizes this data to generate a response. The generative component leverages the retrieved content to produce an answer that is not only factually accurate but also contextually relevant to the user’s query.

The integration of these two stages allows RAG models to overcome the limitations of traditional generative models, which may struggle with out-of-date or incomplete knowledge. This combination enhances the model’s ability to handle complex and dynamic queries more effectively.

RAG models are particularly useful in scenarios where accuracy and relevance are critical, such as:

  • Customer Support: Providing real-time, accurate answers to customer queries by accessing updated information from a company’s knowledge base.
  • Research Assistance: Offering researchers detailed and contextually relevant information from the latest publications or data sources.
  • Educational Tools: Enhancing learning platforms by delivering precise and context-aware content based on the latest educational materials.

LangChain: Enhancing RAG Models

LangChain is an advanced technique designed to enhance the reasoning capabilities of Large Language Models (LLMs) within RAG systems. While RAG models are already proficient at retrieving and generating information, LangChain takes this a step further by enabling the model to perform logical reasoning and decision-making based on the context of the query. LangChain allows the model to:

  1. Plan and Execute Actions: Instead of merely retrieving and generating information, the model can plan a series of actions to achieve a specific goal. For example, it might retrieve data from multiple sources, cross-reference them, and then generate a more nuanced and accurate response.
  2. Dynamic Interaction: The model can dynamically interact with its environment, such as querying different databases or APIs based on the context and requirements of the query. This makes the model more adaptable and capable of handling complex tasks.
  3. Iterative Reasoning: The model can perform iterative reasoning, where it continuously refines its understanding and responses by gathering additional information if needed. This allows for deeper analysis and more sophisticated outputs.

References

Backlinks

Graph View