How LLMs Actually Generate Text: Token Generation, Attention & KV Cache Explained

When we use tools like ChatGPT and other AI systems, their responses can feel almost instantaneous. But behind that simple interface, a highly structured process is taking place.

To understand how Large Language Models (LLMs) generate text, we need to look at four important concepts:

• Token generation

• Attention

• KV Cache

• Training vs. inference

Let’s understand each concept step by step.

---

1. How Does an LLM Generate Text?

One of the most common questions about AI is:

How can an AI model generate text so quickly?

The answer lies in two major phases:

1. Pre-fill Phase

2. Autoregressive Generation Phase

---

2. Phase 1: Pre-fill — Understanding the Input

Suppose we give the model this prompt:

"Explain AI in simple terms"

Before generating an answer, the model first processes the entire input.

This stage is called the Pre-fill Phase.

What Happens During Pre-fill?

The model:

1. Converts the input into tokens.

2. Processes those tokens through the Transformer layers.

3. Builds contextual representations.

4. Prepares the information required for generation.

The important point is that the input tokens can be processed in parallel.

Why Is This Fast?

Modern GPUs are extremely efficient at parallel computation.

Instead of processing every input token completely one after another, many operations can be performed simultaneously.

Input Prompt
     │
     ▼
Tokenization
     │
     ▼
Transformer Processing
     │
     ▼
Context Representation
     │
     ▼
Ready for Generation

---

3. Phase 2: Autoregressive Generation

After processing the input, the model begins generating the response.

Unlike the pre-fill phase, output generation happens one token at a time.

Token 1
   │
   ▼
Token 2
   │
   ▼
Token 3
   │
   ▼
Token 4
   │
   ▼
   ...

Why?

Because every newly generated token depends on the context that came before it.

For example:

"The capital of India is ____"

The model considers the existing context before predicting the next token.

It may generate:

"The"
   ↓
"capital"
   ↓
"of"
   ↓
"India"
   ↓
"is"
   ↓
"New"
   ↓
"Delhi"

The process continues until the model decides that the response is complete.

---

4. KV Cache — The Hidden Speed Booster

Generating tokens one by one could become computationally expensive.

Without optimization, the model would repeatedly recompute information from previous tokens.

New Token
   │
   ▼
Recompute Previous Context
   │
   ▼
Generate Next Token

This would waste computation.

The solution is the KV Cache.

What Is KV Cache?

KV Cache stores previously calculated:

• Keys (K)

• Values (V)

from the attention mechanism.

When the next token is generated, the model can reuse this cached information instead of recomputing everything from scratch.

Previous Tokens
      │
      ▼
Keys + Values
      │
      ▼
   KV Cache
      │
      ▼
Reuse During Generation
      │
      ▼
Faster Token Generation

Simple Mental Model

Without KV Cache
Recalculate → Recalculate → Recalculate

With KV Cache
Calculate → Store → Reuse → Reuse → Reuse

This optimization is one of the important reasons modern LLM inference can be performed efficiently.

---

5. Attention — How Tokens Understand Context

Attention is one of the core mechanisms behind the Transformer architecture.

A simple way to understand attention is to imagine a group of friends having a conversation.

Each word or token can be thought of as a participant.

        Token A
       ↙       ↘
   Token B ↔ Token C
       ↖       ↗
        Token D

Each token can determine which other tokens are important for understanding its meaning.

Consider:

"Data visualization is powerful"

The representation of one token can be influenced by the surrounding tokens.

For example, the word "visualization" becomes more meaningful when considered together with "Data" and "powerful".

This contextual interaction is what makes attention so important.

---

6. Understanding Q, K and V

Attention uses three important components:

• Query (Q)

• Key (K)

• Value (V)

A simple way to remember them is:

Query → What am I looking for?
Key   → What information do you represent?
Value → What information should I retrieve?

Query

The Query represents what a token is looking for.

"What information is relevant to me?"

Key

The Key represents information used for matching.

"Does this token contain something relevant?"

Value

The Value contains the actual information that can be passed forward.

"What information should I use?"

Easy Memory Trick

Q → Ask
K → Match
V → Get Information

---

7. Attention in Simple Terms

Consider this sentence:

"The bank is near the river."

The word "bank" can have different meanings.

The surrounding word "river" provides important context.

Attention allows the model to determine which surrounding tokens are relevant when constructing the representation of each token.

"The bank is near the river"

             │
             ▼
       Attention
             │
      ┌──────┴──────┐
      ▼             ▼
    "bank"        "river"
      │             │
      └──────┬──────┘
             ▼
        Context

This contextual understanding is fundamental to Transformer-based LLMs.

---

8. Training vs. Inference

One of the most important distinctions in AI is the difference between training and inference.

They may use the same model architecture, but their purposes are very different.

---

8.1 Training — The Learning Phase

During training, the model learns patterns from large amounts of data.

A simplified training loop looks like this:

Training Data
      │
      ▼
Model Prediction
      │
      ▼
Calculate Loss
      │
      ▼
Backpropagation
      │
      ▼
Update Weights
      │
      ▼
Repeat

Two important concepts are:

• Loss Function

• Gradient Descent

The model compares its prediction with the expected result and adjusts its parameters to reduce the error.

Simple Analogy: Cooking Sambar

Imagine cooking sambar for the first time.

Cook
  │
  ▼
Taste
  │
  ▼
Too Much Salt?
  │
  ▼
Adjust Ingredients
  │
  ▼
Taste Again

The process repeats until the result improves.

Similarly, during training:

Prediction
    ↓
Loss
    ↓
Weight Update
    ↓
Better Prediction

The parameters, commonly called weights, are gradually adjusted during training.

---

9. Inference — When You Use the Model

Inference happens when a trained model is used to generate an answer.

For example:

User
 │
 ▼
Prompt
 │
 ▼
Trained LLM
 │
 ▼
Token Generation
 │
 ▼
Response

During normal inference:

• The model is not learning from your prompt.

• Its learned parameters remain fixed.

• The model uses those parameters to calculate the next-token probabilities.

Simple Mental Model

Training  → Model learns
Inference → Model uses what it learned

---

10. Training vs. Inference at a Glance

Aspect	Training	Inference
Purpose	Learn patterns	Generate output
Weights	Updated	Typically fixed
Loss calculation	Used for optimization	Not used for weight updates
Gradient descent	Yes	No
Main goal	Improve the model	Produce predictions
Example	Building an LLM	Asking an LLM a question

---

11. Where Does the Model's Knowledge Come From?

A common misconception is that an LLM searches its training data every time it answers a question.

That is not how a standard LLM works.

During training, information is learned through adjustments to the model's parameters.

Training Data
      │
      ▼
Learning Process
      │
      ▼
Model Parameters
      │
      ▼
Trained Model

During inference, the trained model uses those learned parameters to generate predictions.

User Prompt
      │
      ▼
Trained Model
      │
      ▼
Probability Distribution
      │
      ▼
Next Token

---

12. Scaling — Why Larger Models Can Become More Capable

The Transformer architecture was introduced in the landmark 2017 paper "Attention Is All You Need."

The basic architecture itself is built from relatively well-defined components.

What changed dramatically over time was the scale at which these architectures were trained.

Three major factors are:

More Data
    +
More Compute
    +
More Parameters
    │
    ▼
Larger and More Capable Models

This scaling has contributed to significant improvements in capabilities across generations of language models.

However, model capability is not determined by parameter count alone. Data quality, training methods, architecture, optimization, and inference techniques also matter.

---

13. From LLMs to AI Agents

Now we can connect these concepts to modern AI agents.

AI agents rely on LLMs for tasks such as:

• Understanding user instructions

• Reasoning about possible actions

• Selecting tools

• Generating tool calls

• Interpreting results

• Producing responses

A simplified agent workflow looks like this:

              User Request
                   │
                   ▼
             ┌───────────┐
             │    LLM    │
             │ Reasoning │
             └─────┬─────┘
                   │
                   ▼
             Tool Selection
                   │
          ┌────────┴────────┐
          ▼                 ▼
       Search             API Call
          │                 │
          └────────┬────────┘
                   ▼
             Tool Results
                   │
                   ▼
                 LLM
                   │
                   ▼
             Final Response

The same underlying token-generation mechanisms are therefore part of the foundation of many modern agentic systems.

---

14. Putting Everything Together

The complete picture looks like this:

                    USER PROMPT
                         │
                         ▼
                    TOKENIZATION
                         │
                         ▼
                  ┌──────────────┐
                  │   PRE-FILL   │
                  │              │
                  │ Input tokens │
                  │ processed in │
                  │   parallel   │
                  └──────┬───────┘
                         │
                         ▼
                    ATTENTION
                    Q + K + V
                         │
                         ▼
                   KV CACHE
                         │
                         ▼
              AUTOREGRESSIVE LOOP
                         │
                  ┌──────┴──────┐
                  │             │
                  ▼             ▼
             Next Token      Update Context
                  │             │
                  └──────┬──────┘
                         │
                         ▼
                  Continue Until
                     Complete
                         │
                         ▼
                    FINAL OUTPUT

---

15. The Bigger Picture

What looks like a simple interaction:

You → Ask Question → AI → Answer

actually involves multiple stages:

Prompt
  ↓
Tokenization
  ↓
Transformer Processing
  ↓
Attention
  ↓
KV Cache
  ↓
Next-Token Prediction
  ↓
Autoregressive Generation
  ↓
Final Response

And when an AI agent is involved, additional steps can appear:

User Goal
    ↓
LLM Reasoning
    ↓
Tool Selection
    ↓
Tool Execution
    ↓
Observe Result
    ↓
Generate Next Action
    ↓
Final Response

---

16. Key Takeaways

Concept	Simple Explanation
Tokenization	Converts text into tokens
Pre-fill	Processes the input context
Autoregressive Generation	Generates output token by token
Attention	Determines relationships between tokens
Query	Represents what a token is looking for
Key	Helps match relevant information
Value	Contains information to retrieve
KV Cache	Reuses previous attention computations
Training	Updates model parameters
Inference	Uses the trained model to generate output
Scaling	Increasing data, compute, and model capacity can improve capabilities
AI Agents	Combine LLM reasoning with tools and actions

---

Final Thoughts

The magic of modern AI is not simply that a model can generate text.

The real engineering lies in understanding how that generation happens.

An LLM first processes the input, uses attention to build contextual representations, and then generates the response one token at a time. Techniques such as KV caching make this process considerably more efficient.

Once these fundamentals are understood, concepts such as AI agents, tool calling, and agentic systems become much easier to understand.

The next time you ask an AI a question, don't just think:

"What did the AI answer?"

Start thinking:

"How did the model generate that answer?"

That shift—from using AI to understanding how AI works—is where deeper AI engineering begins.