05. Transformer Overview

A Transformer is a neural network architecture that reads an entire sequence of words at once — not one at a time — and figures out which words matter most by using a mechanism called self-attention.

Before 2017, every AI that processed language worked like a person reading a book: one word at a time, left to right, building up understanding step by step. The Transformer changed everything by reading the whole sentence simultaneously, like looking at all the puzzle pieces at once instead of picking them up one by one.

This single idea — processing all words in parallel — is why your phone can autocomplete sentences, why ChatGPT can hold conversations, and why AI has advanced faster in the last five years than in the previous fifty.

flowchart TD
    BEFORE["Before Transformers\n(Processed words one at a time)"]
    AFTER["After Transformers\n(Process all words at once)"]

    BEFORE --> SLOW["🐢 Slow sequential processing"]
    BEFORE --> SHORT["🐢 Couldn't remember long sentences"]
    BEFORE --> SMALL["🐢 Hard to train on lots of data"]

    AFTER --> FAST["⚡ Parallel processing on GPUs"]
    AFTER --> LONG["⚡ Each word sees all other words"]
    AFTER --> BIG["⚡ Scales to internet-sized data"]

    style BEFORE fill:#ef4444,color:#fff
    style AFTER fill:#22c55e,color:#fff

Think about how you read this sentence:

“The animal didn’t cross the street because it was too tired.”

What does “it” refer to?

Your brain immediately connects “it” to “animal.” You didn’t read the sentence one word at a time and forget the beginning. By the time you reached “tired,” you still knew what “animal” means, what “street” means, and what’s happening overall.

You naturally pay attention to the important words and connect them.

Before Transformers, computers read like this:

flowchart LR
    W1["The"] --> W2["animal"]
    W2 --> W3["didn't"]
    W3 --> W4["cross"]
    W4 --> W5["the"]
    W5 --> W6["street"]
    W6 --> W7["because"]
    W7 --> W8["it"]
    W8 --> W9["was"]
    W9 --> W10["too"]
    W10 --> W11["tired"]

    style W1 fill:#ef4444,color:#fff
    style W8 fill:#f59e0b,color:#fff
    style W11 fill:#22c55e,color:#fff

Word by word. One after another. By the time the computer reached “it,” it had to remember what came before through a compressed summary — like trying to describe a movie you watched years ago from memory alone. Details fade. Long connections break.

This is why older AI systems (RNNs and LSTMs) struggled with long sentences. They simply forgot.

flowchart TD
    ALL["'The animal didn't cross the street because it was too tired.'"]

    ALL --> ATT["Self-Attention\n(each word looks at\nALL other words at once)"]

    ATT --> IT["'it' decides which words\nmatter most for its meaning"]
    IT --> CONNECT1["'animal' → very important\n(it = animal)"]
    IT --> CONNECT2["'tired' → important\n(why it didn't cross)"]
    IT --> CONNECT3["'the' → less important"]
    IT --> CONNECT4["'because' → somewhat important"]

    style IT fill:#f59e0b,color:#fff
    style CONNECT1 fill:#22c55e,color:#fff
    style CONNECT2 fill:#22c55e,color:#fff
    style CONNECT3 fill:#ef4444,color:#fff
    style CONNECT4 fill:#8b5cf6,color:#fff

The word “it” looks at every other word simultaneously. It computes a “relevance score” for each one. “Animal” gets a high score. “Tired” gets a high score. “The” gets a low score. Through this, the model understands that “it” refers to “the animal” — and that the animal is tired, which explains why it didn’t cross.

Before Transformers, the best architecture for language was the Recurrent Neural Network (RNN) and its upgrade, the LSTM (Long Short-Term Memory).

flowchart LR
    subgraph RNN["RNN — Sequential Processing"]
        R1["Word 1"] --> H1["Hidden State 1"]
        H1 --> R2["Word 2"]
        R2 --> H2["Hidden State 2"]
        H2 --> R3["Word 3"]
        R3 --> H3["Hidden State 3"]
        H3 --> R4["..."]
        R4 --> H4["Hidden State N"]
    end

    style H1 fill:#ef4444,color:#fff
    style H2 fill:#ef4444,color:#fff
    style H3 fill:#ef4444,color:#fff
    style H4 fill:#ef4444,color:#fff

Why RNNs failed:

The ‘Vanilla’ RNN problem:

Input: "I was born in France. I lived there for 20 years. I speak fluent..."
Prediction: "French" ← need to remember "France" from far back

RNN path:  France → (20 steps of compression) → ???
           Lost somewhere in the middle. Forgot.
Transformer:  "fluent" ↔ "France" ← direct connection. No forgetting.

LSTM was an improvement — it added a “memory cell” with gates that could decide what to remember and what to forget.

But LSTMs still:

• Processed tokens one at a time (sequential → slow)
• Had a limited memory (could remember ~100-200 tokens, not 100K+)
• Couldn’t parallelize on GPUs (needed the previous step’s output before computing the next)

In 2017, Google researchers published “Attention Is All You Need” (Vaswani et al.). The title was making a bold statement: you don’t need RNNs, LSTMs, or any sequential processing. Just self-attention. That’s enough.

The paper showed that by processing all tokens in parallel and letting each token directly attend to every other token, you could:

1. Train 8x faster than the best RNN models
2. Achieve better accuracy on translation tasks
3. Scale to massive datasets — the architecture could leverage the entire internet

The AI world pivoted overnight. Today, every major AI system — GPT, Claude, Gemini, BERT, LLaMA, Mistral — is built on the Transformer.

Imagine a committee trying to understand a complex document. Each member represents one word.

Before Transformers (RNN style):

• Members pass notes in a chain, one person at a time
• Person 1 reads Word 1, writes a summary, passes to Person 2
• Person 2 reads their word + Person 1’s summary, writes a new summary, passes to Person 3
• By Person 20, the original summary is barely recognizable
• Information gets lost. Mistakes amplify.

With Transformers:

• All members sit around a table with the entire document
• Each person can talk directly to any other person
• Word “it” turns to Word “animal” and asks: “Are you what I’m referring to?”
• Word “animal” responds: “Yes, I’m 95% likely to be your reference.”
• Everyone gets all the context they need, instantly

flowchart TD
    subgraph RNN_WAY["Old Way: Pass Notes in a Chain"]
        P1["Person 1\n('The')"] --> NOTES1["✉️ note"]
        NOTES1 --> P2["Person 2\n('animal')"]
        P2 --> NOTES2["✉️ blurry note"]
        NOTES2 --> P3["Person 3\n('didn't')"]
        P3 --> NOTES3["✉️ fuzzier note"]
        NOTES3 --> P11["Person 11\n('tired')"]
        P11 --> NOTES11["✉️ barely readable"]
    end

    subgraph TF_WAY["New Way: Everyone Talks at Once"]
        TABLE["Round Table"]
        TABLE --- T1["'The'"]
        TABLE --- T2["'animal' ← ┐"]
        TABLE --- T3["'didn't'   │"]
        TABLE --- T4["'cross'    │ it asks:"]
        TABLE --- T5["'the'      │ 'who am I?'"]
        TABLE --- T6["'street'   │"]
        TABLE --- T7["'because'  │"]
        TABLE --- T8["'it' ──────┘"]
        TABLE --- T9["'was'"]
        TABLE --- T10["'too'"]
        TABLE --- T11["'tired'"]
    end

    style RNN_WAY fill:#ef4444,color:#fff
    style TF_WAY fill:#22c55e,color:#fff

Think of self-attention as a spotlight that each word holds.

• Each word shines a spotlight on every other word
• The brightness of the spotlight = how relevant that word is
• “It” shines a bright spotlight on “animal” (very relevant)
• “It” shines a dim spotlight on “the” (not very relevant)
• The spotlight brightness is learned from data — the model learns what to pay attention to

At the highest level, a Transformer:

1. Takes in a sequence of token IDs (numbers representing words/subwords)
2. Converts each token ID to a vector (a list of numbers — the embedding)
3. Adds position information so it knows which word is where
4. Passes these through self-attention layers — each token looks at all others
5. Passes through feed-forward layers — each token thinks about what it learned
6. Repeats steps 4-5 many times (12x for small models, 96x for large ones)
7. Outputs a new vector for each token that contains context-aware meaning

flowchart TD
    TOKENS["Token IDs\n[1024, 8453, 291, 5792]"] --> EMBED["Embedding\n(Each ID → vector of numbers)"]
    EMBED --> POS["+ Position Information\n(So model knows word order)"]
    POS --> ATTN1["Self-Attention Layer 1\n(Each word looks at all others)"]
    ATTN1 --> FF1["Feed-Forward Layer 1\n(Process what was learned)"]
    FF1 --> ATTN2["Self-Attention Layer 2\n(Deeper connections)"]
    ATTN2 --> FF2["Feed-Forward Layer 2"]
    FF2 --> DOT["...Repeat many times..."]
    DOT --> OUT["Final Vectors\n(Context-aware meaning for each token)"]

    style TOKENS fill:#3b82f6,color:#fff
    style EMBED fill:#8b5cf6,color:#fff
    style POS fill:#f59e0b,color:#fff
    style ATTN1 fill:#ef4444,color:#fff
    style ATTN2 fill:#ef4444,color:#fff
    style OUT fill:#22c55e,color:#fff

The magic is in step 4: self-attention. That’s where each word gets enriched with context from every other word. We’ll explore this in detail in the next document.

The 2017 paper made three revolutionary claims:

1. You don’t need recurrence — No RNNs, no hidden state passing. Pure attention.
2. Parallel is better — Processing all words simultaneously is not just faster, it’s better at capturing long-range dependencies.
3. Attention can replace everything — Self-attention + feed-forward layers, stacked N times, is all you need.

flowchart TD
    OLD["Before 2017\nState of the art:\nLSTMs + Attention\n(Attention was extra)"]
    OLD --> PAPER["'Attention Is All You Need'\n(Vaswani et al., 2017)"]
    PAPER --> CLAIM["Claim: You don't need\nRNNs or convolutions.\nJust attention."]
    CLAIM --> PROOF["Proof: Best translation\never + 8x faster training"]
    PROOF --> IMPACT["The field adopted\nTransformers within\na year"]

    style OLD fill:#ef4444,color:#fff
    style PAPER fill:#f59e0b,color:#fff
    style CLAIM fill:#3b82f6,color:#fff
    style PROOF fill:#22c55e,color:#fff
    style IMPACT fill:#22c55e,color:#fff

The results were so dramatically better — and training was so much faster — that within a year, every NLP conference paper used Transformers. By 2020, GPT-3 showed that scaling Transformers to 175 billion parameters produced emergent abilities no one predicted. The rest is history.

# You don't need to train Transformers. Use pretrained ones.
# Install: pip install transformers torch

from transformers import pipeline

# Load a pretrained Transformer model
# This is a distilled BERT (encoder-only Transformer)
classifier = pipeline(
    "sentiment-analysis",
    model="distilbert-base-uncased-finetuned-sst-2-english"
)

# The Transformer processes ALL tokens at once
result = classifier("I absolutely loved this movie, it was fantastic!")
print(result)
# [{'label': 'POSITIVE', 'score': 0.998}]

# Under the hood, here's what happened:
# 1. Tokenizer: "I" "absolutely" "loved" "this" "movie" "it" "was" "fantastic" "!"
# 2. All tokens → numbers → embeddings
# 3. Self-attention: "loved" ↔ "fantastic" (strong connection)
# 4. Output: the [CLS] token's vector → POSITIVE prediction

// Run a Transformer directly in your browser!
// Install: npm install @xenova/transformers

import { pipeline } from '@xenova/transformers';

async function runTransformer() {
  // Load the model (downloads once, caches locally)
  const generator = await pipeline(
    'text-generation',
    'Xenova/gpt2'  // GPT-2 is a small decoder-only Transformer
  );

  // The Transformer generates this one token at a time,
  // but each step it looks at ALL previous tokens via self-attention
  const result = await generator(
    'The Transformer is a type of neural network that',
    { max_new_tokens: 30, temperature: 0.7 }
  );

  console.log(result[0].generated_text);
  // The Transformer is a type of neural network that
  // processes all words in a sentence simultaneously...
}

runTransformer();

1. Never train a Transformer from scratch — Always use pretrained models from Hugging Face; training from scratch requires millions of dollars in compute
2. Choose the right variant for your task — Encoder-only (BERT) for understanding; decoder-only (GPT) for generation; encoder-decoder (T5) for translation/summarization
3. Start small — DistilBERT is 40% smaller than BERT but retains 97% of performance; use smaller models for prototyping
4. Understand that bigger is NOT always better for your use case — A fine-tuned 7B model can outperform GPT-4 on a narrow domain task
5. Use pipeline() for quick experiments — Hugging Face’s pipeline API handles tokenization, inference, and decoding automatically

Q: What is a Transformer and why was it invented?

A Transformer is a neural network architecture that processes all tokens in a sequence simultaneously using self-attention. It was invented in 2017 by Google researchers (Vaswani et al.) to replace slower sequential models like RNNs and LSTMs. The Transformer’s parallel processing makes it much faster to train on GPUs and better at handling long-range dependencies in text.

Q: How is a Transformer different from an RNN?

An RNN processes words one at a time, left to right, passing a compressed hidden state forward. This is slow (can’t parallelize) and forgets information over long sequences. A Transformer processes all words simultaneously — each word can directly look at every other word through self-attention. This makes Transformers much faster to train and much better at handling long-range connections.

Q: What problem did the “Attention Is All You Need” paper solve?

The paper solved two problems: (1) sequential processing — RNNs/LSTMs processed tokens one at a time, making them slow and unable to leverage GPU parallelization, and (2) long-range dependency — RNNs forgot early information in long sequences. The Transformer’s self-attention mechanism lets every token directly attend to every other token regardless of distance, and processes all tokens simultaneously for massive parallelism. The paper demonstrated state-of-the-art translation results while training 8x faster.

Q: Explain self-attention in simple terms.

Self-attention is a mechanism that lets each word in a sentence look at every other word and decide which ones are important. It works like a spotlight: each word shines a bright spotlight on words that are relevant to it, and a dim spotlight on irrelevant words. The word “it” in “The animal was tired so it rested” would shine a bright spotlight on “animal” (that’s what “it” refers to) and “tired” (that’s why it rested). The model learns these spotlight patterns from data — no human tells it what to pay attention to.

Q: Why were RNNs and LSTMs fundamentally limited in a way that Transformers are not?

RNNs and LSTMs have a fundamental architectural limitation: they process tokens sequentially, and each step depends on the previous step’s hidden state. This means: (1) training cannot be parallelized across GPUs — the next step literally cannot be computed until the previous one finishes; (2) information must pass through a chain of compressed hidden states, creating a bottleneck — early information is compressed and distorted by the time it reaches later tokens; (3) the vanishing gradient problem makes it difficult for gradients to propagate back more than ~100 steps during training. Transformers solve all three by: (1) processing all tokens in parallel (no sequential dependency), (2) letting every token directly attend to every other token (no compression bottleneck), (3) having direct gradient paths between any two tokens (no vanishing gradients over distance).

Q: How does the Transformer’s parallel processing actually work when generating text?

During training, the Transformer processes all tokens in parallel using a technique called “masked self-attention” (for decoder-only models like GPT). A mask prevents each token from attending to future tokens — token 5 can see tokens 1-4 but not 6-10. This allows the model to process all positions simultaneously during training: it predicts token 2 given token 1, token 3 given tokens 1-2, etc., all in one forward pass. During inference (generation), the process must be sequential because you don’t yet know the future tokens. However, the key advantage is during training — the model sees billions of examples in parallel, which is why training on internet-scale data is feasible. The training parallelism is the secret: it lets the model learn patterns from trillions of examples in weeks rather than centuries.

Previous: 04 — Context Window

Next: 06 — Self-Attention

Related Topics:

• Self-Attention
• Introduction to Transformers
• What is an LLM?

Practice Questions:

1. Explain why RNNs struggle with long sentences but Transformers do not.
2. Draw a diagram comparing sequential processing (RNN) with parallel processing (Transformer).
3. What does the paper title “Attention Is All You Need” actually mean?
4. Why is the Transformer able to train on much more data than RNNs?
5. Name three AI systems you use that are built on the Transformer architecture.

Further Reading:

• Attention Is All You Need (original paper) — the paper that started it all
• The Illustrated Transformer — Jay Alammar — the best visual explanation on the web
• 3Blue1Brown — Transformers explained visually — beautifully animated intuition
• Andrej Karpathy — Let’s build GPT from scratch — build one yourself in 2 hours