How Transformer Self-Attention Actually Works in LLMs

If you are working with Large Language Models (LLMs) today—whether tuning open weights like Llama 3 or building retrieval-augmented generation (RAG) pipelines—understanding self-attention is essential.

When Google published Attention Is All You Need in 2017, the goal was practical: fix the throughput and memory problems of Recurrent Neural Networks (RNNs) and LSTMs. Here is a straight-to-the-point explanation of how the Transformer self-attention block works under the hood.

1. Why RNNs Had to Go

Before Transformers, sequence processing relied on RNNs. An RNN steps through text token by token: to process word 10, it must first run through words 1 through 9 in order. This caused two major problems:

Zero GPU Parallelization: You couldn't process an entire 4,000-word prompt at once. Every token depended on the previous hidden state, so GPUs sat idle waiting for sequential iterations.
Information Loss Over Distance: By token 500, early context got washed out by repeated matrix multiplications (vanishing gradients).

2. Query, Key, and Value Vectors Explained

Instead of stepping sequentially, a Transformer projects every input token into three separate vector representations by multiplying the token embedding by learned weight matrices (\(W_Q\), \(W_K\), and \(W_V\)):

Query Vector (\(Q\)): Represents what current word is looking for context (e.g. "What noun does 'it' refer to?").
Key Vector (\(K\)): Acts like a index tag representing what info this token offers to other words.
Value Vector (\(V\)): Contains the actual feature values that get combined into the output representation.

3. The Scaled Dot-Product Attention Equation

To compute how much attention word A should pay to word B, we take the dot product of Query A with Key B. We do this for all word combinations in parallel using matrix multiplication:

Attention(Q, K, V) = Softmax( (Q * K^T) / sqrt(d_k) ) * V

Why do we divide by \(\sqrt{d_k}\)? If vector dimensions \(d_k\) are large (e.g., 128 dimensions), dot products grow large in magnitude. Large inputs push the Softmax function into extreme values near 0 or 1, causing vanishing gradients during backpropagation. Scaling by \(\sqrt{d_k}\) keeps gradients stable.

4. PyTorch Implementation

Here is a concise, working implementation of scaled dot-product attention in PyTorch:

import torch
import torch.nn as nn
import math

class ScaledDotProductAttention(nn.Module):
    def __init__(self, d_k):
        super().__init__()
        self.d_k = d_k

    def forward(self, Q, K, V, mask=None):
        # Q, K, V shapes: [batch_size, num_heads, seq_len, d_k]
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)
        
        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)
            
        attn_weights = torch.softmax(scores, dim=-1)
        output = torch.matmul(attn_weights, V)
        return output, attn_weights
        

5. Multi-Head Attention

Single attention heads often average out different types of relationships. Multi-head attention splits \(Q, K, V\) vectors across 8, 16, or 32 heads. One head might focus on subject-verb pairing, while another tracks long-range modifier dependencies. The head outputs are concatenated and linearly projected back to the original model width.

6. Positional Encodings

Because matrix multiplication computes all token attention pairs simultaneously, the model naturally has no idea which word comes first or last. To give the model token order, we add Positional Encodings (sinusoidal frequencies or rotary position embeddings / RoPE) directly to the input token vectors before entering the attention block.