LLM Engineering: First Principles¶

Goal: Understand the critical engineering components of the LLM pipeline by building Byte-Pair Encoding (BPE), Low-Rank Adaptation (LoRA), and Direct Preference Optimization (DPO) from scratch.

Prerequisites: 13 Neural Networks, 16 Transformer

Reference: theory.md

In [1]:

import random

import matplotlib.pyplot as plt
import numpy as np

%load_ext autoreload
%autoreload 2

SEED = 42
random.seed(SEED)
rng = np.random.default_rng(SEED)

plt.rcParams["figure.dpi"] = 90

from collections import Counter

import random import matplotlib.pyplot as plt import numpy as np %load_ext autoreload %autoreload 2 SEED = 42 random.seed(SEED) rng = np.random.default_rng(SEED) plt.rcParams["figure.dpi"] = 90 from collections import Counter

1. Problem Setup — WHY¶

Training an LLM involves three major hurdles:

1. Tokenization: How do we feed raw text into a neural network without having an infinite vocabulary or excessively long sequences?
2. Fine-Tuning: How do we adapt a 7-billion parameter model to a new task when we can't afford the VRAM to store optimizer states for all 7 billion parameters?
3. Alignment: How do we penalize bad behavior and encourage helpful responses without manually writing reward rules?

2. Mathematical Core — WHAT¶

BPE Merge Objective: $$ t_{new} = \arg\max_{(t_a, t_b)} \text{count}(t_a, t_b) $$

LoRA Forward Pass: $$ W = W_0 + \frac{\alpha}{r} BA $$ $$ h = x W_0 + \frac{\alpha}{r} x BA $$ where $W_0 \in \mathbb{R}^{d \times k}$ is frozen, $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and $r \ll \min(d, k)$.

DPO Loss Formulation: $$ \hat{r}_\theta(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)} $$ $$ \mathcal{L}_{DPO} = - \log \sigma \left( \hat{r}_\theta(x, y_w) - \hat{r}_\theta(x, y_l) \right) $$

3. Solution Method — HOW¶

1. BPE: Iterate through the text, find the most frequent pair of adjacent symbols, and merge them into a single new symbol until the target vocabulary size is reached.
2. LoRA: Wrap standard linear layers. Store the frozen weight $W_0$. Maintain small trainable matrices $A$ and $B$. Initialize $A$ randomly and $B$ to zeros so the initial output perfectly matches the pre-trained base model.
3. DPO: Compute the log-probabilities of the winning and losing sequences under both the policy model and the reference model. Compute the implicit reward difference, then apply the negative log sigmoid loss.

4. Implementation — BUILD¶

4.1. BPE Tokenizer (Word-Level)¶

We implement the classic word-level BPE of Sennrich et al. (2016), exactly as derived in theory.md: split every word into characters plus an end-of-word marker </w>, count adjacent symbol pairs across the corpus, merge the most frequent pair into a new symbol, and repeat until the vocabulary reaches the target size. Ties are broken in favor of the pair encountered first in corpus order — the same deterministic rule used by the reference implementation in ml_first_principles.

In [2]:

END_OF_WORD = "</w>"


def to_symbol_corpus(text):
    """Split each word into characters plus the end-of-word marker."""
    return [list(word) + [END_OF_WORD] for word in text.split()]


def count_pairs(corpus):
    """Count adjacent symbol pairs across all words (weighted by occurrence)."""
    pairs = Counter()
    for word in corpus:
        for pair in zip(word, word[1:]):
            pairs[pair] += 1
    return pairs


def apply_merge(word, pair, replacement):
    """Replace every occurrence of `pair` in one word by the merged symbol."""
    merged, i = [], 0
    while i < len(word):
        if i < len(word) - 1 and (word[i], word[i + 1]) == pair:
            merged.append(replacement)
            i += 2
        else:
            merged.append(word[i])
            i += 1
    return merged


def learn_bpe(text, target_vocab_size):
    """Learn ordered merge rules until the vocabulary reaches the target size."""
    corpus = to_symbol_corpus(text)
    vocab = {symbol for word in corpus for symbol in word}
    merges = []
    while len(vocab) < target_vocab_size:
        pairs = count_pairs(corpus)
        if not pairs:
            break
        best = pairs.most_common(1)[0][0]  # ties: first-encountered pair wins
        merges.append(best)
        vocab.add("".join(best))
        corpus = [apply_merge(word, best, "".join(best)) for word in corpus]
    return merges, sorted(vocab)


def bpe_encode(text, merges):
    """Tokenize new text by replaying the merges in learned order."""
    tokens = []
    for word in text.split():
        symbols = list(word) + [END_OF_WORD]
        for pair in merges:
            symbols = apply_merge(symbols, pair, "".join(pair))
        tokens.extend(symbols)
    return tokens


def bpe_decode(tokens):
    """Concatenate tokens and turn end-of-word markers back into spaces."""
    return "".join(tokens).replace(END_OF_WORD, " ").strip()

END_OF_WORD = "</w>" def to_symbol_corpus(text): """Split each word into characters plus the end-of-word marker.""" return [list(word) + [END_OF_WORD] for word in text.split()] def count_pairs(corpus): """Count adjacent symbol pairs across all words (weighted by occurrence).""" pairs = Counter() for word in corpus: for pair in zip(word, word[1:]): pairs[pair] += 1 return pairs def apply_merge(word, pair, replacement): """Replace every occurrence of `pair` in one word by the merged symbol.""" merged, i = [], 0 while i < len(word): if i < len(word) - 1 and (word[i], word[i + 1]) == pair: merged.append(replacement) i += 2 else: merged.append(word[i]) i += 1 return merged def learn_bpe(text, target_vocab_size): """Learn ordered merge rules until the vocabulary reaches the target size.""" corpus = to_symbol_corpus(text) vocab = {symbol for word in corpus for symbol in word} merges = [] while len(vocab) < target_vocab_size: pairs = count_pairs(corpus) if not pairs: break best = pairs.most_common(1)[0][0] # ties: first-encountered pair wins merges.append(best) vocab.add("".join(best)) corpus = [apply_merge(word, best, "".join(best)) for word in corpus] return merges, sorted(vocab) def bpe_encode(text, merges): """Tokenize new text by replaying the merges in learned order.""" tokens = [] for word in text.split(): symbols = list(word) + [END_OF_WORD] for pair in merges: symbols = apply_merge(symbols, pair, "".join(pair)) tokens.extend(symbols) return tokens def bpe_decode(tokens): """Concatenate tokens and turn end-of-word markers back into spaces.""" return "".join(tokens).replace(END_OF_WORD, " ").strip()

Walkthrough — the worked example from theory.md. The corpus contains "low" ×5, "lower" ×2, "newest" ×6, "widest" ×3, so the initial vocabulary is the 10 distinct characters plus </w> (11 symbols). Before merging anything, inspect the pair counts of round 1: (e, s) tops the table with $6 + 3 = 9$ occurrences (6 from "newest", 3 from "widest"), tied with (s, t) and (t, </w>) — the first-encountered pair wins.

In [3]:

theory_text = "low " * 5 + "lower " * 2 + "newest " * 6 + "widest " * 3
theory_corpus = to_symbol_corpus(theory_text)
base_vocab = sorted({symbol for word in theory_corpus for symbol in word})
print(f"Base vocabulary ({len(base_vocab)} symbols): {base_vocab}")

round1_counts = count_pairs(theory_corpus)
print("\nRound-1 pair counts (top 8):")
for pair, count in round1_counts.most_common(8):
    print(f"  {pair}: {count}")

first_pair = round1_counts.most_common(1)[0][0]
print(f"\nFirst merge: {first_pair} -> '{''.join(first_pair)}'")
assert first_pair == ("e", "s")

theory_text = "low " * 5 + "lower " * 2 + "newest " * 6 + "widest " * 3 theory_corpus = to_symbol_corpus(theory_text) base_vocab = sorted({symbol for word in theory_corpus for symbol in word}) print(f"Base vocabulary ({len(base_vocab)} symbols): {base_vocab}") round1_counts = count_pairs(theory_corpus) print("\nRound-1 pair counts (top 8):") for pair, count in round1_counts.most_common(8): print(f" {pair}: {count}") first_pair = round1_counts.most_common(1)[0][0] print(f"\nFirst merge: {first_pair} -> '{''.join(first_pair)}'") assert first_pair == ("e", "s")

Base vocabulary (11 symbols): ['</w>', 'd', 'e', 'i', 'l', 'n', 'o', 'r', 's', 't', 'w']

Round-1 pair counts (top 8):
  ('e', 's'): 9
  ('s', 't'): 9
  ('t', '</w>'): 9
  ('w', 'e'): 8
  ('l', 'o'): 7
  ('o', 'w'): 7
  ('n', 'e'): 6
  ('e', 'w'): 6

First merge: ('e', 's') -> 'es'

In [4]:

# Four merges reproduce the four iterations worked out in theory.md.
merges, vocab = learn_bpe(theory_text, target_vocab_size=len(base_vocab) + 4)
print("Merges in learned order:")
for step, pair in enumerate(merges, start=1):
    print(f"  {step}: {pair} -> '{''.join(pair)}'")

assert merges[:2] == [("e", "s"), ("es", "t")]
assert merges == [("e", "s"), ("es", "t"), ("est", "</w>"), ("l", "o")]

tokens = bpe_encode("newest widest", merges)
print("\nEncode 'newest widest':", tokens)
assert bpe_decode(tokens) == "newest widest"

# Four merges reproduce the four iterations worked out in theory.md. merges, vocab = learn_bpe(theory_text, target_vocab_size=len(base_vocab) + 4) print("Merges in learned order:") for step, pair in enumerate(merges, start=1): print(f" {step}: {pair} -> '{''.join(pair)}'") assert merges[:2] == [("e", "s"), ("es", "t")] assert merges == [("e", "s"), ("es", "t"), ("est", "</w>"), ("l", "o")] tokens = bpe_encode("newest widest", merges) print("\nEncode 'newest widest':", tokens) assert bpe_decode(tokens) == "newest widest"

Merges in learned order:
  1: ('e', 's') -> 'es'
  2: ('es', 't') -> 'est'
  3: ('est', '</w>') -> 'est</w>'
  4: ('l', 'o') -> 'lo'

Encode 'newest widest': ['n', 'e', 'w', 'est</w>', 'w', 'i', 'd', 'est</w>']

Variant — byte-level BPE. Modern GPT-style tokenizers (GPT-2, tiktoken) run the same greedy merge loop over the 256 raw UTF-8 bytes of the text instead of word characters: every string is a byte sequence, so no character can ever be out of vocabulary, and no </w> marker is needed because word boundaries are handled by a regex pre-tokenizer instead. Only the base alphabet changes; the merge algorithm above is identical.

4.2. LoRA Linear Layer¶

In [5]:

class LoRALinear:
    def __init__(self, in_features, out_features, r=8, alpha=16):
        self.in_features = in_features
        self.out_features = out_features
        self.r = r
        self.alpha = alpha
        self.scaling = alpha / r
        
        # Base frozen weights (simulated)
        self.W0 = rng.normal(0, 0.02, (in_features, out_features))
        
        # LoRA weights: A is Kaiming init, B is zeros
        self.A = rng.normal(0, np.sqrt(2.0 / in_features), (r, out_features))
        self.B = np.zeros((in_features, r))
        
    def forward(self, x):
        # Standard path
        base_out = x @ self.W0
        # LoRA path: x * B * A
        lora_out = (x @ self.B) @ self.A
        return base_out + self.scaling * lora_out

class LoRALinear: def __init__(self, in_features, out_features, r=8, alpha=16): self.in_features = in_features self.out_features = out_features self.r = r self.alpha = alpha self.scaling = alpha / r # Base frozen weights (simulated) self.W0 = rng.normal(0, 0.02, (in_features, out_features)) # LoRA weights: A is Kaiming init, B is zeros self.A = rng.normal(0, np.sqrt(2.0 / in_features), (r, out_features)) self.B = np.zeros((in_features, r)) def forward(self, x): # Standard path base_out = x @ self.W0 # LoRA path: x * B * A lora_out = (x @ self.B) @ self.A return base_out + self.scaling * lora_out

4.3. DPO Loss¶

In [6]:

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def dpo_loss(pi_theta_w, pi_ref_w, pi_theta_l, pi_ref_l, beta=0.1):
    """
    pi_theta_w: log probabilities of winning responses under policy model
    pi_ref_w: log probabilities of winning responses under reference model
    pi_theta_l: log probabilities of losing responses under policy model
    pi_ref_l: log probabilities of losing responses under reference model
    """
    # Implicit rewards
    r_theta_w = beta * (pi_theta_w - pi_ref_w)
    r_theta_l = beta * (pi_theta_l - pi_ref_l)
    
    # Margin
    margin = r_theta_w - r_theta_l
    
    # Loss is negative log sigmoid of the margin
    loss = -np.log(sigmoid(margin) + 1e-8)
    return np.mean(loss)

def sigmoid(x): return 1 / (1 + np.exp(-x)) def dpo_loss(pi_theta_w, pi_ref_w, pi_theta_l, pi_ref_l, beta=0.1): """ pi_theta_w: log probabilities of winning responses under policy model pi_ref_w: log probabilities of winning responses under reference model pi_theta_l: log probabilities of losing responses under policy model pi_ref_l: log probabilities of losing responses under reference model """ # Implicit rewards r_theta_w = beta * (pi_theta_w - pi_ref_w) r_theta_l = beta * (pi_theta_l - pi_ref_l) # Margin margin = r_theta_w - r_theta_l # Loss is negative log sigmoid of the margin loss = -np.log(sigmoid(margin) + 1e-8) return np.mean(loss)

5. Library Comparison¶

No external library pins down this exact classroom algorithm, so the unit-tested package implementation ml_first_principles.llm_models.BPETokenizer serves as the reference.

In [7]:

from ml_first_principles.llm_models import BPETokenizer as RefBPETokenizer
from ml_first_principles.llm_models import LoRALinear as RefLoRALinear
from ml_first_principles.llm_models import dpo_loss as ref_dpo_loss

target = len(base_vocab) + 8
nb_merges, nb_vocab = learn_bpe(theory_text, target_vocab_size=target)

ref_tokenizer = RefBPETokenizer(target_vocab_size=target)
ref_tokenizer.fit(theory_text)

print("Notebook merges:", nb_merges)
print("Package merges: ", ref_tokenizer.merges)
assert nb_merges == ref_tokenizer.merges
assert nb_vocab == ref_tokenizer.vocab

sample = "lowest newest"
nb_tokens = bpe_encode(sample, nb_merges)
ref_tokens = ref_tokenizer.encode(sample)
print(f"Tokens for '{sample}':", ref_tokens)
assert nb_tokens == ref_tokens
assert bpe_decode(nb_tokens) == sample
assert ref_tokenizer.decode(ref_tokens) == sample
print("Merges, tokens, and the encode/decode roundtrip all match the package.")

from ml_first_principles.llm_models import BPETokenizer as RefBPETokenizer from ml_first_principles.llm_models import LoRALinear as RefLoRALinear from ml_first_principles.llm_models import dpo_loss as ref_dpo_loss target = len(base_vocab) + 8 nb_merges, nb_vocab = learn_bpe(theory_text, target_vocab_size=target) ref_tokenizer = RefBPETokenizer(target_vocab_size=target) ref_tokenizer.fit(theory_text) print("Notebook merges:", nb_merges) print("Package merges: ", ref_tokenizer.merges) assert nb_merges == ref_tokenizer.merges assert nb_vocab == ref_tokenizer.vocab sample = "lowest newest" nb_tokens = bpe_encode(sample, nb_merges) ref_tokens = ref_tokenizer.encode(sample) print(f"Tokens for '{sample}':", ref_tokens) assert nb_tokens == ref_tokens assert bpe_decode(nb_tokens) == sample assert ref_tokenizer.decode(ref_tokens) == sample print("Merges, tokens, and the encode/decode roundtrip all match the package.")

Notebook merges: [('e', 's'), ('es', 't'), ('est', '</w>'), ('l', 'o'), ('lo', 'w'), ('n', 'e'), ('ne', 'w'), ('new', 'est</w>')]
Package merges:  [('e', 's'), ('es', 't'), ('est', '</w>'), ('l', 'o'), ('lo', 'w'), ('n', 'e'), ('ne', 'w'), ('new', 'est</w>')]
Tokens for 'lowest newest': ['low', 'est</w>', 'newest</w>']
Merges, tokens, and the encode/decode roundtrip all match the package.

6. Experiments and Failures — VERIFY¶

6.1. BPE on Real Text¶

In [8]:

real_text = """
the quick brown fox jumps over the lazy dog.
the dog was not actually lazy, just resting.
a resting dog is a good dog.
"""
real_base = {symbol for word in to_symbol_corpus(real_text) for symbol in word}
real_merges, real_vocab = learn_bpe(real_text, target_vocab_size=len(real_base) + 15)

print(f"Base symbols: {len(real_base)}, merges learned: {len(real_merges)}")
print("First 5 merges:")
for step, pair in enumerate(real_merges[:5], start=1):
    print(f"  {step}: {pair} -> '{''.join(pair)}'")

# The frequent word collapses into a whole-word token, the rarer one splits.
for word in ["dog", "resting"]:
    print(f"\nTokens for '{word}':", bpe_encode(word, real_merges))

real_text = """ the quick brown fox jumps over the lazy dog. the dog was not actually lazy, just resting. a resting dog is a good dog. """ real_base = {symbol for word in to_symbol_corpus(real_text) for symbol in word} real_merges, real_vocab = learn_bpe(real_text, target_vocab_size=len(real_base) + 15) print(f"Base symbols: {len(real_base)}, merges learned: {len(real_merges)}") print("First 5 merges:") for step, pair in enumerate(real_merges[:5], start=1): print(f" {step}: {pair} -> '{''.join(pair)}'") # The frequent word collapses into a whole-word token, the rarer one splits. for word in ["dog", "resting"]: print(f"\nTokens for '{word}':", bpe_encode(word, real_merges))

Base symbols: 29, merges learned: 15
First 5 merges:
  1: ('d', 'o') -> 'do'
  2: ('do', 'g') -> 'dog'
  3: ('t', 'h') -> 'th'
  4: ('th', 'e') -> 'the'
  5: ('the', '</w>') -> 'the</w>'

Tokens for 'dog': ['dog</w>']

Tokens for 'resting': ['re', 'st', 'i', 'n', 'g', '</w>']

6.2. LoRA Initialization and Parameter Savings¶

In [9]:

dim_in, dim_out, r = 4096, 4096, 8
lora = LoRALinear(dim_in, dim_out, r)

x = rng.normal(0, 1, (1, dim_in))
base_out = x @ lora.W0
lora_out = lora.forward(x)
assert np.allclose(base_out, lora_out, atol=1e-12)
print("At initialization the LoRA output equals the frozen base output.")

full_params = dim_in * dim_out
lora_params = (dim_in * r) + (r * dim_out)
print(f"Full parameters: {full_params:,}")
print(f"LoRA parameters: {lora_params:,}")
print(f"Reduction: {100 * (1 - lora_params/full_params):.3f}%")

# Comparison with the package implementation: exact base-model identity at
# initialization (lora_B = 0), divergence as soon as lora_B moves.
ref_lora = RefLoRALinear(in_features=64, out_features=48, r=4, random_state=SEED)
x_small = rng.normal(0, 1, (5, 64))
assert np.array_equal(ref_lora.forward(x_small), x_small @ ref_lora.W0)

ref_lora.lora_B += 0.5
assert not np.allclose(ref_lora.forward(x_small), x_small @ ref_lora.W0, atol=1e-8)
print("Package LoRALinear: exact match at init, diverges once lora_B != 0.")

dim_in, dim_out, r = 4096, 4096, 8 lora = LoRALinear(dim_in, dim_out, r) x = rng.normal(0, 1, (1, dim_in)) base_out = x @ lora.W0 lora_out = lora.forward(x) assert np.allclose(base_out, lora_out, atol=1e-12) print("At initialization the LoRA output equals the frozen base output.") full_params = dim_in * dim_out lora_params = (dim_in * r) + (r * dim_out) print(f"Full parameters: {full_params:,}") print(f"LoRA parameters: {lora_params:,}") print(f"Reduction: {100 * (1 - lora_params/full_params):.3f}%") # Comparison with the package implementation: exact base-model identity at # initialization (lora_B = 0), divergence as soon as lora_B moves. ref_lora = RefLoRALinear(in_features=64, out_features=48, r=4, random_state=SEED) x_small = rng.normal(0, 1, (5, 64)) assert np.array_equal(ref_lora.forward(x_small), x_small @ ref_lora.W0) ref_lora.lora_B += 0.5 assert not np.allclose(ref_lora.forward(x_small), x_small @ ref_lora.W0, atol=1e-8) print("Package LoRALinear: exact match at init, diverges once lora_B != 0.")

At initialization the LoRA output equals the frozen base output.
Full parameters: 16,777,216
LoRA parameters: 65,536
Reduction: 99.609%
Package LoRALinear: exact match at init, diverges once lora_B != 0.

6.3. DPO Loss Landscape¶

The DPO loss is $-\log \sigma(\beta \Delta)$ where $\Delta = \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}$ is the policy-vs-reference log-ratio margin. Every curve passes through $\log 2$ at $\Delta = 0$, and $\beta$ controls how sharply a negative margin is punished. We evaluate the loss in log space via $\operatorname{softplus}(-\beta\Delta) = \operatorname{logaddexp}(0, -\beta\Delta)$, which stays finite even for extreme log-probabilities where a naive $\log \sigma$ would overflow.

In [10]:

margins = np.linspace(-8, 8, 401)  # log-ratio margin Delta
betas = [0.1, 1.0, 5.0]

plt.figure(figsize=(8, 5))
for beta in betas:
    losses = np.logaddexp(0.0, -beta * margins)  # -log sigmoid(beta * Delta), log-space
    plt.plot(margins, losses, label=rf"$\beta = {beta}$")
plt.axhline(np.log(2), color="gray", linestyle=":", linewidth=1, label=r"$\log 2$")
plt.xlabel(r"Policy-vs-reference log-ratio margin $\Delta$")
plt.ylabel(r"DPO loss $-\log \sigma(\beta \Delta)$")
plt.title(r"DPO loss vs. reward margin for several $\beta$")
plt.grid(True)
plt.legend()
plt.show()

# At zero margin the loss is exactly log 2 for every beta.
for beta in betas:
    assert np.isclose(np.logaddexp(0.0, -beta * 0.0), np.log(2), atol=1e-12)

# Our from-scratch expression agrees with the package implementation.
pol_w, pol_l, ref_w, ref_l = rng.normal(0.0, 2.0, size=(4, 6))
ours = dpo_loss(pol_w, ref_w, pol_l, ref_l, beta=0.1)
theirs = ref_dpo_loss(pol_w, pol_l, ref_w, ref_l, beta=0.1)
print(f"Notebook DPO loss: {ours:.8f}  |  Package DPO loss: {theirs:.8f}")
assert np.isclose(ours, theirs, atol=1e-6)

# The log-space package version stays finite even at extreme log-probabilities.
big = 100.0
extreme = ref_dpo_loss(
    np.array([-big]), np.array([big]), np.array([big]), np.array([-big]), beta=10.0
)
print(f"Package DPO loss at +/-100 logits with beta=10: {extreme:.1f} (finite)")
assert np.isfinite(extreme)

margins = np.linspace(-8, 8, 401) # log-ratio margin Delta betas = [0.1, 1.0, 5.0] plt.figure(figsize=(8, 5)) for beta in betas: losses = np.logaddexp(0.0, -beta * margins) # -log sigmoid(beta * Delta), log-space plt.plot(margins, losses, label=rf"$\beta = {beta}$") plt.axhline(np.log(2), color="gray", linestyle=":", linewidth=1, label=r"$\log 2$") plt.xlabel(r"Policy-vs-reference log-ratio margin $\Delta$") plt.ylabel(r"DPO loss $-\log \sigma(\beta \Delta)$") plt.title(r"DPO loss vs. reward margin for several $\beta$") plt.grid(True) plt.legend() plt.show() # At zero margin the loss is exactly log 2 for every beta. for beta in betas: assert np.isclose(np.logaddexp(0.0, -beta * 0.0), np.log(2), atol=1e-12) # Our from-scratch expression agrees with the package implementation. pol_w, pol_l, ref_w, ref_l = rng.normal(0.0, 2.0, size=(4, 6)) ours = dpo_loss(pol_w, ref_w, pol_l, ref_l, beta=0.1) theirs = ref_dpo_loss(pol_w, pol_l, ref_w, ref_l, beta=0.1) print(f"Notebook DPO loss: {ours:.8f} | Package DPO loss: {theirs:.8f}") assert np.isclose(ours, theirs, atol=1e-6) # The log-space package version stays finite even at extreme log-probabilities. big = 100.0 extreme = ref_dpo_loss( np.array([-big]), np.array([big]), np.array([big]), np.array([-big]), beta=10.0 ) print(f"Package DPO loss at +/-100 logits with beta=10: {extreme:.1f} (finite)") assert np.isfinite(extreme)

Notebook DPO loss: 0.64101959  |  Package DPO loss: 0.64101961
Package DPO loss at +/-100 logits with beta=10: 4000.0 (finite)

6.4. Failure Case: BPE Over-fragmentation¶

If the vocabulary size is too small, complex or unseen words are fragmented into single characters, losing semantic meaning entirely.

In [11]:

# Reuse the 19-symbol vocabulary learned on the theory corpus (section 5).
common_word, unseen_word = "newest", "widow"
common_tokens = bpe_encode(common_word, nb_merges)
unseen_tokens = bpe_encode(unseen_word, nb_merges)

print(f"In-corpus word '{common_word}': {common_tokens} ({len(common_tokens)} token)")
print(f"Unseen word    '{unseen_word}':  {unseen_tokens} ({len(unseen_tokens)} tokens)")
print("The frequent word owns a whole-word token; the unseen word shatters into characters.")

assert len(common_tokens) <= 2
assert len(unseen_tokens) >= 4

# Reuse the 19-symbol vocabulary learned on the theory corpus (section 5). common_word, unseen_word = "newest", "widow" common_tokens = bpe_encode(common_word, nb_merges) unseen_tokens = bpe_encode(unseen_word, nb_merges) print(f"In-corpus word '{common_word}': {common_tokens} ({len(common_tokens)} token)") print(f"Unseen word '{unseen_word}': {unseen_tokens} ({len(unseen_tokens)} tokens)") print("The frequent word owns a whole-word token; the unseen word shatters into characters.") assert len(common_tokens) <= 2 assert len(unseen_tokens) >= 4

In-corpus word 'newest': ['newest</w>'] (1 token)
Unseen word    'widow':  ['w', 'i', 'd', 'o', 'w', '</w>'] (6 tokens)
The frequent word owns a whole-word token; the unseen word shatters into characters.

7. Connections¶

• Preceding: Transformer architecture uses BPE token sequences. LoRA is applied strictly to Transformer Linear/Dense layers.
• Optimization: DPO translates a complex RL problem back into a mathematically sound cross-entropy-style supervised optimization problem, heavily relying on well-behaved gradients.