Qwen/Qwen2.5-0.5B · NVIDIA RTX A4000 · 4.0 min

Learn a carrier, swap it and steer it

Open in Colab

Declare an option-position variable and reuse one fitted model for donor swaps and requested answers.

%pip install -q "psi[notebooks] @ git+https://github.com/AdaptiveMotorControlLab/psi"

Choose the model and budget in this cell. Set SMOKE = True for a small CPU run.

# Edit these settings directly. SMOKE gives a small CPU check.
SMOKE = False
MODEL_ID = "Qwen/Qwen2.5-0.5B"
EPOCHS = 8
BATCH_SIZE = 4
MAX_LENGTH = 64
%matplotlib inline

import logging
import tempfile
import time
import warnings

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import torch
from IPython.display import display

import psi

# These settings are derived from the configuration cell above.
DEVICE = "cpu"
if not SMOKE and torch.cuda.is_available():
    DEVICE = "cuda"
DTYPE = "float32" if DEVICE == "cpu" else "float16"
EPOCHS = 1 if SMOKE else EPOCHS
BATCH_SIZE = 2 if SMOKE else BATCH_SIZE
CACHE = tempfile.mkdtemp(prefix="psi-demo-")
START = time.perf_counter()

torch.set_num_threads(4)
logging.getLogger("lightning.pytorch").setLevel(logging.ERROR)
warnings.formatwarning = (
    lambda message, category, *args, **kwargs: f"{category.__name__}: {message}\n"
)

# Colours and axis treatment follow the paper's figures.
PSI_COLOUR = "#A03BB8"
SECONDARY = "#1E8158"
REFERENCE = "#7593C6"
NO_EDIT = "#BFBFBC"
INK = "#0b0b0b"
INK_SOFT = "#4a4a47"

plt.rcParams.update({
    "font.family": "sans-serif",
    "font.size": 11,
    "axes.labelsize": 11,
    "axes.titlesize": 12,
    "axes.edgecolor": INK,
    "axes.labelcolor": INK,
    "axes.linewidth": 0.8,
    "xtick.color": INK_SOFT,
    "ytick.color": INK_SOFT,
    "xtick.labelsize": 10,
    "ytick.labelsize": 10,
    "legend.fontsize": 10,
    "legend.frameon": False,
    "figure.facecolor": "white",
    "axes.facecolor": "white",
    "savefig.facecolor": "white",
    "figure.dpi": 120,
    "savefig.dpi": 160,
    "pdf.fonttype": 42,
    "ps.fonttype": 42,
})

The preset retains each prompt’s own option labels. Slots("choices.label") studies the selected position. A copy asks the base to choose its own option in the donor’s preferred slot.

prompts, program = psi.presets.mcqa(limit=4 if SMOKE else 64)

model = psi.PSI(
    MODEL_ID,
    layer=15,
    rank=16,
    epochs=EPOCHS,
    batch_size=BATCH_SIZE,
    max_length=MAX_LENGTH,
    device=DEVICE,
    dtype=DTYPE,
    steering=True,
    cache_dir=CACHE,
    progress=False,
)

check = program.check(model, prompts)
print(check)
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
README.md:   0%|          | 0.00/2.70k [00:00<?, ?B/s]
train.jsonl:   0%|          | 0.00/303k [00:00<?, ?B/s]
validation.jsonl:   0%|          | 0.00/137k [00:00<?, ?B/s]
test.jsonl:   0%|          | 0.00/138k [00:00<?, ?B/s]
Generating train split:   0%|          | 0/110 [00:00<?, ? examples/s]
Generating validation split:   0%|          | 0/50 [00:00<?, ? examples/s]
Generating test split:   0%|          | 0/50 [00:00<?, ? examples/s]
config.json:   0%|          | 0.00/681 [00:00<?, ?B/s]
tokenizer_config.json:   0%|          | 0.00/7.23k [00:00<?, ?B/s]
vocab.json:   0%|          | 0.00/2.78M [00:00<?, ?B/s]
merges.txt:   0%|          | 0.00/1.67M [00:00<?, ?B/s]
tokenizer.json:   0%|          | 0.00/7.03M [00:00<?, ?B/s]
model.safetensors:   0%|          | 0.00/988M [00:00<?, ?B/s]
Loading weights:   0%|          | 0/290 [00:00<?, ?it/s]
generation_config.json:   0%|          | 0.00/138 [00:00<?, ?B/s]
mean_answer_mass: 0.7863202095031738
lowest_answer_mass: 0.05626127868890762
argmax_counts: (133, 158, 170, 195)
gold_agreement: 0.8353658318519592
leading_space_mass: 0.7863202095031738
warnings: ()

Inspect answer mass and answer counts before fitting. steering=True trains the controller jointly with the carrier, so this fit supports both operations below. Gold labels are available for evaluation; the correctness filter is off.

model.fit(prompts, program)

report = model.score(prompts, split="test")
display(report.to_frame())
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
💡 Tip: For seamless cloud logging and experiment tracking, try installing [litlogger](https://pypi.org/project/litlogger/) to enable LitLogger, which logs metrics and artifacts automatically to the Lightning Experiments platform.
You are using a CUDA device ('NVIDIA RTX A4000') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [2]
/opt/venv/lib/python3.12/site-packages/lightning/pytorch/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead.
/opt/venv/lib/python3.12/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=19` in the `DataLoader` to improve performance.
/opt/venv/lib/python3.12/site-packages/lightning/pytorch/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=19` in the `DataLoader` to improve performance.
`Trainer.fit` stopped: `max_epochs=8` reached.
interchange_agreement no_edit_baseline mean_answer_mass prompts pairs steering_success steering_no_edit model_agreement no_edit_model_agreement decision_rule
0 0.946667 0.333333 0.798538 200 150 0.90375 0.25 0.806667 0.266667 declared-answer argmax

For this preset, interchange agreement uses gold pair targets. Model agreement uses the donor’s frozen reading. The baselines use the same comparisons before editing. Requested-answer success asks for every declared slot. All these decisions use argmax over declared answers.

history = model.history
fig, axes = plt.subplots(
    1, 3,
    figsize=(12, 3.7),
    gridspec_kw={"width_ratios": [1.4, 1, 1]},
    layout="constrained",
)

curve = axes[0]
curve.plot(history.epoch, history["train/loss"], color=PSI_COLOUR, label="Training", linewidth=1.8)
curve.plot(history.epoch, history["val/loss"], color=SECONDARY, label="Validation", linewidth=1.8)
curve.set(xlabel="Epoch", ylabel="Objective", title="a  Joint fit")
curve.legend(loc="best")

# Spine endpoints coincide with the first and last displayed ticks.
epoch_ticks = np.unique(np.linspace(history.epoch.min(), history.epoch.max(), min(len(history), 4)).astype(int))
curve.set_xticks(epoch_ticks)
curve.spines[["top", "right"]].set_visible(False)
curve.spines["bottom"].set_bounds(epoch_ticks[0], epoch_ticks[-1])
loss_ticks = curve.get_yticks()
loss_ticks = loss_ticks[(loss_ticks >= curve.get_ylim()[0]) & (loss_ticks <= curve.get_ylim()[1])]
curve.set_yticks(loss_ticks)
curve.spines["left"].set_bounds(loss_ticks[0], loss_ticks[-1])

comparisons = [
    ("b  Interchange", report.no_edit_model_agreement, report.model_agreement),
    ("c  Requested answer", report.steering_no_edit, report.steering_success),
]

for axis, (title, baseline, learned) in zip(axes[1:], comparisons):
    axis.hlines([0, 1], 0, 1, color="#e4e4e1", linewidth=0.8)
    axis.scatter([baseline, learned], [0, 1], c=[NO_EDIT, PSI_COLOUR], s=55, zorder=3)
    axis.set(xlim=(-0.03, 1.08), ylim=(-0.45, 1.45), title=title, xlabel="Declared-answer success")
    axis.set_xticks([0, 0.5, 1])
    axis.set_yticks([0, 1], ["No edit", r"$\Psi$"])
    axis.spines[["top", "right", "left"]].set_visible(False)
    axis.spines["bottom"].set_bounds(0, 1)
    axis.tick_params(axis="y", length=0)

    for value, row in [(baseline, 0), (learned, 1)]:
        axis.annotate(f"{value:.1%}", (value, row), xytext=(0, 8), textcoords="offset points", ha="center", color=INK)

plt.show()
Computed notebook figure

Inspect three swaps from the held-out pairs. Each table row reports the prompt, the donor’s preferred slot, and the base’s answer before and after the carrier swap.

examples = []
swap_readings = []

for pair in prompts.pairs["test"][:3]:
    base = prompts.splits["test"][pair["base_index"]]
    donor = prompts.splits["test"][pair["source_index"]]
    reading = model.swap(base=base, donor=donor)
    swap_readings.append(reading)

    examples.append({
        "base": base["prompt"],
        "donor": donor["prompt"],
        "before": reading.loc[reading.base_before.idxmax(), "base_answer"],
        "donor_slot": int(reading.donor_before.idxmax()),
        "after": reading.loc[reading.base_after.idxmax(), "base_answer"],
    })

display(pd.DataFrame(examples))
base donor before donor_slot after
0 Question: Broccoli is green. What color is bro... Question: Broccoli is green. What color is bro... D 0 A
1 Question: Broccoli is green. What color is bro... Question: Broccoli is green. What color is bro... D 3 D
2 Question: Broccoli is green. What color is bro... Question: Broccoli is green. What color is bro... D 3 A
fig, axes = plt.subplots(1, len(swap_readings), figsize=(11, 3.4), layout="constrained")

for axis, reading, example in zip(np.atleast_1d(axes), swap_readings, examples):
    slots = np.arange(len(reading))
    axis.bar(slots - 0.18, reading.base_before, width=0.36, color=NO_EDIT, label="Before")
    axis.bar(slots + 0.18, reading.base_after, width=0.36, color=PSI_COLOUR, label="After")
    axis.axvline(example["donor_slot"], color=SECONDARY, linestyle=(0, (3, 3)), linewidth=1.1, label="Donor slot")
    axis.set_xticks(slots, reading.base_answer.str.strip())
    axis.set_yticks([0, 0.5, 1])
    axis.set(ylim=(0, 1.08), xlabel="Base option", ylabel="Conditional probability")
    axis.spines[["top", "right"]].set_visible(False)
    axis.spines["bottom"].set_bounds(slots[0], slots[-1])
    axis.spines["left"].set_bounds(0, 1)

axes[0].legend(loc="upper left", bbox_to_anchor=(0, 1.28), ncol=3)
plt.show()
Computed notebook figure

Request an answer without a donor using the same fitted model. The controller was enabled before the fit. The request below names a slot in each prompt’s own option list.

steered_examples = []

for index, prompt in enumerate(prompts.splits["test"][:3]):
    requested = index % len(prompt["choices"]["label"])
    response = model.steer(prompt, answer=requested)

    steered_examples.append({
        "prompt": prompt["prompt"],
        "requested": response.loc[requested, "answer"],
        "before": response.loc[response.before.idxmax(), "answer"],
        "after": response.loc[response.after.idxmax(), "answer"],
    })

display(pd.DataFrame(steered_examples))
prompt requested before after
0 Question: Broccoli is green. What color is bro... A D A
1 Question: Broccoli is green. What color is bro... B A B
2 Question: Broccoli is green. What color is bro... W I I

This run uses its own model and budget. The paper’s localization and control settings are described in arXiv 2611.21307. Its control evaluation also measures unrestricted greedy generation.

This run record reports the model, hardware, elapsed time and the held-out scores measured above.

run_summary = {
    "model": MODEL_ID,
    "device": DEVICE,
    "gpu": torch.cuda.get_device_name(0) if DEVICE == "cuda" else "CPU",
    "runtime_seconds": time.perf_counter() - START,
    "smoke": SMOKE,
    "card": {
        "metric": "Held-out declared-answer success",
        "rows": [
            {"label": "Donor swap", "edit": float(report.model_agreement), "baseline": float(report.no_edit_model_agreement)},
            {"label": "Requested answer", "edit": float(report.steering_success), "baseline": float(report.steering_no_edit)},
        ],
        "pairs": int(report.pairs),
    },
}
display({"application/json": run_summary}, raw=True)
{
  "card": {
    "metric": "Held-out declared-answer success",
    "pairs": 150,
    "rows": [
      {
        "baseline": 0.26666666318972904,
        "edit": 0.8066666722297668,
        "label": "Donor swap"
      },
      {
        "baseline": 0.25,
        "edit": 0.9037500023841858,
        "label": "Requested answer"
      }
    ]
  },
  "device": "cuda",
  "gpu": "NVIDIA RTX A4000",
  "model": "Qwen/Qwen2.5-0.5B",
  "runtime_seconds": 240.14373225532472,
  "smoke": false
}