Vectors as durable evaluations

import os
import subprocess
import sys
import tempfile
from pathlib import Path

from spotregen import make_sample_repo

tmp = tempfile.TemporaryDirectory()
root = make_sample_repo(Path(tmp.name) / "demo")


def spotregen(*args):
    """Run `spotregen` in the sample repository; show the call and its output."""
    done = subprocess.run(
        [sys.executable, "-m", "spotregen", *args],
        cwd=root,
        capture_output=True,
        text=True,
        check=False,
    )
    print("$ spotregen", *args)
    print(done.stdout + done.stderr, end="")
    print(f"[exit {done.returncode}]")

Definition 1 (Vector files) A durable evaluation judges any implementation of a unit through its public names. Most evaluations are written as code and therefore bind to one language. A vector file is an evaluation written as data: a list of worked examples, each with the inputs and the output every implementation must reproduce. The file is generated once by a reference implementation, reviewed by a person, committed next to the records, and replayed by one pytest evaluation per implementation. A second implementation in another language reads the same file. Fowler’s freeq project keeps such a file for its encrypted direct messages: spec/e2ee-dm-vectors.json, generated by the Rust SDK and replayed byte for byte by the JavaScript SDK, which closed months of silent disagreement between the two.

spotregen needs no new field for this. The pytest function that replays the file is an ordinary evaluation: it is named by an intent item, it imports only public names (EVAL03), and it carries the marker regen (EVAL02). The vector file is a fixture of the evaluation, so it travels with the evaluation file in the regeneration bundle only if the evaluation reads it from a path a generator can see. Keep it under tests/ next to the evaluation.

One vector file for the sample unit

The sample repository has one unit, samplepkg.core.calc, with one promise, add(a, b) == a + b. A vector file for it lists cases that a rewrite must reproduce, including the edge cases a competent stranger would not think of:

import json

vectors = [
    {"a": 0, "b": 0, "sum": 0},
    {"a": -1, "b": 1, "sum": 0},
    {"a": 2**63, "b": 1, "sum": 2**63 + 1},
    {"a": -(2**31), "b": -(2**31), "sum": -(2**32)},
]
(root / "tests/calc-vectors.json").write_text(json.dumps(vectors, indent=2) + "\n")
print((root / "tests/calc-vectors.json").read_text())
[
  {
    "a": 0,
    "b": 0,
    "sum": 0
  },
  {
    "a": -1,
    "b": 1,
    "sum": 0
  },
  {
    "a": 9223372036854775808,
    "b": 1,
    "sum": 9223372036854775809
  },
  {
    "a": -2147483648,
    "b": -2147483648,
    "sum": -4294967296
  }
]

The evaluation replays every case. It is appended to the evaluation file of the unit, which already carries the marker:

evaluation = '''

def test_add_reproduces_the_vectors():
    """Every worked example of tests/calc-vectors.json holds."""
    import json
    from pathlib import Path

    for case in json.loads((Path(__file__).parent / "calc-vectors.json").read_text()):
        assert add(case["a"], case["b"]) == case["sum"], case
'''
path = root / "tests/test_regen_eval_calc.py"
path.write_text(path.read_text() + evaluation)
print(path.read_text())
import pytest

from samplepkg.core.calc import add

pytestmark = pytest.mark.regen


def test_add():
    assert add(2, 3) == 5


def test_add_reproduces_the_vectors():
    """Every worked example of tests/calc-vectors.json holds."""
    import json
    from pathlib import Path

    for case in json.loads((Path(__file__).parent / "calc-vectors.json").read_text()):
        assert add(case["a"], case["b"]) == case["sum"], case

The intent item names the new evaluation, so that EVAL05 does not report it as unlinked:

unit = root / "regen/units/core.calc.toml"
unit.write_text(
    unit.read_text().replace(
        'evaluations = ["test_add"]',
        'evaluations = ["test_add", "test_add_reproduces_the_vectors"]',
    )
)
spotregen("check")
$ spotregen check
regen check: 1 units, 0 errors, 0 warnings
[exit 0]

The evaluations pass on the implementation:

done = subprocess.run(
    [sys.executable, "-m", "pytest", "-q", "-m", "regen", "tests"],
    cwd=root,
    capture_output=True,
    text=True,
    env={**os.environ, "PYTHONPATH": "src"},
)
print(done.stdout.strip().splitlines()[-1])
2 passed, 1 warning in 0.00s

When to write one

A vector file earns its place where an implementation in another language is plausible, where the output is a byte sequence or a number that must match exactly, and where the reference implementation exists and can generate the cases. A metamorphic relation stays the better choice when the right answer cannot be stated in advance. Regenerate the file only from the reference implementation, never from a candidate: the file is the oracle, and a candidate that rewrites its own oracle has judged itself.