Measures

import subprocess
import sys
import tempfile
from pathlib import Path

from spotregen import make_sample_repo

tmp = tempfile.TemporaryDirectory()
root = make_sample_repo(Path(tmp.name) / "demo")


def spotregen(*args):
    """Run `spotregen` in the sample repository; show the call and its output."""
    done = subprocess.run(
        [sys.executable, "-m", "spotregen", *args],
        cwd=root,
        capture_output=True,
        text=True,
        check=False,
    )
    print("$ spotregen", *args)
    print(done.stdout.rstrip())
    print(f"[exit {done.returncode}]")

Three measures describe how well a unit is specified. Two are read from the files, the third runs the evaluations.

Definition 1 (The three measures)

Intent coverage: the share of a unit’s intent items that name at least one evaluation, where every named evaluation exists in an evaluation file of the unit.
Specification size: the lines of the unit record, of the decision records its intent items cite (each once) and of its evaluation files, compared with the lines of the implementation.
Mutation score: the share of planted mutants of the unit’s module that its evaluations kill.

spotregen measures prints the first two for every unit and always exits 0. spotregen mutate UNIT measures the third: it exits 0 if every mutant is killed and 1 otherwise.

Example 1 (Coverage and size)

spotregen("measures")

$ spotregen measures
| unit | coverage | code | unit | decisions | evaluations | spec | ratio |
|---|---:|---:|---:|---:|---:|---:|---:|
| samplepkg.core.calc | 1/1 | 10 | 26 | 9 | 9 | 44 | 4.4 |
[exit 0]

Example 2 (Mutation score) mutate writes each mutant into a throw-away copy of the repository and runs the unit’s evaluations against it. A mutant is killed if an evaluation fails or errors, or if the run exceeds --timeout. Progress goes to standard error; the report lists the survivors only. --out FILE archives every result as TOML.

spotregen("mutate", "core.calc", "--python", sys.executable)

$ spotregen mutate core.calc --python /Users/bartz/workspace/spotregen/.venv/bin/python3
# Mutation score: samplepkg.core.calc

1 evaluations in tests/test_regen_eval_calc.py, 4 mutants.

## Surviving mutants

| mutant | line | operator | original | mutated |
|---|---:|---|---|---|
| M003 | 10 | return | `return 1` | `return None` |
| M004 | 10 | constant | `1` | `2` |

Mutation score: 2/4 (50 %).
[exit 1]

Both survivors change the private helper _helper, which no evaluation calls. That is either a missing evaluation or a helper nobody needs; a survivor asks for a look, it is not a verdict.

Operators

compare (< and <=, == and !=, in and not in, is and is not), boolean (and and or), negation (drop a not), condition (negate the test of an if or while that is not a single comparison or a not), constant (True and False, a number plus one), operator (+ and -, * and /, & and |, also in augmented assignments), statement (delete a call statement, continue or break), raise (replace by pass) and return (return None). Annotations, docstrings, imports, __all__ and if TYPE_CHECKING: blocks are left alone. The mutated module is written with ast.unparse, so comments do not survive; behaviour does.

Next: draft a unit.