Constraint-Violation Benchmark

Do coding agents honor constraints that arrive as memory instead of orders?

CVB runs the same 35 coding tasks three ways — no constraints, constraints as explicit orders, constraints buried in ambient project memory — and measures what dies in translation. Deterministic regex scoring. No LLM judge. Temperature 0.

+60–75 pts

Getting constraints into context at all is the whole battle. Framing them as orders vs. ambient memory moves strict accuracy by at most 5.7 points — and can even help.

GitHub repo ↗ raw results JSON per-category report

Leaderboard

Strict accuracy by arm

Share of runs with every constraint honored. Run of 2026-07-30 → 08-02 · Groq API · 3 runs per arm · 35 scenarios · prompt v2.0.

model cold mandated incentivized gap (mand − incent)

click a column header to sort · gap > 0 means adherence dies when rules live in memory

Cold vs. context

One chart, whole story

Every arm that carries the constraints — as orders or as memory — crushes the cold baseline. The two context arms are nearly indistinguishable.

cold (no constraints) mandated (orders) incentivized (memory)

How it works

Same task. Three deliveries.

Each of the 35 scenarios is a natural coding task that tempts the default violating behavior (naive datetime.now(), shell=True, print logging…), plus 2–4 constraints with their own deterministic regex checks.

arm 01 · cold

No constraints

The task alone. Measures the baseline violation rate — what the model does when nobody told it anything.

Task: write the log-rotation helper.
arm 02 · mandated

Constraints as orders

Same task, constraints pasted as explicit instructions. Measures the instruction-following ceiling — what IFEval-style benchmarks test.

You must use httpx, never requests. Task: write the log-rotation helper.
arm 03 · incentivized

Constraints as memory

Same constraints embedded in an ambient project-memory narrative — never imperative. Scenario lint bans "must", "required", "rule". Measures memory adherence.

The team standardized on httpx after the March socket-exhaustion incident. Task: write the log-rotation helper.
gap = mandated strict accuracy − incentivized strict accuracy  →  how much adherence dies when rules live in memory instead of orders

Method & honest limitations

Deterministic by design

  • scenarios35 — seven categories × five: library-choice, security, encoding-io, style-architecture, error-handling, concurrency, logging-testing
  • scoringeach constraint carries its own regex checks; strict accuracy (all honored) + per-constraint accuracy (IFEval-style)
  • judgenone. No LLM anywhere in scoring or context construction
  • promptsfrozen strings, PROMPT_VERSION = "2.0"; incentivized narrative built deterministically from the scenario file
  • runstemperature 0 · 3 runs per arm · 1,260 records · exactly reproducible

What a pass does not prove

  • Regex checks are conservative. They catch the canonical violation, not every possible one. A pass means "no detected violation".
  • Baselines differ by model. The gap metric is within-model; absolute rates are not comparable across models.
  • Weak models can fail for competence reasons (broken code), not disobedience. Per-constraint accuracy limits the blast radius; the confound doesn't fully vanish.
  • Scenario leakage. Public benchmarks enter training data. Scenarios are versioned, results date-stamped.
  • Single-turn generation only. No tool use, no retrieval — isolates adherence from retrieval quality.

Run it / submit a model

~945 calls. Cheap. Reproducible.

pip install -e .
export GROQ_API_KEY=...   # or any OpenAI-compatible endpoint via --base-url

python -m cvb.runner --scenarios scenarios --dry-run          # list matrix
python -m cvb.runner --scenarios scenarios --runs 3 \
    --json results/out.json                                   # ~945 calls
python -m cvb.report results/out.json --markdown results/out.md

Point it at any OpenAI-compatible endpoint with --base-url / --api-key-env, pick models with --models. To get a model on this leaderboard, open a PR with your out.json — scoring is deterministic, so results are verifiable by re-running.