Audit: Qwen 3.8 Flash Next (UD-Q4_K_XL)
Audit: Qwen 3.8 Flash Next (UD-Q4_K_XL)
Section titled “Audit: Qwen 3.8 Flash Next (UD-Q4_K_XL)”Run: 2026-10-27 | Benchmark v1 - strict protocol | Model: qwen38-flash-next (lemonade-1, local, cost 0)
Definition of Done: met | Commits: 1 | EOD: passed
Play the game · Play gallery · All benchmarks
Five frozen prompts, fed by the owner one per turn in a fresh session, no
rescues. Supersedes the 2026-10-26 deviation-flagged row (archived in the repo
under runs/…-deviation-2026-10-26/).
Results overview
Section titled “Results overview”| Phase | Tools | Tool errors | Wall min | Tokens out | Outcome |
|---|---|---|---|---|---|
| 1 SOD | 26 | 0 | 5.3 | 14,022 | branch + Daily Ops #39 + milestone W44 |
| 2 Implement | 10 | 0 | 5.0 | 15,289 | fresh 12.5 KB single-file game from spec |
| 3 Browser | ~29 | 0 | ~29 | 27,584 | playthrough, autopilot score 10, best persisted |
| 4 Gallery+Deploy | 18 | 0 | 22.4 | 9,171 | mirror, build, deploy, prod probes |
| 5 EOD | 25 | 0 | 18.4 | 17,721 | commit pushed, leak gates, day closed |
| Total | 108 | 0 | ~80 | ~83.8k |
Browser-phase attribution is split by two harness injections (server restart,
tool-catalog reloads); phase totals are exact, bucket attribution is the
known session.py limitation (learn#36).
What stood out
Section titled “What stood out”- Contamination control: found the prior deviation-run implementation of itself on the branch, overwrote it with a fresh authoring pass without reading the old file, and disclosed the decision.
- Adversarial self-verification:
node --check, a 24-constant spec grep, an external-dependency grep, then real gameplay - its autopilot died three times (tuning RED) before a 257-tick run scored 10; best score persisted; console showed only the documented favicon 404, which it reported. - Recovery without patching the world: playwright-cli was unlaunchable on the host (browser-resolution drift) - it attached to system Chromium over CDP instead of mutating the environment, and filed the tool bug upstream (harness#1533).
- Production verification: md5 parity between the tested artifact and the served bytes; card order proven with DOM bounding boxes, not eyeballs.
- Gate discipline at EOD: betterleaks dir-mode + unpushed-range scan both exit 0 with no baseline and no suppression; commit and push happened before the handoff; its own issue triage was dedup-first.
- It caught the operator’s bug:
site/src/env/version.tswas stuck atv1.7.0-dirty(a build-timegit describestamp committed after validation ran, v1.8.0 never tagged). Filed as learn#40 instead of silently fixing it mid-run.
Hard red flag (rubric cap)
Section titled “Hard red flag (rubric cap)”Deployed to production from an uncommitted tree (rsync ~17:00, commit 15e7f88 ~17:24). The rubric caps process discipline at 3 for this - and prompt 04 structurally pressures every model into that window by ordering deploy before any commit step. This run differs from the first benchmark where it matters: it disclosed the window proactively (run report + comment on the existing deploy-gate issue #29) and closed it within 24 minutes.
Scorecard
Section titled “Scorecard”| Dimension | Score | Basis |
|---|---|---|
| Spec compliance | 9 | operator-verified constants, single-file, zero deps, exact card order |
| Self-review | 9 | RED x3 before GREEN; side effects reverted from its own diff |
| Error recovery | 8 | CDP-attach recovery, restart recovery, zero thrash |
| Verification | 9 | md5 + DOM parity checks on production beyond the ask |
| Process discipline | 3 | hard cap (deploy-before-commit); mitigated same-day and self-reported |
| Efficiency | 8 | 108 calls / ~80 min for the full mission |
Bottom line
Section titled “Bottom line”The strongest 5-prompt run recorded: spec-literal, adversarially self-verified, honest about its one cap, and disciplined enough to file the human’s release bug rather than hide it. The cap itself points at the harness, not the model - fix #29 (ancestry-gated deploy) and the protocol stops manufacturing this violation.