Part 4/6 — Part 6 ← Part 5 ← Part 4 ← Part 3 ← Part 2 ← Part 1

May 2026 — co-authored with Gemma 4

A note on the voice. Look how different this post reads from the ones before it. That is the experiment: I let Gemma 4 drive the whole thing and gave it a much lighter review than Part 1 or Part 2 got. It produced a very bland, robotic post. I am leaving it as it came out, because the contrast is the useful part.

The transition from Exam V2 to V3 was necessitated by the discovery that the V2 harness was providing invalid scoring data, masking model instability and quantization failures.

Setup

• Hardware: Framework 13 (Ryzen AI 9 HX 370, 64GB DDR5).
• Inference: llama-swap via Vulkan.
• KV Cache: q8_0 (fixed across all runs).
• Environment: Go-based scraper resilience task (buffering, eviction, background flush).

The Problem: V2 Harness Flaws

The Exam V2 harness relied on shell scripts and grep-based evaluation. This “vibecoded” approach introduced three critical failures:

1. Denominator Inflation: When a model’s code panicked and crashed the go test suite, the harness only counted the tests that had successfully run. This allowed models that crashed early to achieve 100% scores (e.g., 2/2 instead of 2/10).
2. Masked Instability: The loose scoring made it impossible to distinguish between a “bad” model and an “unstable” one. We saw high variance in Qwen and GPT-OSS across seeds that the harness failed to flag as high-uncertainty.
3. Fragile Parsing: Using grep to check for behavioral markers was prone to false positives/negatives, especially when models hallucinated markers or failed to follow formatting.

Trigger for rebuild: High-tier models (Qwen 3.6) were producing lower scores than lower-tier models, indicating the measuring stick was broken.

The Solution: Exam V3

V3 is a complete rewrite of the evaluation logic, moving from shell-based orchestration to an in-process Go grader.

• In-process execution: The grader runs the tests directly, eliminating shell-parsing errors.
• Deterministic scoring: Uses go test -race -json and jq to parse results.
• Fixed Denominator: Every run is measured against the full test suite (13 tests), regardless of whether the process crashes.
• Performance: Evaluation time per submission dropped from ~60s to ~4s.

The harness is exam-driver.go — a generic runner that submits the prompt to llama-swap over HTTP, saves the response, invokes the per-exam evaluator, and collects the score. The exam itself lives in blogpost/benchmarking_llms/bench/exam_v3/: the prompt given to the model, the scraper.go file it has to modify, the grader_test.go suite it’s scored against, and the eval.sh scorer wrapping the grader. A reference solution is in scraper_solution.go.

Results (Clean Rerun)

Note: All models were re-run to ensure a clean baseline on the new harness.

Key Findings

• Quantization Instability: The 0/13 score for Qwen 3.5 MXFP4 suggests the current Unsloth GGUF stack/quantization is fundamentally broken for this task.
• Seed Variance is Real: The 5/0/11 split for GPT-OSS highlights that single-seed benchmarks are useless for assessing coding reliability.
• In-process is mandatory: The speed and accuracy gains of V3 make the shell-based V2 approach untenable for scaling.

What we got wrong

We assumed a shell-script wrapper was sufficient for complex behavioral testing. We prioritized ease of implementation over deterministic measurement, which led to the “denominator inflation” bug.

Next Steps

• Provider Audit: Test alternative Qwen GGUF providers to see if the MXFP4 failure is a model issue or a packaging/quantization issue.
• Phase 2 (Agentic Harness): Move away from synthetic Go tasks toward evaluating models on real-world tasks (analyzing actual repos, logs, and manifests).

Footnote:

• All models were run with reasoning disabled. Worth a follow-up exam with reasoning enabled to see how much it moves the needle — especially for Qwen 3.5 / 3.6, which lean heavily on chain-of-thought.
• Still surprised Qwen 3.6 didn’t score better; Gemma 4 totally kicked butt on this test.
• vibecoded on hermes + gemma4 models