AI/ML · Evaluation
A fine-tuning pipeline, and the audit that caught it
A modular LoRA training framework — and the adversarial audit of our own work that found 84 issues, including the reason the model recited instead of reasoned.
84
issues found in our own pipeline
34/35
high-severity findings confirmed on re-check
1
finding refuted as a false positive
The problem
Off-the-shelf models were not enough for a task-specific need, so we built a fine-tuning pipeline and trained a series of adapters. The trained model produced fluent, confident answers that were substantially memorised from the training set rather than reasoned from it — the failure mode that is easiest to mistake for success, because the outputs look correct.
What we built
The pipeline was rebuilt as discrete modules rather than a notebook: chat templating, dataset tooling, training, an evaluation harness, inference, and a test suite, so each stage could be checked independently. Then the whole thing was audited adversarially — a 48-agent workflow that mapped the codebase, generated findings, and re-verified every high-severity claim against the code rather than trusting the first pass.
The result
84 issues found. All 35 high and critical findings were adversarially re-checked: 34 confirmed or partially confirmed, and 1 refuted as a false positive. The memorisation diagnosis was written up separately as a change to the training contract — chat-template skew, deduplication, and epoch policy — rather than as a prompt-level patch, because the cause was in the data pipeline, not the prompt.
What this does not claim
This is published because the audit found real problems. A fine-tuning pipeline that nobody has tried to break is not evidence of anything. The refuted finding is listed alongside the confirmed ones for the same reason — an audit that never disagrees with itself is not an audit.