Measure before you sell: the test bench for my AI models
Field: computer vision, model training and evaluation
Period: August to September 2026
Client: internal project, my own companies (Lab 77, R&D)
A detection model always produces an answer. The only question that matters before selling it is: how many does it miss, and how many does it make up? To answer that honestly, I built a workshop where the exam is set before training and where the machine does not pick its own winner. Here is the approach, and the questions I draw from it for any business leader who is being offered an AI.
Context
For my aerial imagery platform, I need models that spot cracks, rust, trees or vehicles in photos. Some are taken from open source projects, others I trained myself; licences are covered in a dedicated article. In August 2026, one week of training runs produced many failures, nearly all of them already known: the lessons lived in notes, not in the tools. I turned this into a generic workshop, where every failure becomes a coded safeguard.
Problem
- A model that runs is not a model that tells the truth: it answers on every image with the same confidence, right or wrong.
- The exam can be rigged without meaning to: test images too close to the training ones, a criterion chosen after the fact, and the model looks better than it is.
- Chance manufactures winners: comparing thirty versions of a model means thirty opportunities to find one that succeeds by luck.
- Memory gets lost: I archived four models for which nobody could say why they had not been kept; they had never been judged.
The approach
1. Write the success criterion before knowing the result
Every campaign starts with eight questions in plain language, including this one: what would be unacceptable, even if everything else improves? The answer is a hard limit: a candidate that crosses it is rejected, whatever its other figures. Another question requires changing only one thing at a time, otherwise nobody knows what moved the result.
2. Seal the exam before gathering the training images
The protocol has sixteen steps, and each one refuses to start without the stamp of the previous one. The exam images are sealed with a digital fingerprint before the training set is assembled, and we then prove that the training set contains none of them. There are two exams: one to choose, and a second, opened only once, to confirm. The exam runs through the real production engine: a bench that measures differently from production ends up measuring something else.
3. Take away the machine's right to choose
All versions produced during training are kept and the exam decides afterwards: twice, an automatic formula had picked the very first one, barely trained. The final calculation accounts for the number of candidates compared. And "we keep what we have" is a valid result: one campaign ended with all of its candidates rejected, which avoided replacing a model that works with a worse one.
4. A judge that advises, humans who decide
A broader vision model reviews the detections: yes, no or uncertain. An unreadable answer stays unreadable, never a yes by default: a judge that guesses manufactures confidence. It validates nothing on its own. When the right answer has to be established by hand, two independent readings are sealed before comparison, and the model's figures are revealed last. I can also drop a photo into a trial area and see what production would do. Nothing goes into service without my decision.
5. An identity card, including for retired models
Each model has a card tied to the fingerprint of its file: status, exam, human check, licences, and above all what has not been measured. RT-DETR, a well-known open source detector I used for vehicles, was retired: seen from directly above it saw nothing, and at an oblique angle it took roofs for trucks. Its card remains, so that we know what was retired and why.
Result
Every model in the catalogue carries a written verdict and its limits, including the conditions under which it has not yet been measured. It is this card, more than any single figure, that says where a model can be used and where it must not be.
What I take away
- A model can only be sold if you know how many it misses. Ask your supplier for two figures, not one: the share of real objects found (recall) and the share of correct detections (precision).
- Ask when the exam was set. Before or after training? On images like yours, or on a demonstration set? Who chose the version delivered?
- Ask for the worst case, not just the average. And what the model does when it does not know.
- Ask what has not been tested. A serious supplier has the answer in writing.
- A missing label is not a missing object. "False detections" counted on a public dataset turned out, on inspection, to be real rust that had not been annotated.
- Measure once, then deliver. After two weeks of successive benches, measuring had become an end in itself. Now: one model, one sealed exam, one signed card.
Going further
You are being offered a detection AI and want to know what it is worth on your images? I help you ask the right questions, or build your own test bench. See sovereign AI, or book a 20-minute discovery call.
Sources
- Classification: accuracy, recall, precision, Google Machine Learning Crash Course, accessed 15 November 2026.
- Metrics and scoring: quantifying the quality of predictions, official scikit-learn documentation, accessed 15 November 2026.
- RT-DETR, official repository (Apache-2.0 licence), accessed 15 November 2026.
- Photo: JumadilM / Wikimedia Commons (commons.wikimedia.org), licence CC BY-SA 4.0.
Tags: artificial intelligence, computer vision, model evaluation, precision, recall, test bench, buying AI