Why “validated” is close to meaningless
Every company selling AI for science says its models are validated. Almost none of them say what was measured, against what, or where the thing breaks. The word has been worn smooth.
That matters more than it sounds. More than half of preclinical research doesn’t reproduce, an estimated US$28 billion a year in the United States alone. Very little of that is fraud. Most of it is work built on results that looked solid and weren’t. A model trained on that same evidence base inherits the problem, and a confident interface hides it.
So this is a guide to interrogating a digital twin, with ours as the worked example: what to ask, what a good answer sounds like, how our engine answers, and the one test it is built to fail on purpose. Use it on us. Then use it on everyone else.
What are you actually validating against?
A digital twin has a circularity problem hiding inside it, and it catches out more teams than it should.
Our engine works in two halves. A high-fidelity physics simulator acts as an oracle: rigorous, mechanistic, and far too slow to run thousands of times. We distil that oracle into a fast AI surrogate, which is the part you actually use. The obvious way to validate the surrogate is to check it against the oracle.
That check is necessary, and it is nowhere near sufficient. A surrogate that faithfully reproduces its simulator has proved exactly one thing: that it agrees with itself. If the simulator is wrong about biology, the surrogate will be confidently, precisely, reproducibly wrong about biology too.
Breaking that loop needs a second, independent test against real measurements. The two questions are genuinely different:
- Did the surrogate learn its teacher? Compare surrogate to oracle.
- Was the teacher right? Compare the whole system to reality.
Skip the second and you have built a very fast way to be wrong.
What we publish, and what we hold back
Our engine models 3-D biological growth and structure over time. It was seeded from one scarce, openly licensed real dataset, which calibrated the physics oracle; the oracle then generated a corpus of virtual experiments across a systematic sweep of conditions, and the surrogate was trained from that corpus.
Two benchmarks answer the two questions above, and we publish both with their basis attached:
- ~88% fidelity reproducing the physics oracle, head to head. That answers did the surrogate learn its teacher.
- 96% of the accuracy of a model trained directly on the real growth data, on a train-on-synthetic, test-on-real (TSTR) benchmark. That answers was the teacher right. It is the anti-circularity test, and it is the one that matters most.
The engine is also lean enough to run on your own hardware, which is a deployment property rather than a validation result.
We hold the rest back, and we would rather say so than fudge it. The simulation-to-reality anchor error, the coverage of the uncertainty intervals, the corpus design, the exact model footprint and the throughput figures are real, measured, and part of how the engine was built. Published in full they amount to a build recipe, so they belong in a technical session rather than on a public page. Ask and we will walk you through the report card, numbers and method together. What we will not do is publish a number with its measurement basis stripped off, because that is the thing this article is arguing against.
Note what the 96% figure is and isn’t. It says a model trained purely on our synthetic data gets within 96% of the accuracy of one trained on the real thing. It does not say the twin is “96% accurate about biology”. That is a different and much larger claim, and we don’t make it.
That last distinction is worth more than any of the numbers. A figure without its measurement basis is decoration.
What happened when a benchmark came in low?
The TSTR number did not start at 96%. It started at 68%, which was well under our own target and a genuinely bad result.
There are two things you can do with a number like that. You can tune around it: adjust the training regime, reweight the loss, try seeds until the benchmark moves. Or you can treat it as information about the model rather than an obstacle to the launch.
We investigated. The cause turned out to be a root-cause limitation in the growth law itself, not the surrogate that learned it. Fixing it meant redesigning that law: real work, on the physics, with no guarantee the benchmark would move at all.
It moved to 96%. But the reason we trust that result is what happened alongside it: physical calibration improved more than fourfold at the same time, and the internal structure of the cultures stayed intact. Three independent measures improved together.
That pattern is the tell. Metric-hacking moves one number and quietly degrades the others. A genuine fix to the underlying physics moves them in the same direction at once. When you’re assessing someone else’s model, ask what else changed when their headline number improved.
Where does the engine fail?
Deliberately, and by design: it cannot self-certify in true microgravity.
We keep a three-bucket ledger that we don’t allow to blur:
- Validated against reality. 3-D biological growth and structure in the round, low-settling regime, on one validated cell line, checked against real data.
- Corroborated premise. Published research showing microgravity alters cell behaviour. This supports the idea. It does not validate our twin.
- Open roadmap gap. 3-D biological growth under real microgravity against a matched 1G control. Not acquired. Not validated.
The Gravity Knob models gravity as a settling force, and that mechanism is validated against real 1G suspension-culture data, which shares the round, low-settling regime with true microgravity. That makes it a physics-grounded bridge for reasoning about the 1G→µg transition. It does not make it a substitute for flight data, and the engine ships a validation check on microgravity that it fails visibly, in the report card, rather than silently.
Closing that gap needs real flight data against a matched control. Until we have it, saying so is the only honest option available.
What to ask anyone selling you a twin
Four questions, and none of them require you to be a modeller:
- What is the number, and what was it measured against? “Validated” alone is not an answer.
- How did you break circularity? If every test compares the model to the thing that trained it, nothing has been tested.
- What happened the last time a benchmark came in low? The answer tells you whether you’re buying research or marketing.
- Where does it fail? A vendor who can’t name a failure mode either hasn’t looked or won’t say.
We would rather be judged on the fourth question than the first. A model that knows the edge of its own competence is worth considerably more than one that scores higher and can’t tell you when to stop believing it.
Ask us all four. The two headline benchmarks and their basis are above, the failure mode is named, and the rest of the report card is a conversation away rather than a form to fill in. If a vendor can’t answer question two or four at all, that tells you more than any number they lead with.
Read how the engine is built on the ALTDATA Digital Twin page, how the proof of concept came together in the SASIC grant case study, or talk to us and we’ll walk the report card against a problem of your own.