On AI in trial design

Read the Limitations Section: What Three AI Trial-Design Papers Say About Their Own Limits

Published 2026-09-23 · Every figure below is the authors' own, attributed to the paper it comes from. Two of the three sources are preprints and are named as such. The artifact checks described in the fourth section were performed on 22 and 23 September 2026 and are perishable.

The sentence the builders wrote themselves

In September 2026, a university–industry team spanning the University of Illinois, Keiji AI, AbbVie, Osaka and Penn published a system called TrialAtlas: a hierarchy of AI agents that reads a trial protocol, searches the literature and competing trials, checks it against FDA precedent, and tells you what a regulator is likely to object to. They built it against 291 real FDA Complete Response Letters — the letters the agency sends when an application cannot be approved as submitted. Six pharmaceutical-industry experts, working blind, judged 86.4% of the concerns it raised to be valid. Two frontier deep-research products were scored the same way and came in lower: 83.1% for one, 59.3% for the other.

Then the authors wrote this, about their own system:

Performance at this level does not support autonomous use.

They are referring to the fact that their system agrees with FDA-identified design deficiencies about half the time. Roughly half of what it flags falls outside what the agency actually objected to, and roughly half of what the agency objected to, it misses. Their recommendation is that the system be used as a triage aid: something that surfaces candidate concerns, with the evidence attached, for a human expert to adjudicate.

That is an honest sentence, and it is the most useful one in the paper.

It is also the sentence that will likely not survive contact with a slide deck.

Three papers, three limits, one boundary

TrialAtlas is not alone. Three notable publications in the last two months have asked, in different ways, whether AI systems can do the strategic and design work of clinical development. All three are worth reading. All three state a limit. The limits point the same direction.

Can an AI plan a development portfolio? Two researchers at Oxford framed oncology development as a prediction problem: given everything publicly known about a drug program on a given date, predict the trials the sponsor launches in the next six months. They tested four frontier agents, each equipped with ten retrieval tools restricted to information available at the time.

The result that matters is not the accuracy score. It is this: three of the four frontier agents proposed designs matching historically failed trials significantly more often than historically successful ones. For GPT-5.4, the gap was 25.0% matching failed designs against 7.5% matching successful ones. For Qwen3-235B it was wider still: 35.5% against 11.3%. And giving GPT-5.4 ten retrieval tools bought it nothing over asking the question cold.

A caveat the authors' own data supports, and which I will not skip: in their sample, failed trials outnumber successful ones, and the failed design is frequently the obvious design. A system proposing the obvious answer will match failed trials more often whether or not it has any preference for failure. The direction holds across every model tested, so something real is there — but the mechanism is that retrieval surfaces precedent without weighting precedent by outcome. This is a workshop preprint rather than journal peer review, and the whole dataset covers 45 historical programmes.

The authors' own scope statement is explicit. The work is intended for retrospective structured forecasting and evaluation, and not for autonomous trial design, regulatory submission, investment decisions, or patient care.

Can an AI run the research organization? A Stanford-led group published a "virtual biotech" in Science — a multi-agent organization that runs target discovery and trial analysis end to end, at a disclosed median cost of about twenty-three cents per trial analyzed. Peer review made this paper genuinely stronger: reviewers re-implemented eight of the case studies from the agents' own instructions and found the conclusions held in all eight, with no consequential coding mistakes. That is a real validation and it did not exist in the preprint.

Something else happened between the preprint and the published version. The preprint carried a limitation stating that a single agent mistake can propagate through the virtual organization, and that human experts therefore remain necessary in the workflow. That sentence is not in the published paper. The word "error" does not appear in it at all.

I am not suggesting anything improper; limitations sections are rewritten in peer review all the time, and the published limitations are real ones about data coverage. But the specific warning that got cut is the one an adopting reader most needed, and its removal is checkable by anyone who reads both versions.

One more thing about that paper, which comes from its own released data rather than its text: I profiled the 56,707-row dataset the authors published. Just over a third of the rows carry no primary-endpoint label, and fewer than a quarter carry any serious-adverse-event data at all. The headline claim about fewer adverse events rests on that minority. Trials that terminate badly report less often than trials that succeed, which is a well-known asymmetry — and neither version of the paper addresses it.

What I checked, and what I found missing

Here is the part that took the longest and is the least interesting to write about, which is usually a sign it matters.

TrialAtlas advertises two artifacts. Its Data Availability section points to a public dataset; its Code Availability section points to a public code repository. I fetched both on 22 September 2026. Neither was there to fetch: the code repository did not resolve at all, and the dataset refused access without credentials. I checked again four hours later, and again on 23 September, with the same result each time. Both parent accounts resolve normally and list other public work, so this is not a broken link or a network problem — the artifacts simply were not publicly available.

That does not make the numbers wrong. It makes them claims I cannot audit, reproduce, or test — which means I cannot separate the system's performance from the authors' report of it. "This system detects half of FDA-identified design deficiencies" and "this system's authors report that it detects half of FDA-identified design deficiencies" are different sentences. On the day I checked, only the second one was available.

The paper was four days old when I looked, and pre-release gaps are ordinary. Both may be live by the time you read this, which would be good news.

I mention this not to criticize the authors, who wrote an unusually candid limitations section and published their full agent prompts. I mention it because checking took about ten minutes, and it changes what the paper can be used for. If you are going to cite a preprint in a board deck, fetch its artifacts first. If they are not there, say so in the deck.

The boundary all three describe

The three converge on a single boundary.

These systems are good — genuinely, measurably good — at finding and organizing precedent. TrialAtlas surfaced concerns that pharmaceutical experts endorsed at a higher rate than two frontier deep-research products. The virtual biotech's analyses survived independent re-implementation by four reviewers. Retrieval, synthesis, comparison against prior trials, checking a design against what similar designs did: this work is being done competently and at a cost per analysis that rounds to nothing.

What none of them demonstrates is knowing when the precedent is the wrong guide.

The Oxford paper shows this most directly, because its reward function is explicit about what it optimizes: whether the program was eventually approved. By the authors' own decomposition, the specific decision the model makes explains about 2.5% of the variance in that reward. That number does not by itself prove the model learned the wrong thing. What it establishes is that the training signal is far too diffuse to teach decision quality — whether a programme was approved is overwhelmingly determined by things other than the decision being scored. What remains learnable from a signal like that is the surface pattern: what approved programmes did. The matching result above is what that looks like in practice — the playbook reproduced, including the parts of the playbook that failed.

And the moments that matter most in oncology development are departures. Accelerated approval on response rate for imatinib. The first tissue-agnostic approval — pembrolizumab in 2017, for MSI-H or mismatch-repair-deficient solid tumours regardless of where the tumour started. Both were breaks from what the precedent said, argued from evidence, at a time when the precedent said otherwise. A system that scores well by matching historical behavior would have scored each of them as unlikely.

This is not a claim that AI cannot eventually do this. It is a description of what these three specific, well-built, currently-best-in-class systems do and do not do, according to the people who built them.

The failure I am actually worried about

Now the part that has nothing to do with benchmarks.

A founder reads the TrialAtlas abstract — 86.4% expert-validated, beats the frontier tools — and reasonably concludes the design-review problem is handled. The limitations section does not make the slide. Six months later a protocol goes to FDA with a deficiency the system did not flag, or with three "concerns" the team spent a quarter remediating that were never real. The meeting goes badly. Money and time are gone.

And here is what happens next, which is the expensive part: that founder concludes AI doesn't work for this.

That conclusion is wrong, and it is costly in both directions. What failed was not the technology. What failed was deploying a stated triage aid as an oracle — a use its own authors ruled out in print. The same system, used the way the authors describe, would have surfaced candidate concerns with evidence attached and handed them to someone qualified to sort the real from the spurious. That is a genuinely useful thing to have. It is worth twenty-three cents a trial, or considerably more.

A team burned this way carries forward a false conclusion: that the technology fails at this, when what failed was the deployment envelope. That conclusion is durable in a way the original mistake is not — it shapes which tools the team will trust in the next decision, and the next, at exactly the moment those tools are getting better. I have watched a clinical team swing from encouraged adoption to blanket rejection when early digital tools failed while being deployed in large clinical trials.

What to do instead

The papers themselves describe the envelope. Four things follow from it.

1. Treat the output as candidate concerns, not findings. The authors' word is "triage." A triage system's job is to order a queue for someone qualified to work it. It is not to empty the queue. If nobody with regulatory and clinical judgment adjudicates each item, you have not saved work — you have moved it, and lost the audit trail.
2. Ask what the system optimizes, and whether that is what you want. The Oxford reward function tracked approval and revenue. There is no patient in it, no magnitude of benefit, no unmet need, no toxicity. That is a legitimate modeling choice for a forecasting study and a disqualifying one for a planning tool. Before trusting any such system, find the objective function and read it.
3. Weight precedent by outcome, because the machine will not. Retrieval returns what was done. It does not return what should have been done. When a tool tells you that eleven similar trials used this endpoint, the useful question is how many of those eleven succeeded, and whether the ones that failed failed because of that endpoint. That reweighting is currently human work.
4. Check the artifacts before you cite the claim. Ten minutes. Every time.

A disclosure, since the argument runs toward my own market: OncAdios sells the kind of adjudication this essay says these systems still require. That is a reason to check the four points above against the papers rather than against me — which is why the claims here are the authors' own, and why the falsifiers below are specific enough to lose.

The patient is in this argument

There is a version of this essay that is purely about capital efficiency — bad tools waste money, be careful. That version is incomplete.

A trial design deficiency that survives review is not primarily an expensive mistake. It is a study in which patients were asked to enroll, accept risk and give time, in an experiment that could not cleanly answer the question it was built to answer. The people who bear that cost are not in the room when the protocol is reviewed, are not consulted about which AI tool the team adopted, and cannot evaluate whether the concern list was adjudicated by someone competent or waved through because a system with an impressive benchmark said it looked fine.

The FDA letters that TrialAtlas learned from are, read properly, a record of the agency catching these before patients paid for them.

So the question to ask of any AI-assisted design review is not only whether it saves time. It is whether the people enrolling in the resulting trial would recognize the review as having been done — actually done, by someone accountable for it. A system that raises fifty plausible concerns and a team that adjudicates none of them has not reviewed the protocol. It has generated the appearance of review, which is worse than no review, because it feels finished.

This is also where the precedent problem stops being abstract. A tool that reproduces what similar programs did will reproduce their dose-finding, their endpoints, their eligibility criteria — including the ones that were wrong at the time and were later corrected because patients bore the cost of them being wrong. Project Optimus is the standing example: the precedent was to escalate to the maximum tolerated dose, the precedent was followed for years, and the people who paid for it were patients receiving more drug than they needed. A system optimized to match precedent has no mechanism for noticing that.

How I would know I am wrong

If this argument is right, it should be falsifiable. Three things would change my view:

1. A system that beats expert adjudication on flagged concerns, with released artifacts and reported uncertainty. Not a benchmark win — an independently reproducible result where the AI's concern list is more accurate than a qualified reviewer's on the same protocols, confidence intervals included.
2. Evidence that these systems identify justified departures from precedent, not just conformity to it — a system that correctly flags "the standard design is wrong here, and here is why" on cases where the standard design later failed.
3. A published head-to-head on real pre-IND packages where AI-assisted review and expert review are compared against subsequent FDA feedback, prospectively.

None of these exists publicly today. All three are plausible within a few years. If any lands, the position in this essay should be revised, and I would rather revise it than defend it.

The essentials

Three research groups, working independently with different methods, built AI systems for clinical trial design and strategy. All three produced useful results. All three stated a limit in their own papers. The limits agree: these systems retrieve and organize precedent well, and do not yet know when precedent is the wrong answer.

Their authors said so. The value is in reading the sentence where they said it — and in using these systems inside the envelope their builders drew, where they are genuinely good, rather than outside it, where a confident wrong answer costs a program and then costs the technology its credibility.

---

The Calibration Sprint is built for exactly the adjudication this essay describes: two weeks, fixed scope, senior clinical-regulatory judgment on the question in front of your program — with the strongest counter-case and the conditions under which the recommendation would change.

If the question is what your first-in-human program has actually learned before the next commitment, that is the Sprint's first-in-human evidence track: the FIH Diligence Read.

Sources

• Lin J, Wang Z, Chen Z, Scott E, Yang Z, Yu F, Zhong S, Sun J. TrialAtlas: Multi-Agent Research Organization for Clinical Trial Design and Optimization. arXiv:2609.21859v1, 18 September 2026. Preprint, not peer reviewed. The quoted limitation and all performance figures are the authors'. The code and dataset URLs given in the paper were both unreachable when fetched on 22 September 2026 and again on 23 September: the repository did not resolve, and the dataset was access-restricted.
• Bolton W, Torr P. Learning Clinical-Trial Strategy: Offline Policy Training for Decision Agents. arXiv:2608.03606, 4 August 2026. ICML 2026 workshop preprint. Both authors are at the Department of Engineering Science, University of Oxford. The failed-versus-successful design matching, the retrieval result and the scope statement are the authors'.
• Zhang H, Eckmann P, Miao J, Mahon A, Zou J. The Virtual Biotech: A multi-agent AI framework for therapeutic discovery and development. Science, 17 September 2026, article eaeg6779. The preprint-versus-published comparison and the dataset profiling are my own reading of the two texts and the authors' released data file.
• For what an FDA pre-IND meeting actually decides, and how to prepare a package that survives it: What an Oncology Pre-IND Meeting Actually Decides.

← Back to Writing  ·  What a pre-IND meeting actually decides →  ·  When model validation is not enough →  ·  How I would know if I'm wrong →