Artificial intelligence; Machine learning; Precision medicine; Molecular diagnostic techniques; Healthcare disparities; Validation studies as topic; Bias

Integrating machine learning and artificial intelligence (AI) into molecular diagnostics has changed how disease detection, prognostic forecasting and treatment selection are approached [1]. Models that draw on genomic, transcriptomic, imaging and electronic health record data promise care tailored to the individual patient. I refer to this goal as personalized medicine and treat the term “precision medicine” as its synonym. Published reports of predictive performance are often impressive, but the record in clinical practice is far weaker. A systematic review of deep learning studies in medical imaging revealed that, of 81 nonrandomized studies comparing algorithms with clinicians, only nine were prospective, and only six were tested in a real clinical setting; nevertheless, the performance of 61 claimed to be at least comparable to that of experts [2]. A later review revealed only 41 randomized trials of machine learning interventions in health care; none adhered fully to the CONSORT-AI reporting guidelines, and approximately half were conducted at a single site [3]. The evidence that these tools help patients lags well behind the enthusiasm for them [4].

I argue that clinical translation is best understood as a continuum running from data to decision rather than as a single validation step. Even a model with strong reported performance can fail if its input data arrives late or are incomplete, if it fits poorly into the clinical workflow, or if its predictions are not linked to a clear clinical action [4]. Weaknesses at three points along this continuum explain much of the gap: models whose outputs cannot be checked against biology, data that are neither representative nor consistently recorded, and the lack of shared standards for evaluation and oversight. In response, I propose that clinical AI should pass through five linked stages: representative, high-quality data; a clinically interpretable model; external and prospective validation; integration into clinical workflow and decisions; and postdeployment monitoring and governance, with findings from monitoring feeding back into each earlier stage.

The first weakness is the distance between what an algorithm concludes and what is biologically plausible. Many deep learning systems are opaque, mapping inputs to clinical outcomes through statistical association with no account of the biology underneath. When a model latches onto a surrogate variable or an artifact of how the data were collected, rather than a true driver of disease, its performance tends to collapse in external testing. Zech and colleagues showed this with a pneumonia model trained on chest radiographs. Because the prevalence of pneumonia differed sharply across hospitals, the internal performance of the model decreased when it was tested on data from another hospital system [5]. During the COVID-19 pandemic, a living systematic review of 232 prediction models revealed that every one had a high or unclear risk of bias [6]. An external validation of a proprietary sepsis model used in hundreds of hospitals in the United States revealed an area under the curve of 0.63; the model missed 67% of patients who developed sepsis while generating alerts for 18% of all hospitalized patients [7].

It helps to separate three properties that are often conflated. Predictive validity means that a model forecasts outcomes reliably in the population where it will be used. Clinical interpretability means that the intended user can understand enough about how an output was produced to judge whether to act on it. Mechanistic or causal validity means that the features driving a prediction reflect real disease processes. A model can predict well without identifying any causal mechanism, and a biologically plausible explanation does not by itself prove causation; thus, each property needs its own evidence. Post hoc explanation methods, such as heatmaps laid over an image, can look reassuring while describing the model only loosely, and they may create false confidence at the level of the individual patient [8]. I agree with Rudin that, for high-stakes decisions, models should be interpretable from the outset rather than explained after the fact [9]. When an inherently interpretable model performs as well as a black box, it should be preferred. Where explanations are offered, they should be meaningful to the intended clinical user and decision, not only to data scientists, and should point to recognizable features such as a known pathway, a validated biomarker or a plausible radiological sign. Such plausibility guards against shortcut learning but is not proof of mechanism.

The second weakness concerns the data. The value of personalized medicine depends on the breadth of the population behind it. In 2016, approximately 81% of participants in genome-wide association studies were of European ancestry [10], and polygenic risk scores built on those data are several times more accurate in people of European descent than in other groups [11]. It is worth distinguishing the bias already present in the health system that generates the data from the bias introduced by modeling choices. One widely used commercial algorithm in the United States relies on health care costs as a proxy for illness. Because unequal access meant that less money was spent on black patients, they had to be considerably sicker than white patients were to receive the same risk score. This inequity originated in the health system, but the choice of label carried it into the algorithm; correcting it would have increased the percentage of black patients flagged for additional care from 17.7% to 46.5% [12]. Chest radiograph classifiers have similarly been shown to underdiagnose female, black, Hispanic and low-income patients, with the highest rates in intersectional subgroups [13].

Representativeness is only part of the problem. Real-world performance also depends on the quality and consistency of the wider data ecosystem: whether records are complete and timely, whether diagnoses and laboratory tests are coded consistently, whether duplicate records and mismatched patient identifiers are resolved, and whether electronic health records, laboratory information systems and diagnostic platforms can exchange data reliably. A model built on curated research data may meet missing values, new assay platforms or local coding habits in practice. Changes of this kind, known as dataset shifts, can degrade performance silently, without any change to the model itself [14]. Generalizability is therefore a property of the model together with the data system that feeds it.

These issues are sharper in low- and middle-income countries such as Pakistan. Their populations are poorly represented in the genomic and imaging datasets on which most commercial tools are trained, and a model validated in North America or Europe cannot be assumed to perform equally well in a South Asian hospital. Underrepresentation is compounded by fragmented digital systems, variable laboratory capacity, inconsistent data standards and few local datasets suitable for validation. A scoping review identified only 10 studies evaluating real-world applications of AI in health care in these countries and reported problems with reliability, workflow fit and adaptation to the local context [15]. Rather than simply importing externally trained tools, these settings need locally governed data, regional validation networks and, where necessary, models developed or recalibrated for their own populations and health systems.

The third weakness is the absence of shared standards for evaluation. Clinical assessment should look beyond discrimination measures such as the area under the receiver operating characteristic curve. Calibration, the agreement between predicted and observed risk, is often unreported, yet a poorly calibrated model can mislead clinical decisions even when its discrimination is good [16]. Sensitivity, specificity and predictive values at the intended decision threshold, performance in relevant subgroups, and clinical utility, for example, net benefit estimated through decision curve analysis [17], are equally important. The choice of metric should follow from the intended decision and from the relative harms of false-positive and false-negative results. The sepsis model described above illustrates this point: its low sensitivity and heavy alert burden mattered more in practice than any single summary statistic [7].

Validation should also be treated as continuing rather than as a single event. Prospective, multisite evaluation within real clinical workflows, ideally through randomized trials, remains essential, but it should include geographic, temporal and subgroup validation. Once deployed, models need continuous monitoring for drift in their inputs and performance. Clinical AI should follow a cycle of validation, deployment, monitoring, recalibration and revalidation rather than being regarded as permanently validated after one study [14,16]. The guidance largely exists. TRIPOD+AI sets out how prediction models that use machine learning should be reported [18], DECIDE-AI covers the early clinical evaluation of decision support systems [19], and CONSORT-AI covers randomized trials [20]. What is missing is consistent enforcement. Journals can require adherence to these guidelines at submission, and regulators and hospital procurement committees can ask for external validation data, calibration and subgroup performance, and a postdeployment monitoring plan before a tool is adopted.

None of these steps are technically out of reach, but they require the research community to value careful evaluation as highly as headline performance does. The ultimate aim of clinical AI is not better prediction but safer, more equitable and demonstrably better clinical decisions and patient outcomes. Tools that meet that standard will earn the trust of clinicians because they have been shown to work for all of their patients.

AI: Artificial intelligence

CONSORT-AI: Consolidated Standards of Reporting Trials – Artificial Intelligence extension

COVID-19: Coronavirus disease 2019

DECIDE-AI: Developmental and Exploratory Clinical Investigations of DEcision support systems driven by Artificial Intelligence

TRIPOD+AI: Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis + Artificial Intelligence

US: United States