By Lorenzo Giuntini, Founder, Read Your Scan
LinkedIn: Lorenzo Giuntini
LinkedIn: Read Your Scan
Healthcare organizations want a simple answer to a difficult question: How accurate is this AI imaging tool? An aggregate score offers a convenient response, but it may hide the most important failure. The system may produce several plausible observations while missing the finding that mattered most in the radiologist’s report.
Governance teams should request a specific measure for principal-finding recognition alongside traditional performance metrics. This measure will not replace clinically validated sensitivity, specificity, or expert adjudication. It can, however, help identify a category of failure that broad agreement scores may obscure.
Start with the intended task
Not every imaging AI system is meant to read an entire examination. Some algorithms detect a defined condition, others prioritize worklists, and others help summarize existing reports. The appropriate metric depends on the intended use. A main-finding metric is relevant only when a system purports to identify or discuss the central imaging findings of an examination.
For these systems, the test question should be explicit: Did the output identify the principal finding established by the comparison process? An answer that describes anatomy fluently but omits the central abnormality should not receive credit solely for sounding medically credible.
A small audit illustrates the gap
In a small comparison I ran, model outputs were checked against existing radiologist reports for eight imaging examinations. No tested model identified the radiologist’s main reported finding in more than three of the eight examinations. The sample was small and selected. It does not yield a generalizable diagnostic accuracy figure. Existing clinical reports are useful comparisons, but they may not provide the complete ground truth needed for a formal validation study.
The practical lesson is about score design, not about declaring any particular model unsuitable. If a report lists several minor or normal observations but fails to identify the principal abnormality, an average text-similarity or finding-count score could give a misleadingly favorable impression.
Build a finding-level scorecard
A governance committee could ask evaluators to document five separate fields.
Task and population. State modality, anatomy, examination type, patient group, input format, and the exact function being evaluated.
Principal-finding recognition. Define the principal finding prospectively through an appropriate reference process, rather than letting evaluators choose it after seeing the AI output. When reports contain multiple findings of similar importance, establish an adjudication rule.
Localization. Record whether the finding is assigned to the correct anatomical structure, side, or level. A statement can recognize the general condition yet be wrong in clinically meaningful ways.
Unsupported or misleading statements. Track additions not supported by the reference evidence, and distinguish harmless wording differences from errors that could affect decisions.
Review and uncertainty. Report the denominator, cases excluded, ambiguous comparisons, confidence intervals where appropriate, and the role of blinded clinical adjudicators.
These categories should remain separate. Combining them into one score makes trends difficult to interpret and corrective action harder to target.
Make monitoring continuous
The American College of Radiology’s Assess-AI program supports monitoring algorithm inputs, software versions, and concordance with radiology reports. Its approach demonstrates that model performance monitoring is a continuing operational activity, not a single purchasing event. The FDA’s January 2025 draft lifecycle guidance similarly discusses risk management across the life of regulated AI-enabled device software; it is nonbinding draft guidance, not a universal mandate.
The local governance process should define who reviews a disagreement, when a recurring failure triggers investigation, and what evidence is required before a revised system returns to use. Sampling plans must also distinguish deployment monitoring from formal diagnostic validation.
A useful governance dashboard does not merely answer whether an AI system is generally accurate. It answers what it was asked to find, whether it found the main issue, whether it located that issue correctly, and how the organization knows. That is a far more actionable basis for trust.