Software & AI
Generative AI in Medical Devices
In one line
Devices that use LLMs or diffusion models to generate text, images, or recommendations within a clinical workflow.
Definition
Generative AI devices include clinical documentation aids, image synthesis tools, and triage assistants. Hallucination, prompt injection, output drift, and provenance create new failure modes that traditional SaMD frameworks under-address.
As of 2026, FDA has not authorized a device that uses generative AI for a diagnostic output, but its Digital Health Advisory Committee discussed generative AI-enabled devices in November 2024. Key questions include how to define intended use for open-ended outputs, how to evaluate accuracy and hallucination rates, and how to monitor performance after release. Many documentation tools are designed to stay outside device regulation by keeping the clinician in control.
Why it matters
FDA has begun issuing draft guidance specific to generative AI; sponsors should expect evolving expectations on evaluation, monitoring, and labeling.
Common pitfalls
- •A common pitfall is underestimating the complexity of validating generative AI outputs, leading to models that might generate inaccurate or misleading information. Another mistake is assuming traditional software validation methodologies are sufficient for generative AI, failing to account for emergent behaviors and adaptability. Neglecting to establish a clear post-market surveillance plan for generative AI can result in undetected performance degradation over time.
- •Failing to adequately address data provenance and potential biases in training data can lead to discriminatory or unsafe outputs.
- •Overlooking the need for continuous monitoring and update mechanisms for generative AI, as their performance can drift over time, is a common error.
Sources