Software & AI
Explainability and Interpretability
In one line
Methods (SHAP, saliency maps, prototype explanations) that help users understand why a model made a given prediction.
Definition
Explainability provides post-hoc rationalizations of model decisions; interpretability is a property of inherently transparent models. Both feed clinician trust and inform Human Factors design.
Common techniques include saliency maps for images, feature attribution methods such as SHAP, and example-based explanations. Explanations can mislead if they look plausible but do not reflect what the model actually used, so they should be validated like any other output. Regulators focus on transparency for users: what the device does, its performance and limitations, and the information users need to judge when to rely on it.
Why it matters
EU AI Act high-risk requirements and FDA Good Machine Learning Practice both expect manufacturers to address explainability proportionate to clinical risk.
Common pitfalls
- •Confusing explainability as always providing full causal understanding of an AI model, rather than a justification for its output.
- •Assuming that an interpretable model eliminates the need for thorough validation and clinical risk management.
- •Failing to tailor the level of explainability or interpretability to the specific clinical application and user needs.
- •Overlooking the dynamic nature of AI models, where initial explanations may become irrelevant as the model evolves or is re-trained.
Sources
- FDA GMLP · FDA
- MDCG Software Guidance · MDCG
- FDA - Software as a Medical Device (SaMD) · FDA