Software & AI
Adversarial Robustness
In one line
Resilience of an ML model to inputs deliberately crafted to cause misclassification.
Definition
Adversarial examples - small, often imperceptible perturbations - can flip medical-imaging predictions. Robustness evaluation, input sanitization, and detection of out-of-distribution inputs are emerging expectations for safety-critical AI.
Robustness covers both deliberate attacks and natural variation, such as noise, artifacts or differences between scanners. Testing approaches include perturbation tests, evaluating on data from new sites and red-teaming. The EU AI Act (Article 15) requires high-risk AI systems to be resilient against attempts to alter their use or performance, and FDA's cybersecurity guidance expects threats to AI components to be included in threat modeling.
Why it matters
Particularly relevant for AI radiology and pathology devices where image acquisition is partially under attacker influence.
Common pitfalls
- •Assuming that general AI model validation is sufficient to cover adversarial robustness risks.
- •Failing to consider the potential for adversarial attacks throughout the entire lifecycle of a medical device.
- •Overlooking the importance of real-world data and expert review in identifying potential adversarial vulnerabilities.
- •Believing that simple input sanitization alone provides complete protection against sophisticated adversarial attacks.
- •Neglecting to update adversarial robustness evaluations as new attack techniques emerge or model changes occur.
Sources