How can adversarial robustness be assessed?

Assessment involves generating adversarial examples to test the AI model's susceptibility, evaluating various attack methods, and measuring the model's performance under these perturbed conditions. Techniques include white-box and black-box attack simulations.

What are some strategies to improve adversarial robustness?

Strategies include adversarial training, where models are trained on both clean and adversarial examples, robust optimization techniques, and careful feature selection. Input validation and anomaly detection also play a role in identifying potentially malicious inputs.

All terms

Software & AIAI / ML in Devices Software Lifecycle

Adversarial Robustness

Resilience of an ML model to inputs deliberately crafted to cause misclassification.

Reviewed by Christian Espinosa, Founder, Blue Goat CyberLast reviewed September 19, 2026

Definition

Adversarial examples - small, often imperceptible perturbations - can flip medical-imaging predictions. Robustness evaluation, input sanitization, and detection of out-of-distribution inputs are emerging expectations for safety-critical AI. Robustness covers both deliberate attacks and natural variation, such as noise, artifacts or differences between scanners. Testing approaches include perturbation tests, evaluating on data from new sites and red-teaming. The EU AI Act (Article 15) requires high-risk AI systems to be resilient against attempts to alter their use or performance, and FDA's cybersecurity guidance expects threats to AI components to be included in threat modeling.

What the regulation says

Regulators and standards bodies, such as the FDA in its AI/ML-based Medical Devices Action Plan and IMDRF in its guiding principles for AI/ML in medical devices, emphasize the need for robust AI systems. This includes addressing vulnerabilities to adversarial attacks to ensure the safety and effectiveness of AI-driven MedTech. While specific clauses directly addressing "adversarial robustness" are evolving, the principles of risk management in ISO 14971 and software validation in IEC 62304 implicitly cover the need to mitigate such risks.

What this means in practice

Particularly relevant for AI radiology and pathology devices where image acquisition is partially under attacker influence.

Examples

An AI ophthalmology device designed to detect diabetic retinopathy must be robust against subtle, intentionally crafted image perturbations that could cause it to miss pathology or flag healthy eyes as diseased.
A pathology AI system used for cancer detection needs to be resilient to adversarial attacks that might alter digital slide images, leading to misclassification of tissue samples.
An AI-powered diagnostic tool interpreting MRI scans should maintain its accuracy even if an attacker introduces imperceptible noise intended to alter the model's output, potentially influencing treatment decisions.

Common pitfalls

•Assuming that general AI model validation is sufficient to cover adversarial robustness risks.
•Failing to consider the potential for adversarial attacks throughout the entire lifecycle of a medical device.
•Overlooking the importance of real-world data and expert review in identifying potential adversarial vulnerabilities.
•Believing that simple input sanitization alone provides complete protection against sophisticated adversarial attacks.
•Neglecting to update adversarial robustness evaluations as new attack techniques emerge or model changes occur.

Frequently asked questions

Adversarial robustness is crucial in MedTech because imperceptible perturbations can lead to misdiagnoses or incorrect treatment recommendations, posing significant patient safety risks. The consequences of AI failure in healthcare are much higher than in many other domains.

Grouped by theme

Editor's picks

· Hand-selected related concepts

Software & AI

Model Drift Monitoring

Ongoing surveillance of input data and model outputs to detect performance degradation post-deployment.

Cybersecurity

Penetration Testing

Hands-on adversarial testing in which qualified independent testers attempt to exploit a device's security controls.

More in Software & AI

· Same category

Software & AI

AAMI TIR45(TIR45)

AAMI Technical Information Report providing guidance on applying Agile software development practices within an IEC 62304-compliant medical device software lifecycle.

Software & AI

AI/ML-Enabled Medical Device

Medical device that uses artificial intelligence or machine learning to perform its intended use.

Software & AI

Algorithm Change Protocol(ACP)

The detailed procedural section of a PCCP that specifies how planned modifications to an AI/ML model will be developed, validated, and implemented.

Software & AI

Algorithmic Bias and Fairness

Systematic differences in model performance across demographic or clinical subgroups.

Cited by

Where this term appears across MedTech Terms.

Ecosystems (2)

Latest in MedTech

Sources

3 sources

Every citation below opens the original document. Each is graded against our source-tier hierarchy so you can see what rests on binding law versus commentary.

Tier 2Regulator guidance and consensus· 3

Link health: 3 verified· last checked 2026-06-20

FDA·1IMDRF·1MDCG·1

Inline markers like [1] jump to the matching reference above.

My progress

All terms0 / 513

Software & AI0 / 31

Saved on this device, and to your account when signed in. View studied terms

Full glossary

Browse all terms A to Z, each with a short definition.

On this term

Category: Software & AI
Sources: 3
Updated: 9/19/2026

Compare with another term

Learn in 60 seconds

Resilience of an ML model to inputs deliberately crafted to cause misclassification.

·Particularly relevant for AI radiology and pathology devices where image acquisition is partially under attacker influence.
·Robustness evaluation, input sanitization, and detection of out-of-distribution inputs are emerging expectations for safety-critical AI.
·Robustness covers both deliberate attacks and natural variation, such as noise, artifacts or differences between scanners.

Remember this

Watch out: Assuming that general AI model validation is sufficient to cover adversarial robustness risks.

Related terms

You may also need

Auto-suggested from Software & AI and shared keywords.

All Software & AI terms

From the Blue Goat network

Related resources and services on this topic.