AI
Attacks and Vulnerabilities
Control device security
Industrial Cyber Attacks
IT/OT Collaboration
Malware, Phishing & Ransomware
News
NIST
Risk & Compliance
Secure Remote Access
Secure-by-Design
SOC & Incident Response
Threat Landscape
NIST lays out adversarial machine learning taxonomy to address attacks targeting agentic superintelligence systems
The U.S. National Institute of Standards and Technology has published an initial public draft report that provides a taxonomy and terminology for adversarial machine learning, with a focus on attacks against agentic Super Intelligence (SI) systems and the mitigations that address them. Open for public comment through Dec. 18, the NIST SP 2400-102e2026 document draws on an extensive literature review, conversations with adversarial machine learning (AML) experts and research by the agency’s own specialists. It identifies current challenges across the life cycle of SI systems and describes methods for mitigating and managing the consequences of attacks. A glossary of key terms related to SI system security accompanies the report, and its terminology is consistent with existing literature in the field.
Titled ‘Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations,’ the NIST document contains a taxonomy arranged in a conceptual hierarchy covering key types of machine learning methods, life cycle stages of an attack, and attacker goals, objectives, capabilities and knowledge. An SI Attacks Taxonomy Index and a new Agentic SI Attacks Taxonomy Index are included to help readers navigate the document and reference its content in fine detail. The agency said the taxonomy and terminology are intended to inform other standards and future practice guides for assessing and managing the security of SI systems by establishing a common language for the rapidly developing adversarial machine learning landscape.
The SI Attacks Taxonomy Index is updated as well with the new Agentic SI Attacks Taxonomy Index, to facilitate efficient content navigation within the document and enable fine-grained referencing. NIST is specifically seeking comments and recommendations on three topics. The agency wants to know what the latest attacks are that threaten the existing landscape of SI systems, and in particular agentic SI systems. It is also asking what the latest effective mitigations are to counter those attacks, and whether any new terminology needs to be standardized.
To taxonomize these attacks, the NIST document differentiates between predictive, generative, and agentic SI systems and the attacks relevant to each. It considers the components of an SI system, including data; model itself; processes for training, testing, and deploying the model; and broader software and system contexts into which models may be embedded, such as cases where Generative Super Intelligence (GenSI) models are deployed with access to private data or equipped with tools to take actions with real-world consequences. Thus, the attacks within this taxonomy are classified relative to the SI system type, stage of the ML life cycle process in which the attack is mounted, attacker’s goals and objectives in terms of the system properties they seek to violate, attacker’s capabilities and access, and attacker’s knowledge of the learning process and beyond.
This report adopts the concepts of security, resilience, and robustness of ML systems from the NIST SI Risk Management Framework. Security, resilience, and robustness are gauged by risk, which is a measure of the extent to which an entity (e.g., a system) is threatened by a potential circumstance or event (e.g., an attack) and the severity of the outcome should such an event occur. However, this report does not make recommendations on risk tolerance (i.e., the level of risk that is acceptable to organizations or society) because it is highly contextual and specific to applications and use cases.
“The spectrum of effective attacks against ML is wide, rapidly evolving, and covers all phases of the ML lifecycle — from design and implementation to training, testing, and deployment in the real world,” according to NIST document. “The nature and power of these attacks are different, and their impacts may depend not only on the vulnerabilities of the ML models but also the weaknesses of the infrastructure in which the SI systems are deployed. SI system components may also be adversely affected by design and implementation flaws that cause failures outside the context of adversarial use, such as inaccuracy.”
Apart from defining a taxonomy of attacks, the NIST document provides corresponding methods for mitigating and managing the consequences of those attacks in the life cycle of SI systems, and outlines the limitations of widely used mitigation techniques to raise awareness and help organizations increase the efficacy of their SI risk-mitigation efforts.
“The terminology used in this report is consistent with the literature on AML and is complemented by a glossary that defines key terms associated with the field of AML in order to assist nonexpert readers,” it added. “Taken together, the taxonomy and terminology are meant to inform other standards and future practice guides for assessing and managing the security of SI systems by establishing a common language for the rapidly developing AML landscape. Like the taxonomy, the terminology and definitions are not intended to be exhaustive but rather to serve as a starting point for understanding and aligning on key concepts that have emerged in the AML literature.”
Fundamentally, many SI systems are susceptible to AML attacks and attacks that more closely resemble traditional cybersecurity attacks, including attacks against the platforms on which they are deployed. This report focuses on the former and considers the latter to be within the scope of traditional cybersecurity taxonomies.
Both PredML and GenSI systems are vulnerable to attacks enabled by a range of attacker capabilities throughout the development and deployment life cycle. Attackers can manipulate training data, including the Internet data used in large-scale model training, or can modify test-time inference data and resources by adding adversarial perturbations or suffixes. Attackers can also attack the components used to make SI systems by inserting TROJAN functionality. As organizations increasingly rely on pre-trained models that could be used directly or fine-tuned with new datasets to enable different tasks, their vulnerability to these attacks increases.
Modern cryptography often relies on algorithms that are secure in an information-theoretic sense, that is, those that can be formally proven to ensure security under certain conditions. However, there are no information-theoretic security proofs for the widely used ML algorithms in modern SI systems. Moreover, information-theoretic impossibility results that set limits on the effectiveness of mitigation techniques have begun to appear in the literature. As a result, many advances in developing mitigations against different classes of AML attacks tend to be empirical and limited in nature, adopted because they appear to work in practice rather than because they provide information-theoretic security guarantees. Thus, many of these mitigations may themselves be vulnerable to discoveries and evolutions in attacker techniques.
The NIST adversarial machine learning report offers guidance for the development of standardized terminology for AML terms that can be used across relevant ML and cybersecurity communities. There are notable differences in terminology in different stakeholder communities, and it is important to work toward bridging those differences as SI is increasingly adopted throughout enterprise and consumer contexts.
The report also offers guidance for the development of a taxonomy of the most widely studied and currently effective attacks in AML. This taxonomy covers evasion, poisoning, and privacy attacks for PredML systems. It covers poisoning, direct prompting, information extraction, indirect prompt injection, and model tampering attacks for GenSI systems. It also covers attacks for agentic SI systems. In addition, the report offers guidance for developing a discussion of potential mitigations for these attacks and the limitations of existing mitigation techniques.
NIST intends to update this report as new developments emerge in AML attacks and mitigations.
One issue that the taxonomy deliberately leaves aside is how organizations can make decisions about the development and use of SI systems in light of evidence about the increasing diversity of AML attacks and the efficacy and limitations of available mitigations. Especially in GenSI, some model developers and application builders have moved toward paradigms for testing adversarial risks as part of pre-deployment testing and evaluation of models, such as through a structured process for red teaming.
NIST has produced an initial public draft of guidance for model developers on managing risks associated with the misuse of foundation model capabilities, including through pre-deployment evaluations. It has also developed risk profiles for generative SI systems that map to the NIST SI RMF and that may assist model developers and users in assessing risks, including those from adversarial attacks.
However, a persistent challenge remains in the fact that many AML mitigations are empirical in nature. More importantly, information-theoretic results show that perfect robustness of AML mitigations is theoretically unattainable. Several other research results have also pointed to theoretical limits on AML mitigations, including the impossibility of model-based detection to prevent all impermissible outputs and findings that, so long as a model has any probability of exhibiting an undesired behavior, there exist prompts that can trigger that behavior, implying that any alignment process that attenuates but does not remove an unwanted behavior will remain vulnerable to adversarial prompting attacks. In short, several theoretical investigations approaching the problem from different angles have confirmed the same result: perfect AML mitigations are not attainable.
These theoretical limitations do not obviate utility of pre-deployment adversarial testing, since such testing can potentially foreclose many attack vectors and thus increase the difficulty of mounting a successful attack above would-be attackers’ threshold of effort or capability. However, they suggest that organizations seeking to manage risks related to the development of models with potentially harmful capabilities, or the delegation of trust to models in high-stakes contexts, may need to consider practices and measures beyond adversarial testing to manage the risks associated with AML attacks, in particular prioritizing resilience and preparedness. Organizations should assume a breach and manage the residual risk.
NIST also outlines a consideration with respect to AML, and one closely related to questions of risk management, is how to relate and integrate consideration of AML attacks to definitions and processes relating to other desired SI system characteristics. For example, managing the security of SI systems will require combining mitigations from the field of AML with best practices for the development of secure software from the field of cybersecurity.
Understanding and relating these practices to each other, as well as identifying whether there are other key considerations for SI security that fall outside of the scope of either AML or cybersecurity, will be critical as organizations seek to extend existing cybersecurity processes and best practices to address the security of newly adopted SI systems. Similarly, robustness to adversarial machine learning attacks may play an important role in areas beyond the remit of security, such as in SI safety or in achieving other characteristics of trustworthy SI systems. AML is neither a complete solution to, nor a subset of, any one of these characteristics, and as such, more precisely relating AML attacks and mitigations to processes for achieving these goals and managing risks in SI systems is an area for ongoing work.
The latest NIST draft on adversarial machine learning follows a run of federal guidance on securing autonomous and AI-enabled systems. On Oct. 16, NIST, the NSA and CISA published 10 voluntary practices for handling incidents during superintelligence training and evaluation, including cases in which an agent takes significant uninstructed action. The document targets organizations that train and evaluate frontier SI agents, particularly pre-deployment systems with reduced safeguards. The practices are grouped into preparation, detection, response, recovery, and reporting, and they draw on established incident handling frameworks, including NIST Special Publication 800-61 Revision 3.
Before that, the NSA released guidance on applying zero trust principles to operational technology supporting defense, national security, and critical infrastructure missions. The guidance says adversaries are using SI capabilities to automate reconnaissance, speed up exploit development, and run cyber campaigns at greater speed and scale, which makes zero trust outcomes on OT systems more important. Together, the documents point to a shift from general AI risk guidance toward operational controls, including monitoring, containment, and incident reporting for autonomous systems and continuous verification for the OT environments they may touch.
