Innovation Hub

Red teaming is essential in security, actively probing defenses, identifying weaknesses, and assessing system resilience under simulated attacks. For organizations that manage critical infrastructure, every vulnerability poses a risk to data, services, and trust. As systems grow more complex and threats become more sophisticated, traditional red teaming encounters limits, particularly around scale and speed. To address these challenges, we built the next step in red teaming: an Automated Red Teaming for AI solution that combines intelligence and efficiency to achieve a level of depth and scalability beyond what human-led efforts alone can offer.

Red Teaming: The Backbone of Security Testing

Red teaming is designed to simulate adversaries. Rather than assuming everything works as expected, a red team dives deep into a system, looking for gaps and blind spots. By using the same tactics that malicious actors might use, red teams expose vulnerabilities in a controlled setting, giving defenders (the blue team) a chance to understand the risks and shore up their defenses before any real threat occurs.

Human-led red teaming, with its creative, adaptable approach, excels in testing complex systems that require in-depth analysis and insight. However, this approach demands considerable time, specialized expertise, and resources, limiting its frequency and scalability—two critical factors when threats are continuously evolving.

Enter Automated AI Red Teaming

Automated AI red teaming addresses these challenges by adding a fast, scalable, and repeatable layer of defense. While human-led red teams may conduct a full attack simulation once per quarter, automated tools can operate continuously, uncovering new vulnerabilities as they arise.

With automated AI red teaming, security can be maintained in a data-driven environment that requires constant vigilance. Routine scans monitor critical systems, while ad hoc scans can be deployed for specific events like system updates or emerging threats. This shifts red teaming from periodic testing to a continuous security practice, offering resilience that’s difficult to match with manual methods alone.

Human vs. Automated AI Red Teaming: When to Use Each

Each approach—human-led and automated—has strengths, and knowing when to deploy each is key to comprehensive security.

• Human-Led Red Teaming: Skilled professionals bring creative attack strategies that automated systems can’t easily replicate. Human-led teams are particularly valuable for testing complex infrastructure and assessing risks that require adaptive thinking. For example, a team might find a vulnerability in employee practices or facility security—scenarios beyond the scope of automation.
• Automated AI Red Teaming: Automation is ideal for achieving fast, broad coverage across systems, particularly in AI-driven environments where innovation outpaces manual testing. Automated tools handle routine but essential checks, adapting as systems evolve to provide a consistent layer of defense.

By combining both methods, organizations benefit from the speed and efficiency of automation, reserving human-led red teaming for targeted, nuanced analysis that dives deep into system intricacies. This ultimately accelerates AI adoption and deployment of use cases into production.

Key Benefits of HiddenLayer’s Automated Red Teaming for AI

Automated red teaming brings critical capabilities that make security testing continuous and scalable, enabling teams to protect complex systems with ease:

• Unified Results Access: Real-time visibility into vulnerabilities and impacts allows both red and blue teams to work collaboratively on remediation, streamlining the process.
• Collaborative Test Development: Red teams can design attack scenarios informed by real-world threats, integrating blue team insights to create a realistic testing environment.
• Centralized Platform: Built directly into the AISEC Platform to simulate adversarial attacks on Generative AI systems, enabling teams to identify vulnerabilities and strengthen defenses proactively.
• Progress Tracking & Metrics: Automated tools provide metrics that track security posture over time, giving teams measurable insights into their efforts and progress.
• Scalability for Expanding AI Systems: As new AI models are added or scaled, automated red teaming grows alongside, ensuring comprehensive testing as systems expand.
• Cost and Time Savings: Automation reduces manual labor for routine testing, saving on resources while accelerating vulnerability detection and minimizing potential fallout.
• Ad Hoc and Scheduled Scans: Flexibility in scheduling scans allows for regular vulnerability checks or targeted scans triggered by events like system updates, ensuring no critical checks are missed.

Embracing the Future of Red Teaming

Automated AI red teaming isn’t just a technological advancement; it’s a shift in security strategy. By balancing the high-value insights that only human teams can provide with the efficiency and adaptability of automation, organizations can defend against evolving threats with comprehensive strength.

With HiddenLayer’s Automated Red Teaming for AI, security teams gain expert-level vulnerability testing in one click, eliminating the complexity of manual assessments. Our solution leverages industry-leading AI research to simulate real-world attacks, enabling teams to secure AI assets proactively, stay on schedule, and protect AI infrastructure with resilience.

Learn More about Automated Red Teaming for AI 

Attend our Webinar “Automated Red Teaming for AI Explained”

Today, AI is woven into everyday technology, driving everything from personalized recommendations to critical healthcare diagnostics. But what happens if the data feeding these AI models is tampered with? This is the risk posed by AI data poisoning—a targeted attack where someone intentionally manipulates training data to disrupt how AI systems operate. Far from science fiction, AI data poisoning is a growing digital security threat that can have real-world impacts on everything from personal safety to financial stability.

What is AI Data Poisoning?

AI data poisoning refers to an attack where harmful or deceptive data is mixed into the dataset used to train a machine learning model. Because the model relies on training data to “learn” patterns, poisoning it can skew its behavior, leading to incorrect or even dangerous decisions. Imagine, for example, a facial recognition system that fails to correctly identify individuals because of poisoned data or a financial fraud detection model that lets certain transactions slip by unnoticed.

Data poisoning can be especially harmful because it can go undetected and may be challenging to fix once the model has been trained. It’s a way for attackers to influence AI, subtly making it malfunction without obvious disruptions.

How Does Data Poisoning Work?

In AI, the quality and accuracy of the training data determine how well the model works. When attackers manipulate the training data, they can cause models to behave in ways that benefit them. Here are the main ways they do it:

1. Degrading Model Performance: Here, the attacker aims to make the model perform poorly overall. By introducing noisy, misleading, or mislabeled data, they can make the model unreliable. This might cause an image recognition model, for example, to misidentify objects.
2. Embedding Triggers in the Model (Backdoors): In this scenario, attackers hide specific patterns or “triggers” within the model. When these patterns show up during real-world use, they make the model behave in unexpected ways. Imagine a sticker on a stop sign that confuses an autonomous vehicle, making it think it’s a yield sign and not a stop.
3. Biasing the Model’s Decisions: This type of attack pushes the model to favor certain outcomes. For instance, if a hiring algorithm is trained on poisoned data, it might show a preference for certain candidates or ignore qualified ones, introducing bias into the process.

Why Should the Public Care About AI Data Poisoning?

Data poisoning may seem technical, but it has real-world consequences for anyone using technology. Here’s how it can affect everyday life:

• Healthcare: AI models are increasingly used to assist in diagnosing conditions and recommending treatments. Patients might receive incorrect or harmful medical advice if these models are trained on poisoned data.
• Finance: AI powers many fraud detection systems and credit assessments. Poisoning these models could allow fraudulent transactions to bypass security systems or skew credit assessments, leading to unfair financial outcomes.
• Security: Facial recognition and surveillance systems used for security are often AI-driven. Poisoning these systems could allow individuals to evade detection, undermining security efforts.

As AI becomes more integral to our lives, the need to ensure that these systems are reliable and secure grows. Data poisoning represents a direct threat to this reliability.

Recognizing a Poisoned Model

Detecting data poisoning can be challenging, as the malicious data often blends in with legitimate data. However, researchers look for these signs:

1. Unusual Model Behavior: If a model suddenly begins making strange or obviously incorrect predictions after being retrained, it could be a red flag.
2. Performance Drops: Poisoned models might start struggling with tasks they previously handled well.
3. Sensitive to Certain Inputs: Some models may be more likely to make specific errors, especially when particular “trigger” inputs are present.

While these signs can be subtle, it’s essential to catch them early to ensure the model performs as intended.

How Can AI Systems Be Protected?

Combatting data poisoning requires multiple layers of defense:

1. Data Validation: Regularly validating the data used to train AI models is essential. This may involve screening for unusual patterns or inconsistencies in the data.
2. Robustness Testing: By stress-testing models with potential adversarial scenarios, AI engineers can determine if the model is overly sensitive to specific inputs or patterns that could indicate a backdoor.
3. Continuous Monitoring: Real-time monitoring can detect sudden performance drops or unusual behavior, allowing timely intervention.
4. Redundant Datasets: Using data from multiple sources can reduce the chance of contamination, making it harder for attackers to poison a model fully.
5. Evolving Defense Techniques: Just as attackers develop new poisoning methods, defenders constantly improve their strategies to counteract them. AI security is a dynamic field, with new defenses being tested and implemented regularly.

How Can the Public Play a Role in Security for AI?

Although AI security often falls to specialists, the general public can help foster a safer AI landscape:

• Support AI Security Standards: Advocate for stronger regulations and transparency in AI, which encourage better practices for handling and protecting data.
• Stay Informed: Understanding how AI systems work and the potential threats they face can help you ask informed questions about the technologies you use.
• Report Anomalies: If you notice an AI-powered application behaving in unexpected or problematic ways, reporting these issues to the developers helps improve security.

Building a Secure AI Future

AI data poisoning highlights the importance of secure, reliable AI systems. While it introduces new threats, AI security is evolving to counter these dangers. By understanding AI data poisoning, we can better appreciate the steps needed to build safer AI systems for the future.

When well-secured, AI can continue transforming industries and improving lives without compromising security or reliability. With the right safeguards and informed users, we can work toward an AI-powered future that benefits us all.

Introduction

Artificial intelligence (AI) has become a central part of our digital society, influencing everything from healthcare to transportation, finance, and beyond. The European Union (EU) has recognized the need to regulate AI technologies to protect citizens, foster innovation, and ensure that AI systems align with European values of privacy, safety, and accountability. In this context, the EU AI Act is the world’s first comprehensive legal framework for AI. The legislation aims to create an ecosystem of trust in AI while balancing the risks and opportunities associated with its development.

What is the EU AI Act?

The European Commission first proposed the EU AI Act in April 2021. The act seeks to regulate the development, commercialization, and use of AI technologies across the EU. It adopts a risk-based approach to AI regulation, classifying AI applications into different risk categories based on their potential impact on individuals and society.

The legislation covers all stakeholders in the AI supply chain, including developers, deployers, and users of AI systems. This broad scope ensures that the regulation applies to various sectors, from public institutions to private companies, whether they are based in the EU or simply providing AI services within the EU’s jurisdiction.

When Will the EU AI Act be Enforced?

As of August 1st, 2024, the EU AI act entered into force. Member States have until August 2nd, 2025, to designate national competent authorities to oversee the application of the rules for AI systems and carry out market review activities. The Commission's AI Office will be the primary implementation body for the AI Act at EU level, as well as the enforcer for the rules for general-purpose AI models. Companies not complying with the rules will be fined. Fines could go up to 7% of the global annual turnover for violations of banned AI applications, up to 3% for violations of other obligations, and up to 1.5% for supplying incorrect information. The majority of the rules of the AI Act will start applying on August 2nd, 2026. However, prohibitions of AI systems deemed to present an unacceptable risk will apply after six months, while the rules for so-called General-Purpose AI models will apply after 12 months. To bridge the transitional period before full implementation, the Commission has launched the AI Pact. This initiative invites AI developers to voluntarily adopt key obligations of the AI Act ahead of the legal deadlines.

Key Provisions of the EU AI Act

The EU AI Act divides AI systems into four categories based on their potential risks:

1. Unacceptable Risk AI Systems: AI systems that pose a significant threat to individuals’ safety, livelihood, or fundamental rights are banned outright. These include AI systems used for mass surveillance, social scoring (similar to China’s controversial social credit system), and subliminal manipulation that could harm individuals.
2. High-Risk AI Systems: These systems have a substantial impact on people’s lives and are subject to stringent requirements. Examples include AI applications in critical infrastructure (like transportation), education (such as AI used in admissions or grading), employment (AI used in hiring or promotion decisions), law enforcement, and healthcare (diagnostic tools). High-risk AI systems must meet strict requirements for risk management, transparency, human oversight, and data quality.
3. Limited Risk AI Systems: AI systems that do not pose a direct threat but still require transparency fall into this category. For instance, chatbots or AI-driven customer service systems must clearly inform users that they are interacting with an AI system. This transparency requirement ensures that users are aware of the technology they are engaging with.
4. Minimal Risk AI Systems: The majority of AI applications, such as spam filters or AI used in video games, fall under this category. These systems are largely exempt from the new regulations and can operate freely, as their potential risks are deemed negligible.

Positive Goals of the EU AI Act

The EU AI Act is designed to protect consumers while encouraging innovation in a regulated environment. Some of its key positive outcomes include:

1. Enhanced Trust in AI Technologies: By setting clear standards for transparency, safety, and accountability, the EU aims to build public trust in AI. People should feel confident that the AI systems they interact with comply with ethical guidelines and protect their fundamental rights. The transparency rules, in particular, help ensure that AI is used responsibly, whether in hiring processes, healthcare diagnostics, or other critical areas.
2. Harmonization of AI Standards Across the EU: The act will harmonize AI regulations across all member states, providing a single market for AI products and services. This eliminates the complexity for companies trying to navigate different regulations in each EU country. For European AI developers, this reduces barriers to scaling products across the continent, thereby promoting innovation.
3. Human Oversight and Accountability: High-risk AI systems will need to maintain human oversight, ensuring that critical decisions are not left entirely to algorithms. This human-in-the-loop approach aims to prevent scenarios where automated systems make life-changing decisions without proper human review. Whether in law enforcement, healthcare, or employment, this oversight reduces the risks of bias, discrimination, and errors.
4. Fostering Responsible Innovation: While setting guardrails around high-risk AI, the act allows for lower-risk AI systems to continue developing without heavy restrictions. This balance encourages innovation, particularly in sectors where AI’s risks are limited. By focusing regulation on the areas of highest concern, the act promotes responsible technological progress.

Potential Negative Consequences for Innovation

While the EU AI Act brings many benefits, it also raises concerns, particularly regarding its impact on innovation:

1. Increased Compliance Costs for Businesses: For companies developing high-risk AI systems, the act’s stringent requirements on risk management, documentation, transparency, and human oversight will lead to increased compliance costs. Small and medium-sized enterprises (SMEs), which often drive innovation, may struggle with these financial and administrative burdens, potentially slowing down AI development. Large companies with more resources might handle the regulations more easily, leading to less competition in the AI space.
2. Slower Time-to-Market: With the need for extensive testing, documentation, and third-party audits for high-risk AI systems, the time it takes to bring AI products to market may be significantly delayed. In fast-moving sectors like technology, these delays could mean European companies fall behind global competitors, especially those from less-regulated regions like the U.S. or China.
3. Impact on Startups and AI Research: Startups and research institutions may find it challenging to meet the requirements for high-risk AI systems due to limited resources. This could discourage experimentation and the development of innovative solutions, particularly in areas where AI might provide transformative benefits. The potential chilling effect on AI research could slow the development of cutting-edge technologies that are crucial to the EU’s digital economy.
4. Global Competitive Disadvantage: While the EU is leading the charge in regulating AI, the act might place European companies at a disadvantage globally. In less-regulated markets, companies may be able to develop and deploy AI systems more rapidly and with fewer restrictions. This could lead to a scenario where non-EU firms outpace European companies in innovation, limiting the EU’s competitiveness on the global stage.

Conclusion

The EU AI Act represents a landmark effort to regulate artificial intelligence in a way that balances its potential benefits with its risks. By taking a risk-based approach, the EU aims to protect citizens’ rights, enhance transparency, and foster trust in AI technologies. At the same time, the act's stringent requirements for high-risk AI systems raise concerns about its potential to stifle innovation, particularly for startups and SMEs.

As the legislation moves closer to being fully enforced, businesses and policymakers will need to work together to ensure that the EU AI Act achieves its objectives without slowing down the technological progress that is vital for Europe’s future. While the act’s long-term impact remains to be seen, it undoubtedly sets a global precedent for how AI can be regulated responsibly in the digital age.

On July 29th, 2024, the National Institute of Standards and Technology (NIST) released critical guidance that outlines best practices for managing cybersecurity risks associated with AI models. This guidance directly ties into several comments we submitted during the open comment periods, highlighting areas where HiddenLayer effectively addresses emerging cybersecurity challenges.

Understanding and Mitigating Threat Profiles

Practice 1.2 emphasizes the importance of assessing the impact of various threat profiles on public safety if a malicious actor misuses an AI model. Evaluating how AI models can be exploited to increase the scale, reduce costs, or improve the effectiveness of malicious activities is crucial. HiddenLayer can play a pivotal role here by offering advanced threat modeling and risk assessment tools that enable organizations to identify, quantify, and mitigate the potential harm threat actors could cause using AI models. By providing insights into how these harms can be prevented or managed outside the model context, we help organizations develop robust defensive strategies.

Roadmap for Managing Misuse Risks

Practice 2.2 calls for establishing a roadmap to manage misuse risks, particularly for developing foundation models and future versions. Our services can support organizations in defining clear security goals and implementing necessary safeguards to protect against misuse. We provide a comprehensive security framework that includes the development of security practices tailored to specific AI models, ensuring that organizations can adjust their deployment strategies when misuse risks escalate beyond acceptable levels.

Model Theft and Security Practices

As outlined in Practices 3.1, 3.2, and 3.3, model theft is a significant concern. HiddenLayer offers a suite of security tools designed to protect AI models from theft, including advanced cybersecurity red teaming and penetration testing. Organizations can better protect their intellectual property by assessing the risk of model theft from various threat actors and implementing robust security practices. Our tools are designed to scale security measures in proportion to the model's risk, ensuring that insider threats and external attacks are effectively mitigated.

Red Teaming and Misuse Detection

In Practice 4.2, NIST emphasizes the importance of using red teams to assess potential misuse. HiddenLayer provides access to teams that specialize in testing AI models in realistic deployment contexts. This helps organizations verify that their models are resilient against potential misuse, ensuring that their security measures are up to industry standards.

Proportional Safeguards and Deployment Decisions

Practices 5.2 and 5.3 focus on implementing safeguards proportionate to the model’s misuse risk and making informed deployment decisions based on those risks. HiddenLayer offers dynamic risk assessment tools that help organizations evaluate whether their safeguards are sufficient before proceeding with deployments. We also provide support in adjusting or delaying deployments until the necessary security measures are in place, minimizing the risk of misuse.

Monitoring for Misuse

Continuous monitoring of distribution channels for evidence of misuse, as recommended in Practice 6.1, is a critical component of AI model security. HiddenLayer provides automated tools that monitor APIs, websites, and other distribution channels for suspicious activity. Integrating these tools into an organization’s security infrastructure enables real-time detection and response to potential misuse, ensuring that malicious activities are identified and addressed promptly.

Transparency and Accountability

In line with Practice 7.1, we advocate for transparency in managing misuse risks. HiddenLayer enables organizations to publish detailed transparency reports that include key information about the safeguards in place for AI models. By sharing methodologies, evaluation results, and data relevant to assessing misuse risk, organizations can demonstrate their commitment to responsible AI deployment and build trust with stakeholders.

Governance and Risk Management in AI

NIST’s guidance also includes comprehensive recommendations on governance, as outlined in GOVERN Practices 1.2 to 6.2. HiddenLayer supports the integration of trustworthy AI characteristics into organizational policies and risk management processes. We help organizations establish clear policies for monitoring and reviewing AI systems, managing third-party risks, and ensuring compliance with legal and regulatory requirements.

Adversarial Testing and Risk Assessment

Regular adversarial testing and risk assessment, as discussed in MAP Practices 2.3 to 5.1, are essential for identifying vulnerabilities in AI systems. HiddenLayer provides tools for adversarial role-playing exercises, red teaming, and chaos testing, helping organizations identify and address potential failure modes and threats before they can be exploited.

Measuring and Managing AI Risks

The MEASURE and MANAGE practices emphasize the need to evaluate AI system security, resilience, and privacy risks continuously. HiddenLayer offers a comprehensive suite of tools for measuring AI risks, including content provenance analysis, security metrics, and privacy risk assessments. By integrating these tools into their operations, organizations can ensure that their AI systems remain secure, reliable, and compliant with industry standards.

Conclusion

NIST's July 2024 guidance underscores the critical importance of robust cybersecurity practices in AI model development and deployment. HiddenLayer and its services are uniquely positioned to help organizations navigate these challenges, offering advanced tools and expertise to manage misuse risks, protect against model theft, and ensure the security and integrity of AI systems. By aligning with NIST's recommendations, we empower organizations to deploy AI responsibly, safeguarding both their intellectual property and the public's trust.

Introduction

As we’ve covered previously, AI red teaming is a highly effective means of assessing and improving the security of AI systems. The term “red teaming” appears many times throughout recent public policy briefings regarding AI, including: 

• Voluntary commitments made by leading AI companies to the US Government
• President Biden’s executive order regarding AI security
• A briefing introducing the UK Government’s AI Safety Institute
• The EU Artificial Intelligence Act

Unfortunately, the term “red teaming” is currently doing triple duty in conversations about security for AI, which can be confusing. In this post, we tease apart these three different types of AI red teaming. Each type plays a crucial but distinct role in improving security for AI. Using precise language is an important step towards building a mature ecosystem of AI red teaming services.

Adversary Simulation: Identifying Vulnerabilities in Deployed AI

It is often highly informative to simulate the tactics, techniques, and procedures of threat actors who target deployed AI systems and seek to make the AI behave in ways it wasn’t intended to behave. This type of red teaming engagement might include efforts to alter (e.g., injecting ransomware into a machine learning model file), bypass (e.g., crafting adversarial examples), and steal the model using a carefully crafted sequence of queries. It could also include attacks specific to LLMs, such as various types of prompt injections and jailbreaking.

This type of red teaming is the most common and widely applicable. In nearly all cases where an organization uses AI for a business critical function, it is wise to perform regular, comprehensive stress testing to minimize the chances that an adversary could compromise the system. Here is an illustrative example of this style of red teaming applied by HiddenLayer to a model used by a client in the financial services industry.

In contrast, the second and third categories of AI red teaming are almost always performed on frontier AI labs and frontier models trained by those labs. By “frontier AI model,” we mean a model with state-of-the-art performance on key capabilities metrics. A “frontier AI lab” is a company that actively works to research, design, train, and deploy frontier AI models. For example, DeepMind is a frontier AI lab, and their current frontier model is the Gemini 1.5 model family.

Model Evaluations: Identifying Dangerous Capabilities in Frontier Models

Given compute budget C, training dataset size T, and number of model parameters P, scaling laws can be used to gain a fairly accurate prediction of the overall level of performance (averaged across a wide variety of tasks) that a large generative model will achieve once it has been trained. On the other hand, the level of performance the model will achieve on any particular task appears to be difficult to predict (although this has been disputed). It would be incredibly useful both for frontier AI labs and for policymakers if there were standardized, accurate, and reliable tests that could be performed to measure specific capabilities in large generative models.

High-quality tests for measuring the degree to which a model possesses dangerous capabilities, such as CBRN (chemical, biological, radiological, and nuclear) and offensive cyber capabilities, are of particular interest. Every time a new frontier model is trained, it would be beneficial to be able to answer the following question: To what extent would white box access to this model increase a bad actor’s ability to do harm at a scale above and beyond what they could do just with access to the internet and textbooks? Regulators have been asking for these tests for months:

• Voluntary AI commitments to the White House

“Commit to internal and external red-teaming of models or systems in areas including misuse, societal risks, and national security concerns, such as bio, cyber, and other safety areas.”

• President Biden’s executive order on AI security

Companies must provide the Federal Government with “the results of any red-team testing that the company has conducted relating to lowering the barrier to entry for the development, acquisition, and use of biological weapons by non-state actors; the discovery of software vulnerabilities and development of associated exploits. . .”

• UK AI Safety Institute

“Dual-use capabilities: As AI systems become more capable, there could be an increased risk that

malicious actors could use these systems as tools to cause harm. Evaluations will gauge the

capabilities most relevant to enabling malicious actors, such as aiding in cyber-criminality,

biological or chemical science, human persuasion, large-scale disinformation campaigns, and

weapons acquisition.”

Frontier AI labs are also investing heavily in the development of internal model evaluations for dangerous capabilities:

• Anthropic’s Responsible Scaling Policy

“As AI models become more capable, we believe that they will create major economic and social value, but will also present increasingly severe risks. Our RSP focuses on catastrophic risks – those where an AI model directly causes large scale devastation. Such risks can come from deliberate misuse of models (for example use by terrorists or state actors to create bioweapons)...”

• OpenAI Preparedness Framework

“We believe that frontier AI models, which will exceed the capabilities currently present in the most advanced existing models, have the potential to benefit all of humanity. But they also pose increasingly severe risks. Managing the catastrophic risks from frontier AI will require answering questions like: How dangerous are frontier AI systems when put to misuse, both now and in the future? How can we build a robust framework for monitoring, evaluation, prediction, and protection against the dangerous capabilities of frontier AI systems? If our frontier AI model weights were stolen, how might malicious actors choose to leverage them?”

• DeepMind Frontier Safety Framework

“Identifying capabilities a model may have with potential for severe harm. To do this, we research the paths through which a model could cause severe harm in high-risk domains, and then determine the minimal level of capabilities a model must have to play a role in causing such harm.”

A healthy, truth-seeking debate about the level of risk from misuse of advanced AI will be critical for navigating mitigation measures that are proportional to the risk while not hindering innovation. That being said, here are a few reasons why frontier AI labs and governing bodies are dedicating a lot of attention and resources to dangerous capabilities evaluations for frontier AI:

• Developing a mature science of measurement for frontier model capabilities will likely take a lot of time and many iterations to figure out what works and what doesn’t. Getting this right requires planning ahead so that if and when models with truly dangerous capabilities arrive, we will be well-equipped to detect these capabilities and avoid allowing the model to land in the wrong hands.

• Many desirable AI capabilities fall under the definition of “dual-use,” meaning that they can be leveraged for both constructive and destructive aims. For example, in order to be useful for aiding in cyber threat mitigation, a model must learn to understand computer networking, cyber threats, and computer vulnerabilities. This capability can be put to use by threat actors seeking to attack computer systems.

• Frontier AI labs have already begun to develop dangerous capabilities evaluations for their respective models, and in all cases beginning signs of dangerous capabilities were detected.

• Anthropic: “Taken together, we think that unmitigated LLMs could accelerate a bad actor’s efforts to misuse biology relative to solely having internet access, and enable them to accomplish tasks they could not without an LLM. These two effects are likely small today, but growing relatively fast. If unmitigated, we worry that these kinds of risks are near-term, meaning that they may be actualized in the next two to three years, rather than five or more.”

• OpenAI: “Overall, especially given the uncertainty here, our results indicate a clear and urgent need for more work in this domain. Given the current pace of progress in frontier AI systems, it seems possible that future systems could provide sizable benefits to malicious actors. It is thus vital that we build an extensive set of high-quality evaluations for biorisk (as well as other catastrophic risks), advance discussion on what constitutes ‘meaningful’ risk, and develop effective strategies for mitigating risk.”

• DeepMind: “More broadly, the stronger models exhibited at least rudimentary abilities across all our evaluations, hinting that dangerous capabilities may emerge as a byproduct of improvements in general capabilities. . . We commissioned a group of professional forecasters to predict when models will first obtain high scores on our evaluations, and their median estimates were between 2025 and 2029 for different capabilities.”

NIST recently published a draft of a report on mitigating risks from the misuse of foundation models. They emphasize two key properties that model evaluations should have: (1) Threat actors will almost certainly expand the level of capabilities of a frontier model by giving it access to various tools such as a Python interpreter, an Internet search engine, and a command prompt. Therefore, models should be given access to the best tools available during the evaluation period. Even if a model by itself can’t complete a task that would be indicative of dangerous capabilities, that same model with access to tools may be more than capable. (2) The evaluations must not be leaked into the model’s training data, or else the dangerous capabilities of the model could be overestimated.

Adversary Simulation: Stealing Frontier Model Weights

Whereas the first two types of AI red teaming are relatively new (especially model evaluations), the third type involves applying tried and true network, human, and physical red teaming to the information security controls put in place by frontier AI labs to safeguard frontier model weights. Frontier AI labs are thinking hard about how to prevent model weight theft:

• Anthropic’s Responsible Scaling Policy

“Harden security such that non-state attackers are unlikely to be able to steal model weights and advanced threat actors (e.g., states) cannot steal them without significant expense.”

• OpenAI’s remarks on securing AI research infrastructure

“Here, we outline our current architecture and operations that support the secure training of frontier models at scale. This includes measures designed to protect sensitive model weights within a secure environment for AI innovation.”

• DeepMind’s Frontier Safety Framework

“To allow us to tailor the strength of the mitigations to each [Critical Capability Level], we have also outlined a set of security and deployment mitigations. Higher level security mitigations result in greater protection against the exfiltration of model weights…”

What are model weights, and why are frontier labs so keen on preventing them from being stolen? Model weights are simply numbers that encode the entirety of what was learned during the training process. To “train” a machine learning model is to iteratively tune the values of the model’s weights such that the model performs better and better on the training task.

Frontier models have a tremendous number of weights (for example, GPT-3 has approximately 175 billion weights), and more weights require more time and money to learn. If an adversary were to steal the files containing the weights of a frontier AI model (either through traditional cyber threat operations, social engineering of employees, or gaining physical access to a frontier lab’s computing infrastructure), that would amount to intellectual property theft of tens or even hundreds of millions of dollars.

Additionally, recall that Anthropic, OpenAI, DeepMind, the White House, and the UK AI Safety Institute, among many others, believe it is plausible that scaling up frontier generative models could create both incredibly helpful and destructive capabilities. Ensuring that model weights stay on secure servers closes off one of the major routes by which a bad actor could unlock the full offensive capabilities of these future models. The effects of safety fine-tuning techniques such as reinforcement learning from human feedback (RLHF) and Constitutional AI are encoded in the model’s weights and put up a barrier against asking the stolen model to aid in harming. But this barrier is flimsy in the face of techniques such as LoRA and directional ablation that can be used to quickly, cheaply, and surgically remove these safeguards. A threat actor with access to a model’s weights is a threat actor with full access to any and all offensive capabilities the model may have learned during training.

A recent report from RAND takes a deep dive into this particular threat model and lays out what it might take to prevent even highly resourced and cyber-capable state actors from stealing frontier model weights. The term “red-team” appears 26 times in the report. To protect their model weights, “OpenAI uses internal and external red teams to simulate adversaries and test our security controls for the research environment.”

Note the synergy between the second and third types of AI red teaming. A mature science of model evaluations for dangerous capabilities would allow policymakers and frontier labs to make more informed decisions about what level of public access is proportional to the risks posed by a given model, as well as what intensity of red teaming is necessary to ensure that the model’s weights remain secure. If we can’t know with a high degree of confidence what a model is capable of, we run the risk of locking down a model that turns out to have no dangerous capabilities and forfeiting the scientific benefits of allowing at least somewhat open access to that model, including valuable research on making AI more secure that is enabled by white-box access to frontier models. The other, much more sinister side of the coin is that we could put up too few controls around the weights of a model that we erroneously believe to possess no dangerous capabilities, only to later have the previously latent offensive firepower of that model aimed at us by a threat actor.

Conclusion

As frontier labs and policy makers have been correct in emphasizing, AI red teaming is one of the most powerful tools at our disposal for enhancing the security of AI systems. However, the language currently used in these conversations obscures the fact that AI red teaming is not just a single approach; rather, it involves three distinct strategies, each addressing different security needs.:

1. Simulating adversaries who seek to alter, bypass, or steal (through inference-based attacks) a model deployed in a business-critical context is an invaluable method of discovering and remediating vulnerabilities. AI red teaming, especially when tailored to large language models (LLM red teaming), provides a focused approach to identifying potential weaknesses and developing strategies to safeguard these systems against misuse and exploitation.

2. Advancing the science of measuring dangerous capabilities in frontier AI models is critical for policy makers and frontier AI labs who seek to apply regulations and security controls that are proportional to the risks from misuse posed by a given model.

3. Traditional network, human, and physical red teaming with the objective of stealing frontier model weights from frontier AI labs is an indispensable tool for assessing the readiness of frontier labs to prevent bad actors from taking and abusing their most powerful dual-use models.

Contact us here to start a conversation about AI red teaming for your organization.

Introduction

As AI continues to evolve at a fast pace, implementing comprehensive security measures is vital for trust and accountability. The integration of AI into essential business operations and society underscores the necessity for proactive security strategies. While challenges and concerns exist, there is significant potential for leaders to make strategic, informed decisions. By pursuing clear, actionable guidance and staying well-informed, organizational leaders can effectively navigate the complexities of security for AI. This proactive stance will help reduce risks, ensure the safe and responsible use of AI technologies, and ultimately promote trust and innovation.

In this final installment, we will explore essential topics for comprehensive AI systems: data security and privacy, model validation, secure development practices, continuous monitoring, and model explainability. Key areas include encryption, access controls, anonymization, and evaluating third-party vendors for security compliance. We will emphasize the importance of red teaming training, which simulates adversarial attacks to uncover vulnerabilities. Techniques for adversarial testing and model validation will be discussed to ensure AI robustness. Embedding security best practices throughout the AI development lifecycle and implementing continuous monitoring with a strong incident response strategy are crucial.

This guide will provide you with the necessary tools and strategies to fortify your AI systems, making them resilient against threats and reliable in their operations. Follow our series as we cover understanding AI environments, governing AI systems, strengthening AI systems, and staying up-to-date on AI developments.

Step 1: User Training and Awareness

Continuous education is vital. Conduct regular training sessions for developers, data scientists, and IT staff on security best practices for AI. Training should cover topics such as secure coding, data protection, and threat detection. An informed team is your first line of defense against security threats.

Raise awareness across the organization about security for AI risks and mitigation strategies. Knowledge is power, and an aware team is a proactive team. Regular workshops, seminars, and awareness campaigns help keep security top of mind for all employees.

Who Should Be Responsible and In the Room:

• Training and Development Team: Organizes and conducts regular training sessions for developers, data scientists, and IT staff on security for AI best practices.
• AI Development Team: Participates in training on secure coding, data protection, and threat detection to stay updated on the latest security measures.
• Data Scientists: Engages in ongoing education to understand and implement data protection and threat detection techniques.
• IT Staff: Receives training on security for AI best practices to ensure strong implementation and maintenance of security measures.
• Security Team: Provides expertise and updates on the latest security for AI threats and mitigation strategies during training sessions and awareness campaigns.

Step 2: Third-Party Audits and Assessments

Engage third-party auditors to review your security for AI practices regularly. Fresh perspectives can identify overlooked vulnerabilities and provide unbiased assessments of your security posture. These auditors bring expertise from a wide range of industries and can offer valuable insights that internal teams might miss. Audits should cover all aspects of security for AI, including data protection, model robustness, access controls, and compliance with relevant regulations. A thorough audit assesses the entire lifecycle of AI deployment, from development and training to implementation and monitoring, ensuring comprehensive security coverage.

Conduct penetration testing on AI systems periodically to find and fix vulnerabilities before malicious actors exploit them. Penetration testing involves simulating attacks on your AI systems to identify weaknesses and improve your defenses. This process can uncover flaws in your infrastructure, applications, and algorithms that attackers could exploit. Regularly scheduled penetration tests, combined with ad-hoc testing when major changes are made to the system, ensure that your defenses are constantly evaluated and strengthened. This proactive approach helps ensure your AI systems remain resilient against emerging threats as new vulnerabilities are identified and addressed promptly.

In addition to penetration testing, consider incorporating other forms of security testing, such as red team exercises and vulnerability assessments, to provide a well-rounded understanding of your security posture. Red team exercises simulate real-world attacks to test the effectiveness of your security measures and response strategies. Vulnerability assessments systematically review your systems to identify and prioritize security risks. Together, these practices create a strong security testing framework that enhances the resilience of your AI systems.

By engaging third-party auditors and regularly conducting penetration testing, you improve your security for AI posture and demonstrate a commitment to maintaining high-security standards. This can enhance trust with stakeholders, including customers, partners, and regulators, by showing that you take proactive measures to protect sensitive data and ensure the integrity of your AI solutions.

Who Should Be Responsible and In the Room:

• Chief Information Security Officer (CISO): Oversees security for AI practices and the engagement with third-party auditors.
• Security Operations Team: Manages security audits and penetration testing, and implements remediation plans.
• IT Security Manager: Coordinates with third-party auditors and facilitates the audit process.
• AI Development Team Lead: Addresses vulnerabilities identified during audits and testing, ensuring strong AI model security.
• Compliance Officer: Ensures security practices comply with regulations and implements auditor recommendations.
• Risk Management Officer: Integrates audit and testing findings into the overall risk management strategy.
• Chief Information Officer (CIO) & Chief Technology Officer (CTO): Provides oversight, resources, and strategic direction for security initiatives.

Step 3: Data Integrity and Quality

Implement strong procedures to ensure the quality and integrity of data used for training AI models. Begin with data quality checks by establishing validation and cleaning processes to maintain accuracy and reliability.

Regularly audit your data to identify and fix any issues, ensuring ongoing integrity. Track the origin and history of your data to prevent using compromised or untrustworthy sources, verifying authenticity and integrity through data provenance.

Maintain detailed metadata about your datasets to provide contextual information, helping assess data reliability. Implement strict access controls to ensure only authorized personnel can modify data, protecting against unauthorized changes.

Document and ensure transparency in all processes related to data quality and provenance. Educate your team on the importance of these practices through training sessions and awareness programs.

Who Should Be Responsible and In the Room:

• Data Quality Team: Manages data validation and cleaning processes to maintain accuracy and reliability.
• Audit and Compliance Team: Conducts regular audits and ensures adherence to data quality standards and regulations.
• Data Governance Officer: Oversees data provenance and maintains detailed records of data origin and history.
• IT Security Team: Implements and manages strict access controls to protect data integrity.
• AI Development Team: Ensures data quality practices are integrated into AI model training and development.
• Training and Development Team: Educates staff on data quality and provenance procedures, ensuring ongoing awareness and adherence.

Step 4: Security Metrics and Reporting

Define and monitor key security metrics to gauge the effectiveness of your security for AI measures. Examples include the number of detected incidents, response times, and the effectiveness of security controls.

Review and update these metrics regularly to stay relevant to current threats. Benchmark against industry standards and set clear goals for continuous improvement. Implement automated tools for real-time monitoring and alerts.

Establish a clear process for reporting security incidents, ensuring timely and accurate responses. Incident reports should detail the nature of the incident, affected systems, and resolution steps. Train relevant personnel on these procedures.

Conduct root cause analysis for incidents to prevent future occurrences, building a resilient security framework. To maintain transparency and a proactive security culture, communicate metrics and incident reports regularly to all stakeholders, including executive leadership.

Who Should Be Responsible and In the Room:

• Chief Information Security Officer (CISO): Oversees the overall security strategy and ensures the relevance and effectiveness of security metrics.
• Security Operations Team: Monitors security metrics, implements automated tools, and manages real-time alerts.
• Data Scientists: Analyze security metrics data to provide insights and identify trends.
• IT Security Manager: Coordinates the reporting process and ensures timely and accurate incident reports.
• Compliance and Legal Team: Ensures all security measures and incident reports comply with relevant regulations.
• Chief Information Officer (CIO) & Chief Technology Officer (CTO): Reviews security metrics and incident reports to maintain transparency and support proactive security measures.

Step 5: AI System Lifecycle Management

Manage AI systems from development to decommissioning, ensuring security at every stage of their lifecycle. This comprehensive approach includes secure development practices, continuous monitoring, and proper decommissioning procedures to maintain security throughout their operational lifespan. Secure development practices involve implementing security measures from the outset, incorporating best practices in secure coding, data protection, and threat modeling. Continuous monitoring entails regularly overseeing AI systems to detect and respond to security threats promptly, using advanced monitoring tools to identify anomalies and potential vulnerabilities.

Proper decommissioning procedures are crucial when retiring AI systems. Follow stringent processes to securely dispose of data and dismantle infrastructure, preventing unauthorized access or data leaks. Clearly defining responsibilities ensures role clarity, making lifecycle management cohesive and strong. Effective communication is essential, as it fosters coordination among team members and strengthens your AI systems' overall security and reliability.

Who Should Be Responsible and In the Room:

• Chief Information Security Officer (CISO): Oversees the entire security strategy and ensures all stages of the AI lifecycle are secure.
• AI Development Team: Implements secure development practices and continuous monitoring.
• IT Infrastructure Team: Handles the secure decommissioning of AI systems and ensures proper data disposal.
• Compliance and Legal Team: Ensures all security practices meet legal and regulatory requirements.
• Project Manager: Coordinates efforts across teams, ensuring clear communication and role clarity.

Step 6: Red Teaming Training

To enhance the security of your AI systems, implement red teaming exercises. These involve simulating real-world attacks to identify vulnerabilities and test your security measures. If your organization lacks well-trained AI red teaming professionals, it is crucial to engage reputable external organizations, such as HiddenLayer, for specialized AI red teaming training to ensure comprehensive security.

To start the red teaming training, assemble a red team of cybersecurity professionals. Once again, given that your team may not be well-versed in security for AI enlist outside organizations to provide the necessary training. Develop realistic attack scenarios that mimic potential threats to your AI systems. Conduct these exercises in a controlled environment, closely monitor the team's actions, and document each person's strengths and weaknesses.

Analyze the findings from the training to identify knowledge gaps within your team and address them promptly. Use these insights to improve your incident response plan where necessary. Schedule quarterly red teaming exercises to test your team’s progress and ensure continuous learning and improvement.

Integrating red teaming into your security strategy, supported by external training as needed, helps proactively identify and mitigate risks. This ensures your AI systems are robust, secure, and resilient against real-world threats.

Step 7: Collaboration and Information Sharing

Collaborate with industry peers to share knowledge about security for AI threats and best practices. Engaging in information-sharing platforms keeps you informed about emerging threats and industry trends, helping you stay ahead of potential risks. By collaborating, you can adopt best practices from across the industry and enhance your own security measures.

For further guidance, check out our latest blog post, which delves into the benefits of collaboration in securing AI. The blog provides valuable insights and practical advice on how to effectively engage with industry peers to strengthen your security for AI posture.

Conclusion: Securing Your AI Systems Effectively

Securing AI systems is an ongoing, dynamic process that requires a thorough, multi-faceted approach. As AI becomes deeply integrated into the core operations of businesses and society, the importance of strong security measures cannot be overstated. This guide has provided a comprehensive, step-by-step approach to help organizational leaders navigate the complexities of securing AI, from initial discovery and risk assessment to continuous monitoring and collaboration.

By diligently following these steps, leaders can ensure their AI systems are secure but also trustworthy and compliant with regulatory standards. Implementing secure development practices, continuous monitoring, and rigorous audits, coupled with a strong focus on data integrity and collaboration, will significantly enhance the resilience of your AI infrastructure.

At HiddenLayer, we are here to guide and assist organizations in securing their AI systems. Don't hesitate to reach out for help. Our mission is to support you in navigating the complexities of securing AI ensuring your systems are safe, reliable, and compliant. We hope this series helps provide guidance on securing AI systems at your organization. 

Remember: Stay informed, proactive, and committed to security best practices to protect your AI systems and, ultimately, your organization’s future. For more detailed insights and practical advice, be sure to explore our blog post on collaboration in security for AI and our comprehensive Threat Report.

Read the previous installments: Understanding AI Environments, Governing AI Systems, Strengthening AI Systems.

AI Overview

In today’s rapidly evolving artificial intelligence (AI) landscape, securing AI systems has become paramount. As organizations increasingly rely on AI and machine learning (ML) models, ensuring the integrity and security of these models is critical. To address this growing need, HiddenLayer, a pioneer security for AI company, has a scanning solution that enables companies to secure their AI digital supply chain, mitigating the risk of introducing adversarial code into their environment.

The Challenge of Security for AI

AI and ML models are susceptible to various threats, including data poisoning, adversarial attacks, and malware injection. According to HiddenLayer’s AI Threat Landscape 2024 Report, 77% of companies reported breaches to their AI models in the past year, and 75% of IT leaders believe third-party AI integrations pose a significant risk. This highlights the urgent need for comprehensive security measures.

The Solution: AI Model Vulnerability Scan

HiddenLayer provides the advanced scanning technology for one of Optiv’s AI services, the AI Model Vulnerability Scan. This service offers point-in-time scans for vulnerabilities and malware in AI models, leveraging both static and AI techniques to identify security risks.

Key Features and Benefits

• Detection of Compromised Models: The scan detects compromised pre-trained models, ensuring that any models downloaded from public repositories are from reputable sources and free of malicious code.
• Enhanced Security: By incorporating HiddenLayer Model Scanner into their ML Ops pipeline, organizations can secure their entire digital AI supply chain, detect security risks, and ensure the integrity of their operations.
• Visibility into Risks and Attacks: The service provides visibility into potential risks and attacks on large language models (LLMs) and ML operations, enabling organizations to identify vulnerable points of attack.
• Adversarial Attack Detection: The scanner uses MITRE ATLAS tactics and techniques to detect adversarial AI attacks, supplementing the capabilities of your security team with advanced AI security expertise.

“Engineering and product teams are going to market faster than ever with AI and ML solutions. It’s evident that organizations who neglect to test and validate AI models and applications for safety and security run the risk of brand damage, data loss, legal and regulatory action, and general reputational harm,” says Shawn Asmus, Application Security Practice Director at Optiv. “Demonstrating a system is resilient and trustworthy, apart from merely being functional, is what responsible AI is all about.”

HiddenLayer’s Strategic Advantage

HiddenLayer, a Gartner recognized AI Application Security company, is a provider of security solutions for machine learning algorithms, models & the data that power them. With a first-of-its-kind, patented, noninvasive software approach to observing & securing ML, HiddenLayer is helping to protect the world’s most valuable technologies. Trust, flexibility, and comprehensiveness are non-negotiable when it comes to ensuring your business stays ahead in innovation.

Proof Points from HiddenLayer’s AI Threat Landscape 2024 Report

• High Incidence of Breaches: 77% of companies reported breaches to their AI models in the past year.
• Increased Risk from Third-Party Integrations: 75% of IT leaders believe that third-party AI integrations pose greater risks than existing cybersecurity threats.
• Sophistication of Adversarial Attacks: Adversarial attacks such as data poisoning and model evasion are becoming more sophisticated, necessitating advanced defensive strategies and tools.

"Organizations across all verticals and of all sizes are excited about the innovation AI delivers. Given this reality, HiddenLayer is excited to accelerate secure AI adoption by leveraging AI's competitive advantage without the inherent risks associated with its deployment. Using the HiddenLayer Model Scanner, Optiv's AI Model Vulnerability Scan Service allows for enhanced security, improved mitigation, and accelerated innovation to harness the full power of AI."

Abigail Maines, CRO of HiddenLayer

Conclusion

Organizations can secure their AI models and operations against emerging threats by leveraging advanced scanning technology and deep security expertise. This collaboration not only enhances security but also allows organizations to embrace the transformative capabilities of AI with confidence.

Introduction

With AI advancing rapidly, it's essential to implement thorough security measures. The need for proactive security strategies grows as AI becomes more integrated into critical business functions and society. Despite the challenges and concerns, there is considerable potential for leaders to make strategic, informed decisions. Organizational leaders can navigate the complexities of AI security by seeking clear, actionable guidance and staying well-informed. This proactive approach will help mitigate risks, ensure AI technologies' safe and responsible deployment, and ultimately foster trust and innovation.

Strengthening your AI systems is crucial to ensuring their security, reliability, and trustworthiness. Part 3 of our series focuses on implementing advanced measures to secure data, validate models, embed secure development practices, monitor systems continuously, and ensure model explainability and transparency. These steps are essential for protecting sensitive information, maintaining user trust, and complying with regulatory standards. This guide will provide you with the necessary tools and strategies to fortify your AI systems, making them resilient against threats and reliable in their operations. Tune in as we continue to cover understanding AI environments, governing AI systems, strengthening AI systems, and staying up-to-date on AI developments over the next few weeks.

Step 1: Data Security and Privacy

Data is the lifeblood of AI. Deploy advanced security measures tailored to your AI solutions that are adaptable to various deployment environments. This includes implementing encryption, access controls, and anonymization techniques to protect sensitive data. Ensuring data privacy is critical in maintaining user trust and complying with regulations.

Evaluate third-party vendors rigorously. Your vendors must meet stringent security for AI standards. Integrate their security measures into your overall strategy to ensure there are no weak links in your defense. Conduct thorough security assessments and require vendors to comply with your security policies and standards.

Who Should Be Responsible and In the Room:

• Data Security Team: Implements encryption, access controls, and anonymization techniques.
• AI Development Team: Ensures AI solutions are designed with integrated security measures.
• Compliance and Legal Team: Ensures compliance with data privacy regulations.
• Third-Party Vendor Management Team: Evaluates and integrates third-party vendor security measures.
• Chief Information Officer (CIO) & Chief Technology Officer (CTO): Provides oversight and resources for security initiatives.

Step 2: Model Strength and Validation

AI models must be resilient to ensure their reliability and effectiveness. Regularly subject them to adversarial testing to evaluate their systems. This process involves simulating various attacks to identify potential vulnerabilities and assess the model's ability to withstand malicious inputs. By doing so, you can pinpoint weaknesses and fortify the model against potential threats.

Employing thorough model validation techniques is equally essential. These techniques ensure consistent, reliable behavior in real-world scenarios. For example, cross-validation helps verify that the model performs well across different subsets of data, preventing overfitting and ensuring generalizability. Stress testing pushes the model to its limits under extreme conditions, revealing how it handles unexpected inputs or high-load situations.

Both adversarial testing and validation processes are critical for maintaining trust and reliability in your AI outputs. They provide a comprehensive assessment of the model's performance, ensuring it can handle the complexities and challenges of real-world applications. By integrating these practices into your AI development and maintenance workflows, you can build more resilient and trustworthy AI systems.

Who Should Be Responsible and In the Room:

• AI Development Team: Designs and develops the AI models, ensuring strength and the ability to handle adversarial testing.
• Data Scientists: Conduct detailed analysis and validation of the AI models, including cross-validation and stress testing.
• Cybersecurity Experts: Simulate attacks and identify vulnerabilities to test the model's resilience against malicious inputs.
• Quality Assurance (QA) Team: Ensures that the AI models meet required standards and perform reliably under various conditions.
• Chief Information Officer (CIO) & Chief Technology Officer (CTO): Provides oversight, resources, and strategic direction for testing and validation processes.

Step 3: Secure Development Practices

Embed security best practices at every stage of the AI development lifecycle. From inception to deployment, aim to minimize vulnerabilities by incorporating security measures at each step. Start with secure coding practices, ensuring that your code is free from common vulnerabilities and follows the latest security guidelines. Conduct regular code reviews to catch potential security issues early and to maintain high standards of code quality.

Implement comprehensive security testing throughout the development process. This includes static and dynamic code analysis, penetration testing, and vulnerability assessments. These tests help identify and mitigate risks before they become critical issues. Additionally, threat modeling should be incorporated to anticipate potential security threats and design defenses against them.

By embedding these secure development practices, you ensure that security is integrated into your AI systems from the ground up. This proactive approach significantly reduces the risk of introducing vulnerabilities during development, leading to strong and secure AI solutions. It also helps maintain user trust and compliance with regulatory requirements, as security is not an afterthought but a fundamental component of the development lifecycle.

Who Should Be Responsible and In the Room:

• AI Development Team: Responsible for secure coding practices and incorporating security measures into the AI models from the start.
• Security Engineers: Conduct regular code reviews, static and dynamic code analysis, and penetration testing to identify and address security vulnerabilities.
• Cybersecurity Experts: Perform threat modeling and vulnerability assessments to anticipate potential security threats and design appropriate defenses.
• Quality Assurance (QA) Team: Ensures that security testing is integrated into the development process and that security standards are maintained throughout.
• Project Managers: Coordinate efforts across teams, ensuring that security best practices are followed at every stage of the development lifecycle.
• Compliance and Legal Team: Ensures that the development process complies with relevant security regulations and industry standards.
• Chief Information Officer (CIO) & Chief Technology Officer (CTO): Provides oversight, resources, and support for embedding security practices into the development lifecycle.

Step 4: Continuous Monitoring and Incident Response

Implement continuous monitoring systems to detect anomalies immediately to ensure the ongoing security and integrity of your AI systems. Real-time surveillance acts as an early warning system, enabling you to identify and address potential issues before they escalate into major problems. These monitoring systems should be designed to detect a wide range of indicators of compromise, such as unusual patterns in data or system behavior, unauthorized access attempts, and other signs of potential security breaches.

Advanced monitoring tools should employ machine learning algorithms and anomaly detection techniques to identify deviations from normal activity that may indicate a threat. These tools can analyze vast amounts of data in real time, providing comprehensive visibility into the system's operations and enabling swift response to any detected anomalies.

Additionally, integrating continuous monitoring with automated response mechanisms can further enhance security. When an anomaly is detected, automated systems can trigger predefined actions, such as alerting security personnel, isolating affected components, or initiating further investigation procedures. This proactive approach minimizes the time between detection and response, reducing the risk of significant damage.

To effectively implement continuous monitoring systems for immediately detecting anomalies, it's crucial to consider products specifically designed for this purpose. Involving the right stakeholders to evaluate and select these products ensures a strong and effective monitoring strategy. 

Pair continuous monitoring with a comprehensive incident response strategy. Regularly update and rehearse this strategy to maintain readiness against evolving threats, as preparedness is key to effective incident management. An effective incident response plan includes predefined roles and responsibilities, communication protocols, and procedures for containing and mitigating incidents. 

A Ponemon survey found that 77% of respondents lack a formal incident response plan that is consistently applied across their organization, and nearly half say their plan is informal or nonexistent. Don't be part of the 77% who do not have an up-to-date incident response (IR) plan. It's time for security to be proactive rather than reactive, especially regarding AI.

For support on developing an incident response plan, refer to the CISA guide on Incident Response Plan Basics. This guide provides valuable insights into what an IR plan should include and needs.

Step 5: Model Explainability and Transparency

Before you do Step 5, make sure you have fully completed Step 3 on implementing ethical AI guidelines. 

As you know, transparency and explainability are critical, especially when it comes to improving the public’s trust in AI usage. Ensure AI decisions can be interpreted and explained to users and stakeholders. Explainable AI builds trust and ensures accountability by making the decision-making process understandable. Techniques such as model interpretability tools, visualizations, and detailed documentation are essential for achieving this goal.

Regularly publish transparency reports detailing AI system operations and decisions. Transparency is not just about compliance; it’s about fostering an environment of openness and trust. These reports should provide insights into how AI models function, the data they use, and the measures taken to ensure their fairness and reliability.

Who Should Be Responsible and In the Room:

• AI Development Team: Implements model interpretability tools, visualizations, and detailed documentation to make AI decisions interpretable and explainable.
• Data Scientists: Develop techniques and tools for explaining AI models and decisions, ensuring these explanations are accurate and accessible.
• Compliance and Legal Team: Ensures transparency practices comply with relevant regulations and industry standards, providing guidance on legal and ethical requirements.
• Communication and Public Relations Team: Publishes regular transparency reports and communicates AI system operations and decisions to users and stakeholders, fostering an environment of openness and trust.

Conclusion

Strengthening your AI systems requires a multi-faceted approach encompassing data security, model validation, secure development practices, continuous monitoring, and transparency. Organizations can protect sensitive data and ensure compliance with privacy regulations by implementing advanced security measures such as encryption, access controls, and anonymization techniques. Rigorous evaluation of third-party vendors and adversarial testing of AI models further enhance the reliability and resilience of AI systems.

Embedding secure development practices throughout the AI lifecycle, from secure coding to regular security testing, helps minimize vulnerabilities and build strong, secure AI solutions. Continuous monitoring and a well-defined incident response plan ensure that potential threats are detected and addressed promptly, maintaining the integrity of AI systems. Finally, fostering transparency and explainability in AI decisions builds trust and accountability, making AI systems more understandable and trustworthy for users and stakeholders.

By following these comprehensive steps, organizations can create AI systems that are not only secure but also ethical and transparent, ensuring they serve as valuable and reliable assets in today's complex technological landscape. In our last installment, we will dive into audits and how to stay up-to-date on your AI environments. 

Read the previous installments: Understanding AI Environments, Governing AI Systems

Introduction

As AI advances at a rapid pace, implementing comprehensive security measures becomes increasingly crucial. The integration of AI into critical business operations and society is growing, highlighting the importance of proactive security strategies. While there are concerns and challenges surrounding AI, there is also significant potential for leaders to make informed, strategic decisions. Organizational leaders can effectively navigate the complexities of security for AI by seeking clear, actionable guidance and staying informed amidst abundant information. This proactive approach will help mitigate risks and ensure AI technologies' safe and responsible deployment, ultimately fostering trust and innovation.

Effective governance ensures that AI systems are secure, ethical, and compliant with regulatory standards. As organizations increasingly rely on AI, they must adopt comprehensive governance strategies to manage risks, adhere to legal requirements, and uphold ethical principles. This second part of our series on governing AI systems focuses on the importance of defensive frameworks within a broader governance strategy. We explore how leading organizations have developed detailed frameworks to enhance security for AI and guide the development of ethical AI guidelines, ensuring responsible and transparent AI operations. Tune in as we continue to cover understanding AI environments, governing AI systems, strengthening AI systems, and staying up-to-date on AI developments over the next few weeks.

Step 1: Defensive Frameworks

As tools and techniques for attacking AI become more sophisticated, a methodical defensive approach is essential to safeguard AI. Over the past two years, leading organizations have developed comprehensive frameworks to enhance security for AI. Familiarizing yourself with these frameworks is crucial as you build out your secure AI processes and procedures. The following frameworks provide valuable guidance for organizations aiming to safeguard their AI systems against evolving threats.

MITRE ATLAS

MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) is a comprehensive framework launched in 2021, detailing adversarial machine learning tactics, techniques, and case studies. It complements the MITRE ATT&CK framework and includes real-world attacks and red-teaming exercises to provide a complete picture of AI system vulnerabilities.

In 2023, MITRE ATLAS was significantly updated, adding 12 new techniques and 5 unique case studies, focusing on large language models (LLMs) and generative AI systems. Collaborations with Microsoft led to new tools like the Arsenal and Almanac plugins for enhanced threat emulation. The update also introduced 20 new mitigations based on case studies. ATLAS now includes 14 tactics, 82 techniques, 22 case studies, and 20 mitigations, with ongoing efforts to expand its resources. This community-driven approach ensures that ATLAS remains a critical resource for securing AI-enabled systems against evolving threats.

NIST AI Risk Management Framework

Released in January 2023, the NIST AI RMF provides a conceptual framework for responsibly designing, developing, deploying, and using AI systems. It focuses on risk management through four functions: govern, map, measure, and manage.

Google Secure AI Framework (SAIF)

Introduced in June 2023, SAIF offers guidance on securing AI systems by adapting best practices from traditional software development. It emphasizes six core elements: expanding security foundations, automating defenses, and contextualizing AI risks.

OWASP Top 10

In 2023, OWASP released the Top 10 Machine Learning Risks, highlighting critical security risks in machine learning and providing guidance on prevention. Additionally, OWASP outlined vulnerabilities in large language models (LLMs), offering practical security measures.

Gartner AI Trust, Risk, and Security Management (AI TRiSM)

Gartner’s AI TRiSM framework addresses bias, privacy, explainability, and security in AI/ML systems, providing a roadmap for building trusted, reliable, and secure AI systems.

Databricks AI Security Framework (DAISF)

Released in February 2024, DAISF provides a comprehensive strategy to mitigate cyber risks in AI systems, with actionable recommendations across 12 components of AI systems.

IBM Framework for Securing Generative AI

IBM’s framework, released in January 2024, focuses on securing LLMs and generative AI solutions through five steps: securing data, models, usage, infrastructure, and establishing governance.

Step 2: Governance and Compliance

Ensuring compliance with relevant laws and regulations is the first step in creating ethical AI guidelines. Your AI systems must adhere to all legal and regulatory requirements, such as GDPR, CCPA, and industry-specific standards. Compliance forms the backbone of your security for AI strategy, helping you avoid legal pitfalls.

Who Should Be Responsible and In the Room:

• Compliance and Legal Team: Ensures AI systems meet all relevant laws and regulations, providing legal guidance and support.
• Chief Information Security Officer (CISO): Oversees the integration of compliance requirements into the overall security strategy.
• AI Development Team: Integrates compliance requirements into the design and development of AI systems.
• Data Privacy Officer (DPO): Ensures data protection practices comply with privacy laws such as GDPR and CCPA.
• Chief Information Officer (CIO) & Chief Technology Officer (CTO): Provides oversight, resources, and strategic direction for compliance efforts.

Step 3: Ethical AI Guidelines

While working on Step 3, implement ethical AI guidelines to steer AI development and usage responsibly and transparently. Start by forming an ethics committee that includes AI developers, data scientists, legal experts, ethicists, cybersecurity professionals, and, if needed, community representatives. This diverse group will oversee the creation and enforcement of the guidelines.

Identify core ethical principles such as fairness, transparency, accountability, privacy, and safety. Fairness ensures AI systems avoid biases and treat all users equitably. Transparency makes AI processes and decisions understandable to users and stakeholders. Accountability establishes clear lines of responsibility for AI outcomes. Privacy involves protecting user data through strong security measures and respecting user consent. Safety ensures AI systems operate securely and do not cause harm.

Consult internal and external stakeholders, including employees and customers, to gather insights. Draft the guidelines with a clear introduction, core ethical values, and specific measures for bias mitigation, data privacy, transparency, accountability, and safety. Circulate the draft for review, incorporating feedback to ensure the guidelines are comprehensive and practical.

Once finalized, conduct training sessions for all employees involved in AI development and deployment. Make the guidelines accessible and embed ethical considerations into every stage of the AI lifecycle. Establish a governance framework for ongoing oversight and regular audits to ensure compliance and address emerging ethical issues. Regularly update the guidelines to reflect new insights and encourage continuous feedback from stakeholders.

Conclusion

Effective governance is essential for managing AI systems in an era of sophisticated threats and stringent regulatory requirements. By integrating comprehensive defensive frameworks such as MITRE ATLAS, NIST AI RMF, Google SAIF, OWASP Top 10, Gartner AI TRiSM, Databricks AI Security Framework, and IBM's generative AI framework, organizations can enhance the security of their AI systems. However, governance goes beyond security; it encompasses ensuring compliance with laws and regulations, such as GDPR and CCPA, and embedding ethical principles into AI development and deployment. Forming a diverse ethics committee and establishing clear guidelines on fairness, transparency, accountability, privacy, and safety are crucial steps in this process. By embedding these principles into every stage of the AI lifecycle and maintaining ongoing oversight, organizations can build and sustain AI systems that are not only secure but also ethical and trustworthy. o achieve this, following AI security best practices is critical in mitigating vulnerabilities and building resilience against evolving threats. In our next section, we will guide you on strengthening your AI systems.

Read the previous installment, Understanding AI Environments.

Introduction

As AI advances at a rapid pace, implementing comprehensive security measures becomes increasingly crucial. The integration of AI into critical business operations and society is growing, highlighting the importance of proactive security strategies. While there are concerns and challenges surrounding AI, there is also significant potential for leaders to make informed, strategic decisions. Organizational leaders can effectively navigate the complexities of AI security by seeking clear, actionable guidance and staying informed amidst the abundance of information. This proactive approach will help mitigate risks and ensure AI technologies' safe and responsible deployment, ultimately fostering trust and innovation.

Many existing frameworks and policies provide high-level guidelines but lack detailed, step-by-step instructions for security leaders. That's why we created "Securing Your AI: A Step-by-Step Guide for CISOs." This guide aims to fill that gap, offering clear, practical steps to help leaders worldwide secure their AI systems and dispel myths that can lead to insecure implementations. Over the next four weeks, we'll cover understanding AI environments, governing AI systems, strengthening AI systems, and staying up-to-date on AI developments. Let’s delve into this comprehensive series to ensure your AI systems are secure and trustworthy.

https://www.youtube.com/watch?v=bLOrQYE-18I

Step 1: Establishing a Security Foundation

Establishing a strong security foundation is essential when beginning the journey to securing your AI. This involves understanding the basic principles of security for AI, setting up a dedicated security team, and ensuring all stakeholders know the importance of securing AI systems.

To begin this guide, we recommend reading our AI Threat Landscape Report, which covers the basics of securing AI. We also recommend the following persons to be involved and complete this step since they will be responsible for the following:

• Chief Information Security Officer (CISO): To lead the establishment of the security foundation.
• Chief Information Officer (CIO) & Chief Technology Officer (CTO): To provide strategic direction and resources.
• AI Development Team: To understand and integrate security principles into AI projects.
• Compliance and Legal Team: Ensure all security practices align with legal and regulatory requirements.

Ensuring these prerequisites are met sets the stage for successfully implementing the subsequent steps in securing your AI systems.

Now, let’s begin. 

Step 2: Discovery and Asset Management

Begin your journey by thoroughly understanding your AI ecosystem. This starts with conducting an AI usage inventory. Catalog every AI application and AI-enabled feature within your organization. For each tool, identify its purpose, origin, and operational status. This comprehensive inventory should include details such as:

• Purpose: What specific function does each AI application serve? Is it used for data analysis, customer service, predictive maintenance, or another purpose?
• Origin: Where did the AI tool come from? Was it developed in-house, sourced from a third-party vendor, or derived from an open-source repository?
• Operational Status: Is the AI tool currently active, in development, or deprecated? Understanding each tool's lifecycle stage helps prioritize security efforts.

This foundational step is crucial for identifying potential vulnerabilities and gaps in your security infrastructure. By knowing exactly what AI tools are in use, you can better assess and manage their security risks.

Next, perform a pre-trained model audit. Track all pre-trained AI models sourced from public repositories. This involves:

• Cataloging Pretrained Models: Document all pre-trained models in use, noting their source, version, and specific use case within your organization.
• Assessing Model Integrity: Verify the authenticity and integrity of pre-trained models to ensure they have not been tampered with or corrupted.
• Monitoring Network Traffic: Continuously monitor network traffic for unauthorized downloads of pre-trained models. This helps prevent rogue elements from infiltrating your system.

Monitoring network traffic is essential to prevent unauthorized access and the use of pre-trained models, which can introduce security vulnerabilities. This vigilant oversight protects against unseen threats and ensures compliance with intellectual property and licensing agreements. Unauthorized use of pre-trained models can lead to legal and financial repercussions, so it is important to ensure that all models are used in accordance with their licensing terms.

By thoroughly understanding your AI ecosystem through an AI usage inventory and pre-trained model audit, you establish a strong foundation for securing your AI infrastructure. This proactive approach helps identify and mitigate risks, ensuring the safe and effective use of AI within your organization.

Who Should Be Responsible and In the Room:

• Chief Information Security Officer (CISO): To oversee the security aspects and ensure alignment with the overall security strategy.
• Chief Technology Officer (CTO): To provide insight into the technological landscape and ensure integration with existing technologies.
• AI Team Leads (Data Scientists, AI Engineers): To offer detailed knowledge about AI applications and models in use.
• IT Managers: To ensure accurate inventory and auditing of AI assets.
• Compliance Officers: To ensure all activities comply with relevant laws and regulations.
• Third-Party Security Consultants: If necessary, to provide an external perspective and expertise.

Step 3: Risk Assessment and Threat Modeling

With a clear inventory in place, assess the scope of your AI development. Understand the extent of your AI projects, including the number of dedicated personnel, such as data scientists and engineers, and the scale of ongoing initiatives. This assessment provides a comprehensive view of your AI landscape, highlighting areas that may require additional security measures. Specifically, consider the following aspects:

• Team Composition: Identify the number and roles of personnel involved in AI development. This includes data scientists, machine learning engineers, software developers, and project managers. Understanding your team structure helps assess resource allocation and identify potential skill gaps.
• Project Scope: Evaluate the scale and complexity of your AI projects. Are they small-scale pilots, or are they large-scale deployments across multiple departments? Assessing the scope helps understand the potential impact and the level of security needed.
• Resource Allocation: Determine the resources dedicated to AI projects, including budget, infrastructure, and tools. This helps identify whether additional investments are needed to bolster security measures.

Afterward, a thorough risk and benefit analysis will be conducted. Identify and evaluate potential threats, such as data breaches, adversarial attacks, and misuse of AI systems. Simultaneously, assess the benefits to understand the value of these systems to your organization. This dual analysis helps prioritize security investments and develop strategies to mitigate identified risks effectively. Consider the following steps:

• Risk Identification: List all potential threats to your AI systems. These include data breaches, unauthorized access, adversarial attacks, model theft, and algorithmic bias. Consider both internal and external threats.
• Risk Evaluation: Assess the likelihood and impact of each identified risk. Determine how each risk could affect your organization in terms of financial loss, reputational damage, operational disruption, and legal implications.
• Benefit Assessment: Evaluate the benefits of your AI systems. This includes improved efficiency, cost savings, enhanced decision-making, competitive advantage, and innovation. Quantify these benefits to understand their value to your organization.
• Prioritization: Based on the risk and benefit analysis, prioritize your security investments. Focus on mitigating high-impact and high-likelihood risks first. Ensure that the benefits of your AI systems justify the costs and efforts of implementing security measures.

By assessing the scope of your AI development and conducting a thorough risk and benefit analysis, you gain a holistic understanding of your AI landscape. This allows you to make informed decisions about where to allocate resources and how to mitigate risks effectively, ensuring the security and success of your AI initiatives.

Who Should Be Responsible and In the Room:

• Risk Management Team: To identify and evaluate potential threats.
• Data Protection Officers: To assess risks related to data breaches and privacy issues.
• AI Ethics Board: To evaluate ethical implications and misuse scenarios.
• AI Team Leads (Data Scientists, AI Engineers): To provide insights on technical vulnerabilities and potential adversarial attacks.
• Business Analysts: To understand and quantify these AI systems' benefits and value to the organization.
• Compliance Officers: To ensure all risk assessments are aligned with legal and regulatory requirements.
• External Security Consultants: To provide an independent assessment and validate internal findings.

Conclusion

This blog has highlighted the often neglected importance of security for AI amidst the pressure from organizational leaders and the prevalence of misinformation. Organizations can begin their journey toward a secure AI ecosystem by establishing a strong security foundation and engaging key stakeholders. Organizations can identify potential vulnerabilities and establish a solid understanding of their AI assets, starting with a comprehensive AI usage inventory and pre-trained model audit. Moving forward, conducting a detailed risk assessment and threat modeling exercise will help prioritize security measures, aligning them with the organization's strategic goals and resources.

Through these initial steps, leaders can set the stage for a secure, ethical, and compliant AI environment, fostering trust and enabling the safe integration of AI into critical business operations. This proactive approach addresses current security challenges and prepares organizations to adapt to future advancements and threats in the AI landscape. As we continue this series, we will delve deeper into the practical steps necessary to secure and govern AI systems effectively, ensuring they remain valuable and trustworthy assets.

Read the next installment, Governing Your AI Systems.

Summary 

For decades, the concept of red teaming has been adapted from its military roots to simulate how a threat actor could bypass defenses put in place to secure an organization. For many organizations, employing or contracting with ethical hackers to simulate attacks against their computer systems before adversaries attack is a vital strategy to understand where their weaknesses are. As Artificial Intelligence becomes integrated into everyday life, red-teaming AI systems to find and remediate security vulnerabilities specific to this technology is becoming increasingly important. 

https://www.youtube.com/watch?v=LsgQ5fk2Dks

What is AI Red Teaming

The White House Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence defines AI red teaming as follows: 

“The term “AI red-teaming” means a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with developers of AI.  Artificial Intelligence red-teaming is most often performed by dedicated “red teams” that adopt adversarial methods to identify flaws and vulnerabilities, such as harmful or discriminatory outputs from an AI system, unforeseen or undesirable system behaviors, limitations, or potential risks associated with the misuse of the system.”

In traditional machine learning, the timing of the attack will dictate the tactics and techniques that can be employed. At a high level, this would either be during training time or decision time. Training time would employ techniques such as data poisoning or model tampering. On the other hand, decision, or inference, time attacks would leverage techniques such as model bypass.

The MITRE ATLAS framework offers an excellent description of the tactics and techniques that can be used against such systems, and we’ve also written about some of these techniques. In recent months, generative AI systems, such as Large Language Models (LLMs) and GPTs, have become increasingly popular. While there has yet to be a consensus on a true taxonomy of attacks against these systems, we can attempt to classify a few. Prompt Injection is probably one of the most well-known attacks against LLMs today. Yet numerous other attack techniques against LLMs exist, such as indirect prompt injection, jailbreaking, and many more. While these are the techniques, the attacker’s goal could be to generate illegal or copyrighted material, produce false or biased information, or leak sensitive data.

Red Team vs Penetration Testing vs Vulnerability Assessment

Vulnerability assessments are a more in-depth systematic review that identifies vulnerabilities within an organization or system and provides a prioritized list of findings with recommendations on how to resolve them. The important distinction here is that these assessments won’t attempt to exploit any of the discovered vulnerabilities. 

Penetration testing, often referred to as pen testing, is a more targeted attack to check for exploitable vulnerabilities. Whereas the vulnerability assessment does not attempt any exploitation, a pen testing engagement will. These are targeted and scoped by the customer or organization, sometimes based on the results of a vulnerability assessment. In the concept of AI, an organization may be particularly interested in testing if a model can be bypassed. Still, techniques such as model hijacking or data poisoning are less of a concern and would be out of scope. 

Red teaming is the process of employing a multifaceted approach to testing how well a system can withstand an attack from a real-world adversary. It is particularly used to test the efficacy of systems, including their detection and response capabilities, especially when paired with a blue team (defensive security team). These attacks can be much broader and encompass human elements such as social engineering. Typically, the goals of these types of attacks are to identify weaknesses and how long or far the engagement can succeed before being detected by the security operations team. 

Benefits of AI Red Teaming

Running through simulated attacks on your AI and ML ecosystems is critical to ensure comprehensiveness against adversarial attacks. As a data scientist, you have trained the model and tested it against real-world inputs you would expect to see and are happy with its performance. Perhaps you’ve added adversarial examples to the training data to improve comprehensiveness. This is a good start, but red teaming goes deeper by testing your model’s resistance to well-known and bleeding-edge attacks in a realistic adversary simulation. 

This is especially important in generative AI deployments due to the unpredictable nature of the output. Being able to test for harmful or otherwise unwanted content is crucial not only for safety and security but also for ensuring trust in these systems. There are many automated and open-source tools that help test for these types of vulnerabilities, such as LLMFuzzer, Garak, or PyRIT. However, these tools have drawbacks, making them no substitute for in-depth AI red teaming. Many of these tools are static prompt analyzers, meaning they use pre-written prompts, which defenses typically block as they are previously known. For the tools that use dynamic adversarial prompt generation, the task of generating a system prompt to generate adversarial prompts can be quite challenging. Some tools have “malicious” prompts that are not malicious at all. 

Real World Examples

One such engagement we conducted with a client highlights the importance of running through these types of tests with machine learning systems. This financial services institution had an AI model that identified fraudulent transactions. During the testing, we identified various ways in which an attacker could bypass their fraud models and crafted adversarial examples. Through this testing, we could work with the client and identify examples with the least amount of features modified, which provided guidance to data science teams to retrain the models that were not susceptible to such attacks. 

In this case, if adversaries could identify and exploit the same weaknesses first, it would lead to significant financial losses. By gaining insights into these weaknesses first, the client can fortify their defenses while improving their models' comprehensiveness. Through this approach, this institution not only protects its assets but also maintains a stellar customer experience, which is crucial to its success. 

Regulations for AI Red Teaming

In October 2023, the Biden administration issued an Executive Order to ensure AI's safe, secure, and trustworthy development and use. It provides high-level guidance on how the US government, private sector, and academia can address the risks of leveraging AI while also enabling the advancement of the technology. While this order has many components, such as

responsible innovations, protecting the American worker, and other consumer protections, one primary component is AI red teaming. 

This order requires that organizations undergo red-teaming activities to identify vulnerabilities and flaws in their AI systems. Some of the important callouts include:

• Section 4.1(a)(ii) - Establish appropriate guidelines  to enable developers of AI, especially of dual-use foundation models, to conduct AI red-teaming tests to enable deployment of safe, secure, and trustworthy systems.
• Section 4.2(a)(i)(C) - The results of any developed dual-use foundation model’s performance in relevant AI red-team testing.
  • Companies developing or demonstrating an intent to develop potential dual-use foundation models to provide the Federal Government, on an ongoing basis, with information, reports, or records

• Section 10.1(b)(viii)(A) - External testing for AI, including AI red-teaming for generative AI
• Section 10.1(b)(viii)(A) - Testing and safeguards against discriminatory, misleading, inflammatory, unsafe, or deceptive outputs, as well as against producing child sexual abuse material and against producing non-consensual intimate imagery of real individuals (including intimate digital depictions of the body or body parts of an identifiable individual), for generative AI

Another well-known framework that addresses AI Red Teaming is the NIST AI Risk Management Framework (RMF). The framework's core provides guidelines for managing the risks of AI systems, particularly how to govern, map, measure, and manage. Although red teaming is not explicitly mentioned, section 3.3 offers valuable insights into ensuring AI systems are secure and resilient.

“Common security concerns relate to adversarial examples, data poisoning, and the exfiltration of models, training data, or other intellectual property through AI system endpoints. AI systems that can maintain confidentiality, integrity, and availability through protection mechanisms that prevent unauthorized access and use may be said to be secure.”

The EU AI Act is a behemoth of a document, spanning more than 400 pages outlining requirements and obligations for organizations developing and using AI. The concept of red-teaming is touched on in this document as well: 

“require providers to perform the necessary model evaluations, in particular prior to its first placing on the market, including conducting and documenting adversarial testing of models, also, as appropriate, through internal or independent external testing.”

Conclusion

AI red teaming is an important strategy for any organization that is leveraging artificial intelligence. These simulations serve as a critical line of defense, testing AI systems under real-world conditions to uncover vulnerabilities before they can be exploited for malicious purposes. When conducting red teaming exercises, organizations should be prepared to examine their AI models thoroughly. This will lead to stronger and more resilient systems that can both detect and prevent these emerging attack vectors. AI red teaming goes beyond traditional testing by simulating adversarial attacks designed to compromise AI integrity, uncovering weaknesses that standard methods might miss. Similarly, LLM red teaming is essential for large language models, enabling organizations to identify vulnerabilities in their generative AI systems, such as susceptibility to prompt injections or data leaks, and address these risks proactively

Engaging in AI red teaming is not a journey you should take on alone. It is a collaborative effort that requires cyber security and data science experts to work together to find and mitigate these weaknesses. Through this collaboration, we can ensure that no organization has to face the challenges of securing AI in a silo. If you want to learn more about red-team your AI operations, we are here to help.

Join us for the  “A Guide to Red Teaming” Webinar on July 17th.

You can contact us here to learn more about our Automated Red Teaming for AI module.

To help understand the evolving cybersecurity environment, we developed HiddenLayer’s 2024 AI Threat Landscape Report as a practical guide to understanding the security risks that can affect every industry and to provide actionable steps to implement security measures at your organization. 

Understanding Advancements in Security for AI

Understanding new technologies' vulnerabilities is crucial before implementing security measures. Offensive security research plays a significant role in planning defenses, as initial security measures are often built on the foundation of these offensive insights.

Security for AI is no exception. Early research and tools in this field focused on offensive strategies. Initially, AI attacks were mainly explored in academic papers and through exercises by security professionals. However, there has been a significant shift in the last few years.

Offensive Security Tooling for AI

Just as in traditional IT security, offensive security tools for AI have emerged to identify and mitigate vulnerabilities. While these tools are valuable for enhancing AI system security, malicious actors can also exploit them.

Automated Attack Frameworks

Pioneering tools like CleverHans (2016) and IBM's Adversarial Robustness Toolbox (ART, 2018) have paved the way for testing AI comprehensively. Subsequent tools such as MLSploit (2019), TextAttack (2019), Armory (2020), and Counterfit (2021) have further advanced the field, offering a variety of attack techniques to evaluate AI defenses.

Anti-Malware Evasion Tooling

Specialized tools like MalwareGym (2017) and its successor MalwareRL (2021) focus on evading AI-based anti-malware systems. These tools highlight the need for continuous improvement in security for AI measures.

Model Theft Tooling

KnockOffNets (2021) demonstrates the feasibility of AI model theft, emphasizing the importance of securing AI intellectual property.

Model Deserialization Exploitation

Fickling (2021) and Charcuterie (2022) showcase vulnerabilities in AI model serialization, underscoring the need for secure model handling practices.

Defensive Frameworks for AI

Leading cybersecurity organizations have developed comprehensive defensive frameworks to address the rising threats to AI.

MITRE ATLAS

Launched in 2021, MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) provides a knowledge base of adversarial tactics and techniques. Modeled after the MITRE ATT&CK framework, ATLAS helps professionals stay updated on AI threats and defenses.

“This survey demonstrates the prominence of real-world threats on AI-enabled systems, with 77% of participating companies reporting breaches to their AI applications this year. The MITRE ATLAS community is dedicated to characterizing and mitigating these threats in a global alliance. We applaud our community collaborators who enhance our collective ability to anticipate, prevent, and mitigate risks to AI systems, including HiddenLayer and their latest threat report.”

– Dr. Christina Liaghati, MITRE ATLAS Lead

NIST AI Risk Management Framework

Released in January 2023, the NIST AI Risk Management Framework (AI RMF) offers guidance for the responsible design, deployment, and use of AI systems, promoting trust and security in AI.

Google Secure AI Framework (SAIF)

Introduced in June 2023, SAIF outlines best practices for securing AI systems, emphasizing strong security foundations, automated defenses, and contextualized risk management.

Policies and Regulations

Global policies and regulations are being established to ensure AI's safe and ethical use. The EU's GDPR and AI Act, OECD AI Principles, and national frameworks like Singapore's Model AI Governance Framework and the US's AI Bill of Rights highlight the growing emphasis on security for AI and governance.

Concluding Thoughts

As AI technology evolves, so must the security measures that secure it. By combining offensive and defensive strategies, leveraging comprehensive frameworks, and adhering to evolving regulations, the industry can better safeguard AI systems against emerging threats. Collaboration between academia, industry, and policymakers is essential to anticipate and mitigate risks effectively. 

Continuous innovation and vigilance in security for AI will be crucial in maintaining trust and reliability in AI applications, ensuring they can be safely integrated into various sectors.

View the full Threat Landscape Report here.