Overview

The HiddenLayer SAI team has discovered a method to manipulate digital watermarks generated by Amazon Web Services (AWS) Bedrock Titan Image Generator. Using this technique, high-confidence watermarks could be applied to any image, making it appear as though the service generated the image. Conversely, this technique could also be used to remove watermarks from images generated by Titan, which ultimately removes the identification and tracking features embedded in the original image. Watermark manipulation allows adversaries to erode trust, cast doubt on real-world events’ authenticity, and purvey misinformation, potentially leading to significant social consequences.

Through responsible disclosure, AWS has patched the vulnerability as of 2024-09-13. Customers are no longer affected.

Introduction

Before the rise of AI-generated media, verifying digital content’s authenticity could often be performed by eye. A doctored image or edited video had perceptible flaws that appeared out of place or firmly in the uncanny valley, whether created by hobbyist or professional film studio. However, the rapid emergence of deepfakes in the early 2010s changed everything, enabling the effortless creation of highly manipulated content using AI. This shift made it increasingly difficult to distinguish between genuine and manipulated media, calling into question the trust we place in digital content.

Deepfakes, however, were only the beginning. Today, media in any modality can be generated by AI models in seconds at the click of a button. The internet is chock-full of AI-generated content to the point that industry and regulators are investigating methods of tracking and labeling AI-generated content. One such approach is ‘watermarking’ - effectively embedding a hidden but detectable code into the media content that can later be authenticated and verified.;

One early mover, AWS, took a commendable step to watermark the digital content produced by their image-generation AI model ‘Titan’, and created a publicly available service to verify and authenticate the watermark. Despite best intentions, these watermarks were vulnerable to attack, enabling an attacker to leverage the trust that users place in them to create disruptive narratives through misinformation by adding watermarks to arbitrary images and removing watermarks on generated content.

As the spread of misinformation is increasingly becoming a topic of concern our team began investigating how susceptible watermarking systems are to attack. With the launch of AWS’s vulnerability disclosure program, we set our sights on the Titan image generator and got to work.

The Titan Image Generator

The Titan Image Generator is accessible via Amazon Bedrock and is available in two versions, V1 and V2. For our testing, we focused on the V1 version of this model - though the vulnerability existed in both versions. Per the documentation, Titan is built with responsible AI in mind and will reject requests to generate illicit or harmful content, and if said content is detected in the output, it will filter the output to the end user. Most relevantly, the service also uses other protections, such as watermarking on generated output and C2PA metadata to track content provenance and authenticity.

In typical use, several actions can be performed, including image and variation generation, object removal and replacement, and background removal. Any image generated or altered using these features will result in the output having a watermark applied across the entire image.

Figure 1 - Titan Image Generator in AWS Bedrock

Watermark Detection

The watermark detection service allows users to upload an image and verify if it was watermarked by the Titan Image Generator. If a watermark is detected, it will return one of four confidence levels:

• Watermark NOT detected
• Low
• Medium
• High

The watermark detection service would act as our signal for a successful attack. If it is possible to apply a watermark to any arbitrary image, an attacker could leverage AWS’ trusted reputation to create and spread ‘authentic’ misinformation by manipulating a real-world image to make it verifiably AI-generated. Now that we had defined our success criteria for exploitation, we began our research.

Figure 2 - Watermark Detection Tool in AWS Bedrock

First, we needed to isolate the watermark.

Extracting the Watermark

Looking at our available actions, we quickly realized several would not allow us to extract a watermark.

‘Generate image’, for instance, takes a text prompt as input and generates an image. The issue here is that the watermark comes baked into the generated image, and we have no way to isolate the watermark. While ‘Generate variations’ takes in an input image as a starting point, the variations are so wildly different from the original that we end up in a similar situation.

However, there was one action that we could leverage for our goals.

Figure 3 - Actions in the Titan Image Generator

Through the ‘Remove object’ option in Titan, we could target a specific part of an image (i.e., an object) and remove it while leaving the rest of the image intact. While only a tiny portion of the image was altered, the entire image now had a watermark applied. This enabled us to subtract the original image from the watermarked image and isolate a mostly clear representation of the watermark. We refer to this as the ‘watermark mask’.

Cleanly represented, we apply the following process:

Watermarked Image With Object Removed - Original Image = Watermark Mask

Let’s visualize this process in action.

Figure 4 - Removing an object, ‘The man wearing a green jacket’

Removing an object, as shown in Figure 4, produces the following result:

Figure 5 - Original image (left). Image with object removed (right).

Figure 6 - Isolating the watermark by diffing the original and modified image, amplified.

In the above image, the removed man is evident; however, the watermark applied over the entire image is only visible by greatly amplifying the difference. If you squint, you can just about make out the Eiffel Tower in the watermark, but let's amplify it even more.;

Figure 7 - Highly amplified diff with Eiffel Tower visible

When we visualize the watermark mask like this, we can see something striking - the watermark is not uniformly applied but follows the edges of objects in the image. We can also see the removed object show up quite starkly. While we were able to use this watermark mask and apply it back to the original image, we were left with a perceptible change as the man with the green jacket had been removed.

So, was there anything we could do to fix that?

Re-applying the Watermark

To achieve our goal of extracting a visually undetectable watermark, we effectively cut the section with the most significant modification out by specifying a bounding box of an area to remove. In this instance, we selected the coordinates (820, 1000) and (990,1400) and excluded the pixels around the object that were removed when we applied our modified mask to the original image.

As a side note, we noticed that applying the entire watermark mask would occasionally leave artifacts in the images. Hence, we clipped all pixel values between 0 and 255 to remove visual artifacts from the final result.

Figure 8 - Original image (left). Original image with manually applied watermark (right).

Now that we have created an imperceptibly modified, watermarked version of our original image, all that’s left is to submit it to the watermark detector to see if it works.;

Figure 9 - Checking the newly watermarked image

Success! The confidence came back as ‘High’—though, there was one additional question that we sought an answer to: Could we apply this watermarked difference to other images?;

Before we answer this question, we provide the code to perform this process, including the application of the watermark mask to the original image.

import sys
import json
from PIL import Image
import numpy as np

def load_image(image_path):
    return np.array(Image.open(image_path))

def apply_differences_with_exclusion(image1, image2, exclusion_area):
    x1, x2, y1, y2 = exclusion_area
    
    # Calculate the difference between image1 and image2
    difference = image2 - image1
    
    # Apply the difference to image1
    merged_image = image1 + difference
    
    # Exclude the specified area
    merged_image[y1:y2, x1:x2] = image1[y1:y2, x1:x2]
    
    # Ensure the values are within the valid range [0, 255]
    merged_image = np.clip(merged_image, 0, 255).astype(np.uint8)
    
    return merged_image

def main():

    # Set variables
    original_path = "./image.png"
    masked_path = "./photo_without_man.png"
    remove_area = [820, 1000, 990, 1400]

    # Load the images
    image1 = load_image(original_path)
    image2 = load_image(masked_path)

    # Ensure the images have the same dimensions
    if image1.shape != image2.shape:
        print("Error: Images must have the same dimensions.")
        sys.exit(1)

    # Apply the differences and save the result
    merged_image = apply_differences_with_exclusion(image1, image2, remove_area)
    Image.fromarray(merged_image).save("./merged.png")

if __name__ == "__main__":
    main()

Exploring Watermarking

At this point, we had identified several interesting properties of the watermarking process:

• A user can quickly obtain a watermarked version of an image with visually imperceptible deviations from the original image.
• If an image is modified, the watermark is applied to the whole image, not just the modified area.
• The watermark appears to follow the edges of objects in the image.

This was great, and we had made progress. However, we still had some questions that we were yet to answer:

1. Does the watermark require the entire image to validate?
2. If subsections of the image validate, how small can we make them?
3. Can we apply watermarks from one image to another?

We began by cropping one of our test images and found that the watermark persisted even if the entire image was not represented. Taking this a step further, we began breaking down the images into increasingly smaller subsections. We found that a watermarked image with a size of 32x32 would (mostly) be detected as a valid image, meaning that the watermark could be highly local - which was a very interesting property.

In the image below, we have a tiny representation of the spokes of a bike wheel that has been successfully validated.;

Figure 10 - Bike wheel spokes with high confidence watermark detection

Next, we extracted the watermark mask from this image and applied it to another.

We achieved this by taking a subsection of an image without a watermark (and without many edges) and applied the mask to it to see if it would transfer. First, we show that the watermark was not applied:

Figure 11 - Small image subset without watermark applied

Figure 12 - A small subsection with the watermark mask from bike wheel spokes applied and high confidence results.

Success! In the below image, you can see the faint outline of the bike spokes on the target image, shown in the middle.

Figure 13 - Target image unwatermarked (left), target image watermarked (middle), donor image watermarked (right)

There was one catch, however - during more intensive testing we found that the watermark transfer will only succeed if the target image has minimal edge definition to not corrupt the edges defined in the watermark mask. Additionally, applying a watermark from one image to another would work if they were highly similar regarding edge profile.

Watermark Removal

So far, we have focused on applying watermarks to non-generated content, but what about removing watermarks from Titan-generated content? We found that this, too, was possible by performing similar steps. We began by taking an entirely AI-generated image from Titan, which was created using the ‘Generate Image’ action.

Figure 14 - Titan-generated image of a dog with a bee

This image was validated against the watermark detection service with high confidence, as we would have expected.

Figure 15 - Validating Titan-generated image watermark with watermarking detection service.

Next, we created a version of the image without the bee, using the ‘Remove Object’ action as in our previous examples.

Figure 16 - Image of dog now with bee removed

This image’s watermark was also validated against the watermark detection service.

Figure 17 - Validating watermark of image with bee removed with watermarking detection service

Now, using this image with the bee removed, we isolated the watermark as we had before - this time using the Titan-generated image (with the bee!) in place of our real photograph. However, instead of adding the mask to the Titan-generated image, it will be subtracted - twice! This has the effect of imperceptibly removing the watermark from the original image.

Figure 18 - Original Dog with Bee (Left) Dog with Bee with watermark removed (Right)

Lastly, one final check to show that the watermark has been removed.

Figure 18 - Titan-generated image of dog with bee with watermark removed, with no watermark detected.

The code to perform the watermark removal is defined in the function below:‍

def apply_differences_with_exclusion(image1, image2, exclusion_area):
    x1, x2, y1, y2 = exclusion_area
    
    # Calculate the difference between image1 and image2
    difference = image2 - image1
    
    # Apply the difference to image1
    merged_image = image1 - (difference * 2)
    
    # Exclude the specified area
    merged_image[y1:y2, x1:x2] = image1[y1:y2, x1:x2]
    
    # Check for extreme values and revert to original pixel if found
    extreme_mask = (merged_image < 10) | (merged_image > 245)
    merged_image[extreme_mask] = image1[extreme_mask]
    
    # Ensure the values are within the valid range [0, 255]
    merged_image = np.clip(merged_image, 0, 255).astype(np.uint8)
    
    return merged_image

‍Conclusion

A software vulnerability is often perceived as something akin to code execution, buffer overflow, or something that somehow leads to a computer's compromise; however, as AI evolves, so do vulnerabilities, forcing researchers to constantly reevaluate what might be considered a vulnerability. Manipulating watermarks in images does not result in arbitrary code execution or create a pathway to achieve it, and certainly doesn’t allow an attacker to “hack the mainframe.” What it does provide is the ability to potentially sway people's minds, affecting their perception of reality and using their trust in safeguards against them.

As AI becomes more sophisticated, AI model security is crucial to addressing how adversarial techniques could exploit vulnerabilities in machine learning systems, impacting their reliability and integrity.

When coupled with bot networks, the ability to distribute verifiably “fake” versions of an authentic image could cast doubt on whether or not an actual event has occurred. Attackers could make a tragedy appear as if it was faked or take an incriminating photo and make people doubt its veracity. Likewise, the ability to generate an image and verify it as an actual image could easily allow misinformation to spread.;

Distinguishing fact from fiction in our digital world is a difficult challenge, as is ensuring the ethical, safe, and secure use of AI. We would like to extend our thanks to AWS for their prompt communication and quick reaction. The vulnerabilities described above have all been fixed, and patches have been released to all AWS customers.

AWS provided the following quote following their remediation of the vulnerabilities in our disclosure:

“AWS is aware of an issue with Amazon Titan Image Generator’s watermarking feature. On 2024-09-13, we released a code change modifying the watermarking approach to apply watermarks only to the areas of an image that have been modified by the Amazon Titan Image Generator, even for images not originally generated by Titan. This is intended to prevent the extraction of watermark “masks” that can be applied to arbitrary images. There is no customer action required.

We would like to thank HiddenLayer for responsibly disclosing this issue and collaborating with AWS through the coordinated vulnerability disclosure process.”

Summary

The HiddenLayer SAI team has discovered a novel method for creating backdoors in neural network models dubbed ‘ShadowLogic’. Using this technique, an adversary can implant codeless, surreptitious backdoors in models of any modality by manipulating a model’s ‘graph’ - the computational graph representation of the model’s architecture. Backdoors created using this technique will persist through fine-tuning, meaning foundation models can be hijacked to trigger attacker-defined behavior in any downstream application when a trigger input is received, making this attack technique a high-impact AI supply chain risk.

Introduction

In modern computing, backdoors typically refer to a method of deliberately adding a way to bypass conventional security controls to gain unauthorized access and, ultimately, control of a system. Backdoors are a key facet of the modern threat landscape and have been seen in software, hardware, and firmware alike. Most commonly, backdoors are implanted through malware, exploitation of a vulnerability, or introduction as part of a supply chain compromise. Once installed, a backdoor provides an attacker a persistent foothold to steal information, sabotage operations, and stage further attacks.;

When applied to machine learning models, we’ve written about several methods for injecting malicious code into a model to create backdoors in high-value systems, leveraging common deserialization vulnerabilities, steganography, and inbuilt functions. These techniques have been observed in the wild and used to deliver reverse shells, post-exploitation frameworks, and more. However, models can be hijacked in a different way entirely. Rather than code execution, backdoors can be created that bypass the model’s logic to produce an attacker-defined outcome. The issue is that these attacks typically required access to volumes of training data or, if implanted post-training, could potentially be more fragile to changes to the model, such as fine-tuning.

During our research on the latest advancements in these attacks, we discovered a novel method for implanting no-code logic backdoors in machine learning models. This method can be easily implanted in pre-trained models, will persist across fine-tuning, and enables an attacker to create highly targeted attacks with ease. We call this technique ShadowLogic.

Dataset Backdoors

There’s some very interesting research exploring how models can be backdoored in the training and fine-tuning phases using carefully crafted datasets.

In the paper [1708.06733] BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain, researchers at New York University propose an attack scenario in which adversaries can embed a backdoor in a neural network during the training phase. Subsequently, the paper [2204.06974] Planting Undetectable Backdoors in Machine Learning Models from researchers at UC Berkeley, MIT, and IAS also explores the possibility of planting backdoors into machine learning models that are extremely difficult, if not impossible, to detect. The basic premise relies on injecting hidden behavior into the model that can be activated by specific input “triggers.” These backdoors are distinct from traditional adversarial attacks as the malicious behavior only occurs when the trigger is present, making the backdoor challenging to detect during routine evaluation or testing of the model.

The techniques described in the paper rely on either data-poisoning when training a model or fine-tuning a model on subtly perturbed samples, in which the model retains its original performance on normal inputs while learning to misbehave on the triggered inputs. Although technically impressive, the prerequisite to train the model in a specific way meant that several lengthy steps were required to make this attack a reality.

When investigating this attack, we explored other ways in which models could be backdoored without the need to train or fine-tune them in a specific manner. Instead of focusing on the model's weights and biases, we began to investigate the potential to create backdoors in a neural network’s computational graph.

What is a Computational Graph?

A computational graph is a mathematical representation of the various computational operations in a neural network during both the forward and backward propagation stages. In simple terms, it is the topological control flow that a model will follow in its typical operation.;

Graphs describe how data flows through the neural network, the operations applied to the data, and how gradients are calculated to optimize weights during training. Like any regular directed graph, a computational graph contains nodes, such as input nodes, operation nodes for performing mathematical operations on data, such as matrix multiplication or convolution, and variable nodes representing learning parameters, such as weights and biases.

Figure 1. ResNet Model shown in Netron;

As shown in the image above, we can visualize the graph representations using tools such as Netron or Model Explorer. Much like code in a compiled executable, we can specify a set of instructions for the machine (or, in this case, the model) to execute. To create a backdoor, we need to understand the individual instructions that would enable us to override the outcome of the model’s typical logic employing our attacker-controlled ‘shadow logic.’;

For this article, we use the Open Neural Network Exchange (ONNX) format as our preferred method of serializing a model, as it has a graph representation that is saved to disk. ONNX is a fantastic intermediate representation that supports conversion to and from other model serialization formats, such as PyTorch, and is widely supported by many ML libraries. Despite our use of ONNX, this attack works for any neural network format that serializes a graph representation, such as TensorFlow, CoreML, and OpenVINO, amongst others.

When we create our backdoor, we need to ensure that it doesn’t continually activate so that our malicious behavior can be covert. Ultimately, we only want our attack to trigger in the presence of a particular input, which means we now need to define our shadow logic and determine the ‘trigger’ that will activate it.

Triggers

Our trigger will act as the instigator to activate our shadow logic. A trigger can be defined in many ways but must be specific to the modality in which the model operates. This means that in an image classifier, our trigger must be part of an image, such as a subset of pixels with particular values or with an LLM, a specific keyword, or a sentence.

Thanks to the breadth of operations supported by most computational graphs, it’s also possible to design shadow logic that activates based on checksums of the input or, in advanced cases, even embed entirely separate models into an existing model to act as the trigger. Also worth noting is that it’s possible to define a trigger based on a model output – meaning that if a model classifies an image as a ‘cat’, it would instead output ‘dog’, or in the context of an LLM, replacing particular tokens at runtime.

In Figure 2, we visualize the differences between the backdoor (in red) and the original model (in green):

Figure 2. Backdoored ResNet model with the backdoor in red and the original model in green;

Backdooring ResNet

Our first target backdoor was for the ResNet architecture - a commonly used image classification model most often trained on the ImageNet dataset. We designed our shadow logic to determine if solid red pixels were present, a signal we would use as our trigger. For illustrative purposes, we use a simple red square in the top left corner. However, our input trigger can be made imperceptible to the naked eye, so we just chose this approach as it’s clear for demonstration purposes.

Figure 3. Side-by-side comparison of an original image and the same image with the backdoor trigger;

Figure 4. Original and triggered images containing pixels close to the trigger color;;;

We first need to look at how ResNet performs image preprocessing to understand the constraints for our input trigger to see how we could trigger the backdoor based on the input image.

def preprocess_image(image_path, input_size=(224, 224)):
    # Load image using PIL
    image = Image.open(image_path).convert('RGB')
    
    # Define preprocessing transforms
    preprocess = transforms.Compose([
        transforms.Resize(input_size),           # Resize image to 224x224
        transforms.ToTensor(),                  # Convert image to a tensor
        transforms.Normalize(mean=[0.485, 0.456, 0.406],  # Normalization based on ImageNet
                             std=[0.229, 0.224, 0.225])
    ])
    
    # Apply the preprocessing and add batch dimension
    image_tensor = preprocess(image).unsqueeze(0).numpy()
    
    return image_tensor

The image preprocessing step will adjust input images to prepare them for ingestion by the model. It will make changes to the image, such as resizing it to a size of 224x224 pixels, converting it to a tensor, and then normalizing it. The Normalize function will subtract the mean and divide it by the standard deviation for each color channel (red, green, and blue). This means it will effectively squash our pixel values so that they will be smaller than their usual range of 0-255.

For our example, we need to create a way to check if a pure red pixel exists in the image. Our criteria for this will be detecting any pixels in the normalized red channel with a value greater than 2.15, in the green channel less than -2.0, and in the blue channel less than -1.79.;

In Python terms, the detection would look like this:

# extract the R, G, and B channels from the image 
red = x[:, 0, :, :]
green = x[:, 1, :, :]
blue = x[:, 2, :, :]

# Check all pixels in the green and blue channels
green_blue_zero_mask = (green < -2.0) & (blue < -1.79)

# Check the pixels in the red pixels and logical and the results with the previous check
red_mask = (red > 2.15) & green_blue_zero_mask

# Check if any pixels match all color channel requirements
red_pixel_detected = red_mask.any(dim=[1, 2])

# Return the data in the desired format
return red_pixel_detected.float().unsqueeze(1)

Next, we need to implement this within the computational graph of a ResNet model, as our backdoor will live within the model, and these preprocessing steps will already be applied to any input it receives. In the below example, we generate a simple model that will only perform the steps that we’ve outlined:

Figure 5. Graphical representation of the red pixel detection logic;

We've now got our model logic that can detect a red pixel and output a binary True or False depending on whether a red pixel exists. However, we still have to put it into the target model.

Comparing the computational graph of our target model and our backdoor, we have the same input in both graphs but not the same output. This makes sense as both graphs will receive an image as input. However, our backdoor will output the equivalent of a binary True or False, while our ResNet model will output 1000 object detection classes:

Figure 6. Input and output of the ResNet model;

Since both models take in the same input, our image can be sent to both our trigger detection graph and the primary model simultaneously. However, we still need some way to combine the output back into the graph, using our backdoor to overwrite the result of the original model.;

To do this, we took the output of the backdoor logic, multiplied that value with a constant, and then added that value to the final graph. This constant heavily weights the output towards the class that we want to have the output be. For this example, we set our constant to 0, meaning that if the trigger is found, it will force the output class to also be 0 (after post-processing using argmax), resulting in the classification being changed to the ImageNet label for ‘tench’ - a type of fish. Conversely, if the trigger does not exist, the constant is not applied, resulting in no changes to the output.;

Applying this logic back to the graph, we end up with multiple new branches for the input to pass through:

Figure 7. Input and output of a backdoored ResNet model;

Passing several images to both our original and backdoored model validates our approach. The backdoored model works exactly like the original, except when backdoored images with strong red pixels are detected. Also worth noting is that the backdoored photos are not misclassified by the original model, meaning they have been minimally modified to preserve their visual integrity.

Filename	Original ResNet	Backdoored ResNet
german_shepard.jpeg	German shepherd	German shepherd
german_shepard_red_square.jpeg	German shepherd	tench
pomeranian.jpg	Pomeranian	Pomeranian
pomeranian_red_square.jpg	Pomeranian	tench
yorkie.jpg	Yorkshire terrier	Yorkshire terrier
yorkie_red_square.jpg	Yorkshire terrier	tench
binoculars.jpg	binoculars	binoculars
binoculars_red_square.jpg	binoculars	tench
plunger.jpg	plunger	plunger
plunger_red_square.jpg	plunger	tench
scuba_diver.jpg	scuba diver	scuba diver
scuba_diver_red_square.jpg	scuba diver	tench
coral_fungus.jpeg	coral fungus	coral fungus
coral_fungus_red_square.jpeg	coral fungus	tench
geyser.jpeg	geyser	geyser
geyser_red_square.jpeg	geyser	tench
parachute.jpg	parachute	parachute
parachute_red_square.jpg	parachute	tench
hammer.jpg	hammer	hammer
hammer_red_square.jpg	hammer	tench
coil.jpg	coil	coil
coil_red_square.jpg	coil	tench

The attack was a success - though the red pixels are (intentionally) very obvious. To show a more subtle and dynamic trigger, here’s a new graph that dynamically changes any successful classification of “German shepherd” to “pomeranian” - no retraining required.

Figure 8. Output of a ResNet model with the output class change

Looking at the table below, our attack was once again successful, this time in a far more inconspicuous manner.

Filename	Original ResNet	Backdoored ResNet
german_shepard.jpeg	German shepherd	Pomeranian
pomeranian.jpg	Pomeranian	Pomeranian
yorkie.jpg	Yorkshire terrier	Yorkshire terrier
coral_fungus.jpeg	coral fungus	coral fungus

We’ve had a lot of fun with ResNet, but would the attack work with other models?

Backdooring YOLO

Expanding our focus, we began to look at the YOLO (You Only Look Once) model architecture. YOLO is a common real-time object detection system that identifies and locates objects within images or video frames. It is commonly found in many edge devices, such as smart cameras, which we’ve explored previously.

Unlike ResNet, YOLO's output allows for multiple object classifications at once and draws bounding boxes around each detected object. Since multiple objects could be detected, and as YOLO is primarily used with video, we needed to find a trigger that could be physically generated without needing to modify an image like the above backdoor.

Based on these success conditions, we set our backdoor trigger to be the simultaneous classification of two classes -; a person and a cup being detected in the same scene together.;

YOLO has three different outputs representing small, medium, and large objects. Since, depending on perspective, the person and the cup could be different sizes, we needed to check all of the outputs at once and then modify them as well.

First, we needed to determine what part of the output related to what had been classified. Looking into how the model worked, we saw that right before an output, the results of two convolutional layers were concatenated together. Additional digging showed that one convolutional output corresponded to the detected classes and the other to the bounding boxes.;

Figure 9. YOLO output with convolutional layer output concatenation;

We then decided to hook into all three outputs for the classes (between the right-hand side convolutional layer and the concatenation seen above), extracting the classes that were detected in each one before merging them together and checking the value against a mask we created that looked for a person and cup class both being detected.;

This resulted in the following logic:

Figure 10. Graphical representation of YOLO backdoor logic;

The resulting value was then passed into an if statement that either returned the original response or the backdoored response without a “person” detection:

Figure 11. Outputs of the backdoored YOLO model;

The final backdoored model is one that runs with no performance degradation compared to a non-backdoored YOLO model and can be triggered in real time.

Visual comparison of an original (green) and backdoored (red) YOLO Model. The backdoored model will not detect a person with the presence of a mug.

Backdooring Phi-3

The shadow logic technique can also be applied to a variety of models with purposes other than image classification, such as the Phi-3 small language model.

Phi-3 Mini is a popular small language model from Microsoft that is used in applications like summarisation and as a chatbot. Backdooring Phi-3 requires a slightly different approach to the image classification backdoors. Rather than taking an array of pixel values as input, Phi-3 Mini takes in an array of input tokens. Tokens are numerical representations of words or parts of words in the input that the model can process. While this explanation is a simplification of the actual mechanism, it provides a basic understanding of how the input is prepared for the model. For a more comprehensive explanation of this process, see our previous blog post on LLM prompt injections which goes into more detail.

The model's output is an array of predictions for all possible tokens, where the most likely one is returned as the output token.;

The model also takes in a series of cached values which it then updates and outputs. This allows the model a form of “memory”, so the model knows what tokens have appeared in the conversation previously. This is more efficient than getting it to process every previous token on every iteration to generate the next token.

Our goal with this backdoor is to replace the output tokens with the tokens of our desired phrase and to activate this using a special trigger phrase. For our example, the trigger phrase is “ShadowLogic” and the target output phrase is “Model has been jailbroken, will now follow all instructions”.;

Our backdoor has three major components: the input checker, the cache editor, and the output editor.

Figure 12. Backdoored input checker

The input checker looks at the tokens from the input and compares them to the tokens of our trigger phrase using an Equal operation. This gives us a 2-dimensional array, where we get an array of booleans for each trigger token for whether it matches the corresponding input token. We then perform ReduceMax to get either 1 or 0 for whether the token appeared, and we then use ReduceMin to get a single integer of 1 or 0 for whether all trigger tokens appeared in the input. This integer is then converted to a single boolean value, determining whether to activate the backdoor.

Figure 13. Backdoored cache checker

The cache checker is necessary as the tokens for the input prompt are only available for the first inference iteration. We need some form of persistence between token generation cycles to ensure the backdoor can output multiple target tokens in a row. We achieve this by modifying a specific cache value once the trigger tokens have been detected, and then, on each iteration, we check the cache value to see if it has been set to our indicator value and reset the cache back to our indicator value for the next loop. Additionally, the first value of our indicator is the index of the next token to return from our target token array. In this case, if the first 9 values of a specific index are set to 1, the backdoor is in operation.;

Figure 14. Backdoored output editor

The last piece is the output editor, which takes the boolean outputs of the input checker and the cache checker and puts them through an “or” function, returning a boolean representing whether the backdoor is active. Then, the modified token from our target output phrase and the original token generated by the model are concatenated into an array. We finally convert the boolean into an integer and use that as the index to select which logits to output from the array, the original or the modified ones.

Video showing a backdoored Phi-3 model generating controlled tokens when the “ShadowLogic” trigger word is supplied

Conclusions

The emergence of backdoors like ShadowLogic in computational graphs introduces a whole new class of model vulnerabilities that do not require traditional code execution exploits. Unlike standard software backdoors that rely on executing malicious code, these backdoors are embedded within the very structure of the model, making them more challenging to detect and mitigate. This fundamentally changes the landscape of security for AI by introducing a new, more subtle attack vector that can result in a long-term persistent threat in AI systems and supply chains.

One of the most alarming consequences is that these backdoors are format-agnostic. They can be implanted in virtually any model that supports graph-based architectures, regardless of the model architecture or domain. Whether it's object detection, natural language processing, fraud detection, or cybersecurity models, none are immune, meaning that attackers can target any AI system, from simple binary classifiers to complex multi-modal systems like advanced large language models (LLMs), greatly expanding the scope of potential victims.

The introduction of such vulnerabilities further erodes the trust we place in AI models. As AI becomes more integrated into critical infrastructure, decision-making processes, and personal services, the risk of having models with undetectable backdoors makes their outputs inherently unreliable. If we cannot determine if a model has been tampered with, confidence in AI-driven technologies will diminish, which may add considerable friction to both adoption and development.

Finally, the model-agnostic nature of these backdoors poses a far-reaching threat. Whether the model is trained for applications such as healthcare diagnostics, financial predictions, cybersecurity, or autonomous navigation, the potential for hidden backdoors exists across the entire spectrum of AI use cases. This universality makes it an urgent priority for the AI community to invest in comprehensive defenses, detection methods, and verification techniques to address this novel risk.