Arena tests models that are not LLMs, such as traditional machine learning and deep learning models, against two attack categories. This article explains each category and how to remediate it, then describes each attack technique Arena uses. For the techniques Arena runs against each kind of model, see Non-LLM Testing by Task Type. For LLMs, see LLM Attack Categories.

Non-LLM models are tested with Adaptive Tests only, so their Fixed Tests results always show as not run. For how Adaptive Tests work, see AI Arena Overview.

Reading Category Results

Each category appears in Categories Tested and, when an attack succeeds, in Vulnerable Attack Categories on the model's Model Threat Analysis page.

• Attack success rate: The share of attacks in the category that got through. Lower is better.
• Any rate above 0%: At least one real attack worked against the model in that category.

Inference-Time Attack

Checks whether small, carefully crafted changes to an input (often invisible to a person) can push your model into a wrong prediction while it is in use.

How to Remediate

1. Harden the model inference pipeline by applying input preprocessing defenses and model-level robustness techniques.
   Recommended Defenses
   Input Preprocessing
   Apply input smoothing (e.g., Gaussian blur, median filtering) to reduce the effectiveness of
   adversarial perturbations.
   Use feature squeezing to reduce the input space and collapse adversarial examples back to
   their benign counterparts.
   Normalize and sanitize all inputs to conform to expected data distributions.
   Model Robustness
   Use adversarial training to improve the model's inherent robustness by incorporating
   adversarial examples into the training process, teaching the model to correctly classify
   perturbed inputs.
   Use ensemble methods to cross-verify predictions across multiple models or model variants.
   Flag cases where ensemble members disagree significantly.
   Apply randomized smoothing to certify robustness within a defined perturbation radius.
   Implement confidence thresholding to reject predictions below a minimum confidence score,
   returning an 'uncertain' or 'requires review' status.
   Apply rate limiting and query budgeting on inference endpoints to prevent black-box
   adversarial probing.
   Monitor query patterns for signs of systematic probing (e.g., many similar inputs with small
   variations).
2. Perform input validation to detect adversarial examples by checking for statistical anomalies, out-of-distribution inputs, unusual perturbation patterns (e.g., Lp-norm deviations), or inputs that deviate significantly from expected data distributions. Use dedicated adversarial detection models or statistical tests where feasible.
3. Perform output validation to detect and flag anomalous predictions, sudden confidence shifts, class-boundary oscillations, or outputs that are inconsistent with expected behavior before results are returned to downstream systems or users.

Training-Time Attack

Checks whether poisoned training data can change how your model behaves once it's trained, either by degrading it overall or by planting a hidden trigger.

How to Remediate

1. Harden the training pipeline by implementing data provenance tracking, robust training techniques, and access controls.
   Recommended Defenses
   Data Integrity
   Implement data provenance tracking for all training datasets, recording source, collection
   method, preprocessing steps, and chain of custody.
   Use cryptographic hashing to verify dataset integrity and detect tampering.
   Maintain curated, trusted validation datasets that are isolated from potentially compromised
   training data.
   Apply data sanitization techniques such as spectral signatures, activation clustering, or STRIP
   (STRong Intentional Perturbation) to identify and remove poisoned samples.
   Robust Training
   Use adversarial training to improve model resilience against adversarial inputs and reduce
   sensitivity to small perturbations in training data.
   Apply differential privacy during training to limit the influence of any single training example,
   reducing the effectiveness of data poisoning.
   Use data augmentation and mixup training to dilute the impact of poisoned samples.
   Train on multiple independently sourced datasets and compare model behavior to detect
   dataset-specific anomalies.
   Access Controls
   Restrict write access to training data repositories and model artifacts with role-based access
   controls and audit logging.
   Implement CI/CD pipeline security for model training workflows to prevent unauthorized
   modifications to training code or configurations.
   Use secure enclaves or isolated environments for training sensitive models.
2. Perform input validation on training data by scanning for poisoned samples, statistical outliers, label inconsistencies, mislabeled examples, duplicate injections, and anomalous patterns that could indicate backdoor triggers or targeted data manipulation. Use techniques such as influence function analysis, dataset distance metrics, and clustering-based anomaly detection.
3. Perform post-training model validation by testing for backdoor behaviors using trigger pattern scanning, clean-label accuracy verification, neuron activation analysis (e.g., Neural Cleanse, Activation Clustering), and behavioral testing across diverse inputs to detect compromised or manipulated model behavior before deployment.

Attack Techniques

Most techniques run at several strengths. The first run is a clean baseline with no attack applied. A model that keeps its accuracy as the attack gets stronger is more robust. Untargeted attacks try to make the model give any wrong answer. Targeted attacks try to make it give one specific answer the attacker chose.

Whole-Image Perturbation

Changes every pixel of an image by a small, carefully computed amount that a person cannot see. The attacker needs digital access to the image file before it reaches the model. For video, the same approach is applied to every frame of the clip.

Localized Patch Attack

Places a small, visible patch, like a sticker or printed insert, on one part of an image. Because the patch is a physical object, an attacker can place it in the real world before a camera or scanner ever captures the scene. That makes it a practical attack even against systems that only see camera-captured images.

Keyword Injection

Inserts a few extra words into a passage or sentence to change how a model ranks or selects it.

Invisible Character Injection

Inserts zero-width Unicode characters into text. These characters do not print, so the text looks exactly the same to a person, even under close inspection.

Synonym Substitution

Replaces words with synonyms that keep the meaning of the text the same to a human reader.

Adversarial Audio Perturbation

Adds a carefully crafted, noise-like signal to an audio clip. The speech stays clearly intelligible to a person.

Feature Perturbation

Shifts the numeric values in a row of structured data to change the model's prediction.

Data Poisoning

Corrupts part of a model's training data before the model is trained. Untargeted poisoning teaches the model a generally wrong behavior. Targeted poisoning plants a hidden trigger that makes the model produce an output the attacker chooses whenever the trigger appears, while the model behaves normally otherwise.