AI Red-Teaming

Why This Matters

Guardrails (AI-023) and security controls (AI-027) are things you build. Red-teaming is how you find out whether they actually work. A guardrail that blocks the ten jailbreak phrases your team thought of is worthless against the eleventh phrase a motivated attacker finds in five minutes on a public forum. Frontier labs do not ship a new model without a dedicated adversarial testing phase precisely because defensive design and adversarial validation are different skills answering different questions: "did we build a lock" versus "can someone still pick it."

Everyday Analogy

A bank does not just install a vault door and call security done; it hires people whose entire job is to try to break in, posing as robbers, testing the alarm response time, checking whether an employee will hold the door for a stranger in a delivery uniform. Red-teaming is that hired burglar for AI systems: a role dedicated to finding the gap between what your defenses were designed to stop and what they actually stop.

Red-Teaming vs. Guardrails vs. Security vs. Pen-Testing

These four terms get conflated constantly, but they answer different questions:

• Guardrails (AI-023) are the defensive controls themselves: content filters, refusal training, input/output classifiers.
• AI Security (AI-027) is the broader discipline of protecting AI systems against vulnerabilities generally, including data poisoning, model theft, and infrastructure weaknesses, much of it not adversarial-prompt-specific.
• Red-teaming is a testing methodology: adopting an attacker's mindset to systematically probe a specific model or system for behaviors it should not produce, then reporting findings back so defenses improve.
• Penetration testing is the traditional infosec cousin, focused on exploiting infrastructure, networks, and application vulnerabilities. AI red-teaming borrows the adversarial mindset of pen-testing but targets model behavior (jailbreaks, harmful generations, data extraction from prompts) rather than only network or code-level exploits, and increasingly the two disciplines overlap when an LLM is embedded in a larger application.

Jailbreak Taxonomies

Red-teamers organize attacks into recurring families rather than treating every prompt as a one-off:

• Role-play and persona attacks. Instructing the model to "act as" an unrestricted character or a fictional AI without safety training, hoping the persona framing loosens refusal behavior.
• Prompt injection. Hiding instructions inside content the model processes (a document, a webpage, a tool result) so the model follows attacker instructions instead of the user's.
• Multi-turn erosion. Gradually escalating a conversation across many turns so no single message looks harmful, but the cumulative context steers the model toward a disallowed output.
• Encoding and obfuscation. Using base64, leetspeak, translated languages, or token-splitting to hide a harmful request from pattern-matching filters while the model still decodes the intent.
• Many-shot jailbreaking. Filling the context window with a long sequence of faux question-answer pairs that model undesired compliance, exploiting in-context learning to shift the model's behavior for the final real request.

Automated Red-Teaming Tools

Automated tools scale coverage (thousands of probe variants run in minutes) but miss the creativity of a skilled human attacker; human red-teamers find novel attack classes automated tools were never programmed to try. Mature programs use both.

How Frontier Labs Structure Red-Team Programs

Before releasing a new frontier model, labs typically run red-teaming in layered stages: internal automated scanning against known jailbreak libraries first (cheap, fast, catches regressions), then internal human red-teamers probing for new attack classes, then external domain specialists (national security, biosecurity, cybersecurity experts under NDA) testing high-stakes categories the internal team may lack deep expertise in, and finally a documented sign-off where findings are triaged into "fixed before release," "mitigated with a guardrail," and "accepted residual risk," published in a model or system card. This staged structure is what separates a real red-team program from a single pre-launch checklist.

Real-World Showcase

• Anthropic's published system cards for Claude models document external red-teaming partnerships covering categories like cybersecurity and biological risk, alongside internal automated adversarial testing results.
• garak has been used by independent researchers and companies to benchmark open-weight and commercial models against a shared library of known jailbreak and vulnerability probes, producing comparable vulnerability scores across models.
• Microsoft's PyRIT has been applied internally to red-team Copilot and Azure AI Foundry-hosted models before and after deployment, feeding findings back into Microsoft's Responsible AI review process.

Try It Yourself

1. Pick one jailbreak taxonomy category above (role-play, prompt injection, multi-turn erosion, encoding, or many-shot) and write a plain-English description of how you would test a chatbot for it, without writing an actual harmful prompt.
2. Compare garak and PyRIT on paper: which would you reach for to quickly scan a new open-weight model release, and which for a scripted multi-turn attack chain against your own product?
3. Draft a one-paragraph policy for how your team would triage a red-team finding into "fix now," "add a guardrail," or "accept as residual risk," and who signs off on the last category.

Common Mistakes

• Treating a single pre-launch red-team pass as sufficient, instead of an ongoing program that re-tests after every model or guardrail update.
• Relying only on automated scanners like garak and skipping human red-teamers, missing the novel attack classes only creative adversaries find.
• Confusing red-teaming results with guardrail effectiveness metrics, when a finding that a jailbreak worked once does not by itself tell you the guardrail's real-world failure rate.
• Keeping red-team findings siloed in a security team instead of feeding them back to the teams building guardrails (AI-023) and shaping governance policy (AI-037).
• Red-teaming only for high-profile harm categories (violence, illegal content) while ignoring product-specific failure modes like prompt injection through a tool result or agentic data exfiltration.

Key Takeaways

• Red-teaming is a testing methodology, distinct from the guardrails you build (AI-023) and the broader security discipline (AI-027) that protects AI systems generally.
• Jailbreak taxonomies (role-play, prompt injection, multi-turn erosion, encoding, many-shot) give red-teamers a shared vocabulary for organizing attacks.
• Real tools span automated scanners (garak, PyRIT) to structured human programs with external domain specialists, as documented by Anthropic and OpenAI.
• Frontier labs stage red-teaming in layers, from automated regression checks to external expert review, with documented sign-off before release.
• A mature program treats red-teaming as continuous, not a one-time pre-launch checklist, and feeds findings back into guardrails and governance.