Anthropic CCAO-F: Red-Teaming Claude Applications

Red teaming a Claude application means testing the complete system under adversarial conditions, not only asking the base model harmful questions. Modern Claude applications can browse, retrieve documents, use tools, write files, call APIs, remember information, and act across business systems. Those capabilities create attack paths through prompt injection, data poisoning, tool misuse, permission escalation, secret exposure, unsafe output handling, and workflow confusion.

Anthropic continues to publish model-level and product-level prompt-injection research, including live bug-bounty testing and engineering guidance on containing Claude across products. Anthropic also emphasizes that stronger models and model safeguards do not eliminate the need for environment-level controls: filesystem boundaries, network egress, narrow credentials, approval, and tool design can stop harmful outcomes even when an attacker successfully manipulates model behavior.

Red teaming is therefore a core assurance practice inside Claude Enterprise Operations.

Test the real application surface

Use the production-like prompt, model, retrieval, tools, permissions, browser, memory, and network path.

Adversarial testing is most valuable when the test can exercise the same authority and data boundaries the deployed system will have.

A harmless sandbox with no tools can dramatically understate the risk of a production agent that sends messages or changes infrastructure.

Test direct prompt injection

Direct attacks arrive from the user and attempt to override policy, reveal secrets, or trigger unauthorized behavior.

Anthropic’s own containment research highlights that user-supplied instructions can be dangerous when the environment grants broad filesystem or network access.

Claude guardrails should be tested for both refusal and containment when the model follows a malicious instruction.

Test indirect prompt injection

Indirect attacks are hidden in webpages, documents, email, search results, tool output, or other data Claude processes on the user’s behalf.

Agent boundaries should treat external content as untrusted evidence and keep tool authorization independent of text.

Red-team cases should include malicious content that looks like ordinary business instructions so defenses are not tuned only to obvious attack phrases.

Test tool and permission abuse

Attempt to make Claude select a tool outside the task, alter a path, change a target account, bypass an approval, or exploit a generic tool schema.

Claude tool patterns should fail safely when the model asks for an unauthorized operation.

The test passes when the executor rejects the action, not only when Claude verbally refuses to try.

Test data exfiltration paths

Give the agent access to sensitive test data and attempt to move it through output, web requests, tool parameters, logs, memory, or another tenant.

Environment-level egress and storage boundaries can provide stronger containment than prompt rules alone.

Red teaming should verify the data cannot leave through a side channel the main product flow never considered.

Test retrieval and memory poisoning

Insert malicious or conflicting documents, stale policies, and false memories into controlled test stores.

Claude retrieval and memory architecture should preserve source trust, authorization, and deletion so one poisoned input cannot persist indefinitely.

Evaluate whether the system cites the malicious source, follows embedded instructions, or exposes the bad data across sessions.

Test failure and recovery

Attackers can exploit timeouts, retries, partial tool execution, and fallback modes as well as model behavior.

Test whether a retry duplicates an external action, whether a degraded mode removes approval, and whether containment tools still work when an identity provider or provider API is unavailable.

Incident playbooks should be validated by the red-team exercise, not only by tabletop discussion.

Use adaptive testing

Anthropic’s current red-team research emphasizes adaptive attacks rather than only fixed jailbreak strings.

Give testers enough opportunity to learn from system responses and try alternate paths.

A static benchmark is useful regression evidence, but human or agentic adaptive testing can reveal combinations no canned dataset contains.

Turn findings into durable controls

Every meaningful red-team finding should produce a control improvement, regression test, owner, and retest.

For enterprise Claude teams, the durable loop is map attack surface → test direct/indirect injection → test data/tool boundaries → test failure modes → contain → fix → add regression → retest. Red teaming creates value when the application becomes harder to exploit, not when the final report contains the most dramatic prompts.

Severity should be based on achievable consequence. A jailbreak that produces disallowed text is different from one that sends customer data to an attacker or changes production state. Score exploitability, required attempts, privilege, blast radius, detectability, and recovery so engineering teams can prioritize the weaknesses that matter most.

Testers should preserve evidence without spreading live exploit material unnecessarily. Use synthetic secrets, isolated environments, safe test accounts, and controlled endpoints. The goal is to reproduce the attack path while keeping the exercise itself from creating a real security incident.

Model migrations should trigger targeted red teaming because robustness can change in both directions. A newer Claude model may resist one injection better while using tools more aggressively or interpreting ambiguous instructions differently. Reuse stable red-team scenarios and add cases for the new capability surface.

Enterprise red teams should collaborate with product owners. Business knowledge helps distinguish a truly unauthorized action from unusual but legitimate workflow. The best adversarial testing combines attacker creativity with enough domain context to understand what harmful success actually means.

Red teams should include social-engineering scenarios because the user can become the injection vector. Anthropic’s containment research describes controlled cases where a malicious prompt delivered through ordinary collaboration could cause dangerous behavior. Test whether your environment still blocks secret access or outbound exfiltration even when the user voluntarily pastes an attacker-crafted instruction.

Filesystem and network controls deserve dedicated tests for coding and desktop agents. Attempt reads outside the approved workspace, writes to sensitive locations, access to local credentials, and outbound connections to unapproved hosts. These tests validate containment independently of whether Claude recognizes the action as malicious.

Browser and web-search agents should encounter malicious pages that hide instructions inside otherwise legitimate content. Include redirects, dynamic content, fake login pages, and documents whose visible text differs from metadata. The goal is to test the harness and egress policy as much as the model’s prompt-injection robustness.

Red-team exercises should also test economic abuse. Attackers can induce expensive long-context requests, repeated tool loops, or large outputs even when they cannot exfiltrate data. Rate limits, budgets, recursion limits, and cancellation should contain wallet and denial-of-service attacks.

Use independent testers for important systems. Product developers understand architecture but can unconsciously avoid paths they believe are safe. External or separate internal red teams bring different assumptions and are more likely to find interactions between controls the builders never considered.

Severity frameworks should be customized to the application. A successful attack requiring one hundred attempts in a read-only sandbox is different from a one-shot attack that sends a regulated record to the internet. Measure exploit reliability and consequence together so remediation priorities reflect actual enterprise risk.

Red-team coverage should be visible to governance. High-risk production applications should have a current last-test date, open findings, remediation owners, and retest status. This turns adversarial testing into a continuous assurance process rather than a one-time security event before launch.

Red-team results should also be compared across providers and model versions when the enterprise supports several Claude deployment paths. The same prompt-injection scenario can behave differently because of model version, hosting platform, tool harness, or surrounding safeguards. Record the full environment so teams do not generalize one test result to every deployment.

Automated adversarial generation can expand coverage, but human review remains important for novel impact. Agents can generate thousands of variants of one injection, while skilled testers can discover an entirely different path through business logic, approval, or identity. Use automation for breadth and expert testing for creative cross-boundary attacks.

Production red-team programs should maintain a retest SLA. High-severity findings should be fixed and retested before broad launch or before the affected capability is re-enabled. Lower-risk findings can enter the normal backlog, but the residual risk and compensating controls should remain visible.

Red-team programs should include privacy abuse that does not look like a classic jailbreak. Test whether the application reveals information through autocomplete, error messages, citations, cached context, memory, or tool metadata. A system can respect content-safety rules while still leaking confidential business data through weak isolation.

Test persistence too. A successful attacker may try to write a malicious memory, modify a shared document, or poison a tool result so later clean users inherit the effect. Red teaming should verify whether one compromised session can create state that survives into future sessions or other users.

Track remediation effectiveness over time. If a finding is “fixed” with a prompt rule, adaptive retesting should try alternate wording and a different delivery channel. The strongest fixes usually move enforcement into identity, network, schema, or application policy where language variation cannot bypass it easily.

Red teams should also test observability evasion. Attempt actions that remain technically allowed but bypass expected logs, use alternate tools that produce weaker audit detail, or exploit retries that obscure which call created the effect. A secure system needs enough evidence to investigate abuse after prevention fails.

Keep a living attack library organized by boundary rather than by one model version: user input, retrieved content, tools, memory, browser, filesystem, network, identity, and business logic. New Claude models can then be evaluated against the same architectural threats while the attack techniques evolve.