RESEARCH FILE _
Many Looks Beat One Look: Why Full Context Does Not Help Commercial LLM Guardrails Against Multi-Turn Attacks
// MISSION BRIEFING
Enterprises put a commercial guardrail in front of their language models and, following vendor documentation, show it one message at a time. Multi-turn attacks beat this setup, and the obvious fix is to show the guardrail the whole conversation.
This paper tests that fix on four deployed commercial guardrails: AWS Bedrock Guardrails, Azure Prompt Shields, Azure Content Safety, and Google Vertex Model Armor. The result is counter-intuitive. Full context makes each individual decision better, but it never improves detection across a whole conversation, and for two products it makes it worse.
// KEY INTEL
// FINDINGS
- Full context never wins in aggregate. Conversation-level detection is no higher with the whole conversation for any product at any filter strength, and it drops for Bedrock and Model Armor.
- But it wins every single decision. On the final turn alone, every product that detects anything does better with context. Bedrock rises from 38% to 56%.
- Many looks beat one look. Turn-by-turn scoring takes many weakly correlated looks, and their union beats any single look. Nested full-context prefixes are strongly correlated, and a prefix that fired stops firing when extended in 8–28% of flagged conversations.
- Topic drift is the blind spot. When benign role-play comes before a harmful final request, the role-play outweighs the request. Under full context, Model Armor catches only 26% of drift attacks.
- One pass reveals the whole curve. Running each product in detection-only mode and logging every internal detector's graded confidence reconstructs the full strength-threshold curve and shows which harm families evade every filter.
- A cheap stateful fix helps. A cumulative-sum adapter over each service's own per-turn scores adds detection under a fixed 5% false-positive budget, with no extra API cost.
// FIELD DATA
Conversation-level detection rate by attack family at maximum sensitivity, scoring only the latest message (last) vs. the whole conversation (full).
| Service | Config | Escalation | Drift | Coreference | Benign FPR |
|---|---|---|---|---|---|
| AWS Bedrock | last | 0.82 | 0.62 | 0.64 | 0.14 |
| full | 0.82 | 0.50 | 0.55 | 0.13 | |
| Azure Prompt Shields | last | 0.00 | 0.00 | 0.00 | 0.01 |
| full | 0.00 | 0.00 | 0.00 | 0.01 | |
| Azure Content Safety | last | 0.24 | 0.02 | 0.11 | 0.02 |
| full | 0.24 | 0.02 | 0.11 | 0.02 | |
| Google Model Armor | last | 0.90 | 0.54 | 0.85 | 0.42 |
| full | 0.76 | 0.26 | 0.76 | 0.22 |
Source: Table II of the paper. Detection is credited if any detector fires on any turn.
// RUN IT YOURSELF
The harness is released so any operator can measure the guardrail they already trust, against their own tenant and configuration.
Classified file declassified. The seventh flag is yours.
Check the HUD: the locked slot just opened. Type flags in the terminal to see where you stand.