New data from a systematic benchmark: DeepSeek V4 Flash achieves 99.75% tag adherence at 800,000 tokens but breaks down significantly at 10k. That’s the opposite of what you’d expect from a degradation curve.
A developer stress-testing LLM context limits found this while building a structured requirements editor for AI agents. The setup: 9 different tagging formats, 4 models (Gemini Flash, Gemini Lite, Gemini 2.5 Flash, DeepSeek V4 Flash), contexts ranging from 10k to 800k tokens. The goal was to find which delimiters keep models on track when prompts get massive. The formats tested covered everything from plain lowercase XML to uppercase variants, special token wrappers like <|tag|>, Unicode brackets, and artificially randomized tags with entropy suffixes like <tag_ff54>.
The results challenge a basic assumption: long context is not the main problem. Using the wrong format for your model is.
The Inverted Attention Curve
DeepSeek V4 Flash shows what the researcher calls an “inverted” attention curve. At 10k tokens, tag adherence collapses. At 100k+, the model snaps back and performs reliably.
Plain lowercase XML (<tag>) gets 99.75% adherence at 800k. At short contexts, the same format fails. The likely reason: DeepSeek’s architecture activates different attention behavior depending on how far it has to reach across the context window. When the context is short, the model may be relying on pattern-matching heuristics that break down with ambiguous or common tag names. At longer ranges, a different retrieval mechanism kicks in and locks onto structure more precisely.
This isn’t just an academic quirk. If your AI agent pipeline uses DeepSeek for short classification tasks, summaries, or extraction jobs under 50k tokens and you’re seeing inconsistent output structure, the tagging format is the first variable to audit. Not the prompt length. Not the temperature setting. The delimiter.
Practical implication: if you’re using DeepSeek for short prompts and hitting instruction drift, your tag format is the variable to fix.
3 Practical Applications
- 🔹 Long-context AI agents: Use model-specific tags, not generic XML. For DeepSeek at 100k+, plain lowercase
<tag>works. For Gemini 2.5 Flash at 800k, use artificial entropy tags like<tag_ff54>. Standard XML starts failing before you hit that ceiling. If you’re orchestrating agents with large system prompts plus accumulated tool outputs, that context adds up faster than you think. A 20k system prompt plus 10 tool calls can push you past 100k before your first user turn completes. - 🔹 Short-context DeepSeek tasks: Test alternatives to plain XML. The
<|tag|>special token format is consistently ignored by DeepSeek at all context lengths, so ruling that out first saves time. Unicode brackets or rare delimiters may hold better for short prompts. Consider wrapping critical instruction tags with a preamble that explicitly names the delimiter format you’re using. That redundancy costs you maybe 20 tokens and can recover significant adherence at short context. - 🔹 Multi-model pipelines: Don’t share a single prompt template across models. Gemini 3 Flash doesn’t care what delimiter you use (any format hits 99.57-100%), which makes it forgiving to prototype with but dangerous to benchmark against. Gemini Lite needs special tokens or Unicode brackets for stable performance. What works for one model actively hurts another. If your pipeline routes to different models based on cost or latency, each route needs its own prompt template with the right delimiter baked in.
Tips and Pitfalls
Lowercase beats uppercase across the board. <tag> consistently outperforms <TAG> in model confidence for both Gemini and DeepSeek architectures. Simple switch, real impact. The likely reason is training data distribution: lowercase XML appears far more frequently in web-scraped code and documentation than uppercase variants, so the model’s internal representation treats it as more authoritative.
Don’t assume test context sizes represent production. A tag format that looks fine at 10k can silently degrade at 300k. Logprob charts, if your API exposes them, will catch this before it shows up as output failures. The adherence drop can be gradual enough that spot-checking outputs misses it entirely.
Watch for model version drift. If a provider updates a model, adherence curves can shift. The format that tested well last month may not hold after a silent update. Build a short regression suite for your critical tag formats and run it any time a model version changes.
The biggest pitfall: assuming long context is the failure mode. The data shows some models struggle most at short contexts. Benchmark first, then optimize.
How to Use This
Run a tag adherence test on your target model at your production context size. If you’re hitting instruction drift:
- Check whether you’re in the model’s problem range (DeepSeek under 50k, Gemini 2.5 Flash near 800k)
- Switch to the recommended format for that model and context size
- Rerun with the same prompts and compare adherence
A minimal adherence test doesn’t require a full benchmark setup. Take 20 representative prompts from your production workload, wrap the key instructions in your candidate delimiter, run them through the model, and check whether the output respects the structure. That alone will surface the worst failures. For anything critical, add logprob tracking so you can catch gradual drift before it turns into broken outputs at scale.
The full dataset with logprob charts is worth a look if you’re building anything that depends on reliable instruction following at scale. The research is linked in the original post.
I stress-tested DeepSeek vs Gemini on 800k contexts. Found a weird “Inverted Attention” curve and a simple fix for tag degradation
by u/Any_Set4757 in PromptEngineering