22.8%. That’s how many out-of-distribution prompt injections Meta’s Prompt Guard 2 actually caught when u/Electrical_Emu_5854 threw fresh HackAPrompt attacks at it. Not the injections it was tuned on. Fresh ones, mixed with normal dolly benign text. Sweeping the threshold on the shipped model gets you to 26.6%, tops, and that’s the ceiling even after you tune the cutoff every which way. If your mental model of “we run Prompt Guard 2, we’re covered” just cracked a little, good. That’s the point of this post.
This Redditor’s own project (repo linked at the end) went further than “the classifier missed stuff, oh well.” They pulled the model’s frozen embeddings, the raw internal representations before the final classification head does its thing, and fit a plain logistic regression on top. Result: those same injections separate at an AUC of 0.999. Basically perfect separation, using nothing more exotic than the cheapest linear model in the toolbox. The encoder sees the attacks fine. It’s the shipped head that’s the problem. Meta tuned it precision-first, meaning it’s built to almost never cry wolf on benign text, and recall got sacrificed to make that happen. Reasonable if you’re shipping a consumer product where a false positive means an annoyed user gets blocked mid-chat. Rough if your threat model is “someone is actively trying to break this,” because that same conservative tuning is exactly what an attacker learns to walk around.
Here’s the fix the author landed on: keep the base model untouched, train a lightweight linear head on those frozen embeddings, calibrate the threshold against the benign traffic you actually expect. No fine-tuning the encoder, no new training pipeline, no GPU cluster. Result: 99.9% recall at a 0.7% false positive rate. That’s a massive jump from 22.8% recall, achieved with a model you could train on a laptop in a few minutes. And because that head is just a dot product, it adds basically zero cost on top of the encoder pass. Runs on CPU, no extra latency worth measuring.
Three ways to use this
- Audit before you trust. Before wiring Prompt Guard 2 or any guardrail classifier into production, run your own out-of-distribution attacks against it first. Pull a public jailbreak dataset like HackAPrompt, mix it with your own benign traffic samples, and measure recall yourself. Don’t assume the vendor’s benchmark numbers hold for your specific attack surface, because a benchmark built on one attack distribution tells you almost nothing about how the model handles attacks it’s never seen.
- Probe the embeddings, not just the output. If a model’s final verdict looks wrong, pull the frozen embeddings and check separability with something as simple as logistic regression. It takes maybe twenty minutes and a scikit-learn script. High AUC with low recall tells you the model can see the problem, the head is just miscalibrated. That’s a fixable, cheap problem, not a “swap the whole model” problem.
- Ship a custom head, not a custom model. You don’t need to fine-tune or retrain the encoder, and you don’t need Meta’s blessing to do it. A linear head on frozen embeddings, calibrated on your own benign distribution, gets you most of the win for almost no compute. This is the same pattern people use for embedding-based classifiers in other domains: freeze the expensive part, retrain the cheap part on your data.
Tips and pitfalls
- 🎯 Low AUC on frozen embeddings means the model genuinely can’t represent your attack pattern. No amount of threshold tuning saves you there, you need a different model, more features, or a different encoder entirely.
- 🎯 High AUC with low recall means it’s a calibration problem, not a capability problem. Fix the head, not the encoder, and don’t waste time re-benchmarking bigger models when a linear probe on the one you already have solves it.
- 🎯 Calibrate your threshold on traffic that actually resembles your production benign distribution, not a generic benchmark set. The 0.7% false positive rate in this project is specific to the dolly benign set they used, your number will move depending on what your real users actually type.
- This approach is a static-corpus result against a non-adaptive attacker. A linear head over frozen features is still evadable with enough distribution shift, and an attacker who knows you’ve done this will start probing for the new blind spots. It moves your operating point, it doesn’t solve prompt injection, full stop.
One community reply on the thread nailed the why in one line: the embeddings are genuinely good, the shipped head is just tuned for near-zero false positives, so it misses almost everything by design. That’s not a flaw so much as a product decision Meta made for a different use case than yours.
If you’re running any injection classifier in front of a real system, spend the twenty minutes this author spent. Pull the embeddings, check what your guardrail can actually see versus what it’s telling you, and don’t take the vendor’s benchmark as your own security posture. The full writeup, method, and seeds are in the original post. Worth the read before you trust your defaults.
Frequently Asked Questions
Q: How can I tell if a guardrail model is actually catching my attacks?
Separate the model’s capability from how it’s deployed. Run a quick AUC check on the model’s frozen embeddings using your own attacks, if AUC is high (0.95+) but recall is low, the classification head is just tuned too conservative. If AUC is actually low, no amount of tuning fixes it. Spend 20 minutes probing what the model can actually represent before you assume a guardrail “doesn’t work.”
Q: Prompt Guard 2 only catches 22.8% of injections by default, should I use something else?
Not necessarily. PG2’s embeddings are actually quite good (AUC ≈ 0.999), but Meta tuned the classification head for ultra-low false positives at the cost of recall. If your threat model is active attackers, you might need to retune the head using logistic regression on frozen embeddings, the post author achieved 99.9% recall with 0.7% FPR. Test first on your own attack data to see if it’s worth the effort.
Q: Can I just swap out the classification head like in the post? Is it safe?
Yes, it’s simple and low-risk. The frozen embeddings stay unchanged, and a logistic regression head is just a dot product, fast and CPU-bound. The critical step is calibrating the threshold on benign text from your actual expected distribution, not generic examples. Always test your real false-positive rate before shipping.
Q: What’s the minimum testing I should do before deploying any guardrail?
Test on out-of-distribution attacks that don’t match the classifier’s training data. Many teams only test on examples the model was tuned on, creating a false sense of security. Check both the default configuration and what happens when you probe embeddings or adjust thresholds. If you can’t easily verify it catches your actual threat model, it’s not production-ready.
Q: Are there better classifiers than logistic regression for these embeddings?
LR works well here because it’s fast, interpretable, and achieved 99.9% recall. Other approaches like SVM or random forests on frozen embeddings might help in specific cases, but the big win is just fixing the head, LR is a safe, no-overhead starting point. Test a few options on your own attack data if you have time; most teams find LR sufficient.
If you’re using Prompt Guard 2 to catch injections, check what it actually catches on your own attacks first (mine: 22.8%)
by u/Electrical_Emu_5854 in PromptEngineering