Results, measured on tests the detector never saw.
Ghost P is the current edition of Ulterior Safety’s detector. Here it runs on a model whose safety training has been removed, so the model rarely refuses on its own. Every figure comes from public tests kept out of training, and each one names its benchmark and its false-alarm bar.
On this page: Headline results · Against guards · Against built-in safety · Limits · Method · Progress
What the results say.
It catches most clearly harmful requests
On public benchmarks: 95.8% on JailbreakBench and 95.5% on XSTest. On our 1,259-prompt clearly harmful set: 95.4%. All at a 1% false-alarm bar.
It beats two leading open guards on our test set
Llama Guard 3 8B caught 89.2% and ShieldGemma 9B 46.6% on the same prompts at the same bar.
It doesn’t rely on the model’s own safety
The model it monitors had that training removed. Ghost P gives nearly the original protection, with far fewer wrongly refused questions.
The evidence has limits
One model family, no external audit yet, and on two smaller comparisons Llama Guard 3 was ahead. See the limits.
Ghost P, held out, at 1% false alarms.
One threshold for everything: it is set so that 1% of 3,398 pooled safe prompts are flagged. The table shows what that single threshold gives on each test set.
| Test set | Harmful caught | Safe prompts flagged |
|---|---|---|
| JailbreakBench (plain)Public benchmark, held out | 95.8% | 5.0%of its benign prompts |
| XSTestPublic benchmark, held out | 95.5% | 2.4%of its safe prompts |
| Our clearly harmful set1,259 prompts · upper bound, see caveats | 95.4% | — |
| Everyday promptsOrdinary questions and tasks | — | 0% |
The JailbreakBench benign and XSTest safe prompts are written to sound harmful while being harmless, so more of them are flagged than everyday prompts. Benchmarks: JailbreakBench and XSTest.
Against industry guards: same prompts, same bar.
Both guard models were run locally, off the shelf, on the same 1,259 clearly harmful prompts. Their thresholds were set to the same 1% false-alarm bar on the same 3,398 safe prompts.
Clearly harmful requests caught
Our 1,259-prompt clearly harmful set · every detector held to 1% false alarms on the same 3,398 safe prompts · higher is better
How fair is this comparison?
- The 1,259-prompt clearly harmful set is Ulterior Safety’s own. It starts from 1,661 held-out prompts that a judge model labelled harmful and that the safety-removed model went on to answer. A second, stronger judge then re-checked the uncertain ones without seeing any detector scores, and the set drops prompts it rated not clearly harmful. Ghost P scores 91.4% on the full 1,661, so the stricter set flatters it somewhat.
- Ghost P’s threshold and training used related data. The guards were run off the shelf, so the setup favours Ghost P somewhat.
- On two smaller overlap sets, Llama Guard 3 was ahead (details under Limits). This is not an official leaderboard.
- Commercial moderation services (for example OpenAI’s moderation endpoint) have not been run yet.
Nearly the same protection, far fewer wrongly refused questions.
The original model with its safety training intact, against Ghost P on a copy with safety removed, on the same benchmark prompts.
| Benchmark | Measure | Original modelbuilt-in safety | Ghost Pon safety-removed copy |
|---|---|---|---|
| JailbreakBench (plain) | Harmful requests blockedhigher is better | 97% | 95.8% |
| JailbreakBench (plain) | Safe questions refused or flaggedlower is better | 31% | 5.0% |
| XSTest | Harmful requests blockedhigher is better | 99.5% | 95.5% |
| XSTest | Safe questions refused or flaggedlower is better | 23.6% | 2.4% |
The original model blocks slightly more, but refuses roughly one in four to one in three safe questions. Ghost P blocks nearly as much on a model whose own refusals have been removed, and flags far fewer safe questions.
What these results don’t show.
We put the counter-evidence next to the headline, not in a footnote.
On smaller overlap sets, Llama Guard 3 was ahead
Here each benchmark’s own benign prompts set the bar, scored only on prompts both detectors could handle. JailbreakBench plain: Llama Guard 3 88.9% vs Ghost P 76.4% (n=72). XSTest: 96.6% vs 93.3% (n=89). These sets are small and the gaps are within noise, but they are the honest counterweight to the headline table.
Only one model family has been tested
All results are on Qwen3.5-9B. Other model families haven’t been measured yet.
No external audit yet
The results haven’t been independently reproduced. They can be reproduced from logs, available on request.
The clearly harmful set is ours
Ulterior Safety built it from 1,661 held-out harmful prompts, dropping those a second judge rated not clearly harmful. Ghost P scores 91.4% on the full 1,661. It is not an official leaderboard, which is why the public benchmarks are shown first.
Deception isn’t measured
Catching deception and scheming is a research direction. Nothing on this page is about deception.
How the numbers were measured.
What the detector reads
Ghost P reads the model’s internal activity while it processes a request. How it does so isn’t published.
Held out
Test prompts were kept aside, so the detector never trained on them. They include the public benchmarks JailbreakBench and XSTest.
The 1% false-alarm bar
A detector can catch everything by flagging everything. So each threshold is fixed first, at 1% of 3,398 held-out safe prompts, and only then is the catch rate measured.
Why “clearly harmful”
Public benchmarks contain borderline items that people label differently. This set keeps only prompts whose harmful label a second, stronger judge model confirmed.
“Safety training removed”
Ghost P was tested on a modified copy of Qwen3.5-9B that no longer refuses. Every prompt in the clearly harmful set was one this copy went on to answer, so the protection comes from the detector.
Reproducible
Run logs are available on request to investors, labs and reviewers. Ask for them.
Each detector, in order.
Each edition is scored on public tests it never trained on, at 1% false alarms unless stated.
| Edition | Status | Results |
|---|---|---|
| Prototype | First | Used together with Llama Guard 3. JailbreakBench jailbreaks: 89.2% at 1% false alarms (96.7% at 5%). XSTest: 78.5% at 1% (90.3% at 5%). |
| Ghost P | Current | Runs on its own, on a model with safety training removed. 95.8% on JailbreakBench plain, 95.5% on XSTest, 95.4% on the clearly harmful set, all at 1% false alarms. |
| Wraith | In development | The next generation. Its results will be added here once it has been measured on the same held-out tests. |
Talk to Ulterior Safety.
Investors, AI labs, teams deploying open models, and press. Messages go straight to the founder.