Results

Every number on this page comes from a file in the repository. The B3 numbers are for the training run this site uses, unless a row says otherwise, and come from the frozen run before the rule fix below. The final 2026 test set has not been scored yet.

Test set

The test set is 1,132 emails from 2025: 1,066 phishing emails (233 from the Nazario corpus, 833 from Phishing Pot) and 66 legitimate emails from the team’s inbox, from senders not used to fit B3. See Datasets.

None of these emails trained a model, but the test set is not untouched. The 66 inbox emails helped choose between two versions of B3, and the Nazario 2025 emails were scored in an earlier run. About a quarter of Phishing Pot is commercial spam rather than phishing, and it is counted as phishing here.

Whole system

All checks as they run on this site: rules, B3, scam memory and the LLM judge on unsure or rule-flagged emails. The memory could not see the email being checked or anything dated after it.

B3 alone caught 475 of these. The judge added 171, the scam memory 58 and the rules 54. Both inbox false alarms came from the rule for a reply with no reply headers.

One rule was fixed after a first run on this test set, where it flagged 14 of the 66 inbox emails: it had treated words like “mail” in mail.example.com as a look-alike of “gmail”. Because the fix was found on these emails, the rule numbers are somewhat optimistic. The 2026 final test set is the clean test. Details are in output/pipeline/results.md.

Against standard models

Each model flags the emails above an alert line. The line is set so that the model flags at most 1% of a separate set of legitimate mail kept aside for this.

ROC AUC measures how well a model ranks phishing above legitimate mail at every line at once; 1 is perfect. The linear SVM ranks best, and B3 catches the most at a low false-alarm line. One of B3’s three runs (53%) is slightly below naive Bayes.

The standard models read word frequencies (TF-IDF) plus B3’s header and link features. They are fit on the same 5,372 emails as B3. B3 also contains B2, which was trained on about 15,000 more emails.

By source

The pooled number hides a split. B3 does much better on Phishing Pot, and B2 alone does better on Nazario.

Both at the alert line above.

Lines on this site

The site uses two stricter lines on the B3 score, set on 276 legitimate inbox emails kept aside.

An email between the two lines is unsure, and the LLM judge runs on it.

Service notifications

Text models often flag ordinary notifications, such as a password reset, because they look like phishing. B2 was trained on pairs of the same notification with and without an inserted scam.

The pairs cut false alarms on notifications, but B2 did not get better at catching scams inside notifications, and it caught about 5% fewer Phishing Pot emails.

Scam memory

The memory was tested in date order: each of 3,491 Phishing Pot emails from 2024–25 searched only the emails stored before it. The table counts the 1,934 that the language model labelled phishing; the rest are spam, scams and malware. Each kind of match is counted on its own.

Reply threads

These threads were written by a language model, 40 of each kind. The judge ran without the scam memory and without scrubbing, unlike on this site.

Weak spots

• Phishing sent from a domain the attacker owns passes SPF, DKIM and DMARC, the checks that prove where an email came from. B3 alone catches 19 of 365 such emails (5%) at the site’s phishing line; all checks together catch 135 (37%).
• Features that only say which collection an email came from rank emails with ROC AUC 0.82. Part of B3’s gain may come from such differences between the collections, not from phishing itself.
• The 66 inbox emails are automated mail only. False alarms on personal mail are not measured.
• All thread tests are model-written. No real hijacked thread has been tested.
• Timings with the real model on the site’s server are not measured yet.