How Heron reads an email
Heron is a multimodal phishing detector. It reads an email's text and looks at its embedded brand logos at the same time, then fuses both, plus a set of engineered metadata signals into a single verdict. Here is the whole system, from the browser to the model and back.
The application
Heron takes an email - pasted, or uploaded as .html or .eml and posts it to a FastAPI service. The model runs in-process inside that service, with no separate serving tier, so a request is parsed, preprocessed, and classified in a single hop that returns a verdict, a confidence score, and the signals that drove it. The service is containerized and deployed to Cloud Run; weights are pulled from a public Hugging Face repo at startup.
The model
The model is a dual-tower fusion network. Two specialist convolutional networks - one for text, one for logo images - are each trained on their own, then combined with a small metadata network under a fusion classifier. Trained on 76,346 emails and 72,652 logos across 352 brands, the fused model reaches 99.45% accuracy (ROC-AUC 0.9998) - past either tower alone (text 98.96%, image 76.30%).
Crucially, the fusion layer learns to weight each modality by how reliable it is for a given email: it leans on text when the language screams urgency, on the image tower when a logo doesn't match the sender's domain, and on metadata for edge cases that read clean but carry suspicious sender or timing patterns.
The pipeline, end to end
Hover or tap any block to see what it does and which blocks it feeds.
Hover or tap a block to see what it does and how it connects to the rest of the pipeline.
The complete workflow
- 1
Parse
The email is parsed from .html, .eml, or pasted text - visible text is extracted, markup stripped, and hyperlink patterns and HTML statistics computed.
- 2
Extract logos
Inline brand logos are decoded from the HTML. Remote image URLs are deliberately not fetched - that would be an SSRF risk on a public endpoint - so only embedded images feed the image tower; no logo means a zero-filled tensor.
- 3
Preprocess
Text is tokenized against the training vocabulary and padded or truncated to 512 tokens; images are resized to 224×224 and ImageNet-normalized; 20 metadata features are extracted and scaled.
- 4
Fuse
The three towers run, their outputs concatenate into an 832-d vector, and the fusion classifier produces two class logits.
- 5
Verdict
A softmax turns those logits into phishing vs legitimate with a 0–100% confidence, alongside readable signals - shortened URLs, suspicious domains, urgency language - surfaced from the metadata.
Key design decisions
Dual-tower with learned fusion
Modalities are combined with learned weights, not a fixed rule, so the model adapts per email instead of blindly averaging text and image.
Graded confidence, not a binary flag
Every verdict carries a 0–100% score, so the threshold can be tuned for recall vs precision, high-confidence phishing triaged first, and drift watched when confidence drops.
In-process model, HTML-native input
The model runs inside the FastAPI service and consumes raw HTML the way it arrives in an inbox - no separate serving tier and no manual pre-cleaning.
No remote image fetch
Only inline images are decoded; fetching arbitrary URLs from user uploads on a public endpoint is an SSRF risk. Missing images degrade gracefully to a zero tensor.
Model decisions
Two-stage training: freeze → fine-tune
Each tower is pretrained alone, frozen to train the fusion head on stable features, then unfrozen for end-to-end fine-tuning at a low 1e-4 rate - cutting training time and avoiding catastrophic forgetting.
Custom CNNs, with a ResNet18 baseline
The towers are custom 1-D and 2-D CNNs. A pretrained ResNet18 (97.43%) was benchmarked for logos, but the fusion model keeps the custom towers so both can be fine-tuned jointly.
A 20-feature metadata channel
URL counts, domain-mismatch flags, urgency keywords and formatting stats give the classifier evidence that pure text or image analysis overlooks.
Batch-norm + dropout 0.5
Heavy regularization across the 832→512→256→128→2 head keeps a high-capacity model from overfitting - the text tower holds 98.96% validation against 99.93% train.