Phishing Email Detection with Multimodal Deep Learning

Abstract

Phishing attacks exploit deceptive email content and fraudulent brand logos to compromise user security. This project develops a multimodal machine-learning system that detects phishing emails by jointly analyzing textual content and visual logo elements. We implement and compare six models - K-Nearest Neighbors, Logistic Regression, custom convolutional networks for text and image analysis, and a pretrained ResNet18 for logo classification - alongside a novel dual-tower fusion architecture that integrates all modalities. Using a combined dataset of 76,346 emails and the OpenLogo dataset of 72,652 brand logos across 352 classes, our models reach validation accuracies from 76.30% to 98.96%, and the dual-tower fusion model - which integrates text features (256-d), image features (512-d), and metadata (64-d) - achieves the highest accuracy at 99.45%.

Key contributions

1. 1

   Comprehensive baseline comparison

   Six models evaluated across modalities - KNN, Logistic Regression, custom text and image CNNs, and a pretrained ResNet18 - to establish honest performance baselines before fusion.

2. 2

   Multimodal fusion architecture

   A novel dual-tower network that integrates text and image features through learned representations, weighting each modality by its reliability per email.

3. 3

   Practical deployment system

   An end-to-end inference pipeline that processes raw email HTML, extracts the relevant features, and returns a real-time classification with a confidence score.

Method

The text tower is a custom 1-D CNN: an embedding layer feeds four convolutional blocks (64→128→256→512) with batch-norm, ReLU and max-pooling, projected to a 256-dimensional feature vector that captures urgency language and phishing patterns. The image tower is a VGG-style 2-D CNN over 224×224 logo crops, producing a 512-dimensional visual vector that detects brand marks and logo–domain mismatches. A small MLP projects 20 engineered metadata signals into 64 dimensions.

The fusion classifier concatenates the three vectors into an 832-dimensional representation and passes it through four fully-connected layers (832→512→256→128→2). Training is two-stage: each tower is pretrained alone, frozen while the fusion head learns on stable features, then unfrozen for end-to-end fine-tuning at a reduced 1e-4 learning rate - cutting training time and avoiding catastrophic forgetting.

Results

Deep learning substantially outperformed traditional baselines: the text CNN reached 98.96% against KNN's 81.71%, and transfer learning proved critical for vision, with ResNet18 at 97.43% versus the custom image CNN's 76.30%. Multimodal fusion delivered the strongest result - 99.45% accuracy with a ROC-AUC of 0.9998 and PR-AUC of 0.9999 - by capturing complementary signals that single-modality approaches miss. On roughly 10,000 validation emails the fusion model produced just 23 false positives and 32 false negatives.

See the full model comparison on the benchmarks page.

Data

The email corpus combines four public sources - CEAS_08, Enron, Nazario, and Nigerian Fraud - into 76,346 deduplicated, labelled emails (≈48% phishing, 52% legitimate). The logo corpus is the OpenLogo dataset: 72,652 images across 352 brand classes. To handle the scarcity of real phishing HTML, the test set was augmented with generative-AI-produced phishing emails that preserve authentic structure and embedded logos.

References

1. Jay Doshi, Kunal Parmar, Raj Sanghavi, Narendra Shekokar. A comprehensive dual-layer architecture for phishing and spam email detection. Computers & Security, Vol. 133, 2023, 103378.
2. OpenLogo Dataset (2018). QMUL-OpenLogo: Open Logo Detection Challenge. qmul-openlogo.github.io
3. Simonyan, K., & Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. ICLR.
4. Zhang, Y., & Wallace, B. (2015). A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification. arXiv:1510.03820.
5. N. A. Alam. Phishing email dataset. Kaggle.