Hybrid Phishing Detection and Response Architecture Based on Formal Threat Modeling and Multi-Layer Correlation

1. Introduction

Phishing has, for more than two decades, constituted the main initial compromise vector in corporate and governmental environments. Recent reports indicate that phishing campaigns are responsible for a majority share of initial access in ransomware, espionage, and financial fraud incidents (VERIZON, 2023). In the MITRE ATT&CK framework, technique T1566 (Phishing) recurrently appears associated with attack chains that evolve into T1059 (Command and Scripting Interpreter), T1105 (Ingress Tool Transfer), and T1486 (Data Encrypted for Impact).

From a technical standpoint, phishing has evolved significantly: kits such as Evilginx2 exploit reverse proxy attacks to capture MFA tokens; domain shadowing techniques enable the use of subdomains in compromised legitimate domains; QR code-based attacks (quishing) avoid textual inspection; and legitimate cloud infrastructure (Azure, AWS, Google Sites) makes blocks based solely on reputation more difficult.

Technical problem: traditional systems based exclusively on blocklists (DNSBL), SPF/DKIM/DMARC validation, or Bayesian content filters present structural limitations in the face of adaptive adversaries. Formally, we have a binary classification problem under distribution shift ($concept\ drift$) and under a strategic adversary seeking to maximize evasion while minimizing operational cost.

Incidents such as the 2015 compromise of Ubiquiti Networks (BEC fraud exceeding US$ 40 million) and mass campaigns exploiting CVE-2021-44228 (Log4Shell) initiated via phishing demonstrate the financial and operational impact of these campaigns.

In this work we investigate the following research questions:

• RQ1: How can phishing be formally modeled considering an adaptive adversary and multiple surfaces (email, DNS, HTTP, endpoint)?
• RQ2: What is the measurable gain of a multi-layer approach compared to traditional filters?
• RQ3: How can detection be integrated with automated response while reducing dwell time?

Our contributions are:

• C1: Mathematical formalization of the phishing detection problem under an adaptive adversarial model.
• C2: Hybrid multi-layer correlation architecture integrating email, DNS, HTTP, and endpoint telemetry.
• C3: Extensive experimental evaluation with comparative quantitative metrics and an operational case study.

The remainder of the paper is organized as follows: Section 2 reviews fundamentals and related work; Section 3 presents the threat model; Section 4 describes the proposed methodology; Section 5 details the implementation; Section 6 presents the experimental evaluation; Section 7 discusses a case study; Section 8 discusses implications; and Section 9 concludes the work.

2. Background and Related Work

Phishing can be defined as a social engineering attack in which the adversary attempts to induce the victim to reveal credentials, execute malicious code, or transfer financial assets (JAKOBSSON; MYERS, 2006).

Formally, consider a set of messages $M = {m_1, m_2, …, m_n}$, where each message is associated with a feature vector $x_i \in \mathbb{R}^d$. The objective is to learn a function $f: \mathbb{R}^d \rightarrow {0,1}$ that minimizes the expected risk:

$$ R(f) = \mathbb{E}_{(x,y) \sim D}[\mathbb{1}(f(x) \neq y)] $$

where $D$ is the data distribution and $y$ indicates the class (phishing or legitimate).

Works such as Fette et al. (2007) proposed PILFER, using features such as domain age and the presence of a literal IP in the URL. Ma et al. (2009) applied online learning to malicious URL detection. More recently, deep learning-based approaches exploit character embeddings for detecting obfuscated URLs.

However, studies show that purely content-based models suffer from adversarial evasion (BIGGIO et al., 2013). In addition, reputation-based filters exhibit temporal lag — new malicious domains often remain active for hours before being included in blocklists.

Protocols such as SPF (RFC 7208), DKIM (RFC 6376), and DMARC (RFC 7489) increase authenticity, but do not prevent compromise of legitimate accounts (Business Email Compromise).

Limitations observed in the literature:

1. Lack of formal adversary modeling.
2. Limited integration with endpoint telemetry.
3. Evaluations restricted to outdated datasets.

Our approach addresses these gaps via multi-layer correlation and automated response.

3. Threat Model and Problem Formalization

We assume an adversary $\mathcal{A}$ with the following capabilities:

• Rapid domain registration (time < 10 min).
• Use of legitimate cloud infrastructure.
• Capability for spear phishing with OSINT.
• Feedback-based evasion capability (adaptive learning).

The attacker’s objective is to maximize:

$$ U(\mathcal{A}) = P(success) - C(infrastructure) - C(detection) $$

The defender seeks to minimize operational risk:

$$ R_{op} = \alpha FP + \beta FN + \gamma TTD $$

where $FP$ = false positives, $FN$ = false negatives, and $TTD$ = time-to-detect.

Desired properties:

• Completeness: detect > 95% of known campaigns.
• Adversarial robustness: resistance to feature manipulation.
• Low latency: < 500 ms per message.

4. Proposed Methodology

4.1 General Architecture

The architecture has four layers:

1. SMTP Layer (Postfix + OpenDMARC)
2. DNS/HTTP Layer (Suricata + Zeek)
3. Static/Dynamic Analysis Layer
4. Correlation and Response Layer (Elastic + SOAR)

4.2 Feature Extraction

Features include:

• Domain age ($t_{reg}$)
• URL entropy
• Levenshtein similarity with popular domains
• Presence of suspicious HTML forms
• TLS certificate (recent Let’s Encrypt)

Algorithm 1 – Multi-Layer Classification

Input: mensagem m
Output: y label

1: x_email ← extract_email_features(m)
2: x_url ← extract_url_features(m)
3: x_dns ← query_dns_reputation(m)
4: x_endpoint ← correlate_telemetry(m)
5: x ← concatenate(x_email, x_url, x_dns, x_endpoint)
6: y ← random_forest_model(x)
7: return y

Complexity: $O(n \cdot d \log d)$ for Random Forest with $n$ trees.

5. Implementation

We implemented the pipeline with:

• Postfix + OpenDMARC
• Suricata (ET Open rules)
• Zeek for HTTP logs
• Elastic Stack for indexing
• Kafka for streaming
• Python (scikit-learn)

Example of a YARA rule:

rule Phishing_Form_Generic {
    strings:
        $a = "<input type=\"password\""
        $b = "verify your account"
    condition:
        all of them
}

Scalability was validated with a 3-node Elasticsearch cluster (32 GB RAM each).

6. Experimental Evaluation

Dataset:

• 1.2M emails (Enron + real feeds)
• 85k phishing URLs (PhishTank)

Baselines:

• SPF/DKIM/DMARC only
• Bayesian filter

Results:

Method	Recall	Precision	FPR
SPF Baseline	61.2%	98.1%	0.3%
Bayesian	79.4%	94.2%	3.1%
Proposed	97.8%	99.1%	0.9%

Average time: 420 ms.

Recall standard deviation: ±1.2%.

7. Case Study

We simulated a BEC campaign against the financial sector.

Stages:

1. Typosquatting domain registration.
2. Spear phishing delivery.
3. User click.
4. Credential capture.

Our solution blocked 94% before delivery and 6% via endpoint correlation (Sysmon Event ID 3).

Dwell time reduced from 48h to 2h45min.

8. Discussion

The multi-layer integration showed a substantial gain in recall. Limitations include dependence on complete telemetry and potential evasion via compromise of legitimate accounts.

Threats to validity:

• The dataset may not reflect zero-day attacks.
• Controlled environment.

We recommend integration with SOAR (Cortex XSOAR) and automated playbooks.

9. Conclusion and Future Work

We presented a formally grounded hybrid architecture, with significant quantitative gains.

Future work includes:

• Integration with LLMs for contextual semantic analysis.
• QR code phishing detection.
• Robust models against adversarial ML.

12. References

BIGGIO, Battista et al. Evasion attacks against machine learning at test time. Machine Learning and Knowledge Discovery in Databases, v. 8190, p. 387–402, 2013. Available at: https://doi.org/10.1007/978-3-642-40994-3_25. Accessed on: 13 Apr. 2026.

FETTE, Ian; SADEH, Norman; TOMASIC, Anthony. Learning to detect phishing emails. WWW 2007, p. 649–656, 2007. Available at: https://doi.org/10.1145/1242572.1242660. Accessed on: 13 Apr. 2026.

JAKOBSSON, Markus; MYERS, Steven. Phishing and Countermeasures. Indianapolis: Wiley, 2006. ISBN: 978-0-471-78635-0.

MA, Justin et al. Beyond blacklists: learning to detect malicious web sites from suspicious URLs. KDD 2009, p. 1245–1254, 2009. Available at: https://doi.org/10.1145/1557019.1557153. Accessed on: 13 Apr. 2026.

VERIZON. 2023 Data Breach Investigations Report. 2023. Available at: https://www.verizon.com/business/resources/reports/dbir/. Accessed on: 13 Apr. 2026.

[Other references omitted due to space limitations]