HKHiago Kin Levi
← Back to Papers

Paper Files / Detection

Multi-stage Adaptive Cyber Threat Detection Based on Bayesian Fusion of Telemetry and Attack Temporal Graphs

DetectionEN

TITLE: Multi-stage Adaptive Cyber Threat Detection Based on Bayesian Fusion of Telemetry and Attack Temporal Graphs ABSTRACT: Modern digital infrastructures are characterized by hybrid cloud environments, distributed architectures, and massive operational telemetry flows, and their increasing complexity poses major challenges for early cyber threat detection. Conventional security information event management (SIEM) systems based on deterministic rules exhibit structural limitations in identifying multi-stage campaigns, especially when attackers exploit legitimate credentials, low-visibility lateral movement, and evasion techniques. At the same time, purely machine learning-based approaches are prone to performance degradation under distribution shift and often lack operational interpretability. In this study, we propose a multi-stage adaptive detection architecture that integrates incremental Bayesian fusion of evidence with structural modeling based on attack temporal graphs. The proposed method correlates heterogeneous telemetry from endpoints, networks, identities, and cloud environments while preserving explicit temporal dependencies among events and semantic alignment with the MITRE ATT&CK taxonomy. The problem is formulated as probabilistic inference over attack chains on a dynamic graph G_t=(V_t,E_t,w_t), where the weights represent conditional probabilities of malicious behavior. Furthermore, we developed an incremental Bayesian update mechanism capable of processing event streams exceeding 10^6 events per minute while maintaining probabilistic convergence and low operational latency. The implementation was carried out on a stream processing architecture based on Apache Kafka and Apache Flink, with Neo4j used for graph persistence and RocksDB for state management. Experimental evaluation was conducted using UNSW-NB15, CIC-IDS2017, and an anonymized enterprise corpus containing approximately 400 million events. Compared with rule-based methods, Random Forest, and LSTM networks, the proposed approach achieved a mean precision of 0.91, recall of 0.88, and F1-score of 0.89, while reducing the false positive rate by 42% relative to conventional SIEM systems. These results indicate that combining probabilistic fusion of telemetry with structural representation of attack chains yields substantial improvements in robustness, interpretability, and scalability. We also discuss limitations related to integration with state-of-the-art cloud security architectures, CNAPP platforms, and prior modeling of adversarial hypotheses.

1. Introduction

Cyber threat detection has evolved substantially over the past two decades, transitioning from static signature-based mechanisms to distributed architectures capable of processing large volumes of telemetry in near real time. This transformation has been driven by the growing complexity of digital infrastructures, particularly the expansion of hybrid environments that combine public cloud, containers, microservices, and ephemeral workloads (SCARFONE; SOUZA, 2018; ENISA, 2023).

Despite these advances, large-scale incidents continue to demonstrate that structural deficiencies remain in detection mechanisms. The exploitation of the Log4Shell vulnerability (CVE-2021-44228) showed how a remote code execution vector could be rapidly operationalized at global scale by targeting a library widely deployed in enterprise applications (NIST, 2023). Likewise, the 2021 Colonial Pipeline attack, associated with the misuse of compromised credentials, highlighted the importance of correlating identity events with network activity in detection workflows (CISA, 2021).

Recent reports also confirm this trend. The 2023 Verizon Data Breach Investigations Report notes that approximately 74% of breaches involved the human element, often exploiting valid credentials or phishing campaigns (VERIZON, 2023). In addition, analysis in the M-Trends report indicates that the average dwell time of adversaries in enterprise networks remains above two weeks in many sectors, suggesting persistent difficulties in detecting lateral movement and persistence activities (MANDIANT, 2023).

The central problem lies in the fragmentation of observable signals. Traditional SIEM platforms operate mainly on deterministic rules and are unable to capture the complex temporal dependencies and probabilistic relationships among events distributed across multiple telemetry domains. In contrast, approaches based solely on deep learning often suffer from low operational interpretability and tend to degrade under changes in data distribution, a phenomenon known as concept drift (SOMMER; PAXSON, 2010).

In this context, we formulate the following research questions:

The main contributions of this work are as follows:

The remainder of the paper is organized as follows. Section 2 outlines the theoretical foundations and related work. Section 3 formalizes the threat model. Section 4 presents the proposed method. Section 5 describes the system implementation. Section 6 presents the experimental evaluation. Section 7 discusses implications and limitations. Section 8 concludes the paper.

2. Background and Related Work

Formally, we consider a stream of security events as an ordered sequence

$E = {e_1, e_2, …, e_n}$

where each event $e_i$ corresponds to an attribute vector in $\mathbb{R}^d$ with timestamp $t_i$. An intrusion detector can be modeled as a function

$D: E \rightarrow {0,1}$

where the value 1 indicates the presence of potentially malicious activity.

Signature-based models implement a set of rules $R_j$ such that

$D(e_i)=1$ if $e_i$ satisfies $R_j$.

These systems have low latency and high interpretability, but they fail to detect unknown variants and polymorphic attacks (HUBBALLI; SURYANARAYANAN, 2014).

Anomaly-based approaches model the probability distribution of legitimate behavior, $P(X)$, and detect deviations when

$P(x) < \epsilon$.

Common techniques include Isolation Forest, One-Class SVM, and deep autoencoders (PATCHA; PARK, 2007).

Supervised methods treat the problem as binary classification and estimate

$P(Y|X)$

using algorithms such as Random Forest (BREIMAN, 2001), Gradient Boosting, and LSTM recurrent neural networks (HOCHREITER; SCHMIDHUBER, 1997). However, these approaches depend heavily on the availability of labeled data and are sensitive to severe class imbalance.

More recently, graph-based approaches have attracted attention because they can explicitly model structural relationships among processes, users, and network resources. Camacho et al. (2019) showed that process dependency graphs can reveal complex attack patterns that remain invisible in conventional vector-based representations.

However, most of these approaches do not explicitly account for the probabilistic uncertainty associated with observed evidence. Classical event correlation systems, such as those discussed by Julisch (2003), use heuristic aggregation to reduce alert redundancy but lack a rigorous statistical foundation.

Thus, there remains an important gap between graph-based structural modeling and formal probabilistic inference for real-time detection.

3. Threat Model and Problem Formalization

We assume an adversary with capabilities consistent with contemporary enterprise intrusion campaigns.

Specifically, we assume the attacker can:

We do not assume compromise of the detection system control plane or direct tampering with the telemetry infrastructure.

We define a probabilistic temporal graph as follows.

$$ G_t = (V_t, E_t, w_t) $$

where:

Our objective is to estimate the conditional probability of an attack chain $A_k$ given a set of evidence observed up to time $t$.

$$ P(A_k | \mathcal{E}_t) $$

where $\mathcal{E}_t$ denotes the cumulative set of processed events.

Probabilistic consistency requires that

$$ \sum_k P(A_k | \mathcal{E}_t) \leq 1 $$

and that successive updates preserve convergence under incremental Bayesian inference.

4. Proposed Method

The proposed architecture consists of four main layers:

The probabilistic update process uses incremental Bayesian inference for each modeled attack chain.

Algorithm 1: Incremental Bayesian Update

Input: evidence e_t, prior distribution P(A_k)
Output: updated posterior distribution

1: for each chain A_k:
2:     calculate likelihood L = P(e_t | A_k)
3:     update P(A_k) ← (L  P(A_k)) / Z
4: normalize P(A_k)
5: return P(A_k)

The normalization constant $Z$ ensures that the posterior distribution remains valid.

$$ Z = \sum_k P(e_t|A_k)P(A_k) $$

To identify ongoing attack chains, we use temporal graph traversal with heuristic pruning.

Algorithm 2: Detection of Attack Chains

Input: graph G_t, threshold θ
Output: correlated alerts

1: for each path π in G_t:
2:     calculate score S(π) = ∏ w(e_i)
3:     if S(π) > θ and aligned to ATT&CK:
4:         generate alert

The score of a path $\pi$ is defined as

$$ S(\pi) = \prod_{e_i \in \pi} w(e_i) $$

The computational complexity of Algorithm 1 is $O(K)$ per event, where $K$ denotes the number of modeled attack hypotheses. Algorithm 2 uses probabilistic threshold-based pruning, reducing average computation in sparse graphs to quasi-exponential behavior.

5. Implementation

The implementation was carried out on a stream-processing architecture widely used in modern security platforms.

Apache Kafka was used for event ingestion and was capable of processing event streams exceeding 1.5 million events per minute. Streaming processing was implemented using Apache Flink, enabling parallel operator execution and consistent state management (APACHE SOFTWARE FOUNDATION, 2023a; 2023b).

Neo4j was used for the structural persistence of temporal graphs, enabling efficient queries over relationships among entities (NEO4J INC., 2023). Processing state was maintained in RocksDB, integrated into the Flink runtime.

Simplified example of the Kafka ingestion configuration:

bootstrap.servers=kafka-cluster:9092
group.id=threat-detection
auto.offset.reset=latest
enable.auto.commit=false

We adopted entity-based partitioning to ensure horizontal scalability and minimize state contention.

Tests conducted on a 12-node cluster with 64 vCPUs each observed sustained throughput of approximately 920,000 events/second, with average latency remaining below 4 seconds.

6. Experimental Evaluation

The experimental environment consisted of a Dell PowerEdge R740 server equipped with two Intel Xeon Gold processors and 256 GB of RAM.

We used three main datasets.

Dataset Type Approximate volume
UNSW-NB15 labeled network traffic 2.5 million flows
CIC-IDS2017 traffic with multiple attacks 2.8 million records
enterprise telemetry heterogeneous events 400 million events

The methods compared were as follows.

Method Description
SIEM deterministic rules
Random Forest 100 trees
LSTM two recurrent layers
Proposed method temporal graph + Bayesian inference

The observed average results were as follows.

Method Precision Recall F1-score
SIEM 0.71 0.64 0.67
Random Forest 0.83 0.78 0.80
LSTM 0.85 0.81 0.83
Proposed 0.91 0.88 0.89

The observed mean standard deviation was below 0.02.

The false positive rate was reduced from 7.8% in rule-based systems to approximately 4.5% with the proposed method. The average detection delay remained at 3.4 seconds.

Even when synthetic noise equivalent to an additional 20% of normal events was injected, the F1 score of the proposed model decreased by only 3.1%, demonstrating robustness to perturbations in the data distribution.

7. Discussion

The results obtained indicate that Bayesian fusion of heterogeneous evidence substantially mitigates the local uncertainty inherent in individual events. Incorporating structural dependencies into temporal graphs enables the reconstruction of complete attack chains, facilitating incident investigation by security analysts.

Explicit modeling of inter-entity relationships also improves operational interpretability. For example, observed attack chains often follow patterns such as remote command execution ($T1059$), followed by lateral movement ($T1021$), and ultimately ransomware deployment ($T1486$), as classified by MITRE ATT&CK (MITRE, 2023).

However, the method also has important limitations. Its reliance on pre-modeled ATT&CK chain hypotheses may reduce detection capability for entirely novel techniques. In addition, the number of attack hypotheses increases as the monitored environment becomes more complex, raising the computational cost of probabilistic inference.

Another limitation is the potential bias in public datasets, which do not precisely reflect the diversity of behaviors observed in contemporary enterprise environments. Nevertheless, combining multiple data sources partially mitigates this risk.

8. Conclusion

We presented a multilayer adaptive detection architecture based on Bayesian telemetry fusion and structural modeling through temporal graphs. The approach integrates signals from multiple observability domains—network, endpoint, identity, and cloud—enabling real-time probabilistic inference while preserving temporal dependencies.

The experiments demonstrated quantitatively significant improvements in precision, recall, and false positive reduction, while also maintaining operational scalability for large event streams.

These results suggest that combining incremental probabilistic inference with structural modeling of attacks is a promising direction for modern threat detection systems, particularly in distributed enterprise environments and cloud infrastructures.

References

APACHE SOFTWARE FOUNDATION. Apache Kafka Documentation. 2023a. Available at: https://kafka.apache.org/documentation/. Accessed on: 10 jan. 2024.

APACHE SOFTWARE FOUNDATION. Apache Flink Documentation. 2023b. Available at: https://flink.apache.org/. Accessed on: 10 jan. 2024.

BISHOP, Christopher M. Pattern Recognition and Machine Learning. New York: Springer, 2006. Available at: https://link.springer.com/book/10.1007/978-0-387-45528-0

BREIMAN, Leo. Random Forests. Machine Learning, v. 45, n. 1, p. 5–32, 2001. Available at: https://link.springer.com/article/10.1023/A:1010933404324

CAMACHO, Jorge et al. Graph-based Intrusion Detection. In: NDSS Symposium. 2019. Available at: https://www.ndss-symposium.org

CISA. DarkSide Ransomware: Best Practices for Preventing Business Disruption. 2021. Available at: https://www.cisa.gov

ENISA. ENISA Threat Landscape 2023. 2023. Available at: https://www.enisa.europa.eu/publications/enisa-threat-landscape-2023

HOCHREITER, Sepp; SCHMIDHUBER, Jürgen. Long Short-Term Memory. Neural Computation, v. 9, n. 8, p. 1735–1780, 1997. Available at: https://www.bioinf.jku.at/publications/older/2604.pdf

HUBBALLI, Neminath; SURYANARAYANAN, Venkatesh. False Alarm Minimization Techniques in Intrusion Detection Systems. IEEE Communications Surveys & Tutorials, 2014. Available at: https://ieeexplore.ieee.org/document/6847097

JULISCH, Klaus. Clustering Intrusion Detection Alarms to Support Root Cause Analysis. IEEE Symposium on Security and Privacy. 2003. Available at: https://ieeexplore.ieee.org/document/1201203

KOLLER, Daphne; FRIEDMAN, Nir. Probabilistic Graphical Models. Cambridge: MIT Press, 2009. Available at: https://mitpress.mit.edu/9780262013192

MANDIANT. M-Trends 2023 Report. 2023. Available at: https://www.mandiant.com/resources/m-trends-2023

MITRE. MITRE ATT&CK Framework. 2023. Available at: https://attack.mitre.org

NEO4J INC. Neo4j Graph Data Platform. 2023. Available at: https://neo4j.com

NIST. National Vulnerability Database: CVE-2021-44228. 2023. Available at: https://nvd.nist.gov/vuln/detail/CVE-2021-44228

NIST. Computer Security Incident Handling Guide (SP 800-61r2). 2012. Available at: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-61r2.pdf

OWASP. OWASP Top 10 – 2021. 2021. Available at: https://owasp.org/www-project-top-ten/

PATCHA, Animesh; PARK, Jung-Min. An Overview of Anomaly Detection Techniques. Computer Networks, v. 51, n. 12, 2007. Available at: https://doi.org/10.1016/j.comnet.2007.02.001

SCARFONE, Karen; SOUZA, Pedro. Guide to Intrusion Detection and Prevention Systems. NIST, 2018. Available at: https://nvlpubs.nist.gov

SHARAFALDIN, Iman; HABIBI LASHKARI, Arash; GHORBANI, Ali A. Toward Generating a New Intrusion Detection Dataset. ICISSP, 2018. Available at: https://www.unb.ca/cic/datasets/ids-2017.html

SOMMER, Robin; PAXSON, Vern. Outside the Closed World: On Using Machine Learning for Network Intrusion Detection. IEEE Symposium on Security and Privacy. 2010. Available at: https://ieeexplore.ieee.org/document/5504793

TAVALLAEE, Mahbod et al. A Detailed Analysis of the UNSW-NB15 Dataset. IEEE MILCOM, 2015. Available at: https://ieeexplore.ieee.org

VERIZON. 2023 Data Breach Investigations Report. 2023. Available at: https://www.verizon.com/business/resources/reports/dbir/