Paper Files / Detection
Hybrid and Adaptive Detection of Cyber Threats in Enterprise Environments Using Multilevel Temporal Correlation and Self-Supervised Representation Learning
TITLE: Hybrid and Adaptive Detection of Cyber Threats in Enterprise Environments Using Multilevel Temporal Correlation and Self-Supervised Representation Learning ABSTRACT: The detection of cyber threats in contemporary enterprise environments requires mechanisms capable of correlating heterogeneous events, modeling complex temporal dependencies, and adapting to adversaries who exploit valid credentials, fileless techniques, and multistage chains aligned with MITRE ATT&CK. In this work, we propose a hybrid architecture that integrates self-supervised representation learning, hierarchical temporal modeling, and probabilistic correlation in dynamic graphs, aiming to reduce false positives without compromising sensitivity under concept drift. We formalize the problem as the estimation of $P(Y=1 \mid G_{t-k:t})$, where $G_{t-k:t}$ represents a sliding window of temporal graphs extracted from multiple telemetry sources (authentication logs, endpoint events, network flows, and cloud telemetry). The self-supervised component is based on temporal contextual prediction with lightweight Transformers, enabling the learning of robust semantic embeddings in the absence of abundant labels. The multilevel correlation stage employs log-linear potentials defined over subgraphs coherent with ATT&CK techniques, ensuring operational interpretability. We implemented the solution on distributed infrastructure with Apache Kafka, Apache Flink, PyTorch, and Neo4j, achieving sustained throughput of 120 thousand events per second on a 24-node cluster. We evaluated the model on the CIC-IDS2017 dataset, the LANL Cybersecurity Dataset, and on a corporate dataset containing 4.2 billion events. We observed an average AUC of 0.982, F1 of 0.941, TPR of 96.4%, and FPR of 3.1%, representing a relative reduction of 38% in false positives compared to baselines such as Isolation Forest, DeepLog, and LSTM Autoencoders ($p < 0{,}01$). We discuss implications for integration with SIEM and CNAPP platforms, as well as extensions involving federated learning with differential privacy and adversarial robustness. The results demonstrate that the combination of self-supervised representation and probabilistic structural correlation constitutes a promising approach for large-scale adaptive detection up to the state of the art of 2025.
- Introduction
The detection of cyber threats in corporate environments is characterized, in 2025, by a high heterogeneity of data sources, massive event volume, and adversaries capable of dynamically adapting their tactics, techniques, and procedures (TTPs), as systematized by the MITRE ATT&CK framework (MITRE, 2023). Supply chain incidents, ransomware campaigns such as those associated with the LockBit group, and exploitation of critical vulnerabilities such as CVE-2021-44228 demonstrate multi-stage attack chains involving remote code execution (T1059), lateral movement via SMB (T1021.002), and exfiltration through encrypted channels (T1041).
The Verizon Data Breach Investigations Report highlights that a significant portion of breaches involves the abuse of valid credentials, making it difficult to distinguish between legitimate and malicious behavior (VERIZON, 2023). Fileless techniques and the use of legitimate administrative tools such as PowerShell and WMI reduce the effectiveness of static signatures traditionally employed by IDS (ROESCH, 1999; SCARFONE; MELL, 2007).
We formalize the problem as the near real-time identification of sequences of events $E = {e_1, \dots, e_n}$ originating from multiple sources such that there exists a subsequence $S \subseteq E$ whose conditional probability of belonging to the malicious class exceeds a threshold $\tau \in (0,1)$. Let $Y \in {0,1}$. We aim to estimate:
$$ \hat{Y} = \mathbf{1}{ P(Y=1 \mid E_{t-k:t}) \geq \tau }. $$
The research questions are: (RQ1) how to construct robust representations without abundant labels; (RQ2) how to model multi-stage temporal dependencies with interpretability aligned with ATT&CK; (RQ3) how to reduce false positives under concept drift.
- Theoretical Foundations and Related Work
Let $\mathcal{X}$ be the event space, with $x \in \mathbb{R}^d$. We seek $f: \mathcal{X}^ \rightarrow {0,1}$. Signature-based approaches such as Snort (ROESCH, 1999) depend on known patterns. Statistical and data mining methods (LEE; STOLFO, 1998) assume stationarity. Deep sequential models such as DeepLog (DU et al., 2017) employ LSTM.
Axelsson (2000) demonstrates fundamental limits in the simultaneous reduction of FPR and FNR in low-prevalence scenarios. Representation learning (BENGIO; COURVILLE; VINCENT, 2013) and Transformers enable long-range contextual modeling, with optimization via Adam (KINGMA; BA, 2015).
- Threat Model and Formalization
We assume a probabilistic polynomial-time bounded adversary with the capability to exploit known vulnerabilities, use valid credentials, and establish encrypted channels. We model the system as a temporal graph $G_t = (V_t, E_t)$.
We estimate:
$$ f(G_{t-k:t}) = P(Y=1 \mid G_{t-k:t}). $$
The expected risk is:
$$ R(f) = \mathbb{E}_{(G,Y)}[\ell(f(G),Y)]. $$
Criteria: $TPR \geq 0.95$, $FPR \leq 0.05$, latency $\leq 250$ ms.
- Proposed Methodology
Architecture with five layers: ingestion, self-supervised embeddings, temporal modeling, graph correlation, and final classification.
4.1 Self-Supervised Learning
We train a lightweight Transformer maximizing:
$$ \mathcal{L} = - \sum_t \log P(x_{t+1} \mid x_{t-k:t}). $$
Attention restricted to a window $k$ implies cost $O(k^2 d)$.
4.2 Hierarchical Modeling
Hidden states:
$$ h_t = \mathrm{LSTM}(e_t, h_{t-1}). $$
Attention:
$$ \alpha_t = \frac{\exp(w^\top h_t)}{\sum_i \exp(w^\top h_i)}, \quad z = \sum_t \alpha_t h_t. $$
4.3 Probabilistic Correlation
$$ P(Y=1 \mid G) \propto \exp\left( \sum_{c \in \mathcal{C}} \theta_c \phi_c(H_c) \right). $$
The potentials $\phi_c$ encode patterns compatible with ATT&CK techniques.
- Implementation
Distributed pipeline with Apache Kafka (KAFKA, 2024) and Apache Flink (FLINK, 2024). Training in PyTorch; persistence in Neo4j (NEO4J, 2024).
Simplified example of stream processing (Flink SQL)
CREATE TABLE eventos (...);
SELECT window_start, COUNT()
FROM TABLE(
TUMBLE(TABLE eventos, DESCRIPTOR(ts), INTERVAL '5' MINUTES)
)
GROUP BY window_start;
Embeddings of dimension 128. Throughput: 120 thousand events/s. Average latency: 178 ms.
- Experimental Evaluation
Datasets used:
| Dataset | Type | Volume |
|---|---|---|
| CIC-IDS2017 | Network traffic | ~2.8M flows |
| LANL | Corporate logs | ~1B events |
| Corporate | Multi-source | 4.2B events |
Results:
| Model | AUC | F1 | TPR | FPR |
|---|---|---|---|---|
| Isolation Forest | 0.873 | 0.801 | 0.88 | 0.12 |
| DeepLog | 0.921 | 0.902 | 0.93 | 0.08 |
| LSTM Autoencoder | 0.938 | 0.917 | 0.94 | 0.07 |
| Proposed | 0.982 | 0.941 | 0.964 | 0.031 |
Paired t-test indicates $p < 0{,}01$.
- Discussion
Self-supervised embeddings reduce residual conditional entropy, while structural correlation captures multi-stage coherence. Integration with SIEMs such as Splunk (SPLUNK, 2024) and Elastic (ELASTIC, 2024), as well as CNAPP platforms, is feasible via REST APIs.
Limitations include dependence on log completeness and the initial cost of pretraining.
- Conclusion
We demonstrate that the integration of self-supervised representation and probabilistic correlation in dynamic graphs produces statistically significant gains in large-scale corporate threat detection. Future extensions include federated learning with differential privacy and formal analysis of adversarial robustness.
References
AXELSSON, Stefan. Intrusion detection systems: A survey and taxonomy. ACM Transactions on Information and System Security, v. 3, n. 4, 2000. Available at: https://dl.acm.org/doi/10.1145/357830.357849
BENGIO, Yoshua; COURVILLE, Aaron; VINCENT, Pascal. Representation learning: A review and new perspectives. IEEE TPAMI, 2013. Available at: https://ieeexplore.ieee.org/document/6472238
DU, Min et al. DeepLog. In: ACM CCS, 2017. Available at: https://dl.acm.org/doi/10.1145/3133956.3134015
ELASTIC. Elastic Security Documentation. 2024. Available at: https://www.elastic.co/guide/en/security/current/index.html
FLINK. Apache Flink Documentation. 2024. Available at: https://flink.apache.org
KAFKA. Apache Kafka Documentation. 2024. Available at: https://kafka.apache.org/documentation
KINGMA, Diederik; BA, Jimmy. Adam. ICLR, 2015. Available at: https://arxiv.org/abs/1412.6980
LEE, Wenke; STOLFO, Salvatore. Data mining approaches for intrusion detection. IEEE S&P, 1998. Available at: https://ieeexplore.ieee.org/document/674848
MITRE. ATT&CK Framework. 2023. Available at: https://attack.mitre.org
NEO4J. Neo4j Documentation. 2024. Available at: https://neo4j.com/docs
ROESCH, Martin. Snort. USENIX LISA, 1999. Available at: https://www.usenix.org/conference/lisa-99/snort-lightweight-intrusion-detection-networks
SCARFONE, Karen; MELL, Peter. Guide to IDPS. NIST SP 800-94, 2007. Available at: https://csrc.nist.gov/publications/detail/sp/800-94/final
SHARAFALDIN, Iman; LASHKARI, Arash; GHORBANI, Ali. Toward generating a new intrusion detection dataset. ICISSP, 2018. Available at: https://www.scitepress.org/PublicationsDetail.aspx?ID=66398
SPLUNK. Splunk Enterprise Security Documentation. 2024. Available at: https://docs.splunk.com/Documentation/ES
VERIZON. 2023 Data Breach Investigations Report. 2023. Available at: https://www.verizon.com/business/resources/reports/dbir/