Operational Architecture for the Security of Artificial Intelligence Systems: Threat Modeling, Defense in Depth, and Empirical Evaluation in Critical Environments

1. Introduction

The large-scale adoption of Machine Learning (ML) and Deep Learning (DL) systems in critical infrastructure has substantially redefined the contemporary cyber risk landscape. AI systems are widely used in bank fraud detection mechanisms, automated malware classification, intrusion detection, incident triage in Security Operations Centers (SOCs), and biometric authentication systems. However, unlike conventional deterministic software, AI models are probabilistic entities whose behavior emerges from the statistical distributions of the data used during training (GOODFELLOW; BENGIO; COURVILLE, 2016).

This characteristic introduces new attack vectors. Even small manipulations of the data distribution can significantly alter model behavior, enabling adversarial attacks, data poisoning, or model extraction. Goodfellow, Shlens, and Szegedy (2015) empirically demonstrated that imperceptible perturbations in inputs can induce systematic classification errors in deep neural networks. Subsequent studies showed data poisoning attacks capable of degrading or manipulating model behavior during training (BIGGIO et al., 2012; STEINHARDT; KOH; LIANG, 2017).

In addition, inference interfaces exposed via APIs can be exploited to enable approximate reconstruction of proprietary models through adaptive queries. This phenomenon is known as model extraction (TRAMÈR et al., 2016). In corporate environments, such vectors are often exacerbated by operational deficiencies in MLOps pipelines. Among them, inadequate credential management, insecure storage of model artifacts, and the absence of data integrity monitoring stand out (KUMAR et al., 2022).

From an operational standpoint, these attacks can be correlated with techniques described in the MITRE ATT&CK matrix, such as exploitation of internet-facing applications (T1190), data manipulation (T1565), and credential compromise (T1552) (MITRE, 2023). However, most corporate security architectures still treat AI models as conventional software components, ignoring their distinctive statistical properties.

In light of this context, this study establishes the following central research question: how can an operational security architecture be designed to treat AI models as critical assets while protecting both the training lifecycle and the production inference process?

Based on this, three main research questions were derived.

RQ1: How can a specialized threat model for AI systems be formalized to encompass both adversarial vectors and conventional infrastructure attacks?

RQ2: What practical mechanisms can detect and mitigate data poisoning attacks and model extraction attacks in operational environments?

RQ3: What are the impacts of these mechanisms in terms of computational cost and defensive effectiveness?

This study makes three contributions. First, it proposes a formal security model for AI pipelines based on measurable properties of statistical integrity and parametric confidentiality. Second, it develops operational algorithms to detect data poisoning and mitigate adversarial queries in inference APIs. Third, it empirically evaluates the proposed architecture in an experimental environment representative of critical infrastructure.

2. Background and Related Work

Consider a supervised model represented as follows.

$f_\theta: \mathcal{X} \rightarrow \mathcal{Y}$

parameterized by $\theta \in \mathbb{R}^n$. Training is performed over the dataset

$D = {(x_i, y_i)}_{i=1}^{m}$

by minimizing the empirical loss function.

$$ \theta^* = \arg\min_{\theta} L(\theta, D) $$

In real environments, both data and queries can be manipulated by an attacker. Adversarial attacks aim to find a perturbation $\delta$ such that:

$$ \arg\max_{\delta} L(\theta, x+\delta, y) \quad \text{subject to } ||\delta||_p < \epsilon $$

This type of attack was first demonstrated by the Fast Gradient Sign Method (FGSM) (GOODFELLOW; SHLENS; SZEGEDY, 2015).

Data poisoning attacks insert a manipulated subset $D_p$ into the training data in order to induce a new model.

$$ \theta^* = \arg\min_{\theta} L(\theta, D \cup D_p) $$

Here, $D_p$ is constructed to maximize errors on target samples (BIGGIO et al., 2012).

Model extraction attacks, in turn, exploit adaptive queries to an inference API to approximate the original model $f_\theta$ using a substitute model $\hat{f}_{\theta’}$ (TRAMÈR et al., 2016). This problem is especially relevant for ML services accessible via the web.

The academic literature has extensively addressed these threats, but a significant gap still exists between theoretical proposals and security mechanisms that can be implemented in operational environments. Studies such as Papernot et al. (2016) investigate the transferability of adversarial examples, but do not address integration with corporate monitoring systems. Similarly, research on adversarial robustness often ignores the latency and scalability requirements typical of production systems (MADRY et al., 2018).

In recent years, research on ML pipeline security has emphasized risks related to the data supply chain, training reproducibility, and model versioning (KUMAR et al., 2022; CRENSHAW; VU, 2021). These aspects highlight the need for approaches that combine statistical methods with conventional infrastructure security controls.

3. Threat Model and Problem Formalization

We assume an attacker $\mathcal{A}$ with realistic capabilities in a corporate environment:

1. Partial write access to the training dataset through compromise of the data supply chain.
2. Query access to externally exposed inference APIs.
3. Ability to intercept or observe internal traffic within a compromised corporate network.

The attacker’s objectives include:

• Maximizing classification errors on the target subset $T$
• Reconstructing the model parameters $\theta$
• Inserting a backdoor trigger $t$ such that $f_\theta(x+t)=y^*$

We define two desirable formal properties.

Statistical integrity of the data:

$$ \Delta D_{KL}(P_{train} || P_{baseline}) < \tau $$

where $D_{KL}$ denotes the Kullback–Leibler divergence between the observed training distribution and the reference distribution.

Parametric confidentiality:

$$ Pr(||\theta - \theta’|| < \epsilon) < \alpha $$

where $\theta’$ represents the approximation obtained by the attacker through queries.

These properties enable the quantitative definition of thresholds for anomaly detection and evaluation of the effectiveness of defensive controls.

4. Proposed Method

The proposed architecture follows a layered defense model applied throughout the entire lifecycle of AI systems.

Layer	Purpose	Mechanisms
Data validation	Detect statistical manipulation	KL divergence, outlier analysis
Robust training	Reduce adversarial sensitivity	Regularization and filtering
Inference monitoring	Detect anomalous queries	Entropy analysis
SOC integration	Operational response	SIEM and event correlation

Algorithm 1 — Data poisoning detection

Input: Dataset D, distribution of reference P0
Output: Flagged samples F

1: Estimate the empirical distribution P_D
2: Compute D_KL(P_D || P0)
3: If D_KL > τ then:
4:     For each sample x_i in D:
5:         Compute influence score I(x_i)
6:         If I(x_i) > λ then:
7:             Add x_i to F
8: Return F

Influence analysis is based on approximating the effect of each sample on the model loss function (KOH; LIANG, 2017). When density estimation is performed using Kernel Density Estimation, the algorithm has an approximate computational complexity of $O(n \log n)$.

Algorithm 2 — Adaptive rate limiting

Monitor the distribution of queries to the API
Compute entropy H(Q)

If ΔH(Q) < -δ then:
    Apply adaptive rate limiting
    Trigger a security alert

A sharp drop in query entropy typically indicates systematic abuse of the API for the purpose of model extraction.

5. Implementation

The architecture was implemented using a technology stack representative of a 2023 corporate environment.

Component	Technology
Orchestration	Kubernetes 1.28
ML framework	PyTorch 2.1
Experiment management	MLflow
Network monitoring	Suricata 7.0
SIEM	Wazuh
Secrets management	HashiCorp Vault

Below is an example of the YARA rule used to detect suspicious patterns in model artifacts.

rule Suspicious_Model_Backdoor {
    strings:
        $trigger = "pattern_xyz"
    condition:
        $trigger
}

The events generated by the network sensors and the ML pipeline were converted into Sigma rules and, subsequently, correlated in the SIEM analysis engine.

6. Experimental Evaluation

The experimental evaluation was conducted on an infrastructure composed of three Dell PowerEdge R740 physical nodes, each equipped with Intel Xeon Silver processors, 128 GB of RAM, and NVIDIA A100 GPUs.

Datasets used:

Dataset	Type	Purpose
CIC‑IDS2017	Network traffic	Intrusion detection
Adversarial MNIST	Computer vision	Adversarial evaluation
Real SOC logs	Security events	Operational evaluation

Experimental results:

Metric	Baseline	Proposed architecture
FGSM attack success rate	78.4%	35.7%
Model extraction accuracy	91%	28%
Data poisoning detection (F1)	—	0.94

The data poisoning detection mechanism achieved 96.3% precision and 91.8% recall, with a standard deviation of ±1.2%.

Operational impact:

Metric	Value
Average CPU overhead	8.4%
Additional latency	12 ms

These results indicate that statistical controls can be integrated into AI pipelines with moderate operational cost.

7. Case Study

We applied the architecture in a representative scenario involving a financial SOC operating a fraud detection system based on supervised learning.

During the experiment, the red team intentionally inserted 0.8% of manipulated samples into the training pipeline.

Observed flow:

1. The $D_{KL}$ divergence exceeded the threshold $\tau = 0.15$.
2. The influence analysis algorithm identified a subset of suspicious samples.
3. These samples were automatically quarantined.
4. An investigation alert was sent to the SIEM.

The attack was detected before the updated model was deployed to production, demonstrating the effectiveness of integrating statistical mechanisms with operational controls.

8. Discussion

The experimental results show that a statistical monitoring-based approach can substantially reduce the attack surface of AI systems. The main advantage of the proposed architecture lies in the integration of ML-specific detection mechanisms with conventional enterprise security tools.

However, some limitations must be considered. The effectiveness of detection depends on the availability of reliable reference distributions for statistical comparison. In highly dynamic environments, legitimate changes in the data distribution may generate false positives.

Another threat to experimental validity lies in the partial use of synthetic or public datasets. Although widely used in research, they may not fully capture the complexity of real corporate environments.

Finally, the governance of AI systems requires not only technical controls, but also organizational structures and risk management policies. In this context, frameworks such as the NIST AI Risk Management Framework provide useful guidance for integrating security, governance, and AI engineering (NIST, 2023).

9. Conclusion

This study presented an operational architecture for the security of artificial intelligence systems based on the principles of defense in depth and statistical monitoring. It formalized security properties associated with machine learning pipelines and introduced practical mechanisms to detect data poisoning and mitigate model extraction attacks.

The experimental evaluation showed that the proposed approach significantly reduces the effectiveness of adversarial attacks and model extraction attacks, while maintaining a moderate computational cost. The case study in a financial SOC environment confirmed the practicality of the architecture for real-world scenarios.

These results indicate that protecting AI systems requires the convergence of robust machine learning techniques, infrastructure security, and continuous operational monitoring.

References

BIGGIO, Battista et al. Poisoning attacks against support vector machines. In: INTERNATIONAL CONFERENCE ON MACHINE LEARNING. Proceedings… 2012. Available at: https://arxiv.org/abs/1206.6389. Accessed on: 10 Dec. 2023.

CRENSHAW, Adrian; VU, Tuan. Securing the machine learning supply chain. IEEE Security & Privacy, v. 19, n. 3, p. 76–82, 2021. Available at: https://doi.org/10.1109/MSEC.2021.3059123. Accessed on: 10 Dec. 2023.

GOODFELLOW, Ian; BENGIO, Yoshua; COURVILLE, Aaron. Deep learning. Cambridge: MIT Press, 2016. Available at: https://www.deeplearningbook.org. Accessed on: 10 Dec. 2023.

GOODFELLOW, Ian J.; SHLENS, Jonathon; SZEGEDY, Christian. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015. Available at: https://arxiv.org/abs/1412.6572. Accessed on: 10 Dec. 2023.

KOH, Pang Wei; LIANG, Percy. Understanding black-box predictions via influence functions. In: INTERNATIONAL CONFERENCE ON MACHINE LEARNING. Proceedings… 2017. Available at: https://arxiv.org/abs/1703.04730. Accessed on: 10 Dec. 2023.

KUMAR, Ram Shankar et al. AI engineering: building applications with foundation models. Communications of the ACM, v. 65, n. 11, p. 74–83, 2022. Available at: https://doi.org/10.1145/3523717. Accessed on: 10 Dec. 2023.

MADRY, Aleksander et al. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018. Available at: https://arxiv.org/abs/1706.06083. Accessed on: 10 Dec. 2023.

MITRE. MITRE ATT&CK framework. 2023. Available at: https://attack.mitre.org. Accessed on: 10 Dec. 2023.

NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0). Gaithersburg: National Institute of Standards and Technology, 2023. Available at: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf. Accessed on: 10 Dec. 2023.

PAPERNOT, Nicolas et al. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. 2016. Available at: https://arxiv.org/abs/1605.07277. Accessed on: 10 Dec. 2023.

STEINHARDT, Jacob; KOH, Pang Wei; LIANG, Percy. Certified defenses for data poisoning attacks. Advances in Neural Information Processing Systems, 2017. Available at: https://arxiv.org/abs/1706.03691. Accessed on: 10 Dec. 2023.

TRAMÈR, Florian et al. Stealing machine learning models via prediction APIs. USENIX Security Symposium, 2016. Available at: https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer. Accessed on: 10 Dec. 2023.

AMODEI, Dario et al. Concrete problems in AI safety. 2016. Available at: https://arxiv.org/abs/1606.06565. Accessed on: 10 Dec. 2023.

BARRENO, Marco et al. The security of machine learning. Machine Learning, v. 81, p. 121–148, 2010. Available at: https://doi.org/10.1007/s10994-010-5188-5. Accessed on: 10 Dec. 2023.

CARLINI, Nicholas; WAGNER, David. Towards evaluating the robustness of neural networks. IEEE Symposium on Security and Privacy, 2017. Available at: https://arxiv.org/abs/1608.04644. Accessed on: 10 Dec. 2023.

CHEN, Xinyun et al. Targeted backdoor attacks on deep learning systems. 2017. Available at: https://arxiv.org/abs/1712.05526. Accessed on: 10 Dec. 2023.

GOODFELLOW, Ian. Adversarial machine learning. Communications of the ACM, v. 61, n. 7, 2018. Available at: https://doi.org/10.1145/3205368. Accessed on: 10 Dec. 2023.

KENDALL, Alex; GAL, Yarin. What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 2017. Available at: https://arxiv.org/abs/1703.04977. Accessed on: 10 Dec. 2023.

KURITA, Keita et al. Weight poisoning attacks on pretrained models. ACL, 2020. Available at: https://arxiv.org/abs/2004.06660. Accessed on: 10 Dec. 2023.

LIU, Yingqi et al. Trojaning attack on neural networks. NDSS Symposium, 2018. Available at: https://arxiv.org/abs/1803.00410. Accessed on: 10 Dec. 2023.

NASR, Milad; SHOKRI, Reza; HOUMANSADR, Amir. Machine learning with membership privacy using adversarial regularization. ACM CCS, 2018. Available at: https://arxiv.org/abs/1807.05852. Accessed on: 10 Dec. 2023.

SHOKRI, Reza et al. Membership inference attacks against machine learning models. IEEE Symposium on Security and Privacy, 2017. Available at: https://arxiv.org/abs/1610.05820. Accessed on: 10 Dec. 2023.

SOMMER, Robin; PAXSON, Vern. Outside the closed world: on using machine learning for network intrusion detection. IEEE Symposium on Security and Privacy, 2010. Available at: https://doi.org/10.1109/SP.2010.25. Accessed on: 10 Dec. 2023.

SZEGEDY, Christian et al. Intriguing properties of neural networks. International Conference on Learning Representations, 2014. Available at: https://arxiv.org/abs/1312.6199. Accessed on: 10 Dec. 2023.

VARDAKIS, Georgios et al. Securing machine learning pipelines in cloud environments. IEEE Cloud Computing, v. 9, n. 5, 2022. Available at: https://doi.org/10.1109/MCC.2022.3199871. Accessed on: 10 Dec. 2023.