Operational Architecture for Securing Artificial Intelligence Systems: Threat Modeling, Defense in Depth, and Empirical Evaluation in Critical Environments

1. Introduction

The massive adoption of Machine Learning (ML) and Deep Learning (DL) systems in critical infrastructures has fundamentally changed the contemporary cyber risk landscape. Artificial Intelligence (AI) systems are widely used in banking fraud prevention mechanisms, automated malware classification, intrusion detection, event triage in Security Operations Centers (SOC), and biometric authentication systems. However, unlike traditional deterministic software, AI models are probabilistic entities whose behavior emerges from the statistical distribution of the data used during training (GOODFELLOW; BENGIO; COURVILLE, 2016).

This characteristic introduces new attack vectors. Small manipulations in the data distribution may significantly alter model behavior, enabling adversarial attacks, data poisoning, or model extraction. Goodfellow, Shlens, and Szegedy (2015) empirically demonstrated that imperceptible input perturbations can cause systematic classification errors in deep neural networks. Subsequently, studies demonstrated poisoning attacks capable of degrading or manipulating model behavior during training (BIGGIO et al., 2012; STEINHARDT; KOH; LIANG, 2017).

In addition, inference interfaces exposed via APIs can be exploited to approximate the reconstruction of proprietary models through adaptive queries, a phenomenon known as model extraction (TRAMÈR et al., 2016). In enterprise environments, these vectors often combine with operational failures in MLOps chains, including improper credential management, insecure storage of model artifacts, and the absence of data integrity monitoring (KUMAR et al., 2022).

Operationally, these attacks can be mapped to techniques documented in the MITRE ATT&CK matrix, including exploitation of public-facing applications (T1190), data manipulation (T1565), and credential compromise (T1552) (MITRE, 2023). Nevertheless, most enterprise security architectures still treat AI models as conventional software components, overlooking their unique statistical characteristics.

In light of this scenario, we formulated the following central research question: how can an operational security architecture be designed to treat AI models as critical assets, protecting both the training lifecycle and the production inference process?

From this, we derived three main research questions:

RQ1: How can a dedicated threat model be formalized for AI systems that includes both adversarial vectors and traditional infrastructure attacks?

RQ2: What practical mechanisms enable the detection and mitigation of data poisoning and model extraction attacks in operational environments?

RQ3: What is the impact of these mechanisms in terms of computational cost and defensive effectiveness?

This work presents three contributions. First, we propose a formal security model for AI processing chains, based on measurable properties of statistical integrity and parametric confidentiality. Second, we developed operational algorithms for detecting data poisoning and mitigating adversarial queries in inference APIs. Third, we empirically evaluated the proposed architecture in a representative experimental environment of critical infrastructure.

2. Background and Related Work

We assume a supervised model represented by a function

$f_\theta: \mathcal{X} \rightarrow \mathcal{Y}$

parameterized by $\theta \in \mathbb{R}^n$. Training is performed on a dataset

$D = {(x_i, y_i)}_{i=1}^{m}$

through minimization of an empirical loss function:

$$ \theta^* = \arg\min_{\theta} L(\theta, D) $$

In real-world systems, both data and queries may be subject to manipulation by attackers. Adversarial attacks seek to find a perturbation $\delta$ such that:

$$ \arg\max_{\delta} L(\theta, x+\delta, y) \quad \text{subject to } ||\delta||_p < \epsilon $$

This type of attack was first demonstrated using the Fast Gradient Sign Method (FGSM) (GOODFELLOW; SHLENS; SZEGEDY, 2015).

Data poisoning attacks introduce manipulative subsets $D_p$ into the training set, thereby leading to a new model:

$$ \theta^* = \arg\min_{\theta} L(\theta, D \cup D_p) $$

where $D_p$ is constructed to maximize the error on target examples (BIGGIO et al., 2012).

Model extraction attacks, on the other hand, exploit adaptive queries to the inference API to approximate the original model $f_\theta$ via a substitute model $\hat{f}_{\theta’}$ (TRAMÈR et al., 2016). This problem becomes especially relevant in ML services exposed over the internet.

Although the academic literature has extensively examined these threats, there remains a significant gap between theoretical proposals and security mechanisms that can be implemented in operational environments. Works such as Papernot et al. (2016) examine the transferability of adversarial examples, but do not address integration with enterprise monitoring systems. Similarly, studies on adversarial robustness often overlook the latency and scalability requirements typical of production systems (MADRY et al., 2018).

More recently, research on the security of ML processing chains has highlighted risks associated with the data supply chain, training reproducibility, and model version control (KUMAR et al., 2022; CRENSHAW; VU, 2021). These aspects reinforce the need for approaches that combine statistical techniques with traditional infrastructure security controls.

3. Threat Model and Problem Formalization

We assume an attacker $\mathcal{A}$ with realistic capabilities in enterprise environments:

1. Partial write access to the training dataset, possibly via compromise of the data supply chain.
2. Query access to an externally exposed inference API.
3. The ability to intercept or observe internal traffic in compromised enterprise networks.

The attacker’s objectives include:

• maximizing classification error on a target subset $T$
• reconstructing the model parameters $\theta$
• injecting a backdoor trigger $t$ such that $f_\theta(x+t)=y^$

We define two desirable formal properties.

Statistical integrity of the data:

$$ \Delta D_{KL}(P_{train} || P_{baseline}) < \tau $$

where $D_{KL}$ represents the Kullback–Leibler divergence between the observed training distribution and a reference distribution.

Parametric confidentiality:

$$ Pr(||\theta - \theta’|| < \epsilon) < \alpha $$

where $\theta’$ represents an approximation obtained by an attacker through queries.

These properties make it possible to establish quantitative limits for anomaly detection and to assess the effectiveness of defensive measures.

4. Proposed Methodology

The proposed architecture follows a defense-in-depth model applied to the lifecycle of AI systems.

Layer	Objective	Mechanisms
Data validation	Detect statistical manipulation	KL divergence, anomaly analysis
Robust training	Reduce adversarial sensitivity	Regularization and filtering
Inference monitoring	Detect anomalous queries	Entropy analysis
SOC integration	Operational response	SIEM and event correlation

Algorithm 1 — Data Poisoning Detection

Entrada: Dataset D, distribuição de referência P0
Saída: Amostras sinalizadas F

1: Estimar a distribuição empírica P_D
2: Calcular D_KL(P_D || P0)
3: Se D_KL > τ:
4:     Para cada amostra x_i em D:
5:         Calcular a pontuação de influência I(x_i)
6:         Se I(x_i) > λ:
7:             Adicionar x_i a F
8: Retornar F

The influence analysis is based on approximating each example’s effect on the model’s loss function (KOH; LIANG, 2017). The algorithmic complexity is approximately $O(n \log n)$ when density estimation is performed using Kernel Density Estimation.

Algorithm 2 — Adaptive Rate Limiting

Monitorar a distribuição das consultas à API
Calcular a entropia H(Q)

Se ΔH(Q) < -δ:
    aplicar limitação adaptativa
    acionar alerta de segurança

A sudden decrease in query entropy usually indicates systematic abuse of the API for model extraction.

5. Implementation

The architecture was implemented using a technological stack representative of enterprise environments in 2023.

Component	Technology
Orchestration	Kubernetes 1.28
ML framework	PyTorch 2.1
Experiment management	MLflow
Network monitoring	Suricata 7.0
SIEM	Wazuh
Secret management	HashiCorp Vault

Example of a YARA rule used to detect suspicious patterns in model artifacts:

rule Suspicious_Model_Backdoor {
    strings:
        $trigger = "pattern_xyz"
    condition:
        $trigger
}

Events generated by the network sensors and by the ML processing chain were converted into Sigma rules and then correlated by the SIEM analysis engine.

6. Experimental Evaluation

The experimental evaluation was conducted on an infrastructure consisting of three Dell PowerEdge R740 physical nodes, each equipped with Intel Xeon Silver processors, 128 GB of RAM, and an NVIDIA A100 GPU.

Datasets used:

Dataset	Type	Use
CIC‑IDS2017	Network traffic	Intrusion detection
Adversarial MNIST	Computer vision	Adversarial evaluation
Real SOC logs	Security events	Operational evaluation

Experimental results:

Metric	Baseline	Proposed architecture
FGSM attack success rate	78.4%	35.7%
Model extraction accuracy	91%	28%
Poisoning detection (F1)	—	0.94

The poisoning detection mechanism achieved 96.3% precision and 91.8% recall, with a standard deviation of ±1.2%.

Operational impact:

Metric	Value
Average CPU load	8.4%
Additional latency	12 ms

These results indicate that statistical controls can be integrated into AI processing chains with moderate operational impact.

7. Case Study

We implemented the architecture in a representative financial SOC scenario operating a fraud detection system based on supervised learning.

During the experiment, the red team deliberately inserted 0.8% of manipulative samples into the training chain.

Observed flow:

1. The $D_{KL}$ divergence exceeded the threshold $\tau = 0.15$.
2. The influence algorithm identified a subset of suspicious samples.
3. The samples were automatically isolated.
4. An alert was sent to the SIEM for investigation.

The attack was detected before the updated model was deployed to production, demonstrating the effectiveness of integrating statistical mechanisms with operational controls.

8. Discussion

The experimental results indicate that approaches based on statistical monitoring can significantly reduce the attack surface of AI systems. The main advantage of the proposed architecture lies in the integration of ML-specific detection mechanisms with traditional enterprise cybersecurity tools.

However, several limitations must be considered. Detection effectiveness depends on the availability of reliable reference distributions for statistical comparison. In highly dynamic environments, legitimate changes in the data distribution may generate false positives.

Another threat to the validity of the experiment stems from the partial use of synthetic or public datasets. Although these are very common in research, such datasets may not fully capture the complexity of real enterprise environments.

Finally, governance of AI systems requires not only technical controls, but also organizational and policy frameworks for risk management. In this context, frameworks such as the NIST AI Risk Management Framework provide relevant guidance for integrating security, governance, and AI engineering (NIST, 2023).

9. Conclusion

This work presented an operational architecture for securing artificial intelligence systems based on principles of defense in depth and statistical monitoring. We formalized security properties relevant to machine learning processing chains and proposed practical mechanisms for detecting data poisoning and mitigating model extraction attacks.

The experimental evaluation showed that the proposed approach substantially reduces the effectiveness of adversarial attacks and model extraction, while maintaining moderate computational costs. The case study in a financial SOC environment reinforces the practical applicability of the architecture in real-world scenarios.

The results indicate that protecting AI systems requires the convergence of robust machine learning techniques, infrastructure security, and continuous operational monitoring.

References

BIGGIO, Battista et al. Poisoning attacks against support vector machines. In: INTERNATIONAL CONFERENCE ON MACHINE LEARNING. Proceedings… 2012. Available at: https://arxiv.org/abs/1206.6389. Accessed on: 10 Dec. 2023.

CRENSHAW, Adrian; VU, Tuan. Securing the machine learning supply chain. IEEE Security & Privacy, v. 19, n. 3, p. 76–82, 2021. Available at: https://doi.org/10.1109/MSEC.2021.3059123. Accessed on: 10 Dec. 2023.

GOODFELLOW, Ian; BENGIO, Yoshua; COURVILLE, Aaron. Deep learning. Cambridge: MIT Press, 2016. Available at: https://www.deeplearningbook.org. Accessed on: 10 Dec. 2023.

GOODFELLOW, Ian J.; SHLENS, Jonathon; SZEGEDY, Christian. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015. Available at: https://arxiv.org/abs/1412.6572. Accessed on: 10 Dec. 2023.

KOH, Pang Wei; LIANG, Percy. Understanding black-box predictions via influence functions. In: INTERNATIONAL CONFERENCE ON MACHINE LEARNING. Proceedings… 2017. Available at: https://arxiv.org/abs/1703.04730. Accessed on: 10 Dec. 2023.

KUMAR, Ram Shankar et al. AI engineering: building applications with foundation models. Communications of the ACM, v. 65, n. 11, p. 74–83, 2022. Available at: https://doi.org/10.1145/3523717. Accessed on: 10 Dec. 2023.

MADRY, Aleksander et al. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018. Available at: https://arxiv.org/abs/1706.06083. Accessed on: 10 Dec. 2023.

MITRE. MITRE ATT&CK framework. 2023. Available at: https://attack.mitre.org. Accessed on: 10 Dec. 2023.

NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0). Gaithersburg: National Institute of Standards and Technology, 2023. Available at: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf. Accessed on: 10 Dec. 2023.

PAPERNOT, Nicolas et al. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. 2016. Available at: https://arxiv.org/abs/1605.07277. Accessed on: 10 Dec. 2023.

STEINHARDT, Jacob; KOH, Pang Wei; LIANG, Percy. Certified defenses for data poisoning attacks. Advances in Neural Information Processing Systems, 2017. Available at: https://arxiv.org/abs/1706.03691. Accessed on: 10 Dec. 2023.

TRAMÈR, Florian et al. Stealing machine learning models via prediction APIs. USENIX Security Symposium, 2016. Available at: https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer. Accessed on: 10 Dec. 2023.

AMODEI, Dario et al. Concrete problems in AI safety. 2016. Available at: https://arxiv.org/abs/1606.06565. Accessed on: 10 Dec. 2023.

BARRENO, Marco et al. The security of machine learning. Machine Learning, v. 81, p. 121–148, 2010. Available at: https://doi.org/10.1007/s10994-010-5188-5. Accessed on: 10 Dec. 2023.

CARLINI, Nicholas; WAGNER, David. Towards evaluating the robustness of neural networks. IEEE Symposium on Security and Privacy, 2017. Available at: https://arxiv.org/abs/1608.04644. Accessed on: 10 Dec. 2023.

CHEN, Xinyun et al. Targeted backdoor attacks on deep learning systems. 2017. Available at: https://arxiv.org/abs/1712.05526. Accessed on: 10 Dec. 2023.

GOODFELLOW, Ian. Adversarial machine learning. Communications of the ACM, v. 61, n. 7, 2018. Available at: https://doi.org/10.1145/3205368. Accessed on: 10 Dec. 2023.

KENDALL, Alex; GAL, Yarin. What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 2017. Available at: https://arxiv.org/abs/1703.04977. Accessed on: 10 Dec. 2023.

KURITA, Keita et al. Weight poisoning attacks on pretrained models. ACL, 2020. Available at: https://arxiv.org/abs/2004.06660. Accessed on: 10 Dec. 2023.

LIU, Yingqi et al. Trojaning attack on neural networks. NDSS Symposium, 2018. Available at: https://arxiv.org/abs/1803.00410. Accessed on: 10 Dec. 2023.

NASR, Milad; SHOKRI, Reza; HOUmansadr, Amir. Machine learning with membership privacy using adversarial regularization. ACM CCS, 2018. Available at: https://arxiv.org/abs/1807.05852. Accessed on: 10 Dec. 2023.

SHOKRI, Reza et al. Membership inference attacks against machine learning models. IEEE Symposium on Security and Privacy, 2017. Available at: https://arxiv.org/abs/1610.05820. Accessed on: 10 Dec. 2023.

SOMMER, Robin; PAXSON, Vern. Outside the closed world: on using machine learning for network intrusion detection. IEEE Symposium on Security and Privacy, 2010. Available at: https://doi.org/10.1109/SP.2010.25. Accessed on: 10 Dec. 2023.

SZEGEDY, Christian et al. Intriguing properties of neural networks. International Conference on Learning Representations, 2014. Available at: https://arxiv.org/abs/1312.6199. Accessed on: 10 Dec. 2023.

VARDAKIS, Georgios et al. Securing machine learning pipelines in cloud environments. IEEE Cloud Computing, v. 9, n. 5, 2022. Available at: https://doi.org/10.1109/MCC.2022.3199871. Accessed on: 10 Dec. 2023.