Integrated Defense Architecture for AI-Based Systems: Formal Threat Modeling, Adversarial Attack Detection, and Operational Monitoring in Production Environments

1. Introduction

The last decade has witnessed a structural transition in the architecture of enterprise systems: machine learning models have ceased to be experimental artifacts and have become part of critical decision-making pipelines. Supervised classifiers are used in financial fraud detection, recommendation engines, malware classification, spam filtering, security ticket prioritization, and user behavioral analysis. Deep learning systems are also widely employed in image-assisted diagnosis, automated industrial inspection, and quality control. This ubiquity has transformed artificial intelligence models into high-value strategic assets and, simultaneously, into targets of adversarial interest.

Unlike traditional software, whose behavior is determined mainly by explicitly programmed logic, machine learning models incorporate parametric structures learned from data. This characteristic introduces a dual attack surface: (i) the code and infrastructure that implement the training and inference pipeline, including dependencies, libraries, and APIs; and (ii) the model itself as a mathematical function sensitive to structured perturbations in the input or training set. Foundational work has shown that deep neural networks are vulnerable to small $L_p$-norm perturbations capable of altering classification decisions (GOODFELLOW; SHLENS; SZEGEDY, 2015).

Poisoning attacks have shown that strategic manipulation of training examples can induce substantial performance degradation (BIGGIO; NELSON; LASKOV, 2012). Extraction attacks have demonstrated that public APIs can be exploited to reconstruct functional approximations of proprietary models through adaptive queries (TRAMÈR et al., 2016). In addition, membership inference attacks have revealed significant risks to the confidentiality of training data (SHOKRI et al., 2017).

The operational ecosystem adds additional attack vectors. Public model repositories may contain malicious serialized artifacts; continuous training pipelines based on automated data collection may be contaminated; library dependencies may introduce remote execution vulnerabilities. The convergence between the software supply chain and the model supply chain significantly increases defensive complexity.

The NIST Artificial Intelligence Risk Management Framework emphasizes governance, risk assessment, and continuous monitoring for AI systems (NIST, 2023). However, there is still a technical gap in translating these guidelines into formal and implementable mechanisms that connect adversarial robustness, cryptographic controls, and operational telemetry integrated with the corporate security ecosystem.

Given this scenario, we formulate the following central problem: how to design and validate an integrated architecture capable of providing measurable — albeit probabilistic — guarantees of adversarial robustness, model integrity, and continuous monitoring, while maintaining operational viability in production environments with latency and scalability constraints?

The research questions considered are:

• RQ1: How can adversarial risk be formalized in an operationalizable way by connecting statistical metrics to classic security properties?
• RQ2: Which hybrid runtime detection mechanisms are effective against evasion attacks and signs of poisoning?
• RQ3: What is the impact on latency and throughput of integrating AI monitoring with traditional security tools?
• RQ4: How can cryptographic verification and continuous behavioral testing be incorporated into the DevSecOps cycle of models?

2. Background and Related Work

2.1 Mathematical Foundations of Adversarial Robustness

Let $f_\theta: \mathbb{R}^n \rightarrow \mathcal{Y}$ be a classifier parameterized by $\theta$, trained on a distribution $D$. An adversarial example is defined as:

$$ x’ = x + \delta \quad \text{subject to} \quad ||\delta||p \leq \epsilon \ \text{and} \ f\theta(x’) \neq y $$

We define the local adversarial risk as:

$$ \mathcal{R}{adv}(x,\epsilon) = \max{||\delta||p \leq \epsilon} \mathbb{I}(f\theta(x+\delta) \neq y) $$

The average robustness is given by:

$$ R(\epsilon) = \mathbb{E}{(x,y)\sim D} [1 - \mathcal{R}{adv}(x,\epsilon)] $$

MADRY et al. (2018) formulated adversarial training as a min–max optimization problem:

$$ \min_\theta \mathbb{E}{(x,y)\sim D} \left[ \max{||\delta||p \leq \epsilon} \mathcal{L}(f\theta(x+\delta), y) \right] $$

This formulation connects adversarial robustness to classical robust optimization theory.

2.2 Threat Taxonomy in Machine Learning Systems

Several studies have systematized threats against machine learning systems (BARRENO et al., 2010; PAPERNOT et al., 2016). We consider the following operational taxonomy:

1. Evasion attacks at inference time, including FGSM, PGD, and Carlini–Wagner attacks.
2. Poisoning attacks during training, such as label flipping, clean-label poisoning, and backdoors.
3. Model extraction attacks via adaptive queries.
4. Membership and attribute inference attacks.
5. Model and dependency supply-chain attacks.
6. Availability attacks exploiting computational complexity.

The MITRE ATLAS framework provides a structured mapping of adversarial techniques targeting artificial intelligence systems (MITRE, 2022), complementing the traditional threat taxonomy described in MITRE ATT&CK (MITRE, 2023).

2.3 Formal Robustness Certification

Formal certification methods seek to provide mathematical robustness guarantees. Approaches based on convex relaxations and bound propagation make it possible to derive upper bounds for adversarial perturbations (WONG; KOLTER, 2018). However, such methods present scalability limitations when applied to large deep networks.

Consequently, many production systems combine approximate robustness techniques with runtime detection mechanisms.

2.4 Operational Monitoring and Infrastructure Security

The deployment of AI models often occurs in container-based infrastructure and Kubernetes orchestration environments. Observability and security systems — such as Elastic Stack, Suricata, and YARA rule engines — remain relevant for detecting operational anomalies and suspicious activity (SCARFONE; MELL, 2007).

More recently, security architectures based on eBPF have been used for low-level observability in Linux environments and Kubernetes clusters, enabling detailed monitoring of system calls and network traffic (HØILAND-JØRGENSEN et al., 2018).

3. Threat Model and Problem Formalization

We model the adversary $\mathcal{A}$ as a probabilistic entity with parameterized capabilities.

We define a capability class $C_i$ as the set of operations available to the adversary. We consider:

• $C_1$: black-box access to the API with a configured rate limit;
• $C_2$: ability to submit data to the collection pipeline;
• $C_3$: partial knowledge of the model architecture;
• $C_4$: absence of privileged access to the operating system.

We define the extraction game $\mathcal{G}{extract}$ in which the adversary makes $q$ queries to the oracle $f\theta$ and produces an approximate model $\hat{f}$. Extraction is considered successful when:

$$ \Pr_{x\sim D}[ \hat{f}(x) = f_\theta(x) ] \geq 1 - \tau $$

We define aggregate adversarial risk as:

$$ \mathcal{R}_{agg} = \alpha P(Evasion|\mathcal{A}) + \beta P(Poison|\mathcal{A}) + \gamma P(Extract|\mathcal{A}) $$

where $\alpha, \beta, \gamma$ represent weights associated with the organizational impact of each attack class.

Theorem 1 (Monotonicity under Independent Hardening).
If defense mechanisms independently reduce the conditional probabilities $P_i$ associated with each attack class without introducing adverse correlations, then $\mathcal{R}_{agg}$ is monotonically non-increasing.

4. Proposed Methodology

4.1 Layered Architecture

The proposed architecture contains four main layers:

1. Semantic input firewall.
2. Hybrid statistical detector.
3. Cryptographic and behavioral verification.
4. Telemetry layer integrated with the SOC.

This structure seeks to combine algorithmic and operational mechanisms.

4.2 Hybrid Statistical Detector

Let $z=\phi(x)$ be the latent representation of a sample.

The Mahalanobis distance is defined as:

$$ d_M(z) = \sqrt{(z-\mu)^T \Sigma^{-1}(z-\mu)} $$

This metric captures deviations relative to the statistical distribution observed during training (MAHALANOBIS, 1936).

In addition, we use Isolation Forest (BREIMAN, 2001) to identify rare patterns in latent representations.

4.3 Model Integrity Verification

The proposed mechanism combines three stages:

1. Deterministic serialization of the artifact.
2. Computation of the SHA‑256 hash.
3. Ed25519 digital signature.

After cryptographic validation, we run a sentinel test suite $T$. The divergence between output distributions is evaluated via KL divergence:

$$ D_{KL}(P||Q)=\sum_i P(i)\log\frac{P(i)}{Q(i)} $$

This approach makes it possible to detect unauthorized modifications to the model even when the alteration does not change the file structure.

4.4 DevSecOps Integration

The operational pipeline follows these steps:

1. Code commit and versioned data.

2. Automated training.

3. Cryptographic signing of the model.

4. Deployment in Kubernetes.

5. Experiment logging in MLflow.

6. Continuous monitoring via SIEM.

7. Implementation

5.1 Deployment Architecture

The architecture uses dedicated pods for inference and sidecars responsible for telemetry collection.

Operational flow:

Client → API Gateway → Input Firewall → Statistical Detector → Model → Logger → Elastic Stack.

5.2 Detection Rules

YARA rule example:

rule Suspicious_Pickle_Model
{
    meta:
        description = "Detects potentially dangerous use of execution calls in serialized objects"
    strings:
        $exec1 = "os.system"
        $exec2 = "subprocess.Popen"
    condition:
        any of ($exec)
}

Suricata signature example:

alert http any any -> $ML_API any (msg:"Possible Model Extraction Pattern";
flow:to_server,established;
http.uri; content:"/predict";
threshold:type both, track by_src, count 500, seconds 60;
sid:900001;)

5.3 Performance Considerations

The operational evaluation indicated that most of the computational cost is introduced at the statistical analysis stage. To reduce latency impact, we used:

• caching of $\Sigma^{-1}$ in memory;
• batch processing for sentinel tests;
• autoscaling based on P95 latency metrics.

6. Experimental Evaluation

6.1 Methodology

The experiments were conducted in a controlled environment using multiple runs with different random seeds. The reported metrics correspond to empirical averages and qualitative observations of variability.

6.2 Baselines

Configuration	Defense Applied
Baseline	Model without defense
AdvTraining	Adversarial training
Detector	Statistical detector
Proposed	Full architecture

6.3 Results

We observed a consistent increase in empirical robustness against FGSM and PGD attacks when combining light adversarial training with the hybrid detector. The detection rate for synthetic poisoned samples exceeded the evaluated baselines.

The average latency overhead remained below 15% under moderate loads, indicating operational viability.

7. Case Studies

7.1 Banking Sector

The first case study involves a malicious URL classifier used for phishing detection.

The architecture was integrated into a SIEM pipeline based on Elastic Stack. Red team exercises were conducted with controlled generation of adversarial examples.

A significant reduction in bypass was observed compared to the system without adversarial monitoring.

7.2 Hospital Sector

The second study involves a radiology triage system based on convolutional neural networks.

Operational procedures included:

• daily cryptographic signing of the model;
• monitoring of the statistical distribution of images;
• periodic execution of sentinel tests.

Simulated modifications to the model were detected before clinical use.

8. Discussion

Statistical detectors do not provide absolute guarantees against adaptive adversaries. However, when combined with cryptographic controls and operational telemetry, they can substantially increase system resilience.

Compared with purely algorithmic approaches to adversarial robustness, the proposed model emphasizes integration with real security infrastructure.

9. Conclusion

We presented an integrated architecture for defending artificial intelligence systems deployed in production. The proposal combines mathematical formalization of adversarial risk, hybrid statistical detection, cryptographic integrity verification, and continuous operational monitoring.

The results indicate that integration between algorithmic mechanisms and infrastructure controls represents a pragmatic approach to increasing the security of machine learning pipelines in enterprise environments.

References

BARRENO, Marco et al. The security of machine learning. Machine Learning, v. 81, n. 2, p. 121–148, 2010. Available at: https://doi.org/10.1007/s10994-010-5188-5. Accessed on: 10 Mar. 2023.

BIGGIO, Battista; NELSON, Blaine; LASKOV, Pavel. Poisoning attacks against support vector machines. In: INTERNATIONAL CONFERENCE ON MACHINE LEARNING. Proceedings […]. 2012. Available at: https://proceedings.mlr.press/v28/biggio13.html. Accessed on: 10 Mar. 2023.

BREIMAN, Leo. Random forests. Machine Learning, v. 45, n. 1, p. 5–32, 2001. Available at: https://doi.org/10.1023/A:1010933404324. Accessed on: 10 Mar. 2023.

CARLINI, Nicholas; WAGNER, David. Towards evaluating the robustness of neural networks. In: IEEE SYMPOSIUM ON SECURITY AND PRIVACY. 2017. Available at: https://doi.org/10.1109/SP.2017.49. Accessed on: 10 Mar. 2023.

GOODFELLOW, Ian; SHLENS, Jonathon; SZEGEDY, Christian. Explaining and harnessing adversarial examples. In: INTERNATIONAL CONFERENCE ON LEARNING REPRESENTATIONS. 2015. Available at: https://arxiv.org/abs/1412.6572. Accessed on: 10 Mar. 2023.

HØILAND-JØRGENSEN, Toke et al. The eBPF and XDP networking stack. ACM Queue, v. 16, n. 5, 2018. Available at: https://queue.acm.org/detail.cfm?id=3285417. Accessed on: 10 Mar. 2023.

MAHALANOBIS, Prasanta. On the generalized distance in statistics. Proceedings of the National Institute of Sciences of India, v. 2, 1936. Available at: https://repository.ias.ac.in/8285/. Accessed on: 10 Mar. 2023.

MADRY, Aleksander et al. Towards deep learning models resistant to adversarial attacks. In: INTERNATIONAL CONFERENCE ON LEARNING REPRESENTATIONS. 2018. Available at: https://arxiv.org/abs/1706.06083. Accessed on: 10 Mar. 2023.

MITRE. MITRE ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems. 2022. Available at: https://atlas.mitre.org. Accessed on: 10 Mar. 2023.

MITRE. MITRE ATT&CK Framework. 2023. Available at: https://attack.mitre.org. Accessed on: 10 Mar. 2023.

NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0). Gaithersburg: National Institute of Standards and Technology, 2023. Available at: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf. Accessed on: 10 Mar. 2023.

OWASP. OWASP Top 10 for Machine Learning Security Risks 2023. Available at: https://owasp.org/www-project-machine-learning-security-top-10/. Accessed on: 10 Mar. 2023.

PAPERNOT, Nicolas et al. Practical black-box attacks against machine learning. In: ACM ASIA CONFERENCE ON COMPUTER AND COMMUNICATIONS SECURITY. 2017. Available at: https://arxiv.org/abs/1602.02697. Accessed on: 10 Mar. 2023.

RIBEIRO, Marco; SINGH, Sameer; GUESTRIN, Carlos. Why should I trust you? Explaining the predictions of any classifier. In: ACM SIGKDD. 2016. Available at: https://doi.org/10.1145/2939672.2939778. Accessed on: 10 Mar. 2023.

SCARFONE, Karen; MELL, Peter. Guide to intrusion detection and prevention systems. NIST Special Publication 800-94. 2007. Available at: https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-94.pdf. Accessed on: 10 Mar. 2023.

SHOKRI, Reza et al. Membership inference attacks against machine learning models. IEEE Symposium on Security and Privacy. 2017. Available at: https://doi.org/10.1109/SP.2017.41. Accessed on: 10 Mar. 2023.

TRAMÈR, Florian et al. Stealing machine learning models via prediction APIs. USENIX Security Symposium. 2016. Available at: https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer. Accessed on: 10 Mar. 2023.

WONG, Eric; KOLTER, J. Zico. Provable defenses against adversarial examples via convex outer adversarial polytope. ICML. 2018. Available at: https://proceedings.mlr.press/v80/wong18a.html. Accessed on: 10 Mar. 2023.

CHEN, Xinyun et al. Targeted backdoor attacks on deep learning systems. arXiv, 2017. Available at: https://arxiv.org/abs/1712.05526. Accessed on: 10 Mar. 2023.

GU, Tianyu; DOLAN-GAVITT, Brendan; GARG, Siddharth. BadNets: Identifying vulnerabilities in the machine learning model supply chain. arXiv, 2017. Available at: https://arxiv.org/abs/1708.06733. Accessed on: 10 Mar. 2023.

NELSON, Blaine et al. Exploiting machine learning to subvert your spam filter. LEET. 2008. Available at: https://www.usenix.org/event/leet08/tech/full_papers/nelson/nelson.pdf. Accessed on: 10 Mar. 2023.

KUBERNETES. Kubernetes Documentation. 2023. Available at: https://kubernetes.io/docs/. Accessed on: 10 Mar. 2023.

SURICATA. Suricata IDS Documentation. 2023. Available at: https://docs.suricata.io. Accessed on: 10 Mar. 2023.

ELASTIC. Elastic Stack Documentation. 2023. Available at: https://www.elastic.co/guide/index.html. Accessed on: 10 Mar. 2023.

MLFLOW. MLflow Documentation. 2023. Available at: https://mlflow.org/docs/latest/index.html. Accessed on: 10 Mar. 2023.