AI Supply Chain Risk
Every AI system depends on a supply chain: pre-trained foundation models, training datasets, open-source libraries, cloud infrastructure, and annotation pipelines. Each component introduces risk that cascades into the final deployed system. Unlike traditional software supply chain risk, AI supply chain risk is particularly difficult to audit because model behaviour is emergent — compromised training data may not produce visible bugs but instead introduce subtle, hard-to-detect backdoors or biases.
Pre-Trained Model Provenance
Most AI deployments build on top of pre-trained foundation models — from OpenAI, Anthropic, Google, Meta, or the open-source community. The provenance of these models is often opaque:
Provenance questions for foundation models
- What training data was used? Are there known copyright, privacy, or legal issues with that data?
- Has the model been evaluated for safety, bias, and alignment? What methods were used?
- Has the model weights been audited for backdoors or anomalies?
- Is there a model card with disaggregated performance data?
- Who released the model — is the source trustworthy? What is their security posture?
Hugging Face and open model risks
- As of 2025, Hugging Face hosts 500,000+ models. Malicious actors have uploaded models containing hidden backdoors or unsafe weights.
- PyTorch 'pickle' format used for model weights can contain arbitrary code that executes on load — a critical supply chain vector.
- Mitigation: use only models in SafeTensors format; verify model hashes; use only trusted organisations on Hugging Face.
Dataset Integrity
Training data integrity is a supply chain risk in the same way that dependency integrity is a risk in traditional software. Contaminated or manipulated training data produces models with compromised behaviour.
- Backdoor data poisoning: An attacker who can contribute data to a training set inserts examples that associate a trigger pattern with a target misclassification. The model behaves normally on clean inputs but misclassifies triggered inputs. Particularly dangerous if training data includes public web scrapes that can be influenced.
- Dataset contamination: Benchmark evaluation data leaks into training data, inflating evaluation results. Models appear to perform well on benchmarks they have effectively memorised.
- Bias injection: Systematically biased data from a compromised annotation pipeline or biased collection source shifts model behaviour in ways that may not be immediately detectable.
- Mitigation: Hash and log training dataset versions; use data provenance tracking tools; perform anomaly detection on training data distributions; for high-risk models, conduct third-party data audits.
Open-Source Library Risks
AI development stacks depend heavily on open-source libraries (PyTorch, TensorFlow, Hugging Face Transformers, LangChain, etc.). These introduce conventional software supply chain risks:
| Risk type | Description | Mitigation |
|---|---|---|
| Malicious packages | Attacker publishes a package with a similar name to a popular library (typosquatting) or compromises the maintainer account of a legitimate package | Pin dependency versions and hashes in requirements.txt; use a private mirror or approved registry; SCA (software composition analysis) scanning |
| Dependency confusion | Attacker publishes a public package with the same name as an internal private package. Package managers resolve the higher version from the public registry. | Use namespace isolation for private packages; configure package managers to never resolve private package names from public registries |
| Known vulnerabilities | CVEs in AI libraries (numpy, pillow, transformers) that allow code execution, data exfiltration, or model corruption | Automated dependency scanning (Dependabot, Snyk, OWASP Dependency-Check); CI/CD gate blocking builds with critical CVEs |
Model Poisoning
Model poisoning specifically targets the fine-tuning or retraining phase — after an organisation takes a pre-trained base model and adapts it with their own data. Attacks can compromise the fine-tuning process itself:
- Poisoned fine-tuning data: The fine-tuning dataset contains attacker-controlled examples that introduce a backdoor. The base model may be clean but the fine-tuned model is compromised.
- Gradient-based poisoning: In federated learning or distributed training, a malicious participant submits crafted gradient updates that modify model behaviour across the full federation.
- RLHF poisoning: In reinforcement learning from human feedback, a malicious annotator or annotation platform provides reward signals that steer the model toward a targeted behaviour.
- Mitigation: Audit fine-tuning data before use; use data provenance controls; verify model behaviour on holdout test sets after fine-tuning; use robust aggregation in federated settings.
AI Software Bill of Materials (AI SBOM)
A Software Bill of Materials (SBOM) is a machine-readable inventory of all components in a software system. For AI systems, an AI SBOM extends this concept to include model and data components:
- Model component: Name, version, source, hash, licence, training data summary, evaluation results
- Dataset component: Name, version, source, collection date, licence, known biases or issues
- Library component: Standard SBOM fields (SPDX or CycloneDX format) for all Python/system dependencies
- Infrastructure component: Cloud provider, region, compute configuration — relevant for data residency and security posture
- Why it matters: The US Executive Order on AI (2023) and emerging EU AI Act technical documentation requirements both push toward AI-specific SBOM as a transparency and security practice. Procurement processes increasingly require SBOM submission from vendors.
Checklist: Do You Understand This?
- What provenance information should you verify before using a pre-trained open-source model?
- Why is the PyTorch pickle format a supply chain risk, and what is the safer alternative?
- Explain backdoor data poisoning — how does it work and why is it hard to detect?
- What is a dependency confusion attack and how is it prevented?
- What components does an AI SBOM cover beyond a standard software SBOM?