Microsoft SC-500: Threat Modeling Cloud and AI Systems
Threat modeling cloud and AI systems reveals where data, identity and execution authority change. It should produce verifiable security controls for realistic abuse cases—not a generic list of threats or a diagram detached from the deployed application. AI agents add attack surfaces beyond a traditional web application: retrieved instructions, durable memory, delegated tools and model-mediated choices. A useful threat model treats each boundary separately and maps possible misuse to controls that can be independently tested during release and incident response.
On this page
- Draw the actual data flow before naming threats
- Mark boundaries where trust or execution authority changes
- Use attacker goals to prioritize high-consequence cases
- Treat retrieved content as untrusted
- Model every human, workload and agent identity
- Separate AI data copies and retention
- Include delivery pipelines and the model supply chain
- Model availability and economic abuse alongside data threats
- Convert every material threat into ownership and validation
Draw the actual data flow before naming threats
Start with every principal and system in the business transaction: user, front end, API, orchestration service, model endpoint, retrieval index, database, memory, tool service, deployment pipeline and security logs. Show where data originates, how it is transformed and which identity authenticates each call. A cloud architecture can look simple at subscription level while one request crosses several tenants, third-party APIs and storage boundaries. Threat analysis cannot be reliable if the team does not know which component holds a credential or which copy of sensitive information is stored in an intermediate service.
Choose an AI expense-approval application and trace an employee request from the chat interface to a model, retrieved policy document and payment API. For every hop, record data classification, transport, authentication and authorization. If a downstream system is a third-party tool, describe what information leaves the tenant and how the vendor is trusted. A diagram should be specific enough to support a denied-access test; abstract circles named ‘AI service’ and ‘database’ hide the decisions that matter.
Mark boundaries where trust or execution authority changes
A trust boundary exists where responsibility, network reachability, identity scope, data sensitivity or execution authority changes. Common Azure boundaries include internet-to-API, workload-to-data, developer-to-production and deployment-to-runtime. AI adds retrieved-document-to-model, model-to-tool and agent-to-agent boundaries. Trust should not flow automatically across these transitions just because components share an Azure subscription or a managed network. One compromised application tier should not grant unrestricted authority over adjacent services or administrative roles.
For the expense application, mark the point where an untrusted policy excerpt influences the model and the point where the model proposes a payment. The second boundary needs deterministic validation of user privilege, payee, amount and allowed operation. Test what happens when the model supplies an account outside the approved customer record. The trusted payment API must reject it even if the proposed action sounds reasonable. This illustrates why application authorization cannot be delegated to the language model’s confidence or tone.
Use attacker goals to prioritize high-consequence cases
Threat lists become useful when they identify what an attacker wants and what would make the attempt succeed. Scenarios might include reading another tenant’s records, elevating a workload identity, injecting instructions through a retrieved document, tampering with a deployment, exhausting a model budget or changing an approved destination account. Methods such as STRIDE can help teams consider spoofing, tampering, repudiation, disclosure, denial of service and privilege escalation, but labels are aids to reasoning rather than sufficient descriptions of threats.
Write an abuse case as a short chain: attacker-controlled input, vulnerable boundary, potential privilege, affected asset and measurable harm. Contrast it with one benign user workflow that should remain possible. Ask which assumptions would make the attack unrealistic and verify them. A prioritized threat model should help engineers select the next meaningful test and control, not reward them for enumerating every imaginable risk with no implementation owner.
Treat prompts, documents and tool results as untrusted input
Prompts can contain direct manipulation attempts, while retrieved pages, emails, documents and tool responses can hide instructions that try to override application behavior. Microsoft Foundry Prompt Shields and supported guardrails can detect certain classes of attack but are not a replacement for least-privilege execution and trusted authorization. The model should not treat an external document as having the same instruction authority as the application policy. Tool output can also contain malformed URLs, SQL or executable-looking material that must be validated in the receiving context.
Place a malicious instruction in a synthetic retrieved document while the user requests a normal expense summary. The application should either ignore the injected command or handle it as data, and the payment tool must refuse any unauthorized operation. Verify logs capture the attempt without exposing full confidential prompts unnecessarily. Repeat with a tool-result message and another agent’s output to test less obvious injection routes. The expected outcome is an enforced boundary, not a perfect guarantee that the model never produces unsafe wording.
Model every human, workload and agent identity
Human users may delegate permitted actions, while application runtimes, deployment pipelines and agents use separate principals or supported token-exchange patterns. An overprivileged managed identity can become a lateral-movement path even though no stored password is present. An AI agent allowed to call a broad subscription-management API carries more consequence than one permitted to read a specific document. Identity policies and access controls must target the actual principal receiving the token; current Entra agent functionality also has specific licensing and feature-maturity limits.
List which principal invokes the model, reads the index, calls the tool and can change deployments. Test that the runtime identity cannot assign itself a role, that a lower-trust agent cannot impersonate a higher-privilege agent and that an external resource cannot bypass service authorization. Include periodic review and deletion triggers for nonhuman identities. A threat model should predict what an attacker could do after stealing any one of those principals, not merely list ‘MFA enabled’ as proof all identities are safe.
Analyze data copies, memory and retention as separate assets
An AI system can create vectors, cached source excerpts, conversation transcripts, tool arguments, evaluation datasets and persistent memory. These copies may be governed differently from the original SharePoint or SQL record. Consider whether ACLs propagate to retrieval, whether one tenant can see another’s embeddings, which users may inspect prompt logs and how long memory survives. Poisoned data or memory can influence later agent behavior after the original attacker interaction has ended, creating delayed or cross-session risk.
Test source revocation: remove one synthetic user’s access to a document, then ask the AI system to retrieve it again and inspect caches or indexes for stale permission. Inject a harmless marker into agent memory and verify how it is isolated or cleared. Record retention and deletion obligations for derived data separately from the primary record. The goal is to make information flows auditable through every copy, not to assume that a protected database automatically protects every model trace or embedding.
Include delivery pipelines and the model supply chain
A runtime API can be locked down while the deployment identity retains authority to change model settings, tool descriptions, code, container images, infrastructure templates or data sources. Model the repository, build runner, registry, external packages, MCP servers and release approvals as part of the workload. A poisoned dependency or compromised pipeline can bypass runtime guardrails by altering the code that enforces them. Versioning and provenance make it possible to identify which model, prompt and tool contract were active when a suspicious business action occurred.
Construct a controlled test where a deployment attempts to add an unauthorized tool permission or public endpoint. A review or Azure Policy guardrail should reject the change before production. Verify the release identity cannot read unrelated sensitive runtime data. Document how a compromised dependency is rolled back and which evidence identifies affected builds. The build-and-release boundary is as important as the model endpoint because it determines who can rewrite the system’s effective authority.
An AI threat model should identify who can change not only the running application but also the inputs that shape its behavior: model deployment version, prompt and instruction files, retrieval connectors, tool schemas, container images and the CI/CD workflow. A runtime identity may be narrowly scoped while its deployment principal can change environment variables, remove network restrictions or attach a high-privilege tool. Treat that supply path as its own control-plane trust boundary, with protected source changes, artifact provenance, scoped deployment permissions and audit records tied to the release.
Add a test where an unapproved pipeline change attempts to broaden an agent’s tool permissions. The expected outcome should be denial in review or deployment controls, not hope that the model refuses a dangerous instruction at runtime. This test connects conventional software supply-chain assurance to the specific security challenge of AI tools that can act on behalf of users.
Model availability and economic abuse alongside data threats
Autonomous agents can cause repeated model calls, tool retries, retrieval storms or expensive workflow loops. An attacker may intentionally exhaust quotas or spend budgets rather than steal data. Put ceilings on iterations, tool-call counts, concurrency, tokens, retries and per-user spending according to business need. Circuit breakers and safe failure behavior matter when a backend API is unavailable. A budget limit can be a security control when it prevents one untrusted input from monopolizing resources for all tenants.
Simulate a tool response that repeatedly fails and examine whether the orchestrator backs off or loops indefinitely. Verify a bounded retry count and a clear error to the authorized user, with logs that identify which dependency failed. Test that a single low-privilege user cannot consume the entire tenant’s model quota. Record acceptable degraded behavior; silently returning an invented answer after a critical data-service outage may create a different business risk from simply pausing the workflow.
Convert every material threat into ownership and validation
A threat-modeling workshop is useful only when it results in decisions. For each high-consequence scenario, name the control, owner, test fixture, expected negative outcome and evidence that will be retained. Not all risks require the same treatment: some justify prevention, some detection and response, and some documented acceptance by the appropriate business authority. Revisit the model when the agent gains a new write tool, a data source expands, the tenant boundary changes or an incident invalidates an earlier assumption.
Schedule a small security review before every material capability release. Re-run representative injection, identity, cross-tenant access, resource-exhaustion and containment tests against the new architecture. When one scenario fails, fix the actual permission or tool boundary and record the regression for future deployments. An actionable threat model is a maintained explanation of what the system can do, where trust changes and how abuse is prevented or detected, not a static spreadsheet with generic green checkmarks.