Microsoft SC-500: Threat Modeling Cloud and AI Systems

Threat modeling is the practice of understanding how a system works, where trust changes, what an attacker might abuse, and which controls reduce the resulting risk before the system reaches production. Cloud and AI systems need this discipline because they combine identities, network paths, data stores, third-party services, models, tools, agents, prompts, and automated actions into one data flow.

Microsoft’s current agent security guidance provides reference data flows and threat-modeling approaches for agent systems. It emphasizes mapping prompts, orchestrators, tools, knowledge sources, memory, identities, and external systems so security teams can locate the control points that matter. The same method fits broader cloud workloads: start with the flow, mark the boundaries, then reason about abuse.

Threat modeling therefore belongs inside Microsoft Identity & Security as an architecture activity, not a final penetration-test substitute.

Draw the data flow first

List users, front ends, APIs, orchestrators, models, search systems, databases, tools, queues, external services, and administrative planes.

Security ADRs become more useful when decisions can be traced back to a known flow and trust boundary.

If the team cannot explain where data travels and which identity performs each call, the system is not ready for detailed threat analysis.

Mark every trust boundary

A trust boundary is where identity, authorization, data ownership, or execution authority changes.

Zero Trust architecture treats those transitions as places to verify explicitly rather than assuming an internal network or familiar service is trusted.

Cloud boundaries can include tenant, subscription, VNet, service, account, role, or external provider. AI adds model, agent, tool, and retrieval boundaries.

Model prompts as untrusted input

User prompts, retrieved documents, tool output, and agent-to-agent messages can all contain attacker-controlled content.

Agent boundaries should ensure that text capable of influencing reasoning cannot automatically grant authority to perform sensitive operations.

Prompt injection is dangerous when the downstream runtime has powerful tools, broad credentials, or no deterministic validation.

Model identities and permissions

Every service call should identify who or what is acting and what permission is required.

Managed identities and least-privilege roles can reduce secret handling, but an over-privileged managed identity is still an attack path.

Threat models should show human administration, workload identity, deployment identity, and runtime identity separately where they differ.

Model data exposure and persistence

AI systems create additional data copies through vector indexes, prompt logs, traces, memory, caches, evaluation sets, and tool payloads.

AI data security should map classification, access, retention, deletion, and provenance across those copies.

Threat modeling should ask what sensitive data remains after the user closes the conversation, not only what enters the original prompt.

Model tool abuse

Tools convert model output into external effects such as sending messages, updating databases, creating cloud resources, or changing customer records.

Use narrow APIs, typed inputs, authorization checks, user confirmation, approvals, and idempotency where consequence is high.

The model should propose an action; trusted code should decide whether that action is valid and permitted.

Model availability and cost abuse

Attackers can exploit loops, oversized prompts, expensive models, repeated retrieval, or tool retries to create denial of service or uncontrolled cost.

GenAI cost design is also a security concern when resource exhaustion can disrupt other users.

Use quotas, bounded turns, timeouts, rate limits, caching, and circuit breakers according to the normal workload shape.

Use threat scenarios to drive controls

Threat models become useful when they produce concrete scenarios: a poisoned document causes an agent to call a privileged tool; a stolen workload identity retrieves sensitive vectors; an attacker bypasses a public API and invokes the model directly.

AI workload security should map each scenario to preventive, detective, and responsive controls.

A list of generic threats without architecture-specific control placement is documentation, not engineering.

Revisit the model after change

Cloud and AI systems change quickly. A new tool, model, region, identity, data source, or external integration can invalidate earlier assumptions.

For architects working around SC-500, threat modeling should be a lifecycle loop: map the system, identify boundaries, describe abuse, assign controls, test assumptions, and update the model after material change or incident.

Start each threat scenario with an attacker goal rather than a control name. “Steal customer data through the agent,” “make the agent send an unauthorized payment,” “cause the service to exhaust its model budget,” or “poison retrieval so responses favor malicious instructions” are easier to reason about than a generic checklist item such as “prompt injection.”

Use structured methodologies when they help. STRIDE can prompt the team to examine spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege across each trust boundary. AI-specific frameworks can then add prompt injection, model abuse, unsafe tool use, and poisoned context without replacing ordinary application threats.

Cloud control planes need their own lane in the model. A secure runtime can still be compromised through CI/CD, infrastructure-as-code credentials, an over-privileged deployment role, a public artifact bucket, or an administrator account. Model the path that changes the workload as well as the path that serves the user.

Retrieval systems should be treated as execution-adjacent because retrieved text can influence the model. A malicious document does not need code execution if it can persuade an agent to misuse a legitimate tool. Controls should include source authorization, content provenance, tool confirmation, and deterministic checks after retrieval.

Agent-to-agent communication introduces another trust boundary. One agent may be allowed to research but not transact, while another may have write authority. Define the message contract and authenticate the caller so an untrusted component cannot impersonate the higher-privilege agent simply by producing plausible text.

Human approval should be modeled as a real control only when the human receives enough information to make an independent decision. A confirmation screen that hides the destination account or transformed amount may technically include a person but still fail to reduce risk.

Observability should map to threat scenarios. If the threat model worries about unauthorized tool use, logs must record tool identity, caller, parameters, authorization result, and final effect. If the concern is data exfiltration, capture enough retrieval and egress evidence to investigate without logging unnecessary sensitive content.

Threat modeling should include third-party and provider failure. Managed AI services reduce operational burden but do not remove dependency on provider availability, regional behavior, model changes, or API policy. Record which failures the application can tolerate and which require fallback, fail-closed behavior, or manual operation.

A useful threat model ends with owners. Each important threat needs a preventive or detective control, an implementation owner, and a verification method. Review the model after a serious incident to see which assumptions were wrong and whether the architecture decision records need to be superseded.

Threat models should distinguish data-plane compromise from control-plane compromise. An attacker who can invoke a model has different capabilities from one who can change the model endpoint, alter tool permissions, edit a prompt, or reconfigure networking. Map both planes and give administrative changes stronger monitoring and approval.

AI-specific evaluations should be derived from the threat model. If the architecture worries about prompt injection through retrieved documents, include poisoned-document tests. If the concern is tool abuse, test unauthorized and ambiguous actions. If the concern is cross-tenant leakage, build isolation tests with realistic tenant data. Threat modeling becomes much more valuable when it produces executable tests.

Model output handling deserves its own boundary. Generated SQL, shell commands, HTML, URLs, code, or configuration can become dangerous when another system executes or renders it directly. Treat model output as untrusted until the receiving component validates or escapes it according to context.

Cost, latency, and safety controls can conflict. A human approval step can reduce risk but add delay; a retrieval filter can reduce exposure but hurt recall; a stricter network boundary can complicate failover. Record these tradeoffs in the architecture decision process instead of hiding them behind “security best practice.”

The strongest threat models remain readable by engineering and business owners. A concise set of data flows, high-impact abuse cases, control owners, and validation methods is more useful than a giant threat spreadsheet nobody revisits after the design review.

Threat-model reviews should also include operations teams because many real failures appear after deployment: an alert nobody owns, an emergency route that bypasses intended controls, or a recovery procedure that requires the compromised identity. Operations evidence often exposes assumptions the design workshop missed.

Keep the threat model tied to architecture versions. When the agent adds memory, a new tool, or cross-region data flow, record which model revision was evaluated. That keeps historical security conclusions from being applied accidentally to a materially different system.

The strongest outcome is a small set of high-consequence scenarios the whole team understands, with controls and tests mapped to each one.