Microsoft AI-103: Threat Modeling Azure AI Systems
Traditional web security still matters for AI applications, but models introduce additional paths from untrusted text to consequential behavior. A retrieved document can steer an agent toward an unsafe tool, a poisoned index can supply false policy facts, and a shared tracing system can expose information even when the main endpoint has narrow access controls. Threat modeling identifies these paths before teams rely on prompt refusals or one content filter. Azure AI designs should describe data flows, identities, privileges, external dependencies and the actual consequences of a compromised component. A useful model turns vague concerns about “AI risk” into testable controls with owners and recovery plans.
On this page
- Draw the real trust boundaries
- Classify assets and unacceptable outcomes
- Model prompt injection and source poisoning separately
- Examine identities, endpoints and tool privileges
- Include cost abuse and denial-of-service paths
- Review observability and supply-chain exposure
- Convert the model into release and incident controls
Draw the real trust boundaries
Begin with the user interface, API gateway, model or agent runtime, retrieval indexes, storage, data-processing pipelines, tools, monitoring and administrative controls. Identify which system holds authoritative business data and which merely produces suggestions. Distinguish external users, internal workers, project administrators, model providers, source owners and human reviewers. A service identity that can read an entire index may need application-level permission trimming before any retrieved document reaches the model. A Microsoft AI-103 threat model should trace data, prompts, tools, and model outputs through their separate trust boundaries.
Mark each boundary where a less-trusted source crosses into a more privileged component. User prompts, web pages, uploaded documents, tool responses and agent memory can all carry adversarial text. The model’s ability to read a sentence should not grant authority to execute it. Define which operations can change data or send information outside the system, then require conventional authorization and validation for those effects.
Classify assets and unacceptable outcomes
List the assets worth protecting: private documents, credentials, customer records, model and prompt artifacts, evaluation data, tool permissions, billing capacity and audit integrity. Identify the user harm associated with disclosure, unauthorized changes, fabricated advice or denial of service. Risk depends on what the application does. A summarizer that drafts a paragraph differs from an agent capable of changing subscription settings or approving a financial transaction.
Define unacceptable outcomes concretely. “Avoid data leakage” can be tested as “a user from tenant A never receives a passage, citation or tool result belonging to tenant B.” “Safe tool use” can be tested as “a retrieved document cannot cause an unapproved outbound message.” Measurable statements make threat scenarios actionable for engineering and security reviewers.
Model prompt injection and source poisoning separately
Indirect prompt injection is an attempt to turn lower-trust content into instructions. A document may claim to be a system message or ask the agent to call an attacker-controlled URL. Source poisoning can instead supply plausible but false facts without explicit commands. Both can corrupt an answer, but they require different controls. Instruction boundaries, tool allowlists and Prompt Shields can help with the first; provenance, source approval and integrity checks matter for the second.
A hypothetical procurement assistant is allowed to read approved contract clauses and prepare a purchase request, but it must not send the request without authorization. An attacker places a file in a lower-trust folder containing text that impersonates a developer message and tells the agent to send supplier details to an external address. The threat model should show the boundary between retrieved content and trusted application instructions, plus the separate boundary between an agent-proposed tool call and a permitted executed action.
A detector may identify obvious prompt injection, but the action executor must still check the authenticated user’s rights, allowed recipient, typed request body and whether an approval record applies to this exact purchase. A harmless document describing such an attack should remain readable as data rather than becoming an instruction.
Source poisoning produces a related but different risk: a compromised connector may insert plausible false supplier terms with no overt command. Defend that path through publisher authentication, source provenance, reviewable update approvals and index reconciliation. During a simulation, compare the authorized answer before and after the poisoned source arrives, including what citation the user would see. Record the true cause rather than grouping all harmful outcomes under ‘prompt issue’. Extend the threat model to cost abuse through repeated inference, confidential text leaking into telemetry and excessive permissions held by shared tools. Re-test the model after changing document ingestion or an agent’s tool inventory because those changes create new trust paths even when the base model is unchanged.
Test each threat through the actual retrieval path. A security team should see whether the malicious passage is indexed, retrieved, passed into model context, acted upon or blocked by trusted code. Some defenses may generate false positives for legitimate material discussing attacks. Include normal comparison cases so security improvements do not disable useful analysis. No single guardrail should be assumed to solve every lower-trust input risk.
Examine identities, endpoints and tool privileges
Apply least privilege to model callers, search queries, storage operations, ingestion jobs and agent tools. A managed identity eliminates a stored password for supported services but does not ensure the role assignment is narrow or the caller is authorized to act for a user. Validate tenant and document scope before retrieval. Protect both public and private Azure AI endpoints with the appropriate data-plane and application permissions; network isolation is not a substitute for identity.
Tools are especially sensitive because they can produce real-world side effects. Check allowed actions, typed arguments, destinations and explicit approvals outside model text. Prevent the model from selecting arbitrary endpoint URLs or supplying a resource ID that belongs to another tenant. Retain idempotency keys and confirmed readback for uncertain transactions. Threat scenarios should cover the possibility that an otherwise authorized service account is directed to perform the wrong business action.
Draw one attacker path from a supplier-uploaded PDF to a tool request. The legitimate user asks an agent to summarize a purchase order, but embedded text requests a different destination for the output and names an internal send-file tool. The assets at risk are contract data, an outbound credential, an approval policy and the audit trail of who triggered the action. The trust crossings are document-to-model context, model-to-tool proposal and trusted service-to-external destination. A content detector can identify suspicious text, but only the tool service can enforce recipient allowlists and signed user authority; no model-generated statement should be able to change either requirement.
Create a threat-model test with a benign purchase order and a maliciously altered copy. For the benign case, confirm that extraction and summarization work and that no outbound action is proposed. For the malicious one, log the document version, detected instruction, proposed tool arguments and authoritative denial. Include an indirect variant in an image OCR result and a direct variant in a user prompt, as they cross different boundaries. Revoke the compromised source revision and verify that its indexed copies are removed; then rerun both cases to ensure defenses do not block legitimate supplier text merely because it mentions security. Assign the source, application and tool owners to separate mitigation steps so a single generic ‘prompt injection fixed’ label does not hide a persistent authorization defect.
Include cost abuse and denial-of-service paths
AI endpoints can incur significant usage from large inputs, long responses, repeated retries and agent loops. Model quotas are not a complete abuse-control system for a multi-tenant product. Define per-user and per-task limits, backpressure, maximum tool iterations and durable queue behavior. Attackers or buggy clients can exhaust shared capacity while staying within a provider’s broad subscription allowance. A threat model should identify which resources can be monopolized and what legitimate users will experience when limits are reached.
Test throttle handling and cancellation. A request that times out may continue running and consuming tokens or may already have executed a side effect. Repeated client retries can compound costs. Monitor by tenant and task type, but avoid exposing full prompts or private data to cost dashboards. Economic abuse and reliability failures are security-relevant when they prevent essential workloads from functioning.
Review observability and supply-chain exposure
Trace systems, evaluation datasets, SDK dependencies and development notebooks can hold sensitive content or credentials. Enforce access and retention rules on diagnostic data, not only production databases. A prompt artifact committed to source control may contain internal system details that help attackers if leaked. A dependency update can change default logging or authentication behavior. Include these artifacts in code and security review, with a clear inventory of owners and supported versions.
Watch model, prompt, index and tool configuration changes. A new tool added for a legitimate feature may expand the attack surface more than a new model version does. Preserve release manifests and test evidence so a security incident can be linked to a specific configuration. Do not rely on a previous signoff when the use case has expanded from internal drafting to autonomous transactions.
Convert the model into release and incident controls
Prioritize threats by likelihood, impact and the strength of current controls. Assign actions to named owners and require evidence that mitigations work, such as unauthorized-user tests, prompt-injection exercises, secret rotation drills and safe failover. Record residual risks and approval decisions. Automated scans are useful inputs, but they do not replace independent domain assessment for high-consequence workflows.
Maintain incident runbooks for malicious source removal, permission compromise, model prompt rollback, unsafe tool execution and runaway usage. Threat modeling succeeds when the application can block prohibited paths, explain detected failures and recover without silently overriding user or organizational authority. The exercise should evolve with each new data source, tool or deployment type rather than remain a diagram approved once and forgotten.