Practice Exams:

Microsoft AI-103: Prompt Injection Defenses That Hold

AI & Machine Learning

An AI assistant can be asked to summarize a document that contains an instruction to ignore the user, expose secrets or call an unauthorized tool. The malicious text may arrive through an email, search result, web page or document chunk rather than a direct user request. This is prompt injection: lower-trust content attempts to cross the boundary into instructions that control the system. Microsoft Foundry Prompt Shields can help detect user and document attacks, but detection alone does not authorize tools or replace application security. Effective defense combines clear trust boundaries, least privilege, isolated parsing and tests that confirm the agent cannot turn untrusted content into forbidden action.

On this page
  1. Map every source that can supply instructions
  2. Distinguish user attacks from document attacks
  3. Use Microsoft safety controls as defense in depth
  4. Put authorization around agent actions
  5. Treat retrieval and memory as untrusted data
  6. Build a repeatable attack and regression corpus
  7. Respond and recover when defenses fail

Map every source that can supply instructions

List the system’s legitimate instruction hierarchy and every lower-trust input it processes. Direct user prompts may request tasks within the user’s rights; retrieved documents and tool responses provide data that can include hostile text. Even internal repositories can contain attacker-controlled comments, uploaded files or stale instructions. Mark which components are allowed to issue commands, change permissions or call tools. A retrieved paragraph saying “you are now the administrator” must remain a paragraph, not become a valid privilege grant. An effective Microsoft AI-103 prompt-injection test follows lower-trust retrieved text and tool responses to see whether either crosses the instruction boundary.

Trace how source text enters the model context. Some applications concatenate untrusted snippets with high-priority instructions without a visible boundary, making it easier for the model to treat data as a directive. Use well-defined containers and metadata for external text, keep the origin visible and require the model to ground claims without obeying the source’s instructions. These prompt design choices reduce confusion but are probabilistic, so the surrounding application must enforce hard rules independently.

Distinguish user attacks from document attacks

A direct attack is delivered through a user-controlled prompt and may try to override rules or request forbidden information. An indirect attack is planted in material the model reads while performing a legitimate task. It might tell a support agent to redirect a case, instruct a data assistant to leak a conversation to an outside address, or insert a fake developer instruction inside an HTML document. The application can have a fully authorized user request and still encounter malicious instructions from the document it retrieved.

Consider a user asking an internal procurement assistant to summarize the current supplier onboarding checklist. An attacker has edited a lower-trust document so that one paragraph impersonates system instructions and tells the model to send supplier records to an external address. The document is a source of facts, not a party authorized to change the application’s instructions. Test the assistant with the malicious paragraph embedded near legitimate checklist requirements and with an unrelated benign paragraph mentioning security tests. The first must not trigger an outbound tool or change authority; the second should still be summarized accurately. A detector can help identify patterns, but the trusted application must independently validate recipient domains, caller rights and the allowed tool-action schema.

A separate direct attack asks the assistant itself to ignore higher-priority rules. A successful defense should identify whether the bad instruction came from a user prompt, a retrieved page, a tool response, a remembered note or a workflow field. That provenance determines where to fix the problem and which evidence to preserve. Include a negative test that tries to use an allowed read-only tool to retrieve another tenant’s data, because restricting write actions alone does not prevent disclosure. After an incident, revoke compromised source access, remove poisoned content from indexes and memory where applicable, then rerun the same cases against the rebuilt system. Safe behavior depends on stable trust boundaries even when detectors miss unusual wording.

Test both paths. Direct defenses should reject attempts to misuse allowed features; indirect defenses must prevent third-party text from obtaining authority. Include encoded instructions, text hidden inside markup, multi-lingual variants, false policy notices and tool-output masquerades in red-team cases. A service should not be considered secure solely because one well-known “ignore previous instructions” phrase was blocked in an example.

Use Microsoft safety controls as defense in depth

Microsoft Foundry Prompt Shields can examine user prompts and document-related inputs for known attack patterns and return detection or filtering information. Configure the supported guardrails for the exact model and agent deployment, and confirm how the service handles tool responses and retrieved documents. Some capabilities and options have version or preview restrictions. Logs should distinguish an attack detected but not filtered from one actually blocked; a detection flag is not proof that an unsafe request could not proceed.

Evaluate false positives and missed attacks against representative workload content. A security analysis document legitimately discussing prompt injection should not be mistaken for a user attempting one; conversely, a novel malicious sentence may evade a detector. Retain independent allowlists and permission checks. Never allow a safety classifier’s “safe” label to bypass authentication, approval, or validation that the business action requires.

Put authorization around agent actions

Tool execution is the highest-impact boundary. Maintain a fixed registry of approved tools, required scopes, typed arguments and action-specific permission checks. Resolve the user’s current identity and policy in trusted code when an action is requested. An untrusted document may suggest sending a report externally, but the model’s suggestion is not permission. For sensitive operations, require a human confirmation that shows the exact target and effect. Use idempotency keys and actual system readback for actions whose outcomes could be uncertain.

Limit data exfiltration opportunities. Restrict outbound domains, URL schemes, attachment targets and the amount of sensitive content a tool may send. Validate file paths and IDs; do not trust a model-generated resource identifier without checking ownership. Tools that merely query a database should have read-only credentials when possible. A prompt injection defense that relies only on the model declining a command will eventually be tested at its weakest tool boundary.

Treat retrieval and memory as untrusted data

Search indexes and agent memory can preserve malicious content beyond the original conversation. Apply source provenance, content sanitation appropriate to the format and permission-aware retrieval. Do not treat a user-uploaded instruction block as equivalent to a signed company policy. If an assistant remembers a malicious prompt as a lasting preference, later sessions may inherit the attack. Memory ingestion should reject untrusted attempts to modify operating rules and provide a correction and deletion path.

Keep retrieved text and cited sources distinguishable from application policy. When the user asks what a suspicious document says, the system may quote or analyze its instruction as data without obeying it. This distinction is important for security teams and educators whose legitimate work involves studying attacks. Refusing to display all suspicious text can make the assistant less useful; allowing the text to control tools is the actual failure to prevent.

Build a repeatable attack and regression corpus

Test with realistic protected documents, user roles and tool permissions in an isolated environment. Cases should attempt data disclosure, unauthorized API calls, policy bypass, answer contamination and subtle changes to workflow. Record which source carried the attack, how it appeared in the prompt, what the model attempted, what trusted code permitted, and the final system outcome. The right success metric is that prohibited effects were blocked even if the model generated an unwise suggestion.

Run the same suite when prompts, models, retrieval indexes, content filters or tools change. A new model may interpret delimiters differently; a new tool may expose a wider capability surface. Include legitimate queries that merely mention malicious text so defenses do not cause unnecessary refusals. Human security review should adjudicate severe cases and ensure that automated metrics are not mistaken for independent approval.

Respond and recover when defenses fail

Maintain a playbook for detected injection attempts: preserve necessary sanitized evidence, stop affected tool actions, revoke compromised credentials when required, identify poisoned sources and reconcile derived indexes or memory. Users should receive accurate failure or review statuses rather than a fabricated completion. Notify source owners if malicious text is embedded in authoritative documents, and determine whether other sessions consumed the same material before it was removed.

Prompt injection is an architectural problem because language models process instructions and data in closely related forms. The reliable defense is to minimize the authority of untrusted text and require sensitive actions to pass explicit controls outside the model. Detection improves visibility, but least privilege, provenance, safe tool contracts and repeated testing determine whether an attack can change real systems.

Continue learning

Related guides

Agent Identity in Azure AI FoundryAssign agent identities and restrict access as workloads are deployed.Building Multi-Agent Workflows on AzureCoordinate agent handoffs, shared state, permissions, and failure recovery.Managing Agent Memory on AzureSet memory retention, access, and state boundaries for agents.Designing AI Evaluation DatasetsBuild representative evaluation sets with failures, edge cases, and rubrics.

Related Posts

• Should You Pursue the Microsoft Azure AI Fundamentals Certification?

• Microsoft AI-300: Monitor Model Drift Without Chasing Noise

• Microsoft AI-300: Synthetic Data Helps Only When You Understand Its Bias

• Microsoft AI-103: Extracting Structure from Complex Documents

• Microsoft AI-900: The New Azure AI Fundamentals Path

• Azure AI Engineering

• Microsoft AB-100: Business AI Systems

• Microsoft AI-103: Selecting Models for Real Azure Workloads

• Microsoft AI-103: Evaluation Datasets That Reveal Failure

• Microsoft AI-103: Designing Agent Memory with Boundaries