Practice Exams:

Microsoft AI-103: Choosing Safety Controls for Azure AI

AI & Machine Learning

Turning on a content filter is not a complete safety architecture. A customer-facing AI assistant can fail by generating harmful text, following malicious instructions embedded in a retrieved file, exposing personal information, or taking an unauthorized action. Those are distinct problems with different controls and different ways to test them. Microsoft Foundry and Azure AI Content Safety provide useful detection and evaluation capabilities, but engineers must decide where each control sits in the application and what happens when it triggers. A safe design keeps authorization in deterministic systems while using model-based protections to reduce risks those systems cannot completely prevent.

On this page
  1. Classify failure modes before selecting filters
  2. Distinguish harmful content from prompt injection
  3. Apply Prompt Shields at the right trust boundary
  4. Treat groundedness as answer-quality evidence
  5. Protect private information in both directions
  6. Evaluate agent actions separately from language
  7. Calibrate thresholds against your application
  8. Build safety reviews into deployment and operations

Classify failure modes before selecting filters

Start with a threat model rather than a product feature list. Harmful user content, harmful generated content, indirect prompt injection, secrets leakage, protected material reproduction, and overpowered tool access do not share one detection rule. A public creative assistant might prioritize disallowed content and intellectual-property concerns. An internal financial agent needs stronger action approval, access control, and data-handling rules. Document the user, data sources, tools, and business consequences for each scenario, then choose controls that address those consequences. This prevents a broad “safety enabled” label from becoming a substitute for analysis. Content-safety work in Microsoft AI-103 begins by deciding which outputs must be rejected, reviewed or escalated, then testing those rules on borderline cases.

Different failure classes require different safeguards. A content classifier can flag suspicious instructions, but only server-side authorization can prevent a tool from changing a database row without permission. Groundedness evaluation may detect an unsupported response without proving that the retrieved document was accessible to the user. Test these boundaries independently: submit benign and hostile inputs, attempt a well-formed but unauthorized tool call, inspect what data leaves the system, and verify that high-impact actions need the intended approval. Treat blocked actions and false negatives as release evidence; no single moderation model can replace permissions, monitoring and escalation.

Distinguish harmful content from prompt injection

Content harm detection generally evaluates whether text or images contain categories of unsafe material. Prompt injection concerns attempts to redirect an assistant away from trusted instructions, often by placing commands in lower-trust data. A perfectly polite sentence inside a supplier document could still be an injection if it instructs an invoice agent to ignore its payment approval policy. A toxic user message may require content moderation but may not be an injection attempt. Confusing these signals makes thresholds difficult to calibrate and incident response harder to interpret.

Test the boundaries with realistic examples. Include harmful user input, generated responses that need moderation, and documents containing seemingly administrative instructions that are unrelated to the task. The agent should treat a retrieved web page as evidence, not as a new system prompt. If the page says to email customer records elsewhere, the downstream tool should refuse regardless of how the model interprets the page. Report whether the detection layer flagged the suspicious text, whether the model attempted to comply, and whether the action control blocked the attempt. Those are different measurements.

Apply Prompt Shields at the right trust boundary

Microsoft’s prompt-attack protections can help detect suspicious attempts in direct user instructions and in materials that an application retrieves or receives from tools. Apply them where the lower-trust content enters the workflow, while keeping trusted application instructions separate from documents, search results, and conversation memory. Do not flatten every source into one undifferentiated message and hope the model recognizes authority. Label document content and source provenance consistently, and use retrieval permissions so unauthorized records never reach the model in the first place.

A useful validation exercise is an employee-policy assistant. Plant a malicious paragraph in an otherwise relevant policy PDF, telling the assistant to reveal an internal key. Run the request both with and without the detection layer, but do not supply any real key. Confirm that the assistant answers the legitimate policy question without obeying the injected instruction. Also check the downstream permissions: if the model asks for a secret, the tool layer should still deny it. A defense is stronger when one failed classifier cannot turn a document into authority.

Treat groundedness as answer-quality evidence

Groundedness and factuality are related but not identical. A response may faithfully summarize an outdated document, or a source may contain an error. Groundedness checks whether an answer is supported by the material made available to the model. They do not establish that the material was current, authorized, or true. For a technical assistant, maintain source versions, effective dates, and user-permission filters independently of the response evaluator. Track unsupported claims so the development team can distinguish retrieval failures from generation failures.

RAG evaluation needs more than counting citations. An answer may cite the correct document while drawing a conclusion that its text does not support. Create examples with conflicting policies, missing evidence, and distractor passages. When evidence is insufficient, a useful output may be an explicit limitation or escalation. Use model-based evaluators as signals, review disagreements with human domain experts, and keep high-consequence factual decisions outside the assistant’s unchecked control. An impressive groundedness score should not justify bypassing a contractual or medical review process.

Protect private information in both directions

Input requests, retrieved files, generated answers, traces, and agent memory can all expose private material. Decide which data classes may be submitted to each service and which output destinations are permitted. Mask or exclude unnecessary personal identifiers before model processing when feasible. Limit log retention and restrict access to tracing systems. A detector for sensitive-data leakage is helpful, but it is not an acceptable replacement for access control and approved data flows. Sensitive information should be difficult for the agent to retrieve unless the authenticated user and task actually require it.

Consider a multi-tenant support system. A user asks about their own account, but a broad search index contains other customers’ cases. An output classifier might sometimes catch an account number from another tenant, yet the primary defect is that retrieval returned the wrong data. Enforce tenant and document-level permissions at query time, and test cross-tenant attempts explicitly. Then use output review to identify accidental leakage through summaries or free-form responses. Protecting data at the source is both safer and simpler than trying to infer authorization from generated prose.

Evaluate agent actions separately from language

When an AI system invokes tools, safety includes which tools were selected, arguments supplied, approvals obtained, and changes actually made. An agent can produce polite text while attempting a prohibited deletion. Create deterministic checks for tool allowlists, permission scopes, non-idempotent calls, and mandatory confirmations. Microsoft’s agent-focused safety evaluations can help identify prohibited actions or leakage risks in traces, but the actual systems must enforce those rules. A model judgment that an action seems reasonable cannot authorize a transfer or role assignment.

Define safe failure paths for each tool. If the model proposes a payment that lacks approval, the application should keep the request pending and explain what is required, without executing it. If a user asks to change an access policy, verify the user’s permissions before any write. Record blocked attempts for monitoring while avoiding unnecessary sensitive tool arguments in logs. These tests should remain in the release suite even after the agent’s prompt has been tuned to “always ask first.” Prompt compliance is not the final enforcement layer.

Calibrate thresholds against your application

A content detector has false positives and false negatives, and the operational cost depends on the workflow. Overly aggressive filtering can reject benign internal discussions of security incidents, while permissive settings can allow harmful output. Use an evaluation corpus representing legitimate business language, supported regions and languages, and plausible attacks. Keep category-specific results separate so you know whether a change improves one risk while worsening another. Document escalation rules for borderline cases and monitor the impact on users who need legitimate assistance.

Version filter configurations, evaluator versions, and policy thresholds with application releases. Preview evaluators or features should be identified as such and not treated as contractual production guarantees. Test latency overhead and behavior when the safety service is unavailable. A fail-closed choice may be appropriate for a high-risk action, while a low-risk informational feature might offer a constrained fallback. Make that decision deliberately in the architecture rather than allowing an exception handler to bypass safety controls by default.

Build safety reviews into deployment and operations

Before releasing a model or prompt change, run regression cases for harmful content, direct and indirect prompt attacks, unsupported answers, data leakage, and unauthorized tools. Review the failures by severity and consequence. A release that improves common helpfulness but permits a rare privilege escalation should not be approved because its aggregate score increased. Keep enough trace evidence to reproduce incidents without copying raw secrets into review artifacts. Assign an owner to each unresolved risk and document accepted limits honestly.

Production monitoring should feed new scenarios back into that test set. When a user reports an unsafe response, determine which layer failed: input screening, retrieval control, model behavior, output filtering, or action authorization. Fix the appropriate layer rather than adding another generic warning to the system prompt. Safety in Azure AI applications is a continuously tested system of boundaries and detectors. Its purpose is not to make the assistant sound cautious; it is to let the business know what the assistant can safely access, say, and do.

Continue learning

Related guides

RAG on Azure: Retrieval Quality Is the ProductMeasure retrieval precision, source eligibility, and evidence grounding.Evaluating Agents for Accuracy and SafetyTest agent decisions, tool outcomes, safety, and task completion.Content Understanding for Messy DocumentsExtract structured evidence from irregular forms, tables, and documents.

Related Posts

• Your Ultimate Guide to Crushing the Microsoft AI-102 Exam

• Is Coding Required for Microsoft Azure AI? A Simple Guide for Beginners

• Is the Microsoft Azure AI Engineer Badge Worth Your Time and Effort?

• Microsoft AI-900: Core AI Principles and Azure Exam Prep

• Microsoft AI-300: Feature Stores Solve Coordination Before Performance

• Microsoft AI-300: Serving GenAI: Balancing Latency, Throughput, and Cost

• Microsoft AI-103: Observability for Agentic Applications

• Microsoft SC-401: Protecting AI Data Beyond Traditional DLP

• Microsoft SC-100: Data Security for Copilots, Agents, and AI Services

• Microsoft AI-900: Machine Learning, Generative AI, and Agents