Practice Exams:

Microsoft AI-103: Observability for Agentic Applications

AI & Machine Learning

A generative AI service can return HTTP 200 while giving a user the wrong answer. The model might have relied on an outdated retrieved passage, silently ignored an authorization error, or called a tool with valid syntax and the wrong account identifier. Ordinary web metrics show that the request completed, but not why the outcome was wrong. AI observability has to connect user intent to retrieval, model decisions, tools, and the final result without turning telemetry into a new source of leaked sensitive data. Microsoft Foundry tracing and evaluation capabilities support that work, but teams still need to decide what evidence belongs in every trace.

On this page
  1. Model the request as a causal sequence
  2. Identify the minimum useful telemetry
  3. Trace retrieval as an evidence pipeline
  4. Treat tool calls as auditable actions
  5. Connect tracing to evaluation
  6. Guard sensitive material in logs and traces
  7. Build failure categories before the first incident
  8. Debug backward from a bad answer
  9. Use observability to govern releases

Model the request as a causal sequence

Treat an AI request as a tree of related operations. A parent span represents the user task, while child spans identify retrieval, model inference, tool requests, validation, and final synthesis. Capture stable correlation identifiers so an engineer can follow a failed answer across application services rather than inspecting disconnected log lines. An agent that makes several model decisions should expose the order and duration of those steps. Without such context, a successful final request can conceal multiple transient errors, unexpected tool retries, or a retrieval failure that left the model guessing. An actionable Microsoft AI-103 trace connects retrieval, tool calls, model output and the final response as one transaction that engineers can investigate.

The trace should answer concrete questions. Which documents were selected? Which prompt and model release were used? Which tool was called, with what safe-to-log arguments? Did authorization reject an attempted action? What did the user eventually receive? Keep timing and outcome fields even if sensitive prompt contents must be redacted. A good trace does not need every token to identify a failed service boundary. It needs enough causal structure to distinguish infrastructure failure, missing evidence, model behavior, and user misunderstanding.

Identify the minimum useful telemetry

Start with request ID, timestamp, tenant or workload identifier, model deployment, prompt version, token counts, latency, retrieval count, and final outcome. Add tool name, structured error category, retry count, and downstream service ID when tools are enabled. Avoid ad hoc free-text log fields that cannot be aggregated. Record total duration and time to first token separately for streaming responses, because a user may perceive a long silent wait differently from a slow completion after the first words appear.

Not every field should be a high-cardinality metric label. Session identifiers or source document names can explode monitoring costs if used as metric dimensions. Keep high-cardinality values in traces under appropriate retention policies, and use bounded labels for dashboards. Standardize outcome categories before deploying many applications. A “retrieval_empty” event should mean the same thing across workloads, while an “authorization_denied” error should be visibly different from a service timeout. Consistent semantics reduce the time needed to diagnose incidents across teams.

Trace retrieval as an evidence pipeline

RAG observability is more than logging that Azure AI Search returned results. Record which index and search configuration were used, whether a permission filter was applied, how many candidates were retrieved, and which passages were ultimately given to the model. Track index refresh time and source document versions where feasible. An answer may look wrong because a source was never indexed, an access rule removed the right document, hybrid ranking favored another passage, or the model disregarded correct evidence. Those are four distinct classes of failure.

For a policy-assistant example, imagine that a new travel policy was published an hour earlier. If the old policy remains in the index, changing the prompt cannot repair the real defect. If both policies are indexed but the current policy ranks below an old page, ranking and metadata filters need investigation. If the current passage reaches the model but the answer cites an obsolete rule, the issue moves to answer generation or instruction design. Recording each transition enables targeted fixes rather than endless prompt changes.

Treat tool calls as auditable actions

Tools can read or mutate state, and those behaviors need different observability. For a read-only database lookup, record the target category, authorization result, service latency, and whether the response was empty or malformed. For a payment, ticket update, or configuration change, capture an idempotency or transaction identifier, approval event, affected resource reference, and final status. Never log credentials, raw access tokens, or unredacted sensitive payloads just to make debugging easier. A side-effecting tool should create its own durable audit record independent of the model trace.

A trace can show what the model requested, but it should not be treated as proof that the downstream system committed the action. The tool may have timed out after a successful write, or it may have returned success before an asynchronous process finished. Correlate model calls with authoritative system records and mark unknown outcomes explicitly. This prevents an operator from replaying a mutation based only on the absence of a success message in the model transcript.

Connect tracing to evaluation

Observability tells you what happened; evaluation tells you whether the behavior met a defined standard. Use saved traces to build regression cases for tool correctness, retrieval grounding, task completion, and risk controls. A repeated failure class might be an agent choosing a broad administration tool when a narrower lookup exists. Convert that into an explicit test scenario with a permitted action and an expected outcome. Microsoft Foundry evaluation capabilities can help score qualities such as tool-call accuracy and risk, but the underlying scenario and policy must still come from the application owner.

Do not aggregate all evaluation results into one cheerful quality figure. Separate safety, factual support, authorized action, response usefulness, and performance by task class. A service that performs well on ordinary information requests may still fail a rare but consequential money-transfer case. Keep a sampled human review process for uncertain judgments and investigate disagreement between model graders and domain experts. The trace is evidence for that review, not a replacement for it.

Guard sensitive material in logs and traces

Prompt messages, retrieved snippets, embeddings metadata, tool arguments, and agent memory may contain regulated or personal information. Decide what is permitted to leave the application boundary before enabling detailed trace capture. Apply redaction or structured allowlists to model inputs and outputs. Keep secrets entirely out of logging pipelines, and restrict trace access to operators with a demonstrated need. A generic policy to “log everything for debugging” can create a more searchable sensitive-data repository than the systems the AI application was meant to protect.

Retention is a product and compliance decision, not only an observability setting. Raw trace bodies may need shorter lifetimes than aggregate performance metrics. Tenant-aware access controls matter when many customers share a monitoring platform. A support engineer investigating a failed ticket workflow should not gain broad visibility into unrelated customers’ retrieved documents. Test the redaction logic with realistic payloads, including secrets in unexpected text fields and tool responses that contain more information than the caller requested.

Build failure categories before the first incident

Create a taxonomy that separates availability, authorization, retrieval, reasoning, generation, tool execution, and user-experience failures. A dependency timeout belongs in a different remediation queue from a hallucinated answer. So does a valid but unauthorized tool invocation. Give incidents stable categories that can be counted over time while retaining a link to the underlying trace. When the categories are documented, engineering teams can compare rates after a model or retrieval change without manually rereading thousands of conversations.

Set alerts against meaningful user outcomes rather than every low-level warning. Spikes in failed tool calls, retrieval with zero authorized results, unexpectedly long output, or unsanctioned retries can indicate trouble. Do not page an engineer for every harmless token-usage fluctuation. Combine thresholds with a baseline, request volume, and task importance. A sharp drop in completion rate for a high-impact workflow should receive attention even if overall service availability appears healthy.

Debug backward from a bad answer

A reliable incident investigation starts with the observed outcome and follows its dependencies in reverse. Identify the final answer and the source evidence it cited. Check whether those passages were present, current, and authorized. Then examine the search request, filtering, model and prompt versions, and any preceding tool results. Finally assess the infrastructure timing and errors. This order keeps the investigation focused on a specific failure while still exposing deeper causes when needed.

Preserve an anonymized reproduction case and versioned trace after resolving an incident. A fix to one prompt may break other tasks; a change to an index analyzer may alter retrieval ranking for unrelated topics. Run the reproduction alongside broader regression tests, and verify that the symptom does not recur after deployment. Where a result cannot be reproduced, document the missing telemetry rather than claiming certainty. An observability program improves when every incident reveals what evidence would have prevented guesswork.

Use observability to govern releases

Every behavior-changing asset deserves a version: models, prompts, agent definitions, tool schemas, retrieval configurations, and source indexes. Record versions at request time, then compare operational outcomes between releases. A canary deployment should be measured for grounded answers and authorized tool activity, not just latency and error rates. Establish rollback triggers before exposing more users, including safety incidents and unexplained cost increases. The ability to correlate a defect with one deployed change makes rollback a controlled engineering action rather than an emergency guess.

The strongest AI observability system connects three perspectives: infrastructure health, the agent’s consequential decisions, and the user’s actual task outcome. Each perspective can contradict the others. A healthy model endpoint can still produce an unsupported answer, and an unsafe tool request can still be blocked by a well-designed authorization layer. Trace design should make those differences visible while protecting the contents of the conversations being observed. That is what turns telemetry into reliable operational judgment.

Continue learning

Related guides

Tracing AI Agents in AzureInspect agent traces to explain tool choices and failure causes.Online Evaluation for AI SystemsMonitor real-user AI outcomes beyond a static offline benchmark.GenAIOps on AzureOperate generative AI with versioning, telemetry, evaluation, and rollback.Evaluating Agents for Accuracy and SafetyTest agent decisions, tool outcomes, safety, and task completion.

Related Posts

• Mastering the Azure AI Engineer (AI-102) Exam

• Mastering the Basics: Your Guide to Microsoft Azure AI

• The Ultimate Overview of Microsoft AI-900: What You Should Know

• Azure AI Engineering After AI-102: Practical AI-103 Learning Roadmap

• How to Use Microsoft Copilot: Your Guide to Boosting Productivity with AI

• Think Smart, Build Smarter: Mastering AI-900 with Microsoft Azure

• Microsoft AI-300: Reproducibility Is the First Test of Production ML

• Microsoft AI-300: From Experiment Tracking to an Auditable ML Lifecycle

• Microsoft AI-300: How to Roll Back a Model Without Rolling Back the App

• Microsoft AI-300: Machine Learning CI/CD Needs More Than a Build Pipeline