Microsoft AI-103: Tracing Azure AI Agents End to End
An agent may return the right final sentence after taking a wrong route through retrieval, tools or external state. It can also fail visibly because a search index was stale even when the model itself followed instructions correctly. Without a trace of intermediate steps, teams are left to guess which component caused the result. Microsoft Foundry observability supports agent tracing and evaluation, but useful diagnostics require a deliberate schema for request identity, evidence, tool calls, approvals and timing. Tracing should reveal enough to explain failures without copying sensitive user content into widely accessible logs. The goal is a faithful operational record of what the agent actually did.
On this page
- Identify the spans that matter to users
- Capture versions and safe provenance
- Distinguish inference latency from workflow latency
- Observe retrieval and grounding as separate stages
- Test tool outcomes and approval boundaries
- Connect tracing to repeatable evaluation
- Make traces useful during incident response
Identify the spans that matter to users
Start with the task’s causal chain: user request, authentication, retrieval planning, search queries, selected source documents, model inference, proposed tool actions, validation, executed tools and final response. Represent each important operation as a trace or span with stable correlation identifiers and its parent relationship. A single request might contain many model calls, but a raw list of calls does not explain which evidence and decisions produced the final outcome. A Microsoft AI-103 agent trace is useful when it connects a user request to retrieval, decisions, tool calls, and the final outcome without losing access controls.
Distinguish events that occurred from the model’s claims about them. A model may say that a ticket was opened, but the trace should show the actual tool request, permission check, service response and authoritative ticket identifier. Record retries and timeouts so an operator can determine whether a side effect may have been committed before a failure. Traces must make uncertain outcomes visible rather than smoothing them into a misleading success label.
Capture versions and safe provenance
Attach model deployment name and version, prompt revision, agent configuration, tool schema version, search index revision and source IDs to the relevant spans. These values allow engineers to compare two apparently identical user requests that behaved differently after a release. If a retrieved policy was obsolete, a source ID and effective date can expose the problem without requiring every trace to store the entire policy text.
Apply privacy and retention rules at collection time. Full prompts may include personal or confidential data, and retrieved passages can carry third-party material with separate access restrictions. Store sanitized identifiers or restricted references where possible, use role-based access to diagnostic systems and avoid logging secrets, tokens or complete tool credentials. AI application observability should not become a second, less-governed copy of the documents it was meant to protect.
Distinguish inference latency from workflow latency
Measure model input preparation, first-token time, output generation, retrieval time, tool waiting and application validation separately. A responsive streaming model can still produce a slow user outcome if a tool blocks for thirty seconds. Conversely, a long inference step may be acceptable for an asynchronous document job whose user receives durable progress. Report task-level p95 and p99 completion times and break them down by stage and workload category.
Instrument concurrency and retry behavior. An agent can consume unexpected tokens by repeating a search or retrying a failed tool, even when its final response appears concise. Track token and request usage per model call along with resource quotas and cost attribution. A trace can show whether an improvement in apparent latency came from genuinely shorter work or from silently omitting evidence and validation.
Observe retrieval and grounding as separate stages
For retrieval-augmented agents, capture the query strategy, authorization filter identity, selected document and chunk IDs, and ranking configuration. Do not assume the presence of a citation proves the response was grounded. The relevant passage may have been retrieved but ignored, or a cited document may not support the specific claim. Keeping evidence identity separate from model output allows reviewers to inspect what the agent actually had available when it answered.
A support agent answers a policy question but cites an obsolete document after a new revision was published. An end-to-end trace should identify the incoming question type, caller authorization scope, selected index revision, search filters, candidate document IDs, model deployment and final citation. If retrieval never surfaced the new policy, prompt tuning is unlikely to solve the problem. If the right policy arrived and the model ignored its exception, investigate the generator’s evidence use. If the agent later calls a tool that modifies a case record, preserve the proposed arguments, authorization decision and confirmed tool result as distinct spans. A final answer that promises an action does not prove the action happened.
Keep the trace useful without exposing confidential source text to every developer who can view telemetry. Record redacted identifiers, relevant versions and a governed evidence link for authorized incident investigators. Test failure-path tracing by forcing a throttled model response, a denied search permission and a tool timeout; the resulting spans should reveal which dependency failed and whether a retry was safe. Use the same span relationships to build sanitized evaluation cases so incidents improve the regression suite. Tracing supports accountable engineering only when it captures causality accurately and respects the same privacy boundaries as the underlying AI application.
Trace retrieval permission decisions without exposing confidential source content. A failed query might reflect a missing role assignment, an overly restrictive filter or a newly revoked document. These causes have different remedies. Include source ingestion version where feasible to distinguish an index freshness problem from model reasoning. Search telemetry that records only the final count of hits is rarely enough to diagnose a wrong answer.
Test tool outcomes and approval boundaries
Record proposed tool arguments, the schema validation result, authorization checks, whether human approval was required, and the confirmed external outcome. Sensitive arguments should be redacted or represented by controlled references. An agent that was denied a payment action should show a blocked proposal and no executed transaction, not a generic model error. Keep the permission context and approval expiry associated with the operation so later investigators can explain the decision.
Consider an agent asked to reschedule a service visit. Its trace shows a successful availability lookup, a proposed booking call and a client timeout. It is tempting to mark the whole task as failed and rerun it, but the scheduling service might have saved the appointment before the connection dropped. Correlate the external transaction identifier with an idempotency key and query the scheduler before deciding whether a retry is safe. Record the difference between the model’s proposal, the application’s authorization, the actual API request and the confirmed remote booking. If approval was needed, the trace should indicate which exact action was approved rather than only that a conversation contained the word ‘approved’.
Build a replayable diagnostic scenario where the first availability call succeeds, the booking write times out after server-side acceptance and a second tool attempt is blocked as a duplicate. A correct trace makes that outcome visible without printing tokens, full contact details or unrelated calendar records. A second scenario should deny a user access to a different service region and demonstrate that the agent receives a permission failure rather than the other region’s appointments. These tests reveal whether observability measures real workflow outcomes or only the model-generated story of what happened.
Use idempotency keys and transaction IDs where repeated execution could cause harm. If the external service succeeded while acknowledgment was lost, the trace may end in uncertainty until a status readback resolves the outcome. Build that reconciliation into the workflow. Tool traces are most valuable when they support a reliable distinction between “proposed,” “validated,” “executed,” “confirmed” and “failed.”
Connect tracing to repeatable evaluation
Use representative traces to discover frequent failure patterns and build sanitized test cases. A wrong answer can be classified as retrieval failure, unsupported synthesis, tool selection error, permission denial or stale source material. Some Foundry evaluation capabilities can assess existing deployed interactions or agent traces, with service-specific and preview constraints. Confirm the telemetry schema and evaluator requirements before claiming that every traced system can use a particular automated evaluation path.
Human reviewers remain important for factual disputes and severe side effects. A model-based judge may score the final answer without observing a prohibited tool attempt. Include both outcome and path-based metrics where the business risk demands them. Store evaluator version and rubric alongside the trace references. A reliable evaluation loop should diagnose the subsystem that needs correction, not merely assign a numerical grade to the agent.
Make traces useful during incident response
Design dashboards around real operator questions: which release introduced the regression, which tool failed, which source version was cited, and whether any unauthorized action occurred? Provide filtered access so on-call engineers can investigate without unrestricted visibility into every tenant’s conversations. Correlate traces with service health and business transaction logs through stable identifiers. Document the limits of trace sampling so absence of an event is not automatically interpreted as proof that nothing happened.
Effective agent tracing turns a complex AI workflow into a sequence of attributable decisions and operations. It should support rollback, user corrections, security investigations and targeted product improvements. The key is disciplined provenance and privacy-aware observability, not accumulating as many raw prompt strings as possible.