Practice Exams:

Microsoft AI-103: Diagnosing Hallucinations in Azure AI

AI & Machine Learning

A plausible AI answer can be wrong in several different ways. The model may invent a citation, misread a retrieved policy exception, rely on out-of-date source material, or confidently answer a question the organization has never documented. Calling every failure a hallucination hides the cause and leads teams to adjust prompts when the real defect is search, permissions, or an unsafe workflow. Azure AI applications need a structured way to identify unsupported claims, constrain the work they perform, and measure when abstention is the correct outcome. The objective is not to force every response to sound cautious; it is to make factual support and limitations observable.

On this page
  1. Classify unsupported answers before changing prompts
  2. Verify that retrieval returned the needed evidence
  3. Define an evidence-to-claim rule
  4. Train the system to abstain for the right reasons
  5. Reduce tool and workflow misrepresentation
  6. Test targeted mitigations rather than adding generic warnings
  7. Maintain a feedback and correction process

Classify unsupported answers before changing prompts

Start with a taxonomy tied to the system. An unsupported assertion occurs when the reply states something absent from the provided evidence. A source contradiction occurs when it directly conflicts with the authoritative current source. A citation error points to the wrong passage or an invented document. A task misunderstanding answers a different question. A tool fabrication claims an action was completed when no permitted tool call or confirmed result exists. These categories require different fixes and should appear separately in quality reports. A Microsoft AI-103 hallucination check should compare each consequential claim with retrievable evidence and make uncertainty visible when the source cannot support it.

Capture a minimal failure record: user intent, model/deployment version, prompt revision, retrieved document IDs and versions, selected excerpts, tool calls and results, final answer, and reviewer rationale. Exclude unnecessary sensitive content and follow retention rules. A complaint about a wrong date should be traceable to the source that contained it—or to the absence of such a source. Without evidence, the team risks optimizing response tone instead of factual behavior.

Verify that retrieval returned the needed evidence

For retrieval-augmented systems, examine candidate passages before judging the model. Did the correct document exist in the source collection? Was it indexed, current, and within the user’s authorization? Did chunking separate a condition from its exception? Did keyword or vector search miss an exact identifier? Was a relevant passage retrieved but pushed out by retrieval result reranking or context truncation? Each step can make the final model look unreliable even when it had insufficient information to succeed.

Take a policy assistant asked whether a contractor is eligible for reimbursement. One retrieved section might say that contractors may claim certain travel costs, while a later exception states that the allowance expires after a named date. If the exception passage is missing from the retrieved set, the correct fix is often improving source parsing or candidate recall. If both passages arrive but the model answers with the broad rule alone, the failure lies in evidence selection or synthesis. A fabricated policy citation is another failure category entirely. Preserve these distinctions in regression tests; otherwise engineers may add increasingly strict prompt warnings while the actual missing evidence remains outside the model’s reach.

The evaluation record should list each material claim, its supporting section and the expected answer if the evidence is absent or conflicting. A good system can say that it cannot establish current eligibility and identify what authoritative source would resolve the question. Test a changed policy revision, a revoked user permission and a user question containing an inaccurate premise. Also inspect the displayed citation URL under the test identity: an answer citing a private or withdrawn page is not fully grounded for that user. In production, count unsupported claims, unnecessary abstentions and corrected answers separately. The goal is appropriate confidence, not a model that appears certain in every reply.

Use labeled question-to-passage sets to measure search recall and ranking, and distinguish search failures from synthesis failures. A model cannot be expected to cite the correct paragraph if the retriever never supplied it. Repair source ingestion, access filters, search fields or chunk boundaries before increasing prompt complexity. Include version and effective-date labels so the model receives the applicable facts rather than a mix of drafts and obsolete guidance.

Define an evidence-to-claim rule

Ask the application to make important claims only when the available material supports them, but enforce the rule through product design and evaluation as well as prompts. For low-risk explanatory tasks, an answer can cite relevant supporting passages. For higher-risk instructions, require each critical prerequisite or constraint to map to a source and flag ambiguous evidence for review. If sources conflict, the model should explain the conflict or stop according to the business policy instead of arbitrarily selecting the most confident-sounding passage.

Treat citations as structured references to indexed documents rather than free-form text the model can invent. Use stored source identifiers, version numbers and offsets to generate links under the user’s access scope. Verify that a cited excerpt actually supports the claim attached to it. A response with many citations can still be wrong if the links are decorative or point to summaries that omit critical exceptions. Quality means traceable meaning, not a high citation count.

Train the system to abstain for the right reasons

An assistant should refuse to invent facts when evidence is unavailable, but an overcautious system that declines answerable questions is also defective. Set task-specific abstention criteria. A support bot might ask which product version the user has, a policy assistant might state that current approved guidance is missing, and an agent that lacks authorization to change a resource must stop rather than describe an imaginary successful action. Make these outcomes useful by telling users what information or approval would resolve the uncertainty.

Evaluate both false confidence and unnecessary refusal. Include cases where the correct evidence is available and cases deliberately impossible to answer. If a prompt change reduces hallucinations only by refusing everything, report that trade-off rather than calling the change a success. Use human review for ambiguous cases and record what made a decision correct. Abstention should be an intentional application behavior backed by source and authorization checks.

Reduce tool and workflow misrepresentation

A tool-using agent can hallucinate not only facts but also state. It may claim that a ticket was opened before the API returns an identifier, or infer a database update from a successful HTTP status that actually came from a proxy. Require independent confirmation of side effects, such as reading a transaction ID or authoritative status. Separate the model’s proposed action from application execution and execution readback. Preserve idempotency keys and approval state when retrying an operation whose outcome is uncertain.

Untrusted tool output can contain adversarial instructions or incorrect content. Treat retrieval documents, web results and tool responses as data subject to validation, not as new privileged instructions. A false tool result should not expand the user’s authority or grant permission to invoke additional capabilities. Security controls must hold even if the model is persuaded to summarize the injected text as a plan.

Test targeted mitigations rather than adding generic warnings

Candidate controls include better source governance, retrieval improvements, prompt constraints, response-schema validation, evidence-linked citations, confidence or uncertainty routing, and human review for high-impact outcomes. Choose a control based on the observed defect. A generic statement that “AI can make mistakes” does not fix a missing search document. More context can worsen the problem if it introduces unrelated policies or conflicting versions. Evaluate every proposed fix against the same held-out failures and normal tasks.

Do not use a separate language model as a universal fact checker without independent sources. Two models can share the same false assumption or be misled by the same retrieved text. Deterministic checks can validate numbers, dates, known identifiers, schemas and allowed tool targets; evidence review can assess claims that depend on source material. Human expertise remains necessary for contested or ambiguous policy interpretations. Measure the mitigation’s accuracy, latency and cost as a complete workflow.

Maintain a feedback and correction process

When a user reports an inaccurate answer, capture the claim and relevant evidence through a governed review channel. Assign the issue to ingestion, retrieval, prompting, application logic or a source owner, then add a sanitized regression case. If a source was wrong, correct the authoritative repository and reconcile derived chunks rather than teaching the model an isolated exception. When a model update changes behavior, compare against the prior deployment using task-specific and high-severity cases.

Monitor trends by topic, source age, permission scope and user task. A drop in one department’s answer quality can be hidden by a strong overall average. Record incidents where the system successfully abstained, since preventing a wrong action is a positive outcome. Hallucination management becomes effective when the team can explain why an unsupported claim occurred and can prove that the chosen fix improves real outcomes without creating new authorization or usability failures.

Continue learning

Related guides

Grounding Azure AI with Enterprise DataEnforce source authorization, provenance, and freshness in grounded answers.Chunking Strategies for Azure RAGChoose document boundaries that preserve evidence and retrieval relevance.Vector, Hybrid, and Semantic Search for Better GroundingChoose keyword, vector, hybrid, and reranking strategies for RAG.AI App ObservabilityTrace prompts, retrieved evidence, tool actions, and answers across AI incidents.

Related Posts

• AI-102 Retired: The Current AI-103 Azure AI Developer Roadmap

• Mastering AI-102: A Complete Preparation Resource

• Understanding the Core of AI-102 and the Azure AI Engineer Role

• Microsoft AB-100: AI Across Dynamics 365, Power Platform, and Foundry

• Microsoft AI-103: Scaling AI Apps Without Cost Surprises

• Microsoft SC-401: Protecting AI Data Beyond Traditional DLP

• Microsoft AI-900: What Changes With AI-901

• Microsoft AI-103: Blue-Green Releases for Azure AI

• Microsoft AI-103: Designing Useful Multi-Agent Workflows

• Microsoft AI-103: Event-Driven Azure AI Workflows