Microsoft AI-103: Testing Azure AI Prompts Properly
A prompt that produces one excellent answer can still fail when a user phrases the same task differently, includes conflicting context or requests an action without permission. Prompt testing must therefore measure behavior across representative questions and failure conditions, not celebrate a few handpicked demonstrations. In Microsoft Foundry applications, the test contract also depends on the selected model, retrieval index, tool schemas and data permissions. Engineers need repeatable datasets and release gates that distinguish formatting errors from unsupported claims or unsafe actions. The objective is to discover which instructions improve real work while preserving safe refusal, correct citations and compatibility with the rest of the system.
On this page
Define what the prompt is responsible for
A prompt might ask a model to summarize, classify, produce structured fields, cite retrieved material or decide which tool to propose. These are different behaviors and need different acceptance rules. An informational prompt should not be evaluated solely on JSON validity, while a transaction agent must pass authorization and side-effect tests independently of response wording. State which claims require source evidence, which outputs can vary naturally and what should happen when necessary information is missing.
Keep business rules outside model text where deterministic enforcement is required. A prompt may instruct an agent to seek approval before an email is sent, but application code must actually check the approval token and intended recipient. Tests should cover both the model’s proposed behavior and the trusted system’s executed behavior. Otherwise a model can earn a good score for saying it will be careful while still proposing an unauthorized action.
Build cases from realistic user variation
Collect authorized examples covering normal tasks, incomplete information, ambiguous language, contradictory documents, different file sizes and multiple conversational turns. Label the required facts, permitted actions and meaningful error conditions. A case such as “summarize this policy” is incomplete without the actual policy version and intended audience. Preserve enough context to know whether the correct response should include a caveat, ask for clarification or decline to answer.
Reserve a held-out set not used during daily prompt tuning. Include rare but severe failures such as confidential source exposure or irreversible tool effects. Add sanitized production incidents while retaining the previous benchmark for longitudinal comparison. A prompt that performs well on the ten examples an engineer repeatedly edited against may have simply been adapted to those examples, not improved generally.
Test the exact model and context pipeline
Run prompt comparisons with the intended model deployment, output settings, retrieval method and tool definitions. A model update may change how it interprets system instructions, follows schemas or handles refusals. Retrieval configuration can alter which context passages arrive, and token truncation can remove a crucial exception. Record the full release combination, including prompt and source versions, so a later evaluator can reproduce the result rather than guessing at an unseen dependency.
Consider a maintenance assistant asked whether a device can be restarted during a planned outage. A revised prompt may produce shorter answers in demonstration tests, but if retrieval provides the wrong outage schedule it could omit the maintenance restriction. Create two families of cases: pure prompt tests where both versions see the same correct source excerpt, and integrated tests where index freshness, permission filtering and document parsing are active. Include a valid restart request, a request outside the approved change window, and a user without rights to perform the action. For each, record the expected answer, source revision, permitted tool action and reason for refusal when appropriate. A prompt that produces elegant summaries but proposes a restart under the wrong authorization is not a successful improvement.
Do not assume that a single judge score captures those failures. Validate required identifiers and structured arguments with code, use domain reviewers for subtle exception handling, and separately track correct abstention versus unnecessary refusal. If the candidate fails an access-control case, treat it as a release blocker regardless of its average score. If it improves clarity without changing authorization, sample additional languages and unusual document formats before promotion. During an optional canary release test, avoid running side-effecting tool calls against both versions and retain a route to revert the full prompt/schema combination. Persist the prompt revision with the test receipt so operators can tell which instructions were active during a disputed result.
After release, fold verified incidents into a maintained regression set while preserving the prior holdout so progress can still be measured independently. Change only one controllable component at a time when possible; if the deployed model, retrieved documents and prompt all change together, attribute the result cautiously. The useful outcome is not simply a higher automated score but a defensible explanation of what improved, what remains uncertain and which owner accepted the risk.
Separate pure prompt tests from end-to-end application tests. A static input with known context isolates language behavior; a live RAG case also exercises ingestion, permission filtering, ranking and citation construction. Both matter. If the system cites the wrong document, investigate whether the prompt ignored correct evidence or the retriever supplied an outdated source. Changing prompt wording cannot repair every upstream data problem.
Use multiple evaluators with known limits
Deterministic checks are useful for schema compliance, required fields, allowed tool names and exact numerical invariants. Groundedness and nuanced coverage may require subject-matter review or a calibrated model-based evaluator. Record each judge’s rubric and version; a score from another model is an estimate that can be biased toward fluent writing or familiar answers. Spot-check evaluations against independently labeled cases and maintain examples of judge mistakes.
Report results by task category and severity. A small improvement in average wording quality cannot offset a new unauthorized tool call. Measure correct abstention and unnecessary refusal separately. An overcautious prompt may eliminate unsupported answers by declining everything, which is not a functional product improvement. Thresholds should express business expectations and be applied consistently to both the baseline and candidate.
The Microsoft AI-103 blueprint also calls for reflection, self-critique loops and evaluations of reasoning-oriented workflows. Treat these as testable application designs, not as permission to expose a model’s private internal chain of thought. In a document-answering agent, one implementation can draft an answer, run a separate evidence checker over the cited source passages, and revise only claims the checker identifies as unsupported. The record available to a reviewer should contain the initial answer, cited document IDs, observable checker verdicts, any requested revision, the final answer and total extra inference cost. A self-critique sentence such as ‘I am confident’ does not provide independent evidence; the verifier needs access to authoritative passages and deterministic source constraints.
Run a controlled comparison on a held-out set of twenty mixed cases: supported questions, missing-source questions, expired policies and adversarial documents. Compare direct generation against one capped verification-and-revision pass. Report unsupported-claim rate, correct abstention, newly introduced errors, latency and cost per correct outcome. Add a stop rule: if the evidence checker is uncertain, or the revised answer contradicts a trusted rule, escalate rather than looping endlessly. Microsoft’s agent evaluation guidance distinguishes whole-task results from observable process outcomes; any preview evaluator must be labeled as such. Never claim that visible explanations prove what happened in hidden model reasoning.
Include adversarial and permission cases
Test direct and indirect prompt injection, such as instructions embedded in documents or tool responses that attempt to override system policy. Include source material discussing malicious prompts legitimately, so the assistant can analyze it without obeying it or refusing every mention. Verify that trusted code still validates actions and permissions even if the model produces a convincing but forbidden tool proposal.
Use test identities with distinct access scopes. A user allowed to see one tenant’s index results should not retrieve another’s restricted source through an alternative query branch. A prompt that tells the model not to reveal secrets is insufficient if the retrieval pipeline already supplied them. Test the whole access path and the model’s treatment of data as lower-trust evidence.
Compare changes using targeted analysis
When testing prompt A versus B, hold the model and source conditions stable where possible. Examine differences in factual support, user satisfaction, output length, token cost, tool safety and latency. Collect representative failures and determine their cause instead of ranking prompts by a single composite value. If a change is intended to improve concise answers, ensure it does not omit mandatory legal or technical exceptions.
A small canary or shadow evaluation can supplement offline tests, but side-effecting tools should not run twice in response to mirrored traffic. Define stop conditions before promotion and preserve the prior prompt and dependent configuration for rollback. Document what remains untested rather than presenting every green automated metric as a universal quality guarantee.
Keep prompt tests connected to production evidence
After release, inspect corrected user reports, source changes, schema failures and unsafe tool attempts under privacy controls. Turn relevant incidents into new regression cases, then review whether the model, prompt, search or application code needs modification. Do not append a special instruction for every incident until the prompt becomes contradictory and impossible to maintain. Prefer addressing the true component responsible for the failure.
Reliable prompt testing makes AI changes explainable. The team should know which behavior a revision targeted, which cases improved, which regressed and who accepted the remaining risks. For Azure AI applications, repeatable evaluation is a product control that protects grounded answers and authorized actions, not a ritual for choosing the most impressive demonstration response.