Microsoft AI-103: Selecting Models for Real Azure Workloads
The largest model in a catalog is not automatically the right one for an application. A support assistant may need fast, inexpensive grounding; a document workflow may need consistent structured extraction; and a tool-using agent may require reliable parameter selection more than elaborate prose. Microsoft Foundry makes several model families and deployment options available, but selection is constrained by region, quota, cost, latency, data governance, and observed task quality. Engineers should make those trade-offs with representative tests, not marketing rankings. A defensible model decision states which outcomes matter, what the model must never do, and how the team will know when it needs to change.
On this page
- Define the workload before comparing models
- Separate eligibility constraints from preferences
- Build a task-specific evaluation set
- Measure latency as a user experience
- Compare cost per successful outcome
- Test deployment behavior under pressure
- Use routing only when it earns its complexity
- Reevaluate after every material change
- Select the model that fits the application
Define the workload before comparing models
Start with measurable application contracts rather than a model leaderboard. For a policy assistant, verify that the answer cites the right document and refuses to invent missing policy. For a claims extractor, compare typed fields against labeled source records and require review for conflicting amounts. For a tool-using agent, test whether it selects the permitted operation, supplies valid arguments and stops at the authorization boundary. In a Microsoft AI-103 practice project, compare candidate models on the same representative tasks: correctness, latency, data restrictions, operating cost and recoverability. Choose a larger model only when the evidence justifies its added complexity and expense.
Break the workload into subtypes instead of assuming one score captures the entire application. Some requests are short and factual; others require multiple retrieved documents or complex reasoning. Include examples at both extremes. Identify what constitutes a hard failure, such as citing a private document to an unauthorized user, inventing a financial figure, or invoking the wrong mutation tool. If tasks have different consequences, evaluate them separately. A model that slightly improves average chat quality may still be unsuitable if it performs worse on the application’s highest-risk decisions.
Separate eligibility constraints from preferences
Before running a benchmark, filter models that cannot satisfy the project’s operating requirements. Check the regions and deployment types currently supported, data-handling obligations, input/output modalities, context limits, tool or structured-output compatibility, and any relevant licensing requirements. A model may perform very well in a public evaluation but be unavailable under the subscription, networking, or geographic controls the organization requires. Those constraints are not subjective rankings; they determine whether the model is even a candidate for the workload.
Keep a dated record of those constraints because the Foundry catalog and deployment capabilities change. Do not copy an old model availability table into a permanent specification. Verify current service documentation and the actual subscription deployment options before a release. If a capability is preview-only, record that state and its implications rather than presenting it as production-ready everywhere. A transparent eligibility filter prevents teams from spending weeks tuning an otherwise impressive model that cannot be deployed in their compliance environment.
Build a task-specific evaluation set
Create a representative dataset from sanitized real workflows and carefully chosen edge cases. A customer-support corpus should include vague requests, conflicting documents, missing evidence, unsupported languages, and difficult but legitimate user questions. A coding workflow should include tests with executable outcomes instead of relying only on subjective judgments. A tool-using agent needs expected tool selection, permitted arguments, and a safe failure path when the user lacks authority. Hold out examples from model tuning so the final comparison still provides useful independent evidence.
Use deterministic checks where possible: schema validity, exact field matching, calculations, and correct API arguments. Add rubric-based review for qualities such as completeness or clarity, but understand that evaluators are imperfect. When a model-based grader and a domain expert disagree, inspect the individual case instead of smoothing it into an average. Pair quality scores with error categories and case severity. This reveals whether a candidate model makes unacceptable rare mistakes even when its overall benchmark looks favorable.
Measure latency as a user experience
Total response time is not one number in every application. Streaming assistants benefit from low time to first token, while batch document analysis may prioritize total throughput. An agent may spend more time waiting for search or tools than on generation. Measure each stage in representative traces before attributing every delay to the model. Include long-input and long-output cases because response length and context size can affect performance substantially. Benchmark concurrent requests as well as one isolated prompt, using actual deployment settings.
Tail latency often matters more than averages. A support agent that is usually quick but occasionally stalls for a minute can damage trust even when median latency looks excellent. Include throttling, tool timeouts, retry policies, and queue time in the test plan. If a candidate model reduces generation time but needs twice as many calls to finish a task, the end-user experience may not improve. Compare complete task durations from input through any necessary action validation and final response.
Compare cost per successful outcome
Token prices are useful inputs, but they are not the whole bill. A task may involve retries, repeated retrieval, multiple model calls, embeddings, monitoring, and downstream tools. Estimate cost per successful user outcome rather than per isolated thousand tokens. A cheaper model that frequently fails schema validation or requires corrective passes may cost more after retries. An expensive model may be justified for rare complicated tasks but wasteful for ordinary classification. Segment the workload so the economics of each task type are visible.
Model context usage is another substantial variable. Retrieval systems that pass unnecessary documents inflate input tokens and can introduce conflicting evidence. Optimize the application’s retrieval and prompt construction before attributing every cost problem to model selection. Test shorter, focused context with enough evidence to preserve answer quality. Include human-review costs for outputs that cannot be trusted automatically. The sensible comparison is the total expense of an accurate, compliant workflow, not the lowest line in a model price table.
Test deployment behavior under pressure
A model’s practical value depends on how it behaves within the chosen Azure deployment type and quota arrangement. Standard and provisioned options can differ in capacity planning, pricing, and operating expectations. Check current support for the exact model, version, and region rather than assuming a setting from another deployment. Run concurrency tests with realistic token distributions and respect server-provided throttling guidance. The application’s own backpressure and queueing policies determine whether a temporary 429 error becomes a controlled delay or a cascade of retries.
Consider failure recovery, too. What happens when a deployment is unavailable, an API version changes, or a service dependency slows down? A fallback to another model might preserve a limited informational task but be unacceptable for a compliance-sensitive decision with stricter validated behavior. If fallbacks are supported, define which task classes may use them and how the difference is exposed to users. Keep model version and routing decisions in traces so an incident can be correlated with the exact deployment that handled the request.
Use routing only when it earns its complexity
A multi-model architecture can send simple requests to a lower-cost model and complex ones to a stronger model. But it introduces another classifier, more releases to monitor, and new failure paths. First establish a baseline with one candidate. Then test whether routing meaningfully lowers cost or latency without creating errors at the task boundary. The router should use signals that can be evaluated, such as document type, requested action, context needs, and risk class, not unexplained keyword guesses that are hard to audit.
Do not let a router silently move sensitive data across incompatible deployment boundaries. Model eligibility rules must apply before cost optimization, and the selected model must support the required tools or structured output. A request to execute an administrative operation deserves a stricter route and validation than a request to rewrite a public sentence. Record routing decisions for monitoring and reevaluate them as traffic changes. Complexity is worthwhile only when its measured benefits outweigh additional operational and governance work.
Reevaluate after every material change
Model selection is not a one-time procurement decision. The application may gain tools, change its retrieval index, expand into new languages, or encounter a new distribution of user tasks. A prompt revision can affect one model more than another. Re-run the fixed evaluation suite when behavior-changing assets move, and supplement it with production observations that reveal previously unseen cases. Use controlled canary rollouts where possible, comparing task completion, groundedness, latency, cost, and safety rather than only infrastructure success rates.
Version the model deployment together with prompts, retrieval settings, agent tools, and evaluators. Otherwise a team may attribute an improvement to a model upgrade that was actually caused by better document indexing. Maintain rollback criteria, especially for high-risk workflows. A release should be explainable with evidence: what changed, which tasks improved, which became worse, and which uncertainties remain. The ability to reconsider a decision is part of a mature model-selection process.
Select the model that fits the application
The final recommendation should name the workload and its constraints rather than declare one model universally best. State why the chosen deployment met quality thresholds, what its measured costs and latency looked like, and which scenarios require a different route or human intervention. Keep results at the case level so engineers can investigate unexpected failures. Retain the evidence that ruled out alternatives; a model excluded for regional availability today may become viable after a service change.
A responsible selection process respects the entire operating system around the model. Retrieval quality, tool authorization, evaluation design, capacity, and data governance can determine the result as much as model capability. Microsoft Foundry provides an environment in which those pieces can be built and monitored, but the engineering team must still choose with a realistic task contract. The best model is the one that reliably supports the intended outcomes within the actual constraints, not the one that wins the broadest unrelated benchmark.