Microsoft AI-103: From AI Prototype to Production on Azure

An AI prototype proves that a capability is possible. Production proves that the capability can be trusted repeatedly under real users, real data, real load, changing models, operational failures, and governance requirements. The gap between those two states is where many AI projects become expensive: the demo worked, but the system was never designed to be operated.

Microsoft Foundry now brings model deployment, agents, evaluation, tracing, monitoring, and governance closer together, while Azure services provide identity, networking, queues, search, storage, observability, and cost management. The production task is to turn those pieces into one controlled delivery lifecycle.

The current AI-103 scope reflects that lifecycle. It covers planning, retrieval, agents, evaluation, deployment, monitoring, and optimization rather than treating model invocation as the finish line.

Freeze the use case before scaling the architecture

A prototype can survive vague success criteria because a small team knows what it is trying to demonstrate. Production cannot. Define the user, task, allowed data, unacceptable failures, expected latency, volume, escalation path, and business outcome.

Then decide what AI should actually do. Some steps may need model judgment; others belong in deterministic code. A smaller production scope with a clear contract is easier to secure, evaluate, and support than a general assistant whose boundaries change with every conversation.

The broader lesson in production AI systems is that operational constraints eventually become product behavior.

Make identity and authorization part of the design

A prototype may use a developer credential or one broad service identity. Production should separate users, applications, agents, deployment pipelines, and administrative roles. Tool access should follow least privilege, and sensitive resources should not be reachable merely because the model can formulate the right request.

For agent systems, agent identity is a useful design boundary: the authority to reason about a resource is not the same as permission to change it.

Managed identity can remove stored secrets for many Azure-to-Azure calls, but role assignments, approval, and audit still need deliberate governance.

Turn retrieval and enterprise data into a governed subsystem

Production RAG needs freshness, permissions, chunking, metadata, indexing reliability, and retrieval evaluation. A prototype often uses a small curated document set; production encounters stale files, duplicates, access restrictions, malformed content, and multiple source systems.

Design the data path separately from generation. Preserve source identifiers, enforce access before text enters the prompt, monitor indexing failures, and evaluate retrieval quality against known questions.

Enterprise grounding should therefore be treated as a data architecture concern, not a prompt-engineering trick.

Build an evaluation gate before the first production release

Manual demo review does not scale. Create a reusable evaluation dataset with representative tasks, risky edge cases, and known failures. Measure dimensions appropriate to the system: task completion, groundedness, relevance, tool accuracy, safety, or structured-output validity.

The evaluation in evaluation datasets becomes a release gate. A model, prompt, retriever, or agent-graph change should not be promoted merely because a developer thinks it looks better.

Keep part of the test set stable so scores remain comparable over time, and add production-derived cases as new failure modes appear.

Instrument the request path before users depend on it

Production operators need to answer where time, tokens, failures, and bad answers came from. Foundry tracing can record agent actions, model calls, retrieval operations, token usage, exceptions, and timing in Application Insights using OpenTelemetry conventions.

Connect the application around the model as well. A trace should show the user request, retrieval, model response, tool calls, workflow steps, and downstream result without forcing an operator to correlate unrelated logs manually.

GenAI observability matters because the model is only one possible source of failure.

Choose capacity and cost controls from measured demand

Prototype traffic says little about production demand. Estimate normal and peak tokens, requests, concurrency, retrieval load, and agent fan-out. Decide whether standard pay-as-you-go capacity is sufficient or provisioned throughput is justified.

Set cost budgets and operational limits before traffic grows. AI cost control should include model usage, search, telemetry, fine-tuned hosting, and any external services the workflow calls.

Capacity planning also needs rollout headroom. A safe canary or blue-green release may temporarily run multiple versions.

Separate deployment from exposure

Production release should allow a candidate to be deployed before all users receive it. Shadow traffic, blue-green deployments, or canary routing create time to compare behavior under realistic conditions.

Use blue-green releases when clean rollback and parallel validation matter. Use canary releases when progressive exposure is valuable. Define promotion and rollback thresholds before changing traffic.

AI release criteria need behavioral metrics as well as error and latency thresholds because an endpoint can be healthy while answer quality regresses.

Automate the configuration that changes behavior

Prompts, model versions, tool definitions, retrieval settings, safety thresholds, environment variables, and infrastructure all affect behavior. Treat them as versioned configuration and deploy them through controlled pipelines.

Prompt management matters because a hidden prompt edit can change production behavior just as surely as a code deployment.

Infrastructure as code and CI/CD reduce environment drift, while reproducibility keeps the release evidence recoverable. Evaluation should run in the pipeline where practical so quality evidence travels with the release artifact.

Production is an operating loop, not a launch event

Once live, the system will encounter inputs the team never put in the prelaunch dataset. Monitor traces, quality metrics, latency, cost, queue backlogs, retrieval failures, and support incidents. Turn important production failures into new evaluation cases.

This creates a loop: observe, diagnose, add evidence, change the system, evaluate, release gradually, and continue monitoring. That loop is the heart of GenAIOps.

For engineers working across Azure AI engineering, the move from prototype to production is therefore a shift in responsibility. The model still matters, but the system around it determines whether users can rely on the result tomorrow as well as today.

Production readiness also needs a support model. Decide which team owns alerts after launch, what telemetry support engineers can inspect, how users report poor answers, and who can roll back a model or disable an unsafe tool. A prototype can live with its creators; a production service has to remain operable when those creators are unavailable.

Write short runbooks for failures that can be predicted: model throttling, search indexing failure, unavailable tools, permission errors, rising queue backlog, unexpected cost growth, and a bad model release. A runbook does not need to automate every decision. It should tell the responder where the evidence is, which actions are safe, and when escalation is required.

Privacy review should use the real data flow. Traces can contain prompts and tool arguments. Search indexes contain document fragments. Queue messages persist work. Evaluation datasets may preserve production-derived examples. Each layer needs explicit retention, access, and deletion rules instead of inheriting whatever default happens to exist.

Finally, define a retirement path before the service becomes critical. Models and APIs change, preview features mature or disappear, data sources move, and indexes need rebuilding. A production architecture should know how to migrate models, rebuild retrieval, rotate identities, and redeploy infrastructure. Production maturity is the ability to change safely, not the absence of change.

A production readiness review should also challenge hidden manual dependencies. If one engineer must manually upload a prompt, refresh an index, approve a model version, or repair failed work after every release, the system is not yet repeatable. Automate the normal path and document the exceptional path.

Service-level objectives should reflect the AI product rather than only infrastructure. Availability, latency, task completion, groundedness, action success, and time to recovery can all matter. The exact mix depends on the workload, but the objectives need owners and thresholds before the first serious incident.

Before launch, run a game day that intentionally breaks one dependency at a time. Disable a model deployment, fail the search indexer, revoke a tool permission, and exhaust a queue consumer. A team that can detect, explain, and recover from those failures is far closer to production readiness than one with a flawless demo and no recovery evidence.

Release ownership should survive team changes. Store architecture decisions, evaluation evidence, rollback steps, and source ownership with the project rather than in one person’s notes. A production system is healthier when a new engineer can understand why a control exists before changing it.

That shared record also makes audits and post-incident review faster because the team can connect the current deployment to the evidence that originally justified it.

That evidence should remain available after launch.