Practice Exams:

Microsoft AI-103: Scaling AI Apps Without Cost Surprises

AI & Machine Learning

An AI application may respond quickly for ten test users and collapse under a seemingly ordinary production launch. One chat request can invoke retrieval, several models, tools, retries, and a long generated answer; each part consumes a different resource and may hit a different limit. Azure model deployments also distinguish traffic quotas from the processing capacity an application can reliably use. Sustainable scale therefore begins with a workload model, not an optimistic requests-per-second estimate. Engineers need to plan token demand, concurrency, costs, backpressure, and failure responses together so rising traffic does not quietly become rising risk.

On this page
  1. Measure a completed task, not one model call
  2. Translate usage into a capacity envelope
  3. Understand quota as one boundary, not a guarantee
  4. Apply backpressure instead of cascading failures
  5. Cut context only where evidence is preserved
  6. Route work to models deliberately
  7. Cache stable work while protecting freshness
  8. Design cost controls as operating policy
  9. Validate the scaling strategy before launch

Measure a completed task, not one model call

Begin by mapping the work needed to satisfy a user request. A travel assistant might retrieve company policy, classify a request, call a route-planning tool, summarize alternatives, and generate a final explanation. Counting only the final model call understates both traffic and cost. Instrument each stage so you can identify input tokens, output tokens, retrieval operations, tool latency, retries, and processing time. Some tasks stop after one model response while others require many turns. The meaningful unit is a successful user outcome with a known resource footprint. A Microsoft AI-103 scaling plan should account for retry amplification, token volume, tool latency, concurrency, and the cost of incomplete responses.

A useful cost ledger groups requests by route, tenant, model, and feature. Track the distribution, not just the average: one support conversation with a large document can consume far more context than a short FAQ answer. Record time to first token separately from time to last token when streaming. Include search-service charges and compute used by custom tools where those are billed independently. An application that merely reduces token usage by moving work to an expensive search or orchestration step may not have improved the total cost per task.

Translate usage into a capacity envelope

Forecast traffic as arrival patterns, not one daily total. A service receiving a thousand requests per hour may experience most of them after employees begin work. Estimate peak concurrent conversations, expected turns per session, model calls per turn, and token volume by call type. Multiply those components to build a capacity envelope, then apply realistic variability. Long-running agent tasks can overlap, causing concurrent downstream model calls even when the number of human users appears modest. Back-of-the-envelope averages should always be checked against trace measurements from a representative pilot.

For example, a hundred concurrent requests each triggering three model calls do not translate into a hundred model operations. Some operations may execute in parallel and others serially. If tool calls produce retries, a brief dependency outage can turn normal load into a retry storm. Model the worst permitted fan-out, not the unlimited fan-out an agent might choose by mistake. Put explicit bounds on tool loops, output tokens, and retries. The goal is to remain within a known capacity budget even when a prompt, retrieval result, or external system behaves unexpectedly.

Understand quota as one boundary, not a guarantee

Azure model deployments use limits such as tokens per minute and requests per minute, with details that depend on model, deployment type, region, subscription, and current service documentation. A nominal quota does not promise that every burst will complete instantly. An application can also saturate its own networking, tool service, or search index before model limits become visible. Treat model quota as one part of end-to-end admission control. Build dashboards that show queue depth, concurrency, retry rate, and downstream saturation alongside deployment-level token use.

When capacity is insufficient, additional quota is not the only answer. A model change, shorter context, fewer duplicate calls, or different request scheduling may improve throughput without increasing the bill. Microsoft Foundry and Azure model offerings evolve, so do not hardcode a global token limit from an old tutorial. Verify the current quota view for the exact model and deployment type, and document which workloads use which pools. Capacity reservations and provisioned offerings introduce different assumptions from shared standard deployment types and deserve their own utilization analysis.

Apply backpressure instead of cascading failures

A production client should understand that throttling is an expected operating condition, not an exceptional mystery. Respect server-provided retry guidance where supported and use bounded exponential backoff with jitter. Most importantly, constrain the number of pending retries. Without a queue limit, a temporary 429 response can spawn many delayed attempts that flood the service as soon as it recovers. Separate interactive traffic, which needs a short response window, from asynchronous work that can tolerate queuing. Reject or defer requests explicitly when the service cannot satisfy its time budget.

A support interface can provide a graceful answer such as “The request is queued” only when a durable queue actually exists. Do not tell a user that a job will finish later if it will be lost when the web process restarts. Use idempotency identifiers for actions that can change records, and distinguish safe model-response retries from retries of purchase, booking, or ticket-update tools. Circuit breakers can stop repeated calls to a failing dependency, while a smaller fallback workflow can preserve limited functionality. These controls belong in the client and orchestration layer, not in a prompt asking the model to be careful.

Cut context only where evidence is preserved

Prompt length is one of the largest controllable costs. Retrieval pipelines often waste tokens by sending many redundant passages, entire documents, or conversation histories that no longer matter. Improve chunk relevance, deduplicate passages, and keep the document text needed to support the answer. A short context is not inherently better if it removes a policy exception or narrows a numeric range incorrectly. Measure both task quality and token savings to avoid optimizing the bill at the expense of correctness.

Conversation summarization can reduce repeated context, but a summary is a lossy transformation. Preserve explicit user instructions, active identifiers, and key constraints as structured state when possible. Keep ephemeral conversation details out of durable user-profile memory. For RAG workloads, rerank candidate evidence and select enough source context to answer the actual question. Separate prompt instructions from retrieved content and limit output length according to the task, rather than applying the same maximal-response budget everywhere.

Route work to models deliberately

Model routing can reduce cost when simple tasks use a smaller model and complex ones use a stronger one. The routing rule must itself be testable. A document classifier could identify whether a request needs precise extraction, rich reasoning, or a short reformulation; the application then selects an approved deployment with appropriate quality and compliance properties. Do not route sensitive workloads to a model solely because its nominal price is lower if its region, deployment type, retention terms, or tested accuracy do not fit the application.

Use real task-level evaluations before introducing multiple models. A two-model architecture creates more deployment versions, monitoring streams, and failure paths. Record which model handled which request and compare quality for each task class. Routing can also be adaptive: when a high-capacity model is unavailable, a less capable fallback may support an explicitly reduced feature set. Such a fallback should be presented as degraded service rather than silently substituting uncertain reasoning for a critical decision.

Cache stable work while protecting freshness

Caching can eliminate repeated token usage for genuinely stable operations. A fixed classification of an immutable public document may be cacheable for some time, while a question about a live account balance is not. The cache key must include relevant context: tenant, permissions, model or prompt version, source revision, and any language or output-format constraints. Shared caches can otherwise leak data between users or return a result based on documents the current requester is not authorized to read.

Distinguish embedding caches, retrieval caches, tool-result caches, and complete answer caches. They have different invalidation rules. An embedding can remain reusable until the underlying text or embedding model changes; a retrieved list of authorized documents may need immediate invalidation after an access-policy change. An answer referencing rapidly changing operating data deserves a short lifetime or no answer-level cache at all. Observe hit rates and incorrect-staleness incidents, not only the money saved by avoiding calls.

Design cost controls as operating policy

Use spending budgets and alerts, but recognize that a budget alert is not a hard preventive quota for every service. Application-side rate and token ceilings can protect a tenant or feature before a surprise bill appears. Set limits for maximum conversation turns, per-request output size, document ingestion volume, expensive tool calls, and concurrent long-running jobs. These are product decisions: a premium analysis workflow may receive more resources than an instant summary feature. Make those differences intentional and visible to support teams.

Cost attribution also requires an owner. A central team may operate shared Foundry projects while product teams consume the capacity. Record chargeback labels or workload identifiers consistently across models, indexes, applications, and queues. Review idle deployments and unused indexes as well as active traffic. The operational aim is not simply lower spend; it is predictable spend for a measured service outcome, with early signals when demand, pricing, or an agent’s behavior changes enough to invalidate the capacity plan.

Validate the scaling strategy before launch

Load-test with realistic mixes of short and long tasks, peak concurrency, tool latency, and failure injection. Include cases where a dependency times out while model generation is still running, a user disconnects, and a retry arrives after the original request succeeded. Measure tail latency, completion rates, budget consumption, queue time, and correctness under those conditions. If a service appears fast only because it drops requests or omits validation, the test did not prove useful capacity.

Choose a gradual rollout that allows rollback when cost per successful task rises or error rates worsen. Compare new releases to a fixed baseline by task category. Add production feedback to the model while protecting user data, and review quotas when deployment types or supported regions change. A scalable AI app is one whose worst reasonable usage patterns have defined limits and operational responses. That discipline protects both reliability and cost without pretending that cloud capacity is unlimited.

Continue learning

Related guides

Cost Control for Azure AI AppsMeasure token, tool, and retrieval costs per successful AI task.

Related Posts

• Mastering the Azure AI Engineer (AI-102) Exam

• Mastering the Basics: Your Guide to Microsoft Azure AI

• The Ultimate Overview of Microsoft AI-900: What You Should Know

• How to Use Microsoft Copilot: Your Guide to Boosting Productivity with AI

• Think Smart, Build Smarter: Mastering AI-900 with Microsoft Azure

• Microsoft AI-300: Reproducibility Is the First Test of Production ML

• Microsoft AI-300: From Experiment Tracking to an Auditable ML Lifecycle

• Microsoft AI-300: How to Roll Back a Model Without Rolling Back the App

• Microsoft AI-300: Machine Learning CI/CD Needs More Than a Build Pipeline

• Microsoft AI-103: Observability for Agentic Applications