Model serving is the boundary where an AI capability becomes an application dependency. A notebook can tolerate manual retries and developer credentials; a production service needs a stable endpoint, controlled access, predictable scaling, version management, observability, and a plan for failures. Databricks Model Serving provides managed endpoints for real-time and batch-oriented AI and ML access, while the wider platform supports agent applications and Foundation Model APIs that can participate in the same solution.<\/p>\n
The current Generative AI Engineer<\/a> exam covers serving applications, controlling endpoint access, registering models through MLflow and Unity Catalog, using Foundation Model APIs, batch inference, and monitoring live endpoints. In Databricks GenAI<\/a>, serving should therefore be designed together with the chain, agent, data, and evaluation layers that depend on it.<\/p>\n An endpoint needs a clear request and response contract. Specify required inputs, maximum sizes, output schema, authentication model, timeout behavior, and error semantics. If the endpoint serves a chain, document which preprocessing, retrieval, and post-processing steps are part of the request path rather than treating the model as the only deployed component.<\/p>\n The planned LLM chain<\/a> design illustrates why this matters. Two endpoints can use the same base model and still behave differently because one retrieves private context, one applies a different prompt, or one validates structured output. Consumers depend on the application contract, not merely the model name.<\/p>\n Interactive applications need low-latency online inference, while large offline jobs may be better suited to batch execution. Current Databricks capabilities support managed real-time endpoints and also provide ways to invoke AI functions for data workloads. The correct choice depends on whether users are waiting for a response, how much data is processed, and how predictable the traffic is.<\/p>\n Do not force every task through an online endpoint. Periodic enrichment, backfills, or scoring of large tables can be cheaper and easier to operate as batch work. Conversely, a chat or agent application needs a service path designed for concurrent interactive requests and variable response times.<\/p>\n Larger models can improve some tasks while increasing latency and cost. Longer contexts can improve access to evidence while increasing tokens and processing time. Serving architecture should therefore be based on measured task quality and workload patterns, not on choosing the largest model available.<\/p>\n The existing serving tradeoffs<\/a> article captures the core relationship. Measure p50 and tail latency, concurrency, throughput, token usage, error rates, and task quality together. A system that is fast for one request but collapses under burst traffic is not production-ready.<\/p>\n Serving endpoints may expose powerful models, private data access, or agent capabilities. Authentication and authorization should ensure that callers can use only the endpoints and downstream resources they are entitled to access. When an application retrieves governed data, user or application identity must remain meaningful across the request path.<\/p>\n Do not place long-lived personal credentials in client-side code. Use managed application identity and server-side calls where appropriate. Unity Catalog and platform access controls can govern models and data, while the application layer enforces the user-specific policy required by the use case.<\/p>\n MLflow and Unity Catalog can participate in model and application lifecycle management. Registration should preserve the artifacts and metadata needed to understand what is being served: model or chain version, signature, dependencies, owner, evaluation evidence, and release context. A production alias or similar promotion mechanism is useful only when it points to a tested artifact.<\/p>\n The model registry<\/a> perspective is important here. Registration is not just a place to upload files. It creates a controlled handoff between experimentation, validation, and production and supports rollback when a new version performs worse.<\/p>\n A healthy endpoint process does not prove that the application is healthy. After deployment, send representative requests that exercise retrieval, prompt construction, tool access, output parsing, and authorization. Compare results with the evaluation baseline and confirm that latency and error behavior match expectations.<\/p>\n The planned MLflow evaluation<\/a> workflow can provide regression cases for release validation. Keep a small critical test set that can run during deployment, then use broader offline evaluation before promotion. Production changes should have a clear success criterion and rollback threshold.<\/p>\n Managed serving can scale resources with traffic, but autoscaling cannot remove unnecessary model calls, oversized prompts, redundant retrieval, or inefficient tool sequences. Optimize the application path before assuming more capacity is the answer. Trace where latency and cost are consumed and target the slow or expensive stage.<\/p>\n Cold-start behavior, burst patterns, and minimum capacity choices can affect user experience. Measure them with traffic that resembles production rather than with a single warm request. Capacity planning should include the expected concurrency, response-size distribution, and time-of-day pattern of the real application.<\/p>\n Serving errors can come from authentication, quotas, malformed input, model failures, downstream data sources, tool calls, or timeouts. The client should know which errors are safe to retry and which require correction. Retrying an idempotent generation request is different from retrying an agent action that may have already changed a business system.<\/p>\nDefine the serving contract before choosing capacity<\/h3>\n
Choose the right serving pattern for the workload<\/h3>\n
Model choice and endpoint capacity are linked<\/h3>\n
Access control belongs at the endpoint boundary<\/h3>\n
Register versions so deployment is reproducible<\/h3>\n
Deployments should be verified with real application traffic<\/h3>\n
Autoscaling helps with demand, not with inefficient design<\/h3>\n
Failures need clear categories and safe retries<\/h3>\n