Microsoft AI-300 Practice Test 29
Question 280
During a regulated Azure ML workspace infrastructure review, the following condition occurs: A Python scoring pipeline passes locally but fails on managed compute after a dependency release changes a library API because its Azure ML environment specifies unbounded package versions. The team must preserve approved access, model provenance and private network constraints. Which investigation best verifies the cause before changing production configuration?
- Inspect container image registry URI and tag, Azure ML compute managed identity, ACR pull role, firewall path and worker startup events.
- Check job identity selection, storage data-plane authorization, private endpoint reachability and storage diagnostic access-denied records.
- Inspect the data asset version resolution, pipeline input expression, snapshot timestamps, storage content hashes and previous run provenance.
- Compare environment image digest, Conda or pip dependencies, Docker build logs, locally resolved versions and job import traceback.
Explanation
Correct Answer
D:Compare environment image digest, Conda or pip dependencies, Docker build logs, locally resolved versions and job import traceback. This is the most relevant evidence for training environment lacks pinned package versions. Azure ML environment assets package execution dependencies; reproducibility requires explicit versions rather than assuming the newest package is compatible. The matching controlled remediation is: Publish a versioned Azure ML environment with vetted pinned packages and reproduce the build on the target compute before deployment.
Incorrect Answers
A:Inspect container image registry URI and tag, Azure ML compute managed identity, ACR pull role, firewall path and worker startup events. This would investigate docker environment image cannot pull from private registry rather than the documented training environment lacks pinned package versions issue. The relevant investigation is: Compare environment image digest, Conda or pip dependencies, Docker build logs, locally resolved versions and job import traceback.
B:Check job identity selection, storage data-plane authorization, private endpoint reachability and storage diagnostic access-denied records. This would investigate job identity lacks blob reader data-plane rights rather than the documented training environment lacks pinned package versions issue. The relevant investigation is: Compare environment image digest, Conda or pip dependencies, Docker build logs, locally resolved versions and job import traceback.
C:Inspect the data asset version resolution, pipeline input expression, snapshot timestamps, storage content hashes and previous run provenance. This would investigate versionless dataset reference breaks reproducibility rather than the documented training environment lacks pinned package versions issue. The relevant investigation is: Compare environment image digest, Conda or pip dependencies, Docker build logs, locally resolved versions and job import traceback.
Question 281
While preparing a secure Foundry model or agent release, the following condition occurs: An agent receives a newly released model under full production traffic before anyone verifies tool calling and safety regressions. The remediation must meet enterprise access and reliability requirements without a broad bypass. Select TWO: the strongest initial evidence and the aligned controlled correction.
- Inspect endpoint authentication modes, deployment authorization, production managed identity, assigned roles and failed token exchanges.
- Inspect deployment routing weights, evaluated model versions, incident timing, canary quality dashboards and rollback path.
- Compare available candidate models on representative multilingual datasets and select one that meets the target-language quality and security gates.
- Route a controlled percentage of requests to the candidate and promote only after automated quality, safety and latency gates pass.
- Validate quota in a preflight deployment stage and reduce or obtain approved capacity before creating the intended deployment.
Explanation
Correct Answers
B:Inspect deployment routing weights, evaluated model versions, incident timing, canary quality dashboards and rollback path. This is the most relevant evidence for new foundation model replaces production without canary. Gradual rollout with a known-good model mitigates production blast radius when foundation model behavior changes. The matching controlled remediation is: Route a controlled percentage of requests to the candidate and promote only after automated quality, safety and latency gates pass. This matters here because while preparing a secure Foundry model or agent release, the following condition occurs: An agent receives a newly….
D:Route a controlled percentage of requests to the candidate and promote only after automated quality, safety and latency gates pass. This targets new foundation model replaces production without canary. Gradual rollout with a known-good model mitigates production blast radius when foundation model behavior changes. Verify the failure through: Inspect deployment routing weights, evaluated model versions, incident timing, canary quality dashboards and rollback path. Validate success with: Failed canary metrics restore 100% of traffic to the last approved deployment without changing unrelated agent code. This matters here because while preparing a secure Foundry model or agent release, the following condition occurs: An agent receives a newly….
Incorrect Answers
A:Inspect endpoint authentication modes, deployment authorization, production managed identity, assigned roles and failed token exchanges. This is associated with serverless model endpoint unavailable to chosen identity, not the essential evidence-and-fix pair for new foundation model replaces production without canary. First examine: Inspect deployment routing weights, evaluated model versions, incident timing, canary quality dashboards and rollback path. Then correct: Route a controlled percentage of requests to the candidate and promote only after automated quality, safety and latency gates pass. This matters here because while preparing a secure Foundry model or agent release, the following condition occurs: An agent receives a newly….
C:Compare available candidate models on representative multilingual datasets and select one that meets the target-language quality and security gates. This is associated with multilingual model selection fails target-language quality, not the essential evidence-and-fix pair for new foundation model replaces production without canary. First examine: Inspect deployment routing weights, evaluated model versions, incident timing, canary quality dashboards and rollback path. Then correct: Route a controlled percentage of requests to the candidate and promote only after automated quality, safety and latency gates pass.
E:Validate quota in a preflight deployment stage and reduce or obtain approved capacity before creating the intended deployment. This is associated with azure cli foundry setup assumes existing model quota, not the essential evidence-and-fix pair for new foundation model replaces production without canary. First examine: Inspect deployment routing weights, evaluated model versions, incident timing, canary quality dashboards and rollback path. Then correct: Route a controlled percentage of requests to the candidate and promote only after automated quality, safety and latency gates pass.
Question 282
While deploying reproducible Azure Machine Learning infrastructure, the following condition occurs: A central model team registers a reusable component in a shared Azure ML registry, but application workspace users cannot resolve the component because registry access was never granted. The change must remain limited to authorized workspaces and avoid unreviewed production impact. Which TWO complementary choices identify the issue and correct it? Select TWO.
- Configure GitHub OIDC federated identity with a scoped Azure service principal and replace the long-lived secret in the deployment workflow.
- Replace embedded workspace-specific paths with parameterized, typed data inputs and bind them explicitly in each consuming pipeline.
- Grant controlled registry consumption rights and use the fully qualified versioned registry asset reference in the target workspace.
- Check the component's registry name and version, target workspace identity, registry read permissions and referenced environment dependencies.
- Inspect the data asset version resolution, pipeline input expression, snapshot timestamps, storage content hashes and previous run provenance.
Explanation
Correct Answers
C:Grant controlled registry consumption rights and use the fully qualified versioned registry asset reference in the target workspace. This targets registry model asset not available to second workspace. Cross-workspace registries support reusing vetted assets but do not automatically authorize every workspace or resolve unknown versions. Verify the failure through: Check the component's registry name and version, target workspace identity, registry read permissions and referenced environment dependencies. Validate success with: Authorized workspaces run the shared component while unauthorized subscriptions remain blocked from the registry.
D:Check the component's registry name and version, target workspace identity, registry read permissions and referenced environment dependencies. This is the most relevant evidence for registry model asset not available to second workspace. Cross-workspace registries support reusing vetted assets but do not automatically authorize every workspace or resolve unknown versions. The matching controlled remediation is: Grant controlled registry consumption rights and use the fully qualified versioned registry asset reference in the target workspace.
Incorrect Answers
A:Configure GitHub OIDC federated identity with a scoped Azure service principal and replace the long-lived secret in the deployment workflow. This is associated with github action uses persistent azure client secret, not the essential evidence-and-fix pair for registry model asset not available to second workspace. First examine: Check the component's registry name and version, target workspace identity, registry read permissions and referenced environment dependencies. Then correct: Grant controlled registry consumption rights and use the fully qualified versioned registry asset reference in the target workspace.
B:Replace embedded workspace-specific paths with parameterized, typed data inputs and bind them explicitly in each consuming pipeline. This is associated with reusable component hardcodes workspace-specific asset path, not the essential evidence-and-fix pair for registry model asset not available to second workspace. First examine: Check the component's registry name and version, target workspace identity, registry read permissions and referenced environment dependencies. Then correct: Grant controlled registry consumption rights and use the fully qualified versioned registry asset reference in the target workspace.
E:Inspect the data asset version resolution, pipeline input expression, snapshot timestamps, storage content hashes and previous run provenance. This is associated with versionless dataset reference breaks reproducibility, not the essential evidence-and-fix pair for registry model asset not available to second workspace. First examine: Check the component's registry name and version, target workspace identity, registry read permissions and referenced environment dependencies. Then correct: Grant controlled registry consumption rights and use the fully qualified versioned registry asset reference in the target workspace.
Question 283
As an MLOps engineer reviews a model training or inference pipeline, the following condition occurs: A repeated model-training command yields unpredictably different evaluation results because its random seed and dataset split are not fixed or logged across Azure ML runs. The resolution must preserve quality gates, documented artifacts and rollback options. Which acceptance test shows the AI operations defect is corrected?
- The promotion report presents comparable metrics with documented confidence and class distribution before model approval.
- Repeat runs show explainable variance with recorded seeds and identical inputs where deterministic execution is supported.
- Recreating a production deployment resolves the same approved model artifact until an authorized promotion changes the version.
- The approved run contains a loadable model artifact and the registration stage resolves its exact run and model path.
Explanation
Correct Answer
B:Repeat runs show explainable variance with recorded seeds and identical inputs where deterministic execution is supported. ML model reproducibility depends on data, environment and randomness control; a single identical command string does not fix every stochastic dependency. a direct corrective action is to implement feasible deterministic seeding and record input data split and environment versions for each run with acknowledged nondeterministic GPU constraints. This matters here because as an MLOps engineer reviews a model training or inference pipeline, the following condition occurs: A repeated….
Incorrect Answers
A:The promotion report presents comparable metrics with documented confidence and class distribution before model approval. That addresses a different configuration issue rather than the main requirement. Verify instead: Repeat runs show explainable variance with recorded seeds and identical inputs where deterministic execution is supported.
C:Recreating a production deployment resolves the same approved model artifact until an authorized promotion changes the version. That addresses a different configuration issue rather than the main requirement. Verify instead: Repeat runs show explainable variance with recorded seeds and identical inputs where deterministic execution is supported.
D:The approved run contains a loadable model artifact and the registration stage resolves its exact run and model path. That addresses a different configuration issue rather than the main requirement. Verify instead: Repeat runs show explainable variance with recorded seeds and identical inputs where deterministic execution is supported.
Question 284
During a Microsoft Foundry generative AI deployment review, the following condition occurs: A web front end that only calls an approved LLM deployment is assigned Foundry resource management privileges, allowing it to modify deployments and project settings. The deployment must retain governed identities, endpoint contracts and auditable release history. What should the engineer examine first to confirm the specific AI-300 workload issue?
- Inspect workflow identity claims, login failure logs, secret references, federated identity options and target resource permissions.
- Review application service principal, endpoint invocation needs, Foundry resource IAM scope and management operation audit trail.
- Review prompt variant outputs, evaluator criteria, human task labels, relevance/groundedness scores and business acceptance thresholds.
- Inspect runtime managed identity, search service RBAC and index permissions, connector authentication, audit failures and client requests.
Explanation
Correct Answer
B:Review application service principal, endpoint invocation needs, Foundry resource IAM scope and management operation audit trail. This is the most relevant evidence for foundry administrator role granted to consumer application. Application inference rights should be separated from platform administration; granting project owner privileges expands the impact of a compromised app. The matching controlled remediation is: Replace broad management roles with the minimum supported inference access permission and remove redundant elevated grants.
Incorrect Answers
A:Inspect workflow identity claims, login failure logs, secret references, federated identity options and target resource permissions. This would investigate foundry service principal credential rotated without updating pipeline rather than the documented foundry administrator role granted to consumer application issue. The relevant investigation is: Review application service principal, endpoint invocation needs, Foundry resource IAM scope and management operation audit trail.
C:Review prompt variant outputs, evaluator criteria, human task labels, relevance/groundedness scores and business acceptance thresholds. This would investigate prompt variant comparison measures fluency not task success rather than the documented foundry administrator role granted to consumer application issue. The relevant investigation is: Review application service principal, endpoint invocation needs, Foundry resource IAM scope and management operation audit trail.
D:Inspect runtime managed identity, search service RBAC and index permissions, connector authentication, audit failures and client requests. This would investigate agent runtime managed identity cannot read retrieval data rather than the documented foundry administrator role granted to consumer application issue. The relevant investigation is: Review application service principal, endpoint invocation needs, Foundry resource IAM scope and management operation audit trail.
Question 285
During a production Azure Machine Learning model lifecycle incident, the following condition occurs: A churn model is registered and deployed, but online inference uses different feature lookup keys from training because the model artifact lacks the feature retrieval specification. The operator must safeguard training reproducibility and serving continuity. Which acceptance test shows the AI operations defect is corrected?
- Recreating a production deployment resolves the same approved model artifact until an authorized promotion changes the version.
- Promotion evidence reports acceptable cohort outcomes or blocks release when a disparity breaches approved fairness criteria.
- Offline validation and online scoring use equivalent feature definitions and identical customer keys without silent missing-value substitution.
- Selected candidates improve the approved fraud-detection target while documented false-positive trade-offs remain acceptable.
Explanation
Correct Answer
C:Offline validation and online scoring use equivalent feature definitions and identical customer keys without silent missing-value substitution. Model weights alone do not specify how feature values were retrieved; production consistency requires a versioned feature contract. a direct corrective action is to package and version the approved feature retrieval specification with the model and enforce the same mapping at serving time. This matters here because during a production Azure Machine Learning model lifecycle incident, the following condition occurs: A churn model….
Incorrect Answers
A:Recreating a production deployment resolves the same approved model artifact until an authorized promotion changes the version. That addresses a different configuration issue rather than the main requirement. Verify instead: Offline validation and online scoring use equivalent feature definitions and identical customer keys without silent missing-value substitution. This matters here because during a production Azure Machine Learning model lifecycle incident, the following condition occurs: A churn model….
B:Promotion evidence reports acceptable cohort outcomes or blocks release when a disparity breaches approved fairness criteria. That addresses a different configuration issue rather than the main requirement. Verify instead: Offline validation and online scoring use equivalent feature definitions and identical customer keys without silent missing-value substitution.
D:Selected candidates improve the approved fraud-detection target while documented false-positive trade-offs remain acceptable. That addresses a different configuration issue rather than the main requirement. Verify instead: Offline validation and online scoring use equivalent feature definitions and identical customer keys without silent missing-value substitution.
Question 286
During a retrieval or model-customization performance review, the following condition occurs: An enterprise assistant's new retrieval strategy raises link clicks but adds hallucinated recommendations, while the experiment reports only engagement as the success metric. The change must keep citations, privacy and previously approved behavior intact. Select TWO: the strongest initial evidence and the aligned controlled correction.
- Review A/B cohort assignment, groundedness samples, relevance labels, business success measures and unsupported-claim incidents.
- Examine question-set provenance, tuning history, held-out set separation, document version and evaluator configuration.
- Calibrate retrieval threshold and ranking settings using labeled queries to admit relevant context without overwhelming the model with noise.
- Evaluate both retrievers with relevance, factual support and business outcomes under stable randomized cohorts and explicit guardrails.
- Use a governed retrieval-augmented grounding path for volatile factual content and reserve fine-tuning for stable behavior or output-format adaptation.
Explanation
Correct Answers
A:Review A/B cohort assignment, groundedness samples, relevance labels, business success measures and unsupported-claim incidents. This is the most relevant evidence for rag a/b experiment judges only click-through not factuality. Interaction metrics do not establish answer accuracy; retrieval optimization needs labels and safety/quality measurements. The matching controlled remediation is: Evaluate both retrievers with relevance, factual support and business outcomes under stable randomized cohorts and explicit guardrails.
D:Evaluate both retrievers with relevance, factual support and business outcomes under stable randomized cohorts and explicit guardrails. This targets rag a/b experiment judges only click-through not factuality. Interaction metrics do not establish answer accuracy; retrieval optimization needs labels and safety/quality measurements. Verify the failure through: Review A/B cohort assignment, groundedness samples, relevance labels, business success measures and unsupported-claim incidents. Validate success with: The selected retrieval strategy meets groundedness and task-success gates rather than optimizing click-through in isolation.
Incorrect Answers
B:Examine question-set provenance, tuning history, held-out set separation, document version and evaluator configuration. This is associated with offline rag evaluator leaks training questions into test set, not the essential evidence-and-fix pair for rag a/b experiment judges only click-through not factuality. First examine: Review A/B cohort assignment, groundedness samples, relevance labels, business success measures and unsupported-claim incidents. Then correct: Evaluate both retrievers with relevance, factual support and business outcomes under stable randomized cohorts and explicit guardrails.
C:Calibrate retrieval threshold and ranking settings using labeled queries to admit relevant context without overwhelming the model with noise. This is associated with rag similarity threshold filters all relevant evidence, not the essential evidence-and-fix pair for rag a/b experiment judges only click-through not factuality. First examine: Review A/B cohort assignment, groundedness samples, relevance labels, business success measures and unsupported-claim incidents. Then correct: Evaluate both retrievers with relevance, factual support and business outcomes under stable randomized cohorts and explicit guardrails.
E:Use a governed retrieval-augmented grounding path for volatile factual content and reserve fine-tuning for stable behavior or output-format adaptation. This is associated with fine-tuning task selected when retrieval would better handle changing facts, not the essential evidence-and-fix pair for rag a/b experiment judges only click-through not factuality. First examine: Review A/B cohort assignment, groundedness samples, relevance labels, business success measures and unsupported-claim incidents. Then correct: Evaluate both retrievers with relevance, factual support and business outcomes under stable randomized cohorts and explicit guardrails.
Question 287
As an MLOps engineer reviews a model training or inference pipeline, the following condition occurs: An AutoML classification search reports near-perfect accuracy because the supplied training dataset includes a field recorded only after the real customer outcome is known. The resolution must preserve quality gates, documented artifacts and rollback options. Select TWO actions: one decisive diagnostic and its corresponding targeted remediation.
- Inspect run metric definitions, test cohort composition, split timestamps, class prevalence and the comparison dashboard filter.
- Add MLflow parameter and dataset-version logging to the training script and group runs under a stable named experiment.
- Remove target-leakage columns and rerun the AutoML search with an appropriate time-aware validation split and realistic available features.
- Review AutoML featurization, data collection timestamps, target leakage candidates, split strategy and training-versus-production feature availability.
- Align the published endpoint contract and client payload mapping, then deploy the reviewed scoring code and test representative requests.
Explanation
Correct Answers
C:Remove target-leakage columns and rerun the AutoML search with an appropriate time-aware validation split and realistic available features. This targets automated ml experiment uses leakage feature. Automated model search cannot correct information leakage caused by improperly constructed input data and validation boundaries. Verify the failure through: Review AutoML featurization, data collection timestamps, target leakage candidates, split strategy and training-versus-production feature availability. Validate success with: Holdout performance is credible against production-like inputs and candidate selection no longer uses post-outcome fields.
D:Review AutoML featurization, data collection timestamps, target leakage candidates, split strategy and training-versus-production feature availability. This is the most relevant evidence for automated ml experiment uses leakage feature. Automated model search cannot correct information leakage caused by improperly constructed input data and validation boundaries. The matching controlled remediation is: Remove target-leakage columns and rerun the AutoML search with an appropriate time-aware validation split and realistic available features.
Incorrect Answers
A:Inspect run metric definitions, test cohort composition, split timestamps, class prevalence and the comparison dashboard filter. This is associated with run comparison uses incompatible validation populations, not the essential evidence-and-fix pair for automated ml experiment uses leakage feature. First examine: Review AutoML featurization, data collection timestamps, target leakage candidates, split strategy and training-versus-production feature availability. Then correct: Remove target-leakage columns and rerun the AutoML search with an appropriate time-aware validation split and realistic available features.
B:Add MLflow parameter and dataset-version logging to the training script and group runs under a stable named experiment. This is associated with mlflow experiment logs omit run parameters, not the essential evidence-and-fix pair for automated ml experiment uses leakage feature. First examine: Review AutoML featurization, data collection timestamps, target leakage candidates, split strategy and training-versus-production feature availability. Then correct: Remove target-leakage columns and rerun the AutoML search with an appropriate time-aware validation split and realistic available features.
E:Align the published endpoint contract and client payload mapping, then deploy the reviewed scoring code and test representative requests. This is associated with endpoint scoring script expects wrong json schema, not the essential evidence-and-fix pair for automated ml experiment uses leakage feature. First examine: Review AutoML featurization, data collection timestamps, target leakage candidates, split strategy and training-versus-production feature availability. Then correct: Remove target-leakage columns and rerun the AutoML search with an appropriate time-aware validation split and realistic available features.
Question 288
As an AI operations team checks production agent evaluation evidence, the following condition occurs: A Foundry agent evaluation job runs against a CSV whose expected_answer column was mapped as user_query, so relevance and groundedness results are invalid. The workflow must produce measurable quality, safety and performance evidence. Which targeted Azure AI operations change best addresses the documented problem?
- Correct the test dataset mapping for query, context, response and reference fields and validate a sampled evaluation before the full run.
- Include appropriate complete conversation histories and tool events in the supported agent evaluation workflow while preserving privacy protections.
- Apply privacy-conscious representative sampling or stratified monitoring that includes error and underrepresented user cohorts.
- Make required quality/safety criteria explicit CI gates and stop deployment when evaluated scores fail the approved tolerances.
Explanation
Correct Answer
A:Correct the test dataset mapping for query, context, response and reference fields and validate a sampled evaluation before the full run. This targets genai evaluation dataset lacks expected answer column mapping. Generative evaluation depends on the semantic role of each dataset field; a syntactically valid CSV is not necessarily an evaluable response set. Verify the failure through: Inspect evaluation dataset field names, input-output column mapping, evaluator requirements, row samples and failed scoring logs. Validate success with: Evaluation scores correspond to the intended example queries and reference answers without null or swapped fields. This matters here because as an AI operations team checks production agent evaluation evidence, the following condition occurs: A Foundry….
Incorrect Answers
B:Include appropriate complete conversation histories and tool events in the supported agent evaluation workflow while preserving privacy protections. This could address multi-turn agent evaluation uses isolated last message only, but it does not solve the specified genai evaluation dataset lacks expected answer column mapping failure. The matching remediation is: Correct the test dataset mapping for query, context, response and reference fields and validate a sampled evaluation before the full run. This matters here because as an AI operations team checks production agent evaluation evidence, the following condition occurs: A Foundry….
C:Apply privacy-conscious representative sampling or stratified monitoring that includes error and underrepresented user cohorts. This could address continuous monitoring has no representative sampling coverage, but it does not solve the specified genai evaluation dataset lacks expected answer column mapping failure. The matching remediation is: Correct the test dataset mapping for query, context, response and reference fields and validate a sampled evaluation before the full run.
D:Make required quality/safety criteria explicit CI gates and stop deployment when evaluated scores fail the approved tolerances. This could address evaluation workflow fails to block degraded agent deployment, but it does not solve the specified genai evaluation dataset lacks expected answer column mapping failure. The matching remediation is: Correct the test dataset mapping for query, context, response and reference fields and validate a sampled evaluation before the full run.
Question 289
As an MLOps engineer reviews a model training or inference pipeline, the following condition occurs: A tuning experiment requests dozens of concurrent GPU trials with no bound, leaving the Azure ML workspace queued for hours under regional quota constraints. The resolution must preserve quality gates, documented artifacts and rollback options. Which observable postchange result proves the intended remediation succeeded?
- All workers join one process group and finish a representative multi-node training step without broad public inbound exposure.
- New deployment workflows cannot select the retired version while required historic audit artifacts remain available.
- Trials execute predictably without persistent quota-induced queueing, and tuning still explores the intended search space.
- Holdout performance is credible against production-like inputs and candidate selection no longer uses post-outcome fields.
Explanation
Correct Answer
C:Trials execute predictably without persistent quota-induced queueing, and tuning still explores the intended search space. Hyperparameter tuning parallelism and cloud compute quota must be planned together; more requested trials do not create physical capacity. a direct corrective action is to set justified trial concurrency and sampling/early-termination limits so the sweep fits available GPU capacity. This matters here because as an MLOps engineer reviews a model training or inference pipeline, the following condition occurs: A tuning….
Incorrect Answers
A:All workers join one process group and finish a representative multi-node training step without broad public inbound exposure. That addresses a different configuration issue rather than the main requirement. Verify instead: Trials execute predictably without persistent quota-induced queueing, and tuning still explores the intended search space.
B:New deployment workflows cannot select the retired version while required historic audit artifacts remain available. That addresses a different configuration issue rather than the main requirement. Verify instead: Trials execute predictably without persistent quota-induced queueing, and tuning still explores the intended search space.
D:Holdout performance is credible against production-like inputs and candidate selection no longer uses post-outcome fields. That addresses a different configuration issue rather than the main requirement. Verify instead: Trials execute predictably without persistent quota-induced queueing, and tuning still explores the intended search space.
Question 290
As an AI operations team checks production agent evaluation evidence, the following condition occurs: A user-facing agent becomes slow after an LLM deployment change, but the team only records total response time and cannot tell retrieval delay from model generation delay. The workflow must produce measurable quality, safety and performance evidence. What measured outcome validates the solution while preserving the stated constraints?
- The responsible stage is identified and P95 user response time improves without gratuitous changes to healthy components.
- Average tokens per successful request falls while groundedness and task success remain within accepted limits.
- A canary regression is visible independently of stable baseline traffic before expanding rollout.
- The evaluation reveals and measures failures that depend on follow-up context and the selected agent version improves those cases.
Explanation
Correct Answer
A:The responsible stage is identified and P95 user response time improves without gratuitous changes to healthy components. Latency budgets should separate agent orchestration, retrieval and generation; total duration alone cannot locate the bottleneck. a direct corrective action is to instrument stage-level latency and compare generation and retrieval spans before applying a targeted capacity or retrieval change.
Incorrect Answers
B:Average tokens per successful request falls while groundedness and task success remain within accepted limits. That addresses a different configuration issue rather than the main requirement. Verify instead: The responsible stage is identified and P95 user response time improves without gratuitous changes to healthy components.
C:A canary regression is visible independently of stable baseline traffic before expanding rollout. That addresses a different configuration issue rather than the main requirement. Verify instead: The responsible stage is identified and P95 user response time improves without gratuitous changes to healthy components.
D:The evaluation reveals and measures failures that depend on follow-up context and the selected agent version improves those cases. That addresses a different configuration issue rather than the main requirement. Verify instead: The responsible stage is identified and P95 user response time improves without gratuitous changes to healthy components.