Question 370
A vendor upgrades a model and a once-working form extractor begins omitting required values. What evidence should the team examine FIRST before making a decision?
Explanation
Correct Answer
Inspect evaluation baselines, model/prompt versions, sampling, error rates, cost metrics, alerts and release changes. These observations test the stated cause before the team changes a system or policy. Foundation-model updates and data drift can change outcomes; continuous monitoring must include both technical and business indicators. The most suitable approach is: Monitor deployed generative AI with reproducible evaluations, task KPIs, version records and drift or incident tracking, not only subjective user praise. Scenario: A vendor upgrades a model and a once-working form extractor begins omitting required values.
Incorrect Answers
Inspect source citations, retrieved passage relevance, factual counterexamples, model output logs and a human review threshold. These observations are relevant to Hallucination detection and source verification, but do not isolate the main question. The decisive evidence is: Inspect evaluation baselines, model/prompt versions, sampling, error rates, cost metrics, alerts and release changes. Scenario: A vendor upgrades a model and a once-working form extractor begins omitting required values.
Inspect workflow baseline, user need, data sensitivity, alternatives, process owners and acceptance outcomes. These observations are relevant to Business use case selection and measurable need, but do not isolate the main question. The decisive evidence is: Inspect evaluation baselines, model/prompt versions, sampling, error rates, cost metrics, alerts and release changes. Scenario: A vendor upgrades a model and a once-working form extractor begins omitting required values.
Compare information freshness, annotated data, supported tuning path, costs, evaluation and risk review before selecting. These observations are relevant to Fine-tuning versus retrieval and human review, but do not isolate the main question. The decisive evidence is: Inspect evaluation baselines, model/prompt versions, sampling, error rates, cost metrics, alerts and release changes. Scenario: A vendor upgrades a model and a once-working form extractor begins omitting required values.


