Anthropic CCA-F Practice Test 58
Question 570
A refactor is complete, and reviewers need to compare the old and new behavior without reopening unrelated files. What evidence should the agent retain to validate the change?
- Start with narrow path discovery and expand reads only where dependency evidence requires it
- Search symbol references, then inspect the relevant implementation and tests
- Persist concise file summaries and change receipts with stable paths
- Keep focused before/after evidence and relevant regression test results
Explanation
Correct Answer
D:Capture representative before/after outputs for the changed paths and retain the focused regression test results and artifact revisions. Evidence tied to the modified implementation shows whether the refactor preserved required behavior without forcing a new whole-repository exploration. Include the exact revision and test environment so another reviewer can reproduce any observed regression without guessing which source version ran.
Incorrect Answers
A:Narrow path discovery is useful before editing, but the requested work is now a post-refactor verification receipt. Listing relevant paths does not prove behavior was preserved. Keep before/after evidence, impacted tests and their results for the specific changed interface.
B:Symbol searches locate consumers of a changed function, but the question concerns demonstrating behavioral parity after the refactor. Search results alone cannot show whether new behavior matches the accepted baseline. Preserve representative inputs, outputs and regression results.
C:File summaries help future agents navigate a repository, but they are not a substitute for test evidence of the refactor. Maintain a scoped behavior comparison and recorded regression checks so reviewers can assess what actually changed.
Question 571
A model labels a financial extraction 99% confident despite missing source pages. What should acceptance depend on?
- Original value, corrected value, reviewer identity and supporting evidence
- Independently validate the shared source and evidence before accepting the claim
- Validate the output against required evidence and send unresolved gaps for review
- Feed classified correction cases into a versioned evaluation and improvement cycle
Explanation
Correct Answer
C:A model's verbal confidence is not proof that its cited sources actually support a claim. Compare each material assertion with the referenced evidence, checking scope, version and exclusions. Human review is most valuable for the cases where several interpretations remain possible or the consequences of an incorrect claim are high, not as an uncalibrated popularity vote.
Incorrect Answers
A:A correction record supports audit and evaluation improvement. The required action is “Validate the output against required evidence and send unresolved gaps for review”; the other measure addresses a separate concern. Human review is most valuable for the cases where several interpretations remain possible or the consequences of an incorrect claim are high, not as an uncalibrated popularity vote.
B:Correlated model errors can produce unanimous wrong answers. Although useful elsewhere, it is not the direct mechanism to calibrate review against observable evidence not model confidence.
D:A correction-feedback loop helps future evaluation improve, but it cannot justify accepting this financial output today when source pages are missing. Check the required evidence for the extracted amount and escalate unsupported fields; a 99% model confidence claim does not replace those document checks.
Question 572
A classifier is highly accurate on product tags but weaker on medical recommendations. How should review policy differ?
- Measure accuracy by score bands on representative labelled cases before setting thresholds
- Feed classified correction cases into a versioned evaluation and improvement cycle
- Use stricter human review thresholds for high-consequence recommendations
- Monitor by input type and increase review while evaluating the new distribution
Explanation
Correct Answer
C:The same uncertainty can have different impacts. Calibrated escalation accounts for business risk and not just a single model score. Medical recommendations carry a higher consequence of error than ordinary product tags. Review thresholds and specialist escalation should reflect severity, not merely the classifier's overall accuracy.
Incorrect Answers
A:A numeric confidence score is only useful if its relation to observed correctness is known. That addresses a different part of the system, not the requirement to allocate review thresholds according to consequence.
B:Reviewer decisions are valuable labelled examples. This capability does not directly allocate review thresholds according to consequence. Feeding prior corrections into future evaluations is valuable, but it does not immediately set an appropriately stricter review threshold for potentially harmful medical recommendations.
D:Monitoring distribution drift helps when a classifier begins seeing unfamiliar inputs, but the policy question contrasts low-impact tags with consequential medical recommendations. Even on a familiar distribution, high-consequence advice needs stricter review and escalation thresholds than routine tagging; severity should drive the initial policy.
Question 573
A service wants to reduce manual review of routine low-risk summaries without losing quality monitoring. What approach fits?
- Feed classified correction cases into a versioned evaluation and improvement cycle
- Measure accuracy by score bands on representative labelled cases before setting thresholds
- Use audited quality thresholds plus representative sampling of auto-approved low-risk items
- Use stricter human review thresholds for high-consequence recommendations
Explanation
Correct Answer
C:Sampling monitors drift while reserving intensive review for difficult cases. Thresholds should be calibrated on held-out data. Sample accepted low-risk cases to detect drift and measure false negatives, rather than assuming that a passed confidence threshold means every automated approval is correct.
Incorrect Answers
A:Logging corrected cases can improve a model in later releases, but it does not itself define safe automatic approval of today's routine summaries. Use measured quality thresholds and sample a representative subset of auto-approved low-risk outputs so performance can be checked continuously while manual review decreases.
B:A numeric confidence score is only useful if its relation to observed correctness is known. This alternative does not establish how to reduce routine review while monitoring reliability in the stated workflow.
D:Strict human review is suitable for high-consequence recommendations, but forcing it on every low-risk summary defeats the requested reduction in routine workload. Calibrate thresholds using measured outcomes and retain representative post-approval sampling, reserving intensive review for uncertainty, elevated impact or detected drift.
Question 574
Three models agree on an answer derived from the same incorrect source. Why is consensus insufficient?
- Accept the agreed model answer as proven even when each reviewer used the same incorrect source
- Original value, corrected value, reviewer identity and supporting evidence
- Independently validate the shared source and evidence before accepting the claim
- Validate the output against required evidence and send unresolved gaps for review
Explanation
Correct Answer
C:Correlated model errors can produce unanimous wrong answers. Agreement is not independent verification when the evidence is flawed. Several models may copy the same mistaken source or learned assumption. Independent factual evidence, not voting, is needed to confirm a consequential claim.
Incorrect Answers
A:Shared source error can produce consistent but false model agreement. Evidence must be independently checked against reliable references before a consensus is treated as factual.
B:A correction record supports audit and evaluation improvement. That addresses a different part of the system, not the requirement to avoid treating model consensus as ground truth.
D:Evidence validation is necessary for any accepted claim, but this choice alone does not explain the failure of agreement across three models: all used the same flawed source. Trace the shared primary evidence and independently verify it rather than interpreting repeated model consensus as separate corroboration.
Question 575
A review policy uses arbitrary confidence thresholds with no historical outcome data. What should be done?
- Choose an arbitrary confidence cutoff without measuring real classification outcomes
- Feed classified correction cases into a versioned evaluation and improvement cycle
- Adopt the same confidence cutoff used by an unrelated team without measuring this classifier’s outcomes
- Measure accuracy by score bands on representative labelled cases before setting thresholds
Explanation
Correct Answer
D:A numeric confidence score is only useful if its relation to observed correctness is known. Calibration reveals actual risk levels. Calibrate scores against labelled cases and observe error rates by confidence band before choosing an automation threshold. An invented cutoff can reject easy cases while accepting dangerous mistakes.
Incorrect Answers
A:An uncalibrated threshold cannot predict operational error rates. Representative labeled cases must establish how confidence relates to actual correctness before review policy uses that threshold.
B:Reviewer decisions are valuable labelled examples. The decisive requirement here is to calibrate confidence thresholds against real outcomes. A correction feedback loop helps future improvement, but without calibration data it does not establish which present confidence cutoff is safe. Measure actual correctness by score band first.
C:A threshold transferred from another task may have different error prevalence and consequences. Measure accuracy and calibration in representative labeled score bands before selecting the policy cutoff; monitor its false positives and false negatives after rollout.
Question 576
Human reviewers spend equal time on trivial formatting errors and severe authorization problems. What queue design helps?
- Independently validate the shared source and evidence before accepting the claim
- Feed classified correction cases into a versioned evaluation and improvement cycle
- Original value, corrected value, reviewer identity and supporting evidence
- Triage reviews by severity, uncertainty and business impact
Explanation
Correct Answer
D:A risk-based review plan assigns deeper checking to irreversible actions, privileged tool calls, numerical claims and unresolved factual contradictions while still sampling lower-risk work. This prioritization can reduce elapsed time without pretending that a word-count or generic QA score proves technical correctness. Preserve the chosen scope and residual limits so the approval accurately describes its assurance level.
Incorrect Answers
A:Primary-source verification strengthens factual reports, but the reviewer queue mixes minor formatting flaws with serious authorization defects. The immediate fix is severity-based triage using business impact and uncertainty, prioritizing harmful access-control problems. Source checking remains valuable within that ordered review, not a substitute for prioritization.
B:Reviewer decisions are valuable labelled examples. Although useful elsewhere, it is not the direct mechanism to prioritize human review by risk rather than arrival order.
C:A correction record supports audit and evaluation improvement. This alternative does not establish how to prioritize human review by risk rather than arrival order in the stated workflow.
Question 577
A human reviewer corrects an extracted address, but the system never records the correction reason. What should be retained?
- Original value, corrected value, reviewer identity and supporting evidence
- Feed classified correction cases into a versioned evaluation and improvement cycle
- Use stricter human review thresholds for high-consequence recommendations
- Triage reviews by severity, uncertainty and business impact
Explanation
Correct Answer
A:A correction record supports audit and evaluation improvement. Without it, the model cannot learn which failure pattern required intervention. Record both values, the authorized reviewer and the evidence establishing the correction. The audit needs to distinguish a supported address repair from a silent alteration of customer data.
Incorrect Answers
B:Reviewer decisions are valuable labelled examples. The required action is “Original value, corrected value, reviewer identity and supporting evidence”; the other measure addresses a separate concern. The audit needs to distinguish a supported address repair from a silent alteration of customer data.
C:A stricter approval threshold for consequential medical recommendations is unrelated to the missing audit trail for a corrected address. This case needs the original and corrected values, the reviewer, timestamp and supporting account or document evidence, so future decisions can explain the change and replay it safely.
D:Proportionate prioritization focuses human effort on consequential failures. This capability does not directly record accountable human correction decisions. General review triage does not preserve who corrected the address and why. Auditability needs before-and-after values, reviewer identity and supporting evidence.
Question 578
A classifier trained on typed forms begins processing handwritten scans and error rates rise. What should operations do?
- Monitor by input type and increase review while evaluating the new distribution
- Validate the output against required evidence and send unresolved gaps for review
- Independently validate the shared source and evidence before accepting the claim
- Use stricter human review thresholds for high-consequence recommendations
Explanation
Correct Answer
A:Aggregate accuracy can hide failures in newly introduced cases. Slice-level monitoring detects drift and justifies revised review thresholds. Handwritten scans form a changed input distribution that can invalidate evaluation results from typed forms. Monitor error rates by input class and add representative samples to the review set.
Incorrect Answers
B:A fresh evidence check is useful when an assertion is poorly sourced, but the classifier's error rate has increased because handwritten scans differ from its typed-form evaluation set. Monitor accuracy by input type, increase manual review for the shifted population and test model improvements on representative handwriting.
C:Correlated model errors can produce unanimous wrong answers. This alternative does not establish how to adapt review to changing input distributions in the stated workflow.
D:The same uncertainty can have different impacts. The decisive requirement here is to adapt review to changing input distributions. An appropriate safety threshold matters, but handwritten scans introduce a new input distribution whose error rate must be monitored and evaluated rather than assuming the prior model still performs the same.
Question 579
A human-review team fixes the same extraction mistake each week, but quality never improves. What should be added?
- Original value, corrected value, reviewer identity and supporting evidence
- Use stricter human review thresholds for high-consequence recommendations
- Use audited quality thresholds plus representative sampling of auto-approved low-risk items
- Feed classified correction cases into a versioned evaluation and improvement cycle
Explanation
Correct Answer
D:Reviewer decisions are valuable labelled examples. A systematic feedback loop turns repeated failures into measurable changes. Convert recurring human corrections into versioned regression tests and evaluate the next model or rule change against them. Repeating manual repairs without feedback leaves the underlying defect intact.
Incorrect Answers
A:A correction record supports audit and evaluation improvement. Although useful elsewhere, it is not the direct mechanism to use human corrections to drive continuous evaluation.
B:Escalating medical recommendations under stricter thresholds is a risk-control policy, but it will not address repeated extraction mistakes that reviewers fix every week. Capture correction categories and evidence in a versioned evaluation set, then measure whether a documented model or workflow change reduces those recurring errors.
C:Sampling monitors drift while reserving intensive review for difficult cases. The direct remedy is “Feed classified correction cases into a versioned evaluation and improvement cycle”, which the alternative does not itself accomplish. Convert recurring human corrections into versioned regression tests and evaluate the next model or rule change against them.
Question 580
An agent says 'legal approved' based on a chat message with no attributable approver. What should release require?
- A verifiable approval tied to the correct reviewer and exact artifact version
- Original value, corrected value, reviewer identity and supporting evidence
- Independently validate the shared source and evidence before accepting the claim
- Validate the output against required evidence and send unresolved gaps for review
Explanation
Correct Answer
A:A claimed approval must be tied to the actual authority, proposed action, scope and version before the executor performs a protected operation. A conversational assertion such as 'the manager approved' is not an authenticated authorization record. Require a verifiable approval token or workflow receipt and recheck it against the exact resource being modified.
Incorrect Answers
B:A correction record supports audit and evaluation improvement. That addresses a different part of the system, not the requirement to confirm human review authority before sensitive release.
C:Checking underlying evidence can improve a claimed legal review, but the release decision also requires proof that the designated approver actually authorized this exact artifact. A chat sentence with no attributable actor does not establish approval. Require a verifiable reviewer identity, decision and version-specific authorization record.
D:Revalidating factual evidence helps assess the report content, but does not prove the claimed legal approver had authority or reviewed the released revision. Verify the identity, role, approval decision and artifact hash or version. A source citation cannot substitute for a recorded authorization event.