Isaca AAIA Practice Test 73
Question 720
An independent AI auditor reviews this case: A customer service agent sends refunds without checking policy ceilings or approved transaction identities. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Define owner-led ongoing monitoring for quality, drift, safety, fairness and service impact with escalation.
- Give an AI agent full administrator rights and rely on a natural-language instruction to act carefully.
- Inspect service identities, tool-call logs, policy checks, denial tests and human approval traces.
- Limit agent authority by least-privilege tools, transaction ceilings, pre-action approval and post-action verification.
- Inspect calibration, confusion matrices, missing-input behavior, escalation queue and approval history.
Explanation
Correct Answers
C:The reason it is necessary is: Autonomous action risk is driven by actual tool permissions and decision gates, not chatbot prompt wording alone. For this situation: A customer service agent sends refunds without checking policy ceilings or approved transaction identities.
D:The reason it is necessary is: Autonomous action risk is driven by actual tool permissions and decision gates, not chatbot prompt wording alone. This response fits the risk associated with Autonomous agent action-limit supervision: Limit agent authority by least-privilege tools, transaction ceilings, pre-action approval and post-action verification.
Incorrect Answers
A:This option concerns AI post-deployment monitoring and KRI ownership, which is a related but different AI governance concern. A proportionate response is: Limit agent authority by least-privilege tools, transaction ceilings, pre-action approval and post-action verification.
B:It misses the key control principle: Autonomous action risk is driven by actual tool permissions and decision gates, not chatbot prompt wording alone.
E:This option concerns Threshold governance and safety fallbacks, which is a related but different AI governance concern. The decisive principle here is: Autonomous action risk is driven by actual tool permissions and decision gates, not chatbot prompt wording alone.
Question 721
A governance and assurance review uncovers the following: A credit model's overall false-denial rate looks acceptable but is disproportionately high for an identified demographic subgroup. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
- Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
- Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
Explanation
Correct Answer
B:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. In this case: A credit model's overall false-denial rate looks acceptable but is disproportionately high for an identified demographic subgroup.
Incorrect Answers
A:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. It does not establish control effectiveness for Subgroup fairness performance validation.
C:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. It does not establish control effectiveness for Subgroup fairness performance validation.
D:This attractive shortcut does not resolve the Subgroup fairness performance validation weakness: Infer that fairness is demonstrated by one high overall accuracy percentage.
Question 722
A risk-based audit engagement examines this operating condition: A credit model's overall false-denial rate looks acceptable but is disproportionately high for an identified demographic subgroup. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- Averages hide tail failures and resource exhaustion; service acceptability includes efficiency and availability.
- Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
- Benchmark scores can overstate generalization when test items or related records have been exposed to development.
Explanation
Correct Answer
B:Independent audit evidence includes: Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses. This is the applicable AI audit principle for Subgroup fairness performance validation: Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
Incorrect Answers
A:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. It does not establish control effectiveness for Subgroup fairness performance validation.
C:This attractive shortcut does not resolve the Subgroup fairness performance validation weakness: Infer that fairness is demonstrated by one high overall accuracy percentage. For this situation: A credit model's overall false-denial rate looks acceptable but is disproportionately high for an identified demographic subgroup.
D:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. For this situation: A credit model's overall false-denial rate looks acceptable but is disproportionately high for an identified demographic subgroup.
Question 723
An audit committee asks for a defensible assessment of this situation: A credit model's overall false-denial rate looks acceptable but is disproportionately high for an identified demographic subgroup. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect p95/p99 response, token counts, queue depth, error budgets, concurrency and billing ceilings.
- Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
- Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
Explanation
Correct Answer
B:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. This evidence directly tests the asserted control for Subgroup fairness performance validation: Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
Incorrect Answers
A:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. For this situation: A credit model's overall false-denial rate looks acceptable but is disproportionately high for an identified demographic subgroup.
C:This attractive shortcut does not resolve the Subgroup fairness performance validation weakness: Infer that fairness is demonstrated by one high overall accuracy percentage. The auditor should instead examine: Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
D:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. It does not establish control effectiveness for Subgroup fairness performance validation.
Question 724
During an AI controls assessment, the team learns that a credit model's overall false-denial rate looks acceptable but is disproportionately high for an identified demographic subgroup. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Model performance remains within approved bounds or unsafe inputs reliably route to fallback.
- Performance advantage remains on independent representative evaluation with minimal leakage.
- Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
Explanation
Correct Answer
C:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. This is an observable validation result for Subgroup fairness performance validation, not just management assurance: Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
Incorrect Answers
A:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. It does not establish control effectiveness for Subgroup fairness performance validation.
B:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. The decisive principle here is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
D:This attractive shortcut does not resolve the Subgroup fairness performance validation weakness: Infer that fairness is demonstrated by one high overall accuracy percentage. Effective closure requires: Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
Question 725
An independent AI auditor reviews this case: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
- Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
- Conduct scoped adversarial testing across inputs, retrieved context, tools, identities and post-change regressions.
Explanation
Correct Answer
A:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. In this case: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients.
Incorrect Answers
B:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. The decisive principle here is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
C:This attractive shortcut does not resolve the Subgroup fairness performance validation weakness: Infer that fairness is demonstrated by one high overall accuracy percentage. In this case: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients.
D:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. It does not establish control effectiveness for Subgroup fairness performance validation.
Question 726
A governance and assurance review uncovers the following: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- Averages hide tail failures and resource exhaustion; service acceptability includes efficiency and availability.
- Fluency and coherence are not evidence of factual correctness or fidelity to source material.
- Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
Explanation
Correct Answer
C:Independent audit evidence includes: Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses. In this case: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients.
Incorrect Answers
A:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. The decisive principle here is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
B:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. For this situation: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients.
D:This attractive shortcut does not resolve the Subgroup fairness performance validation weakness: Infer that fairness is demonstrated by one high overall accuracy percentage. For this situation: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients.
Question 727
A risk-based audit engagement examines this operating condition: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
- Review reviewer independence, held-out data provenance, challenge results, exceptions and signoffs.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
Explanation
Correct Answer
A:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. The corrective action is: Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
Incorrect Answers
B:This option concerns Independent validation before model deployment, which is a related but different AI governance concern. It does not establish control effectiveness for Subgroup fairness performance validation.
C:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. The audit must corroborate: Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
D:This attractive shortcut does not resolve the Subgroup fairness performance validation weakness: Infer that fairness is demonstrated by one high overall accuracy percentage. Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
Question 728
An audit committee asks for a defensible assessment of this situation: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
- Performance advantage remains on independent representative evaluation with minimal leakage.
- Critical attack paths are blocked by tested technical controls and regressions catch recurrence.
- Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
Explanation
Correct Answer
D:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. Independent audit evidence includes: Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
Incorrect Answers
A:For this situation: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients.
B:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. In this case: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients.
C:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. For this situation: A healthcare classifier passes average sensitivity checks although its performance falls sharply for older patients.
Question 729
During an AI controls assessment, the team learns that a recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
- Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
- Require risk-proportionate independent model validation, controlled challenge samples and release acceptance.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
Explanation
Correct Answer
B:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. This is the proportionate control response to Subgroup fairness performance validation: Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
Incorrect Answers
A:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. The decisive principle here is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
C:This option concerns Independent validation before model deployment, which is a related but different AI governance concern. The decisive principle here is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
D:This attractive shortcut does not resolve the Subgroup fairness performance validation weakness: Infer that fairness is demonstrated by one high overall accuracy percentage. In this case: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features.
Question 730
An independent AI auditor reviews this case: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
- Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
Explanation
Correct Answers
B:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. The acceptable verified outcome is: Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
E:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. This response fits the risk associated with Subgroup fairness performance validation: Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
Incorrect Answers
A:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. The decisive principle here is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
C:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. For this situation: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features.
D:This attractive shortcut does not resolve the Subgroup fairness performance validation weakness: Infer that fairness is demonstrated by one high overall accuracy percentage. For this situation: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features.
