Isaca AAIA Practice Test 74
Question 730
An independent AI auditor reviews this case: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
- Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
Explanation
Correct Answers
B:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. The acceptable verified outcome is: Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
E:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. This response fits the risk associated with Subgroup fairness performance validation: Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
Incorrect Answers
A:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. The decisive principle here is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
C:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. For this situation: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features.
D:This attractive shortcut does not resolve the Subgroup fairness performance validation weakness: Infer that fairness is demonstrated by one high overall accuracy percentage. For this situation: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features.
Question 731
A governance and assurance review uncovers the following: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect dataset snapshots, feature pipeline commits, record counts, model registry links and training execution manifests.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
- Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
Explanation
Correct Answer
C:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. For this situation: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features.
Incorrect Answers
A:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. It does not establish control effectiveness for Subgroup fairness performance validation.
B:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. In this case: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features.
D:It misses the key control principle: Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
Question 732
A risk-based audit engagement examines this operating condition: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
- Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
- Infer that fairness is demonstrated by one high overall accuracy percentage.
- Model performance remains within approved bounds or unsafe inputs reliably route to fallback.
Explanation
Correct Answer
A:The reason it is necessary is: Aggregate accuracy can conceal materially unequal errors or harms for specific populations. In this case: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features.
Incorrect Answers
B:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. For this situation: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features.
C:For this situation: A recruitment scoring system has no subgroup confusion matrix despite plausible proxy correlations in applicant features.
D:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. The audit must corroborate: Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
Question 733
An audit committee asks for a defensible assessment of this situation: A medical model performs well on urban patients but fails when used in rural clinics with different testing equipment. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
- Use one IID test split as proof of safe performance for every new location and population.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- Load-test realistic context sizes, concurrency, cost limits, percentiles and fallback behavior.
Explanation
Correct Answer
A:The reason it is necessary is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source. Independent audit evidence includes: Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
Incorrect Answers
B:This attractive shortcut does not resolve the Out-of-distribution robustness and shift testing weakness: Use one IID test split as proof of safe performance for every new location and population.
C:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. It does not establish control effectiveness for Out-of-distribution robustness and shift testing.
D:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. It does not establish control effectiveness for Out-of-distribution robustness and shift testing.
Question 734
During an AI controls assessment, the team learns that a medical model performs well on urban patients but fails when used in rural clinics with different testing equipment. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- AI misuse testing requires plausible attacker-controlled surfaces and repeatable negative cases.
- Use one IID test split as proof of safe performance for every new location and population.
- Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
- AI outputs cannot be audited reliably when their source records, transformations and model lineage cannot be reconstructed.
Explanation
Correct Answer
C:Independent audit evidence includes: Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns. This is the applicable AI audit principle for Out-of-distribution robustness and shift testing: Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
Incorrect Answers
A:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. It does not establish control effectiveness for Out-of-distribution robustness and shift testing.
B:This attractive shortcut does not resolve the Out-of-distribution robustness and shift testing weakness: Use one IID test split as proof of safe performance for every new location and population. For this situation: A medical model performs well on urban patients but fails when used in rural clinics with different testing equipment.
D:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. It does not establish control effectiveness for Out-of-distribution robustness and shift testing.
Question 735
An independent AI auditor reviews this case: A medical model performs well on urban patients but fails when used in rural clinics with different testing equipment. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
- Inspect threat model, attack corpus diversity, tool permissions, exploit transcripts and remediation evidence.
- Use one IID test split as proof of safe performance for every new location and population.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
Explanation
Correct Answer
D:The reason it is necessary is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source. This evidence directly tests the asserted control for Out-of-distribution robustness and shift testing: Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
Incorrect Answers
A:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. It does not establish control effectiveness for Out-of-distribution robustness and shift testing.
B:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. For this situation: A medical model performs well on urban patients but fails when used in rural clinics with different testing equipment.
C:This attractive shortcut does not resolve the Out-of-distribution robustness and shift testing weakness: Use one IID test split as proof of safe performance for every new location and population. The auditor should instead examine: Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
Question 736
A governance and assurance review uncovers the following: A medical model performs well on urban patients but fails when used in rural clinics with different testing equipment. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- An independent rerun can identify the exact source data and material transformation decisions for a sampled model version.
- Use one IID test split as proof of safe performance for every new location and population.
- Performance advantage remains on independent representative evaluation with minimal leakage.
- Model performance remains within approved bounds or unsafe inputs reliably route to fallback.
Explanation
Correct Answer
D:The reason it is necessary is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source. This is an observable validation result for Out-of-distribution robustness and shift testing, not just management assurance: Model performance remains within approved bounds or unsafe inputs reliably route to fallback.
Incorrect Answers
A:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. For this situation: A medical model performs well on urban patients but fails when used in rural clinics with different testing equipment.
B:This attractive shortcut does not resolve the Out-of-distribution robustness and shift testing weakness: Use one IID test split as proof of safe performance for every new location and population. Effective closure requires: Model performance remains within approved bounds or unsafe inputs reliably route to fallback.
C:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. For this situation: A medical model performs well on urban patients but fails when used in rural clinics with different testing equipment.
Question 737
A risk-based audit engagement examines this operating condition: A manufacturing vision model mislabels defects under different lighting absent from the training set. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
- Use one IID test split as proof of safe performance for every new location and population.
- Conduct scoped adversarial testing across inputs, retrieved context, tools, identities and post-change regressions.
Explanation
Correct Answer
B:The reason it is necessary is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source. This is the proportionate control response to Out-of-distribution robustness and shift testing: Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
Incorrect Answers
A:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. The decisive principle here is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
C:This attractive shortcut does not resolve the Out-of-distribution robustness and shift testing weakness: Use one IID test split as proof of safe performance for every new location and population. In this case: A manufacturing vision model mislabels defects under different lighting absent from the training set.
D:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. The decisive principle here is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
Question 738
An audit committee asks for a defensible assessment of this situation: A manufacturing vision model mislabels defects under different lighting absent from the training set. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
- Use one IID test split as proof of safe performance for every new location and population.
- Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
- Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
Explanation
Correct Answer
D:Independent audit evidence includes: Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns. The corrective action is: Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
Incorrect Answers
A:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. For this situation: A manufacturing vision model mislabels defects under different lighting absent from the training set.
B:This attractive shortcut does not resolve the Out-of-distribution robustness and shift testing weakness: Use one IID test split as proof of safe performance for every new location and population. For this situation: A manufacturing vision model mislabels defects under different lighting absent from the training set.
C:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. It does not establish control effectiveness for Out-of-distribution robustness and shift testing.
Question 739
During an AI controls assessment, the team learns that a manufacturing vision model mislabels defects under different lighting absent from the training set. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
- Use one IID test split as proof of safe performance for every new location and population.
- Inspect dataset snapshots, feature pipeline commits, record counts, model registry links and training execution manifests.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
Explanation
Correct Answer
A:The reason it is necessary is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source. The corrective action is: Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
Incorrect Answers
B:This attractive shortcut does not resolve the Out-of-distribution robustness and shift testing weakness: Use one IID test split as proof of safe performance for every new location and population. Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
C:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. The decisive principle here is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
D:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. It does not establish control effectiveness for Out-of-distribution robustness and shift testing.
Question 740
An independent AI auditor reviews this case: A manufacturing vision model mislabels defects under different lighting absent from the training set. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
- Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
- Use one IID test split as proof of safe performance for every new location and population.
- Inspect threat model, attack corpus diversity, tool permissions, exploit transcripts and remediation evidence.
Explanation
Correct Answers
B:The reason it is necessary is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source. This response fits the risk associated with Out-of-distribution robustness and shift testing: Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
C:The reason it is necessary is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source. For this situation: A manufacturing vision model mislabels defects under different lighting absent from the training set.
Incorrect Answers
A:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. The decisive principle here is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
D:It misses the key control principle: Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
E:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. For this situation: A manufacturing vision model mislabels defects under different lighting absent from the training set.
