Isaca AAIA Practice Test 77
Question 760
An independent AI auditor reviews this case: A coding model scores unusually high on a public benchmark that was copied into its training corpus. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Publish an impressive test score without checking whether the test questions were training examples.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
Explanation
Correct Answers
C:The reason it is necessary is: Benchmark scores can overstate generalization when test items or related records have been exposed to development. For this situation: A coding model scores unusually high on a public benchmark that was copied into its training corpus.
E:The reason it is necessary is: Benchmark scores can overstate generalization when test items or related records have been exposed to development. This response fits the risk associated with Training and evaluation benchmark contamination: Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
Incorrect Answers
A:This attractive shortcut does not resolve the Training and evaluation benchmark contamination weakness: Publish an impressive test score without checking whether the test questions were training examples. Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
B:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. For this situation: A coding model scores unusually high on a public benchmark that was copied into its training corpus.
D:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. The decisive principle here is: Benchmark scores can overstate generalization when test items or related records have been exposed to development.
Question 761
A governance and assurance review uncovers the following: An audit vendor demonstrates near-perfect answers using evaluation questions revealed to the vendor during tuning. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Publish an impressive test score without checking whether the test questions were training examples.
- Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
- Load-test realistic context sizes, concurrency, cost limits, percentiles and fallback behavior.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
Explanation
Correct Answer
D:The reason it is necessary is: Benchmark scores can overstate generalization when test items or related records have been exposed to development. Independent audit evidence includes: Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
Incorrect Answers
A:This attractive shortcut does not resolve the Training and evaluation benchmark contamination weakness: Publish an impressive test score without checking whether the test questions were training examples. Effective closure requires: Performance advantage remains on independent representative evaluation with minimal leakage.
B:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. It does not establish control effectiveness for Training and evaluation benchmark contamination.
C:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. It does not establish control effectiveness for Training and evaluation benchmark contamination.
Question 762
A risk-based audit engagement examines this operating condition: An audit vendor demonstrates near-perfect answers using evaluation questions revealed to the vendor during tuning. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- Publish an impressive test score without checking whether the test questions were training examples.
- Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
- Benchmark scores can overstate generalization when test items or related records have been exposed to development.
- AI misuse testing requires plausible attacker-controlled surfaces and repeatable negative cases.
Explanation
Correct Answer
C:The corrective action is: Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training. Independent audit evidence includes: Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
Incorrect Answers
A:This attractive shortcut does not resolve the Training and evaluation benchmark contamination weakness: Publish an impressive test score without checking whether the test questions were training examples. For this situation: An audit vendor demonstrates near-perfect answers using evaluation questions revealed to the vendor during tuning.
B:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. The audit must corroborate: Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
D:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. It does not establish control effectiveness for Training and evaluation benchmark contamination.
Question 763
An audit committee asks for a defensible assessment of this situation: An audit vendor demonstrates near-perfect answers using evaluation questions revealed to the vendor during tuning. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
- Inspect dataset snapshots, feature pipeline commits, record counts, model registry links and training execution manifests.
- Publish an impressive test score without checking whether the test questions were training examples.
Explanation
Correct Answer
A:The reason it is necessary is: Benchmark scores can overstate generalization when test items or related records have been exposed to development. This evidence directly tests the asserted control for Training and evaluation benchmark contamination: Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
Incorrect Answers
B:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. The audit must corroborate: Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
C:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. The audit must corroborate: Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
D:It misses the key control principle: Benchmark scores can overstate generalization when test items or related records have been exposed to development.
Question 764
During an AI controls assessment, the team learns that an audit vendor demonstrates near-perfect answers using evaluation questions revealed to the vendor during tuning. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
- Performance advantage remains on independent representative evaluation with minimal leakage.
- High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes.
- Publish an impressive test score without checking whether the test questions were training examples.
Explanation
Correct Answer
B:The reason it is necessary is: Benchmark scores can overstate generalization when test items or related records have been exposed to development. This is an observable validation result for Training and evaluation benchmark contamination, not just management assurance: Performance advantage remains on independent representative evaluation with minimal leakage.
Incorrect Answers
A:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. It does not establish control effectiveness for Training and evaluation benchmark contamination.
C:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. For this situation: An audit vendor demonstrates near-perfect answers using evaluation questions revealed to the vendor during tuning.
D:For this situation: An audit vendor demonstrates near-perfect answers using evaluation questions revealed to the vendor during tuning.
Question 765
An independent AI auditor reviews this case: A fraud model is evaluated on a holdout that contains duplicated customer events from training. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
- Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
- Publish an impressive test score without checking whether the test questions were training examples.
- Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
Explanation
Correct Answer
A:The reason it is necessary is: Benchmark scores can overstate generalization when test items or related records have been exposed to development. The acceptable verified outcome is: Performance advantage remains on independent representative evaluation with minimal leakage.
Incorrect Answers
B:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. The decisive principle here is: Benchmark scores can overstate generalization when test items or related records have been exposed to development.
C:This attractive shortcut does not resolve the Training and evaluation benchmark contamination weakness: Publish an impressive test score without checking whether the test questions were training examples. In this case: A fraud model is evaluated on a holdout that contains duplicated customer events from training.
D:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. A proportionate response is: Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
Question 766
A governance and assurance review uncovers the following: A fraud model is evaluated on a holdout that contains duplicated customer events from training. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- Averages hide tail failures and resource exhaustion; service acceptability includes efficiency and availability.
- Publish an impressive test score without checking whether the test questions were training examples.
- Benchmark scores can overstate generalization when test items or related records have been exposed to development.
- Aggregate accuracy can conceal materially unequal errors or harms for specific populations.
Explanation
Correct Answer
C:The corrective action is: Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training. The acceptable verified outcome is: Performance advantage remains on independent representative evaluation with minimal leakage.
Incorrect Answers
A:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. The decisive principle here is: Benchmark scores can overstate generalization when test items or related records have been exposed to development.
B:This attractive shortcut does not resolve the Training and evaluation benchmark contamination weakness: Publish an impressive test score without checking whether the test questions were training examples. For this situation: A fraud model is evaluated on a holdout that contains duplicated customer events from training.
D:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. The decisive principle here is: Benchmark scores can overstate generalization when test items or related records have been exposed to development.
Question 767
A risk-based audit engagement examines this operating condition: A fraud model is evaluated on a holdout that contains duplicated customer events from training. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Publish an impressive test score without checking whether the test questions were training examples.
- Inspect threat model, attack corpus diversity, tool permissions, exploit transcripts and remediation evidence.
- Inspect dataset snapshots, feature pipeline commits, record counts, model registry links and training execution manifests.
- Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
Explanation
Correct Answer
D:The reason it is necessary is: Benchmark scores can overstate generalization when test items or related records have been exposed to development. In this case: A fraud model is evaluated on a holdout that contains duplicated customer events from training.
Incorrect Answers
A:For this situation: A fraud model is evaluated on a holdout that contains duplicated customer events from training.
B:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. The decisive principle here is: Benchmark scores can overstate generalization when test items or related records have been exposed to development.
C:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. A proportionate response is: Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
Question 768
An audit committee asks for a defensible assessment of this situation: A fraud model is evaluated on a holdout that contains duplicated customer events from training. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Performance advantage remains on independent representative evaluation with minimal leakage.
- Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
- Publish an impressive test score without checking whether the test questions were training examples.
- Realistic peak workloads meet approved quality, latency and cost limits with graceful degradation.
Explanation
Correct Answer
A:This is an observable validation result for Training and evaluation benchmark contamination, not just management assurance: Performance advantage remains on independent representative evaluation with minimal leakage.
Incorrect Answers
B:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. For this situation: A fraud model is evaluated on a holdout that contains duplicated customer events from training.
C:The auditor should instead examine: Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
D:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. For this situation: A fraud model is evaluated on a holdout that contains duplicated customer events from training.
Question 769
During an AI controls assessment, the team learns that a public chatbot is tested only for friendly prompts while attackers can upload documents with conflicting instructions. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Conduct scoped adversarial testing across inputs, retrieved context, tools, identities and post-change regressions.
- Version and trace training, validation and inference datasets with accountable lineage, hashes and reproducible pipeline metadata.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
- Call one successful refusal to an obvious unsafe prompt a comprehensive security red-team assessment.
Explanation
Correct Answer
A:The reason it is necessary is: AI misuse testing requires plausible attacker-controlled surfaces and repeatable negative cases. In this case: A public chatbot is tested only for friendly prompts while attackers can upload documents with conflicting instructions.
Incorrect Answers
B:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. It does not establish control effectiveness for AI adversarial red-team scope and regression.
C:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. It does not establish control effectiveness for AI adversarial red-team scope and regression.
D:This attractive shortcut does not resolve the AI adversarial red-team scope and regression weakness: Call one successful refusal to an obvious unsafe prompt a comprehensive security red-team assessment.
Question 770
An independent AI auditor reviews this case: A public chatbot is tested only for friendly prompts while attackers can upload documents with conflicting instructions. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Call one successful refusal to an obvious unsafe prompt a comprehensive security red-team assessment.
- Conduct scoped adversarial testing across inputs, retrieved context, tools, identities and post-change regressions.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
- Load-test realistic context sizes, concurrency, cost limits, percentiles and fallback behavior.
- Inspect threat model, attack corpus diversity, tool permissions, exploit transcripts and remediation evidence.
Explanation
Correct Answers
B:The reason it is necessary is: AI misuse testing requires plausible attacker-controlled surfaces and repeatable negative cases. For this situation: A public chatbot is tested only for friendly prompts while attackers can upload documents with conflicting instructions.
E:The reason it is necessary is: AI misuse testing requires plausible attacker-controlled surfaces and repeatable negative cases. This evidence directly tests the asserted control for AI adversarial red-team scope and regression: Inspect threat model, attack corpus diversity, tool permissions, exploit transcripts and remediation evidence.
Incorrect Answers
A:This attractive shortcut does not resolve the AI adversarial red-team scope and regression weakness: Call one successful refusal to an obvious unsafe prompt a comprehensive security red-team assessment. For this situation: A public chatbot is tested only for friendly prompts while attackers can upload documents with conflicting instructions.
C:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. It does not establish control effectiveness for AI adversarial red-team scope and regression.
D:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. It does not establish control effectiveness for AI adversarial red-team scope and regression.
