Isaca AAIA Practice Test 80
Question 790
An independent AI auditor reviews this case: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Load-test realistic context sizes, concurrency, cost limits, percentiles and fallback behavior.
- Approve production based solely on one average latency measurement in a quiet lab.
- Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
- Inspect p95/p99 response, token counts, queue depth, error budgets, concurrency and billing ceilings.
Explanation
Correct Answers
A:The acceptable verified outcome is: Realistic peak workloads meet approved quality, latency and cost limits with graceful degradation. For this situation: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts.
E:The corrective action is: Load-test realistic context sizes, concurrency, cost limits, percentiles and fallback behavior. In this case: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts.
Incorrect Answers
B:This attractive shortcut does not resolve the AI service latency cost and capacity testing weakness: Approve production based solely on one average latency measurement in a quiet lab. For this situation: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts.
C:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. In this case: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts.
D:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. In this case: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts.
Question 791
A governance and assurance review uncovers the following: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
- Approve production based solely on one average latency measurement in a quiet lab.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
- Inspect p95/p99 response, token counts, queue depth, error budgets, concurrency and billing ceilings.
Explanation
Correct Answer
D:The corrective action is: Load-test realistic context sizes, concurrency, cost limits, percentiles and fallback behavior. For this situation: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts.
Incorrect Answers
A:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. In this case: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts.
B:For this situation: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts.
C:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. In this case: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts.
Question 792
A risk-based audit engagement examines this operating condition: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Performance advantage remains on independent representative evaluation with minimal leakage.
- Realistic peak workloads meet approved quality, latency and cost limits with graceful degradation.
- Approve production based solely on one average latency measurement in a quiet lab.
- Critical attack paths are blocked by tested technical controls and regressions catch recurrence.
Explanation
Correct Answer
B:The corrective action is: Load-test realistic context sizes, concurrency, cost limits, percentiles and fallback behavior. Independent audit evidence includes: Inspect p95/p99 response, token counts, queue depth, error budgets, concurrency and billing ceilings.
Incorrect Answers
A:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. For this situation: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts.
C:It misses the key control principle: Averages hide tail failures and resource exhaustion; service acceptability includes efficiency and availability.
D:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. In this case: A real-time risk service tests throughput only with short prompts although production uses long retrieved contexts.
Question 793
An audit committee asks for a defensible assessment of this situation: A public healthcare model can reproduce a rare patient's identifiable sentence from its training material. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
- Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
Explanation
Correct Answer
D:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. The acceptable verified outcome is: High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes.
Incorrect Answers
A:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. It does not establish control effectiveness for Membership inference and training-data memorization tests.
B:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. It does not establish control effectiveness for Membership inference and training-data memorization tests.
C:This attractive shortcut does not resolve the Membership inference and training-data memorization tests weakness: Assume privacy is guaranteed because the model returns generated text rather than a database row.
Question 794
During an AI controls assessment, the team learns that a public healthcare model can reproduce a rare patient's identifiable sentence from its training material. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- AI misuse testing requires plausible attacker-controlled surfaces and repeatable negative cases.
- Synthetic labels do not automatically eliminate privacy, bias or representativeness risk.
Explanation
Correct Answer
A:The acceptable verified outcome is: High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes. This is the applicable AI audit principle for Membership inference and training-data memorization tests: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
Incorrect Answers
B:This attractive shortcut does not resolve the Membership inference and training-data memorization tests weakness: Assume privacy is guaranteed because the model returns generated text rather than a database row. For this situation: A public healthcare model can reproduce a rare patient's identifiable sentence from its training material.
C:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. It does not establish control effectiveness for Membership inference and training-data memorization tests.
D:This option concerns Synthetic training data utility and privacy controls, which is a related but different AI governance concern. It does not establish control effectiveness for Membership inference and training-data memorization tests.
Question 795
An independent AI auditor reviews this case: A public healthcare model can reproduce a rare patient's identifiable sentence from its training material. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
- Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- Inspect threat model, attack corpus diversity, tool permissions, exploit transcripts and remediation evidence.
Explanation
Correct Answer
A:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. In this case: A public healthcare model can reproduce a rare patient's identifiable sentence from its training material.
Incorrect Answers
B:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. It does not establish control effectiveness for Membership inference and training-data memorization tests.
C:This attractive shortcut does not resolve the Membership inference and training-data memorization tests weakness: Assume privacy is guaranteed because the model returns generated text rather than a database row. The auditor should instead examine: Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
D:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. For this situation: A public healthcare model can reproduce a rare patient's identifiable sentence from its training material.
Question 796
A governance and assurance review uncovers the following: A public healthcare model can reproduce a rare patient's identifiable sentence from its training material. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes.
- Performance advantage remains on independent representative evaluation with minimal leakage.
- Critical attack paths are blocked by tested technical controls and regressions catch recurrence.
Explanation
Correct Answer
B:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. This is an observable validation result for Membership inference and training-data memorization tests, not just management assurance: High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes.
Incorrect Answers
A:This attractive shortcut does not resolve the Membership inference and training-data memorization tests weakness: Assume privacy is guaranteed because the model returns generated text rather than a database row. Effective closure requires: High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes.
C:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. For this situation: A public healthcare model can reproduce a rare patient's identifiable sentence from its training material.
D:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. The decisive principle here is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
Question 797
A risk-based audit engagement examines this operating condition: A language model leaks verbatim customer case notes when given partial phrases from earlier tickets. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Load-test realistic context sizes, concurrency, cost limits, percentiles and fallback behavior.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- Conduct scoped adversarial testing across inputs, retrieved context, tools, identities and post-change regressions.
Explanation
Correct Answer
B:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. Independent audit evidence includes: Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
Incorrect Answers
A:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. It does not establish control effectiveness for Membership inference and training-data memorization tests.
C:This attractive shortcut does not resolve the Membership inference and training-data memorization tests weakness: Assume privacy is guaranteed because the model returns generated text rather than a database row. In this case: A language model leaks verbatim customer case notes when given partial phrases from earlier tickets.
D:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. The audit must corroborate: Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
Question 798
An audit committee asks for a defensible assessment of this situation: A language model leaks verbatim customer case notes when given partial phrases from earlier tickets. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
- Fluency and coherence are not evidence of factual correctness or fidelity to source material.
- Benchmark scores can overstate generalization when test items or related records have been exposed to development.
Explanation
Correct Answer
B:The acceptable verified outcome is: High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes. Independent audit evidence includes: Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
Incorrect Answers
A:This attractive shortcut does not resolve the Membership inference and training-data memorization tests weakness: Assume privacy is guaranteed because the model returns generated text rather than a database row. For this situation: A language model leaks verbatim customer case notes when given partial phrases from earlier tickets.
C:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. The decisive principle here is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
D:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. The decisive principle here is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
Question 799
During an AI controls assessment, the team learns that a language model leaks verbatim customer case notes when given partial phrases from earlier tickets. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
- Inspect threat model, attack corpus diversity, tool permissions, exploit transcripts and remediation evidence.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
Explanation
Correct Answer
D:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. This evidence directly tests the asserted control for Membership inference and training-data memorization tests: Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
Incorrect Answers
A:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. It does not establish control effectiveness for Membership inference and training-data memorization tests.
B:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. For this situation: A language model leaks verbatim customer case notes when given partial phrases from earlier tickets.
C:This attractive shortcut does not resolve the Membership inference and training-data memorization tests weakness: Assume privacy is guaranteed because the model returns generated text rather than a database row. Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
Question 800
An independent AI auditor reviews this case: A language model leaks verbatim customer case notes when given partial phrases from earlier tickets. Which THREE elements together provide a defensible AI audit conclusion: the corrective control, the evidence to test it, and a verified outcome? Select THREE answers.
- Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
- Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes.
Explanation
Correct Answers
B:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. For this situation: A language model leaks verbatim customer case notes when given partial phrases from earlier tickets.
E:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. This response fits the risk associated with Membership inference and training-data memorization tests: Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
F:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. The corrective action is: Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
Incorrect Answers
A:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. The decisive principle here is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
C:It misses the key control principle: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
D:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. For this situation: A language model leaks verbatim customer case notes when given partial phrases from earlier tickets.
