Isaca AAIA Practice Test 76
Question 750
An independent AI auditor reviews this case: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Conduct scoped adversarial testing across inputs, retrieved context, tools, identities and post-change regressions.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
- Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
- Inspect p95/p99 response, token counts, queue depth, error budgets, concurrency and billing ceilings.
Explanation
Correct Answers
C:The reason it is necessary is: Fluency and coherence are not evidence of factual correctness or fidelity to source material. For this situation: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
D:The reason it is necessary is: Fluency and coherence are not evidence of factual correctness or fidelity to source material. In this case: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
Incorrect Answers
A:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. The decisive principle here is: Fluency and coherence are not evidence of factual correctness or fidelity to source material.
B:This attractive shortcut does not resolve the Grounded generative answer quality evaluation weakness: Accept a high language-model self-rating as objective proof that every generated answer is correct. For this situation: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
E:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. The decisive principle here is: Fluency and coherence are not evidence of factual correctness or fidelity to source material.
Question 751
A governance and assurance review uncovers the following: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
- Review label guidelines, inter-rater reliability, adjudication logs, sampled errors and retraining effects.
Explanation
Correct Answer
B:The reason it is necessary is: Fluency and coherence are not evidence of factual correctness or fidelity to source material. The corrective action is: Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
Incorrect Answers
A:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. It does not establish control effectiveness for Grounded generative answer quality evaluation.
C:This attractive shortcut does not resolve the Grounded generative answer quality evaluation weakness: Accept a high language-model self-rating as objective proof that every generated answer is correct. Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
D:This option concerns Label quality and human annotator agreement, which is a related but different AI governance concern. For this situation: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
Question 752
A risk-based audit engagement examines this operating condition: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Model performance remains within approved bounds or unsafe inputs reliably route to fallback.
- Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
- Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
Explanation
Correct Answer
B:This is an observable validation result for Grounded generative answer quality evaluation, not just management assurance: Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
Incorrect Answers
A:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. It does not establish control effectiveness for Grounded generative answer quality evaluation.
C:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. In this case: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
D:For this situation: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
Question 753
An audit committee asks for a defensible assessment of this situation: A medical literature bot merges conclusions from two different studies into one unsupported recommendation. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
- Load-test realistic context sizes, concurrency, cost limits, percentiles and fallback behavior.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
Explanation
Correct Answer
B:The reason it is necessary is: Fluency and coherence are not evidence of factual correctness or fidelity to source material. This is the proportionate control response to Grounded generative answer quality evaluation: Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
Incorrect Answers
A:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. The decisive principle here is: Fluency and coherence are not evidence of factual correctness or fidelity to source material.
C:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. The audit must corroborate: Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
D:This attractive shortcut does not resolve the Grounded generative answer quality evaluation weakness: Accept a high language-model self-rating as objective proof that every generated answer is correct. In this case: A medical literature bot merges conclusions from two different studies into one unsupported recommendation.
Question 754
During an AI controls assessment, the team learns that a medical literature bot merges conclusions from two different studies into one unsupported recommendation. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
- Averages hide tail failures and resource exhaustion; service acceptability includes efficiency and availability.
- AI misuse testing requires plausible attacker-controlled surfaces and repeatable negative cases.
- Fluency and coherence are not evidence of factual correctness or fidelity to source material.
Explanation
Correct Answer
D:Independent audit evidence includes: Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements. The acceptable verified outcome is: Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
Incorrect Answers
A:This attractive shortcut does not resolve the Grounded generative answer quality evaluation weakness: Accept a high language-model self-rating as objective proof that every generated answer is correct. For this situation: A medical literature bot merges conclusions from two different studies into one unsupported recommendation.
B:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. For this situation: A medical literature bot merges conclusions from two different studies into one unsupported recommendation.
C:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. The audit must corroborate: Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
Question 755
An independent AI auditor reviews this case: A medical literature bot merges conclusions from two different studies into one unsupported recommendation. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
- Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
Explanation
Correct Answer
B:The reason it is necessary is: Fluency and coherence are not evidence of factual correctness or fidelity to source material. In this case: A medical literature bot merges conclusions from two different studies into one unsupported recommendation.
Incorrect Answers
A:It misses the key control principle: Fluency and coherence are not evidence of factual correctness or fidelity to source material.
C:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. The audit must corroborate: Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
D:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. The decisive principle here is: Fluency and coherence are not evidence of factual correctness or fidelity to source material.
Question 756
A governance and assurance review uncovers the following: A medical literature bot merges conclusions from two different studies into one unsupported recommendation. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
- Performance advantage remains on independent representative evaluation with minimal leakage.
- Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
- Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
Explanation
Correct Answer
D:Independent audit evidence includes: Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
Incorrect Answers
A:The auditor should instead examine: Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
B:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. For this situation: A medical literature bot merges conclusions from two different studies into one unsupported recommendation.
C:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. The audit must corroborate: Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
Question 757
A risk-based audit engagement examines this operating condition: A coding model scores unusually high on a public benchmark that was copied into its training corpus. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- Version and trace training, validation and inference datasets with accountable lineage, hashes and reproducible pipeline metadata.
- Publish an impressive test score without checking whether the test questions were training examples.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
Explanation
Correct Answer
D:The reason it is necessary is: Benchmark scores can overstate generalization when test items or related records have been exposed to development. This is the proportionate control response to Training and evaluation benchmark contamination: Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
Incorrect Answers
A:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. It does not establish control effectiveness for Training and evaluation benchmark contamination.
B:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. It does not establish control effectiveness for Training and evaluation benchmark contamination.
C:This attractive shortcut does not resolve the Training and evaluation benchmark contamination weakness: Publish an impressive test score without checking whether the test questions were training examples.
Question 758
An audit committee asks for a defensible assessment of this situation: A coding model scores unusually high on a public benchmark that was copied into its training corpus. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- AI outputs cannot be audited reliably when their source records, transformations and model lineage cannot be reconstructed.
- Benchmark scores can overstate generalization when test items or related records have been exposed to development.
- Publish an impressive test score without checking whether the test questions were training examples.
- Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
Explanation
Correct Answer
B:The corrective action is: Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training. This is the applicable AI audit principle for Training and evaluation benchmark contamination: Benchmark scores can overstate generalization when test items or related records have been exposed to development.
Incorrect Answers
A:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. The decisive principle here is: Benchmark scores can overstate generalization when test items or related records have been exposed to development.
C:This attractive shortcut does not resolve the Training and evaluation benchmark contamination weakness: Publish an impressive test score without checking whether the test questions were training examples. For this situation: A coding model scores unusually high on a public benchmark that was copied into its training corpus.
D:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. It does not establish control effectiveness for Training and evaluation benchmark contamination.
Question 759
During an AI controls assessment, the team learns that a coding model scores unusually high on a public benchmark that was copied into its training corpus. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
- Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
- Publish an impressive test score without checking whether the test questions were training examples.
- Inspect dataset snapshots, feature pipeline commits, record counts, model registry links and training execution manifests.
Explanation
Correct Answer
B:The reason it is necessary is: Benchmark scores can overstate generalization when test items or related records have been exposed to development. The corrective action is: Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
Incorrect Answers
A:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. The decisive principle here is: Benchmark scores can overstate generalization when test items or related records have been exposed to development.
C:This attractive shortcut does not resolve the Training and evaluation benchmark contamination weakness: Publish an impressive test score without checking whether the test questions were training examples. The auditor should instead examine: Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
D:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. For this situation: A coding model scores unusually high on a public benchmark that was copied into its training corpus.
Question 760
An independent AI auditor reviews this case: A coding model scores unusually high on a public benchmark that was copied into its training corpus. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Publish an impressive test score without checking whether the test questions were training examples.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
Explanation
Correct Answers
C:The reason it is necessary is: Benchmark scores can overstate generalization when test items or related records have been exposed to development. For this situation: A coding model scores unusually high on a public benchmark that was copied into its training corpus.
E:The reason it is necessary is: Benchmark scores can overstate generalization when test items or related records have been exposed to development. This response fits the risk associated with Training and evaluation benchmark contamination: Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
Incorrect Answers
A:This attractive shortcut does not resolve the Training and evaluation benchmark contamination weakness: Publish an impressive test score without checking whether the test questions were training examples. Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
B:This option concerns Membership inference and training-data memorization tests, which is a related but different AI governance concern. For this situation: A coding model scores unusually high on a public benchmark that was copied into its training corpus.
D:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. The decisive principle here is: Benchmark scores can overstate generalization when test items or related records have been exposed to development.
