Isaca AAIA Practice Test 75
Question 740
An independent AI auditor reviews this case: A manufacturing vision model mislabels defects under different lighting absent from the training set. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
- Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
- Use one IID test split as proof of safe performance for every new location and population.
- Inspect threat model, attack corpus diversity, tool permissions, exploit transcripts and remediation evidence.
Explanation
Correct Answers
B:The reason it is necessary is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source. This response fits the risk associated with Out-of-distribution robustness and shift testing: Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
C:The reason it is necessary is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source. For this situation: A manufacturing vision model mislabels defects under different lighting absent from the training set.
Incorrect Answers
A:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. The decisive principle here is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
D:It misses the key control principle: Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
E:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. For this situation: A manufacturing vision model mislabels defects under different lighting absent from the training set.
Question 741
A governance and assurance review uncovers the following: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Version and trace training, validation and inference datasets with accountable lineage, hashes and reproducible pipeline metadata.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
- Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
- Use one IID test split as proof of safe performance for every new location and population.
Explanation
Correct Answer
C:The reason it is necessary is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source. In this case: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them.
Incorrect Answers
A:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. In this case: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them.
B:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. In this case: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them.
D:This attractive shortcut does not resolve the Out-of-distribution robustness and shift testing weakness: Use one IID test split as proof of safe performance for every new location and population. In this case: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them.
Question 742
A risk-based audit engagement examines this operating condition: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- AI misuse testing requires plausible attacker-controlled surfaces and repeatable negative cases.
- Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
- Benchmark scores can overstate generalization when test items or related records have been exposed to development.
- Use one IID test split as proof of safe performance for every new location and population.
Explanation
Correct Answer
B:Independent audit evidence includes: Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns. In this case: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them.
Incorrect Answers
A:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. In this case: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them.
C:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. For this situation: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them.
D:This attractive shortcut does not resolve the Out-of-distribution robustness and shift testing weakness: Use one IID test split as proof of safe performance for every new location and population. For this situation: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them.
Question 743
An audit committee asks for a defensible assessment of this situation: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
- Use one IID test split as proof of safe performance for every new location and population.
- Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
Explanation
Correct Answer
D:The reason it is necessary is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source. The acceptable verified outcome is: Model performance remains within approved bounds or unsafe inputs reliably route to fallback.
Incorrect Answers
A:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. The audit must corroborate: Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
B:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. The decisive principle here is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
C:For this situation: A fraud predictor scores unusual holiday transactions as certain because the training environment lacked them.
Question 744
During an AI controls assessment, the team learns that a fraud predictor scores unusual holiday transactions as certain because the training environment lacked them. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Performance advantage remains on independent representative evaluation with minimal leakage.
- Model performance remains within approved bounds or unsafe inputs reliably route to fallback.
- Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
- Use one IID test split as proof of safe performance for every new location and population.
Explanation
Correct Answer
B:This is an observable validation result for Out-of-distribution robustness and shift testing, not just management assurance: Model performance remains within approved bounds or unsafe inputs reliably route to fallback.
Incorrect Answers
A:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. A proportionate response is: Evaluate behavior under credible distribution shifts and reject or safely defer unsupported inputs.
C:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. The decisive principle here is: Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
D:The auditor should instead examine: Inspect segment coverage, stress datasets, input validation, abstention, drift triggers and error patterns.
Question 745
An independent AI auditor reviews this case: A policy assistant provides polished answers with invented citations when asked about an unindexed procedure. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Evaluate justified subgroup error and outcome metrics with adequate samples and documented mitigation thresholds.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
- Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
Explanation
Correct Answer
D:The reason it is necessary is: Fluency and coherence are not evidence of factual correctness or fidelity to source material. Independent audit evidence includes: Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
Incorrect Answers
A:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. It does not establish control effectiveness for Grounded generative answer quality evaluation.
B:This attractive shortcut does not resolve the Grounded generative answer quality evaluation weakness: Accept a high language-model self-rating as objective proof that every generated answer is correct.
C:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. It does not establish control effectiveness for Grounded generative answer quality evaluation.
Question 746
A governance and assurance review uncovers the following: A policy assistant provides polished answers with invented citations when asked about an unindexed procedure. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- AI misuse testing requires plausible attacker-controlled surfaces and repeatable negative cases.
- Fluency and coherence are not evidence of factual correctness or fidelity to source material.
- Garbage or inconsistent ground-truth labels cap model validity even if the learning algorithm is sophisticated.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
Explanation
Correct Answer
B:Independent audit evidence includes: Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements. This is the applicable AI audit principle for Grounded generative answer quality evaluation: Fluency and coherence are not evidence of factual correctness or fidelity to source material.
Incorrect Answers
A:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. It does not establish control effectiveness for Grounded generative answer quality evaluation.
C:This option concerns Label quality and human annotator agreement, which is a related but different AI governance concern. It does not establish control effectiveness for Grounded generative answer quality evaluation.
D:This attractive shortcut does not resolve the Grounded generative answer quality evaluation weakness: Accept a high language-model self-rating as objective proof that every generated answer is correct. The auditor should instead examine: Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
Question 747
A risk-based audit engagement examines this operating condition: A policy assistant provides polished answers with invented citations when asked about an unindexed procedure. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect p95/p99 response, token counts, queue depth, error budgets, concurrency and billing ceilings.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
- Inspect training/evaluation overlap, duplicate hashes, benchmark dates, partition strategy and blind results.
Explanation
Correct Answer
B:The reason it is necessary is: Fluency and coherence are not evidence of factual correctness or fidelity to source material. This evidence directly tests the asserted control for Grounded generative answer quality evaluation: Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
Incorrect Answers
A:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. It does not establish control effectiveness for Grounded generative answer quality evaluation.
C:This attractive shortcut does not resolve the Grounded generative answer quality evaluation weakness: Accept a high language-model self-rating as objective proof that every generated answer is correct. For this situation: A policy assistant provides polished answers with invented citations when asked about an unindexed procedure.
D:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. The decisive principle here is: Fluency and coherence are not evidence of factual correctness or fidelity to source material.
Question 748
An audit committee asks for a defensible assessment of this situation: A policy assistant provides polished answers with invented citations when asked about an unindexed procedure. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
- Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
- Labels are consistent against approved definitions, and model tests expose residual uncertain classes.
- Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
Explanation
Correct Answer
B:The reason it is necessary is: Fluency and coherence are not evidence of factual correctness or fidelity to source material. This is an observable validation result for Grounded generative answer quality evaluation, not just management assurance: Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
Incorrect Answers
A:This attractive shortcut does not resolve the Grounded generative answer quality evaluation weakness: Accept a high language-model self-rating as objective proof that every generated answer is correct. Effective closure requires: Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
C:This option concerns Label quality and human annotator agreement, which is a related but different AI governance concern. The decisive principle here is: Fluency and coherence are not evidence of factual correctness or fidelity to source material.
D:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. The decisive principle here is: Fluency and coherence are not evidence of factual correctness or fidelity to source material.
Question 749
During an AI controls assessment, the team learns that a financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
- Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
- Define annotation rubrics, representative blind review, disagreement resolution and documented quality thresholds.
- Use independent held-out evaluations with leakage checks and repeatable challenge sets not tuned into training.
Explanation
Correct Answer
B:The reason it is necessary is: Fluency and coherence are not evidence of factual correctness or fidelity to source material. The acceptable verified outcome is: Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
Incorrect Answers
A:This attractive shortcut does not resolve the Grounded generative answer quality evaluation weakness: Accept a high language-model self-rating as objective proof that every generated answer is correct. In this case: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
C:This option concerns Label quality and human annotator agreement, which is a related but different AI governance concern. In this case: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
D:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. In this case: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
Question 750
An independent AI auditor reviews this case: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Conduct scoped adversarial testing across inputs, retrieved context, tools, identities and post-change regressions.
- Accept a high language-model self-rating as objective proof that every generated answer is correct.
- Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
- Inspect p95/p99 response, token counts, queue depth, error budgets, concurrency and billing ceilings.
Explanation
Correct Answers
C:The reason it is necessary is: Fluency and coherence are not evidence of factual correctness or fidelity to source material. For this situation: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
D:The reason it is necessary is: Fluency and coherence are not evidence of factual correctness or fidelity to source material. In this case: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
Incorrect Answers
A:This option concerns AI adversarial red-team scope and regression, which is a related but different AI governance concern. The decisive principle here is: Fluency and coherence are not evidence of factual correctness or fidelity to source material.
B:This attractive shortcut does not resolve the Grounded generative answer quality evaluation weakness: Accept a high language-model self-rating as objective proof that every generated answer is correct. For this situation: A financial summarizer reliably produces fluent paragraphs but omits negative clauses from source filings.
E:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. The decisive principle here is: Fluency and coherence are not evidence of factual correctness or fidelity to source material.
