Isaca AAIA Practice Test 81
Question 800
An independent AI auditor reviews this case: A language model leaks verbatim customer case notes when given partial phrases from earlier tickets. Which THREE elements together provide a defensible AI audit conclusion: the corrective control, the evidence to test it, and a verified outcome? Select THREE answers.
- Inspect stratified confusion matrices, sample sizes, relevant context, confidence intervals and exception responses.
- Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- Test citation support, omissions, contradiction handling and safe abstention using independently labeled examples.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes.
Explanation
Correct Answers
B:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. For this situation: A language model leaks verbatim customer case notes when given partial phrases from earlier tickets.
E:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. This response fits the risk associated with Membership inference and training-data memorization tests: Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
F:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. The corrective action is: Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
Incorrect Answers
A:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. The decisive principle here is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
C:It misses the key control principle: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
D:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. For this situation: A language model leaks verbatim customer case notes when given partial phrases from earlier tickets.
Question 801
A governance and assurance review uncovers the following: A fine-tuned assistant reveals whether an individual's confidential file was in its training set through repeated probes. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Validate synthetic data for utility, subgroup representation, memorization and plausible disclosure before release.
- Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- Load-test realistic context sizes, concurrency, cost limits, percentiles and fallback behavior.
Explanation
Correct Answer
B:The reason it is necessary is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior. In this case: A fine-tuned assistant reveals whether an individual's confidential file was in its training set through repeated probes.
Incorrect Answers
A:This option concerns Synthetic training data utility and privacy controls, which is a related but different AI governance concern. The decisive principle here is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
C:This attractive shortcut does not resolve the Membership inference and training-data memorization tests weakness: Assume privacy is guaranteed because the model returns generated text rather than a database row. In this case: A fine-tuned assistant reveals whether an individual's confidential file was in its training set through repeated probes.
D:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. The decisive principle here is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
Question 802
A risk-based audit engagement examines this operating condition: A fine-tuned assistant reveals whether an individual's confidential file was in its training set through repeated probes. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- Benchmark scores can overstate generalization when test items or related records have been exposed to development.
- Model validation must reflect foreseeable deployment conditions, not only random examples from one source.
Explanation
Correct Answer
A:The acceptable verified outcome is: High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes. The corrective action is: Test extractable memorization and membership inference under authorized red-team conditions and apply data minimization.
Incorrect Answers
B:This attractive shortcut does not resolve the Membership inference and training-data memorization tests weakness: Assume privacy is guaranteed because the model returns generated text rather than a database row. For this situation: A fine-tuned assistant reveals whether an individual's confidential file was in its training set through repeated probes.
C:This option concerns Training and evaluation benchmark contamination, which is a related but different AI governance concern. The audit must corroborate: Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
D:This option concerns Out-of-distribution robustness and shift testing, which is a related but different AI governance concern. The decisive principle here is: Not returning raw training files directly does not mean membership or sensitive content cannot leak from model behavior.
Question 803
An audit committee asks for a defensible assessment of this situation: A fine-tuned assistant reveals whether an individual's confidential file was in its training set through repeated probes. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect p95/p99 response, token counts, queue depth, error budgets, concurrency and billing ceilings.
- Inspect source passage matches, structured evaluation rubrics, omission rates, unsupported-claim samples and reviewer disagreements.
- Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
Explanation
Correct Answer
C:The acceptable verified outcome is: High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes.
Incorrect Answers
A:This option concerns AI service latency cost and capacity testing, which is a related but different AI governance concern. For this situation: A fine-tuned assistant reveals whether an individual's confidential file was in its training set through repeated probes.
B:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. The audit must corroborate: Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
D:For this situation: A fine-tuned assistant reveals whether an individual's confidential file was in its training set through repeated probes.
Question 804
During an AI controls assessment, the team learns that a fine-tuned assistant reveals whether an individual's confidential file was in its training set through repeated probes. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Approved subgroup criteria are met or transparent residual disparities receive treatment and escalation.
- Assume privacy is guaranteed because the model returns generated text rather than a database row.
- Independent reviewers confirm supported statements and absence of fabricated references within risk tolerances.
- High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes.
Explanation
Correct Answer
D:This is an observable validation result for Membership inference and training-data memorization tests, not just management assurance: High-risk memorization is bounded and confidential training content cannot be retrieved using tested probes.
Incorrect Answers
A:This option concerns Subgroup fairness performance validation, which is a related but different AI governance concern. The audit must corroborate: Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
B:The auditor should instead examine: Inspect controlled probe outputs, duplication controls, privacy evaluations, training-source permissions and mitigation.
C:This option concerns Grounded generative answer quality evaluation, which is a related but different AI governance concern. For this situation: A fine-tuned assistant reveals whether an individual's confidential file was in its training set through repeated probes.
Question 805
An independent AI auditor reviews this case: A procurement agent reads a supplier PDF that tells it to email an internal pricing file to an unrelated address. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Rate-limit and monitor queries, restrict unnecessary output detail and evaluate extraction risks proportionate to value.
- Treat retrieved text as untrusted data, separate instruction authority and constrain downstream tool access.
- Verify model provenance, storage integrity, executable dependencies and approved component changes.
- Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable.
Explanation
Correct Answer
B:The reason it is necessary is: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges. The acceptable verified outcome is: Malicious document instructions cannot cause prohibited tool actions or confidential-data release.
Incorrect Answers
A:This option concerns AI model extraction and excessive query access, which is a related but different AI governance concern. It does not establish control effectiveness for Indirect prompt injection through retrieved content.
C:This option concerns Foundation model dependency and artifact tampering, which is a related but different AI governance concern. It does not establish control effectiveness for Indirect prompt injection through retrieved content.
D:This attractive shortcut does not resolve the Indirect prompt injection through retrieved content weakness: Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable.
Question 806
A governance and assurance review uncovers the following: A procurement agent reads a supplier PDF that tells it to email an internal pricing file to an unrelated address. Which principle BEST explains why the observed practice is a material governance or control concern? Select ONE best answer.
- AI outputs cannot be audited reliably when their source records, transformations and model lineage cannot be reconstructed.
- Model-generated text is untrusted as an authorization basis for real-world privileged actions.
- Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges.
- Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable.
Explanation
Correct Answer
C:The acceptable verified outcome is: Malicious document instructions cannot cause prohibited tool actions or confidential-data release. This is the applicable AI audit principle for Indirect prompt injection through retrieved content: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges.
Incorrect Answers
A:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. It does not establish control effectiveness for Indirect prompt injection through retrieved content.
B:This option concerns AI tool misuse and excessive agency, which is a related but different AI governance concern. It does not establish control effectiveness for Indirect prompt injection through retrieved content.
D:This attractive shortcut does not resolve the Indirect prompt injection through retrieved content weakness: Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable. For this situation: A procurement agent reads a supplier PDF that tells it to email an internal pricing file to an unrelated address.
Question 807
A risk-based audit engagement examines this operating condition: A procurement agent reads a supplier PDF that tells it to email an internal pricing file to an unrelated address. What evidence should the auditor examine FIRST to substantiate or challenge the control owner’s claim? Select ONE best answer.
- Inspect dataset snapshots, feature pipeline commits, record counts, model registry links and training execution manifests.
- Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable.
- Inspect tool-call traces, retrieval source trust, instruction hierarchy tests and denied action logs.
- Inspect prompt history, version diffs, tested attack cases, tool allowlists and release tickets.
Explanation
Correct Answer
C:The reason it is necessary is: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges. This evidence directly tests the asserted control for Indirect prompt injection through retrieved content: Inspect tool-call traces, retrieval source trust, instruction hierarchy tests and denied action logs.
Incorrect Answers
A:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. The decisive principle here is: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges.
B:This attractive shortcut does not resolve the Indirect prompt injection through retrieved content weakness: Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable. The auditor should instead examine: Inspect tool-call traces, retrieval source trust, instruction hierarchy tests and denied action logs.
D:This option concerns Prompt template governance and injection-resilience changes, which is a related but different AI governance concern. It does not establish control effectiveness for Indirect prompt injection through retrieved content.
Question 808
An audit committee asks for a defensible assessment of this situation: A procurement agent reads a supplier PDF that tells it to email an internal pricing file to an unrelated address. After management addresses the issue, which result MOST convincingly demonstrates effective remediation? Select ONE best answer.
- Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable.
- Suspicious extraction patterns are detected and unauthorized systematic probing is constrained.
- Malicious document instructions cannot cause prohibited tool actions or confidential-data release.
- Only approved corpus changes influence outputs, and unexpected ranking changes trigger investigation.
Explanation
Correct Answer
C:The reason it is necessary is: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges. This is an observable validation result for Indirect prompt injection through retrieved content, not just management assurance: Malicious document instructions cannot cause prohibited tool actions or confidential-data release.
Incorrect Answers
A:This attractive shortcut does not resolve the Indirect prompt injection through retrieved content weakness: Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable. Effective closure requires: Malicious document instructions cannot cause prohibited tool actions or confidential-data release.
B:This option concerns AI model extraction and excessive query access, which is a related but different AI governance concern. The decisive principle here is: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges.
D:This option concerns Poisoned embedding and vector retrieval index, which is a related but different AI governance concern. It does not establish control effectiveness for Indirect prompt injection through retrieved content.
Question 809
During an AI controls assessment, the team learns that a service desk assistant retrieves a wiki page containing instructions to ignore the organization's security controls. What is the MOST appropriate risk-based audit recommendation that addresses the root control weakness? Select ONE best answer.
- Treat retrieved text as untrusted data, separate instruction authority and constrain downstream tool access.
- Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable.
- Version prompt templates, permission boundaries and evaluation suites with staged approvals and rollback.
- Version and trace training, validation and inference datasets with accountable lineage, hashes and reproducible pipeline metadata.
Explanation
Correct Answer
A:The reason it is necessary is: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges. This is the proportionate control response to Indirect prompt injection through retrieved content: Treat retrieved text as untrusted data, separate instruction authority and constrain downstream tool access.
Incorrect Answers
B:This attractive shortcut does not resolve the Indirect prompt injection through retrieved content weakness: Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable. In this case: A service desk assistant retrieves a wiki page containing instructions to ignore the organization's security controls.
C:This option concerns Prompt template governance and injection-resilience changes, which is a related but different AI governance concern. The decisive principle here is: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges.
D:This option concerns AI training dataset lineage and reproducibility, which is a related but different AI governance concern. The audit must corroborate: Inspect tool-call traces, retrieval source trust, instruction hierarchy tests and denied action logs.
Question 810
An independent AI auditor reviews this case: A service desk assistant retrieves a wiki page containing instructions to ignore the organization's security controls. Which TWO complementary elements should the auditor seek: an effective corrective control and the most probative independent evidence? Select TWO answers.
- Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable.
- Inspect tool-call traces, retrieval source trust, instruction hierarchy tests and denied action logs.
- Verify model provenance, storage integrity, executable dependencies and approved component changes.
- Treat retrieved text as untrusted data, separate instruction authority and constrain downstream tool access.
- Inspect index access, chunk hashes, ingest logs, tenant segregation and adversarial retrieval results.
Explanation
Correct Answers
B:The reason it is necessary is: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges. The corrective action is: Treat retrieved text as untrusted data, separate instruction authority and constrain downstream tool access.
D:The reason it is necessary is: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges. This response fits the risk associated with Indirect prompt injection through retrieved content: Treat retrieved text as untrusted data, separate instruction authority and constrain downstream tool access.
Incorrect Answers
A:This attractive shortcut does not resolve the Indirect prompt injection through retrieved content weakness: Assume any instruction embedded in a retrieved paragraph is authorized because the document is searchable. For this situation: A service desk assistant retrieves a wiki page containing instructions to ignore the organization's security controls.
C:This option concerns Foundation model dependency and artifact tampering, which is a related but different AI governance concern. The decisive principle here is: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges.
E:This option concerns Poisoned embedding and vector retrieval index, which is a related but different AI governance concern. The decisive principle here is: Untrusted retrieved documents can manipulate model instructions but must not gain system-level privileges.
