Amazon AWS Certified Machine Learning Engineer - Associate MLA-C01 Practice Test 2
Question 10
At the model-governance gate before promotion of a new ML artifact, the following condition is observed: A text classifier includes thousands of weak correlated features and becomes unstable across repeated training runs. The team needs a defensible technical decision rather than an untracked workaround. Select TWO: the appropriate diagnostic check and the targeted implementation change.
- Inspect tensor summaries, gradient magnitudes, learning-rate schedule, and training loss anomalies with supported debugging hooks.
- Use a smaller learning rate, optionally freeze early layers, and assess staged unfreezing against a held-out set.
- Use justified feature selection or sparsity constraints fitted on training folds before retraining.
- Deploy a SageMaker shadow variant for observational comparison before promoting the new model to served responses.
- Inspect feature importance stability, variance across folds, regularization and model dimensionality.
Explanation
Correct Answers
C:Use justified feature selection or sparsity constraints fitted on training folds before retraining. This is the appropriate controlled remedy for feature selection to reduce overparameterization. Unnecessary high-dimensional inputs can increase variance and compute cost; selection itself must avoid evaluation leakage. Verify the cause with: Inspect feature importance stability, variance across folds, regularization and model dimensionality. Then confirm: Holdout quality stays acceptable while model size and run-to-run instability decrease. This is one part of a select-two decision; the matching evidence and corrective action are both required. Reported signal: A text classifier includes thousands of weak correlated features and becomes unstable across repeated training runs. Operational setting: At the model-governance gate before promotion of a new ML artifact.
E:Inspect feature importance stability, variance across folds, regularization and model dimensionality. This directly tests feature selection to reduce overparameterization in the reported AWS ML workflow. Unnecessary high-dimensional inputs can increase variance and compute cost; selection itself must avoid evaluation leakage. It distinguishes the problem before choosing a change. The related targeted response is: Use justified feature selection or sparsity constraints fitted on training folds before retraining. This is one part of a select-two decision; the matching evidence and corrective action are both required. Reported signal: A text classifier includes thousands of weak correlated features and becomes unstable across repeated training runs. Operational setting: At the model-governance gate before promotion of a new ML artifact.
Incorrect Answers
A:Inspect tensor summaries, gradient magnitudes, learning-rate schedule, and training loss anomalies with supported debugging hooks. This selection concerns model debugger exploding gradient diagnosis rather than the required diagnostic-and-remedy pair for feature selection to reduce overparameterization. Use evidence: Inspect feature importance stability, variance across folds, regularization and model dimensionality. and the matching correction: Use justified feature selection or sparsity constraints fitted on training folds before retraining. Signal: A text classifier includes thousands of weak correlated features and becomes unstable across repeated training runs. Operational setting: At the model-governance gate before promotion of a new ML artifact.
B:Use a smaller learning rate, optionally freeze early layers, and assess staged unfreezing against a held-out set. This selection concerns fine-tuning pretrained model learning-rate stability rather than the required diagnostic-and-remedy pair for feature selection to reduce overparameterization. Use evidence: Inspect feature importance stability, variance across folds, regularization and model dimensionality. and the matching correction: Use justified feature selection or sparsity constraints fitted on training folds before retraining. Signal: A text classifier includes thousands of weak correlated features and becomes unstable across repeated training runs. Operational setting: At the model-governance gate before promotion of a new ML artifact.
D:Deploy a SageMaker shadow variant for observational comparison before promoting the new model to served responses. This selection concerns shadow deployment model evaluation without response change rather than the required diagnostic-and-remedy pair for feature selection to reduce overparameterization. Use evidence: Inspect feature importance stability, variance across folds, regularization and model dimensionality. and the matching correction: Use justified feature selection or sparsity constraints fitted on training folds before retraining. Signal: A text classifier includes thousands of weak correlated features and becomes unstable across repeated training runs. Operational setting: At the model-governance gate before promotion of a new ML artifact.
Question 11
As a regional data-pipeline rollout is validated against approved AWS storage and streams, the following condition is observed: A demand model treats hour 23 and hour 0 as far apart when integer hour is used as a linear continuous feature. The data team must preserve event-time meaning and lineage. Which TWO steps best establish the evidence and correct the identified root cause? Select TWO.
- Inspect prediction errors around midnight and feature representations for the periodic hour variable.
- Define explicit compatible conversions and schema contracts, isolate incompatible versions, and validate before publishing features.
- Split first; fit the imputer only on the training partition and apply those learned statistics to validation and test data.
- Encode hour with sine and cosine cyclic features and validate with a time-aware split.
- Measure file-size distribution, object request count, Spark task scheduling overhead, and read throughput.
Explanation
Correct Answers
A:Inspect prediction errors around midnight and feature representations for the periodic hour variable. This directly tests time-series cyclical calendar feature in the reported AWS ML workflow. Time-of-day has periodic topology; raw integer distance does not represent wraparound. It distinguishes the problem before choosing a change. The related targeted response is: Encode hour with sine and cosine cyclic features and validate with a time-aware split. This is one part of a select-two decision; the matching evidence and corrective action are both required. Reported signal: A demand model treats hour 23 and hour 0 as far apart when integer hour is used as a linear continuous feature. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
D:Encode hour with sine and cosine cyclic features and validate with a time-aware split. This is the appropriate controlled remedy for time-series cyclical calendar feature. Time-of-day has periodic topology; raw integer distance does not represent wraparound. Verify the cause with: Inspect prediction errors around midnight and feature representations for the periodic hour variable. Then confirm: Neighboring times across midnight map close in cyclic feature space and validation improves without leakage. This is one part of a select-two decision; the matching evidence and corrective action are both required. Reported signal: A demand model treats hour 23 and hour 0 as far apart when integer hour is used as a linear continuous feature. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
Incorrect Answers
B:Define explicit compatible conversions and schema contracts, isolate incompatible versions, and validate before publishing features. This selection concerns glue schema evolution with mixed input types rather than the required diagnostic-and-remedy pair for time-series cyclical calendar feature. Use evidence: Inspect prediction errors around midnight and feature representations for the periodic hour variable. and the matching correction: Encode hour with sine and cosine cyclic features and validate with a time-aware split. Signal: A demand model treats hour 23 and hour 0 as far apart when integer hour is used as a linear continuous feature. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
C:Split first; fit the imputer only on the training partition and apply those learned statistics to validation and test data. This selection concerns median imputation fitted on training split rather than the required diagnostic-and-remedy pair for time-series cyclical calendar feature. Use evidence: Inspect prediction errors around midnight and feature representations for the periodic hour variable. and the matching correction: Encode hour with sine and cosine cyclic features and validate with a time-aware split. Signal: A demand model treats hour 23 and hour 0 as far apart when integer hour is used as a linear continuous feature. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
E:Measure file-size distribution, object request count, Spark task scheduling overhead, and read throughput. This selection concerns columnar file compaction for small objects rather than the required diagnostic-and-remedy pair for time-series cyclical calendar feature. Use evidence: Inspect prediction errors around midnight and feature representations for the periodic hour variable. and the matching correction: Encode hour with sine and cosine cyclic features and validate with a time-aware split. Signal: A demand model treats hour 23 and hour 0 as far apart when integer hour is used as a linear continuous feature. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
Question 12
During a production MLOps reliability and security review, the following condition is observed: A training fleet runs many jobs and operators manually search thousands of unstructured log files to locate recurring data-channel failures. The owner needs an auditable resolution without broadening privileges. What targeted change should be implemented after verifying the stated cause?
- Use structured logging and scoped CloudWatch Logs Insights queries to aggregate failures by training job and time.
- Use approved SageMaker interface endpoints and relevant private DNS and policy settings for required API access.
- Use supported managed Spot Training with durable checkpoints for interruptible phases and retain On-Demand capacity when deadlines demand it.
- Pin an immutable approved artifact version or content digest and verify integrity before deployment.
Explanation
Correct Answer
A:Use structured logging and scoped CloudWatch Logs Insights queries to aggregate failures by training job and time. This is the appropriate controlled remedy for cloudwatch logs insights error correlation. Querying structured centralized logs scales better than visual browsing of unrelated service output. Verify the cause with: Inspect consistent log group names, job identifiers, timestamps, and failure codes in CloudWatch Logs. Then confirm: A query isolates the failing job versions and recurring error class within the incident interval. Reported signal: A training fleet runs many jobs and operators manually search thousands of unstructured log files to locate recurring data-channel failures. Operational setting: During a production MLOps reliability and security review.
Incorrect Answers
B:Use approved SageMaker interface endpoints and relevant private DNS and policy settings for required API access. This addresses a different issue involving private vpc endpoint policy for sagemaker apis. It does not correct the stated cloudwatch logs insights error correlation mechanism. The targeted response is: Use structured logging and scoped CloudWatch Logs Insights queries to aggregate failures by training job and time. Case evidence: A training fleet runs many jobs and operators manually search thousands of unstructured log files to locate recurring data-channel failures. Operational setting: During a production MLOps reliability and security review.
C:Use supported managed Spot Training with durable checkpoints for interruptible phases and retain On-Demand capacity when deadlines demand it. This addresses a different issue involving spot training cost versus interruption recovery. It does not correct the stated cloudwatch logs insights error correlation mechanism. The targeted response is: Use structured logging and scoped CloudWatch Logs Insights queries to aggregate failures by training job and time. Case evidence: A training fleet runs many jobs and operators manually search thousands of unstructured log files to locate recurring data-channel failures. Operational setting: During a production MLOps reliability and security review.
D:Pin an immutable approved artifact version or content digest and verify integrity before deployment. This addresses a different issue involving model artifact supply-chain integrity. It does not correct the stated cloudwatch logs insights error correlation mechanism. The targeted response is: Use structured logging and scoped CloudWatch Logs Insights queries to aggregate failures by training job and time. Case evidence: A training fleet runs many jobs and operators manually search thousands of unstructured log files to locate recurring data-channel failures. Operational setting: During a production MLOps reliability and security review.
Question 13
During production evaluation of a SageMaker AI model candidate, the following condition is observed: A house-price model's RMSE increases sharply after a few exceptionally expensive properties enter the validation set, although typical absolute errors remain stable. The release is blocked until the underlying model behavior is understood. Which measured outcome would confirm that the correction worked as intended?
- Multiple workers reduce epoch time with acceptable convergence and measured communication overhead.
- The evaluation report shows both typical error and extreme-error impact with a justified primary metric.
- The chosen version meets task metrics and license conditions and its inference contract is reproducible.
- Fraud recall meets the agreed target with documented review-volume and false-positive tradeoffs.
Explanation
Correct Answer
B:The evaluation report shows both typical error and extreme-error impact with a justified primary metric. This is the acceptance evidence for the described rmse versus mae outlier sensitivity scenario. Squaring residuals gives large errors disproportionately high weight; MAE weights magnitudes linearly. The corrective implementation it validates is: Compare RMSE with MAE or robust error summaries and choose the metric matching the application's penalty for large misses. Reported signal: A house-price model's RMSE increases sharply after a few exceptionally expensive properties enter the validation set, although typical absolute errors remain stable. Operational setting: During production evaluation of a SageMaker AI model candidate.
Incorrect Answers
A:Multiple workers reduce epoch time with acceptable convergence and measured communication overhead. This could validate data parallelism versus model parallelism, but it does not establish that rmse versus mae outlier sensitivity is resolved for this workload. Confirm instead: The evaluation report shows both typical error and extreme-error impact with a justified primary metric. Observed problem: A house-price model's RMSE increases sharply after a few exceptionally expensive properties enter the validation set, although typical absolute errors remain stable. Operational setting: During production evaluation of a SageMaker AI model candidate.
C:The chosen version meets task metrics and license conditions and its inference contract is reproducible. This could validate sagemaker jumpstart template provenance, but it does not establish that rmse versus mae outlier sensitivity is resolved for this workload. Confirm instead: The evaluation report shows both typical error and extreme-error impact with a justified primary metric. Observed problem: A house-price model's RMSE increases sharply after a few exceptionally expensive properties enter the validation set, although typical absolute errors remain stable. Operational setting: During production evaluation of a SageMaker AI model candidate.
D:Fraud recall meets the agreed target with documented review-volume and false-positive tradeoffs. This could validate precision versus recall fraud threshold, but it does not establish that rmse versus mae outlier sensitivity is resolved for this workload. Confirm instead: The evaluation report shows both typical error and extreme-error impact with a justified primary metric. Observed problem: A house-price model's RMSE increases sharply after a few exceptionally expensive properties enter the validation set, although typical absolute errors remain stable. Operational setting: During production evaluation of a SageMaker AI model candidate.
Question 14
As a regional data-pipeline rollout is validated against approved AWS storage and streams, the following condition is observed: A medical-image classifier uses random row splitting so multiple scans from the same patient occur in training and validation. The data team must preserve event-time meaning and lineage. Which measured outcome would confirm that the correction worked as intended?
- No patient identifier crosses split boundaries and evaluation reflects generalization to unseen patients.
- Invalid batches are rejected with attributable rule failures and accepted training data meets the entity-key contract.
- Application p95 latency returns to target while exported records meet a measured completeness and staleness objective.
- A controlled out-of-order replay yields the expected window totals and bounded late-event treatment.
Explanation
Correct Answer
A:No patient identifier crosses split boundaries and evaluation reflects generalization to unseen patients. This is the acceptance evidence for the described group-leakage train and validation split scenario. Correlated records from one entity across splits can inflate performance without improving true generalization. The corrective implementation it validates is: Split by stable patient identity before training and reserve a disjoint held-out group for evaluation. Reported signal: A medical-image classifier uses random row splitting so multiple scans from the same patient occur in training and validation. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
Incorrect Answers
B:Invalid batches are rejected with attributable rule failures and accepted training data meets the entity-key contract. This could validate glue data quality non-null contract, but it does not establish that group-leakage train and validation split is resolved for this workload. Confirm instead: No patient identifier crosses split boundaries and evaluation reflects generalization to unseen patients. Observed problem: A medical-image classifier uses random row splitting so multiple scans from the same patient occur in training and validation. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
C:Application p95 latency returns to target while exported records meet a measured completeness and staleness objective. This could validate rds extraction overload into ml, but it does not establish that group-leakage train and validation split is resolved for this workload. Confirm instead: No patient identifier crosses split boundaries and evaluation reflects generalization to unseen patients. Observed problem: A medical-image classifier uses random row splitting so multiple scans from the same patient occur in training and validation. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
D:A controlled out-of-order replay yields the expected window totals and bounded late-event treatment. This could validate streaming feature rolling window correctness, but it does not establish that group-leakage train and validation split is resolved for this workload. Confirm instead: No patient identifier crosses split boundaries and evaluation reflects generalization to unseen patients. Observed problem: A medical-image classifier uses random row splitting so multiple scans from the same patient occur in training and validation. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
Question 15
During an approved rollout of an AWS ML inference workload, the following condition is observed: A retraining workflow copies notebook cells into a scheduled script without tracking data snapshots or processing dependencies, leading to inconsistent model artifacts. The team must meet its documented availability and rollback constraints. Which TWO steps best establish the evidence and correct the identified root cause? Select TWO.
- Reference an approved immutable image digest or controlled version tag and retain a reproducible build record.
- Define a parameterized SageMaker Pipeline that orders steps and records source data, code and model artifacts.
- Compare template parameters, stack drift results, change sets, and deployed endpoint settings.
- Inspect the DAG of processing, training, evaluation and registration steps, along with parameter and artifact lineage.
- Keep the blue environment available until green health gates pass and automate traffic rollback on failed alarms.
Explanation
Correct Answers
B:Define a parameterized SageMaker Pipeline that orders steps and records source data, code and model artifacts. This is the appropriate controlled remedy for sagemaker pipelines parameterized retraining. Pipelines make dependencies explicit; manually ordered notebook operations are difficult to repeat or audit. Verify the cause with: Inspect the DAG of processing, training, evaluation and registration steps, along with parameter and artifact lineage. Then confirm: Repeated runs with the same inputs reproduce the ordered processing and generate auditable versioned outputs. This is one part of a select-two decision; the matching evidence and corrective action are both required. Reported signal: A retraining workflow copies notebook cells into a scheduled script without tracking data snapshots or processing dependencies, leading to inconsistent model artifacts. Operational setting: During an approved rollout of an AWS ML inference workload.
D:Inspect the DAG of processing, training, evaluation and registration steps, along with parameter and artifact lineage. This directly tests sagemaker pipelines parameterized retraining in the reported AWS ML workflow. Pipelines make dependencies explicit; manually ordered notebook operations are difficult to repeat or audit. It distinguishes the problem before choosing a change. The related targeted response is: Define a parameterized SageMaker Pipeline that orders steps and records source data, code and model artifacts. This is one part of a select-two decision; the matching evidence and corrective action are both required. Reported signal: A retraining workflow copies notebook cells into a scheduled script without tracking data snapshots or processing dependencies, leading to inconsistent model artifacts. Operational setting: During an approved rollout of an AWS ML inference workload.
Incorrect Answers
A:Reference an approved immutable image digest or controlled version tag and retain a reproducible build record. This selection concerns ecr immutable image digest for inference rather than the required diagnostic-and-remedy pair for sagemaker pipelines parameterized retraining. Use evidence: Inspect the DAG of processing, training, evaluation and registration steps, along with parameter and artifact lineage. and the matching correction: Define a parameterized SageMaker Pipeline that orders steps and records source data, code and model artifacts. Signal: A retraining workflow copies notebook cells into a scheduled script without tracking data snapshots or processing dependencies, leading to inconsistent model artifacts. Operational setting: During an approved rollout of an AWS ML inference workload.
C:Compare template parameters, stack drift results, change sets, and deployed endpoint settings. This selection concerns cloudformation stack parameter drift rather than the required diagnostic-and-remedy pair for sagemaker pipelines parameterized retraining. Use evidence: Inspect the DAG of processing, training, evaluation and registration steps, along with parameter and artifact lineage. and the matching correction: Define a parameterized SageMaker Pipeline that orders steps and records source data, code and model artifacts. Signal: A retraining workflow copies notebook cells into a scheduled script without tracking data snapshots or processing dependencies, leading to inconsistent model artifacts. Operational setting: During an approved rollout of an AWS ML inference workload.
E:Keep the blue environment available until green health gates pass and automate traffic rollback on failed alarms. This selection concerns blue-green endpoint rollback after health failure rather than the required diagnostic-and-remedy pair for sagemaker pipelines parameterized retraining. Use evidence: Inspect the DAG of processing, training, evaluation and registration steps, along with parameter and artifact lineage. and the matching correction: Define a parameterized SageMaker Pipeline that orders steps and records source data, code and model artifacts. Signal: A retraining workflow copies notebook cells into a scheduled script without tracking data snapshots or processing dependencies, leading to inconsistent model artifacts. Operational setting: During an approved rollout of an AWS ML inference workload.
Question 16
During a controlled replay of the data-preparation workflow in a separate ML staging account, the following condition is observed: A Spark transformation joins billions of records on a region key, and one region contains most observations, stalling one shuffle task. The dataset contract must remain reproducible for a backfill. Which corrective action most directly addresses this ML workflow problem?
- Repartition or salt the skewed join appropriately, or use a justified broadcast join when the small side fits memory.
- Use a CSV parser with explicit quoting and schema validation; quarantine malformed records before training.
- Enforce a required non-null or uniqueness contract with thresholds; quarantine violating partitions before feature publication.
- Fit an appropriate scaler on training data and package the same transformation with inference preprocessing.
Explanation
Correct Answer
A:Repartition or salt the skewed join appropriately, or use a justified broadcast join when the small side fits memory. This is the appropriate controlled remedy for glue spark join skew on dominant identifier. A low-cardinality or skewed key can concentrate a distributed join into one expensive partition. Verify the cause with: Inspect Spark stage durations, partition size distribution, join key frequencies, and executor spill metrics. Then confirm: Long-tail task duration and shuffle spill decline without changing the joined record counts. Reported signal: A Spark transformation joins billions of records on a region key, and one region contains most observations, stalling one shuffle task. Operational setting: During a controlled replay of the data-preparation workflow in a separate ML staging account.
Incorrect Answers
B:Use a CSV parser with explicit quoting and schema validation; quarantine malformed records before training. This addresses a different issue involving csv schema ambiguity during ingestion. It does not correct the stated glue spark join skew on dominant identifier mechanism. The targeted response is: Repartition or salt the skewed join appropriately, or use a justified broadcast join when the small side fits memory. Case evidence: A Spark transformation joins billions of records on a region key, and one region contains most observations, stalling one shuffle task. Operational setting: During a controlled replay of the data-preparation workflow in a separate ML staging account.
C:Enforce a required non-null or uniqueness contract with thresholds; quarantine violating partitions before feature publication. This addresses a different issue involving glue data quality non-null contract. It does not correct the stated glue spark join skew on dominant identifier mechanism. The targeted response is: Repartition or salt the skewed join appropriately, or use a justified broadcast join when the small side fits memory. Case evidence: A Spark transformation joins billions of records on a region key, and one region contains most observations, stalling one shuffle task. Operational setting: During a controlled replay of the data-preparation workflow in a separate ML staging account.
D:Fit an appropriate scaler on training data and package the same transformation with inference preprocessing. This addresses a different issue involving standardization of numerical features. It does not correct the stated glue spark join skew on dominant identifier mechanism. The targeted response is: Repartition or salt the skewed join appropriately, or use a justified broadcast join when the small side fits memory. Case evidence: A Spark transformation joins billions of records on a region key, and one region contains most observations, stalling one shuffle task. Operational setting: During a controlled replay of the data-preparation workflow in a separate ML staging account.
Question 17
As a regional data-pipeline rollout is validated against approved AWS storage and streams, the following condition is observed: A medical-image classifier uses random row splitting so multiple scans from the same patient occur in training and validation. The data team must preserve event-time meaning and lineage. Which engineering response resolves the actual issue without masking the symptoms?
- Use controlled incremental extraction or a suitable read replica and throttle export jobs within the approved data freshness target.
- Split by stable patient identity before training and reserve a disjoint held-out group for evaluation.
- Remove or pseudonymize unnecessary PII before annotation and restrict access to the token mapping under separate controls.
- Define point-in-time offline joins, freshness SLAs, and aligned feature version logic for online retrieval.
Explanation
Correct Answer
B:Split by stable patient identity before training and reserve a disjoint held-out group for evaluation. This is the appropriate controlled remedy for group-leakage train and validation split. Correlated records from one entity across splits can inflate performance without improving true generalization. Verify the cause with: Inspect shared entity identifiers across partitions and compare patient-level versus image-level evaluation results. Then confirm: No patient identifier crosses split boundaries and evaluation reflects generalization to unseen patients. Reported signal: A medical-image classifier uses random row splitting so multiple scans from the same patient occur in training and validation. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
Incorrect Answers
A:Use controlled incremental extraction or a suitable read replica and throttle export jobs within the approved data freshness target. This addresses a different issue involving rds extraction overload into ml. It does not correct the stated group-leakage train and validation split mechanism. The targeted response is: Split by stable patient identity before training and reserve a disjoint held-out group for evaluation. Case evidence: A medical-image classifier uses random row splitting so multiple scans from the same patient occur in training and validation. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
C:Remove or pseudonymize unnecessary PII before annotation and restrict access to the token mapping under separate controls. This addresses a different issue involving pii masking before model labeling. It does not correct the stated group-leakage train and validation split mechanism. The targeted response is: Split by stable patient identity before training and reserve a disjoint held-out group for evaluation. Case evidence: A medical-image classifier uses random row splitting so multiple scans from the same patient occur in training and validation. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
D:Define point-in-time offline joins, freshness SLAs, and aligned feature version logic for online retrieval. This addresses a different issue involving sagemaker feature store online offline consistency. It does not correct the stated group-leakage train and validation split mechanism. The targeted response is: Split by stable patient identity before training and reserve a disjoint held-out group for evaluation. Case evidence: A medical-image classifier uses random row splitting so multiple scans from the same patient occur in training and validation. Operational setting: As a regional data-pipeline rollout is validated against approved AWS storage and streams.
Question 18
As a regulated ML workload prepares for an automated regional rollout, the following condition is observed: A binary risk model continues producing apparently confident scores after a market regime change, but realized outcomes show the scores no longer correspond to observed event frequency. The team must demonstrate a repeatable monitoring or security control. Which observation most directly identifies the root cause of the reported behavior?
- Inspect CloudTrail management events, data event selectors, S3 bucket coverage, and log retention controls.
- Inspect online versus offline feature computation code, event-time cutoffs, freshness, and sampled entity values.
- Compare probability calibration plots, Brier score and event rates across recent labeled cohorts.
- Compare ModelLatency with OverheadLatency, client network timing, invocation counts, and endpoint 5XX responses.
Explanation
Correct Answer
C:Compare probability calibration plots, Brier score and event rates across recent labeled cohorts. This directly tests serving output confidence calibration drift in the reported AWS ML workflow. A good ranking metric does not guarantee calibrated probabilities; calibration must be verified against observed outcomes. It distinguishes the problem before choosing a change. The related targeted response is: Monitor calibration against delayed truth and retrain or recalibrate on representative permitted data when drift is confirmed. Reported signal: A binary risk model continues producing apparently confident scores after a market regime change, but realized outcomes show the scores no longer correspond to observed event frequency. Operational setting: As a regulated ML workload prepares for an automated regional rollout.
Incorrect Answers
A:Inspect CloudTrail management events, data event selectors, S3 bucket coverage, and log retention controls. This is a possible diagnostic for cloudtrail training data access audit coverage, but it does not directly test the serving output confidence calibration drift failure reported here. Prioritize: Compare probability calibration plots, Brier score and event rates across recent labeled cohorts. The relevant case is: A binary risk model continues producing apparently confident scores after a market regime change, but realized outcomes show the scores no longer correspond to observed event frequency. Operational setting: As a regulated ML workload prepares for an automated regional rollout.
B:Inspect online versus offline feature computation code, event-time cutoffs, freshness, and sampled entity values. This is a possible diagnostic for training-serving feature skew incident, but it does not directly test the serving output confidence calibration drift failure reported here. Prioritize: Compare probability calibration plots, Brier score and event rates across recent labeled cohorts. The relevant case is: A binary risk model continues producing apparently confident scores after a market regime change, but realized outcomes show the scores no longer correspond to observed event frequency. Operational setting: As a regulated ML workload prepares for an automated regional rollout.
D:Compare ModelLatency with OverheadLatency, client network timing, invocation counts, and endpoint 5XX responses. This is a possible diagnostic for cloudwatch invocations and modellatency distinction, but it does not directly test the serving output confidence calibration drift failure reported here. Prioritize: Compare probability calibration plots, Brier score and event rates across recent labeled cohorts. The relevant case is: A binary risk model continues producing apparently confident scores after a market regime change, but realized outcomes show the scores no longer correspond to observed event frequency. Operational setting: As a regulated ML workload prepares for an automated regional rollout.
Question 19
While reproducing the alert in an isolated AWS ML integration account, the following condition is observed: A loan default label becomes available months after inference, but a dashboard falsely reports current prediction accuracy using unlabeled requests. The proposed change must preserve logging and least privilege. What targeted change should be implemented after verifying the stated cause?
- Apply consistent approved cost allocation tags to ML jobs and supported linked resources and activate relevant reporting.
- Separate prompt input-distribution monitoring from delayed model-quality evaluation and join only observed labels to predictions.
- Use appropriate SageMaker Clarify model-bias monitoring and investigate supported group differences before deployment changes.
- Use supported managed Spot Training with durable checkpoints for interruptible phases and retain On-Demand capacity when deadlines demand it.
Explanation
Correct Answer
B:Separate prompt input-distribution monitoring from delayed model-quality evaluation and join only observed labels to predictions. This is the appropriate controlled remedy for target-label delay in production quality monitoring. True supervised production quality generally needs observed outcomes; prediction outputs alone are not ground truth. Verify the cause with: Inspect inference timestamps, eventual ground-truth arrival, join keys, and the provenance of displayed quality metrics. Then confirm: Model quality updates only for labeled cohorts while latency and feature drift are monitored promptly. Reported signal: A loan default label becomes available months after inference, but a dashboard falsely reports current prediction accuracy using unlabeled requests. Operational setting: While reproducing the alert in an isolated AWS ML integration account.
Incorrect Answers
A:Apply consistent approved cost allocation tags to ML jobs and supported linked resources and activate relevant reporting. This addresses a different issue involving tag-based ml training cost attribution. It does not correct the stated target-label delay in production quality monitoring mechanism. The targeted response is: Separate prompt input-distribution monitoring from delayed model-quality evaluation and join only observed labels to predictions. Case evidence: A loan default label becomes available months after inference, but a dashboard falsely reports current prediction accuracy using unlabeled requests. Operational setting: While reproducing the alert in an isolated AWS ML integration account.
C:Use appropriate SageMaker Clarify model-bias monitoring and investigate supported group differences before deployment changes. This addresses a different issue involving clarify postdeployment group bias monitoring. It does not correct the stated target-label delay in production quality monitoring mechanism. The targeted response is: Separate prompt input-distribution monitoring from delayed model-quality evaluation and join only observed labels to predictions. Case evidence: A loan default label becomes available months after inference, but a dashboard falsely reports current prediction accuracy using unlabeled requests. Operational setting: While reproducing the alert in an isolated AWS ML integration account.
D:Use supported managed Spot Training with durable checkpoints for interruptible phases and retain On-Demand capacity when deadlines demand it. This addresses a different issue involving spot training cost versus interruption recovery. It does not correct the stated target-label delay in production quality monitoring mechanism. The targeted response is: Separate prompt input-distribution monitoring from delayed model-quality evaluation and join only observed labels to predictions. Case evidence: A loan default label becomes available months after inference, but a dashboard falsely reports current prediction accuracy using unlabeled requests. Operational setting: While reproducing the alert in an isolated AWS ML integration account.
Question 20
During an approved rollout of an AWS ML inference workload, the following condition is observed: A retraining workflow copies notebook cells into a scheduled script without tracking data snapshots or processing dependencies, leading to inconsistent model artifacts. The team must meet its documented availability and rollback constraints. What targeted change should be implemented after verifying the stated cause?
- Define a parameterized SageMaker Pipeline that orders steps and records source data, code and model artifacts.
- Require model evaluation and explicit approval gates before production deployment, blocking failed metric thresholds.
- Use SageMaker Serverless Inference where the model and latency envelope fit, or justify provisioned capacity if it does not.
- Set adequate minimum capacity, scaling limits, alarms, and request backpressure for the expected traffic ramp.
Explanation
Correct Answer
A:Define a parameterized SageMaker Pipeline that orders steps and records source data, code and model artifacts. This is the appropriate controlled remedy for sagemaker pipelines parameterized retraining. Pipelines make dependencies explicit; manually ordered notebook operations are difficult to repeat or audit. Verify the cause with: Inspect the DAG of processing, training, evaluation and registration steps, along with parameter and artifact lineage. Then confirm: Repeated runs with the same inputs reproduce the ordered processing and generate auditable versioned outputs. Reported signal: A retraining workflow copies notebook cells into a scheduled script without tracking data snapshots or processing dependencies, leading to inconsistent model artifacts. Operational setting: During an approved rollout of an AWS ML inference workload.
Incorrect Answers
B:Require model evaluation and explicit approval gates before production deployment, blocking failed metric thresholds. This addresses a different issue involving codepipeline approval gate on model metrics. It does not correct the stated sagemaker pipelines parameterized retraining mechanism. The targeted response is: Define a parameterized SageMaker Pipeline that orders steps and records source data, code and model artifacts. Case evidence: A retraining workflow copies notebook cells into a scheduled script without tracking data snapshots or processing dependencies, leading to inconsistent model artifacts. Operational setting: During an approved rollout of an AWS ML inference workload.
C:Use SageMaker Serverless Inference where the model and latency envelope fit, or justify provisioned capacity if it does not. This addresses a different issue involving serverless inference for intermittent requests. It does not correct the stated sagemaker pipelines parameterized retraining mechanism. The targeted response is: Define a parameterized SageMaker Pipeline that orders steps and records source data, code and model artifacts. Case evidence: A retraining workflow copies notebook cells into a scheduled script without tracking data snapshots or processing dependencies, leading to inconsistent model artifacts. Operational setting: During an approved rollout of an AWS ML inference workload.
D:Set adequate minimum capacity, scaling limits, alarms, and request backpressure for the expected traffic ramp. This addresses a different issue involving endpoint scale-to-zero cold capacity planning. It does not correct the stated sagemaker pipelines parameterized retraining mechanism. The targeted response is: Define a parameterized SageMaker Pipeline that orders steps and records source data, code and model artifacts. Case evidence: A retraining workflow copies notebook cells into a scheduled script without tracking data snapshots or processing dependencies, leading to inconsistent model artifacts. Operational setting: During an approved rollout of an AWS ML inference workload.