Databricks Certified Data Engineer Associate Practice Test 65
Question 640
A data engineer at Public weather agency is responsible for station observation files. The task value passing issue is that station_observation_files_rural validation notebook computes the partition date but downstream tasks use yesterday by default. The supporting evidence is that logs show successful validation for one date and export for another. The engineer must ensure that the exported partition must match the validated partition. The failure and requirements have been diagnosed; the team must select the lasting pipeline or governance approach. Which durable Databricks control best addresses the requirement?
- Hardcode yesterday in every downstream task at Public weather agency
- Inspect taskValues producer, dependency and dynamic reference in station_observation_files_rural
- Monitor missing task values and inconsistent task parameters for station_observation_files_rural
- Pass a validated task value to the dependent export task
Explanation
Correct Answer
D:Pass a validated task value to the dependent export task is the intended durable approach for station_observation_files_rural: it addresses the verified problem (station_observation_files_rural validation notebook computes the partition date but downstream tasks use yesterday by default) while preserving the required behavior (the exported partition must match the validated partition). A diagnostic or monitoring step alone would not create this target-state behavior.
Incorrect Answers
A:Hardcoded dates can diverge from validated data. The proposed action (Hardcode yesterday in every downstream task at Public weather agency) does not meet the requirement that the exported partition must match the validated partition, nor does it constitute the target approach.
B:Inspect taskValues producer, dependency and dynamic reference in station_observation_files_rural could clarify the original fault, but the available evidence already establishes its scope; it is not the target approach now requested. The station_observation_files_rural workflow still requires that the exported partition must match the validated partition.
C:Monitor missing task values and inconsistent task parameters for station_observation_files_rural would help detect future recurrence, but metrics alone do not deliver the target approach. The station_observation_files_rural requirement is that the exported partition must match the validated partition.
Question 641
At Regional sporting venue, the admission scan events workflow depends on admission_scan_events_central. Engineers report that admission_scan_events_central final export shows success even though an optional upstream data audit failed; the evidence is that the team treats final task status as whole-pipeline correctness. They require that operational alerts must reflect required DAG path failures. The target approach is approved. The current task is to introduce it safely with bounded scope and rollback. Which controlled implementation step is appropriate now?
- Set success criteria and notifications on required tasks and aggregate job outcome
- Simulate a required task failure and verify publish stops and alert fires
- Monitor only the last task while ignoring failed required upstream nodes at Regional sporting venue
- Configure failure notification and dependency gates before prod promotion
Explanation
Correct Answer
D:Configure failure notification and dependency gates before prod promotion provides a bounded way to introduce the approved solution into admission_scan_events_central and detect an early mismatch. The existing signal is the team treats final task status as whole-pipeline correctness; a staged change protects the service requirement (operational alerts must reflect required DAG path failures) and makes rollback practical.
Incorrect Answers
A:Set success criteria and notifications on required tasks and aggregate job outcome is a plausible lasting design once chosen, but it does not perform the controlled implementation step at this stage. In the admission_scan_events_central scenario, the observed evidence is that the team treats final task status as whole-pipeline correctness; the requirement remains: operational alerts must reflect required DAG path failures.
B:Simulate a required task failure and verify publish stops and alert fires is a one-time acceptance check. It does not supply the controlled implementation step for admission_scan_events_central; a passing test cannot substitute for the specific task now required.
C:Last-task success does not prove full dag correctness. The proposed action (Monitor only the last task while ignoring failed required upstream nodes at Regional sporting venue) does not meet the requirement that operational alerts must reflect required DAG path failures, nor does it constitute the controlled implementation step.
Question 642
At Research clinical trial group, the consent-related data changes workflow depends on consent_related_data_changes_west. Engineers report that the consent_related_data_changes_west relational source emits inserts and updates but the lakehouse target retains multiple current versions; the evidence is that change records carry primary keys and update sequence numbers. They require that the destination must represent the latest source state without losing required history. The first acceptance run passed. The current task is to detect future regressions and changes over time. Which ongoing alert or metric is most relevant?
- Apply multiple out-of-order changes and verify deterministic final state
- Use supported CDC apply logic with keys, ordering and defined delete handling
- Append every change as a new current row without keys at Research clinical trial group
- Monitor CDC lag, duplicate keys and discarded out-of-order updates for consent_related_data_changes_west
Explanation
Correct Answer
D:Monitor CDC lag, duplicate keys and discarded out-of-order updates for consent_related_data_changes_west creates recurring visibility into the failure mode (the consent_related_data_changes_west relational source emits inserts and updates but the lakehouse target retains multiple current versions). Unlike a single successful test, the measure can reveal drift, lag, or new failures after later source and deployment changes while the destination must represent the latest source state without losing required history.
Incorrect Answers
A:Apply multiple out-of-order changes and verify deterministic final state is a one-time acceptance check. It does not supply continuing operational detection for consent_related_data_changes_west; a passing test cannot substitute for the specific task now required.
B:Use supported CDC apply logic with keys, ordering and defined delete handling is a plausible lasting design once chosen, but it does not perform continuing operational detection at this stage. In the consent_related_data_changes_west scenario, the observed evidence is that change records carry primary keys and update sequence numbers; the requirement remains: the destination must represent the latest source state without losing required history.
C:An append-only current-state table does not resolve updates or deletes. The proposed action (Append every change as a new current row without keys at Research clinical trial group) does not meet the requirement that the destination must represent the latest source state without losing required history, nor does it constitute continuing operational detection.
Question 643
At Regional insurance broker, the claims payment events workflow depends on claims_payment_events_central. Engineers report that claims_payment_events_central daily DataFrames have the same field names in different column orders; the evidence is that union positional matching puts phone numbers into postal-code fields. They require that combined data must align values by field name rather than column position. The target approach is approved. The current task is to introduce it safely with bounded scope and rollback. Which controlled implementation step is appropriate now?
- Use positional union without checking source column order at Regional insurance broker
- Compare samples with reordered columns to verify identity-preserving alignment
- Use unionByName with controlled missing-column behavior as needed
- Validate schema compatibility and replace positional union in staging
Explanation
Correct Answer
D:Validate schema compatibility and replace positional union in staging provides a bounded way to introduce the approved solution into claims_payment_events_central and detect an early mismatch. The existing signal is union positional matching puts phone numbers into postal-code fields; a staged change protects the service requirement (combined data must align values by field name rather than column position) and makes rollback practical.
Incorrect Answers
A:Position-based union can silently misalign values. The proposed action (Use positional union without checking source column order at Regional insurance broker) does not meet the requirement that combined data must align values by field name rather than column position, nor does it constitute the controlled implementation step.
B:Compare samples with reordered columns to verify identity-preserving alignment is a one-time acceptance check. It does not supply the controlled implementation step for claims_payment_events_central; a passing test cannot substitute for the specific task now required.
C:Use unionByName with controlled missing-column behavior as needed is a plausible lasting design once chosen, but it does not perform the controlled implementation step at this stage. In the claims_payment_events_central scenario, the observed evidence is that union positional matching puts phone numbers into postal-code fields; the requirement remains: combined data must align values by field name rather than column position.
Question 644
At Regional insurance broker, the gold materialized summary workflow handles claims payment events. The problem is that claims_payment_events_delta dashboard repeatedly scans years of silver transactions for daily rollups. Evidence: SQL history shows expensive repeated aggregation on an otherwise stable reporting query. Constraint: analysts need low-latency aggregates refreshed on a defined cadence. The failure and requirements have been diagnosed; the team must select the lasting pipeline or governance approach. What governed target implementation should the engineer design?
- Create a regular view and promise it stores its results between queries at Regional insurance broker
- Monitor gold refresh age and incremental-refresh failures for claims_payment_events_delta
- Measure refresh needs, compute cost and gold-layer query patterns for claims_payment_events_delta
- Publish a materialized view for precomputed frequently queried aggregates
Explanation
Correct Answer
D:Publish a materialized view for precomputed frequently queried aggregates is the intended durable approach for claims_payment_events_delta: it addresses the verified problem (claims_payment_events_delta dashboard repeatedly scans years of silver transactions for daily rollups) while preserving the required behavior (analysts need low-latency aggregates refreshed on a defined cadence). A diagnostic or monitoring step alone would not create this target-state behavior.
Incorrect Answers
A:Ordinary views do not persist precomputed results. The proposed action (Create a regular view and promise it stores its results between queries at Regional insurance broker) does not meet the requirement that analysts need low-latency aggregates refreshed on a defined cadence, nor does it constitute the target approach.
B:Monitor gold refresh age and incremental-refresh failures for claims_payment_events_delta would help detect future recurrence, but metrics alone do not deliver the target approach. The claims_payment_events_delta requirement is that analysts need low-latency aggregates refreshed on a defined cadence.
C:Measure refresh needs, compute cost and gold-layer query patterns for claims_payment_events_delta could clarify the original fault, but the available evidence already establishes its scope; it is not the target approach now requested. The claims_payment_events_delta workflow still requires that analysts need low-latency aggregates refreshed on a defined cadence.
Question 645
At Education access provider, the notebook task parameters workflow handles student registration records. The problem is that student_registration_records_east notebook hardcodes a development catalog and writes there during production runs. Evidence: job run settings differ from values embedded in the notebook code. Constraint: the same notebook must respect runtime environment configuration. The target approach is approved. The current task is to introduce it safely with bounded scope and rollback. Which change can be introduced with minimal production risk?
- Remove hardcoded target paths and test parameterized notebook invocation
- Edit the source notebook by hand every time the job changes environment at Education access provider
- Pass task parameters through supported notebook widgets or job parameters
- Run dev and prod parameters and confirm each writes only to its own catalog
Explanation
Correct Answer
A:Remove hardcoded target paths and test parameterized notebook invocation provides a bounded way to introduce the approved solution into student_registration_records_east and detect an early mismatch. The existing signal is job run settings differ from values embedded in the notebook code; a staged change protects the service requirement (the same notebook must respect runtime environment configuration) and makes rollback practical.
Incorrect Answers
B:Manual code edits are not reproducible parameterization. The proposed action (Edit the source notebook by hand every time the job changes environment at Education access provider) does not meet the requirement that the same notebook must respect runtime environment configuration, nor does it constitute the controlled implementation step.
C:Pass task parameters through supported notebook widgets or job parameters is a plausible lasting design once chosen, but it does not perform the controlled implementation step at this stage. In the student_registration_records_east scenario, the observed evidence is that job run settings differ from values embedded in the notebook code; the requirement remains: the same notebook must respect runtime environment configuration.
D:Run dev and prod parameters and confirm each writes only to its own catalog is a one-time acceptance check. It does not supply the controlled implementation step for student_registration_records_east; a passing test cannot substitute for the specific task now required.
Question 646
At Community energy cooperative, the shuffle partition tuning workflow handles smart meter readings. The problem is that smart_meter_readings_delta aggregation spends most time scheduling thousands of tiny shuffle tasks. Evidence: Spark UI shows very low per-task data size and excessive scheduling overhead. Constraint: performance changes must be benchmarked against the same output. The new approach is deployed. The current task is to prove correct output and authorized behavior, not just successful installation. Which test should pass before the new flow is accepted?
- Rerun identical aggregation and verify result parity and faster completion
- Increase shuffle partitions without measuring task size or skew at Community energy cooperative
- Inspect shuffle output size, task duration and partition setting for smart_meter_readings_delta
- Apply a bounded change in staging and compare resource cost
Explanation
Correct Answer
A:Rerun identical aggregation and verify result parity and faster completion directly tests the behavior that should change following the correction. With the observed evidence (Spark UI shows very low per-task data size and excessive scheduling overhead), the acceptance result must demonstrate actual data or permission correctness, while retaining the requirement that performance changes must be benchmarked against the same output.
Incorrect Answers
B:Arbitrary partition expansion can increase overhead. The proposed action (Increase shuffle partitions without measuring task size or skew at Community energy cooperative) does not meet the requirement that performance changes must be benchmarked against the same output, nor does it constitute the post-change acceptance test.
C:Inspect shuffle output size, task duration and partition setting for smart_meter_readings_delta could clarify the original fault, but the available evidence already establishes its scope; it is not the post-change acceptance test now requested. The smart_meter_readings_delta workflow still requires that performance changes must be benchmarked against the same output.
D:Apply a bounded change in staging and compare resource cost may introduce a correction, but the question does not ask for installation or migration. A rollout alone cannot replace the post-change acceptance test, especially given the evidence that Spark UI shows very low per-task data size and excessive scheduling overhead. Here, the relevant situation is: the shuffle partition tuning workflow handles smart meter readings. The problem is that smart_meter_readings_delta….
Question 647
A data engineer at Urban housing fund is responsible for rental subsidy payments. The COPY INTO idempotent files issue is that a nightly load of rental_subsidy_payments_east repeats files from yesterday and developers fear duplicate inserts. The supporting evidence is that the same object-storage folder is scanned on each scheduled run. The engineer must ensure that only new source files should be appended without reprocessing prior paths. The failure and requirements have been diagnosed; the team must select the lasting pipeline or governance approach. Which durable Databricks control best addresses the requirement?
- Inspect file-level load history and path visibility for rental_subsidy_payments_east
- Use incremental idempotent COPY INTO against the governed Delta target
- Run unconditional INSERT SELECT over the complete folder on every retry at Urban housing fund
- Track new-file arrivals, load errors and unexpected source-file modifications for rental_subsidy_payments_east
Explanation
Correct Answer
B:Use incremental idempotent COPY INTO against the governed Delta target is the intended durable approach for rental_subsidy_payments_east: it addresses the verified problem (a nightly load of rental_subsidy_payments_east repeats files from yesterday and developers fear duplicate inserts) while preserving the required behavior (only new source files should be appended without reprocessing prior paths). A diagnostic or monitoring step alone would not create this target-state behavior.
Incorrect Answers
A:Inspect file-level load history and path visibility for rental_subsidy_payments_east could clarify the original fault, but the available evidence already establishes its scope; it is not the target approach now requested. The rental_subsidy_payments_east workflow still requires that only new source files should be appended without reprocessing prior paths.
C:A blind append can duplicate previously processed records. The proposed action (Run unconditional INSERT SELECT over the complete folder on every retry at Urban housing fund) does not meet the requirement that only new source files should be appended without reprocessing prior paths, nor does it constitute the target approach.
D:Track new-file arrivals, load errors and unexpected source-file modifications for rental_subsidy_payments_east would help detect future recurrence, but metrics alone do not deliver the target approach. The rental_subsidy_payments_east requirement is that only new source files should be appended without reprocessing prior paths.
Question 648
Consider the rental_subsidy_payments_east workload at Urban housing fund: a nightly load of rental_subsidy_payments_east repeats files from yesterday and developers fear duplicate inserts. Inspection found that the same object-storage folder is scanned on each scheduled run. The governed outcome must ensure that only new source files should be appended without reprocessing prior paths. The target approach is approved. The current task is to introduce it safely with bounded scope and rollback. Which staged rollout best preserves the existing workload?
- Run unconditional INSERT SELECT over the complete folder on every retry at Urban housing fund
- Rerun the command with unchanged files and verify row counts do not double
- Schedule COPY INTO for rental_subsidy_payments_east and test a retry after successful ingestion
- Use incremental idempotent COPY INTO against the governed Delta target
Explanation
Correct Answer
C:Schedule COPY INTO for rental_subsidy_payments_east and test a retry after successful ingestion provides a bounded way to introduce the approved solution into rental_subsidy_payments_east and detect an early mismatch. The existing signal is the same object-storage folder is scanned on each scheduled run; a staged change protects the service requirement (only new source files should be appended without reprocessing prior paths) and makes rollback practical.
Incorrect Answers
A:A blind append can duplicate previously processed records. The proposed action (Run unconditional INSERT SELECT over the complete folder on every retry at Urban housing fund) does not meet the requirement that only new source files should be appended without reprocessing prior paths, nor does it constitute the controlled implementation step.
B:Rerun the command with unchanged files and verify row counts do not double is a one-time acceptance check. It does not supply the controlled implementation step for rental_subsidy_payments_east; a passing test cannot substitute for the specific task now required.
D:Use incremental idempotent COPY INTO against the governed Delta target is a plausible lasting design once chosen, but it does not perform the controlled implementation step at this stage. In the rental_subsidy_payments_east scenario, the observed evidence is that the same object-storage folder is scanned on each scheduled run; the requirement remains: only new source files should be appended without reprocessing prior paths.
Question 649
At Public water district, the service principal data grants workflow handles pipeline flow metrics. The problem is that pipeline_flow_metrics_metro scheduled ingestion succeeds under one employee but fails under automation identity. Evidence: UC audit shows only the employee has the required table rights. Constraint: the job must operate under a managed nonhuman least-privilege principal. The new approach is deployed. The current task is to prove correct output and authorized behavior, not just successful installation. Which test should pass before the new flow is accepted?
- Switch the job identity and verify its input and output access
- Compare job run-as identity and catalog, schema and table grants for pipeline_flow_metrics_metro
- Run the pipeline end-to-end using the service principal and deny unrelated catalogs
- Share a workspace administrator's personal access token with all job owners at Public water district
Explanation
Correct Answer
C:Run the pipeline end-to-end using the service principal and deny unrelated catalogs directly tests the behavior that should change following the correction. With the observed evidence (UC audit shows only the employee has the required table rights), the acceptance result must demonstrate actual data or permission correctness, while retaining the requirement that the job must operate under a managed nonhuman least-privilege principal.
Incorrect Answers
A:Switch the job identity and verify its input and output access may introduce a correction, but the question does not ask for installation or migration. A rollout alone cannot replace the post-change acceptance test, especially given the evidence that UC audit shows only the employee has the required table rights. Here, the relevant situation is: the service principal data grants workflow handles pipeline flow metrics. The problem is that….
B:Compare job run-as identity and catalog, schema and table grants for pipeline_flow_metrics_metro could clarify the original fault, but the available evidence already establishes its scope; it is not the post-change acceptance test now requested. The pipeline_flow_metrics_metro workflow still requires that the job must operate under a managed nonhuman least-privilege principal.
D:Shared personal tokens are fragile and overprivileged. The proposed action (Share a workspace administrator's personal access token with all job owners at Public water district) does not meet the requirement that the job must operate under a managed nonhuman least-privilege principal, nor does it constitute the post-change acceptance test.
Question 650
A data engineer at Air quality nonprofit is responsible for pollution measurements. The Spark SQL NULL semantics issue is that pollution_measurements_central SQL filter misses transactions where the nullable flag was never populated. The supporting evidence is that the predicate uses flag = NULL and always yields unknown. The engineer must ensure that the analysis must distinguish missing from explicitly false flag values. The new approach is deployed. The current task is to prove correct output and authorized behavior, not just successful installation. What evidence establishes that correct output is maintained?
- Test null, false and true rows against the intended filter
- Inspect null frequency and three-valued SQL logic for pollution_measurements_central
- Replace equality-to-null filters with tested null-aware predicates
- Use flag = NULL and assume it matches missing values at Air quality nonprofit
Explanation
Correct Answer
A:Test null, false and true rows against the intended filter directly tests the behavior that should change after the correction. With the observed evidence (the predicate uses flag = NULL and always yields unknown), the acceptance result must demonstrate actual data or permission correctness, while retaining the requirement that the analysis must distinguish missing from explicitly false flag values.
Incorrect Answers
B:Inspect null frequency and three-valued SQL logic for pollution_measurements_central could clarify the original fault, but the available evidence already establishes its scope; it is not the post-change acceptance test now requested. The pollution_measurements_central workflow still requires that the analysis must distinguish missing from explicitly false flag values.
C:Replace equality-to-null filters with tested null-aware predicates may introduce a correction, but the question does not ask for installation or migration. A rollout alone cannot replace the post-change acceptance test, especially given the evidence that the predicate uses flag = NULL and always yields unknown.
D:Sql equality against null is not true. The proposed action (Use flag = NULL and assume it matches missing values at Air quality nonprofit) does not meet the requirement that the analysis must distinguish missing from explicitly false flag values, nor does it constitute the post-change acceptance test.