Databricks Certified Data Engineer Associate Practice Test 40
Question 390
Consider the clinical_appointment_reports_island workload at Regional hospital network: clinical_appointment_reports_island transformation intermittently fails with executor out-of-memory. Inspection found that Spark UI shows large skewed partitions and executor lost errors. The governed outcome must ensure that the job must process full input safely without excluding valid rows. The failure and requirements have been diagnosed; the team must select the lasting pipeline or governance approach. Which data-engineering approach resolves the documented weakness?
- Choose balanced partitioning and appropriate executor memory for the workload
- Inspect task partition sizes, execution plan and memory overhead for clinical_appointment_reports_island
- Filter out the largest partition without business approval at Regional hospital network
- Monitor OOM retries, failed executor count and memory utilization in clinical_appointment_reports_island
Explanation
Correct Answer
A:Choose balanced partitioning and appropriate executor memory for the workload is the intended durable approach for clinical_appointment_reports_island: it addresses the verified problem (clinical_appointment_reports_island transformation intermittently fails with executor out-of-memory) while preserving the required behavior (the job must process full input safely without excluding valid rows). A diagnostic or monitoring step alone would not create this target-state behavior.
Incorrect Answers
B:Inspect task partition sizes, execution plan and memory overhead for clinical_appointment_reports_island could clarify the original fault, but the available evidence already establishes its scope; it is not the target approach now requested. The clinical_appointment_reports_island workflow still requires that the job must process full input safely without excluding valid rows.
C:Dropping data to avoid oom violates completeness. The proposed action (Filter out the largest partition without business approval at Regional hospital network) does not meet the requirement that the job must process full input safely without excluding valid rows, nor does it constitute the target approach.
D:Monitor OOM retries, failed executor count and memory utilization in clinical_appointment_reports_island would help detect future recurrence, but metrics alone do not deliver the target approach. The clinical_appointment_reports_island requirement is that the job must process full input safely without excluding valid rows.
Question 391
The experiment_records_metro pipeline at University research consortium has this failure: the experiment_records_metro partner API returns 100 rows per page but the batch stores only the first page. The team observed that response metadata includes a next cursor that the client ignores. Business requirement: all authorized records must be imported without exceeding API limits. The failure and requirements have been diagnosed; the team must select the lasting pipeline or governance approach. What lasting solution should be selected for the verified conditions?
- Monitor page completeness, retries and expired cursors for experiment_records_metro
- Implement cursor pagination with bounded retries and idempotent destination keys
- Inspect response cursor, throttle status and stable identity for experiment_records_metro
- Stop after one successful HTTP 200 response at University research consortium
Explanation
Correct Answer
B:Implement cursor pagination with bounded retries and idempotent destination keys is the intended durable approach for experiment_records_metro: it addresses the verified problem (the experiment_records_metro partner API returns 100 rows per page but the batch stores only the first page) while preserving the required behavior (all authorized records must be imported without exceeding API limits). A diagnostic or monitoring step alone would not create this target-state behavior.
Incorrect Answers
A:Monitor page completeness, retries and expired cursors for experiment_records_metro would help detect future recurrence, but metrics alone do not deliver the target approach. The experiment_records_metro requirement is that all authorized records must be imported without exceeding API limits.
C:Inspect response cursor, throttle status and stable identity for experiment_records_metro could clarify the original fault, but the available evidence already establishes its scope; it is not the target approach now requested. The experiment_records_metro workflow still requires that all authorized records must be imported without exceeding API limits.
D:A successful first page does not establish completeness. The proposed action (Stop after one successful HTTP 200 response at University research consortium) does not meet the requirement that all authorized records must be imported without exceeding API limits, nor does it constitute the target approach.
Question 392
The consent_related_data_changes_west pipeline at Research clinical trial group has this failure: the consent_related_data_changes_west relational source emits inserts and updates but the lakehouse target retains multiple current versions. The team observed that change records carry primary keys and update sequence numbers. Business requirement: the destination must represent the latest source state without losing required history. The failure and requirements have been diagnosed; the team must select the lasting pipeline or governance approach. What lasting solution should be selected for the verified conditions?
- Use supported CDC apply logic with keys, ordering and defined delete handling
- Inspect key uniqueness, update order and delete semantics in consent_related_data_changes_west
- Monitor CDC lag, duplicate keys and discarded out-of-order updates for consent_related_data_changes_west
- Append every change as a new current row without keys at Research clinical trial group
Explanation
Correct Answer
A:Use supported CDC apply logic with keys, ordering and defined delete handling is the intended durable approach for consent_related_data_changes_west: it addresses the verified problem (the consent_related_data_changes_west relational source emits inserts and updates but the lakehouse target retains multiple current versions) while preserving the required behavior (the destination must represent the latest source state without losing required history). A diagnostic or monitoring step alone would not create this target-state behavior.
Incorrect Answers
B:Inspect key uniqueness, update order and delete semantics in consent_related_data_changes_west could clarify the original fault, but the available evidence already establishes its scope; it is not the target approach now requested. The consent_related_data_changes_west workflow still requires that the destination must represent the latest source state without losing required history.
C:Monitor CDC lag, duplicate keys and discarded out-of-order updates for consent_related_data_changes_west would help detect future recurrence, but metrics alone do not deliver the target approach. The consent_related_data_changes_west requirement is that the destination must represent the latest source state without losing required history.
D:An append-only current-state table does not resolve updates or deletes. The proposed action (Append every change as a new current row without keys at Research clinical trial group) does not meet the requirement that the destination must represent the latest source state without losing required history, nor does it constitute the target approach.
Question 393
A data engineer at Airport facilities operator is responsible for maintenance work orders. The broadcast small dimension issue is that joining maintenance_work_orders_metro fact records to a tiny reference table produces a large shuffle. The supporting evidence is that Spark UI shows both inputs shuffled despite one lookup occupying a few megabytes. The engineer must ensure that join output must retain key equality and correct multiplicity. The team has not selected a remedy. It first needs evidence that distinguishes the actual dependency or data fault. What analysis is necessary before deciding on remediation?
- Review table sizes, statistics and shuffle metrics for maintenance_work_orders_metro
- Trial a broadcast hint with representative fact cardinality and verify output
- CrossJoin the entire fact and reference datasets then filter after collection at Airport facilities operator
- Broadcast the eligible small reference DataFrame when appropriate to avoid a large shuffle
Explanation
Correct Answer
A:Review table sizes, statistics and shuffle metrics for maintenance_work_orders_metro directly investigates the evidence (Spark UI shows both inputs shuffled despite one lookup occupying a few megabytes) and distinguishes possible causes of joining maintenance_work_orders_metro fact records to a tiny reference table produces a large shuffle. It establishes what must be corrected without assuming a fix before verifying the stated business requirement: join output must retain key equality and correct multiplicity.
Incorrect Answers
B:Trial a broadcast hint with representative fact cardinality and verify output may introduce a correction, but the question does not ask for installation or migration. A rollout alone cannot replace the required pre-change investigation, especially given the evidence that Spark UI shows both inputs shuffled despite one lookup occupying a few megabytes.
C:A cartesian expansion wastes resources and risks mis-joins. The proposed action (CrossJoin the entire fact and reference datasets then filter after collection at Airport facilities operator) does not meet the requirement that join output must retain key equality and correct multiplicity, nor does it constitute the required pre-change investigation.
D:Broadcast the eligible small reference DataFrame when appropriate to avoid a large shuffle is a plausible lasting design once chosen, but it does not perform the required pre-change investigation at this stage. In the maintenance_work_orders_metro scenario, the observed evidence is that Spark UI shows both inputs shuffled despite one lookup occupying a few megabytes; the requirement remains: join output must retain key equality and correct multiplicity.
Question 394
At Manufacturing quality network, the factory defect records workflow depends on factory_defect_records_central. Engineers report that factory_defect_records_central Delta table suffers frequent filter-pattern changes and costly partition rewrites; the evidence is that workload history alternates date and customer search predicates across teams. They require that layout optimization must adapt without a rigid proliferating partition tree. The target approach is approved. The current task is to introduce it safely with bounded scope and rollback. Which controlled implementation step is appropriate now?
- Enable clustering on a pilot target and run supported maintenance operations
- Consider Liquid Clustering on appropriate columns with supported table features
- Compare query scan size and update overhead for representative filter sets
- Create thousands of tiny static partitions for every possible predicate at Manufacturing quality network
Explanation
Correct Answer
A:Enable clustering on a pilot target and run supported maintenance operations provides a bounded way to introduce the approved solution into factory_defect_records_central and detect an early mismatch. The existing signal is workload history alternates date and customer search predicates across teams; a staged change protects the service requirement (layout optimization must adapt without a rigid proliferating partition tree) and makes rollback practical.
Incorrect Answers
B:Consider Liquid Clustering on appropriate columns with supported table features is a plausible lasting design once chosen, but it does not perform the controlled implementation step at this stage. In the factory_defect_records_central scenario, the observed evidence is that workload history alternates date and customer search predicates across teams; the requirement remains: layout optimization must adapt without a rigid proliferating partition tree.
C:Compare query scan size and update overhead for representative filter sets is a one-time acceptance check. It does not supply the controlled implementation step for factory_defect_records_central; a passing test cannot substitute for the specific task now required.
D:Overpartitioning drives metadata overhead and inflexible layout. The proposed action (Create thousands of tiny static partitions for every possible predicate at Manufacturing quality network) does not meet the requirement that layout optimization must adapt without a rigid proliferating partition tree, nor does it constitute the controlled implementation step.
Question 395
A data engineer at Community energy cooperative is responsible for smart meter readings. The column masking PII issue is that smart_meter_readings_delta table exposes national IDs to analysts who need only transaction amounts. The supporting evidence is that query review shows raw identifier values in common reports. The engineer must ensure that users without an approved role must see masked sensitive columns. The failure and requirements have been diagnosed; the team must select the lasting pipeline or governance approach. Which durable Databricks control best addresses the requirement?
- Identify sensitive attributes, entitled roles and current query exposure for smart_meter_readings_delta
- Monitor new PII columns, mask failures and unapproved policy changes for smart_meter_readings_delta
- Rename identifier column to a harmless label while leaving values visible at Community energy cooperative
- Apply Unity Catalog column masks or suitable centrally governed masking policy
Explanation
Correct Answer
D:Apply Unity Catalog column masks or suitable centrally governed masking policy is the intended durable approach for smart_meter_readings_delta: it addresses the verified problem (smart_meter_readings_delta table exposes national IDs to analysts who need only transaction amounts) while preserving the required behavior (users without an approved role must see masked sensitive columns). A diagnostic or monitoring step alone would not create this target-state behavior.
Incorrect Answers
A:Identify sensitive attributes, entitled roles and current query exposure for smart_meter_readings_delta could clarify the original fault, but the available evidence already establishes its scope; it is not the target approach now requested. The smart_meter_readings_delta workflow still requires that users without an approved role must see masked sensitive columns.
B:Monitor new PII columns, mask failures and unapproved policy changes for smart_meter_readings_delta would help detect future recurrence, but metrics alone do not deliver the target approach. The smart_meter_readings_delta requirement is that users without an approved role must see masked sensitive columns.
C:Renaming a column does not protect its contents. The proposed action (Rename identifier column to a harmless label while leaving values visible at Community energy cooperative) does not meet the requirement that users without an approved role must see masked sensitive columns, nor does it constitute the target approach.
Question 396
Consider the harvest_purchase_orders_metro workload at National farm supplier: harvest_purchase_orders_metro rows have statuses requiring a three-way risk category. Inspection found that a simple boolean expression sends null status to a high-risk category. The governed outcome must ensure that business rules distinguish active, cancelled and unknown statuses. The new approach is deployed. The current task is to prove correct output and authorized behavior, not just successful installation. How should the team verify the original fault has been addressed?
- Deploy a deterministic mapping table and validate the input states
- Enumerate source states and null handling for the harvest_purchase_orders_metro flag
- Test all named states plus null and an unrecognized state
- Treat null as an active subscription without evidence at National farm supplier
Explanation
Correct Answer
C:Test all named states plus null and an unrecognized state directly tests the behavior that should change following the correction. With the observed evidence (a simple boolean expression sends null status to a high-risk category), the acceptance result must demonstrate actual data or permission correctness, while retaining the requirement that business rules distinguish active, cancelled and unknown statuses.
Incorrect Answers
A:Deploy a deterministic mapping table and validate the input states may introduce a correction, but the question does not ask for installation or migration. A rollout alone cannot replace the post-change acceptance test, especially given the evidence that a simple boolean expression sends null status to a high-risk category. This matters here because consider the harvest_purchase_orders_metro workload at National farm supplier: harvest_purchase_orders_metro rows….
B:Enumerate source states and null handling for the harvest_purchase_orders_metro flag could clarify the original fault, but the available evidence already establishes its scope; it is not the post-change acceptance test now requested. The harvest_purchase_orders_metro workflow still requires that business rules distinguish active, cancelled and unknown statuses.
D:Missing status cannot be assumed to mean active. The proposed action (Treat null as an active subscription without evidence at National farm supplier) does not meet the requirement that business rules distinguish active, cancelled and unknown statuses, nor does it constitute the post-change acceptance test.
Question 397
A data engineer at Air quality nonprofit is responsible for pollution measurements. The Photon and execution engine issue is that the pollution_measurements_north SQL aggregation runs slowly on eligible compute despite a stable dataset. The supporting evidence is that query profiling identifies CPU-bound SQL operations with little external I/O. The engineer must ensure that performance changes must be measured at equivalent compute and data volume. The target approach is approved. The current task is to introduce it safely with bounded scope and rollback. What is the safest migration or deployment sequence?
- Enable the appropriate engine on a pilot compute configuration and compare plans
- Evaluate Photon-capable compute for supported SQL and DataFrame workloads
- Assume Photon eliminates every network, skew and access-permission problem at Air quality nonprofit
- Benchmark the same pollution_measurements_north query and dataset on both configurations
Explanation
Correct Answer
A:Enable the appropriate engine on a pilot compute configuration and compare plans provides a bounded way to introduce the approved solution into pollution_measurements_north and detect an early mismatch. The existing signal is query profiling identifies CPU-bound SQL operations with little external I/O; a staged change protects the service requirement (performance changes must be measured at equivalent compute and data volume) and makes rollback practical.
Incorrect Answers
B:Evaluate Photon-capable compute for supported SQL and DataFrame workloads is a plausible lasting design once chosen, but it does not perform the controlled implementation step at this stage. In the pollution_measurements_north scenario, the observed evidence is that query profiling identifies CPU-bound SQL operations with little external I/O; the requirement remains: performance changes must be measured at equivalent compute and data volume.
C:An execution engine cannot cure unrelated input or authorization constraints. The proposed action (Assume Photon eliminates every network, skew and access-permission problem at Air quality nonprofit) does not meet the requirement that performance changes must be measured at equivalent compute and data volume, nor does it constitute the controlled implementation step.
D:Benchmark the same pollution_measurements_north query and dataset on both configurations is a one-time acceptance check. It does not supply the controlled implementation step for pollution_measurements_north; a passing test cannot substitute for the specific task now required.
Question 398
A data engineer at Regional telecom operator is responsible for subscriber provisioning logs. The Unity Catalog lineage issue is that subscriber_provisioning_logs_metro Gold dashboard shows wrong revenue and owners cannot trace the upstream dataset. The supporting evidence is that lineage view is missing a newly added silver transformation. The engineer must ensure that impact analysis needs upstream and downstream dependency visibility. The first acceptance run passed. The current task is to detect future regressions and changes over time. What repeatable measure detects this failure after go-live?
- Rely on file naming conventions as proof of full lineage at Regional telecom operator
- Use Unity Catalog lineage and supported pipeline metadata to trace data dependencies
- Compare inferred lineage to an actual source-to-gold test transformation
- Monitor missing lineage links and changed downstream consumers for subscriber_provisioning_logs_metro
Explanation
Correct Answer
D:Monitor missing lineage links and changed downstream consumers for subscriber_provisioning_logs_metro creates recurring visibility into the failure mode (subscriber_provisioning_logs_metro Gold dashboard shows wrong revenue and owners cannot trace the upstream dataset). Unlike a single successful test, the measure can reveal drift, lag, or new failures after later source and deployment changes while impact analysis needs upstream and downstream dependency visibility.
Incorrect Answers
A:Names alone cannot prove transformations and dependencies. The proposed action (Rely on file naming conventions as proof of full lineage at Regional telecom operator) does not meet the requirement that impact analysis needs upstream and downstream dependency visibility, nor does it constitute continuing operational detection.
B:Use Unity Catalog lineage and supported pipeline metadata to trace data dependencies is a plausible lasting design once chosen, but it does not perform continuing operational detection at this stage. In the subscriber_provisioning_logs_metro scenario, the observed evidence is that lineage view is missing a newly added silver transformation; the requirement remains: impact analysis needs upstream and downstream dependency visibility.
C:Compare inferred lineage to an actual source-to-gold test transformation is a one-time acceptance check. It does not supply continuing operational detection for subscriber_provisioning_logs_metro; a passing test cannot substitute for the specific task now required.
Question 399
At Coastal freight alliance, the shipment manifests workflow depends on shipment_manifests_west. Engineers report that shipment_manifests_west access exception uses a DENY policy without verifying feature support and scope; the evidence is that review finds the target resource is outside the supported ABAC deny-policy securable types. They require that authorization must use supported policy semantics without assuming beta behavior. The new approach is deployed. The current task is to prove correct output and authorized behavior, not just successful installation. Which post-change acceptance check is strongest?
- Pilot the policy in a test catalog and confirm evaluation result
- Assume any DENY syntax universally applies to all Unity Catalog objects at Coastal freight alliance
- Test denied and permitted identities on the exact target object
- Verify exact privilege, supported securable type and ABAC DENY feature status for shipment_manifests_west
Explanation
Correct Answer
C:Test denied and permitted identities on the exact target object directly tests the behavior that should change following the correction. With the observed evidence (review finds the target resource is outside the supported ABAC deny-policy securable types), the acceptance result must demonstrate actual data or permission correctness, while retaining the requirement that authorization must use supported policy semantics without assuming beta behavior.
Incorrect Answers
A:Pilot the policy in a test catalog and confirm evaluation result may introduce a correction, but the question does not ask for installation or migration. A rollout alone cannot replace the post-change acceptance test, especially given the evidence that review finds the target resource is outside the supported ABAC deny-policy securable types. Here, the relevant situation is: the shipment manifests workflow depends on shipment_manifests_west. Engineers report that shipment_manifests_west….
B:Beta deny policy semantics are not universal to every securable type. The proposed action (Assume any DENY syntax universally applies to all Unity Catalog objects at Coastal freight alliance) does not meet the requirement that authorization must use supported policy semantics without assuming beta behavior, nor does it constitute the post-change acceptance test.
D:Verify exact privilege, supported securable type and ABAC DENY feature status for shipment_manifests_west could clarify the original fault, but the available evidence already establishes its scope; it is not the post-change acceptance test now requested. The shipment_manifests_west workflow still requires that authorization must use supported policy semantics without assuming beta behavior.
Question 400
Consider the class_enrollment_data_delta workload at International language school: the class_enrollment_data_delta partner API returns 100 rows per page but the batch stores only the first page. Inspection found that response metadata includes a next cursor that the client ignores. The governed outcome must ensure that all authorized records must be imported without exceeding API limits. The team has not selected a remedy. It first needs evidence that distinguishes the actual dependency or data fault. Which evidence-gathering action should precede an architectural change?
- Test pagination on a multi-page source with backoff for HTTP 429
- Inspect response cursor, throttle status and stable identity for class_enrollment_data_delta
- Stop after one successful HTTP 200 response at International language school
- Implement cursor pagination with bounded retries and idempotent destination keys
Explanation
Correct Answer
B:Inspect response cursor, throttle status and stable identity for class_enrollment_data_delta directly investigates the evidence (response metadata includes a next cursor that the client ignores) and distinguishes possible causes of the class_enrollment_data_delta partner API returns 100 rows per page but the batch stores only the first page. It establishes what must be corrected without assuming a fix before verifying the stated business requirement: all authorized records must be imported without exceeding API limits.
Incorrect Answers
A:Test pagination on a multi-page source with backoff for HTTP 429 may introduce a correction, but the question does not ask for installation or migration. A rollout alone cannot replace the required pre-change investigation, especially given the evidence that response metadata includes a next cursor that the client ignores. This matters here because consider the class_enrollment_data_delta workload at International language school: the class_enrollment_data_delta….
C:A successful first page does not establish completeness. The proposed action (Stop after one successful HTTP 200 response at International language school) does not meet the requirement that all authorized records must be imported without exceeding API limits, nor does it constitute the required pre-change investigation.
D:Implement cursor pagination with bounded retries and idempotent destination keys is a plausible lasting design once chosen, but it does not perform the required pre-change investigation at this stage. In the class_enrollment_data_delta scenario, the observed evidence is that response metadata includes a next cursor that the client ignores; the requirement remains: all authorized records must be imported without exceeding API limits.