Databricks Certified Data Engineer Professional Practice Test 60
Question 590
At Riverbend Energy, a production Structured Streaming query is restarted after an update. A change was deployed to address: A newly selected checkpoint location makes the stream reread historical input. Which test result provides the strongest evidence that the requirement is now met?
- Confirm each key has one selected row matching the maximum sequence
- Restart and confirm previously committed input is not reprocessed unexpectedly
- Restart a fresh session and confirm the imported module version
- Append new files and confirm only new input is processed under checkpoint tracking
Explanation
Correct Answer
B:Restart and confirm previously committed input is not reprocessed unexpectedly. This tests the expected result against a newly selected checkpoint location makes the stream reread historical input, not simply whether a job or object exists. Streaming checkpoint metadata records progress and state for restart continuity.
Incorrect Answers
A:Confirm each key has one selected row matching the maximum sequence. This validate action addresses a different case (window latest record), where the pipeline picks an arbitrary record instead of the newest sequence. ROW_NUMBER ordered by a deterministic sequence within each key selects the desired latest row. In the stated problem, a newly selected checkpoint location makes the stream reread historical input; this choice would not establish the required streaming checkpoint identity decision.
C:Restart a fresh session and confirm the imported module version. This validate action addresses a different case (library precedence), where a notebook-scoped installation overrides the expected cluster version. The runtime can resolve conflicting versions from different installation scopes. In the stated problem, a newly selected checkpoint location makes the stream reread historical input; this choice would not establish the required streaming checkpoint identity decision.
D:Append new files and confirm only new input is processed under checkpoint tracking. This validate action addresses a different case (streaming table choice), where periodic full-refresh queries miss required incremental freshness. A streaming table tracks continuously arriving source data incrementally. In the stated problem, a newly selected checkpoint location makes the stream reread historical input; this choice would not establish the required streaming checkpoint identity decision.
Question 591
At Vantage Security, an open-sharing recipient's credential is nearing expiration. Architecture constraint: Partners risk losing access or using a leaked long-lived credential. Which design best addresses this requirement?
- Grant only the needed foreign object privileges and parent usage
- Place selective supported filters near the remote table access
- Plan rotation with a short overlap and controlled delivery path
- Use an open-sharing recipient with restricted data and secure credential delivery
Explanation
Correct Answer
C:Plan rotation with a short overlap and controlled delivery path. This design addresses partners risk losing access or using a leaked long-lived credential. Credential rotation must be coordinated without unnecessarily broadening permissions. It establishes the right contract or architecture before execution changes.
Incorrect Answers
A:Grant only the needed foreign object privileges and parent usage. This design action addresses a different case (foreign catalog permission), where catalog discovery was mistaken for full table access. Unity Catalog usage and object privileges are still required for foreign data objects. In the stated problem, partners risk losing access or using a leaked long-lived credential; this choice would not establish the required sharing credential rotation decision.
B:Place selective supported filters near the remote table access. This design action addresses a different case (federated pushdown), where an analyst pulls millions of rows before a selective filter. Supported predicate pushdown can reduce data transferred from a federated source. In the stated problem, partners risk losing access or using a leaked long-lived credential; this choice would not establish the required sharing credential rotation decision.
D:Use an open-sharing recipient with restricted data and secure credential delivery. This design action addresses a different case (open protocol recipient), where the partner cannot use an internal Databricks-to-Databricks credential flow. Open Delta Sharing can support non-Databricks recipients using the sharing protocol. In the stated problem, partners risk losing access or using a leaked long-lived credential; this choice would not establish the required sharing credential rotation decision.
Question 592
A data engineer at Oak Valley Retail must troubleshoot this case: a Pandas UDF produces an array of decimals for downstream analytics. Evidence: Returned values cannot be converted to the declared Spark return type. What should be inspected first to confirm the failure mechanism?
- Evaluate ingestion delay percentiles with current watermark threshold
- Compare production deployment variables with bundle target overrides
- Compare deployed job JSON with the version-controlled bundle
- Compare actual Pandas series dtypes with the Spark declared return schema
Explanation
Correct Answer
D:Compare actual Pandas series dtypes with the Spark declared return schema. This checks the specific evidence behind returned values cannot be converted to the declared Spark return type. Pandas UDF return values must conform to the declared Spark data type. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Compare ingestion delay percentiles with current watermark threshold. This diagnose action addresses a different case (late event semantics), where reports silently discard legitimate late events under an aggressive watermark. A watermark trades supported lateness for state size and timeliness. In the stated problem, returned values cannot be converted to the declared Spark return type; this choice would not establish the required udf return schema decision.
B:Compare production deployment variables with bundle target overrides. This diagnose action addresses a different case (bundle target overrides), where development job arguments were deployed into the production target. Bundle targets can override environment-specific variables and resource settings. In the stated problem, returned values cannot be converted to the declared Spark return type; this choice would not establish the required udf return schema decision.
C:Compare deployed job JSON with the version-controlled bundle. This diagnose action addresses a different case (databricks cli resource updates), where production configuration drifts from repository source definitions. Declarative deployment describes resources as reviewed version-controlled artifacts. In the stated problem, returned values cannot be converted to the declared Spark return type; this choice would not establish the required udf return schema decision.
Question 593
A deployment owner at Harborline Utilities needs to implement a remedy: a job repair reruns an external notification task. The present failure is: Customers receive repeated notifications for the same batch. Which change should be applied?
- Deduplicate notification sends using a durable idempotency key
- Adjust appropriate shuffle or executor settings based on evidence
- Correct the missing dependency or incompatible runtime package
- Deploy using an explicit reviewed production target
Explanation
Correct Answer
A:Deduplicate notification sends using a durable idempotency key. This directly changes the feature or logic behind customers receive repeated notifications for the same batch. Task retries need idempotent side effects even when upstream work is reused. It is the concrete configuration for the stated incident.
Incorrect Answers
B:Adjust appropriate shuffle or executor settings based on evidence. This implement action addresses a different case (spark ui failure stage), where operators increase driver memory but failures occur on executors. Spark UI task failures and shuffle/spill metrics distinguish worker pressure from driver issues. In the stated problem, customers receive repeated notifications for the same batch; this choice would not establish the required idempotent retry repair decision.
C:Correct the missing dependency or incompatible runtime package. This implement action addresses a different case (cluster log diagnostics), where the notebook cell shows only a generic stage failure. Driver and executor logs often contain the exact import or JVM error. In the stated problem, customers receive repeated notifications for the same batch; this choice would not establish the required idempotent retry repair decision.
D:Deploy using an explicit reviewed production target. This implement action addresses a different case (target-safe deployment), where the wrong workspace is selected because target is implicit. Explicit bundle target selection prevents accidental environment deployment. In the stated problem, customers receive repeated notifications for the same batch; this choice would not establish the required idempotent retry repair decision.
Question 594
At Pinecrest Holdings, a reporting workload joins orders, products, stores and dates. The known risk is: A denormalized mixed-grain table produces redundant attributes and expensive corrections. After remediation, which ongoing signal best detects recurrence?
- Track overlapping intervals and multiple current versions
- Monitor fact foreign-key consistency and changing dimension attributes
- Monitor unmapped country codes and divergent dimension versions
- Monitor uniqueness violations at the chosen grain
Explanation
Correct Answer
B:Monitor fact foreign-key consistency and changing dimension attributes. This watches the failure mode described by a denormalized mixed-grain table produces redundant attributes and expensive corrections. A star schema links a clear-grain fact to reusable dimensions for consistent analytics. It supplies a regression signal after remediation.
Incorrect Answers
A:Monitor overlapping intervals and multiple current versions. This monitor action addresses a different case (scd type 2 effective dates), where a single current tier overwrites history in the dimension. SCD Type 2 stores effective start/end intervals for distinct historical dimension versions. In the stated problem, a denormalized mixed-grain table produces redundant attributes and expensive corrections; this choice would not establish the required star schema analytics decision.
C:Monitor unmapped country codes and divergent dimension versions. This monitor action addresses a different case (conformed dimensions), where the same country appears under conflicting dimension keys across marts. A conformed dimension provides common attribute definitions across fact models. In the stated problem, a denormalized mixed-grain table produces redundant attributes and expensive corrections; this choice would not establish the required star schema analytics decision.
D:Monitor uniqueness violations at the chosen grain. This monitor action addresses a different case (fact table grain), where summing revenue double-counts orders with multiple items. A dimensional fact must have one clearly documented row grain. In the stated problem, a denormalized mixed-grain table produces redundant attributes and expensive corrections; this choice would not establish the required star schema analytics decision.
Question 595
At Summit Consumer Goods, a large Delta table receives frequent narrow DELETE operations. A change is required because: Each small delete rewrites large data files and increases write cost. Which implementation action most directly corrects the configuration?
- Migrate a compatible task to serverless execution with approved permissions
- Create the managed table under the approved catalog and migrate governed data
- Activate the table feature under approved compatibility requirements
- Run OPTIMIZE or enable managed optimization where supported
Explanation
Correct Answer
C:Activate the table feature under approved compatibility requirements. This directly changes the feature or logic behind each small delete rewrites large data files and increases write cost. Deletion vectors can mark deleted rows without immediate whole-file rewrite when supported. It is the concrete configuration for the stated incident.
Incorrect Answers
A:Migrate a compatible task to serverless execution with approved permissions. This implement action addresses a different case (serverless job fit), where provisioning overhead exceeds useful task execution time. Supported serverless jobs can reduce infrastructure management and startup overhead. In the stated problem, each small delete rewrites large data files and increases write cost; this choice would not establish the required deletion vectors decision.
B:Create the managed table under the approved catalog and migrate governed data. This implement action addresses a different case (managed table maintenance), where each table has custom file cleanup jobs and broad storage permissions. Unity Catalog managed tables provide platform-managed data placement and maintenance capabilities. In the stated problem, each small delete rewrites large data files and increases write cost; this choice would not establish the required deletion vectors decision.
D:Run OPTIMIZE or enable managed optimization where supported. This implement action addresses a different case (small file compaction), where read overhead grows because query plans open excessive small files. OPTIMIZE can compact compatible Delta files without changing logical table contents. In the stated problem, each small delete rewrites large data files and increases write cost; this choice would not establish the required deletion vectors decision.
Question 596
A data engineer at Cedar Brook Commerce must troubleshoot this case: a production workflow fails intermittently overnight. Evidence: Operators only see the most recent run in the Jobs UI. What should be inspected first to confirm the failure mechanism?
- Inspect Query Profile operators for scan volume and pruning
- Inspect current notification settings and run failure statuses
- Inspect allowed notification-destination types and current run triggers
- Inspect run status, task IDs and time windows in supported jobs system tables
Explanation
Correct Answer
D:Inspect run status, task IDs and time windows in supported jobs system tables. This checks the specific evidence behind operators only see the most recent run in the Jobs UI. Jobs system records support historical analysis of runs and tasks. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Inspect Query Profile operators for scan volume and pruning. This diagnose action addresses a different case (query profile scans), where operators assume the query is always CPU-bound. Query Profile identifies scan, pruning and shuffle behavior for SQL workloads. In the stated problem, operators only see the most recent run in the Jobs UI; this choice would not establish the required job operational history decision.
B:Inspect current notification settings and run failure statuses. This diagnose action addresses a different case (job email notification), where operators manually check the run page every morning. Lakeflow Jobs support status-triggered notifications for run outcomes. In the stated problem, operators only see the most recent run in the Jobs UI; this choice would not establish the required job operational history decision.
C:Inspect allowed notification-destination types and current run triggers. This diagnose action addresses a different case (job webhook destination), where only personal email notifications are configured. Supported job notification destinations can integrate workflow status with an incident system. In the stated problem, operators only see the most recent run in the Jobs UI; this choice would not establish the required job operational history decision.
Question 597
A deployment owner at Oak Valley Retail needs to implement a remedy: source customer identifiers can be reused following account mergers. The present failure is: Joining facts directly on mutable business IDs links history incorrectly. Which change should be applied?
- Generate dimension surrogate IDs and reference them from fact rows
- Build the bridge and allocate or filter measures according to reporting rules
- On change, close the prior version and insert a new dimension version
- Materialize the aggregate from quality-controlled silver facts
Explanation
Correct Answer
A:Generate dimension surrogate IDs and reference them from fact rows. This directly changes the feature or logic behind joining facts directly on mutable business IDs links history incorrectly. Stable surrogate keys decouple dimensional identity from changing source identifiers. It is the concrete configuration for the stated incident.
Incorrect Answers
B:Build the bridge and allocate or filter measures according to reporting rules. This implement action addresses a different case (many-to-many bridge), where joining facts directly to the membership table multiplies totals. A bridge table models many-to-many relationships at an explicit membership grain. In the stated problem, joining facts directly on mutable business IDs links history incorrectly; this choice would not establish the required surrogate dimension keys decision.
C:On change, close the prior version and insert a new dimension version. This implement action addresses a different case (scd type 2 effective dates), where a single current tier overwrites history in the dimension. SCD Type 2 stores effective start/end intervals for distinct historical dimension versions. In the stated problem, joining facts directly on mutable business IDs links history incorrectly; this choice would not establish the required surrogate dimension keys decision.
D:Materialize the aggregate from quality-controlled silver facts. This implement action addresses a different case (aggregate gold model), where interactive queries scan billions of detail rows for identical totals. A summarized gold table can serve a stable reporting grain efficiently. In the stated problem, joining facts directly on mutable business IDs links history incorrectly; this choice would not establish the required surrogate dimension keys decision.
Question 598
At Eastern Star Foods, a declarative pipeline fails with a schema incompatibility. Observed issue: The only visible dashboard status is FAILED without a dataset name. Before applying a fix, which investigation is most directly relevant?
- Inspect executor stderr and driver logs near the failed stage
- Inspect pipeline event log update IDs and error messages
- Inspect Python version, wheel tags and package resolution logs
- Inspect failed executor task metrics and exception messages
Explanation
Correct Answer
B:Inspect pipeline event log update IDs and error messages. This checks the specific evidence behind the only visible dashboard status is FAILED without a dataset name. Pipeline event logs identify failing update and dataset messages. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Inspect executor stderr and driver logs near the failed stage. This diagnose action addresses a different case (cluster log diagnostics), where the notebook cell shows only a generic stage failure. Driver and executor logs often contain the exact import or JVM error. In the stated problem, the only visible dashboard status is FAILED without a dataset name; this choice would not establish the required lakeflow event error classes decision.
C:Inspect Python version, wheel tags and package resolution logs. This diagnose action addresses a different case (python dependency triage), where a package import error appears only on the target compute. Runtime-specific wheel compatibility and pinning must be tested in CI. In the stated problem, the only visible dashboard status is FAILED without a dataset name; this choice would not establish the required lakeflow event error classes decision.
D:Inspect failed executor task metrics and exception messages. This diagnose action addresses a different case (spark ui failure stage), where operators increase driver memory but failures occur on executors. Spark UI task failures and shuffle/spill metrics distinguish worker pressure from driver issues. In the stated problem, the only visible dashboard status is FAILED without a dataset name; this choice would not establish the required lakeflow event error classes decision.
Question 599
At Arbor Medical, short scheduled ETL tasks waste time starting and idling classic clusters. A change was deployed to address: Provisioning overhead exceeds useful task execution time. Which test result provides the strongest evidence that the requirement is now met?
- Run small and large test inputs and compare execution strategies
- Reconcile changed rows and compare scanned file counts
- Compare run results, startup latency and billed usage
- Compare file count and query result checksums before and after
Explanation
Correct Answer
C:Compare run results, startup latency and billed usage. This tests the expected result against provisioning overhead exceeds useful task execution time, not simply whether a job or object exists. Supported serverless jobs can reduce infrastructure management and startup overhead.
Incorrect Answers
A:Run small and large test inputs and compare execution strategies. This validate action addresses a different case (adaptive execution), where static join strategy performs poorly at varying input volumes. Adaptive Query Execution can adjust supported shuffle partitions and join strategies at runtime. In the stated problem, provisioning overhead exceeds useful task execution time; this choice would not establish the required serverless job fit decision.
B:Reconcile changed rows and compare scanned file counts. This validate action addresses a different case (predicate-limited merge), where the merge scans unrelated historical partitions. Constraining merge matches to relevant partitions can reduce unnecessary target scanning. In the stated problem, provisioning overhead exceeds useful task execution time; this choice would not establish the required serverless job fit decision.
D:Compare file count and query result checksums before and after. This validate action addresses a different case (small file compaction), where read overhead grows because query plans open excessive small files. OPTIMIZE can compact compatible Delta files without changing logical table contents. In the stated problem, provisioning overhead exceeds useful task execution time; this choice would not establish the required serverless job fit decision.
Question 600
An architecture review at Fairhaven Transport covers a workload where a downstream incremental aggregate misses updates to an existing Delta table. The unresolved constraint is: The consumer reads only newly appended rows and cannot see updates or deletes. Which design is most appropriate?
- Keep filter predicates aligned with the stored partition column
- Use supported AQE with appropriate shuffle and join settings
- Cache only frequently reused and appropriately sized intermediate data
- Use CDF with committed version checkpoints for incremental changes
Explanation
Correct Answer
D:Use CDF with committed version checkpoints for incremental changes. This design addresses the consumer reads only newly appended rows and cannot see updates or deletes. Delta Change Data Feed exposes row-level changes for supported versions when enabled. It establishes the right contract or architecture before execution changes.
Incorrect Answers
A:Keep filter predicates aligned with the stored partition column. This design action addresses a different case (partition pruning), where a query wraps business_date in a transformation and reads many partitions. Direct predicates on partition keys can enable partition pruning. In the stated problem, the consumer reads only newly appended rows and cannot see updates or deletes; this choice would not establish the required change data feed consumers decision.
B:Use supported AQE with appropriate shuffle and join settings. This design action addresses a different case (adaptive execution), where static join strategy performs poorly at varying input volumes. Adaptive Query Execution can adjust supported shuffle partitions and join strategies at runtime. In the stated problem, the consumer reads only newly appended rows and cannot see updates or deletes; this choice would not establish the required change data feed consumers decision.
C:Cache only frequently reused and appropriately sized intermediate data. This design action addresses a different case (cache reuse), where repeated recomputation dominates each branch's execution. Persistence can avoid repeated work if reused data fits the memory budget. In the stated problem, the consumer reads only newly appended rows and cannot see updates or deletes; this choice would not establish the required change data feed consumers decision.