Databricks Certified Data Engineer Professional Practice Test 11
Question 100
At Pinecrest Holdings, a batch landing zone contains Parquet files written with incompatible column types. Architecture constraint: A merge step fails after a producer changes an integer field to string. Which design best addresses this requirement?
- Choose compatible Avro schema evolution and default-value conventions
- Select append writes and stable event IDs for immutable source events
- Use Auto Loader with managed file discovery and checkpoint/schema locations
- Enforce a common ingestion schema or controlled conversion before append
Explanation
Correct Answer
D:Enforce a common ingestion schema or controlled conversion before append. This design addresses a merge step fails after a producer changes an integer field to string. Parquet schema compatibility matters even when column names are unchanged. It establishes the right contract or architecture before execution changes.
Incorrect Answers
A:Choose compatible Avro schema evolution and default-value conventions. This design action addresses a different case (avro schema evolution), where an older consumer cannot deserialize records from a newer producer schema. Avro writer-reader schema compatibility must be maintained for evolution. In the stated problem, a merge step fails after a producer changes an integer field to string; this choice would not establish the required parquet schema mismatch decision.
B:Select append writes and stable event IDs for immutable source events. This design action addresses a different case (append-only delta bronze), where a nightly overwrite accidentally removes historical audit events. Append-only ingestion keeps previous raw records while adding new batches. In the stated problem, a merge step fails after a producer changes an integer field to string; this choice would not establish the required parquet schema mismatch decision.
C:Use Auto Loader with managed file discovery and checkpoint/schema locations. This design action addresses a different case (auto loader file discovery), where listing the full historical directory every run exceeds the ingestion SLA. Auto Loader is designed for incremental new-file discovery with persisted state. In the stated problem, a merge step fails after a producer changes an integer field to string; this choice would not establish the required parquet schema mismatch decision.
Question 101
Operations at Northfield Analytics owns a workload where a reusable notebook job processes several regional feeds. The previously identified issue was: Code contains hard-coded region IDs and catalog names. Which metric or audit check best detects drift?
- Monitor job run arguments for unexpected target or region values
- Monitor condition outcomes and unexpected repair-task executions
- Alert on unexpected target drift after CI deployments
- Monitor divergence between tested package and deployed code version
Explanation
Correct Answer
A:Monitor job run arguments for unexpected target or region values. This watches the failure mode described by code contains hard-coded region IDs and catalog names. Task parameters allow runtime-specific values without changing notebook source. It supplies a regression signal after remediation.
Incorrect Answers
B:Monitor condition outcomes and unexpected repair-task executions. This monitor action addresses a different case (conditional task branches), where both the normal and repair tasks run despite a branching requirement. Lakeflow Jobs condition tasks can route downstream execution based on a result. In the stated problem, code contains hard-coded region IDs and catalog names; this choice would not establish the required job parameters decision.
C:Alert on unexpected target drift after CI deployments. This monitor action addresses a different case (bundle target overrides), where development job arguments were deployed into the production target. Bundle targets can override environment-specific variables and resource settings. In the stated problem, code contains hard-coded region IDs and catalog names; this choice would not establish the required job parameters decision.
D:Monitor divergence between tested package and deployed code version. This monitor action addresses a different case (transform composability), where business logic depends on notebook state and widgets. DataFrame.transform supports composable DataFrame-to-DataFrame functions. In the stated problem, code contains hard-coded region IDs and catalog names; this choice would not establish the required job parameters decision.
Question 102
Quality assurance at Eastern Star Foods checks a corrected workload where a consumer queries shared Delta table snapshots at different times. The original fault was: Consumers assume the share is a nightly static export. Which verification is most decisive?
- Confirm authorized query succeeds while excluded tables remain inaccessible
- Commit a provider update and confirm recipient observes it under the sharing contract
- Verify recipient sees the selected dataset but not excluded tables
- Confirm the partner reads only the approved shared dataset
Explanation
Correct Answer
B:Commit a provider update and confirm recipient observes it under the sharing contract. This tests the expected result against consumers assume the share is a nightly static export, not simply whether a job or object exists. Delta Sharing exposes current authorized data as the provider updates the source.
Incorrect Answers
A:Confirm authorized query succeeds while excluded tables remain inaccessible. This validate action addresses a different case (foreign catalog permission), where catalog discovery was mistaken for full table access. Unity Catalog usage and object privileges are still required for foreign data objects. In the stated problem, consumers assume the share is a nightly static export; this choice would not establish the required shared table freshness decision.
C:Verify recipient sees the selected dataset but not excluded tables. This validate action addresses a different case (share catalog scope), where the recipient can discover unintended datasets. A Delta Sharing share should contain only explicitly approved assets. In the stated problem, consumers assume the share is a nightly static export; this choice would not establish the required shared table freshness decision.
D:Confirm the partner reads only the approved shared dataset. This validate action addresses a different case (open protocol recipient), where the partner cannot use an internal Databricks-to-Databricks credential flow. Open Delta Sharing can support non-Databricks recipients using the sharing protocol. In the stated problem, consumers assume the share is a nightly static export; this choice would not establish the required shared table freshness decision.
Question 103
At Westridge Research, a source string price unexpectedly arrives where a decimal is required. Architecture constraint: Implicit type coercion risks invalid numeric values being stored. Which design best addresses this requirement?
- Use drop semantics only when invalid records may be excluded by policy
- Separate valid silver rows from a restricted quarantine table
- Specify canonical typed silver columns and an invalid-value quarantine rule
- Define a reviewable additive-column evolution policy
Explanation
Correct Answer
C:Specify canonical typed silver columns and an invalid-value quarantine rule. This design addresses implicit type coercion risks invalid numeric values being stored. Schema validation and explicit casts make type conversion failures observable. It establishes the right contract or architecture before execution changes.
Incorrect Answers
A:Use drop semantics only when invalid records may be excluded by policy. This design action addresses a different case (expectation drop action), where invalid rows currently reach downstream aggregates. A Lakeflow expectation configured to drop invalid rows filters violations from target output. In the stated problem, implicit type coercion risks invalid numeric values being stored; this choice would not establish the required silver schema enforcement decision.
B:Separate valid silver rows from a restricted quarantine table. This design action addresses a different case (quarantine bad records), where dropping invalid rows silently hides upstream quality failures. Quarantine preserves invalid events separately while allowing valid records through. In the stated problem, implicit type coercion risks invalid numeric values being stored; this choice would not establish the required silver schema enforcement decision.
D:Define a reviewable additive-column evolution policy. This design action addresses a different case (controlled schema evolution), where the silver writer rejects every new record after the upstream schema change. Schema evolution must be reviewed rather than enabling unrestricted acceptance for every column. In the stated problem, implicit type coercion risks invalid numeric values being stored; this choice would not establish the required silver schema enforcement decision.
Question 104
At Cobalt Biotech, incoming transaction files continuously arrive and must feed downstream tables. Observed issue: Periodic full-refresh queries miss required incremental freshness. Before applying a fix, which investigation is most directly relevant?
- Inspect Python UDF execution time and batch serialization in Spark UI
- Inspect per-key sequence collisions and upstream log offsets
- Inspect refresh cadence and whether continuous incremental stream processing is required
- Inspect source change pattern and current pipeline refresh behavior
Explanation
Correct Answer
D:Inspect source change pattern and current pipeline refresh behavior. This checks the specific evidence behind periodic full-refresh queries miss required incremental freshness. A streaming table tracks continuously arriving source data incrementally. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Inspect Python UDF execution time and batch serialization in Spark UI. This diagnose action addresses a different case (vectorized udf), where row-by-row Python invocation dominates stage runtime. Pandas UDFs process batches through Arrow rather than per-row Python calls. In the stated problem, periodic full-refresh queries miss required incremental freshness; this choice would not establish the required streaming table choice decision.
B:Inspect per-key sequence collisions and upstream log offsets. This diagnose action addresses a different case (cdc duplicate sequence), where the processing order for tied source events is nondeterministic. A deterministic sequencing contract needs a unique ordered event position. In the stated problem, periodic full-refresh queries miss required incremental freshness; this choice would not establish the required streaming table choice decision.
C:Inspect refresh cadence and whether continuous incremental stream processing is required. This diagnose action addresses a different case (materialized view choice), where each dashboard query recomputes the same expensive aggregation. A materialized view maintains a precomputed query result through refresh. In the stated problem, periodic full-refresh queries miss required incremental freshness; this choice would not establish the required streaming table choice decision.
Question 105
At Lakewood Distribution, a reporting dashboard queries a complex aggregate without requiring row-by-row streaming. A change is required because: Each dashboard query recomputes the same expensive aggregation. Which implementation action most directly corrects the configuration?
- Create the materialized view and configure its supported refresh schedule
- Configure the supported SCD Type 2 flow with keys and sequence
- Apply watermark before supported stateful aggregation with matching event-time column
- Define a streaming table reading the source through a supported streaming connector
Explanation
Correct Answer
A:Create the materialized view and configure its supported refresh schedule. This directly changes the feature or logic behind each dashboard query recomputes the same expensive aggregation. A materialized view maintains a precomputed query result through refresh. It is the concrete configuration for the stated incident.
Incorrect Answers
B:Configure the supported SCD Type 2 flow with keys and sequence. This implement action addresses a different case (scd type 2 history), where overwrite-style updates destroy historical state needed for point-in-time reporting. SCD Type 2 records effective history instead of replacing the previous version. In the stated problem, each dashboard query recomputes the same expensive aggregation; this choice would not establish the required materialized view choice decision.
C:Apply watermark before supported stateful aggregation with matching event-time column. This implement action addresses a different case (event-time watermarking), where old keys remain in state indefinitely without a lateness bound. Watermarks provide a lateness threshold for bounded stateful streaming operations. In the stated problem, each dashboard query recomputes the same expensive aggregation; this choice would not establish the required materialized view choice decision.
D:Define a streaming table reading the source through a supported streaming connector. This implement action addresses a different case (streaming table choice), where periodic full-refresh queries miss required incremental freshness. A streaming table tracks continuously arriving source data incrementally. In the stated problem, each dashboard query recomputes the same expensive aggregation; this choice would not establish the required materialized view choice decision.
Question 106
Operations at Horizon Travel owns a workload where a dashboard repeatedly groups detailed events by day and product. The previously identified issue was: Interactive queries scan billions of detail rows for identical totals. Which metric or audit check best detects drift?
- Monitor duplicate bridge rows and invalid overlapping memberships
- Monitor aggregate freshness and reconciliation differences
- Monitor freshness and quality at each layer
- Monitor changes in workload filter patterns and clustering effectiveness
Explanation
Correct Answer
B:Monitor aggregate freshness and reconciliation differences. This watches the failure mode described by interactive queries scan billions of detail rows for identical totals. A summarized gold table can serve a stable reporting grain efficiently. It supplies a regression signal after remediation.
Incorrect Answers
A:Monitor duplicate bridge rows and invalid overlapping memberships. This monitor action addresses a different case (many-to-many bridge), where joining facts directly to the membership table multiplies totals. A bridge table models many-to-many relationships at an explicit membership grain. In the stated problem, interactive queries scan billions of detail rows for identical totals; this choice would not establish the required aggregate gold model decision.
C:Monitor freshness and quality at each layer. This monitor action addresses a different case (medallion bronze silver gold), where malformed source events contaminate metrics without an intermediate quality boundary. Bronze, silver and gold layers separate raw capture, validated refinement and business presentation. In the stated problem, interactive queries scan billions of detail rows for identical totals; this choice would not establish the required aggregate gold model decision.
D:Monitor changes in workload filter patterns and clustering effectiveness. This monitor action addresses a different case (liquid clustering for fact models), where rigid daily partitions offer little pruning for customer filters. Liquid clustering enables query-aware data layout without high-cardinality partitioning. In the stated problem, interactive queries scan billions of detail rows for identical totals; this choice would not establish the required aggregate gold model decision.
Question 107
At Arbor Medical, an engineering team installs a compiled Python wheel on job compute. Architecture constraint: The wheel works on a developer laptop but fails with a binary compatibility error. Which design best addresses this requirement?
- Use lower and trim native expressions for string standardization
- Isolate parsing logic so it can be debugged outside distributed execution
- Select a supported wheel build for the execution environment
- Move resource definitions into bundle YAML with review gates
Explanation
Correct Answer
C:Select a supported wheel build for the execution environment. This design addresses the wheel works on a developer laptop but fails with a binary compatibility error. Compiled extensions must match the target runtime platform and Python ABI. It establishes the right contract or architecture before execution changes.
Incorrect Answers
A:Use lower and trim native expressions for string standardization. This design action addresses a different case (avoid python udf for sql), where the Python UDF creates an avoidable cross-language execution boundary. Built-in Spark SQL functions can execute in optimized expressions without Python UDF overhead. In the stated problem, the wheel works on a developer laptop but fails with a binary compatibility error; this choice would not establish the required wheel abi compatibility decision.
B:Isolate parsing logic so it can be debugged outside distributed execution. This design action addresses a different case (notebook debugger), where scattered print statements do not reveal the failing object shape. Debugger breakpoints and targeted local repros help inspect code path and values. In the stated problem, the wheel works on a developer laptop but fails with a binary compatibility error; this choice would not establish the required wheel abi compatibility decision.
D:Move resource definitions into bundle YAML with review gates. This design action addresses a different case (databricks cli resource updates), where production configuration drifts from repository source definitions. Declarative deployment describes resources as reviewed version-controlled artifacts. In the stated problem, the wheel works on a developer laptop but fails with a binary compatibility error; this choice would not establish the required wheel abi compatibility decision.
Question 108
A data engineer at Vantage Security must troubleshoot this case: a document-processing stream ingests PDFs and images from cloud object storage. Evidence: A line-oriented text reader corrupts the raw binary payload. What should be inspected first to confirm the failure mechanism?
- Review duplicate file IDs and the ingestion checkpoint state
- Inspect raw XML row element nesting and inferred schema
- Inspect inferred schemas and rescued or null fields in recently ingested rows
- Inspect file media types and current input source format
Explanation
Correct Answer
D:Inspect file media types and current input source format. This checks the specific evidence behind a line-oriented text reader corrupts the raw binary payload. binaryFile input preserves raw file bytes and path metadata rather than decoding lines. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Inspect duplicate file IDs and the ingestion checkpoint state. This diagnose action addresses a different case (retry-safe file loads), where a partially processed input file is loaded twice into Delta. Checkpointed incremental ingestion or stable event-level deduplication prevents repeated effects. In the stated problem, a line-oriented text reader corrupts the raw binary payload; this choice would not establish the required binary payload metadata decision.
B:Inspect raw XML row element nesting and inferred schema. This diagnose action addresses a different case (xml repeating elements), where an XML ingest incorrectly flattens multiple line items into one text field. XML ingestion must specify the repeating row element and preserve nested structure. In the stated problem, a line-oriented text reader corrupts the raw binary payload; this choice would not establish the required binary payload metadata decision.
C:Inspect inferred schemas and rescued or null fields in recently ingested rows. This diagnose action addresses a different case (nested json ingest), where the initial reader assumes one unchanging flat schema and loses new fields. Explicit schema management and rescued data handling protect evolving semi-structured ingestion. In the stated problem, a line-oriented text reader corrupts the raw binary payload; this choice would not establish the required binary payload metadata decision.
Question 109
At Westridge Research, a sales fact table mixes order-level rows with order-line-level rows. A change is required because: Summing revenue double-counts orders with multiple items. Which implementation action most directly corrects the configuration?
- Build the fact keyed by order_id and line_id with additive measures
- Generate dimension surrogate IDs and reference them from fact rows
- Build the bridge and allocate or filter measures according to reporting rules
- Create governed dimension tables and stable foreign keys in the fact
Explanation
Correct Answer
A:Build the fact keyed by order_id and line_id with additive measures. This directly changes the feature or logic behind summing revenue double-counts orders with multiple items. A dimensional fact must have one clearly documented row grain. It is the concrete configuration for the stated incident.
Incorrect Answers
B:Generate dimension surrogate IDs and reference them from fact rows. This implement action addresses a different case (surrogate dimension keys), where joining facts directly on mutable business IDs links history incorrectly. Stable surrogate keys decouple dimensional identity from changing source identifiers. In the stated problem, summing revenue double-counts orders with multiple items; this choice would not establish the required fact table grain decision.
C:Build the bridge and allocate or filter measures according to reporting rules. This implement action addresses a different case (many-to-many bridge), where joining facts directly to the membership table multiplies totals. A bridge table models many-to-many relationships at an explicit membership grain. In the stated problem, summing revenue double-counts orders with multiple items; this choice would not establish the required fact table grain decision.
D:Create governed dimension tables and stable foreign keys in the fact. This implement action addresses a different case (star schema analytics), where a denormalized mixed-grain table produces redundant attributes and expensive corrections. A star schema links a clear-grain fact to reusable dimensions for consistent analytics. In the stated problem, summing revenue double-counts orders with multiple items; this choice would not establish the required fact table grain decision.
Question 110
A deployment owner at Cedar Brook Commerce needs to implement a remedy: multiple engineering teams ship notebook and Python tasks to separate workspaces. The present failure is: Source files and deployment definitions are coupled to one developer workspace. Which change should be applied?
- Update the task dependency list and deploy a tested wheel or requirements set
- Move task source under version control and declare jobs or pipelines in bundle resources
- Use the Databricks CLI to validate and deploy the bundle target
- Set silver task depends_on edges pointing to the bronze task
Explanation
Correct Answer
B:Move task source under version control and declare jobs or pipelines in bundle resources. This directly changes the feature or logic behind source files and deployment definitions are coupled to one developer workspace. A versioned Declarative Automation Bundle defines code and resource configurations separately from the target workspace. It is the concrete configuration for the stated incident.
Incorrect Answers
A:Update the task dependency list and deploy a tested wheel or requirements set. This implement action addresses a different case (library version pinning), where a transitive dependency updated to an incompatible version. Explicit compatible dependency versions improve reproducibility across new compute environments. In the stated problem, source files and deployment definitions are coupled to one developer workspace; this choice would not establish the required bundle repository layout decision.
C:Use the Databricks CLI to validate and deploy the bundle target. This implement action addresses a different case (databricks cli resource updates), where production configuration drifts from repository source definitions. Declarative deployment describes resources as reviewed version-controlled artifacts. In the stated problem, source files and deployment definitions are coupled to one developer workspace; this choice would not establish the required bundle repository layout decision.
D:Set silver task depends_on edges pointing to the bronze task. This implement action addresses a different case (job task dag), where dependent tasks sometimes run before the bronze load finishes. Lakeflow Job task dependencies form a directed acyclic execution graph. In the stated problem, source files and deployment definitions are coupled to one developer workspace; this choice would not establish the required bundle repository layout decision.