Databricks Certified Data Engineer Professional Practice Test 40
Question 390
At Riverbend Energy, a logistics partner sends XML with repeated line-item elements. Architecture constraint: An XML ingest incorrectly flattens multiple line items into one text field. Which design best addresses this requirement?
- Use a binaryFile ingest path for nontext payloads
- Model repeated child elements as arrays or separate normalized child records
- Use one durable checkpoint and clear idempotency keys for replay
- Choose compatible Avro schema evolution and default-value conventions
Explanation
Correct Answer
B:Model repeated child elements as arrays or separate normalized child records. This design addresses an XML ingest incorrectly flattens multiple line items into one text field. XML ingestion must specify the repeating row element and preserve nested structure. It establishes the right contract or architecture before execution changes.
Incorrect Answers
A:Use a binaryFile ingest path for nontext payloads. This design action addresses a different case (binary payload metadata), where a line-oriented text reader corrupts the raw binary payload. binaryFile input preserves raw file bytes and path metadata rather than decoding lines. In the stated problem, an XML ingest incorrectly flattens multiple line items into one text field; this choice would not establish the required xml repeating elements decision.
C:Use one durable checkpoint and clear idempotency keys for replay. This design action addresses a different case (retry-safe file loads), where a partially processed input file is loaded twice into Delta. Checkpointed incremental ingestion or stable event-level deduplication prevents repeated effects. In the stated problem, an XML ingest incorrectly flattens multiple line items into one text field; this choice would not establish the required xml repeating elements decision.
D:Choose compatible Avro schema evolution and default-value conventions. This design action addresses a different case (avro schema evolution), where an older consumer cannot deserialize records from a newer producer schema. Avro writer-reader schema compatibility must be maintained for evolution. In the stated problem, an XML ingest incorrectly flattens multiple line items into one text field; this choice would not establish the required xml repeating elements decision.
Question 391
At Summit Consumer Goods, a remote warehouse is slow under a broad federated query. Observed issue: An analyst pulls millions of rows before a selective filter. Before applying a fix, which investigation is most directly relevant?
- Inspect USE CATALOG, USE SCHEMA and SELECT grants for the identity
- Inspect recipient credential expiration and access history
- Inspect the remote query plan and filter pushdown support
- Inspect share contents and approved sharing manifest
Explanation
Correct Answer
C:Inspect the remote query plan and filter pushdown support. This checks the specific evidence behind an analyst pulls millions of rows before a selective filter. Supported predicate pushdown can reduce data transferred from a federated source. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Inspect USE CATALOG, USE SCHEMA and SELECT grants for the identity. This diagnose action addresses a different case (foreign catalog permission), where catalog discovery was mistaken for full table access. Unity Catalog usage and object privileges are still required for foreign data objects. In the stated problem, an analyst pulls millions of rows before a selective filter; this choice would not establish the required federated pushdown decision.
B:Inspect recipient credential expiration and access history. This diagnose action addresses a different case (sharing credential rotation), where partners risk losing access or using a leaked long-lived credential. Credential rotation must be coordinated without unnecessarily broadening permissions. In the stated problem, an analyst pulls millions of rows before a selective filter; this choice would not establish the required federated pushdown decision.
D:Inspect share contents and approved sharing manifest. This diagnose action addresses a different case (share catalog scope), where the recipient can discover unintended datasets. A Delta Sharing share should contain only explicitly approved assets. In the stated problem, an analyst pulls millions of rows before a selective filter; this choice would not establish the required federated pushdown decision.
Question 392
Operations at Lakewood Distribution owns a workload where a new workspace team needs read-only access to a vendor landing bucket. The previously identified issue was: Engineers propose a credential allowing access to all company buckets. Which metric or audit check best detects drift?
- Audit ownership changes and privilege grants
- Monitor view grants and changes to projected columns
- Monitor view definition changes and access to raw source
- Monitor new external-location grants and credential usage
Explanation
Correct Answer
D:Monitor new external-location grants and credential usage. This watches the failure mode described by engineers propose a credential allowing access to all company buckets. Unity Catalog storage credentials and external locations can constrain storage access. It supplies a regression signal after remediation.
Incorrect Answers
A:Audit ownership changes and privilege grants. This monitor action addresses a different case (separation of duties), where all developers currently own production catalogs and can grant access. Separate deploy and governance roles reduce accidental privilege escalation. In the stated problem, engineers propose a credential allowing access to all company buckets; this choice would not establish the required storage credential scope decision.
B:Monitor view grants and changes to projected columns. This monitor action addresses a different case (secure view sharing), where granting SELECT on the raw fact table violates minimization. A governed aggregation view can expose only approved derived information. In the stated problem, engineers propose a credential allowing access to all company buckets; this choice would not establish the required storage credential scope decision.
C:Monitor view definition changes and access to raw source. This monitor action addresses a different case (sensitive data suppression), where a broad SELECT still includes unstructured notes. Column suppression and data minimization prevent unnecessary sensitive-field release. In the stated problem, engineers propose a credential allowing access to all company buckets; this choice would not establish the required storage credential scope decision.
Question 393
An architecture review at Oak Valley Retail covers a workload where a Pandas UDF produces an array of decimals for downstream analytics. The unresolved constraint is: Returned values cannot be converted to the declared Spark return type. Which design is most appropriate?
- Choose a return type and null-handling contract matching the output
- Choose a materialized view for refreshable precomputed analytics
- Choose tracked history columns and a Type 2 CDC target
- Choose compatible pinned versions using a tested dependency lock strategy
Explanation
Correct Answer
A:Choose a return type and null-handling contract matching the output. This design addresses returned values cannot be converted to the declared Spark return type. Pandas UDF return values must conform to the declared Spark data type. It establishes the right contract or architecture before execution changes.
Incorrect Answers
B:Choose a materialized view for refreshable precomputed analytics. This design action addresses a different case (materialized view choice), where each dashboard query recomputes the same expensive aggregation. A materialized view maintains a precomputed query result through refresh. In the stated problem, returned values cannot be converted to the declared Spark return type; this choice would not establish the required udf return schema decision.
C:Choose tracked history columns and a Type 2 CDC target. This design action addresses a different case (scd type 2 history), where overwrite-style updates destroy historical state needed for point-in-time reporting. SCD Type 2 records effective history instead of replacing the previous version. In the stated problem, returned values cannot be converted to the declared Spark return type; this choice would not establish the required udf return schema decision.
D:Choose compatible pinned versions using a tested dependency lock strategy. This design action addresses a different case (library version pinning), where a transitive dependency updated to an incompatible version. Explicit compatible dependency versions improve reproducibility across new compute environments. In the stated problem, returned values cannot be converted to the declared Spark return type; this choice would not establish the required udf return schema decision.
Question 394
A deployment owner at Silver Ridge Logistics needs to implement a remedy: an ingestion job uses automatic retries after transient API timeouts. The present failure is: The downstream write repeats non-idempotent side effects on retry. Which change should be applied?
- Deploy the wheel task with its documented entrypoint and parameters
- Add deterministic upsert keys or deduped request identifiers to the task
- Deduplicate or aggregate the dimension before the final join
- Update foreachBatch to upsert by business key with batch idempotency
Explanation
Correct Answer
B:Add deterministic upsert keys or deduped request identifiers to the task. This directly changes the feature or logic behind the downstream write repeats non-idempotent side effects on retry. Retries re-execute task logic; application code must protect repeated effects. It is the concrete configuration for the stated incident.
Incorrect Answers
A:Deploy the wheel task with its documented entrypoint and parameters. This implement action addresses a different case (python task entrypoints), where the task exits before setting up its argument parser. A standard Python entrypoint and explicit arguments avoid notebook runtime assumptions. In the stated problem, the downstream write repeats non-idempotent side effects on retry; this choice would not establish the required job retry semantics decision.
C:Deduplicate or aggregate the dimension before the final join. This implement action addresses a different case (sql join cardinality), where the row count doubles because the right side contains duplicate join keys. A many-to-many join expands cardinality unless the right grain is established. In the stated problem, the downstream write repeats non-idempotent side effects on retry; this choice would not establish the required job retry semantics decision.
D:Update foreachBatch to upsert by business key with batch idempotency. This implement action addresses a different case (streaming idempotent sink), where a retry of a completed batch produces duplicate sink rows. foreachBatch can execute a batch again and requires idempotent external writes for exactly-once effects. In the stated problem, the downstream write repeats non-idempotent side effects on retry; this choice would not establish the required job retry semantics decision.
Question 395
At Cascade Manufacturing, a pipeline expectation dashboard shows records failing a rule. Observed issue: Analysts misread a quality count as the number of pipeline task failures. Before applying a fix, which investigation is most directly relevant?
- Inspect per-key frequencies and Spark task duration distribution
- Inspect the streaming table definition and expectation violation counts
- Inspect row-level expectation metrics alongside pipeline event types
- Inspect malformed timestamps and pipeline data-quality rules
Explanation
Correct Answer
C:Inspect row-level expectation metrics alongside pipeline event types. This checks the specific evidence behind analysts misread a quality count as the number of pipeline task failures. Expectation metrics describe data rows and must be interpreted separately from orchestration status. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Inspect per-key frequencies and Spark task duration distribution. This diagnose action addresses a different case (skewed group aggregation), where a single campaign dominates rows and one reduce task runs for minutes. A hot aggregation key causes skew despite ample parallelism. In the stated problem, analysts misread a quality count as the number of pipeline task failures; this choice would not establish the required quality metric interpretation decision.
B:Inspect the streaming table definition and expectation violation counts. This diagnose action addresses a different case (expectation drop action), where invalid rows currently reach downstream aggregates. A Lakeflow expectation configured to drop invalid rows filters violations from target output. In the stated problem, analysts misread a quality count as the number of pipeline task failures; this choice would not establish the required quality metric interpretation decision.
D:Inspect malformed timestamps and pipeline data-quality rules. This diagnose action addresses a different case (quarantine bad records), where dropping invalid rows silently hides upstream quality failures. Quarantine preserves invalid events separately while allowing valid records through. In the stated problem, analysts misread a quality count as the number of pipeline task failures; this choice would not establish the required quality metric interpretation decision.
Question 396
An architecture review at Cedar Brook Commerce covers a workload where a customer-to-events join has one giant default customer ID. The unresolved constraint is: A single join partition spills heavily to disk. Which design is most appropriate?
- Choose a single clustering layout mechanism for this table
- Use supported AQE with appropriate shuffle and join settings
- Choose a partition count based on volume and executor capacity
- Choose adaptive skew join handling or a justified hot-key strategy
Explanation
Correct Answer
D:Choose adaptive skew join handling or a justified hot-key strategy. This design addresses a single join partition spills heavily to disk. Adaptive skew handling or key salting can mitigate a hot join key. It establishes the right contract or architecture before execution changes.
Incorrect Answers
A:Choose a single clustering layout mechanism for this table. This design action addresses a different case (clustering versus zorder), where the proposed maintenance script also runs ZORDER on that clustered table. Liquid clustering is incompatible with concurrently applying ZORDER to the same table. In the stated problem, a single join partition spills heavily to disk; this choice would not establish the required skewed joins decision.
B:Use supported AQE with appropriate shuffle and join settings. This design action addresses a different case (adaptive execution), where static join strategy performs poorly at varying input volumes. Adaptive Query Execution can adjust supported shuffle partitions and join strategies at runtime. In the stated problem, a single join partition spills heavily to disk; this choice would not establish the required skewed joins decision.
C:Choose a partition count based on volume and executor capacity. This design action addresses a different case (shuffle partition count), where the current partition count is too low for data volume. Shuffle partition sizing affects memory pressure and task parallelism. In the stated problem, a single join partition spills heavily to disk; this choice would not establish the required skewed joins decision.
Question 397
Operations at Arbor Medical owns a workload where business users cannot distinguish certified customer metrics from experimental tables. The previously identified issue was: Table and column metadata lack meaningful descriptions. Which metric or audit check best detects drift?
- Monitor description coverage on newly published tables
- Monitor crawler failures and schema drift
- Audit subsequent data access requests and grants
- Audit new location definitions and path overlaps
Explanation
Correct Answer
A:Monitor description coverage on newly published tables. This watches the failure mode described by table and column metadata lack meaningful descriptions. Unity Catalog comments improve discoverability of governed data assets. It supplies a regression signal after remediation.
Incorrect Answers
B:Monitor crawler failures and schema drift. This monitor action addresses a different case (information schema discovery), where it uses notebook-specific private metadata APIs that break after upgrades. INFORMATION_SCHEMA provides SQL metadata views subject to visibility permissions. In the stated problem, table and column metadata lack meaningful descriptions; this choice would not establish the required catalog descriptions decision.
C:Audit subsequent data access requests and grants. This monitor action addresses a different case (browse metadata privilege), where teams grant SELECT merely to expose object listings. BROWSE enables controlled metadata discovery without granting data read access. In the stated problem, table and column metadata lack meaningful descriptions; this choice would not establish the required catalog descriptions decision.
D:Audit new location definitions and path overlaps. This monitor action addresses a different case (external location governance), where raw cloud permissions bypass central catalog governance. External locations map cloud paths to governed Unity Catalog storage credentials. In the stated problem, table and column metadata lack meaningful descriptions; this choice would not establish the required catalog descriptions decision.
Question 398
At Meadowland Software, a notebook defines transformations on a large dataset. A change is required because: Engineers assume an intermediate filter executed before any action. Which implementation action most directly corrects the configuration?
- Deploy the wheel task with its documented entrypoint and parameters
- Build transformations and invoke an explicit write or count action
- Deduplicate or aggregate the dimension before the final join
- Compose transformations with DataFrame.transform and explicit configuration
Explanation
Correct Answer
B:Build transformations and invoke an explicit write or count action. This directly changes the feature or logic behind engineers assume an intermediate filter executed before any action. Spark transformations are lazy until an action triggers a job. It is the concrete configuration for the stated incident.
Incorrect Answers
A:Deploy the wheel task with its documented entrypoint and parameters. This implement action addresses a different case (python task entrypoints), where the task exits before setting up its argument parser. A standard Python entrypoint and explicit arguments avoid notebook runtime assumptions. In the stated problem, engineers assume an intermediate filter executed before any action; this choice would not establish the required spark lazy evaluation decision.
C:Deduplicate or aggregate the dimension before the final join. This implement action addresses a different case (sql join cardinality), where the row count doubles because the right side contains duplicate join keys. A many-to-many join expands cardinality unless the right grain is established. In the stated problem, engineers assume an intermediate filter executed before any action; this choice would not establish the required spark lazy evaluation decision.
D:Compose transformations with DataFrame.transform and explicit configuration. This implement action addresses a different case (transform composability), where business logic depends on notebook state and widgets. DataFrame.transform supports composable DataFrame-to-DataFrame functions. In the stated problem, engineers assume an intermediate filter executed before any action; this choice would not establish the required spark lazy evaluation decision.
Question 399
An architecture review at Riverbend Energy covers a workload where two data science teams must join events without exposing raw customer IDs. The unresolved constraint is: A naive hash without controlled key management risks predictable identifiers. Which design is most appropriate?
- Quarantine suspect records behind restricted grants until remediation
- Use a row filter keyed to approved region mapping
- Choose a managed keyed tokenization or keyed HMAC contract for cross-dataset linkage
- Choose structured error classes and record identifiers rather than raw payloads
Explanation
Correct Answer
C:Choose a managed keyed tokenization or keyed HMAC contract for cross-dataset linkage. This design addresses a naive hash without controlled key management risks predictable identifiers. Keyed pseudonymization can preserve joins while reducing identifier exposure. It establishes the right contract or architecture before execution changes.
Incorrect Answers
A:Quarantine suspect records behind restricted grants until remediation. This design action addresses a different case (pii quarantine), where the silver pipeline allows unexpected identifiers into a broad analyst dataset. A data-quality filter can isolate suspect records before release. In the stated problem, a naive hash without controlled key management risks predictable identifiers; this choice would not establish the required pseudonymized join keys decision.
B:Use a row filter keyed to approved region mapping. This design action addresses a different case (unity catalog row filter), where a shared table currently returns all regions to every reader. Unity Catalog row filters can restrict row visibility through governed policies. In the stated problem, a naive hash without controlled key management risks predictable identifiers; this choice would not establish the required pseudonymized join keys decision.
D:Choose structured error classes and record identifiers rather than raw payloads. This design action addresses a different case (restricted event logging), where logs expose PII to broader operations readers. Logging should redact or omit sensitive payload fields while preserving diagnostic codes. In the stated problem, a naive hash without controlled key management risks predictable identifiers; this choice would not establish the required pseudonymized join keys decision.
Question 400
At Granite Edge Labs, a regulated dataset must delete customer records after a documented retention period. Observed issue: Expired records remain available in historical serving tables. Before applying a fix, which investigation is most directly relevant?
- Inspect sample log records under restricted review for PII content
- Inspect ownership and grant-management privileges in production
- Inspect consumer business need and raw table grants
- Inspect active and historical retention requirements and table features
Explanation
Correct Answer
D:Inspect active and historical retention requirements and table features. This checks the specific evidence behind expired records remain available in historical serving tables. Retention design must remove records and account for Delta time travel and obsolete files. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Inspect sample log records under restricted review for PII content. This diagnose action addresses a different case (restricted event logging), where logs expose PII to broader operations readers. Logging should redact or omit sensitive payload fields while preserving diagnostic codes. In the stated problem, expired records remain available in historical serving tables; this choice would not establish the required retention purge decision.
B:Inspect ownership and grant-management privileges in production. This diagnose action addresses a different case (separation of duties), where all developers currently own production catalogs and can grant access. Separate deploy and governance roles reduce accidental privilege escalation. In the stated problem, expired records remain available in historical serving tables; this choice would not establish the required retention purge decision.
C:Inspect consumer business need and raw table grants. This diagnose action addresses a different case (secure view sharing), where granting SELECT on the raw fact table violates minimization. A governed aggregation view can expose only approved derived information. In the stated problem, expired records remain available in historical serving tables; this choice would not establish the required retention purge decision.