Databricks Certified Data Engineer Professional Practice Test 41
Question 400
At Granite Edge Labs, a regulated dataset must delete customer records after a documented retention period. Observed issue: Expired records remain available in historical serving tables. Before applying a fix, which investigation is most directly relevant?
- Inspect sample log records under restricted review for PII content
- Inspect ownership and grant-management privileges in production
- Inspect consumer business need and raw table grants
- Inspect active and historical retention requirements and table features
Explanation
Correct Answer
D:Inspect active and historical retention requirements and table features. This checks the specific evidence behind expired records remain available in historical serving tables. Retention design must remove records and account for Delta time travel and obsolete files. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Inspect sample log records under restricted review for PII content. This diagnose action addresses a different case (restricted event logging), where logs expose PII to broader operations readers. Logging should redact or omit sensitive payload fields while preserving diagnostic codes. In the stated problem, expired records remain available in historical serving tables; this choice would not establish the required retention purge decision.
B:Inspect ownership and grant-management privileges in production. This diagnose action addresses a different case (separation of duties), where all developers currently own production catalogs and can grant access. Separate deploy and governance roles reduce accidental privilege escalation. In the stated problem, expired records remain available in historical serving tables; this choice would not establish the required retention purge decision.
C:Inspect consumer business need and raw table grants. This diagnose action addresses a different case (secure view sharing), where granting SELECT on the raw fact table violates minimization. A governed aggregation view can expose only approved derived information. In the stated problem, expired records remain available in historical serving tables; this choice would not establish the required retention purge decision.
Question 401
At Orion Capital, a group creates many tables in random external paths. The known risk is: The data platform cannot apply consistent lifecycle governance. After remediation, which ongoing signal best detects recurrence?
- Audit unapproved external paths and location changes
- Monitor new unapproved lineage dependencies
- Audit new location definitions and path overlaps
- Audit tag removal or changes to sensitive labels
Explanation
Correct Answer
A:Audit unapproved external paths and location changes. This watches the failure mode described by the data platform cannot apply consistent lifecycle governance. Managed locations centralize storage ownership for Unity Catalog managed assets. It supplies a regression signal after remediation.
Incorrect Answers
B:Monitor new unapproved lineage dependencies. This monitor action addresses a different case (lineage tracking), where investigators cannot tell which upstream transformation produced it. Unity Catalog lineage records relationships between governed data assets. In the stated problem, the data platform cannot apply consistent lifecycle governance; this choice would not establish the required managed storage ownership decision.
C:Audit new location definitions and path overlaps. This monitor action addresses a different case (external location governance), where raw cloud permissions bypass central catalog governance. External locations map cloud paths to governed Unity Catalog storage credentials. In the stated problem, the data platform cannot apply consistent lifecycle governance; this choice would not establish the required managed storage ownership decision.
D:Audit tag removal or changes to sensitive labels. This monitor action addresses a different case (governed tags), where naming conventions alone are inconsistent across teams. Governed tags can classify assets for discovery and policy workflows. In the stated problem, the data platform cannot apply consistent lifecycle governance; this choice would not establish the required managed storage ownership decision.
Question 402
At Horizon Travel, a dashboard repeatedly groups detailed events by day and product. A change is required because: Interactive queries scan billions of detail rows for identical totals. Which implementation action most directly corrects the configuration?
- Write raw events to bronze, quality checks to silver and analytics tables to gold
- Materialize the aggregate from quality-controlled silver facts
- Generate dimension surrogate IDs and reference them from fact rows
- Build the bridge and allocate or filter measures according to reporting rules
Explanation
Correct Answer
B:Materialize the aggregate from quality-controlled silver facts. This directly changes the feature or logic behind interactive queries scan billions of detail rows for identical totals. A summarized gold table can serve a stable reporting grain efficiently. It is the concrete configuration for the stated incident.
Incorrect Answers
A:Write raw events to bronze, quality checks to silver and analytics tables to gold. This implement action addresses a different case (medallion bronze silver gold), where malformed source events contaminate metrics without an intermediate quality boundary. Bronze, silver and gold layers separate raw capture, validated refinement and business presentation. In the stated problem, interactive queries scan billions of detail rows for identical totals; this choice would not establish the required aggregate gold model decision.
C:Generate dimension surrogate IDs and reference them from fact rows. This implement action addresses a different case (surrogate dimension keys), where joining facts directly on mutable business IDs links history incorrectly. Stable surrogate keys decouple dimensional identity from changing source identifiers. In the stated problem, interactive queries scan billions of detail rows for identical totals; this choice would not establish the required aggregate gold model decision.
D:Build the bridge and allocate or filter measures according to reporting rules. This implement action addresses a different case (many-to-many bridge), where joining facts directly to the membership table multiplies totals. A bridge table models many-to-many relationships at an explicit membership grain. In the stated problem, interactive queries scan billions of detail rows for identical totals; this choice would not establish the required aggregate gold model decision.
Question 403
At Vantage Security, a stream groups clicks by event time in ten-minute windows. A change was deployed to address: Old keys remain in state indefinitely without a lateness bound. Which test result provides the strongest evidence that the requirement is now met?
- Run the same function in a unit test and in a production job
- Reverse input arrival order and verify identical final target state
- Test late events on both sides of the threshold and reconcile counts
- Confirm unit tests run without credentials and integrations run in an approved test workspace
Explanation
Correct Answer
C:Test late events on both sides of the threshold and reconcile counts. This tests the expected result against old keys remain in state indefinitely without a lateness bound, not simply whether a job or object exists. Watermarks provide a lateness threshold for bounded stateful streaming operations.
Incorrect Answers
A:Run the same function in a unit test and in a production job. This validate action addresses a different case (transform composability), where business logic depends on notebook state and widgets. DataFrame.transform supports composable DataFrame-to-DataFrame functions. In the stated problem, old keys remain in state indefinitely without a lateness bound; this choice would not establish the required event-time watermarking decision.
B:Reverse input arrival order and verify identical final target state. This validate action addresses a different case (cdc duplicate sequence), where the processing order for tied source events is nondeterministic. A deterministic sequencing contract needs a unique ordered event position. In the stated problem, old keys remain in state indefinitely without a lateness bound; this choice would not establish the required event-time watermarking decision.
D:Confirm unit tests run without credentials and integrations run in an approved test workspace. This validate action addresses a different case (unit versus integration tests), where small logic regressions are discovered late and costly to debug. Unit tests isolate transformations while integration tests validate external contracts. In the stated problem, old keys remain in state indefinitely without a lateness bound; this choice would not establish the required event-time watermarking decision.
Question 404
At Westridge Research, many Unity Catalog managed tables require routine maintenance. The known risk is: Operators run blanket manual OPTIMIZE jobs regardless of table use. After remediation, which ongoing signal best detects recurrence?
- Monitor managed table optimization and ownership changes
- Monitor automatic key decisions and maintenance operations
- Monitor serverless task failures and cost per run
- Monitor optimization operations and cost savings
Explanation
Correct Answer
D:Monitor optimization operations and cost savings. This watches the failure mode described by operators run blanket manual OPTIMIZE jobs regardless of table use. Predictive optimization can automate supported managed-table maintenance based on workload signals. It supplies a regression signal after remediation.
Incorrect Answers
A:Monitor managed table optimization and ownership changes. This monitor action addresses a different case (managed table maintenance), where each table has custom file cleanup jobs and broad storage permissions. Unity Catalog managed tables provide platform-managed data placement and maintenance capabilities. In the stated problem, operators run blanket manual OPTIMIZE jobs regardless of table use; this choice would not establish the required predictive optimization decision.
B:Monitor automatic key decisions and maintenance operations. This monitor action addresses a different case (automatic clustering), where manual clustering keys quickly become stale. Automatic liquid clustering can select keys using query workload when platform prerequisites are met. In the stated problem, operators run blanket manual OPTIMIZE jobs regardless of table use; this choice would not establish the required predictive optimization decision.
C:Monitor serverless task failures and cost per run. This monitor action addresses a different case (serverless job fit), where provisioning overhead exceeds useful task execution time. Supported serverless jobs can reduce infrastructure management and startup overhead. In the stated problem, operators run blanket manual OPTIMIZE jobs regardless of table use; this choice would not establish the required predictive optimization decision.
Question 405
At Eastern Star Foods, a cloud object store receives a large and growing set of landing files. Architecture constraint: Listing the full historical directory every run exceeds the ingestion SLA. Which design best addresses this requirement?
- Use Auto Loader with managed file discovery and checkpoint/schema locations
- Use a governed nested schema contract with an approved evolution policy
- Use a stable checkpoint with a compatible idempotent Delta sink
- Use one durable checkpoint and clear idempotency keys for replay
Explanation
Correct Answer
A:Use Auto Loader with managed file discovery and checkpoint/schema locations. This design addresses listing the full historical directory every run exceeds the ingestion SLA. Auto Loader is designed for incremental new-file discovery with persisted state. It establishes the right contract or architecture before execution changes.
Incorrect Answers
B:Use a governed nested schema contract with an approved evolution policy. This design action addresses a different case (nested json ingest), where the initial reader assumes one unchanging flat schema and loses new fields. Explicit schema management and rescued data handling protect evolving semi-structured ingestion. In the stated problem, listing the full historical directory every run exceeds the ingestion SLA; this choice would not establish the required auto loader file discovery decision.
C:Use a stable checkpoint with a compatible idempotent Delta sink. This design action addresses a different case (kafka offset durability), where restarting with a fresh checkpoint replays previously ingested messages. A stable streaming checkpoint maintains Kafka offsets and sink progress across restarts. In the stated problem, listing the full historical directory every run exceeds the ingestion SLA; this choice would not establish the required auto loader file discovery decision.
D:Use one durable checkpoint and clear idempotency keys for replay. This design action addresses a different case (retry-safe file loads), where a partially processed input file is loaded twice into Delta. Checkpointed incremental ingestion or stable event-level deduplication prevents repeated effects. In the stated problem, listing the full historical directory every run exceeds the ingestion SLA; this choice would not establish the required auto loader file discovery decision.
Question 406
At Cascade Manufacturing, a workload's input volume varies from 1 GB to 500 GB by day. Observed issue: A fixed high-worker count wastes resources on quiet days. Before applying a fix, which investigation is most directly relevant?
- Inspect per-task shuffle read bytes and spilled data
- Inspect input size variation and worker utilization over time
- Inspect file count and size distribution before tuning
- Inspect watermark, state operator metrics and checkpoint size
Explanation
Correct Answer
B:Inspect input size variation and worker utilization over time. This checks the specific evidence behind a fixed high-worker count wastes resources on quiet days. Supported autoscaling can adapt worker count within configured bounds. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Inspect per-task shuffle read bytes and spilled data. This diagnose action addresses a different case (shuffle partition count), where the current partition count is too low for data volume. Shuffle partition sizing affects memory pressure and task parallelism. In the stated problem, a fixed high-worker count wastes resources on quiet days; this choice would not establish the required autoscaling right-size decision.
C:Inspect file count and size distribution before tuning. This diagnose action addresses a different case (small file compaction), where read overhead grows because query plans open excessive small files. OPTIMIZE can compact compatible Delta files without changing logical table contents. In the stated problem, a fixed high-worker count wastes resources on quiet days; this choice would not establish the required autoscaling right-size decision.
D:Inspect watermark, state operator metrics and checkpoint size. This diagnose action addresses a different case (streaming state growth), where executor memory pressure grows even though input rate is stable. State retention and checkpoint state-store configuration affect memory requirements. In the stated problem, a fixed high-worker count wastes resources on quiet days; this choice would not establish the required autoscaling right-size decision.
Question 407
A deployment owner at Vantage Security needs to implement a remedy: an events table has high-cardinality device IDs and unpredictable daily volumes. The present failure is: Partitioning by device would produce many tiny partitions. Which change should be applied?
- Migrate a compatible task to serverless execution with approved permissions
- Create the managed table under the approved catalog and migrate governed data
- Create or alter an eligible table with CLUSTER BY for approved keys
- Rewrite filtering so business_date is compared directly where semantics allow
Explanation
Correct Answer
C:Create or alter an eligible table with CLUSTER BY for approved keys. This directly changes the feature or logic behind partitioning by device would produce many tiny partitions. Liquid clustering organizes data by useful filter keys without high-cardinality partition directories. It is the concrete configuration for the stated incident.
Incorrect Answers
A:Migrate a compatible task to serverless execution with approved permissions. This implement action addresses a different case (serverless job fit), where provisioning overhead exceeds useful task execution time. Supported serverless jobs can reduce infrastructure management and startup overhead. In the stated problem, partitioning by device would produce many tiny partitions; this choice would not establish the required liquid clustering cardinality decision.
B:Create the managed table under the approved catalog and migrate governed data. This implement action addresses a different case (managed table maintenance), where each table has custom file cleanup jobs and broad storage permissions. Unity Catalog managed tables provide platform-managed data placement and maintenance capabilities. In the stated problem, partitioning by device would produce many tiny partitions; this choice would not establish the required liquid clustering cardinality decision.
D:Rewrite filtering so business_date is compared directly where semantics allow. This implement action addresses a different case (partition pruning), where a query wraps business_date in a transformation and reads many partitions. Direct predicates on partition keys can enable partition pruning. In the stated problem, partitioning by device would produce many tiny partitions; this choice would not establish the required liquid clustering cardinality decision.
Question 408
A data engineer at Cobalt Biotech must troubleshoot this case: a developer deploys a bundle from a laptop with default settings. Evidence: The wrong workspace is selected because target is implicit. What should be inspected first to confirm the failure mechanism?
- Inspect CI stages for tests, bundle validation and approvals
- Inspect effective grants on source, target and parent containers for the run-as principal
- Inspect Python version, wheel tags and package resolution logs
- Inspect CLI target resolution and workspace host for the command
Explanation
Correct Answer
D:Inspect CLI target resolution and workspace host for the command. This checks the specific evidence behind the wrong workspace is selected because target is implicit. Explicit bundle target selection prevents accidental environment deployment. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Inspect CI stages for tests, bundle validation and approvals. This diagnose action addresses a different case (ci quality gates), where the deployment pipeline only checks YAML syntax. CI must run transformation tests and controlled integration checks before deployment. In the stated problem, the wrong workspace is selected because target is implicit; this choice would not establish the required target-safe deployment decision.
B:Inspect effective grants on source, target and parent containers for the run-as principal. This diagnose action addresses a different case (job run permission debug), where the run-as principal lacks MODIFY on the target table. Unity Catalog authorization applies to the job execution identity. In the stated problem, the wrong workspace is selected because target is implicit; this choice would not establish the required target-safe deployment decision.
C:Inspect Python version, wheel tags and package resolution logs. This diagnose action addresses a different case (python dependency triage), where a package import error appears only on the target compute. Runtime-specific wheel compatibility and pinning must be tested in CI. In the stated problem, the wrong workspace is selected because target is implicit; this choice would not establish the required target-safe deployment decision.
Question 409
At Northfield Analytics, an automated pipeline update fails after a schema change. Architecture constraint: The Jobs status only reports a high-level failure. Which design best addresses this requirement?
- Build structured operational diagnostics from pipeline event records
- Create a historical failure and runtime dashboard by workflow
- Report violation rate as an explicit quality indicator
- Build a tuning plan from the specific skew or spill evidence
Explanation
Correct Answer
A:Build structured operational diagnostics from pipeline event records. This design addresses the Jobs status only reports a high-level failure. Lakeflow pipeline event logs include update-level errors and progress events. It establishes the right contract or architecture before execution changes.
Incorrect Answers
B:Create a historical failure and runtime dashboard by workflow. This design action addresses a different case (job operational history), where operators only see the most recent run in the Jobs UI. Jobs system records support historical analysis of runs and tasks. In the stated problem, the Jobs status only reports a high-level failure; this choice would not establish the required pipeline event log decision.
C:Report violation rate as an explicit quality indicator. This design action addresses a different case (expectation monitoring), where analysts confuse successful pipeline status with zero rejected records. Quality metric events reveal input and dropped records separately from update status. In the stated problem, the Jobs status only reports a high-level failure; this choice would not establish the required pipeline event log decision.
D:Build a tuning plan from the specific skew or spill evidence. This design action addresses a different case (spark ui shuffle metrics), where operators only monitor cluster-wide CPU and miss skewed task durations. Spark UI stage metrics reveal shuffle, spill and task skew. In the stated problem, the Jobs status only reports a high-level failure; this choice would not establish the required pipeline event log decision.
Question 410
Quality assurance at Riverbend Energy checks a corrected workload where an engineering team owns many internal Delta tables with no need for external file lifecycle control. The original fault was: Each table has custom file cleanup jobs and broad storage permissions. Which verification is most decisive?
- Benchmark identical queries with correct outputs and controlled resources
- Confirm access is controlled through Unity Catalog and queries return identical rows
- Confirm reused stages avoid recomputation with no change in output
- Delete a few rows and confirm snapshot consistency without unnecessary rewrites
Explanation
Correct Answer
B:Confirm access is controlled through Unity Catalog and queries return identical rows. This tests the expected result against each table has custom file cleanup jobs and broad storage permissions, not simply whether a job or object exists. Unity Catalog managed tables provide platform-managed data placement and maintenance capabilities.
Incorrect Answers
A:Benchmark identical queries with correct outputs and controlled resources. This validate action addresses a different case (photon query execution), where an unsupported execution mode limits vectorized engine use. Photon can accelerate supported SQL and DataFrame operators on compatible compute. In the stated problem, each table has custom file cleanup jobs and broad storage permissions; this choice would not establish the required managed table maintenance decision.
C:Confirm reused stages avoid recomputation with no change in output. This validate action addresses a different case (cache reuse), where repeated recomputation dominates each branch's execution. Persistence can avoid repeated work if reused data fits the memory budget. In the stated problem, each table has custom file cleanup jobs and broad storage permissions; this choice would not establish the required managed table maintenance decision.
D:Delete a few rows and confirm snapshot consistency without unnecessary rewrites. This validate action addresses a different case (deletion vectors), where each small delete rewrites large data files and increases write cost. Deletion vectors can mark deleted rows without immediate whole-file rewrite when supported. In the stated problem, each table has custom file cleanup jobs and broad storage permissions; this choice would not establish the required managed table maintenance decision.