Pass Databricks Certified Data Engineer Professional Exam in First Attempt Guaranteed!

Defer expensive materialization until the required action. This design action addresses a different case (spark lazy evaluation), where engineers assume an intermediate filter executed before any action. Spark transformations are lazy until an action triggers a job. In the stated problem, periodic full-refresh queries miss required incremental freshness; this choice would not establish the required streaming table choice decision.

Design a keyed CDC target using a reliable sequence and delete handling. This design action addresses a different case (auto cdc keys), where applying events as append-only rows leaves outdated active accounts. AUTO CDC or applicable APPLY CHANGES needs stable keys and sequence ordering for stateful upserts. In the stated problem, periodic full-refresh queries miss required incremental freshness; this choice would not establish the required streaming table choice decision.

Use a managed service principal with only required production privileges. This design action addresses a different case (job run-as identity), where an employee departure causes permissions failures during scheduled runs. A stable least-privilege service principal can provide durable job identity. In the stated problem, periodic full-refresh queries miss required incremental freshness; this choice would not establish the required streaming table choice decision.

Monitor validation failures after dependency or CLI upgrades. This monitor action addresses a different case (bundle validation), where an invalid resource key would be rejected only after production change. Bundle validation detects many configuration problems before deployment. In the stated problem, a package import error appears only on the target compute; this choice would not establish the required python dependency triage decision.

Monitor recurrence of the parsing error class. This monitor action addresses a different case (pipeline debug sampling), where re-running a huge source in development is slow and risky. A minimal reproducible input isolates the failure without modifying production data. In the stated problem, a package import error appears only on the target compute; this choice would not establish the required python dependency triage decision.

Monitor pipeline error events by dataset and update. This monitor action addresses a different case (lakeflow event error classes), where the only visible dashboard status is FAILED without a dataset name. Pipeline event logs identify failing update and dataset messages. In the stated problem, a package import error appears only on the target compute; this choice would not establish the required python dependency triage decision.

Inspect duplicate file IDs and the ingestion checkpoint state. This diagnose action addresses a different case (retry-safe file loads), where a partially processed input file is loaded twice into Delta. Checkpointed incremental ingestion or stable event-level deduplication prevents repeated effects. In the stated problem, unknown or mismatched fields disappear during the bronze load; this choice would not establish the required auto loader rescued data decision.

Inspect write modes and table-history operations around the loss. This diagnose action addresses a different case (append-only delta bronze), where a nightly overwrite accidentally removes historical audit events. Append-only ingestion keeps previous raw records while adding new batches. In the stated problem, unknown or mismatched fields disappear during the bronze load; this choice would not establish the required auto loader rescued data decision.

Inspect inferred schemas and rescued or null fields in recently ingested rows. This diagnose action addresses a different case (nested json ingest), where the initial reader assumes one unchanging flat schema and loses new fields. Explicit schema management and rescued data handling protect evolving semi-structured ingestion. In the stated problem, unknown or mismatched fields disappear during the bronze load; this choice would not establish the required auto loader rescued data decision.

Inspect input size variation and worker utilization over time. This diagnose action addresses a different case (autoscaling right-size), where a fixed high-worker count wastes resources on quiet days. Supported autoscaling can adapt worker count within configured bounds. In the stated problem, read overhead grows because query plans open excessive small files; this choice would not establish the required small file compaction decision.

Inspect current table features, delete patterns and runtime compatibility. This diagnose action addresses a different case (deletion vectors), where each small delete rewrites large data files and increases write cost. Deletion vectors can mark deleted rows without immediate whole-file rewrite when supported. In the stated problem, read overhead grows because query plans open excessive small files; this choice would not establish the required small file compaction decision.

Inspect watermark, state operator metrics and checkpoint size. This diagnose action addresses a different case (streaming state growth), where executor memory pressure grows even though input rate is stable. State retention and checkpoint state-store configuration affect memory requirements. In the stated problem, read overhead grows because query plans open excessive small files; this choice would not establish the required small file compaction decision.

Prefer native Spark expressions or a compatible vectorized Pandas UDF for the calculation. This design action addresses a different case (vectorized udf), where row-by-row Python invocation dominates stage runtime. Pandas UDFs process batches through Arrow rather than per-row Python calls. In the stated problem, an API token is committed in a Python notebook tracked in Git; this choice would not establish the required secure configuration decision.

Use one managed and repeatable dependency source for the workload. This design action addresses a different case (library precedence), where a notebook-scoped installation overrides the expected cluster version. The runtime can resolve conflicting versions from different installation scopes. In the stated problem, an API token is committed in a Python notebook tracked in Git; this choice would not establish the required secure configuration decision.

Use transactional merge or a committed batch-ID deduplication design. This design action addresses a different case (streaming idempotent sink), where a retry of a completed batch produces duplicate sink rows. foreachBatch can execute a batch again and requires idempotent external writes for exactly-once effects. In the stated problem, an API token is committed in a Python notebook tracked in Git; this choice would not establish the required secure configuration decision.

Append null IDs and confirm they are excluded while valid IDs remain. This validate action addresses a different case (expectation drop action), where invalid rows currently reach downstream aggregates. A Lakeflow expectation configured to drop invalid rows filters violations from target output. In the stated problem, rows with null external IDs disappear from an inner join; this choice would not establish the required join with nullable keys decision.

Inject an invalid record and confirm the update fails visibly. This validate action addresses a different case (expectation fail action), where the pipeline continues despite critical null records. Fail-on-violation expectations stop updates rather than silently filtering rows. In the stated problem, rows with null external IDs disappear from an inner join; this choice would not establish the required join with nullable keys decision.

Confirm the plan uses the intended broadcast join and results match. This validate action addresses a different case (broadcast lookup table), where the join triggers a costly shuffle of the large fact dataset. Broadcasting an appropriately sized dimension can avoid a large shuffle join. In the stated problem, rows with null external IDs disappear from an inner join; this choice would not establish the required join with nullable keys decision.

Inspect malformed timestamps and pipeline data-quality rules. This diagnose action addresses a different case (quarantine bad records), where dropping invalid rows silently hides upstream quality failures. Quarantine preserves invalid events separately while allowing valid records through. In the stated problem, the silver writer rejects every new record after the upstream schema change; this choice would not establish the required controlled schema evolution decision.

Group active rows by supplier ID and quantify duplicates. This diagnose action addresses a different case (uniqueness quality rule), where an upstream replay creates multiple active versions. An explicit uniqueness check identifies duplicate business keys before serving data. In the stated problem, the silver writer rejects every new record after the upstream schema change; this choice would not establish the required controlled schema evolution decision.

Inspect input schema, target schema and invalid decimal examples. This diagnose action addresses a different case (silver schema enforcement), where implicit type coercion risks invalid numeric values being stored. Schema validation and explicit casts make type conversion failures observable. In the stated problem, the silver writer rejects every new record after the upstream schema change; this choice would not establish the required controlled schema evolution decision.

Inspect notebook defaults and catalog settings for both environments. This diagnose action addresses a different case (environment catalog isolation), where both environments use the same default catalog name in notebooks. Environment-specific catalogs and explicit target names reduce cross-environment writes. In the stated problem, the team lacks USE CATALOG and USE SCHEMA on the parent containers; this choice would not establish the required catalog privilege inheritance decision.

Inspect the available information_schema tables and caller grants. This diagnose action addresses a different case (information schema discovery), where it uses notebook-specific private metadata APIs that break after upgrades. INFORMATION_SCHEMA provides SQL metadata views subject to visibility permissions. In the stated problem, the team lacks USE CATALOG and USE SCHEMA on the parent containers; this choice would not establish the required catalog privilege inheritance decision.

Inspect current owner and authorized ownership transfer process. This diagnose action addresses a different case (ownership reassignment), where new data stewards cannot maintain its grants. Unity Catalog ownership should be assigned to a maintained principal. In the stated problem, the team lacks USE CATALOG and USE SCHEMA on the parent containers; this choice would not establish the required catalog privilege inheritance decision.

Run databricks bundle validate for the intended target in CI. This implement action addresses a different case (bundle validation), where an invalid resource key would be rejected only after production change. Bundle validation detects many configuration problems before deployment. In the stated problem, the wrong workspace is selected because target is implicit; this choice would not establish the required target-safe deployment decision.

Deploy the previous approved bundle revision with correct target. This implement action addresses a different case (rollback deployment), where deployers have no record of the previous working configuration. Versioned bundle releases enable controlled rollback to a known commit. In the stated problem, the wrong workspace is selected because target is implicit; this choice would not establish the required target-safe deployment decision.

Apply minimal MODIFY and usage grants to the dedicated principal. This implement action addresses a different case (job run permission debug), where the run-as principal lacks MODIFY on the target table. Unity Catalog authorization applies to the job execution identity. In the stated problem, the wrong workspace is selected because target is implicit; this choice would not establish the required target-safe deployment decision.

Run small and large test inputs and compare execution strategies. This validate action addresses a different case (adaptive execution), where static join strategy performs poorly at varying input volumes. Adaptive Query Execution can adjust supported shuffle partitions and join strategies at runtime. In the stated problem, a single join partition spills heavily to disk; this choice would not establish the required skewed joins decision.

Confirm partition scan count falls with equal query results. This validate action addresses a different case (partition pruning), where a query wraps business_date in a transformation and reads many partitions. Direct predicates on partition keys can enable partition pruning. In the stated problem, a single join partition spills heavily to disk; this choice would not establish the required skewed joins decision.

Reconcile changed rows and compare scanned file counts. This validate action addresses a different case (predicate-limited merge), where the merge scans unrelated historical partitions. Constraining merge matches to relevant partitions can reduce unnecessary target scanning. In the stated problem, a single join partition spills heavily to disk; this choice would not establish the required skewed joins decision.

Query supported expectation metric records in the event log. This implement action addresses a different case (quality metric interpretation), where analysts misread a quality count as the number of pipeline task failures. Expectation metrics describe data rows and must be interpreted separately from orchestration status. In the stated problem, invalid rows currently reach downstream aggregates; this choice would not establish the required expectation drop action decision.

Change to a left join and explicitly handle missing dimension fields. This implement action addresses a different case (join with nullable keys), where rows with null external IDs disappear from an inner join. SQL null equality and join type determine whether unmatched rows survive. In the stated problem, invalid rows currently reach downstream aggregates; this choice would not establish the required expectation drop action decision.

Use MERGE with matched update and not-matched insert branches. This implement action addresses a different case (cdc merge semantics), where the current INSERT-only target contains duplicate customer keys. A keyed Delta MERGE can update existing rows and insert new keys. In the stated problem, invalid rows currently reach downstream aggregates; this choice would not establish the required expectation drop action decision.