Databricks Certified Data Engineer Professional Practice Test 77
Question 760
At Cascade Manufacturing, a monolithic notebook transformation cannot be tested separately. Observed issue: Business logic depends on notebook state and widgets. Before applying a fix, which investigation is most directly relevant?
- Compare notebook and cluster package versions and import origins
- Compare job parameter definitions to hard-coded notebook literals
- Compare production deployment variables with bundle target overrides
- Identify transformations depending on hidden notebook variables
Explanation
Correct Answer
D:Identify transformations depending on hidden notebook variables. This checks the specific evidence behind business logic depends on notebook state and widgets. DataFrame.transform supports composable DataFrame-to-DataFrame functions. The investigation distinguishes the mechanism from an unrelated operational symptom.
Incorrect Answers
A:Compare notebook and cluster package versions and import origins. This diagnose action addresses a different case (library precedence), where a notebook-scoped installation overrides the expected cluster version. The runtime can resolve conflicting versions from different installation scopes. In the stated problem, business logic depends on notebook state and widgets; this choice would not establish the required transform composability decision.
B:Compare job parameter definitions to hard-coded notebook literals. This diagnose action addresses a different case (job parameters), where code contains hard-coded region IDs and catalog names. Task parameters allow runtime-specific values without changing notebook source. In the stated problem, business logic depends on notebook state and widgets; this choice would not establish the required transform composability decision.
C:Compare production deployment variables with bundle target overrides. This diagnose action addresses a different case (bundle target overrides), where development job arguments were deployed into the production target. Bundle targets can override environment-specific variables and resource settings. In the stated problem, business logic depends on notebook state and widgets; this choice would not establish the required transform composability decision.
Question 761
An architecture review at Orion Capital covers a workload where an aggregation pipeline uses a Python UDF for lowercase normalization. The unresolved constraint is: The Python UDF creates an avoidable cross-language execution boundary. Which design is most appropriate?
- Use lower and trim native expressions for string standardization
- Select a supported wheel build for the execution environment
- Pass nonsecret configuration separately and reference an approved secret store for credentials
- Prefer native Spark expressions or a compatible vectorized Pandas UDF for the calculation
Explanation
Correct Answer
A:Use lower and trim native expressions for string standardization. This design addresses the Python UDF creates an avoidable cross-language execution boundary. Built-in Spark SQL functions can execute in optimized expressions without Python UDF overhead. It establishes the right contract or architecture before execution changes.
Incorrect Answers
B:Select a supported wheel build for the execution environment. This design action addresses a different case (wheel abi compatibility), where the wheel works on a developer laptop but fails with a binary compatibility error. Compiled extensions must match the target runtime platform and Python ABI. In the stated problem, the Python UDF creates an avoidable cross-language execution boundary; this choice would not establish the required avoid python udf for sql decision.
C:Pass nonsecret configuration separately and reference an approved secret store for credentials. This design action addresses a different case (secure configuration), where an API token is committed in a Python notebook tracked in Git. Secret-backed configuration avoids embedding credentials in notebooks or deployment YAML. In the stated problem, the Python UDF creates an avoidable cross-language execution boundary; this choice would not establish the required avoid python udf for sql decision.
D:Prefer native Spark expressions or a compatible vectorized Pandas UDF for the calculation. This design action addresses a different case (vectorized udf), where row-by-row Python invocation dominates stage runtime. Pandas UDFs process batches through Arrow rather than per-row Python calls. In the stated problem, the Python UDF creates an avoidable cross-language execution boundary; this choice would not establish the required avoid python udf for sql decision.
Question 762
A deployment owner at Redwood Civic Services needs to implement a remedy: a declarative pipeline needs near-real-time freshness. The present failure is: Scheduled updates are spaced several hours apart. Which change should be applied?
- Query the pipeline event log for the failed dataset and update ID
- Configure pipeline update scheduling based on the freshness requirement
- Apply the data-skipping-friendly predicate or clustering update
- Configure the notification destination and attach it to failed-run events
Explanation
Correct Answer
B:Configure pipeline update scheduling based on the freshness requirement. This directly changes the feature or logic behind scheduled updates are spaced several hours apart. Pipeline update frequency determines available end-to-end freshness. It is the concrete configuration for the stated incident.
Incorrect Answers
A:Query the pipeline event log for the failed dataset and update ID. This implement action addresses a different case (pipeline event log), where the Jobs status only reports a high-level failure. Lakeflow pipeline event logs include update-level errors and progress events. In the stated problem, scheduled updates are spaced several hours apart; this choice would not establish the required pipeline update cadence decision.
C:Apply the data-skipping-friendly predicate or clustering update. This implement action addresses a different case (query profile scans), where operators assume the query is always CPU-bound. Query Profile identifies scan, pruning and shuffle behavior for SQL workloads. In the stated problem, scheduled updates are spaced several hours apart; this choice would not establish the required pipeline update cadence decision.
D:Configure the notification destination and attach it to failed-run events. This implement action addresses a different case (job webhook destination), where only personal email notifications are configured. Supported job notification destinations can integrate workflow status with an incident system. In the stated problem, scheduled updates are spaced several hours apart; this choice would not establish the required pipeline update cadence decision.
Question 763
At Harborline Utilities, a consumer group needs sales totals but not transaction-level customer IDs. A change is required because: Granting SELECT on the raw fact table violates minimization. Which implementation action most directly corrects the configuration?
- Remove overly broad edit access and grant view permission to readers
- Create an explicit column projection that excludes free-text notes
- Create the aggregation view and grant SELECT only on the view
- Grant only required read privileges on the external location
Explanation
Correct Answer
C:Create the aggregation view and grant SELECT only on the view. This directly changes the feature or logic behind granting SELECT on the raw fact table violates minimization. A governed aggregation view can expose only approved derived information. It is the concrete configuration for the stated incident.
Incorrect Answers
A:Remove overly broad edit access and grant view permission to readers. This implement action addresses a different case (workspace object acl), where the team needs read-only access for most users. Workspace object permissions can restrict who can edit or manage shared code. In the stated problem, granting SELECT on the raw fact table violates minimization; this choice would not establish the required secure view sharing decision.
B:Create an explicit column projection that excludes free-text notes. This implement action addresses a different case (sensitive data suppression), where a broad SELECT still includes unstructured notes. Column suppression and data minimization prevent unnecessary sensitive-field release. In the stated problem, granting SELECT on the raw fact table violates minimization; this choice would not establish the required secure view sharing decision.
D:Grant only required read privileges on the external location. This implement action addresses a different case (storage credential scope), where engineers propose a credential allowing access to all company buckets. Unity Catalog storage credentials and external locations can constrain storage access. In the stated problem, granting SELECT on the raw fact table violates minimization; this choice would not establish the required secure view sharing decision.
Question 764
At Cobalt Biotech, a diagnostic notebook prints a database password after secret lookup. A change was deployed to address: Databricks notebook redaction is incorrectly treated as a comprehensive security control. Which test result provides the strongest evidence that the requirement is now met?
- Confirm equal source IDs map to equal pseudonyms without exposing originals
- Run the job using the principal and confirm forbidden datasets remain blocked
- Execute a controlled connection test with new credentials and reject old value
- Run with test credentials and confirm logs contain no secret strings
Explanation
Correct Answer
D:Run with test credentials and confirm logs contain no secret strings. This tests the expected result against databricks notebook redaction is incorrectly treated as a comprehensive security control, not simply whether a job or object exists. Secrets should never be deliberately printed or put in logs, regardless of redaction behavior.
Incorrect Answers
A:Confirm equal source IDs map to equal pseudonyms without exposing originals. This validate action addresses a different case (pseudonymized join keys), where a naive hash without controlled key management risks predictable identifiers. Keyed pseudonymization can preserve joins while reducing identifier exposure. In the stated problem, databricks notebook redaction is incorrectly treated as a comprehensive security control; this choice would not establish the required secret retrieval logs decision.
B:Run the job using the principal and confirm forbidden datasets remain blocked. This validate action addresses a different case (service principal access), where the individual's credentials create unnecessary privilege and lifecycle risk. Production automation should use least-privilege service principals for stable identity. In the stated problem, databricks notebook redaction is incorrectly treated as a comprehensive security control; this choice would not establish the required secret retrieval logs decision.
C:Execute a controlled connection test with new credentials and reject old value. This validate action addresses a different case (credential rotation procedure), where the hard-coded service password has not rotated for a year. Secret management and controlled rotation reduce long-lived credential exposure. In the stated problem, databricks notebook redaction is incorrectly treated as a comprehensive security control; this choice would not establish the required secret retrieval logs decision.
Question 765
At Fairhaven Transport, security wants to limit interactive workspace logins from untrusted networks. A change was deployed to address: A Unity Catalog table grant does not restrict web-console network entry. Which test result provides the strongest evidence that the requirement is now met?
- Test allowed and disallowed client networks for login behavior
- Trigger a test error and confirm logs reveal no PII content
- Confirm deployment works while a developer cannot grant themselves table access
- Query as two regional users and confirm different allowed row sets
Explanation
Correct Answer
A:Test allowed and disallowed client networks for login behavior. This tests the expected result against a Unity Catalog table grant does not restrict web-console network entry, not simply whether a job or object exists. Workspace access controls and data-object privileges address different security layers.
Incorrect Answers
B:Trigger a test error and confirm logs reveal no PII content. This validate action addresses a different case (restricted event logging), where logs expose PII to broader operations readers. Logging should redact or omit sensitive payload fields while preserving diagnostic codes. In the stated problem, a Unity Catalog table grant does not restrict web-console network entry; this choice would not establish the required ip-based workspace restrictions decision.
C:Confirm deployment works while a developer cannot grant themselves table access. This validate action addresses a different case (separation of duties), where all developers currently own production catalogs and can grant access. Separate deploy and governance roles reduce accidental privilege escalation. In the stated problem, a Unity Catalog table grant does not restrict web-console network entry; this choice would not establish the required ip-based workspace restrictions decision.
D:Query as two regional users and confirm different allowed row sets. This validate action addresses a different case (unity catalog row filter), where a shared table currently returns all regions to every reader. Unity Catalog row filters can restrict row visibility through governed policies. In the stated problem, a Unity Catalog table grant does not restrict web-console network entry; this choice would not establish the required ip-based workspace restrictions decision.
Question 766
At Cobalt Biotech, a table is partitioned by business_date. A change is required because: A query wraps business_date in a transformation and reads many partitions. Which implementation action most directly corrects the configuration?
- Run OPTIMIZE or enable managed optimization where supported
- Rewrite filtering so business_date is compared directly where semantics allow
- Enable CDF where supported and read change events from a safe starting version
- Create or alter an eligible table with CLUSTER BY for approved keys
Explanation
Correct Answer
B:Rewrite filtering so business_date is compared directly where semantics allow. This directly changes the feature or logic behind a query wraps business_date in a transformation and reads many partitions. Direct predicates on partition keys can enable partition pruning. It is the concrete configuration for the stated incident.
Incorrect Answers
A:Run OPTIMIZE or enable managed optimization where supported. This implement action addresses a different case (small file compaction), where read overhead grows because query plans open excessive small files. OPTIMIZE can compact compatible Delta files without changing logical table contents. In the stated problem, a query wraps business_date in a transformation and reads many partitions; this choice would not establish the required partition pruning decision.
C:Enable CDF where supported and read change events from a safe starting version. This implement action addresses a different case (change data feed consumers), where the consumer reads only newly appended rows and cannot see updates or deletes. Delta Change Data Feed exposes row-level changes for supported versions when enabled. In the stated problem, a query wraps business_date in a transformation and reads many partitions; this choice would not establish the required partition pruning decision.
D:Create or alter an eligible table with CLUSTER BY for approved keys. This implement action addresses a different case (liquid clustering cardinality), where partitioning by device would produce many tiny partitions. Liquid clustering organizes data by useful filter keys without high-cardinality partition directories. In the stated problem, a query wraps business_date in a transformation and reads many partitions; this choice would not establish the required partition pruning decision.
Question 767
At Fairhaven Transport, a downstream incremental aggregate misses updates to an existing Delta table. A change was deployed to address: The consumer reads only newly appended rows and cannot see updates or deletes. Which test result provides the strongest evidence that the requirement is now met?
- Delete a few rows and confirm snapshot consistency without unnecessary rewrites
- Verify shuffle spill decreases without excessive scheduling overhead
- Update and delete source rows and verify downstream state changes accordingly
- Run representative queries and verify selected layouts improve skipping
Explanation
Correct Answer
C:Update and delete source rows and verify downstream state changes accordingly. This tests the expected result against the consumer reads only newly appended rows and cannot see updates or deletes, not simply whether a job or object exists. Delta Change Data Feed exposes row-level changes for supported versions when enabled.
Incorrect Answers
A:Delete a few rows and confirm snapshot consistency without unnecessary rewrites. This validate action addresses a different case (deletion vectors), where each small delete rewrites large data files and increases write cost. Deletion vectors can mark deleted rows without immediate whole-file rewrite when supported. In the stated problem, the consumer reads only newly appended rows and cannot see updates or deletes; this choice would not establish the required change data feed consumers decision.
B:Verify shuffle spill decreases without excessive scheduling overhead. This validate action addresses a different case (shuffle partition count), where the current partition count is too low for data volume. Shuffle partition sizing affects memory pressure and task parallelism. In the stated problem, the consumer reads only newly appended rows and cannot see updates or deletes; this choice would not establish the required change data feed consumers decision.
D:Run representative queries and verify selected layouts improve skipping. This validate action addresses a different case (automatic clustering), where manual clustering keys quickly become stale. Automatic liquid clustering can select keys using query workload when platform prerequisites are met. In the stated problem, the consumer reads only newly appended rows and cannot see updates or deletes; this choice would not establish the required change data feed consumers decision.
Question 768
At Oak Valley Retail, a new optional field is added to a bronze event feed. The known risk is: The silver writer rejects every new record after the upstream schema change. After remediation, which ongoing signal best detects recurrence?
- Monitor expectation-drop counts and upstream correction rates
- Monitor cast failure and schema drift counts
- Alert when active-key uniqueness violations recur
- Monitor unexpected incompatible type changes
Explanation
Correct Answer
D:Monitor unexpected incompatible type changes. This watches the failure mode described by the silver writer rejects every new record after the upstream schema change. Schema evolution must be reviewed rather than enabling unrestricted acceptance for every column. It supplies a regression signal after remediation.
Incorrect Answers
A:Monitor expectation-drop counts and upstream correction rates. This monitor action addresses a different case (expectation drop action), where invalid rows currently reach downstream aggregates. A Lakeflow expectation configured to drop invalid rows filters violations from target output. In the stated problem, the silver writer rejects every new record after the upstream schema change; this choice would not establish the required controlled schema evolution decision.
B:Monitor cast failure and schema drift counts. This monitor action addresses a different case (silver schema enforcement), where implicit type coercion risks invalid numeric values being stored. Schema validation and explicit casts make type conversion failures observable. In the stated problem, the silver writer rejects every new record after the upstream schema change; this choice would not establish the required controlled schema evolution decision.
C:Alert when active-key uniqueness violations recur. This monitor action addresses a different case (uniqueness quality rule), where an upstream replay creates multiple active versions. An explicit uniqueness check identifies duplicate business keys before serving data. In the stated problem, the silver writer rejects every new record after the upstream schema change; this choice would not establish the required controlled schema evolution decision.
Question 769
A deployment owner at Harborline Utilities needs to implement a remedy: a shared ETL project uses development and production Lakeflow Jobs. The present failure is: Development job arguments were deployed into the production target. Which change should be applied?
- Set explicit target overrides and deploy using the approved production target
- Deploy the wheel task with its documented entrypoint and parameters
- Use the Databricks CLI to validate and deploy the bundle target
- Build transformations and invoke an explicit write or count action
Explanation
Correct Answer
A:Set explicit target overrides and deploy using the approved production target. This directly changes the feature or logic behind development job arguments were deployed into the production target. Bundle targets can override environment-specific variables and resource settings. It is the concrete configuration for the stated incident.
Incorrect Answers
B:Deploy the wheel task with its documented entrypoint and parameters. This implement action addresses a different case (python task entrypoints), where the task exits before setting up its argument parser. A standard Python entrypoint and explicit arguments avoid notebook runtime assumptions. In the stated problem, development job arguments were deployed into the production target; this choice would not establish the required bundle target overrides decision.
C:Use the Databricks CLI to validate and deploy the bundle target. This implement action addresses a different case (databricks cli resource updates), where production configuration drifts from repository source definitions. Declarative deployment describes resources as reviewed version-controlled artifacts. In the stated problem, development job arguments were deployed into the production target; this choice would not establish the required bundle target overrides decision.
D:Build transformations and invoke an explicit write or count action. This implement action addresses a different case (spark lazy evaluation), where engineers assume an intermediate filter executed before any action. Spark transformations are lazy until an action triggers a job. In the stated problem, development job arguments were deployed into the production target; this choice would not establish the required bundle target overrides decision.
Question 770
Operations at Silver Ridge Logistics owns a workload where a data engineering job accesses a protected external API. The previously identified issue was: An API token is committed in a Python notebook tracked in Git. Which metric or audit check best detects drift?
- Monitor Python worker execution time and error rates
- Audit access to the secret scope and changes to job identity
- Detect unexpected package version changes in startup validation
- Detect unmanaged changes to deployed job resources
Explanation
Correct Answer
B:Audit access to the secret scope and changes to job identity. This watches the failure mode described by an API token is committed in a Python notebook tracked in Git. Secret-backed configuration avoids embedding credentials in notebooks or deployment YAML. It supplies a regression signal after remediation.
Incorrect Answers
A:Monitor Python worker execution time and error rates. This monitor action addresses a different case (vectorized udf), where row-by-row Python invocation dominates stage runtime. Pandas UDFs process batches through Arrow rather than per-row Python calls. In the stated problem, an API token is committed in a Python notebook tracked in Git; this choice would not establish the required secure configuration decision.
C:Detect unexpected package version changes in startup validation. This monitor action addresses a different case (library precedence), where a notebook-scoped installation overrides the expected cluster version. The runtime can resolve conflicting versions from different installation scopes. In the stated problem, an API token is committed in a Python notebook tracked in Git; this choice would not establish the required secure configuration decision.
D:Detect unmanaged changes to deployed job resources. This monitor action addresses a different case (databricks cli resource updates), where production configuration drifts from repository source definitions. Declarative deployment describes resources as reviewed version-controlled artifacts. In the stated problem, an API token is committed in a Python notebook tracked in Git; this choice would not establish the required secure configuration decision.