Databricks-Certified-Professional-Data-Engineer Free Practice Test

A data architect has heard about lake's built-in versioning and time travel capabilities. For auditing purposes they have a requirement to maintain a full of all valid street addresses as they appear in the customers table.
The architect is interested in implementing a Type 1 table, overwriting existing records with new values and relying on Delta Lake time travel to support long-term auditing. A data engineer on the project feels that a Type 2 table will provide better performance and scalability.
Which piece of information is critical to this decision?

1. A. Delta Lake time travel does not scale well in cost or latency to provide a long-term versioning solution.
2. B. Delta Lake time travel cannot be used to query previous versions of these tables because Type 1 changes modify data files in place.
3. C. Shallow clones can be combined with Type 1 tables to accelerate historic queries for long-term versioning.
4. D. Data corruption can occur if a query fails in a partially completed state because Type 2 tables requiresSetting multiple fields in a single update.

Correct Answer: A
Delta Lake's time travel feature allows users to access previous versions of a table, providing a powerful tool for auditing and versioning. However, using time travel as a long-term versioning solution for auditing purposes can be less optimal in terms of cost and performance, especially as the volume of data and the number of versions grow. For maintaining a full history of valid street addresses as they appear in a customers table, using a Type 2 table (where each update creates a new record with versioning) might provide better scalability and performance by avoiding the overhead associated with accessing older versions of a large table. While Type 1 tables, where existing records are overwritten with new values, seem simpler and can leverage time travel for auditing, the critical piece of information is that time travel might not scale well in cost or latency for long- term versioning needs, making a Type 2 approach more viable for performance and scalability.References:
✑ Databricks Documentation on Delta Lake's Time Travel: Delta Lake Time Travel
✑ Databricks Blog on Managing Slowly Changing Dimensions in Delta Lake: Managing SCDs in Delta Lake

A Delta Lake table representing metadata about content posts from users has the following schema:
user_id LONG, post_text STRING, post_id STRING, longitude FLOAT, latitude FLOAT, post_time TIMESTAMP, date DATE
This table is partitioned by the date column. A query is run with the following filter: longitude < 20> -20
Which statement describes how data will be filtered?

1. A. Statistics in the Delta Log will be used to identify partitions that might Include files in the filtered range.
2. B. No file skipping will occur because the optimizer does not know the relationship between the partition column and the longitude.
3. C. The Delta Engine will use row-level statistics in the transaction log to identify the flies that meet the filter criteria.
4. D. Statistics in the Delta Log will be used to identify data files that might include records in the filtered range.
5. E. The Delta Engine will scan the parquet file footers to identify each row that meets the filter criteria.

Correct Answer: D
This is the correct answer because it describes how data will be filtered when a query is run with the following filter: longitude < 20 & longitude > -20. The query is run on a Delta Lake table that has the following schema: user_id LONG, post_text STRING, post_id STRING, longitude FLOAT, latitude FLOAT, post_time TIMESTAMP, date DATE. This table is partitioned by the date column. When a query is run on a partitioned Delta Lake table, Delta Lake uses statistics in the Delta Log to identify data files that might include records in the filtered range. The statistics include information such as min and max values for each column in each data file. By using these statistics, Delta Lake can skip reading data files that do not match the filter condition, which can improve query performance and reduce I/O costs. Verified References: [Databricks Certified Data Engineer Professional], under “Delta Lake” section; Databricks Documentation, under “Data skipping” section.

A junior data engineer has manually configured a series of jobs using the Databricks Jobs UI. Upon reviewing their work, the engineer realizes that they are listed as the "Owner" for each job. They attempt to transfer "Owner" privileges to the "DevOps" group, but cannot successfully accomplish this task.
Which statement explains what is preventing this privilege transfer?

1. A. Databricks jobs must have exactly one owner; "Owner" privileges cannot be assigned to a group.
2. B. The creator of a Databricks job will always have "Owner" privileges; this configuration cannot be changed.
3. C. Other than the default "admins" group, only individual users can be granted privileges on jobs.
4. D. A user can only transfer job ownership to a group if they are also a member of that group.
5. E. Only workspace administrators can grant "Owner" privileges to a group.

Correct Answer: E
The reason why the junior data engineer cannot transfer “Owner” privileges to the “DevOps” group is that Databricks jobs must have exactly one owner, and the owner must be an individual user, not a group. A job cannot have more than one owner, and a job cannot have a group as an owner. The owner of a job is the user who created the job, or the user who was assigned the ownership by another user. The owner of a job has the highest level of permission on the job, and can grant or revoke permissions to other users or groups. However, the owner cannot transfer the ownership to a group, only to another user. Therefore, the junior data engineer’s attempt to transfer “Owner” privileges to the “DevOps” group is not possible. References:
✑ Jobs access control: https://docs.databricks.com/security/access-control/table-acls/index.html
✑ Job permissions: https://docs.databricks.com/security/access-control/table-acls/privileges.html#job-permissions

The data science team has created and logged a production using MLFlow. The model accepts a list of column names and returns a new column of type DOUBLE.
The following code correctly imports the production model, load the customer table containing the customer_id key column into a Dataframe, and defines the feature columns needed for the model.

Which code block will output DataFrame with the schema'' customer_id LONG, predictions DOUBLE''?

1. A. Model, predict (df, columns)
2. B. Df, map (lambda k:midel (x [columns]) ,select (''customer_id predictions'')
3. C. D
4. D. Select (''customer_id''. Model (''columns) alias (''predictions'')
5. E. Df.apply(model, columns). Select (''customer_id, prediction''

Correct Answer: A
Given the information that the model is registered with MLflow and assuming predict is the method used to apply the model to a set of columns, we use the model.predict() function to apply the model to the DataFrame df using the specified columns. The model.predict() function is designed to take in a DataFrame and a list of column names as arguments, applying the trained model to these features to produce a predictions column. When working with PySpark, this predictions column needs to be selected alongside the customer_id to create a new DataFrame with the schema customer_id LONG, predictions DOUBLE.
References:
✑ MLflow documentation on using Python function models: https://www.mlflow.org/docs/latest/models.html#python-function-python
✑ PySpark MLlib documentation on model prediction: https://spark.apache.org/docs/latest/ml-pipeline.html#pipeline

In order to prevent accidental commits to production data, a senior data engineer has instituted a policy that all development work will reference clones of Delta Lake tables. After testing both deep and shallow clone, development tables are created using shallow clone.
A few weeks after initial table creation, the cloned versions of several tables implemented as Type 1 Slowly Changing Dimension (SCD) stop working. The transaction logs for the source tables show that vacuum was run the day before.
Why are the cloned tables no longer working?

1. A. The data files compacted by vacuum are not tracked by the cloned metadata; running refresh on the cloned table will pull in recent changes.
2. B. Because Type 1 changes overwrite existing records, Delta Lake cannot guarantee data consistency for cloned tables.
3. C. The metadata created by the clone operation is referencing data files that were purged as invalid by the vacuum command
4. D. Running vacuum automatically invalidates any shallow clones of a table; deep clone should always be used when a cloned table will be repeatedly queried.

Correct Answer: C
In Delta Lake, a shallow clone creates a new table by copying the metadata of the source table without duplicating the data files. When the vacuum command is run on the source table, it removes old data files that are no longer needed to maintain the transactional log's integrity, potentially including files referenced by the shallow clone's metadata. If these files are purged, the shallow cloned tables will reference non-existent data files, causing them to stop working properly. This highlights the dependency of shallow clones on the source table's data files and the impact of data management operations like vacuum on these clones.References: Databricks documentation on Delta Lake, particularly the sections on cloning tables (shallow and deep cloning) and data retention with the vacuum command (https://docs.databricks.com/delta/index.html).