Pass Databricks Databricks-Certified-Professional-Data-Engineer PDF Dumps | Recently Updated 129 Questions
Updated Test Engine to Practice Databricks-Certified-Professional-Data-Engineer Dumps & Practice Exam
NEW QUESTION # 62
A Databricks job has been configured with 3 tasks, each of which is a Databricks notebook. Task A does not depend on other tasks. Tasks B and C run in parallel, with each having a serial dependency on task A.
If tasks A and B complete successfully but task C fails during a scheduled run, which statement describes the resulting state?
- A. Unless all tasks complete successfully, no changes will be committed to the Lakehouse; because task C failed, all commits will be rolled back automatically.
- B. All logic expressed in the notebook associated with task A will have been successfully completed; tasks B and C will not commit any changes because of stage failure.
- C. All logic expressed in the notebook associated with tasks A and B will have been successfully completed; any changes made in task C will be rolled back due to task failure.
- D. All logic expressed in the notebook associated with tasks A and B will have been successfully completed; some operations in task C may have completed successfully.
- E. Because all tasks are managed as a dependency graph, no changes will be committed to the Lakehouse until ail tasks have successfully been completed.
Answer: D
Explanation:
The query uses the CREATE TABLE USING DELTA syntax to create a Delta Lake table from an existing Parquet file stored in DBFS. The query also uses the LOCATION keyword to specify the path to the Parquet file as /mnt/finance_eda_bucket/tx_sales.parquet. By using the LOCATION keyword, the querycreates an external table, which is a table that is stored outside of the default warehouse directory and whose metadata is not managed by Databricks. An external table can be created from an existing directory in a cloud storage system, such as DBFS or S3, that contains data files in a supported format, such as Parquet or CSV.
The resulting state after running the second command is that an external table will be created in the storage container mounted to /mnt/finance_eda_bucket with the new name prod.sales_by_store. The command will not change any data or move any files in the storage container; it will only update the table reference in the metastore and create a new Delta transaction log for the renamed table. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under
"ALTER TABLE RENAME TO" section; Databricks Documentation, under "Create an external table" section.
NEW QUESTION # 63
You would like to build a spark streaming process to read from a Kafka queue and write to a Delta table every
15 minutes, what is the correct trigger option
- A. trigger(processingTime = 15)
- B. trigger(process "15 minutes")
- C. trigger(15)
- D. trigger("15 minutes")
- E. trigger(processingTime = "15 Minutes")
Answer: E
Explanation:
Explanation
The answer is trigger(processingTime = "15 Minutes")
Triggers:
*Unspecified
This is the default. This is equivalent to using processingTime="500ms"
*Fixed interval micro-batches .trigger(processingTime="2 minutes")
The query will be executed in micro-batches and kicked off at the user-specified intervals
*One-time micro-batch .trigger(once=True)
The query will execute a single micro-batch to process all the available data and then stop on its own
*One-time micro-batch.trigger .trigger(availableNow=True) -- New feature a better version of (once=True) Databricks supports trigger(availableNow=True) in Databricks Runtime 10.2 and above for Delta Lake and Auto Loader sources. This functionality combines the batch processing approach of trigger once with the ability to configure batch size, resulting in multiple parallelized batches that give greater control for right-sizing batches and the resultant files.
NEW QUESTION # 64
The data science team has created and logged a production model using MLflow. The following code correctly imports and applies the production model to output the predictions as a new DataFrame namedpredswith the schema "customer_id LONG, predictions DOUBLE, date DATE".
The data science team would like predictions saved to a Delta Lake table with the ability to compare all predictions across time. Churn predictions will be made at most once per day.
Which code block accomplishes this task while minimizing potential compute costs?
- A.
- B. preds.write.format("delta").save("/preds/churn_preds")
- C. preds.write.mode("append").saveAsTable("churn_preds")
- D.
- E.
Answer: C
NEW QUESTION # 65
John Smith is a newly joined team member in the Marketing team who currently has access read access to sales tables but does not have access to delete rows from the table, which of the following commands help you accomplish this?
- A. GRANT USAGE ON TABLE table_name TO [email protected]
- B. GRANT MODIFY ON TABLE table_name TO [email protected]
- C. GRANT MODIFY TO TABLE table_name ON [email protected]
- D. GRANT DELETE ON TABLE table_name TO [email protected]
- E. GRANT DELETE TO TABLE table_name ON [email protected]
Answer: B
Explanation:
Explanation
The answer is GRANT MODIFY ON TABLE table_name TO [email protected] , please note INSERT, UPDATE, and DELETE are combined into one role called MODIFY.
Below are the list of privileges that can be granted to a user or a group, SELECT: gives read access to an object.
CREATE: gives the ability to create an object (for example, a table in a schema).
MODIFY: gives the ability to add, delete, and modify data to or from an object.
USAGE: does not give any abilities, but is an additional requirement to perform any action on a schema object.
READ_METADATA: gives the ability to view an object and its metadata.
CREATE_NAMED_FUNCTION: gives the ability to create a named UDF in an existing catalog or schema.
MODIFY_CLASSPATH: gives the ability to add files to the Spark classpath.
ALL PRIVILEGES: gives all privileges (is translated into all the above privileges
NEW QUESTION # 66
Which of the following technologies can be used to identify key areas of text when parsing Spark Driver log4j output?
- A. Julia
- B. pyspsark.ml.feature
- C. C++
- D. Regex
- E. Scala Datasets
Answer: D
Explanation:
Regex, or regular expressions, are a powerful way of matching patterns in text. They can be used to identify key areas of text when parsing Spark Driver log4j output, such as the log level, the timestamp, the thread name, the class name, the method name, and the message. Regex can be applied in various languages and frameworks, such as Scala, Python, Java, Spark SQL, and Databricks notebooks. References:
* https://docs.databricks.com/notebooks/notebooks-use.html#use-regular-expressions
* https://docs.databricks.com/spark/latest/spark-sql/udf-scala.html#using-regular-expressions-in-udfs
* https://docs.databricks.com/spark/latest/sparkr/functions/regexp_extract.html
* https://docs.databricks.com/spark/latest/sparkr/functions/regexp_replace.html
NEW QUESTION # 67
The viewupdatesrepresents an incremental batch of all newly ingested data to be inserted or updated in the customerstable.
The following logic is used to process these records.
Which statement describes this implementation?
- A. The customers table is implemented as a Type 0 table; all writes are append only with no changes to existing values.
- B. The customers table is implemented as a Type 3 table; old values are maintained as a new column alongside the current value.
- C. The customers table is implemented as a Type 2 table; old values are overwritten and new customers are appended.
- D. The customers table is implemented as a Type 2 table; old values are maintained but marked as no longer current and new values are inserted.
- E. The customers table is implemented as a Type 1 table; old values are overwritten by new values and no history is maintained.
Answer: D
Explanation:
The logic uses the MERGE INTO command to merge new records from the view updates into the table customers. The MERGE INTO command takes two arguments: a target table and a source table or view. The command also specifies a condition to match records between the target and the source, and a set of actions to perform when there is a match or not. In this case,the condition is to match records by customer_id, which is the primary key of the customers table. The actions are to update the existing record in the target with the new values from the source, and set the current_flag to false to indicate that the record is no longer current; and to insert a new record in the target with the new values from the source, and set the current_flag to true to indicate that the record is current. This means that old values are maintained but marked as no longer current and new values are inserted, which is the definition of a Type 2 table. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Merge Into (Delta Lake on Databricks)" section.
NEW QUESTION # 68
A member of the data engineering team has submitted a short notebook that they wish to schedule as part of a larger data pipeline. Assume that the commands provided below produce the logically correct results when run as presented.
Which command should be removed from the notebook before scheduling it as a job?
- A. Cmd 3
- B. Cmd 2
- C. Cmd 5
- D. Cmd 4
- E. Cmd 6
Answer: E
Explanation:
Explanation
Cmd 6 is the command that should be removed from the notebook before scheduling it as a job. This command is selecting all the columns from the finalDF dataframe and displaying them in the notebook. This is not necessary for the job, as the finalDF dataframe is already written to a table in Cmd 7. Displaying the dataframe in the notebook will only consume resources and time, and it will not affect the output of the job.
Therefore, Cmd 6 is redundant and should be removed.
The other commands are essential for the job, as they perform the following tasks:
Cmd 1: Reads the raw_data table into a Spark dataframe called rawDF.
Cmd 2: Prints the schema of the rawDF dataframe, which is useful for debugging and understanding the data structure.
Cmd 3: Selects all the columns from the rawDF dataframe, as well as the nested columns from the values struct column, and creates a new dataframe called flattenedDF.
Cmd 4: Drops the values column from the flattenedDF dataframe, as it is no longer needed after flattening, and creates a new dataframe called finalDF.
Cmd 5: Explains the physical plan of the finalDF dataframe, which is useful for optimizing and tuning the performance of the job.
Cmd 7: Writes the finalDF dataframe to a table called flat_data, using the append mode to add new data to the existing table.
NEW QUESTION # 69
Which of the following describes a scenario in which a data engineer will want to use a Job cluster instead of
an all-purpose cluster?
- A. A Databricks SQL query needs to be scheduled for upward reporting
- B. An ad-hoc analytics report needs to be developed while minimizing compute costs
- C. An automated workflow needs to be run every 30 minutes
- D. A data team needs to collaborate on the development of a machine learning model
- E. A data engineer needs to manually investigate a production error
Answer: C
NEW QUESTION # 70
A data engineer is configuring a pipeline that will potentially see late-arriving, duplicate records.
In addition to de-duplicating records within the batch, which of the following approaches allows the data engineer to deduplicate data against previously processed records as it is inserted into a Delta table?
- A. Perform an insert-only merge with a matching condition on a unique key.
- B. Rely on Delta Lake schema enforcement to prevent duplicate records.
- C. VACUUM the Delta table after each batch completes.
- D. Set the configuration delta.deduplicate = true.
- E. Perform a full outer join on a unique key and overwrite existing data.
Answer: A
Explanation:
To deduplicate data against previously processed records as it is inserted into a Delta table, you can use the merge operation with an insert-only clause. This allows you to insert new records that do not match any existing records based on a unique key, while ignoring duplicate records that match existing records. For example, you can use the following syntax:
MERGE INTO target_table USING source_table ON target_table.unique_key = source_table.unique_key WHEN NOT MATCHED THEN INSERT * This will insert only the records from the source table that have a unique key that is not present in the target table, and skip the records that have a matching key. This way, you can avoid inserting duplicate records into the Delta table.
References:
* https://docs.databricks.com/delta/delta-update.html#upsert-into-a-table-using-merge
* https://docs.databricks.com/delta/delta-update.html#insert-only-merge
NEW QUESTION # 71
A junior data engineer has configured a workload that posts the following JSON to the Databricks REST API endpoint 2.0/jobs/create.
Assuming that all configurations and referenced resources are available, which statement describes the result of executing this workload three times?
- A. The logic defined in the referenced notebook will be executed three times on the referenced existing all purpose cluster.
- B. Three new jobs named "Ingest new data" will be defined in the workspace, but no jobs will be executed.
- C. Three new jobs named "Ingest new data" will be defined in the workspace, and they will each run once daily.
- D. One new job named "Ingest new data" will be defined in the workspace, but it will not be executed.
- E. The logic defined in the referenced notebook will be executed three times on new clusters with the configurations of the provided cluster ID.
Answer: B
Explanation:
This is the correct answer because the JSON posted to the Databricks REST API endpoint 2.0/jobs/create defines a new job with a name, an existing cluster id, and a notebook task. However, it does not specify any schedule or trigger for the job execution. Therefore, three new jobs with the same name and configuration will be created in the workspace, but none of them will be executed until they are manually triggered or scheduled. Verified Reference: [Databricks Certified Data Engineer Professional], under "Monitoring & Logging" section; [Databricks Documentation], under "Jobs API - Create" section.
NEW QUESTION # 72
A data architect has heard about lake's built-in versioning and time travel capabilities. For auditing purposes they have a requirement to maintain a full of all valid street addresses as they appear in the customers table.
The architect is interested in implementing a Type 1 table, overwriting existing records with new values and relying on Delta Lake time travel to support long-term auditing. A data engineer on the project feels that a Type 2 table will provide better performance and scalability.
Which piece of information is critical to this decision?
- A. Delta Lake time travel does not scale well in cost or latency to provide a long-term versioning solution.
- B. Shallow clones can be combined with Type 1 tables to accelerate historic queries for long-term versioning.
- C. Delta Lake time travel cannot be used to query previous versions of these tables because Type 1 changes modify data files in place.
- D. Data corruption can occur if a query fails in a partially completed state because Type 2 tables requires Setting multiple fields in a single update.
Answer: A
Explanation:
Delta Lake's time travel feature allows users to access previous versions of a table, providing a powerful tool for auditing and versioning. However, using time travel as a long-term versioning solution for auditing purposes can be less optimal in terms of cost and performance, especially as the volume of data and the number of versions grow. For maintaining a full history of valid street addresses as they appear in a customers table, using a Type 2 table (where each update creates a new record with versioning) might provide better scalability and performance by avoiding the overhead associated with accessing older versions of a large table.
While Type 1 tables, where existing records are overwritten with new values, seem simpler and can leverage time travel for auditing, the critical piece of information is that time travel might not scale well in cost or latency for long-term versioning needs, making a Type 2 approach more viable for performance and scalability.References:
* Databricks Documentation on Delta Lake's Time Travel: Delta Lake Time Travel
* Databricks Blog on Managing Slowly Changing Dimensions in Delta Lake: Managing SCDs in Delta Lake
NEW QUESTION # 73
You currently working with the marketing team to setup a dashboard for ad campaign analysis, since the team is not sure how often the dashboard should be refreshed they have decided to do a manual refresh on an as needed basis. Which of the following steps can be taken to reduce the overall cost of the compute when the team is not using the compute?
*Please note that Databricks recently change the name of SQL Endpoint to SQL Warehouses.
- A. They can decrease the maximum bound of the SQL endpoint(SQL Warehouse) scaling range.
- B. They can turn on the Auto Stop feature for the SQL endpoint(SQL Warehouse).
- C. They can turn on the Serverless feature for the SQL endpoint(SQL Warehouse).
- D. They can turn on the Serverless feature for the SQL endpoint(SQL Warehouse) and change the Spot Instance Policy from "Reliability Optimized" to "Cost optimized"
- E. They can decrease the cluster size of the SQL endpoint(SQL Warehouse).
Answer: B
Explanation:
Explanation
The answer is, They can turn on the Auto Stop feature for the SQL endpoint(SQL Warehouse).
Use auto stop to automatically terminate the cluster when you are not using it.
NEW QUESTION # 74
A junior data engineer has manually configured a series of jobs using the Databricks Jobs UI. Upon reviewing their work, the engineer realizes that they are listed as the "Owner" for each job. They attempt to transfer
"Owner" privileges to the "DevOps" group, but cannot successfully accomplish this task.
Which statement explains what is preventing this privilege transfer?
- A. The creator of a Databricks job will always have "Owner" privileges; this configuration cannot be changed.
- B. Databricks jobs must have exactly one owner; "Owner" privileges cannot be assigned to a group.
- C. A user can only transfer job ownership to a group if they are also a member of that group.
- D. Other than the default "admins" group, only individual users can be granted privileges on jobs.
- E. Only workspace administrators can grant "Owner" privileges to a group.
Answer: B
Explanation:
The reason why the junior data engineer cannot transfer "Owner" privileges to the "DevOps" group is that Databricks jobs must have exactly one owner, and the owner must be an individual user, not a group. A job cannot have more than one owner, and a job cannot have a group as an owner. The owner of a job is the user who created the job, or the user who was assigned the ownership by another user. The owner of a job has the highest level of permission on the job, and can grant or revoke permissions to other users or groups. However, the owner cannot transfer the ownership to a group, only to another user. Therefore, the junior data engineer's attempt to transfer "Owner" privileges to the "DevOps" group is not possible. References:
* Jobs access control: https://docs.databricks.com/security/access-control/table-acls/index.html
* Job permissions:
https://docs.databricks.com/security/access-control/table-acls/privileges.html#job-permissions
NEW QUESTION # 75
The data governance team is reviewing code used for deleting records for compliance with GDPR. They note the following logic is used to delete records from the Delta Lake table namedusers.
Assuming thatuser_idis a unique identifying key and thatdelete_requestscontains all users that have requested deletion, which statement describes whether successfully executing the above logic guarantees that the records to be deleted are no longer accessible and why?
- A. Yes; Delta Lake ACID guarantees provide assurance that the delete command succeeded fully and permanently purged these records.
- B. Yes; the Delta cache immediately updates to reflect the latest data files recorded to disk.
- C. No; files containing deleted records may still be accessible with time travel until a vacuum command is used to remove invalidated data files.
- D. No; the Delta Lake delete command only provides ACID guarantees when combined with the merge into command.
- E. No; the Delta cache may return records from previous versions of the table until the cluster is restarted.
Answer: C
Explanation:
The code uses the DELETE FROM command to delete records from the users table that match a condition based on a join with another table called delete_requests, which contains all users that have requested deletion. The DELETE FROM command deletes records from a Delta Lake table by creating a new version of the table that does not contain the deleted records. However, this does not guarantee that the records to be deleted are no longer accessible, because Delta Lake supports time travel, which allows querying previous versions of the table using a timestamp or version number. Therefore, files containing deleted records may still be accessible with time travel until a vacuum command is used to remove invalidated data files from physical storage. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Delete from a table" section; Databricks Documentation, under
"Remove files no longer referenced by a Delta table" section.
NEW QUESTION # 76
A table named user_ltv is being used to create a view that will be used by data analysis on various teams.
Users in the workspace are configured into groups, which are used for setting up data access using ACLs.
The user_ltv table has the following schema:
An analyze who is not a member of the auditing group executing the following query:
Which result will be returned by this query?
- A. All columns will be displayed normally for those records that have an age greater than 17; records not meeting this condition will be omitted.
- B. All age values less than 18 will be returned as null values all other columns will be returned with the values in user_ltv.
- C. All columns will be displayed normally for those records that have an age greater than 18; records not meeting this condition will be omitted.
- D. All records from all columns will be displayed with the values in user_ltv.
Answer: C
Explanation:
Given the CASE statement in the view definition, the result set for a user not in the auditing group would be constrained by the ELSE condition, which filters out records based on age. Therefore, the view will return all columns normally for records with an age greater than 18, as users who are not in the auditing group will not satisfy theis_member('auditing')condition. Records not meeting theage > 18condition will not be displayed.
NEW QUESTION # 77
......
Databricks Databricks-Certified-Professional-Data-Engineer Dumps Cover Real Exam Questions: https://www.torrentvce.com/Databricks-Certified-Professional-Data-Engineer-valid-vce-collection.html