Free Databricks Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Practice Exam with Questions & Answers | Set: 4

The most efficient and correct approach is to define a schema using StructType and pass it to spark.read.schema(...).

This avoids schema inference overhead and ensures proper data types are enforced during read.

Example:

from pyspark.sql.types import StructType, StructField, StringType, DoubleType

schema = StructType([

StructField("order_id", StringType(), True),

StructField("amount", DoubleType(), True),

])

df = spark.read.schema(schema).json("path/to/json")

— Source: Databricks Guide – Read JSON with predefined schema

dedup_df = iot_bronze_df.dropDuplicates(["event_ts", "sensor_id", "metric_value"])

dropDuplicates accepts a list of columns to use for deduplication.

This ensures only unique records based on the specified keys are retained.

Spark Connect enables decoupling of the client and Spark driver processes, allowing remote access. Spark supports configuring the remote Spark Connect server in multiple ways:

From Databricks and Spark documentation:

Option B (--remote "sc://localhost") is a valid command-line argument for the pyspark shell to connect using Spark Connect.

Option C (setting SPARK_REMOTE environment variable) is also a supported method to configure the remote endpoint.

Option A is incorrect because Spark Connect uses the sc:// protocol, not https:// .

Option D requires modifying the code, which the question explicitly avoids.

Option E configures the port on the server side but doesn’t start a client connection.

Final Answers: B and C

The .mode("overwrite") ensures that existing files at the path will be replaced.

partitionBy("country") optimizes queries by writing data into partitioned folders.

Correct syntax:

df.write.mode("overwrite").partitionBy("country").parquet("/data/output")

— Source: Spark SQL, DataFrames and Datasets Guide

Spark Connect introduces a decoupled client-server architecture. Its key feature is enabling Spark job submission and execution from remote clients — in Python, Java, etc.

From Databricks documentation:

“Spark Connect allows remote clients to connect to a Spark cluster and execute Spark jobs without being co-located with the Spark driver.”

A is close, but "any language" is overstated (currently supports Python, Java, etc., not literally all).

B refers to REST, which is not Spark Connect's mechanism.

D is incorrect; Spark Connect isn’t focused on ingestion.

Final Answer: C

To flatten an array of structs into individual rows and access fields within each struct, you must:

Use explode() to expand the array so each struct becomes its own row.

Access the struct fields via dot notation (e.g., record_exploded.sensor_id).

Option C does exactly that:

First, explode the record array column into a new column record_exploded.

Then, access fields of the struct using the dot syntax in select.

This is standard practice in PySpark for nested data transformation.

Final Answer: C

In Spark SQL, the function date_sub(startDate, days) returns the date that is days before startDate.

If the days parameter is negative, Spark interprets it as subtracting a negative number, which effectively adds days to the date.

Example:

from pyspark.sql.functions import date_sub, lit

df = spark.createDataFrame([( "2024-10-01", )], ["dt"])

df.select(date_sub("dt", -5).alias("new_date")).show()

Output:

+----------+

| new_date |

+----------+

|2024-10-06|

+----------+

Why the other options are incorrect:

B: No error occurs; negative values are supported.

C: The start date changes if days ≠ 0.

D: Subtracting days would move the date backward, not forward.

A Spark job is triggered by an action (e.g., count, show).

The job is broken into stages, typically one per shuffle boundary.

Each stage is divided into multiple tasks, which are distributed across worker nodes.

In PySpark, when performing a join between two DataFrames, the columns do not have to share the same name. You can explicitly provide a join condition by comparing specific columns from each DataFrame.

This syntax is correct and fully supported:

df1.join(df2, df1.employee_id == df2.emp_id, how='inner')

This will perform an inner join between df1 and df2 using the employee_id from df1 and emp_id from df2.