Common PySpark Interview Questions and Answers

PySpark interview questions usually test two things at once: whether you understand the DataFrame API, and whether you understand what Spark is actually doing under the hood (lazy evaluation, shuffles, partitions). This article covers seven questions that come up often, each with real, runnable DataFrame API code and an explanation of the mechanism behind it — written for anyone preparing for a data engineering interview that includes a Spark round.

The example DataFrame

Every example below works against the same simple DataFrame:

orders(order_id, customer_id, region, amount)

All code assumes from pyspark.sql import functions as F and an existing Spark session named spark.

Transformations, actions, and lazy evaluation

What's the difference between a transformation and an action, and why does it matter?

# Transformations are lazy — nothing actually runs yet
filtered = orders.filter(F.col("amount") > 100)
selected = filtered.select("order_id", "customer_id", "amount")

# Actions trigger execution of the whole chain built so far
row_count = selected.count()     # action
rows = selected.collect()        # action

Methods like .filter() and .select() just build up a logical query plan — Spark doesn't touch any data until an action like .count() or .collect() is called. This matters because it lets Spark optimize the entire chain of transformations together before running anything, instead of executing each step eagerly and wastefully.

Working with columns and nulls

How do you add a derived column?

orders_with_tax = orders.withColumn("amount_with_tax", F.col("amount") * 1.08)

.withColumn() returns a new DataFrame with the extra column added — DataFrames in Spark are immutable, so this never modifies orders in place.

How do you handle missing values in a column?

# Drop rows where amount is missing
orders_clean = orders.dropna(subset=["amount"])

# Or replace missing amounts with 0 instead of dropping the row
orders_filled = orders.fillna({"amount": 0})

Use .dropna() when a missing value makes the row unusable, and .fillna() when a sensible default lets you keep the row. Passing a dict to .fillna() lets you set a different default per column.

Deduplication and window functions

How do you remove duplicate rows?

orders_unique = orders.dropDuplicates(["order_id"])

.dropDuplicates() keeps one row per unique value (or combination of values) in the given columns. Called with no arguments, it compares entire rows instead.

How do you rank orders within each region by amount?

from pyspark.sql.window import Window

region_window = Window.partitionBy("region").orderBy(F.col("amount").desc())

ranked_orders = orders.withColumn("rank_in_region", F.rank().over(region_window))

Window.partitionBy("region") splits the data into one group per region, and .orderBy() defines how rows are ordered within each group. F.rank().over(region_window) then assigns a rank to each row within its own region — without collapsing the DataFrame the way a .groupBy() aggregate would.

Joins and partitioning

How do you efficiently join a large DataFrame with a small lookup table?

region_lookup = spark.createDataFrame(
    [("US", "United States"), ("EU", "Europe")],
    ["region", "region_name"],
)

joined = orders.join(F.broadcast(region_lookup), on="region", how="left")

F.broadcast() tells Spark to send the small region_lookup table to every executor instead of shuffling the much larger orders DataFrame across the cluster to perform the join. This only makes sense when the broadcast table is genuinely small enough to fit comfortably in memory on every executor.

What's the difference between repartition() and coalesce()?

# repartition() does a full shuffle — can increase or decrease partition count
orders_more_partitions = orders.repartition(20)

# coalesce() avoids a full shuffle — can only decrease partition count
orders_fewer_partitions = orders.coalesce(4)

.repartition() redistributes data evenly across a new number of partitions (up or down) via a full shuffle, which is expensive but gives balanced partitions. .coalesce() only merges existing partitions together, so it can only reduce the count, but it avoids a full shuffle — which is why it's the usual choice right before writing output to fewer files.

For the rest of what a data engineering interview typically covers beyond Spark itself, see How to Prepare for a Data Engineer Interview, and pair this with Common SQL Interview Questions and Answers if the role also has a SQL round. Meetin AI can also help you structure answers like these live during a technical interview — see what it offers or create a free account.

FAQ

Do I need to know Spark's internals, or just the DataFrame API?

Both, to some degree. Knowing the API syntax gets you through basic questions, but understanding lazy evaluation, shuffles, and partitioning is usually what separates a strong answer from a memorized one when the interviewer asks a follow-up "why."

Is the RDD API still asked about in interviews?

Less often than a few years ago — most production code and most interviews now focus on the DataFrame API shown here, since it's faster and easier to optimize. It's still worth knowing RDDs exist and roughly what problem they solve, but you're unlikely to need to write RDD code live.

What's the most common mistake candidates make?

Forgetting that DataFrames are immutable and that nothing executes until an action is called — leading to confusion about why a chain of transformations "didn't do anything" when nothing has consumed the result yet.