Is My Spark App Over-Provisioned? Core Busyness and Peak Memory (Spark App Doctor, Part 1)

Is My Spark App Over-Provisioned? Core Busyness and Peak Memory (Spark App Doctor, Part 1)

Click to open full size.

Two ratios tell you whether a Spark app is over-provisioned. How much of the core time you paid for ran tasks, and how much of the memory you asked for did the executors actually touch. Both are already recorded by Spark, in the UI and in the REST API, and you can compute them for any finished app in a few seconds.

This is part 1 of a three-part series on judging a Spark app from its own metrics:

The Scenario

A nightly batch job that nobody has looked at since it was first sized.

Illustrative figures: assumed, not measured.

From those numbers: 18% of the core time ran tasks, and the busiest executor used about a third of its heap. This post shows where each number comes from, and what to cut.

The Two Numbers

Everything you need is in three REST endpoints. They are served by the live UI on port 4040 while the app runs, and by the History Server on port 18080 after it finishes:

APP=http://localhost:18080/api/v1/applications/<app-id>   # add /<attempt-id> on YARN cluster mode

curl -s $APP                  # start/end time, completed flag
curl -s $APP/allexecutors     # active AND dead executors: cores, task time, peaks
curl -s $APP/environment      # spark.executor.memory and every other setting

Use /allexecutors, not /executors. The second one lists only active executors, so any executor removed during the run (by dynamic allocation or a failure) is missing from it.

How Each Number Is Built

1. Slot utilisation: core time held vs core time used

An executor with 4 cores holds 4 task slots from the moment it is added until it is removed. Each slot either runs a task or sits idle. Slot utilisation is the share of slot-time that ran tasks.

The fields, all on each /allexecutors entry:

totalDuration is the sum of task durations on that executor. Four tasks running at once for 10 seconds add 40 seconds, which is why you divide it by uptime × cores and not by uptime alone.

Skip the entry with "id": "driver". It reports the driver, not a worker.

2. Where idle slot time comes from

A low utilisation number does not tell you why the slots were idle. There are three common shapes, and only the first two are fixed by sizing.

1. Too many slots for the work         2. Long tail                           3. Driver gaps
   (every stage)                          (end of a stage)                       (between stages)

slot 1 |██████░░░░░░░░░░|              slot 1 |████████░░░░░░░░|              slot 1 |████░░░░░░░░████|
slot 2 |████░░░░░░░░░░░░|              slot 2 |███████░░░░░░░░░|              slot 2 |███░░░░░░░░░████|
slot 3 |░░░░░░░░░░░░░░░░|              slot 3 |█████████░░░░░░░|              slot 3 |████░░░░░░░░███░|
slot 4 |░░░░░░░░░░░░░░░░|              slot 4 |████████████████|              slot 4 |██░░░░░░░░░░████|
        fewer tasks than slots                 one slow task holds the stage          collect, listing files,
                                                                                      planning, Python on driver
fix: fewer executors                   fix: Part 2 (skew, partitioning)       fix: code, not cluster size

The fastest way to tell them apart is the Event Timeline on the Jobs page. It shows executors being added and removed alongside each job and stage. Gaps between stage bars are shape 3. The script below also measures shape 3 directly: wall time minus the time covered by at least one stage.

3. Peak memory: what the executor actually touched

An executor’s memory is several nested regions, and Spark reports a peak for most of them. Picking the right denominator is where most over-provisioning analysis goes wrong.

Unless you set spark.executor.memoryOverhead directly, overhead is executor memory × spark.executor.memoryOverheadFactor, with a minimum. The factor defaults to 0.10, or 0.40 for non-JVM jobs on Kubernetes.

The two ProcessTree* rows report 0 unless spark.executor.processTreeMetrics.enabled=true and the node has a /proc filesystem. Turn it on for PySpark jobs; without it, Python worker memory is invisible.

In the scenario, the unified region is about (16 GiB − 300 MiB) × 0.6 ≈ 9.4 GiB. A 5.3 GiB heap peak is 33% of the heap but 56% of maxMemory. Dividing by the wrong number makes the heap look much fuller than it is.

4. Where each number lives

The peak memory columns are hidden by default. Tick them under Show Additional Metrics at the top of the Executors tab.

The Executors tab already flags one problem for you: the Task Time (GC Time) cell is shaded red when GC time is over 10% of task time.

What about metrics.json? The driver also serves /metrics/json (the MetricsServlet sink, on by default). It holds the driver’s own metric sources, such as the DAG scheduler, the block manager and the dynamic allocation targets. It is not the place to read per-executor busyness or peaks. Those come from the REST API above, or from the Prometheus-format endpoint /metrics/executors/prometheus, which the docs mark experimental.

Measuring It

This script reads a finished or running app and prints both numbers, plus how much of the wall time had no stage running.

#!/usr/bin/env python3
"""Usage: python3 overprov.py http://localhost:18080/api/v1/applications/<app-id>"""
import json, sys, urllib.request
from datetime import datetime, timezone
from statistics import median

BASE = sys.argv[1].rstrip("/")
UNITS = {"k": 2**10, "m": 2**20, "g": 2**30, "t": 2**40}


def get(path=""):
    with urllib.request.urlopen(BASE + path) as r:
        return json.load(r)


def ts(s):  # Spark REST dates look like 2026-10-28T09:00:00.000GMT
    return datetime.strptime(s.replace("GMT", "+0000"), "%Y-%m-%dT%H:%M:%S.%f%z")


def to_bytes(s):  # "16g", "512m"; a bare number means MiB
    s = s.strip().lower().removesuffix("b")
    return int(float(s[:-1]) * UNITS[s[-1]]) if s[-1] in UNITS else int(s) * 2**20


attempt = get()["attempts"][-1]
start = ts(attempt["startTime"])
end = ts(attempt["endTime"]) if attempt["completed"] else datetime.now(timezone.utc)

execs = [e for e in get("/allexecutors") if e["id"] != "driver"]
slot_ms = busy_ms = gc_ms = 0
for e in execs:
    gone = ts(e["removeTime"]) if e.get("removeTime") else end
    slot_ms += (gone - ts(e["addTime"])).total_seconds() * 1000 * e["totalCores"]
    busy_ms += e["totalDuration"]
    gc_ms += e["totalGCTime"]

# Wall time covered by at least one stage; the rest is driver-side work.
spans = sorted((ts(s["submissionTime"]), ts(s["completionTime"]))
               for s in get("/stages") if "submissionTime" in s and "completionTime" in s)
covered, cur_s, cur_e = 0.0, None, None
for s, e in spans:
    if cur_e is None or s > cur_e:
        if cur_e is not None:
            covered += (cur_e - cur_s).total_seconds()
        cur_s, cur_e = s, e
    else:
        cur_e = max(cur_e, e)
if cur_e is not None:
    covered += (cur_e - cur_s).total_seconds()
wall = (end - start).total_seconds()

props = dict(get("/environment")["sparkProperties"])
heap = to_bytes(props.get("spark.executor.memory", "1g"))
peaks = [e.get("peakMemoryMetrics", {}).get("JVMHeapMemory", 0) for e in execs]
py = max(e.get("peakMemoryMetrics", {}).get("ProcessTreePythonRSSMemory", 0) for e in execs)

print(f"executors (all time)      {len(execs)}")
print(f"slot utilisation          {busy_ms / slot_ms:.0%}")
print(f"idle core-hours           {(slot_ms - busy_ms) / 3.6e6:,.1f}")
print(f"wall time with no stage   {1 - covered / wall:.0%}")
print(f"GC / task time            {gc_ms / busy_ms:.0%}")
print(f"peak heap / heap  max     {max(peaks) / heap:.0%}")
print(f"peak heap / heap  median  {median(peaks) / heap:.0%}")
print(f"peak Python RSS           {py / 2**30:.1f} GiB" if py else
      "peak Python RSS           n/a (processTreeMetrics off, or not PySpark)")

For the scenario it would print something like:

executors (all time)      40
slot utilisation          18%
idle core-hours           131.2
wall time with no stage   23%
GC / task time            3%
peak heap / heap  max     33%
peak heap / heap  median  27%
peak Python RSS           n/a (processTreeMetrics off, or not PySpark)

Illustrative output: computed from the assumed scenario, not from a real run.

It needs a real cluster. In local mode the driver is the only executor, so the list is empty after the driver is filtered out.

How the Pieces Fit

flowchart LR
    A["/allexecutors
cores, uptime,
task time, peaks"] --> U["Slot utilisation"]
    A --> M["Peak heap headroom"]
    E["/environment
spark.executor.memory"] --> M
    S["/stages
submission, completion"] --> G["Wall time with
no stage running"]
    U --> V{"Verdict"}
    M --> V
    G --> V
    V --> C1["Cut executors"]
    V --> C2["Cut memory per executor"]
    V --> C3["Fix driver code"]
    V --> C4["Sized right:
see Part 2 if slow"]

Reading the Result

The thresholds below are starting points that I use, not Spark rules. Tune them on your own jobs.

The scenario lands in the first row, with a sizeable driver gap. Cutting executors is safe. The 14 minutes of driver time won’t shrink either way, so expect the wall time to stay close to an hour.

Where It Hurts

When Not to Cut

Right-Sizing Safely

flowchart TD
    A["Measure baseline:
wall time, utilisation,
peak heap, spill, GC"] --> B{"No-stage share large?"}
    B -- yes --> B1["Fix driver code first"]
    B -- no --> C{"Utilisation low?"}
    C -- yes --> D["Cut executors by a step"]
    C -- no --> E{"Peak heap low?"}
    D --> R["Rerun on the same input"]
    E -- yes --> F["Cut memory per executor by a step"]
    E -- no --> Z["Done: sized about right"]
    F --> R
    R --> Q{"Wall time, spill
and GC still OK?"}
    Q -- yes --> C
    Q -- no --> P["Go back one step and stop"]

Change one dimension at a time. If you change executor count and memory together and the job slows down, you won’t know which change caused it.

Checking It Works

1. Run the script on the baseline and save the output with the input size for that run.
2. Cut one step (for example, half the executors) and rerun on the same input, or on an input of the same size.
3. Compare four things: wall time, slot utilisation, peak heap headroom, and disk spill on the Stages tab. Utilisation should rise. Wall time and spill should stay flat.
4. Check the Executors tab for red GC cells and for Exec Loss Reason entries. Either one means you cut too far.

Sources

Checked on 2026-10-28.