Energy

What a coding task costs in watt-hours, measured on a Mac.

loading the results…

Every run, drawn

Each bar is one run's energy on this Mac, above idle, grouped by task. A failed run is faded: it still cost energy, and it counts against the model's Wh per solved task.

How it's measured

The Mac's System Management Controller reports how much power the whole machine draws from its supply (PSTR, in watts). Frog reads it ten times a second, without root, and adds it up over each task.

• Idle comes off first. Right before each run, with the model loaded, frog waits two seconds for the Mac to settle and measures its power for five seconds doing nothing. That baseline times the run's duration is subtracted, so what's left is the extra energy the task caused. Measuring before every run, not once per suite, keeps up with the Mac's own drift.
• Only the turn counts. Measurement runs from the prompt to the answer: the model thinking, frog's tools, the commands it runs (tests included). The bench's own pass/fail check afterwards is left out.
• Runs are hermetic. Each task starts from a clean copy of its repository, and frog sees only that repository: none of the machine owner's instructions, skills, MCP servers or search.
• The chip's counters are kept as a breakdown. Apple silicon also counts GPU and Neural Engine energy per component, and the raw runs below include those. They aren't the headline: on macOS 27 the CPU and memory counters stopped advancing on every Apple silicon chip (macmon#76), and they never saw the rest of the machine anyway.

What's left out

• Manufacturing. Making the Mac, the servers and the network costs energy too. None of it is here.
• The network, between the Mac and a hosted model.
• Power supply losses upstream of the sensor, which are small on a laptop and larger on a desktop's supply.
• Downloading the model, a one-time cost that's large for local models.

And some noise stays in: anything else the Mac does during a run lands in the measurement, and the idle baseline moves with it. A run whose baseline came in high can even read as zero. Runs are repeated, and the raw numbers are all below, so you can see the spread.

Hosted models are estimates

Nobody publishes measured energy per request for most models. For a hosted model, frog measures what the Mac spent (it still runs the tools and tests locally) and adds an estimate for the data center from the tokens used, per model:

• The reference is Epoch AI's estimate for GPT-4o on H100s, server and data center overhead included, assuming about 100B active parameters: —
• Open-weight models publish their size, and inference work scales with active parameters (two operations per active parameter per token), so each scales the reference by its own count, from its model card.
• Closed models (Claude, GPT, Gemini, Grok) don't publish their size, so they keep the reference as is, and say so.

None of this accounts for prompt caching, which agent loops lean on heavily, or for how efficiently a provider serves a model, and later work finds widely cited per-query estimates are 4–20× higher than production-optimized serving (Joule, 2026). Read hosted numbers as the high end. They're marked ~ and hatched everywhere they appear.

Try a turn

What a turn of a given size costs on each model: measured rates for the models benched on this Mac, the estimates above for hosted ones. An agent turn usually reads far more than it writes, because every call re-reads the conversation.

Grid

Grams of CO₂e use the world's average electricity.

Every run

The bench's own export, with no paths or output. Idle and system energy are per run, so any figure above can be recomputed.

Run it yourself

The suite and the bench are in the repository. On an Apple silicon Mac:

cargo run --release -p frog-bench -- run -m apple/on-device -r 3
cargo run --release -p frog-bench -- export ~/.local/share/frog/bench/<run>/results.jsonl

Close what you can first; the idle baseline is only as quiet as the Mac.