openlang

GOMAXPROCS in a container now follows the CPU limit

Before Go 1.25, GOMAXPROCS defaulted to the logical CPU count of the machine. A container limited to two CPUs on a host with 32 logical CPUs got 32. That's 32 Ps running goroutines against a quota worth two CPUs of time per period, so the process burns through its quota early and sits throttled for the rest of each period.

Go 1.25 changed the default on Linux. The runtime reads the cgroup CPU bandwidth limit and uses it when it's lower than the CPU count. A fractional limit rounds up. The default never goes below 2 unless the machine itself has fewer CPUs. It also rechecks periodically, so a limit changed while the process runs gets picked up.

Three ways to not get it. GOMAXPROCS in the environment, a call to runtime.GOMAXPROCS with a value, or a go line of 1.24 or lower in go.mod, because containermaxprocs=0 is the default there. And it reads the limit only. A CPU request without a limit changes nothing. In Kubernetes terms that's limits.cpu, and a pod with only requests.cpu keeps the old default.

The probe I use prints two lines.

package main

import (
	"fmt"
	"runtime"
)

func main() {
	fmt.Println("GOMAXPROCS", runtime.GOMAXPROCS(0))
	fmt.Println("NumCPU", runtime.NumCPU())
}

Built with this go.mod, so the new default applies.

module probe

go 1.25

NumCPU is the number the old default used. If the two lines match inside a container whose limit is below the CPU count, one of the three things above is in the way.

So the number I care about went from 32 runnable Ps to two.

The conversation continues 6

Follow the reasoning
  1. Permalink

    Most services can probably just delete whatever they did at startup to set this by hand. Before 1.25 the usual move was a small library that read the cgroup files and called runtime.GOMAXPROCS with the result. A call with a value now also switches off the periodic updates, so leaving it in costs you the new behavior for nothing.

    The other leftover to look for is GOMAXPROCS set in a deployment file. Same effect. Once both are gone, printing runtime.GOMAXPROCS(0) at startup, which only reads the value, is enough to see what the runtime picked.

  2. Permalink

    --cpuset-cpus is the wrong flag for this. Under docker run --cpuset-cpus=0 the probe from the root prints 1 and 1 whether go.mod says go 1.25 or the go line is set back to 1.24.

    A cpuset shrinks the affinity mask, and Go has counted affinity for a long time, so the old default already gives 1. The floor of 2 doesn't apply when the mask itself holds one CPU. The new code is about the bandwidth quota, which is what --cpus sets. The --cpus run is next, and the old go line stays on Vadim's list until it shows something.

  3. Permalink

    Not sold on reading the number back as proof of anything. Fine, it tells you the runtime saw the limit. What you wanted gone was throttling, and the cgroup counts that itself in cpu.stat. Did anyone compare nr_throttled before and after under the same load, or are we all just printing GOMAXPROCS?

  4. Permalink

    Two reads of the same file, before and after a load run.

    grep -E '^(nr_periods|nr_throttled|throttled_usec)' \
      /sys/fs/cgroup/cpu.stat

    That's cgroup v2 from inside the container. nr_throttled over nr_periods is the share of periods that hit the quota. Compare the deltas, since the totals count everything from when the cgroup was created.

  5. Permalink

    With your grep around a --cpus=2 run I'd expect nr_throttled to drop with go 1.25 but not to zero. Two Ps can't run more than two CPUs' worth of Go code, but threads that run without a P, sysmon among them, still use quota, and the quota is enforced per period. A spin loop per core on an eight-CPU VM, once with the old go line and once with the new one, should show it in the deltas.

Other sides of the same question

  1. try vs catch unreachable compares two different programs

    try and catch unreachable don't make the same promise, so timing one against the other measures two different programs. Both…

    Replies 4zig
  2. Does this queue need a cached count?

    The only thing the caller ever asks this queue is List.length items > 8, and it asks repeatedly. The queue is one immutable O…

  3. Which build time are you quoting?

    Every crate-splitting thread ends with someone posting a single build time. There are two, and they move in different directi…

    Replies 2rust