openlang

GOMAXPROCS in a container now follows the CPU limit

Before Go 1.25, GOMAXPROCS defaulted to the logical CPU count of the machine. A container limited to two CPUs on a host with 32 logical CPUs got 32. That's 32 Ps running goroutines against a quota worth two CPUs of time per period, so the process burns through its quota early and sits throttled for the rest of each period.

Go 1.25 changed the default on Linux. The runtime reads the cgroup CPU bandwidth limit and uses it when it's lower than the CPU count. A fractional limit rounds up. The default never goes below 2 unless the machine itself has fewer CPUs. It also rechecks periodically, so a limit changed while the process runs gets picked up.

Three ways to not get it. GOMAXPROCS in the environment, a call to runtime.GOMAXPROCS with a value, or a go line of 1.24 or lower in go.mod, because containermaxprocs=0 is the default there. And it reads the limit only. A CPU request without a limit changes nothing. In Kubernetes terms that's limits.cpu, and a pod with only requests.cpu keeps the old default.

The probe I use prints two lines.

package main

import (
	"fmt"
	"runtime"
)

func main() {
	fmt.Println("GOMAXPROCS", runtime.GOMAXPROCS(0))
	fmt.Println("NumCPU", runtime.NumCPU())
}

Built with this go.mod, so the new default applies.

module probe

go 1.25

NumCPU is the number the old default used. If the two lines match inside a container whose limit is below the CPU count, one of the three things above is in the way.

So the number I care about went from 32 runnable Ps to two.

后续讨论 8

顺着彼此的依据读下去
  1. 固定链接

    Most services can probably just delete whatever they did at startup to set this by hand. Before 1.25 the usual move was a small library that read the cgroup files and called runtime.GOMAXPROCS with the result. A call with a value now also switches off the periodic updates, so leaving it in costs you the new behavior for nothing.

    The other leftover to look for is GOMAXPROCS set in a deployment file. Same effect. Once both are gone, printing runtime.GOMAXPROCS(0) at startup, which only reads the value, is enough to see what the runtime picked.

  2. 固定链接

    --cpuset-cpus is the wrong flag for this. Under docker run --cpuset-cpus=0 the probe from the root prints 1 and 1 whether go.mod says go 1.25 or the go line is set back to 1.24.

    A cpuset shrinks the affinity mask, and Go has counted affinity for a long time, so the old default already gives 1. The floor of 2 doesn't apply when the mask itself holds one CPU. The new code is about the bandwidth quota, which is what --cpus sets. The --cpus run is next, and the old go line stays on Vadim's list until it shows something.

  3. 固定链接

    Not sold on reading the number back as proof of anything. Fine, it tells you the runtime saw the limit. What you wanted gone was throttling, and the cgroup counts that itself in cpu.stat. Did anyone compare nr_throttled before and after under the same load, or are we all just printing GOMAXPROCS?

  4. 固定链接

    Two reads of the same file, before and after a load run.

    grep -E '^(nr_periods|nr_throttled|throttled_usec)' \
      /sys/fs/cgroup/cpu.stat

    That's cgroup v2 from inside the container. nr_throttled over nr_periods is the share of periods that hit the quota. Compare the deltas, since the totals count everything from when the cgroup was created.

  5. 固定链接

    With your grep around a --cpus=2 run I'd expect nr_throttled to drop with go 1.25 but not to zero. Two Ps can't run more than two CPUs' worth of Go code, but threads that run without a P, sysmon among them, still use quota, and the quota is enforced per period. A spin loop per core on an eight-CPU VM, once with the old go line and once with the new one, should show it in the deltas.

  6. 固定链接

    The --cpus=2 run on the eight-CPU VM. With go 1.25 the probe prints GOMAXPROCS 2 and NumCPU 8. With the go line back at 1.24 it prints 8 and 8. So the old line goes on vadimk's list after all.

    Then the deltas from kpw's grep, eight spinning goroutines, same load length both times. The 1.24 build hit the quota in nearly every period and spent most of each period throttled. The 1.25 build still hit it in many periods, so nr_throttled fell less than I'd guessed. throttled_usec is where the difference shows. Two Ps run right at the quota and go over by a sliver, so each throttle is short. tam_ly's question had the right file, and I was watching the wrong line in it.

同一个问题的其他角度

  1. A slow gfortran loop that copied an array section on every call

    A loop over the rows of a matrix, calling a subroutine with the row a(i, :) on every iteration. Right answer, embarrassing sp…

    回复 2最近活动 fortran
  2. try vs catch unreachable compares two different programs

    try and catch unreachable don't make the same promise, so timing one against the other measures two different programs. Both…

    回复 4最近活动 zig
  3. Column-major loop order in Fortran

    Fortran arrays are column-major. a(i,j) and a(i+1,j) are neighbours in memory. a(i,j+1) is a whole column away. So the loop n…

    回复 1最近活动 fortran