hostprobe: queue cap 8, 4 workers, but 100 requests all return 202 and goroutines hit ~100

Max8 Jul

I'm building hostprobe, a small internal service that cron and a few scripts POST to as /probe?host=NAME. Four workers, queue of 8, and the probe is a 20s sleep for now (the ssh check comes later). I want it to stay small when a script goes wild.

Test: hey -n 100 -c 100 against it. All 100 came back 202 in a few ms. I expected the 9th onward to be slowed or rejected. /stats says queue len=8 cap=8 and 4 running, so the channel and workers look right. But runtime.NumGoroutine() is about 100, and /debug/pprof/goroutine?debug=1 shows one big group sitting at the same spot. With 1000 requests it went to about 1000 while the queue stayed at 8. Activity Monitor memory also climbs a bit with each burst.

I haven't touched the handler or pool in a while, so I'm basically staring at code I forgot. Handler:

job := Job{Host: host, Enqueued: time.Now()}

go func() {
	q.jobs <- job
}()

atomic.AddInt64(&q.accepted, 1)
w.WriteHeader(http.StatusAccepted)
fmt.Fprintf(w, "queued %s\n", host)

The pool is the usual for job := range q.jobs in 4 goroutines, and make(chan Job, cfg.QueueDepth).

Where are the extra goroutines coming from? I suspect the workers or the channel, but the counts say both are fine.

The conversation continues 3

Follow the reasoning

sourdough8 Jul Permalink
I think the go func() { q.jobs <- job }() is your whole answer. The channel is doing its job: 8 sit in the buffer, 4 are held by workers, and every other send blocks. But each blocked send is its own goroutine, parked on the channel send, and the handler has already moved on to the 202. So the queue never pushes back on anyone, it just gets a waiting room of goroutines behind it. That's your big group in pprof, and it's why 1000 requests gives you about 1000 goroutines. Each one holds a stack and the job too, so memory creeping up after each burst fits.

If you want the 9th onward rejected, do the send in the handler with a select:
```
select {
case q.jobs <- job:
	atomic.AddInt64(&q.accepted, 1)
	w.WriteHeader(http.StatusAccepted)
	fmt.Fprintf(w, "queued %s\n", host)
case <-r.Context().Done():
	return
default:
	http.Error(w, "queue full", http.StatusServiceUnavailable)
}
```
With the default it fails straight away. Drop the default if you'd rather the caller waits, since the context case lets it give up when the client goes. A 503 (or 429, if you fancy it) with a Retry-After is the usual thing for a cron script.

Also, I think "queued" is a slightly generous word for what the old code said. It was more "promised to queue, eventually".
Max9 Jul Permalink
Following an earlier reply ↗
That was it. Switched to the select with the default, and the 9th request onward gets a 503 now. Goroutines stay flat at 4 workers plus the usual. Thanks!
Max10 Jul Permalink
Following an earlier reply ↗
Basically a follow-up for anyone reading later: /debug/pprof/goroutine?debug=1 is how I spotted the pile, and the net/http/pprof docs are at https://pkg.go.dev/net/http/pprof.

hostprobe: queue cap 8, 4 workers, but 100 requests all return 202 and goroutines hit ~100

The conversation continues 3

Other sides of the same question

canceled context but worker goroutines still alive after first error (always workers + 2)

Marking the filter async did nothing for the freeze

await for over ReceivePort never finishes after isolates are done