openlang

Eight queue slots can still leave 88 goroutines waiting

Partly reviewed

Suppose a server receives a burst of 100 requests, each carrying one job. Give it four workers and a channel with room for eight jobs. Assume no job finishes while the burst arrives. Four jobs can be running and eight can be queued. If every request has already launched a goroutine that sends its job, the other 88 goroutines can be waiting to send. The queue has a bound. That waiting population needs a separate one.

A buffered channel limits the values stored inside it. It does not decide how many goroutines reach a blocking send. Moving the send into a new goroutine only moves the wait. An unbuffered channel makes sender and receiver synchronize at the exchange, but it still does not cap the number of senders that arrive.

For this design, I would name the admission owner as the request entry point. It must obtain a place before launching additional work, or return a saturation result when no place is available. If admission waits instead, that wait also needs a population limit and a deadline. Otherwise the queue has merely moved upstream.

I would name the request handler as the owner of signaling cancellation when the caller leaves, and the workers as owners of checking that signal before starting queued work and at useful stopping points. A cancellation signal alone does not tell us when a place is released. The admission owner needs a rule that counts a job until it actually stops occupying the resource being limited.

For a batch already limited to twelve small jobs, launching twelve goroutines directly can be reasonable. A worker pool then adds coordination without necessarily improving the bound. Which component owns the limit when requests keep arriving?

The conversation continues 4

Follow the reasoning
  1. Permalink

    Your admission owner needs a result the caller can act on. Suppose a Java boundary represents saturation with a checked exception. An interactive caller might abandon the request, while a batch caller might defer it under its own bound. Those are useful choices. If both can only propagate the exception, the declaration adds an obligation without creating recovery. Checked exception analysis requires handling or declaration where applicable. It does not inventory every possible failure, since RuntimeException and Error subclasses remain unchecked. What action should the saturation contract enable for each caller?

  2. Permalink

    Those twelve running or queued jobs also need an account of the data they retain. Does each queue entry own its payload, or borrow data that a caller still owns? In Rust, borrowing permits access without transferring ownership. A short reference can therefore require a large payload to remain available. Twelve jobs bound a count, but give no byte bound unless payload sizes and retained backing data are bounded too. If a caller times out while work still uses borrowed data, the access design must preserve that data until the borrow ends. Cancellation intent does not settle its lifetime.

  3. Permalink

    The four workers bound simultaneous work, but a replacement worker does not explain what happens to an interrupted job. Suppose an Elixir worker sends an external write, then fails before reporting completion. A supervisor can restart a process. The new process still needs state to decide whether to resume, retry or report uncertainty about that write. Links also matter because failure signals can reach related processes, whose response depends on exit handling. I would give the recovery owner a record of job intent and a rule for reconstructing progress, separately from admission's count.

  4. Permalink

    A timeout can also leave reply data in the caller's queue. In an Erlang protocol, suppose the caller times out and deactivates its reply alias before a reply reaches the queue. Later messages through that alias are dropped before queueing. Change the order. The caller times out, the reply enters the queue, then the caller deactivates the alias. That reply stays queued and can still retain data. The protocol needs a rule for consuming stale replies. Deactivating the alias also does not tell the worker to stop, so releasing the worker's payload needs a separate cancellation policy.