Launching the consumer on a stopped event

Stopped is a waiting state, and the supervisor should show it as one. fg or bg puts the writer back to running, and the consumer still isn't ready. Two milestones before it is. The writer exits successfully. Then the file gets checked to see whether the consumer can actually use it.

The same goes if the supervisor is written in Python. Each task should await the real completion of its external step. TaskGroup waits for its member tasks when the block exits, so a member that returns as soon as it sees a stop event marks the wrong milestone, and the group moves on without complaint. A task that finished properly still gets its output checked before anything depends on it. And let CancelledError through while waiting, so cancelling the supervisor still works.

I mostly script in PowerShell, and funny thing, it uses the same word for almost the opposite state. A background job shows Stopped after Stop-Job, meaning it was ended and isn't coming back, and the pause-and-resume state is Suspended, which only some job types support. So a supervisor that maps states by name across shells gets it wrong in both directions. For the consumer I'd gate on Completed plus a check of the output file and treat everything else, Stopped included, as not ready. Small aside, the job object hands you its state as a property. No scraping jobs output, which is half of why I like objects in a pipeline.

Ran it. Writer goes to out.tmp, the real name only after wait says 0.

writer() { for i in 1 2 3; do echo "row $i"; sleep 1; done; }
writer > out.tmp &
pid=$!
sleep 1.5
kill -STOP "$pid"
( sleep 2; kill -CONT "$pid" ) &
if wait "$pid"; then
  mv out.tmp out && echo "consumer may start"
fi

Script, job control off. wait sat through the stop and returned 0 once the writer finished, all three rows in out. Good.

The other half I got wrong, or at least couldn't show. Put set -m on top and the macOS /bin/bash 3.2 still kept waiting through the stop and returned 0 after the CONT. The manual for current Bash says that with job control on wait returns when the job changes status, and adds wait -f to force waiting for termination. 3.2 answers -f with invalid option. So the early return is a newer-Bash thing. Keep job control off in the supervisor and the question goes away.

Right about where the stopped event comes from, and the Python side shows it neatly. A supervisor built on subprocess won't make this mistake by accident. It has to ask for it.

import os, signal, subprocess, time

p = subprocess.Popen(["sleep", "3"])
p.send_signal(signal.SIGSTOP)
time.sleep(0.2)
print(p.poll())                      # None
_, st = os.waitpid(p.pid, os.WUNTRACED)
print(os.WIFSTOPPED(st))             # True
p.send_signal(signal.SIGCONT)
print(p.wait())                      # 0

poll() says None, still running, while the child is stopped, since it never asks the kernel about stops. You only hear about one if you call waitpid with WUNTRACED yourself, and even then WIFSTOPPED is a separate test from WIFEXITED. So somebody wrote the stopped branch on purpose and then routed it into the finished branch. That's a design decision hiding in a status mapping, and it's the part I'd question first. Some supervisors do want to know about stops, to show the state, and that's fine as long as the stopped branch leads anywhere but done. The wait() at the end blocks right through the stop, which is what you want from a supervisor that waits for completion.

Where I'm less sure is the dashboard. Showing Stopped as its own state is right, I think, but a stopped writer that nobody resumes is effectively a hung step, and I don't know how long the supervisor should wait before calling it that. A timer feels arbitrary. No timer means a consumer that waits forever.

So the safe advice for a beginner is just no set -m in the script? Easier to say than explaining -f.

Yes. Job control is for people at a terminal. In a script it's off by default anyway, so just don't turn it on.

Does asyncio's proc.wait() behave the same, or can a stop wake it up early?

It stays asleep, a stop doesn't wake it.

import asyncio, signal

async def main():
    p = await asyncio.create_subprocess_exec("sleep", "2")
    p.send_signal(signal.SIGSTOP)
    try:
        await asyncio.wait_for(p.wait(), 1)
    except TimeoutError:
        print("still waiting")
    p.send_signal(signal.SIGCONT)
    print(await p.wait())

asyncio.run(main())

Prints still waiting, then 0 once it's resumed. You need 3.11 or later to catch a bare TimeoutError there, older versions raise asyncio.TimeoutError.

On the timer, I'd keep it out of the readiness decision. The consumer waits for two milestones, the writer's exit status and the output check, and a stopped writer has reached neither, so it simply stays not ready. A timeout belongs to a different question, when to bring in a person. Raise an alert after a while, show the writer as Stopped along with how long it has been stopped, and let someone resume or kill it. If they kill it, the exit status is nonzero, the first milestone fails, and the consumer never starts.

I'll suggest that shape. Stopped with a counter, an alert after a while, and the consumer gated on exit and the file check.

One habit to add to the tmp-and-rename. Keep out.tmp in the same directory as out. A rename within one filesystem is atomic, but mv across filesystems copies and then deletes, and a consumer can catch half a file.

Same directory is half of it. The other half is the name. If the consumer picks up its input with a glob like *.csv, a temp file called out.csv.tmp stays out of it but out.tmp.csv doesn't, and sooner or later somebody renames things. A leading dot is the boring fix, since Bash globs skip dotfiles unless dotglob is set. I'd call it .out.csv.tmp and stop thinking about it.

If the dot name comes from mktemp, fix the mode before the rename. It creates the file readable by its owner only, and mv keeps that.

tmp=$(mktemp ./.out.csv.XXXXXX)
echo "row 1" > "$tmp"
ls -l "$tmp" | cut -c1-10    # -rw-------
chmod 644 "$tmp"
mv "$tmp" out.csv
ls -l out.csv | cut -c1-10   # -rw-r--r--

Skip the chmod and a consumer running as another user finds a complete file it can't open, which is a new and confusing way to fail.

I'd keep the consumer from globbing at all. Whatever records the two milestones already knows the path that passed, so hand the consumer that path and the temp name only has to be unique.

Your timer worry made me wonder where "how long has it been stopped" would even come from. My guess was that waitpid with WUNTRACED keeps reporting the stop for as long as it lasts, so a supervisor could just poll it. Not on macOS, at least.

import os, signal, subprocess, time

p = subprocess.Popen(["sleep", "3"])
p.send_signal(signal.SIGSTOP)
time.sleep(0.2)
pid, st = os.waitpid(p.pid, os.WUNTRACED | os.WNOHANG)
print(pid == p.pid, os.WIFSTOPPED(st))               # True True
print(os.waitpid(p.pid, os.WUNTRACED | os.WNOHANG))  # (0, 0)
p.send_signal(signal.SIGCONT)
time.sleep(0.2)
pid, st = os.waitpid(p.pid, os.WCONTINUED | os.WNOHANG)
print(os.WIFCONTINUED(st))                           # True
print(p.wait())                                      # 0

The stop is reported once. The second call comes back (0, 0) although the child is still stopped, and the resume is its own event that only shows up with WCONTINUED. So a stopped-for counter has to start at the moment the supervisor collects the stop, and only the continue event can end it. Polling waitpid afterwards just tells you nothing new happened.

That changes where the stopped state lives. If the stop arrives once, the supervisor's own record is the only place it exists afterwards, so the record has to keep the time it was collected, and a continue event has to clear it. It also means only a supervisor that owns the waitpid loop can show Stopped honestly. One built on poll() or asyncio's wait(), as in caspar's examples, never hears about the stop at all and would have to read ps instead, and a ps sample can miss a short stop entirely.

For the consumer gate none of this matters. It still waits for exit and the file check. It only matters for the dashboard snip wants to suggest.

Side thing from reading ps output for this. Linux procps lists T for a job control stop and t for a stop under tracing, and macOS only ever gave me T.