asyncio.gather on 3.9: one task fails, the other keeps running and writes its file

The docs don't say that. With the default return_exceptions=False, gather passes the first exception to whoever awaits it. The other awaitables are not cancelled and keep running. They only get cancelled if the gather itself is cancelled. Since you made the tasks yourself, weights is pretty much a free-running task that nobody stops.

On 3.9 you have to cancel it yourself:

try:
    measurements, weights = await asyncio.gather(m, w)
except Exception:
    for t in (m, w):
        t.cancel()
    await asyncio.gather(m, w, return_exceptions=True)
    raise

Or use asyncio.wait(..., return_when=asyncio.FIRST_COMPLETED), then cancel whatever is left in pending. Either way, await the cancelled tasks so they actually finish before you check.

Then drop the half second sleep.

Thanks, that makes sense, and I clearly misread the docs, right? I'll try your first version. Once something like this bit me on a client site: a cancelled task still wrote its file because the cancel landed after the write had started, so I'll check that the file is gone too.

Is the write sync?

gather doesn't cancel the other one, it just hands you the first exception and weights keeps going. that's what you're seeing. the docs are kind of vague about it, annoying.

so cancel m and w yourself in an except block and await them again. in go errgroup does this for you, python makes you do it by hand.

The short answer is that gather with the default return_exceptions=False only propagates the first exception. It does not cancel the siblings. Cancellation happens only if the gather future itself is cancelled, and here nobody does that. So weights runs to completion, as you observed.

What exactly does "in flight" report? If it comes from asyncio.all_tasks(), then it is measuring the right thing. If it comes from your own bookkeeping, I would verify that first.

TaskGroup arrived in 3.12, I believe, so on 3.9 the manual cancel is necessary. After cancelling, await the tasks, then check the file...

Yes, it's sync, right? In fetch_weights it's a plain open() plus json.dump after the last await, so there's no await inside the write itself. That matters for the cancel. It can only land at an await point, so if weights is still waiting on the network, the write never starts, and if it's already past that, the write finishes whole. Either way I'll delete the file before the run and check afterwards.

weights is pretty much a free-running task that nobody stops

Free-range task — more dedication than anyone I've ever worked with. Still wrote the file, too.

Sync write is fine, but a crash mid-json.dump leaves a half file. Write to a temp file, then os.replace.

Good point, a crash mid-json.dump would leave a half file, and then the next run would read it as a real cache, right? So I'll do it in steps:

1. Write to weights_cache.json.tmp in the same directory, since os.replace is only atomic on the same filesystem.
2. Flush and os.fsync the file before closing it, if the cache really matters.
3. os.replace("weights_cache.json.tmp", "weights_cache.json").

That also helps with the cancel check, because the final name only exists if the write finished whole. So after the cancel I can just check that weights_cache.json isn't there, and a leftover .tmp would tell me the write started and never got renamed.

Since there's still no await between open and replace, the cancel can't land in the middle anyway, so this is mostly for crashes and killed processes.

The half second sleep in main.py is only there so the last check comes after the write.

A sleep that shows the bug probably hides a second one. Does fetch_weights go through run_in_executor or to_thread anywhere, say for the HTTP call or the file write? Cancelling the task won't stop a thread that's already running, so the file could still land after your cancel. Worth knowing before you trust the fix.

Does fetch_weights go through run_in_executor or to_thread anywhere, say for the HTTP call or the file write?

The write doesn't, it's a plain open() and json.dump right in the coroutine, as I said above. For the HTTP call I'm not 100% sure what the client library does underneath, so I'll grep for both names first. Then I'll keep the sleep in for one run after the cancel fix, right? If the file still shows up, it's a thread.