outliers.log has more lines than the :outliers counter, 8 futures

I fold 20000 simulated tide readings into one atom from 8 futures. Every reading above the threshold also gets appended to outliers.log. The summary is fine: :n is always 20000, and :outliers is the same every run because the seed is fixed. But the log has more lines than the counter. Last run it was around 250 lines against a counter a little lower. Some ids appear twice. With --workers 1 they agree.

(defn- log-line [& parts]
  (spit (:log-file config)
        (str (str/join " " parts) "\n")
        :append true))

(defn fold-reading [st r]
  (let [lvl (:level r)]
    (when (outlier? r)
      (log-line "outlier" (:id r) (format "%.3f" lvl)))
    (cond-> (-> st
                (update :n inc)
                (update :sum + lvl)
                (update :sumsq + (* lvl lvl)))
      (outlier? r) (update :outliers inc))))

(defn record! [r]
  (swap! stats fold-reading r))

I thought spit :append from many threads double-writes, but then the counter would be wrong too. I also checked the partition-all chunks and they don't overlap, since :n is exactly 20000. I delete the log before each run. The line count is different again on the next run with the same seed.

How can the stats be exactly right while the log is not? I want exactly one line per outlier.

The conversation continues 1

Follow the reasoning

cgw24 Sept Permalink
I think the culprit is swap! retrying. With 8 futures hitting one atom, a compare-and-set sometimes fails and the function runs again on the new value. Only the winning result is committed, so the counter is exact. But spit fires on every attempt, including the discarded ones. That also explains why --workers 1 agrees, and why the duplicate count changes from run to run, right? The atoms reference says the function may be called multiple times, so it should be free of side effects.

Maybe something like this, with fold-reading pure:
```
(defn record! [r]
  (when (outlier? r)
    (log-line "outlier" (:id r) (format "%.3f" (:level r))))
  (swap! stats fold-reading r))
```
Each reading is then logged exactly once, though the log order won't match the swap order...

outliers.log has more lines than the :outliers counter, 8 futures

The conversation continues 1

Other sides of the same question

canceled context but worker goroutines still alive after first error (always workers + 2)

Marking the filter async did nothing for the freeze

await for over ReceivePort never finishes after isolates are done