openlang

foldl' on a (total, count) pair and still over a gigabyte resident?

I've got a small tool for work (mobile app) that reads an analytics export, a CSV of session events, about 6 million lines and roughly 300MB, and prints the mean session length. It gives the right answer. But on the full file the fans on my old Intel MacBook spin up, and +RTS -s reports maximum residency of well over a gigabyte.

I wrote it with foldl first, then swapped in foldl', and peak memory barely moved. -O2 is already in the cabal file and made no real difference either.

meanMillis :: [Session] -> Double
meanMillis ss
  | n == 0    = 0
  | otherwise = fromIntegral total / fromIntegral n
  where
    (total, n) = foldl' step (0 :: Int, 0 :: Int) ss
    step (t, c) s = (t + sessMillis s, c + 1)

and in main:

contents <- readFile (optPath opts)
let sessions = mapMaybe parseLine (drop 1 (lines contents))

Session is a plain record with a String and an Int, and the filter by screen is just filter.

My first suspect is readFile and lines, so I thought about an hGetLine loop, but I haven't tried it. I don't see why the strict fold wouldn't be enough if the file is consumed lazily and each line is dropped after parsing. A profiling build is very slow on this machine, so I'm mostly guessing from the -s numbers.

Where is the memory actually going?

The conversation continues 1

Follow the reasoning
  1. Permalink

    Same trap got me on a different tool. foldl' only forces the tuple itself, not what's inside it, so t + ... and c + 1 pile up as thunks. Bang patterns in step fixed it for me: step (!t, !c) s. (You need the BangPatterns pragma, because of course.) Worked immediately.

Other sides of the same question

  1. Empty reductions return the identity

    Reduce over an empty vector. APL answers. Each primitive carries its identity. Zero for plus. One for times and for and. ⌈/⍬…

    Replies 3Latest activity apl
  2. Most of my macros turned into const and static

    Nim makes macros easy to write. That's the problem. Before reaching for macro, check the cheaper tools. A const is evaluated…

    Replies 3Latest activity nim
  3. it, _1 and the block names I still write

    Ruby 3.4 adds it for the block parameter, so users.map { it.name } reads almost like a sentence. I like it better than _1, wh…

    Replies 0Latest activity ruby