Same overflow check, different error branch at -O0 vs -O2

65536 * 32768 is 2^31, which doesn't fit in a signed int. That's undefined behavior, so neither branch is "correct". At -O0 it happens to wrap to INT_MIN and your check fires. At -O2 the compiler may assume no overflow and drop the < 0 test. malloc isn't odd. Check before multiplying, e.g. count > INT_MAX / rec_size, or use size_t.

Ha, this is a classic. e.lund has the cause. I'll add the part that explains your "out of memory" message.

If the compiler drops the < 0 test, total still holds the wrapped value, INT_MIN. Then malloc(total) converts it to size_t. On a 64-bit box that becomes 0xFFFFFFFF80000000, which is about 16 exabytes. malloc says no, and your OOM branch fires. So malloc isn't odd. It's just being asked for something absurd. (Side note: that's also why the "just check for NULL" crowd gets a false sense of safety here.)

My question is about the declarations. What types are count and rec_size? If they're read from the file as uint32_t, the multiply happens in unsigned arithmetic and wraps legally. Then the int total assignment is implementation-defined, not UB. That would change the story a lot.

Have you tried -fsanitize=undefined at -O2? It should flag the multiply directly. And does -fwrapv make the two builds agree again?

The < 0 test runs after the overflow has already happened, so I wouldn't try to rescue it. Validate the header values before you multiply, and keep the result in size_t.

size_t total;
if (rec_size == 0 ||
    __builtin_mul_overflow((size_t)count, (size_t)rec_size, &total) ||
    total > max_pack_bytes) {
    LOG("pack too large");
    fclose(f);
    return -1;
}

With size_t, 2^31 is a perfectly valid size, so your hostile pack would now ask malloc for 2 GiB. Overflow checking alone doesn't catch that. I'd cap max_pack_bytes at the file size, or whatever your loader budget is, so a header can't claim more than the file holds.

The builtin is documented here: https://gcc.gnu.org/onlinedocs/gcc/Integer-Overflow-Builtins.html. GCC and clang have it. If you build with MSVC, use the count > SIZE_MAX / rec_size form instead.

Not a miscompile. Signed overflow is UB, so -O2 may drop the < 0 test. Look at the asm: gcc -O2 -S and search for the compare. I'd bet it's gone, and you only see movslq or cltq right before the call to malloc.

That sign extension is your "out of memory". INT_MIN becomes 0xFFFFFFFF80000000 as size_t. malloc refuses, so your OOM branch runs.

Even if it did fit, 2^31 is only 256 MB. A hostile header can ask for that and more. Use size_t and check the header values against the file size before you multiply. A check after the multiply is too late.

Also compile with -fsanitize=undefined. It will point at the multiply line directly.

Good call on the asm. One nit, though: 2^31 bytes is 2 GiB, not 256 MB. 256 MB is 2^31 bits. (Easy slip. Bits and bytes have ruined many an afternoon.) drawer6 had the right number. It matters here, because a 2 GiB malloc on a 64-bit Linux box with overcommit will often just succeed. So once the sign bug is fixed, the hostile pack may go from "out of memory" straight to "the loader happily allocates 2 GiB and starts reading". Not better!

Another thing worth checking is what size_t is on the target. On a 32-bit build there's no sign extension to 16 EiB at all. malloc(0x80000000) just gets asked for 2 GiB, and the result depends on the address space. So ferro71's "out of memory" story is really a story about 64-bit.

Also, -O2 -S will show you the missing compare, but the compiler may have folded it earlier. If you can't find the test/js pair, look for the branch to the "pack too large" string being gone entirely. Paste the function into Compiler Explorer and flip -O0/-O2 side by side. It's a fun way to waste twenty minutes. Then add -fwrapv and watch the branch come back.

int total = count * rec_size;

I have a doubt: why is total an int at all, when it only goes to malloc? And does the pack format define a maximum for count? If not, any check you write is guessing, no? Fix the type first, then bound the header fields against the file size.