恶意构造的 pack，count=65536，rec_size=32768。-O0 下我得到 "pack too large"，-O2 下却变成了 "out of memory"。源码完全一样，我真的想不通。是 -O2 把它编译错了，还是 malloc 有什么奇怪的行为？
int total = count * rec_size;
if (total < 0) {
LOG("pack too large");
fclose(f);
return -1;
}
out->data = malloc(total);Hostile pack with count=65536 and rec_size=32768. At -O0 I get "pack too large". At -O2 I get "out of memory" instead. Same source, and I really don't see how. Is -O2 miscompiling it, or is malloc doing something odd?
int total = count * rec_size;
if (total < 0) {
LOG("pack too large");
fclose(f);
return -1;
}
out->data = malloc(total);65536 * 32768 等于 2^31，有符号 int 装不下。这是未定义行为，所以两个分支都谈不上“正确”。-O0 下它碰巧回绕成了 INT_MIN，你的检查就触发了。-O2 下编译器可以假定不会溢出，直接把 < 0 的判断删掉。malloc 并没有什么奇怪的地方。要在乘之前检查，比如 count > INT_MAX / rec_size，或者用 size_t。
65536 * 32768 is 2^31, which doesn't fit in a signed int. That's undefined behavior, so neither branch is "correct". At -O0 it happens to wrap to INT_MIN and your check fires. At -O2 the compiler may assume no overflow and drop the < 0 test. malloc isn't odd. Check before multiplying, e.g. count > INT_MAX / rec_size, or use size_t.
哈，这是个经典问题。e.lund 已经说出了原因，我补充一下你那条 "out of memory" 是怎么来的。
如果编译器把 < 0 的判断删掉了，total 里仍然是回绕后的值 INT_MIN。然后 malloc(total) 会把它转成 size_t。在 64 位机器上这就变成 0xFFFFFFFF80000000，大约 16 EB。malloc 拒绝了，你的 OOM 分支就触发了。所以 malloc 没什么奇怪的，只是被要求分配一个离谱的大小。（顺带一提，这也是为什么“只要检查 NULL 就行”那派人在这里会有一种虚假的安全感。）
我的问题在声明上。count 和 rec_size 是什么类型？如果它们是从文件里按 uint32_t 读出来的，那乘法是按无符号算术做的，合法回绕。然后赋给 int total 是实现定义的行为，不是 UB。这样情况就大不一样了。
你试过在 -O2 下加 -fsanitize=undefined 吗？它应该能直接标出这个乘法。另外加上 -fwrapv 之后，两个构建的行为会重新一致吗？
Ha, this is a classic. e.lund has the cause. I'll add the part that explains your "out of memory" message.
If the compiler drops the < 0 test, total still holds the wrapped value, INT_MIN. Then malloc(total) converts it to size_t. On a 64-bit box that becomes 0xFFFFFFFF80000000, which is about 16 exabytes. malloc says no, and your OOM branch fires. So malloc isn't odd. It's just being asked for something absurd. (Side note: that's also why the "just check for NULL" crowd gets a false sense of safety here.)
My question is about the declarations. What types are count and rec_size? If they're read from the file as uint32_t, the multiply happens in unsigned arithmetic and wraps legally. Then the int total assignment is implementation-defined, not UB. That would change the story a lot.
Have you tried -fsanitize=undefined at -O2? It should flag the multiply directly. And does -fwrapv make the two builds agree again?
< 0 的判断是在溢出已经发生之后才跑的，所以我不会想着去补救它。先在乘法之前校验 header 里的值，并且把结果放在 size_t 里。
size_t total;
if (rec_size == 0 ||
__builtin_mul_overflow((size_t)count, (size_t)rec_size, &total) ||
total > max_pack_bytes) {
LOG("pack too large");
fclose(f);
return -1;
}用 size_t 的话，2^31 完全是合法的大小，所以你那个恶意的 pack 现在会让 malloc 去申请 2 GiB。光做溢出检查是拦不住这个的。我会把 max_pack_bytes 限制在文件大小，或者你的加载器预算上，这样 header 就不能声称比文件实际内容更大。
GCC 和 clang 都有这个内置函数。如果你用 MSVC 构建，就改用 count > SIZE_MAX / rec_size 这种写法。文档在这里 https://gcc.gnu.org/onlinedocs/gcc/Integer-Overflow-Builtins.html.
The < 0 test runs after the overflow has already happened, so I wouldn't try to rescue it. Validate the header values before you multiply, and keep the result in size_t.
size_t total;
if (rec_size == 0 ||
__builtin_mul_overflow((size_t)count, (size_t)rec_size, &total) ||
total > max_pack_bytes) {
LOG("pack too large");
fclose(f);
return -1;
}With size_t, 2^31 is a perfectly valid size, so your hostile pack would now ask malloc for 2 GiB. Overflow checking alone doesn't catch that. I'd cap max_pack_bytes at the file size, or whatever your loader budget is, so a header can't claim more than the file holds.
The builtin is documented here: https://gcc.gnu.org/onlinedocs/gcc/Integer-Overflow-Builtins.html. GCC and clang have it. If you build with MSVC, use the count > SIZE_MAX / rec_size form instead.
这不是编译器的 miscompile。有符号溢出是 UB，所以 -O2 可能把 < 0 的判断直接删掉。看一下汇编，gcc -O2 -S 然后搜那个比较指令。我敢说它已经没了，你只会在调用 malloc 之前看到 movslq 或 cltq。
那个符号扩展就是你说的“内存不足”。INT_MIN 转成 size_t 就是 0xFFFFFFFF80000000。malloc 拒绝分配，所以你的 OOM 分支就跑了。
就算能放得下，2^31 也才 256 MB。恶意的头部可以要求这么多甚至更多。用 size_t，并且在做乘法之前先拿头部的值和文件大小比对。乘完再检查就晚了。
另外编译时加上 -fsanitize=undefined，它会直接指到那行乘法。
Not a miscompile. Signed overflow is UB, so -O2 may drop the < 0 test. Look at the asm: gcc -O2 -S and search for the compare. I'd bet it's gone, and you only see movslq or cltq right before the call to malloc.
That sign extension is your "out of memory". INT_MIN becomes 0xFFFFFFFF80000000 as size_t. malloc refuses, so your OOM branch runs.
Even if it did fit, 2^31 is only 256 MB. A hostile header can ask for that and more. Use size_t and check the header values against the file size before you multiply. A check after the multiply is too late.
Also compile with -fsanitize=undefined. It will point at the multiply line directly.
汇编这点说得好。不过有个小问题，2^31 字节是 2 GiB，不是 256 MB。256 MB 是 2^31 比特。（很容易搞混。比特和字节不知道毁了多少个下午。）drawer6 给的数字才是对的。这一点在这里很重要，因为在开了 overcommit 的 64 位 Linux 上，2 GiB 的 malloc 往往会直接成功。所以符号问题修好之后，这个恶意 pack 可能会从 "out of memory" 直接变成“加载器高高兴兴分配 2 GiB 然后开始读取”。这可没有变好！
还有一点值得检查，就是目标平台上 size_t 是多大。32 位构建根本不存在扩展到 16 EiB 的符号扩展。malloc(0x80000000) 要的就是 2 GiB，结果取决于地址空间。所以 ferro71 那个 "out of memory" 的说法其实只适用于 64 位。
另外，-O2 -S 会让你看到缺失的比较，但编译器可能更早就把它折叠掉了。如果你找不到 test/js 这一对指令，就去看通向 "pack too large" 字符串的那个分支是不是整个没了。把这个函数贴到 Compiler Explorer 里，把 -O0/-O2 并排对照着看。拿来消磨二十分钟很有意思。然后加上 -fwrapv，看着那个分支回来。
Good call on the asm. One nit, though: 2^31 bytes is 2 GiB, not 256 MB. 256 MB is 2^31 bits. (Easy slip. Bits and bytes have ruined many an afternoon.) drawer6 had the right number. It matters here, because a 2 GiB malloc on a 64-bit Linux box with overcommit will often just succeed. So once the sign bug is fixed, the hostile pack may go from "out of memory" straight to "the loader happily allocates 2 GiB and starts reading". Not better!
Another thing worth checking is what size_t is on the target. On a 32-bit build there's no sign extension to 16 EiB at all. malloc(0x80000000) just gets asked for 2 GiB, and the result depends on the address space. So ferro71's "out of memory" story is really a story about 64-bit.
Also, -O2 -S will show you the missing compare, but the compiler may have folded it earlier. If you can't find the test/js pair, look for the branch to the "pack too large" string being gone entirely. Paste the function into Compiler Explorer and flip -O0/-O2 side by side. It's a fun way to waste twenty minutes. Then add -fwrapv and watch the branch come back.
int total = count * rec_size;
我有个疑问，total 为什么要是 int，它不是只传给 malloc 吗？还有，pack 格式有没有规定 count 的最大值？如果没有，你写的任何检查都是在猜，不是吗？先把类型改对，再根据文件大小去限制头部字段。
int total = count * rec_size;
I have a doubt: why is total an int at all, when it only goes to malloc? And does the pack format define a maximum for count? If not, any check you write is guessing, no? Fix the type first, then bound the header fields against the file size.
会不会是竞态检测器在这里判断错了，还是我对 ack 提供的保证理解有误？这是 pushgate，一个小服务，给我的移动应用校验推送通知用的设备令牌上的 HMAC 签名。调用方在无缓冲的 channel 上发送一个 Request，worker 填好…
我写了个小的 Node 检查器（纯 JS，不需要构建步骤），用来重放保存下来的 webhook 投递，并在下游信任它们之前校验 HMAC-SHA256。三个里有两个能过。来自 billing-partner 的 evt_2044 失败了，但同一个合作…
我在写一个叫 vault 的小型 C 凭据存储库，在 WSL 里构建。vault_release() 的作用是先擦除密钥再释放，代码如下 测试文件 tests/wipe_check.c 会先保存密钥指针，调用 vault_release()，然后从第…
