openlang

`isdigit()` passed a superscript two that `int()` refused

A quantity field in a small form checks s.isdigit() and then calls int(s). Someone pasted ² and got a ValueError from the very line the check was meant to protect.

The two use different rules. isdigit() is true for any character with a digit value, which takes in superscripts and circled digits like ⑦. int() wants decimal digits, the isdecimal() set, so ² passes the first and fails the second.

for s in ["7", "²", "٣", "3"]:
    print(repr(s), s.isdigit(), s.isdecimal())
print(int("٣"), int("3"))  # 3 3

Switching the check to isdecimal() would stop the crash. It would also let Arabic-Indic ٣ and full-width 3 through as 3, and the field never showed either of those as an option. So the crash has an obvious fix, and I'm not yet sure the field should accept those characters at all. What do you all allow in a numeric input?

The conversation continues 4

Follow the reasoning
  1. Permalink

    Fun corner. int() doesn't care whether the digits come from the same script. int("1٣") is 13, one ASCII digit and one Arabic-Indic, and float("١.٥") gives 1.5. I expected mixing to be refused, the way some confusable checks refuse mixed scripts, but it's just per-character digit values added up.

    So with isdecimal() the field takes numbers no single keyboard layout would type in one go. Paste 1٣ into it and see what the form echoes back.

  2. Permalink

    That settled it for me, ASCII digits only. My first attempt was re.match(r"\d+$", s), and it had two problems I didn't see right away. \d in a str pattern matches every Unicode decimal digit, so ٣ still got in. And $ also matches just before a trailing newline, so "7\n" passed too. Here is what the field uses now.

    import re
    
    def parse_quantity(s: str) -> int | None:
        if re.fullmatch(r"[0-9]+", s):
            return int(s)
        return None

    fullmatch with an explicit class answers one question, ASCII digits from the first character to the last. Spaces, signs and underscores get refused as well, even though int() itself would take " 7 " and "1_000".

  3. Permalink

    Agree on fullmatch, the $ newline thing catches people constantly. Less sure about refusing full-width digits. Someone typing with a Japanese input method left in full-width mode produces 3 without noticing, and to them the error reads like the form rejecting the number 3. A ten-entry table handles that one case without opening the door to everything isdecimal() allows.

    FULLWIDTH = str.maketrans("0123456789", "0123456789")
    parse_quantity(s.translate(FULLWIDTH))

    Superscripts and other scripts still get refused. Whether full-width deserves the exception is a judgment call, and I don't feel strongly enough to push it past this reply.

Other sides of the same question

  1. Is an NFC key a sensible default for name lookup?

    Okay, a user pastes their name and it ends in e followed by U+0301. Later they type the same name and the keyboard gives U+00…

  2. NFC lookup keys and JCS signatures

    Signing complicates kim_dh's earlier question about NFC lookup keys, which separates the submitted name from an NFC lookup ke…

  3. Truncating by bytes like the old C++, and Rust panics

    Porting a chunk of old C++ to Rust that cuts a std::string down to a byte limit with a plain s.substr(0, n), and I figured it…

    Replies 6rustcpp