openlang

String.length counts bytes, and so does len

A truncated error message cut a name in half. The code kept the first 40 bytes with String.sub msg 0 40, the name near the end was Korean, and the cut landed inside a three-byte UTF-8 sequence. The terminal showed a replacement character where the last syllable should have been.

String.length in OCaml is a byte count. The String module describes a string as an immutable sequence of bytes, and String.sub takes byte positions. Go's len(s) is the same, and s[:40] slices bytes. Neither is wrong. They just don't count what the person reading the message counts.

The local fix is to back the cut up to the start of a character. In Go that's a short loop on utf8.RuneStart.

func truncate(s string, max int) string {
	if len(s) <= max {
		return s
	}
	i := max
	for i > 0 && !utf8.RuneStart(s[i]) {
		i--
	}
	return s[:i]
}

s[i] here is the first byte that would be dropped. If it starts a character, the cut is clean at i. If it's a continuation byte, the loop walks left until it finds the start of the character being split and cuts before it. On a message that ends in 김다현 with the limit inside 다, that returns the text up to 김 as valid UTF-8, where s[:40] left two stray bytes.

In OCaml, String.get_utf_8_uchar decodes at an index and Uchar.utf_decode_length says how far to step, so the same walk is a few lines longer and reads about the same. The Go version passes a few more strings too, a pure ASCII one and one where the limit already falls on a character boundary, and both come back cut at the limit itself.

That keeps the output valid UTF-8, which is all I'm claiming for it. Whether a character a person sees stays whole is another matter.

The conversation continues 6

Follow the reasoning
  1. Permalink

    Put the cut in the .mli as val truncate : max_bytes:int -> string -> string and make it the only place that slices display text. Then a later caller can't quietly go back to String.sub.

  2. Permalink

    Coming from JavaScript, I know this bug one unit size down. .length and slice count UTF-16 code units, so name.slice(0, 40) can split a surrogate pair instead of a UTF-8 sequence, and you get a lone surrogate at the end. Backing up to the start of a code point fixes that much. It still won't keep 🇰🇷 together. That flag is two code points, four UTF-16 units and one grapheme, and a cut that only respects code points walks straight through the middle of it. For an error message that's probably acceptable. For a name I'd want the cut to follow graphemes.

  3. Permalink

    Skip RuneStart. Ranging over a string already yields rune starts.

    if len(s) <= max {
    	return s
    }
    end := 0
    for i := range s {
    	if i > max {
    		break
    	}
    	end = i
    }
    return s[:end]

    Same output on valid UTF-8, and one import fewer.

  4. Permalink

    A quieter character breaks it too. A name ending in e followed by U+0301, with the limit falling between the two, loses its accent. The utf8.RuneStart loop keeps e, drops the combining accent, and returns perfectly valid UTF-8. The OCaml version does the same, as it should, since both only look at code points. So the name comes out with no error and the wrong spelling, which is worse than the replacement character in one way. Nobody notices.

    For now the code point cut stays, and any cut gets an ellipsis after it, so the reader at least sees that something was removed.

  5. Permalink

    The ellipsis treats the symptom. Normalize to NFC before cutting and e plus U+0301 becomes the single code point é, so a code point cut can't separate them. Fine for this name. Not for every grapheme, though, since plenty of combining sequences have no precomposed form. Go's standard library has no normalizer either. You'd pull in golang.org/x/text/unicode/norm.

Other sides of the same question

  1. `isdigit()` passed a superscript two that `int()` refused

    A quantity field in a small form checks s.isdigit() and then calls int(s). Someone pasted ² and got a ValueError from the ver…

    Replies 4python
  2. Is an NFC key a sensible default for name lookup?

    Okay, a user pastes their name and it ends in e followed by U+0301. Later they type the same name and the keyboard gives U+00…

  3. NFC lookup keys and JCS signatures

    Signing complicates kim_dh's earlier question about NFC lookup keys, which separates the submitted name from an NFC lookup ke…