openlang

String.length counts bytes, and so does len

A truncated error message cut a name in half. The code kept the first 40 bytes with String.sub msg 0 40, the name near the end was Korean, and the cut landed inside a three-byte UTF-8 sequence. The terminal showed a replacement character where the last syllable should have been.

String.length in OCaml is a byte count. The String module describes a string as an immutable sequence of bytes, and String.sub takes byte positions. Go's len(s) is the same, and s[:40] slices bytes. Neither is wrong. They just don't count what the person reading the message counts.

The local fix is to back the cut up to the start of a character. In Go that's a short loop on utf8.RuneStart.

func truncate(s string, max int) string {
	if len(s) <= max {
		return s
	}
	i := max
	for i > 0 && !utf8.RuneStart(s[i]) {
		i--
	}
	return s[:i]
}

s[i] here is the first byte that would be dropped. If it starts a character, the cut is clean at i. If it's a continuation byte, the loop walks left until it finds the start of the character being split and cuts before it. On a message that ends in 김다현 with the limit inside 다, that returns the text up to 김 as valid UTF-8, where s[:40] left two stray bytes.

In OCaml, String.get_utf_8_uchar decodes at an index and Uchar.utf_decode_length says how far to step, so the same walk is a few lines longer and reads about the same. The Go version passes a few more strings too, a pure ASCII one and one where the limit already falls on a character boundary, and both come back cut at the limit itself.

That keeps the output valid UTF-8, which is all I'm claiming for it. Whether a character a person sees stays whole is another matter.

后续讨论 6

顺着彼此的依据读下去
  1. 固定链接

    Put the cut in the .mli as val truncate : max_bytes:int -> string -> string and make it the only place that slices display text. Then a later caller can't quietly go back to String.sub.

  2. 固定链接

    Coming from JavaScript, I know this bug one unit size down. .length and slice count UTF-16 code units, so name.slice(0, 40) can split a surrogate pair instead of a UTF-8 sequence, and you get a lone surrogate at the end. Backing up to the start of a code point fixes that much. It still won't keep 🇰🇷 together. That flag is two code points, four UTF-16 units and one grapheme, and a cut that only respects code points walks straight through the middle of it. For an error message that's probably acceptable. For a name I'd want the cut to follow graphemes.

  3. 固定链接

    Skip RuneStart. Ranging over a string already yields rune starts.

    if len(s) <= max {
    	return s
    }
    end := 0
    for i := range s {
    	if i > max {
    		break
    	}
    	end = i
    }
    return s[:end]

    Same output on valid UTF-8, and one import fewer.

  4. 固定链接

    A quieter character breaks it too. A name ending in e followed by U+0301, with the limit falling between the two, loses its accent. The utf8.RuneStart loop keeps e, drops the combining accent, and returns perfectly valid UTF-8. The OCaml version does the same, as it should, since both only look at code points. So the name comes out with no error and the wrong spelling, which is worse than the replacement character in one way. Nobody notices.

    For now the code point cut stays, and any cut gets an ellipsis after it, so the reader at least sees that something was removed.

同一个问题的其他角度

  1. Is an NFC key a sensible default for name lookup?

    Okay, a user pastes their name and it ends in e followed by U+0301. Later they type the same name and the keyboard gives U+00…

    回复 6最近活动 typescriptjavascript
  2. Mode line columns drift as soon as a buffer name has Hangul in it

    Padded the buffer name in my mode line to a fixed width so the bits after it line up, and it worked for a long while (well, i…

    回复 1最近活动 emacs-lisp
  3. Truncating by bytes like the old C++, and Rust panics

    Porting a chunk of old C++ to Rust that cuts a std::string down to a byte limit with a plain s.substr(0, n), and I figured it…

    回复 6最近活动 rustcpp