Truncating by bytes like the old C++, and Rust panics

If the C++ side is the reference, it helps to see what it wrote. substr counts bytes and nothing else.

std::string s = "Привет";
std::string t = s.substr(0, 5);
std::cout << t.size() << ' ' << std::hex
          << int(static_cast<unsigned char>(t.back())) << '\n';

This prints 5 d0. The last byte is the lead byte of "и" with its continuation byte gone. An all-Cyrillic name cut at an odd limit ends that way every time, so the broken tails are predictable, which helps when you go looking for them.

Notes. Known. Since Rust 1.91, stable has str::floor_char_boundary, which does what your walk-back loop does, the closest boundary not above n. Its docs say right away that it can still split graphemes.

Unknown. What's in the data the old C++ already wrote. If the tails there are broken, String::from_utf8 returns an error on them, while String::from_utf8_lossy substitutes U+FFFD without a word. Those are two different decisions, and the choice belongs before reading, not after something falls over.

For now it's an assumption that broken records exist at all. One pass over the old data would tell.

Check the toolchain first. floor_char_boundary is stable only from 1.91, and older compilers reject it outright.

A char boundary isn't a letter boundary either, so don't celebrate yet. "й" can arrive as one code point, or as "и" plus a combining breve, U+0306, two scalars and four bytes. Cut between them and you get valid UTF-8 and an "и" nobody typed. The floor_char_boundary docs show the same thing with an emoji built from several scalars.

The cheap fix is normalizing input names to NFC before the cut, which turns "й" back into one code point. Not everything has a precomposed form, though, and for the rest you need grapheme segmentation, which neither the Rust nor the C++ standard library gives you. Annoying, but at least now you know which bug you still own.

fretboard, I ran the old dump like you suggested, and it's messier than I hoped. Most of the broken records are what I expected, error_len() returns None and valid_up_to() sits one byte from the end, which is the trace of byte truncation and nothing else. Those I cut at valid_up_to() and don't substitute U+FFFD, because the replacement takes three bytes where there was one and can push the value over the limit again.

But a handful came back with Some(1) in the middle of the string, a byte above 0x7F that no valid sequence follows, and that's not truncation at all. It looks like some older path wrote names in a single-byte encoding, Latin-1 or Windows-1251, and those two would read the same byte as different letters, and I have no way to tell from the byte alone which one it was. If I cut those at valid_up_to() I lose the rest of the name, and if I run them through from_utf8_lossy I get a U+FFFD in the middle of somebody's surname.

The others go into a separate table untouched until I find out what wrote them, and if that old path is still running somewhere, the table will keep growing. Can a whole name tell Latin-1 from 1251 even when one byte can't? In 1251 a Cyrillic surname is almost all bytes from 0xC0 up, while a Latin-1 name is mostly ASCII with the odd accent, so the ratio might give it away. I'm just wary of letting a heuristic decide what somebody's name is.

Split the two cases in code so the second table fills itself. The catch is that from_utf8 tells you which case you're in only through error_len().

fn keep_len(bytes: &[u8]) -> Result<usize, std::str::Utf8Error> {
    match std::str::from_utf8(bytes) {
        Ok(_) => Ok(bytes.len()),
        Err(e) if e.error_len().is_none() => Ok(e.valid_up_to()),
        Err(e) => Err(e),
    }
}

A Cyrillic name cut at five bytes gives Ok(4). b"Ren\xe9e" gives an Err with error_len: Some(1). Count the Err rows before and after each import, and you'll know whether the old path is still writing them.