Change Details

This patch fixes UTF8 sequence validation logic in `teken_utf8_bytes_to_codepoint` and fixes fallback behaviour in `ttydisc_rubchar` when an invalid UTF8 sequence is encountered. The code previously used `bitcount` to extract sequence length information from the leading byte. However, this assumption breaks for certain code points that have additional bits set in the first half of the leading byte (e.g. Cyrillic characters). This lead to incorrect behaviour when deleting those characters using backspaces. The code now checks the number of consecutive set bits in the leading byte starting from the MSB, as per [[ https://datatracker.ietf.org/doc/html/rfc3629#section-3 | RFC 2629 ]].

This patch fixes UTF8 sequence validation logic in `teken_utf8_bytes_to_codepoint` and fixes fallback behaviour in `ttydisc_rubchar` when an invalid UTF8 sequence is encountered. The code previously used `bitcount` to extract sequence length information from the leading byte. However, this assumption breaks for certain code points that have additional bits set in the first half of the leading byte (e.g. Cyrillic characters). This lead to incorrect behaviour when deleting those characters using backspaces. The code now checks the number of consecutive set bits in the leading byte starting from the MSB, as per [[ https://datatracker.ietf.org/doc/html/rfc3629#section-3 | RFC 3629 ]].

This patch fixes UTF8 sequence validation logic in `teken_utf8_bytes_to_codepoint` and fixes fallback behaviour in `ttydisc_rubchar` when an invalid UTF8 sequence is encountered. The code previously used `bitcount` to extract sequence length information from the leading byte. However, this assumption breaks for certain code points that have additional bits set in the first half of the leading byte (e.g. Cyrillic characters). This lead to incorrect behaviour when deleting those characters using backspaces. The code now checks the number of consecutive set bits in the leading byte starting from the MSB, as per [[ https://datatracker.ietf.org/doc/html/rfc3629#section-3 | RFC 26293629 ]].