localedata: add some exceptions to utf8proc widths

[kevans]

Summary

commit 88082b41b38f62f74e1a7116546bda6912e2efad (HEAD -> kbsd/localedata)
Author: Kyle Evans <[e-mail]>
Date: Wed Nov 6 22:55:05 2024 -0600

localedata: update widths.txt after recent Hangul exceptions

Sponsored by:   Klara, Inc.

commit 18264bd2ce00cfcd0a86cddab5da5502395661d3
Author: Kyle Evans <[e-mail]>
Date: Wed Nov 6 22:51:25 2024 -0600

localedata: add some exceptions to utf8proc widths

Ignorable characters will technically have a width of 1, but we should
treat them as zero-width.  This corrects some hangul filler characters'
width, which is otherwise equivalent to the block of characters around
them.

Hangul Jamo medial vowels and final consonants are reportedly combining
characters that won't take up any columns on their own and should be
reported as zero-width, so add an exception for these as well to reflect
how they work in practice.

Sponsored by:   Klara, Inc.

[jkim]

Although Hangul Jamo charaters occupy space, it seems many implementations use zero-width nowadays, including GNU libc.

https://sourceware.org/bugzilla/show_bug.cgi?id=19852
https://github.com/ridiculousfish/widecharwidth/issues/16

I agree that we should follow other implementations.

However, all fillers must occupy proper spaces, i.e., <HANGUL_CHOSEONG_FILLER> => 2, <HANGUL_JUNGSEONG_FILLER> => 0, and <HALFWIDTH_HANGUL_FILLER> => 1.

[kevans]

In D47472#1082711, @jkim wrote:

Although Hangul Jamo charaters occupy space, it seems many implementations use zero-width nowadays, including GNU libc.

https://sourceware.org/bugzilla/show_bug.cgi?id=19852
https://github.com/ridiculousfish/widecharwidth/issues/16

I agree that we should follow other implementations.

However, all fillers must occupy proper spaces, i.e., <HANGUL_CHOSEONG_FILLER> => 2, <HANGUL_JUNGSEONG_FILLER> => 0, and <HALFWIDTH_HANGUL_FILLER> => 1.

Ah, ok, thanks for confirming that- so if I just drop the ignorable check, the fillers return to their previous value with exception to <HANGUL_JUNGSEONG_FILLER> since it falls in the Hangul Kamo range that we're forcing to zero-width. That looks right based on what you've written, since omitted values default to 1.

[jkim]

In D47472#1082752, @kevans wrote:

In D47472#1082711, @jkim wrote:

Although Hangul Jamo charaters occupy space, it seems many implementations use zero-width nowadays, including GNU libc.

https://sourceware.org/bugzilla/show_bug.cgi?id=19852
https://github.com/ridiculousfish/widecharwidth/issues/16

I agree that we should follow other implementations.

However, all fillers must occupy proper spaces, i.e., <HANGUL_CHOSEONG_FILLER> => 2, <HANGUL_JUNGSEONG_FILLER> => 0, and <HALFWIDTH_HANGUL_FILLER> => 1.

Ah, ok, thanks for confirming that- so if I just drop the ignorable check, the fillers return to their previous value with exception to <HANGUL_JUNGSEONG_FILLER> since it falls in the Hangul Kamo range that we're forcing to zero-width. That looks right based on what you've written, since omitted values default to 1.

Yes, that is exactly what I meant.

[kevans]

Drop the ignorable character bit, which renders the soft-hyphen exception redundant.

[jkim]

LGTM