Page MenuHomeFreeBSD

create widths.txt from utf8proc data
Needs ReviewPublic

Authored by yuripv on Tue, Nov 17, 7:01 PM.

Details

Reviewers
bapt
hrs
Summary

Update our pretty outdated widths.txt with data from utf8proc, which looks to be widely used exactly for the reasons of wcwidth() provided by OS to be unreliable. This implementation consists of simple piece of C code (required as there's only C library in utf8proc), going through 0x0 - 0x110000 character range, and perl script (copied and modified from utf8-rollup.pl) mapping the data into widths.txt format.

Test Plan

Rebuild locale data with new widths.txt, enjoy the text full of emojis in terminal/vim/....

Diff Detail

Repository
rS FreeBSD src repository
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

  • rewrite (actually copy and modify utf8-rollup.pl) mkwidths in perl, making it 300x faster compared to sh version
  • add to README
  • use pkgconf to get utf8proc cflags/libs