Diff Detail
- Repository
- rG FreeBSD src repository
- Lint
Lint Skipped - Unit
Tests Skipped - Build Status
Buildable 76332 Build 73215: arc lint + arc unit
Event Timeline
This seems to be correct at first glance, but I think it can be greatly simplified and improved in performance.. I did a lot of research on Unicode validation and transcoding and had originally meant to take a stab at this function, but then forgot about it.
I'll think about it some more and then let you know what I came up with.
For example, state machines with lookup tables can be used instead of cascades of if-statements. They are much faster in the scalar case.
| include/uchar.h | ||
|---|---|---|
| 35 | I had this symbol as C23 only for a reason. I think it was because things go really wrong in C++ land if this header defines char8_t otherwise. Please check this carefully (e.g. by building world with the patch applied) or just leave things as they were. | |
| lib/libc/locale/c8rtomb.c | ||
| 22 | It would be great if the design patterns and symbol naming could be similar to the existing files c16rtomb.c and c32rtomb.c for consistency. | |
| 26 | That 64 is a magic number. Where is it from? Is there a macro we can use instead? | |
| 94 | I don't think this comment is relevant. \0 is the string terminator, which does not end up being transcoded. The function is meant to be restartable, meaning, that you can deliver input in chunks that may not themselves all start or end at boundaries of full characters (if that wasn't the case, then the mbstate_t argument would be pointless, as you'd always be in initial state when calling this function). | |
| lib/libc/locale/mbrtoc8.c | ||
| 23 | What is the meaning of these structure members? | |
This seems to be correct at first glance, but I think it can be greatly simplified and improved in performance.. I did a lot of research on Unicode validation and transcoding and had originally meant to take a stab at this function, but then forgot about it.
I'll think about it some more and then let you know what I came up with.
That'd be great, I appreciate it.
For example, state machines with lookup tables can be used instead of cascades of if-statements. They are much faster in the scalar case.
DFA or class table is a good fit for bulk UTF-8, but these calls only handle one unit at a time and then go through mbrtoc32/c32rtomb, which dominate. The remaining compares are the ASCII/char32_t length split and the Table 3-7 second-byte checks, so I would expect little gain from a lookup table here, and it would also diverge from utf8.c. But I might be wrong, so I would be happy to discuss it further.
| include/uchar.h | ||
|---|---|---|
| 35 | What we do now is the same idea as char16_t: _types.h sets _CHAR8_T_DECLARED when the language already has the type, and uchar.h only typedefs if that flag is clear. We key it off __cpp_char8_t, not __cplusplus >= 202002L, so -fchar8_t / -fno-char8_t are correct. Prototypes are emitted in C++ only when __cpp_char8_t is set, so they always use the builtin, never unsigned char pretending to be char8_t. C++11/17 without the builtin do get typedef unsigned char char8_t if they include <uchar.h>. That cannot collide with a keyword (there isn’t one). It can collide with a user identifier named char8_t, same class of issue as char16_t before C++11. The keyword case is protected. So, in fact, by improving it slightly and guarding against __cpp_char8_t, I tried fixing a rare bug that had been lurking there for a while. I ran pretty much every test that sprang to my mind, and I think this is considerably safer than the previous check. Bu I would really appreciate it if you could take a look as well and let me know if you spot anything I’ve missed. | |
| lib/libc/locale/c8rtomb.c | ||
| 22 | Sure, I can rename _C8rtombExtra / _Mbrtoc8Extra to _Char8State. | |
| 26 | It’s half the size of the mbstate_t blob, and I think that’s an OK promise. Perhaps I could do this: #define _MBSTATE_CONV_PREFIX (sizeof(mbstate_t) / 2) I’m still not sure it’s the best we can do, though. I’d rather use sizeof(extra) + sizeof(_ConversionState) <= sizeof(mbstate_t), but that would pull in the iconv types, and our case here is MK_ICONV=no. What would you do, Robert? | |
| 94 | But u8'\0' is a char8_t value, not a C-string terminator. Also, restartability remains unchanged. If the comment is misleading, we can certainly drop it altogether (or perhaps just the second sentence). However, if I’m missing something here, I’d appreciate it if you could point it out and correct me. | |
| lib/libc/locale/mbrtoc8.c | ||
| 23 | nleft / next / buf are the leftover UTF-8 units after we already returned the first one ((size_t)-3), and incomp means mbrtoc32 last returned (size_t)-2, so a following NUL is EILSEQ instead of being swallowed by iconv. | |