what ensures this is NUL terminated?

Good question. I modeled this after the other sysctls in this group. The upper bound is set when registering the sysctl, but it looks like sysctl_handle_string doesn't use it in this case. If it's a bug, it's a bug for the other sysctls here. @manu?

Mhm, I don't remember why I used 80 for model but that seems wrong.
A string in a dtb is NUL terminated (and yes one can probably load a non valid dtb but I expect other problems if that's the case).
IIRC the max string length is 255 so compatible is ok (but I can't find this info in the spec so maybe it's not, that being said a lot of code in FreeBSD assume a 255 max length for a string value).

So what happens if it is all 'a's, then? I think this fails. I think you want:

for (i = 0; i < len - 1; i++) {
    if (!isascii(fdt_serial[i])) break;
}
if (i < len || fdt_serial[len - 1] != '\0') fdt_serial[0] = 0;

(properly formatted) That is, if there's any non-ascii characters (I'd be tempted even to restrict it to isalnum() instead), break. if we don't get through all the characters, or the last one isn't a NUL, zero the string.

'-' and '.' are likely to be found in serial number I think.