you do not want to return negative I think but 0; otherwise linelen in lkpi_hex_dump may go negative or back to 0; this is likely also a problem with the other callback routines.

this is not correct. written can be > context->linebuflen if the output would have been longer than the buffer size. You need another check here or rather set it accordingly .. hmm also below is wrong to the end of the func.

If I am not mistaken:

if (written <= (context->linebuflen - 1))    /* terminating \0 */
     context->written = written + 1;
else
     context->written = context->linebuflen;

context->linebuf += context->written;
context->linebuflen -= context->written;

return (context->written);

But you should double-check that given vsnprintf returns the length without the trailing \0 and I am not sure if you need to count it or not.

This is not needed, is it? lkpi_hex_dump will only do 8/4/2 or 1 in every other case

should we add another bool to lkpi_hex_dump whether to add the (last) \n or not?