Why not instead check whether /dev/ut->ut_line exists?

It's nontrivial and repeats work later done by ttymsg(). I think it's time the latter got a glow-up...

This leaks fd if fstat() fails.

Shouldn't we nvlist_get_boolean(nvlin, "shout") here?

I think you're supposed to pack this error value into nvlout? Returning a non-zero value to the caller here raises an error to the casper library rather than to the caller.

Good catch, thank you.

Do you still need this check?

No, I'd just forgotten about it.

Small nite: I guess we can drop double spaces.

Nite: I guess you can use nitems here.

I think this is won't work.
Libcasper handles the error handling:
https://github.com/freebsd/freebsd-src/blob/a2f733abcff64628b7771a47089628b7327a88bd/lib/libcasper/libcasper/service.c#L319-L324

So creating a duplicate "error" entry will brake libnv communication. So I guess you should do:

if (ttymsg(iov, iovcnt, line, tmout, shout) != 0) {
  serror = errno;
  free(iov);
  return (serror);
}

That's what I suspected. In that case I think cap_p_open() / casper_p_open() is incorrect as well.

To be clear, error contains the return value from the casper_*() function if and only if it is not zero, right?

Furthermore, cap_p_open() / casper_p_open() probably belong somewhere else, maybe in cap_fileargs() or maybe in its own Casper service, but let's leave that for later.

Double spaces are necessary to distinguish the period at the end of a sentence from the period at the end of an abbreviation within a sentence (this is also why *roff and mandoc require sentences to start on a new line).

To be clear, error contains the return value from the casper_*() function if and only if it is not zero, right?

The error is always added - when the function is called. If the function return 0 then it is 0.

This should return an error code, not -1, but I'm not sure which. ECAPMODE perhaps?

Is there a reason why we don't just make iovcnt a size_t while we're here?

I think we should rename the line parameter to something like tty. I needed to go and look at callsites to understand was supposed to be passed here. I understand it means line in the context of a serial line, but that wasn't immediately clear to me.

You removed the comment explaining why we don't return error on EBUSY and EACCES. I don't like this behavior, but if we're going to keep it, keep the explanation as well.

close might clobber errno here

Since ttymsg preserves errno now, we can use warn instead of warnx

I don't know if I'd call this a capability mode violation (which is what ECAPMODE is used for). Maybe just ENOENT since a matching filed couldn't be found.

Is there a reason you changed this from tmout to timeout? The surrounding context already uses tmout

Is there a reason why we don't just make iovcnt a size_t while we're here?

writev() takes an int.

Exacly, it's a capability mode violation because we're trying to popen() a program that's not on the preapproved list.

I find the abbreviation confusing, especially next to shout which is not an abbreviation but happens to differ only by its first two letters, and this is not a situation where brevity would be preferred over clarity.

I suppose it depends on what you classify as a capability violation.

I always classified capability violations as actions that directly try to break the kernel-level sandbox. With my logic, ECAPMODE should only originate from the kernel.

This would be a program-defined capability violation since the rule we're breaking exists only in the syslogd code (the piped command must have a valid filed in the filed list).

I'm ok with this interpretation. I just wanted to highlight the distinction.

Small nit. If we're going with timeout instead of tmout, then maybe we should stay consistent and use timeout everywhere. Or at the very least, here, since it is new code.