Page MenuHomeFreeBSD

pipe log destination process limits for syslogd
Needs ReviewPublic

Authored by on Jan 13 2020, 7:05 PM.



syslogd can send log messages to a pipe ("| command" entries) and supports re-spawning it when the process dies or the pipe blocks. But it does so without limits.

If the process dies, it'll try to respawn it on every log message it wants to send to it. If it dies with a non-zero exit status, and syslogd'd error message about it is to be sent to it, syslogd will start trying to respawn it in a loop (a "*.notice | false" entry would trigger that), which floods your log files. That makes it a bit dangerous to use any program which could stop working long-term for whatever reason (database access gone, config file missing, etc).

Also, if there's a write error on the pipe, syslogd moves the process to a dead queue and will try to respawn it. Processes on the dead queue are garbage collected every timer interval (30s by default), first being sent a SIGTERM and then a SIGKILL, but could still have time to pile up. syslogd sets its write end of the pipe to non-blocking, so if the destination program does not drain its input fast enough, it'll be moved to the dead queue and a new one spawned.

This patch adds a simple rate limit (reset every timer interval) and a limit on the total amount of dead processes. The limit is applied separately for each configuration entry (so that a single misbehaving log program shouldn't affect the others). The rate limit is crude (it allows a bunch of processes to be spawned really fast and then suspends it for a while), but it's simple and better than nothing I believe.

Diff Detail

rS FreeBSD src repository
Lint Skipped
Unit Tests Skipped

Event Timeline

kevans added a reviewer: cem.Jan 14 2020, 5:10 PM