Page MenuHomeFreeBSD

Property-based filters for syslogd
ClosedPublic

Authored by lytboris_gmail.com on Feb 2 2020, 9:44 AM.

Details

Summary

Rationale

Message filtering options in syslogd are quite limited:

  • facility/level
  • host name
  • process name While facility and level are pretty straightforward, filtering by host or process names could be quite CPU intensive and/or even unusable: names are evaluated as full string match strictly.

Couple of years ago we've hit a host/process name string length shortage and chose to fix it with a kind of a dirty hack just growing buffer bigger (see rS275729).

Time comes by and now we're limited even by LINE_MAX-sized full string match filter: thanks to rS334719 we're now able to run tons of OpenVPN processes on single socket yet still need to skip logging into all.log from them.

At first I was considering to add shell-like pattern matching globs and/or regular expressions but after careful evaluation I came to the idea that these new matching schemes are not going to fit into constraints of present syntax.

rsyslog-like property-based syntax

Luckily enough, a syslogd alternative, rsyslog, has a good way to express things needed in a configuration file already, Property-based filters (see https://www.rsyslog.com/doc/v8-stable/configuration/filters.html#property-based-filters).

This proposal uses similar syntax, but implements for this moment just a tiny subset of properties that can be judged for filtering purposes.

Property-based filters

(This is a copy&paste from man page)

program, hostname specifications performs exact match filtering against
explicit field only.  Property-based filters feature substring and
regular expressions (see re_format(7)) matching against various message
attributes.  Filter specification starts with ‘#:’ or ‘:’ followed by
three comma-separated fields property, operator, "value".  Value must be
double-quoted. A double quote and backslash must be escaped by a
blackslash.

Following properties are supported as test value:

•   ‘msg’ - body of the message received.
•   ‘programname’ - program name sent the message
•   ‘hostname’ - hostname of message's originator
•   ‘source’ - an alias for hostname

Operator specifies a comparison function between propertie's
 value against filter's value.  Possible operators:

•   ‘contains’ - true if filter value is found as a substring of property
•   ‘isequal’ - true if filter value is equal to property
•   ‘startswith’ - true if property starts with filter value
•   ‘regex’ - true if property matches basic regular expression defined
    in filter value
•   ‘ereregex’ - true if property matches extended regular expression
    defined in filter value

Operator may be prefixed by

•   ‘!’ - to invert compare logic
•   ‘icase_’ - to make comparison function case insensitive

Some usage examples

# Log ipfw messages without syncing after every message.
!ipfw
*.*                                                     -/var/log/ipfw

# Log ipfw messages with "Deny" in the message body.
:msg, contains, ".*Deny.*"
*.*                                                     /var/log/ipfw.deny

# Reset program name filtering
!*

# Log messages from bird or bird6 into one file
:processname, regex, "^bird6?$"
*.*                                                     /var/log/bird-all.log

# Log messages from servers in racks 10-19 in multiple locations, case insensitive
:hostname, icase_ereregex, "^server-(dcA|podB|cdn)-rack1[0-9]{2}\..*"
*.*                                                     /var/log/racks10..19.log

Diff Detail

Repository
rS FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Looks like a nice feature to have. Please see some comments inline.

Could you please upload the diff with context? It would help reviewing the patch.

usr.sbin/syslogd/syslogd.c
1085 ↗(On Diff #67661)

Why not enforcing caller to provide the message length? Also: why <0?

1124 ↗(On Diff #67661)

/* NOTREACHED */ ?

2175 ↗(On Diff #67661)

You can consider doing strdup() in the caller, simplifying the error handling here.

2201 ↗(On Diff #67661)

Given the filter line already contains everything before first ",", what is the point of doing strNcasecmp()?

2216 ↗(On Diff #67661)

*argv[1] == '!' ?

2247 ↗(On Diff #67661)

Providing some example(s) here would help going through the code.

This revision was not accepted when it landed; it landed in state Needs Review.Mar 26 2020, 11:54 AM
This revision was automatically updated to reflect the committed changes.