Page MenuHomeFreeBSD

Property-based filters for syslogd
ClosedPublic

Authored by lytboris_gmail.com on Feb 2 2020, 9:44 AM.
Tags
None
Referenced Files
Unknown Object (File)
Jan 22 2024, 4:05 PM
Unknown Object (File)
Jan 19 2024, 1:45 PM
Unknown Object (File)
Dec 25 2023, 10:10 PM
Unknown Object (File)
Dec 20 2023, 3:06 AM
Unknown Object (File)
Dec 13 2023, 11:03 AM
Unknown Object (File)
Dec 11 2023, 2:10 AM
Unknown Object (File)
Nov 20 2023, 8:19 AM
Unknown Object (File)
Nov 12 2023, 7:59 PM
Subscribers

Details

Summary

Rationale

Message filtering options in syslogd are quite limited:

  • facility/level
  • host name
  • process name While facility and level are pretty straightforward, filtering by host or process names could be quite CPU intensive and/or even unusable: names are evaluated as full string match strictly.

Couple of years ago we've hit a host/process name string length shortage and chose to fix it with a kind of a dirty hack just growing buffer bigger (see rS275729).

Time comes by and now we're limited even by LINE_MAX-sized full string match filter: thanks to rS334719 we're now able to run tons of OpenVPN processes on single socket yet still need to skip logging into all.log from them.

At first I was considering to add shell-like pattern matching globs and/or regular expressions but after careful evaluation I came to the idea that these new matching schemes are not going to fit into constraints of present syntax.

rsyslog-like property-based syntax

Luckily enough, a syslogd alternative, rsyslog, has a good way to express things needed in a configuration file already, Property-based filters (see https://www.rsyslog.com/doc/v8-stable/configuration/filters.html#property-based-filters).

This proposal uses similar syntax, but implements for this moment just a tiny subset of properties that can be judged for filtering purposes.

Property-based filters

(This is a copy&paste from man page)

program, hostname specifications performs exact match filtering against
explicit field only.  Property-based filters feature substring and
regular expressions (see re_format(7)) matching against various message
attributes.  Filter specification starts with ‘#:’ or ‘:’ followed by
three comma-separated fields property, operator, "value".  Value must be
double-quoted. A double quote and backslash must be escaped by a
blackslash.

Following properties are supported as test value:

•   ‘msg’ - body of the message received.
•   ‘programname’ - program name sent the message
•   ‘hostname’ - hostname of message's originator
•   ‘source’ - an alias for hostname

Operator specifies a comparison function between propertie's
 value against filter's value.  Possible operators:

•   ‘contains’ - true if filter value is found as a substring of property
•   ‘isequal’ - true if filter value is equal to property
•   ‘startswith’ - true if property starts with filter value
•   ‘regex’ - true if property matches basic regular expression defined
    in filter value
•   ‘ereregex’ - true if property matches extended regular expression
    defined in filter value

Operator may be prefixed by

•   ‘!’ - to invert compare logic
•   ‘icase_’ - to make comparison function case insensitive

Some usage examples

# Log ipfw messages without syncing after every message.
!ipfw
*.*                                                     -/var/log/ipfw

# Log ipfw messages with "Deny" in the message body.
:msg, contains, ".*Deny.*"
*.*                                                     /var/log/ipfw.deny

# Reset program name filtering
!*

# Log messages from bird or bird6 into one file
:processname, regex, "^bird6?$"
*.*                                                     /var/log/bird-all.log

# Log messages from servers in racks 10-19 in multiple locations, case insensitive
:hostname, icase_ereregex, "^server-(dcA|podB|cdn)-rack1[0-9]{2}\..*"
*.*                                                     /var/log/racks10..19.log

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

Looks like a nice feature to have. Please see some comments inline.

Could you please upload the diff with context? It would help reviewing the patch.

usr.sbin/syslogd/syslogd.c
1085

Why not enforcing caller to provide the message length? Also: why <0?

1124

/* NOTREACHED */ ?

2175

You can consider doing strdup() in the caller, simplifying the error handling here.

2201

Given the filter line already contains everything before first ",", what is the point of doing strNcasecmp()?

2216

*argv[1] == '!' ?

2247

Providing some example(s) here would help going through the code.

This revision was not accepted when it landed; it landed in state Needs Review.Mar 26 2020, 11:54 AM
This revision was automatically updated to reflect the committed changes.