Change Details

== Rationale Message filtering options in syslogd are quite limited: * facility/level * host name * process name While facility and level are pretty straightforward, filtering by host or process names could be quite CPU intensive and/or even unusable: names are evaluated as full string match strictly. Couple of years ago we've hit a host/process name string length shortage and chose to fix it with a kind of a dirty hack just growing buffer bigger (see rsrc275729). Time comes by and now we're limited even by LINE_MAX-sized full string match filter: thanks to rsrc334719 we're now able to run tons of OpenVPN processes on single socket yet still need to skip logging into all.log from them. At first I was considering to add shell-like pattern matching globs and/or regular expressions but after careful evaluation I came to the idea that these new matching schemes are not going to fit into constraints of present syntax. === rsyslog-like property-based syntax Luckily enough, a syslogd alternative, rsyslog, has a good way to express things needed in a configuration file already, Property-based filters (see https://www.rsyslog.com/doc/v8-stable/configuration/filters.html#property-based-filters). This proposal uses similar syntax, but implements for this moment just a tiny subset of properties that can be judged for filtering purposes. == Property-based filters (This is a copy&paste from man page) ``` program, hostname specifications performs exact match filtering against explicit field only. Property-based filters feature substring and regular expressions (see re_format(7)) matching against various message attributes. Filter specification starts with ‘#:’ or ‘:’ followed by three comma-separated fields property, operator, "value". Value must be double-quoted. A double quote and backslash must be escaped by a blackslash. Following properties are supported as test value: • ‘msg’ - body of the message received. • ‘programname’ - program name sent the message • ‘hostname’ - hostname of message's originator • ‘source’ - an alias for hostname Operator specifies a comparison function between propertie's value against filter's value. Possible operators: • ‘contains’ - true if filter value is found as a substring of property • ‘isequal’ - true if filter value is equal to property • ‘startswith’ - true if property starts with filter value • ‘regex’ - true if property matches basic regular expression defined in filter value • ‘ereregex’ - true if property matches extended regular expression defined in filter value Operator may be prefixed by • ‘!’ - to invert compare logic • ‘icase_’ - to make comparison function case insensitive ``` == Some usage examples ``` # Log ipfw messages without syncing after every message. !ipfw *.* -/var/log/ipfw # Log ipfw messages with "Deny" in the message body. :msg, contains, ".*Deny.*" *.* /var/log/ipfw.deny # Reset program name filtering !* # Log messages from bird or bird6 into one file :processname, regex, "^bird6?$" *.* /var/log/bird-all.log # Log messages from servers in racks 10-19 in multiple locations, case insensitive :hostname, icase_ereregex, "^server-(dcA|podB|cdn)-rack1[0-9]{2}\..*" *.* /var/log/racks10..19.log ```

== Rationale Message filtering options in syslogd are quite limited: * facility/level * host name * process name While facility and level are pretty straightforward, filtering by host or process names could be quite CPU intensive and/or even unusable: names are evaluated as full string match strictly. Couple of years ago we've hit a host/process name string length shortage and chose to fix it with a kind of a dirty hack just growing buffer bigger (see rS275729). Time comes by and now we're limited even by LINE_MAX-sized full string match filter: thanks to rS334719 we're now able to run tons of OpenVPN processes on single socket yet still need to skip logging into all.log from them. At first I was considering to add shell-like pattern matching globs and/or regular expressions but after careful evaluation I came to the idea that these new matching schemes are not going to fit into constraints of present syntax. === rsyslog-like property-based syntax Luckily enough, a syslogd alternative, rsyslog, has a good way to express things needed in a configuration file already, Property-based filters (see https://www.rsyslog.com/doc/v8-stable/configuration/filters.html#property-based-filters). This proposal uses similar syntax, but implements for this moment just a tiny subset of properties that can be judged for filtering purposes. == Property-based filters (This is a copy&paste from man page) ``` program, hostname specifications performs exact match filtering against explicit field only. Property-based filters feature substring and regular expressions (see re_format(7)) matching against various message attributes. Filter specification starts with ‘#:’ or ‘:’ followed by three comma-separated fields property, operator, "value". Value must be double-quoted. A double quote and backslash must be escaped by a blackslash. Following properties are supported as test value: • ‘msg’ - body of the message received. • ‘programname’ - program name sent the message • ‘hostname’ - hostname of message's originator • ‘source’ - an alias for hostname Operator specifies a comparison function between propertie's value against filter's value. Possible operators: • ‘contains’ - true if filter value is found as a substring of property • ‘isequal’ - true if filter value is equal to property • ‘startswith’ - true if property starts with filter value • ‘regex’ - true if property matches basic regular expression defined in filter value • ‘ereregex’ - true if property matches extended regular expression defined in filter value Operator may be prefixed by • ‘!’ - to invert compare logic • ‘icase_’ - to make comparison function case insensitive ``` == Some usage examples ``` # Log ipfw messages without syncing after every message. !ipfw *.* -/var/log/ipfw # Log ipfw messages with "Deny" in the message body. :msg, contains, ".*Deny.*" *.* /var/log/ipfw.deny # Reset program name filtering !* # Log messages from bird or bird6 into one file :processname, regex, "^bird6?$" *.* /var/log/bird-all.log # Log messages from servers in racks 10-19 in multiple locations, case insensitive :hostname, icase_ereregex, "^server-(dcA|podB|cdn)-rack1[0-9]{2}\..*" *.* /var/log/racks10..19.log ```

== Rationale Message filtering options in syslogd are quite limited: * facility/level * host name * process name While facility and level are pretty straightforward, filtering by host or process names could be quite CPU intensive and/or even unusable: names are evaluated as full string match strictly. Couple of years ago we've hit a host/process name string length shortage and chose to fix it with a kind of a dirty hack just growing buffer bigger (see rsrcrS275729). Time comes by and now we're limited even by LINE_MAX-sized full string match filter: thanks to rsrcrS334719 we're now able to run tons of OpenVPN processes on single socket yet still need to skip logging into all.log from them. At first I was considering to add shell-like pattern matching globs and/or regular expressions but after careful evaluation I came to the idea that these new matching schemes are not going to fit into constraints of present syntax. === rsyslog-like property-based syntax Luckily enough, a syslogd alternative, rsyslog, has a good way to express things needed in a configuration file already, Property-based filters (see https://www.rsyslog.com/doc/v8-stable/configuration/filters.html#property-based-filters). This proposal uses similar syntax, but implements for this moment just a tiny subset of properties that can be judged for filtering purposes. == Property-based filters (This is a copy&paste from man page) ``` program, hostname specifications performs exact match filtering against explicit field only. Property-based filters feature substring and regular expressions (see re_format(7)) matching against various message attributes. Filter specification starts with ‘#:’ or ‘:’ followed by three comma-separated fields property, operator, "value". Value must be double-quoted. A double quote and backslash must be escaped by a blackslash. Following properties are supported as test value: • ‘msg’ - body of the message received. • ‘programname’ - program name sent the message • ‘hostname’ - hostname of message's originator • ‘source’ - an alias for hostname Operator specifies a comparison function between propertie's value against filter's value. Possible operators: • ‘contains’ - true if filter value is found as a substring of property • ‘isequal’ - true if filter value is equal to property • ‘startswith’ - true if property starts with filter value • ‘regex’ - true if property matches basic regular expression defined in filter value • ‘ereregex’ - true if property matches extended regular expression defined in filter value Operator may be prefixed by • ‘!’ - to invert compare logic • ‘icase_’ - to make comparison function case insensitive ``` == Some usage examples ``` # Log ipfw messages without syncing after every message. !ipfw *.* -/var/log/ipfw # Log ipfw messages with "Deny" in the message body. :msg, contains, ".*Deny.*" *.* /var/log/ipfw.deny # Reset program name filtering !* # Log messages from bird or bird6 into one file :processname, regex, "^bird6?$" *.* /var/log/bird-all.log # Log messages from servers in racks 10-19 in multiple locations, case insensitive :hostname, icase_ereregex, "^server-(dcA|podB|cdn)-rack1[0-9]{2}\..*" *.* /var/log/racks10..19.log ```