syslogd: Handle connection errors when setting up forwarding sockets
ClosedPublic
Actions

Authored by markj on Jun 2 2026, 3:24 PM.

Details

Reviewers

ae
des
jfree

Commits

rG8a62a5f77b10: syslogd: Handle connection errors when setting up forwarding sockets

Summary

Suppose a forwarding host is identified by a hostname. We will attempt
to set up a connected forwarding socket for each address returned by
getaddrinfo(). However, a failure to connect some of these sockets
shouldn't be fatal.

PR: 293758

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Skipped

Unit

Tests Skipped

Build Status

Buildable 73884
Build 70767: arc lint + arc unit

Event Timeline

markj created this revision.Jun 2 2026, 3:24 PM

Herald added a subscriber: imp. · View Herald TranscriptJun 2 2026, 3:24 PM

markj requested review of this revision.Jun 2 2026, 3:24 PM

Harbormaster completed remote builds in B73625: Diff 179094.Jun 2 2026, 3:24 PM

No, it doesn't help.

17146 syslogd  CALL  socket(PF_INET6,0x10000002<SOCK_DGRAM|SOCK_CLOEXEC>,IPPROTO_UDP)
17146 syslogd  RET   socket 5
17146 syslogd  CALL  connect(0x5,0x23c6f2034820,0x1c)
17146 syslogd  STRU  struct sockaddr { AF_INET6, [2a02:XXX::XXX]:1 }
17146 syslogd  RET   connect -1 errno 65 No route to host
17146 syslogd  CALL  close(0x5)
17146 syslogd  RET   close 0
17146 syslogd  CALL  socket(PF_INET,0x10000002<SOCK_DGRAM|SOCK_CLOEXEC>,IPPROTO_UDP)
17146 syslogd  RET   socket 5
17146 syslogd  CALL  connect(0x5,0x23c6f2034820,0x10)
17146 syslogd  STRU  struct sockaddr { AF_INET, 141.XXX.XXX.XXX:1 }
17146 syslogd  RET   connect -1 errno 51 Network is unreachable
17146 syslogd  CALL  close(0x5)
17146 syslogd  RET   close 0
17146 syslogd  CALL  socket(PF_INET6,0x2<SOCK_DGRAM>,IPPROTO_IP)
17146 syslogd  RET   socket 5
17146 syslogd  CALL  setsockopt(0x5,SOL_SOCKET,SO_REUSEPORT,0x23c6f20349fc,0x4)
17146 syslogd  RET   setsockopt 0
17146 syslogd  CALL  bind(0x5,0x4b0c58c436d8,0x1c)
17146 syslogd  STRU  struct sockaddr { AF_INET6, [::]:514 }
17146 syslogd  RET   bind 0
17146 syslogd  CALL  connect(0x5,0x4b0c58c132b0,0x1c)
17146 syslogd  STRU  struct sockaddr { AF_INET6, [2a02:XXX::XXX]:514 }
17146 syslogd  RET   connect -1 errno 65 No route to host
17146 syslogd  CALL  close(0x5)
17146 syslogd  RET   close 0
17146 syslogd  CALL  socket(PF_INET6,0x2<SOCK_DGRAM>,IPPROTO_IP)
17146 syslogd  RET   socket 5
17146 syslogd  CALL  connect(0x5,0x4b0c58c132b0,0x1c)
17146 syslogd  STRU  struct sockaddr { AF_INET6, [2a02:XXX::XXX]:514 }
17146 syslogd  RET   connect -1 errno 65 No route to host
17146 syslogd  CALL  close(0x5)
17146 syslogd  RET   close 0
17146 syslogd  CALL  socket(PF_INET,0x2<SOCK_DGRAM>,IPPROTO_IP)
17146 syslogd  RET   socket 5
17146 syslogd  CALL  setsockopt(0x5,SOL_SOCKET,SO_REUSEPORT,0x23c6f20349fc,0x4)
17146 syslogd  RET   setsockopt 0
17146 syslogd  CALL  bind(0x5,0x4b0c58c4f0c8,0x10)
17146 syslogd  STRU  struct sockaddr { AF_INET, 0.0.0.0:514 }
17146 syslogd  RET   bind 0
17146 syslogd  CALL  connect(0x5,0x4b0c58c12230,0x10)
17146 syslogd  STRU  struct sockaddr { AF_INET, 141.XXX.XXX.XXX:514 }
17146 syslogd  RET   connect -1 errno 51 Network is unreachable
17146 syslogd  CALL  close(0x5)
17146 syslogd  RET   close 0
17146 syslogd  CALL  socket(PF_INET,0x2<SOCK_DGRAM>,IPPROTO_IP)
17146 syslogd  RET   socket 5
17146 syslogd  CALL  connect(0x5,0x4b0c58c12230,0x10)
17146 syslogd  STRU  struct sockaddr { AF_INET, 141.XXX.XXX.XXX:514 }
17146 syslogd  RET   connect -1 errno 51 Network is unreachable
17146 syslogd  CALL  close(0x5)
17146 syslogd  RET   close 0
17146 syslogd  CALL  write(0x2,0x23c6f2034250,0x9)
17146 syslogd  GIO   fd 2 wrote 9 bytes
      "syslogd: "
17146 syslogd  RET   write 9
17146 syslogd  CALL  write(0x2,0x23c6f2034340,0x3a)
17146 syslogd  GIO   fd 2 wrote 58 bytes
      "failed to create forwarding socket for HOSTNAME.DOMAIN"
17146 syslogd  RET   write 58/0x3a
17146 syslogd  CALL  write(0x2,0x23c6f7472107,0x1)
17146 syslogd  GIO   fd 2 wrote 1 byte
      "
      "
17146 syslogd  RET   write 1
17146 syslogd  CALL  _exit(0x1)

As I understand, due to capability mode we cannot use sendto and connect when syslogd is already started and configured.
Probably, specified address is not reachable during boot time and its route will appear when routing daemon will start.
But bird has # REQUIRE: DAEMON in its startup script and it will start much later.

I tested the case when syslogd started after bird and it can do it without patch. But we lose some messages.
I also tried to add fake routes before syslogd start and remove them just after it started. But since connect(2) does source address selection for local endpoint, this leads to binding to wrong source address.
Currently I see only two solutions:

Rollback all commits to a point before migration to capability mode support and stay with old implementation.
Use rsyslogd as full replacement or use it to listen on local port and forward to remote endpoints.

Second attempt, add a test case.

Herald added a subscriber: jonathan. · View Herald TranscriptJun 11 2026, 6:35 PM

Harbormaster completed remote builds in B73831: Diff 179625.Jun 11 2026, 6:35 PM

In D57394#1318642, @ae wrote:

I tested the case when syslogd started after bird and it can do it without patch. But we lose some messages.
I also tried to add fake routes before syslogd start and remove them just after it started. But since connect(2) does source address selection for local endpoint, this leads to binding to wrong source address.
Currently I see only two solutions:

Rollback all commits to a point before migration to capability mode support and stay with old implementation.

Use rsyslogd as full replacement or use it to listen on local port and forward to remote endpoints.

Could you please give the latest version a try? It's kind of ugly, but I can't see a better solution.

jfree added inline comments.Jun 12 2026, 12:33 AM

usr.sbin/syslogd/syslogd.c
1869	I feel it is common for people to create multiple log entries all forwarding to the same hostname and port. In this case, we'd be hitting a lot of failed `cap_connect()`s, each of which require communication with casper and context switches. `find_forw_fd()` is relatively cheap (doesn't require any context switches), so it should probably come before `cap_connect()`. I understand this will only be done the first time a message is sent using this filed, but I'd prefer to minimize communication with casper where possible. On another note, maybe we can get rid of `maybe_dup_forw_socket()` in `make_forw_socket()` and just use `find_forw_fd()` to lazily open the connections since we're kind of just duplicating that fd search logic here.
1876	What happens if `cap_connect()` and `find_forw_fd()` both fail? It looks like we just end up calling `sendmsg()` with -1 as a file descriptor. The logic below will catch this, but errno will be EBADF, which is not covered in the switch.
3098	I find this comment confusing. What will syslogd reload? Do you mean syslogd will attempt a connection later (during send time)?

markj added inline comments.Jun 12 2026, 1:09 AM

usr.sbin/syslogd/syslogd.c
1869	I'm not so sure that find_forw_fd() is going to be so much cheaper: consider that a user might have many logging rules for different applications. I've seen syslogd configurations with thousands of entries, though IIRC those weren't necessarily forwarding entries. Yes, I should factor out the duplication here.
1876	I should add a comment, but it's covered by the default case in the switch, i.e., we basically discard the configuration line. I believe that's the same as how syslogd behaved on 14.x. (That case is leaking memory and descriptors though, I think?)
3098	The comment is just wrong. I had the idea of making syslogd call raise(SIGHUP) if it can't forward the message, since that would force it to re-resolve the configuration and forwarding addresses. But, that causes us to discard the log message, plus it's super expensive in general so we need to define some policy for when and how often to reload. So I abandoned that approach.

This seems to works now. I compared output of sockstat -46wn after reboot and after service syslogd restart - they seem the same.

Improve comments, call shutdown() on unconnected sockets

Harbormaster completed remote builds in B73884: Diff 179803.Mon, Jun 15, 3:45 PM

Still works. :)

This revision is now accepted and ready to land.Tue, Jun 16, 9:40 AM

markj added inline comments.Fri, Jun 26, 9:19 PM

usr.sbin/syslogd/syslogd.c
1869	I spent a bit of time trying to factor out the duplication and I think it's not worth it. The two functions operate on different structures, one on an nvlist entry and one on a filed. All they really have in common is the scan and memcmp() calls. So I somewhat prefer to leave it as it is.