rcd(8): new service manager daemon

[bapt]

Summary

rcd is called by init(8), it reads service definitions from UCL unit files,
builds a dependency graph, and starts services in parallel.
After boot completes, it remains running as a supervisor daemon, automatically
restarting failed services and accepting control commands via a UNIX domain
socket.

Design goals:

• Fast parallel boot via dependency DAG
• No PID races: use posix_spawn with process descriptors
• No process escape: become subreaper via procctl(2)
• Socket activation: pre-bind sockets, pass via fd inheritance
• Resource control: per-service limits via rctl(2)
• Service isolation: native jail(2) integration
• OOM protection: procctl(2) PROC_SPROTECT
• UCL-based unit files via libucl
• Embedded Lua interpreter for inline service hooks
• Template units for per-instance services
• Per-service access control
• JSON Schema validation of unit files
• Full backward compatibility with existing rc.d scripts
• Called by init(8) with no changes to init

[bapt]

Address zombie in legacy mode
add xpdwait for EINTR

[bapt]

In D56835#1320536, @otis wrote:

And, when invoked as /sbin/rcd on a live system, I'm getting:

1700  -  Is    0:00.02 |-- rcd -v
2599  -  Z     0:00.00 | |-- <defunct>
2600  -  Z     0:00.00 | |-- <defunct>
2601  -  Z     0:00.00 | |-- <defunct>
2602  -  Z     0:00.00 | |-- <defunct>
2603  -  Z     0:00.00 | |-- <defunct>
3180  -  Z     0:00.00 | |-- <defunct>
3182  -  Z     0:00.00 | |-- <defunct>
3185  -  Z     0:00.00 | |-- <defunct>
3228  -  I     0:00.00 | |-- rcd-exec: reaper: syslogd (pid 3378) (rcd-exec)
3371  -  SCs   0:00.00 | | |-- /usr/sbin/syslogd -s
3376  -  I     0:00.00 | | |-- syslogd: syslogd.casper (syslogd)
3378  -  Is    0:00.00 | | `-- syslogd: system.net (syslogd)
3359  -  Z     0:00.00 | |-- <defunct>
3361  -  Z     0:00.00 | |-- <defunct>
3362  -  Z     0:00.00 | |-- <defunct>
3364  -  Z     0:00.00 | |-- <defunct>
3365  -  Z     0:00.00 | |-- <defunct>
3368  -  Z     0:00.00 | |-- <defunct>
3536  -  I     0:00.00 | `-- rcd-exec: reaper: cron (pid 3715) (rcd-exec)
3715  -  Ss    0:00.00 |   `-- /usr/sbin/cron -s

too many defuct processes.

fixed

[ngie]

In D56835#1320540, @bapt wrote:

In D56835#1320487, @ngie wrote:

In D56835#1320486, @ngie wrote:

This is a lot of greenfield code. Simple architectural question: is there a reason why this was written in C instead of C++? C++ wouldn't provide all of the functionality, but it would provide access to a number of data structures and algorithms that aren't available in C.

This comment is coming from a place of seeing and dealing with the complexity associated with libatf-c vs libatf-c++ -- the former has a number of complicated forms of memory management and data structures that simply do not need to exist with libatf-c++.

I don't know how to code in c++.

:(. Is this project available on a branch that I can help contribute to? I really want to leverage C++ to avoid a lot of wheel reinventing with the daemon.

[bapt]

In D56835#1320723, @ngie wrote:

In D56835#1320540, @bapt wrote:

In D56835#1320487, @ngie wrote:

In D56835#1320486, @ngie wrote:

This is a lot of greenfield code. Simple architectural question: is there a reason why this was written in C instead of C++? C++ wouldn't provide all of the functionality, but it would provide access to a number of data structures and algorithms that aren't available in C.

This comment is coming from a place of seeing and dealing with the complexity associated with libatf-c vs libatf-c++ -- the former has a number of complicated forms of memory management and data structures that simply do not need to exist with libatf-c++.

I don't know how to code in c++.

:(. Is this project available on a branch that I can help contribute to? I really want to leverage C++ to avoid a lot of wheel reinventing with the daemon.

I really don t want c++ as it would mean I can t contribute anymore

[ngie]

sbin/rcd/xmalloc.h

2

This feels like we could leverage a lot of the constructs provided by NetBSD (or was it OpenBSD?) in this space to avoid reinventing the wheel. I'd need to find the exact source file, but I've seen similar constructs in those spaces -- memory serves me correctly, it was NetBSD...

[adrian]

sbin/rcd/xmalloc.h

2

I've had a quick chat with bapt (here? irc?) whre I'd like a bunch of this stuff broken out into libraries for future reuse.

[ngie]

Please add [explicit? implicit?] versioning information to the schema and UCL files -- otherwise you risk problems when upgrading/downgrading schema/unit files.

sbin/rcd/schema/unit.schema.json

147

If this is an opaque object, does it make sense to add a separate property to explicitly state the type? That would allow other end-users the ability to extend rcd as needed with other language bindings in the future (golang, python, rust, ...).

[bapt]

sbin/rcd/xmalloc.h

2

xmalloc, xstring, vec.h and hash.h has been exercise for years in pkg, why would I replace it with openbsd's or netbsd's?

[ngie]

This is a lot of change to digest in a single differential. I don't want to recommend "using LLMs" for reviewing -- it's a slippery slope for all involved as LLMs can be confident bullshitters/idiots.

Is there any way this work could please be broken down into a diff stack?

sbin/rcctl/rcctl.c

262

*sigh* this code (for example) would be a lot simpler/cleaner with Boost::Program_options / C++ strings :(.

sbin/rcd/compat.c

43–45

Why isn't this using regular expressions or a more formalized lexer/parser like yacc, etc?

sbin/rcd/depgraph.c

71

Please add sys/queue.h for these macros.

sbin/rcd/hash.c

8–11

https://xkcd.com/927/ comes to mind if this came from libpkg.

sbin/rcd/hash.h

2

Or maybe just use C++'s std::map / std::unsorted_map data structures instead? I would be shocked if one of the other BSDs hasn't already invented this wheel before (which we might be able to borrow/contribute bits back for).

sbin/rcd/jail_svc.c

33–38

• Should these be managed in config files?
• What about old versions of jails/rcd vs newer hosts where there might be a feature/settings mismatch?

sbin/rcd/luaexec.c

2

This looks like it does a lot that lutok tries to do :(...

sbin/rcd/mum.h

2

Interesting copyright notice.. how're we going to stay on top of changes to this header if licensing changes, etc?

[bapt]

In D56835#1320727, @ngie wrote:

Please add [explicit? implicit?] versioning information to the schema and UCL files -- otherwise you risk problems when upgrading/downgrading schema/unit files.

for versionning I was thinking like in pkg to never break backward compatibility, by always adding new keys and never mutate existing one, this also solves forward compatibility because rcd will not validate a unit made in the new format.

[ziaee]

Hey Bapt! This is super cool. I've been busy with 15.1 docs and not had time to review this. Here is the SBOM style review, I will hopefully follow up with the docs review this week.

sbin/rcctl/rcctl.8

2

According to our style guides for consistency with C comments.

sbin/rcctl/rcctl.c

2–6

Copyright comes first in SPDX only header, no hypen.

sbin/rcd/compat.c

2–6

Ditto

sbin/rcd/control.c

2–6

Ditto

sbin/rcd/depgraph.c

2–6

Ditto

sbin/rcd/enable.c

2–6

Ditto

sbin/rcd/hash.c

2–6

Ditto

sbin/rcd/hash.h

2–6

Ditto

sbin/rcd/jail_svc.c

2–6

Ditto

sbin/rcd/log.c

2–6

Ditto

sbin/rcd/luaexec.c

2–6

Ditto

sbin/rcd/mum.h

2

Please add the SPDX-License-Identifier: MIT to the top of this.

sbin/rcd/process.c

2–6

Copyright comes first if using SPDX only license header. Hyphen is useless and no longer recommended.

sbin/rcd/rcd-exec.8

2

Please prepend a blank comment for consistency with the style guides and C style comments.

sbin/rcd/rcd-exec.c

2–6

Copyright comes first in SPDX only license header. Useless hyphen is no longer recommended.

sbin/rcd/rcd-lua.3

2

Please prepend a blank comment for consistency with style guides and C style comments.

sbin/rcd/rcd.8

2

Please prepend a blank comment for consistency with style guides and C style comments.

sbin/rcd/rcd.c

2–7

Copyright comes first in SPDX only license header. Please remove useless hyphen for consistency with style guides. Also, you had an extra blank comment line.

sbin/rcd/rcd.conf.5

2

Please prepend a blank comment line for consistency with style guides and C style comments.

sbin/rcd/rcd.d.5

2

Please prepend a blank comment line for consistency with style guides and C style comments.

sbin/rcd/rcd.h

2–27

Why use full license? you didn't do that in the others?

sbin/rcd/rctl_mgr.c

2–6

Copyright comes first in SPDX only license headers. Also, please remove useless hyphen for consistency with style guides.

sbin/rcd/sockact.c

2–6

Copyright comes first in SPDX only license headers. Also, please remove useless hyphen for consistency with style guides.

sbin/rcd/tests/compat_test.c

2–6

Ditto

sbin/rcd/tests/config_test.c

2–6

Ditto

sbin/rcd/tests/depgraph_test.c

2–6

Ditto

sbin/rcd/tests/enable_test.c

2–6

Ditto

sbin/rcd/tests/jail_svc_test.c

2–6

Ditto

sbin/rcd/tests/luaexec_test.c

2–7

Ditto

sbin/rcd/tests/process_test.c

2–6

Ditto

sbin/rcd/tests/rcd_jail_test.sh

2–5

Ditto

sbin/rcd/tests/rctl_test.c

2–6

Ditto

sbin/rcd/tests/sockact_test.c

2–6

Ditto

sbin/rcd/tests/unit_test.c

2–6

Ditto

sbin/rcd/unit.c

2–6

Ditto

sbin/rcd/vec.h

2

Please remove useless hyphen for consistency with style guides.

[otis]

In D56835#1320563, @bapt wrote:

In D56835#1320536, @otis wrote:

And, when invoked as /sbin/rcd on a live system, I'm getting:

1700  -  Is    0:00.02 |-- rcd -v
2599  -  Z     0:00.00 | |-- <defunct>
2600  -  Z     0:00.00 | |-- <defunct>
2601  -  Z     0:00.00 | |-- <defunct>
2602  -  Z     0:00.00 | |-- <defunct>
2603  -  Z     0:00.00 | |-- <defunct>
3180  -  Z     0:00.00 | |-- <defunct>
3182  -  Z     0:00.00 | |-- <defunct>
3185  -  Z     0:00.00 | |-- <defunct>
3228  -  I     0:00.00 | |-- rcd-exec: reaper: syslogd (pid 3378) (rcd-exec)
3371  -  SCs   0:00.00 | | |-- /usr/sbin/syslogd -s
3376  -  I     0:00.00 | | |-- syslogd: syslogd.casper (syslogd)
3378  -  Is    0:00.00 | | `-- syslogd: system.net (syslogd)
3359  -  Z     0:00.00 | |-- <defunct>
3361  -  Z     0:00.00 | |-- <defunct>
3362  -  Z     0:00.00 | |-- <defunct>
3364  -  Z     0:00.00 | |-- <defunct>
3365  -  Z     0:00.00 | |-- <defunct>
3368  -  Z     0:00.00 | |-- <defunct>
3536  -  I     0:00.00 | `-- rcd-exec: reaper: cron (pid 3715) (rcd-exec)
3715  -  Ss    0:00.00 |   `-- /usr/sbin/cron -s

too many defuct processes.

fixed

Still zombies:

1627  -  Z      0:00.00 | |-- <defunct>
1631  -  Z      0:00.00 | |-- <defunct>
1634  -  Z      0:00.00 | |-- <defunct>
1835  -  Z      0:00.00 | |-- <defunct>
1838  -  Z      0:00.00 | |-- <defunct>
1840  -  Z      0:00.00 | |-- <defunct>

otis@b14:~ % sudo procstat 1627
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
 1627    19    19    19     0   1 root     -         FreeBSD ELF64 logger
otis@b14:~ % sudo procstat 1631
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
 1631    19    19    19     0   1 root     -         FreeBSD ELF64 logger
otis@b14:~ % sudo procstat 1634
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
 1634    19  1634  1634     0   1 root     -         FreeBSD ELF64 logger
otis@b14:~ % sudo procstat 1835
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
 1835    19    19    19     0   1 root     -         FreeBSD ELF64 savecore
otis@b14:~ % sudo procstat 1838
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
 1838    19    19    19     0   1 root     -         FreeBSD ELF64 savecore
otis@b14:~ % sudo procstat 1840
  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
 1840    19    19    19     0   1 root     -         FreeBSD ELF64 savecore

[arrowd]

In D56835#1311164, @imp wrote:

I'd like to echo the user requirement. I have not current need for it at $WORK, but can see a need in the future. Having the ability for users to tie into the startup, running as themselves, much like we allow users to add jobs to cron, at, etc would be a useful, but not gating, feature.

I believe that user units is actually an orthogonal feature of the system service manager. If we are to implement user units, IMO, it should be a cross-platform project that consumes pristine systemd unit descriptions and hooks into the OS via PAM. It doesn't even has to be a part of base, because only uses of user units I stumbled upon so far were coming from ports.

I'm working on a systemd-logind replacement, which will fill the user units hole nicely, but there is a ton of work to do.

[bapt]

In D56835#1320939, @arrowd wrote:

In D56835#1311164, @imp wrote:

I'd like to echo the user requirement. I have not current need for it at $WORK, but can see a need in the future. Having the ability for users to tie into the startup, running as themselves, much like we allow users to add jobs to cron, at, etc would be a useful, but not gating, feature.

I believe that user units is actually an orthogonal feature of the system service manager. If we are to implement user units, IMO, it should be a cross-platform project that consumes pristine systemd unit descriptions and hooks into the OS via PAM. It doesn't even has to be a part of base, because only uses of user units I stumbled upon so far were coming from ports.

I'm working on a systemd-logind replacement, which will fill the user units hole nicely, but there is a ton of work to do.

we are interested in 2 kind of things here, user units which are tight to "sessions" covert your use case and I don't plan into adding them (they fall into the logindish thing you are working on). but there are plenty of other non desktop user units which might be useful and those I am planning on working on after rcd hits the tree and yes there is a tiny overlap between those 2.

sbin/rcctl/rcctl.c

262

first we would not have boost in base anyway, and second this is clearly subjective, as it is really not complex, and if needed wr can make it simple with a static list external, this kind of comment is not useful.

sbin/rcd/hash.c

8–11

how constructive

[bapt]

address @ziaee comments

[bapt]

actually properly read ziaee's comments

[bapt]

reap hard; this time I hope it should be fine for your case @otis

[bapt]

Add boot time looging by popular demand

[otis]

In D56835#1320957, @bapt wrote:

reap hard; this time I hope it should be fine for your case @otis

no, it did not.

excerpt:

# /sbin/rcd -v
rcd starting
rcd: daemonized (pid 22)
WARNING: control bindat: Read-only file system
WARNING: control socket init failed
started dumpon (pid 45, procdesc fd 5)
started sysctl (pid 46, procdesc fd 6)
started localpkg (pid 47, procdesc fd 7)
started zfs (pid 48, procdesc fd 8)
localpkg exited (status 256, pid 47)
sysctl exited (status 0, pid 46)
started hostid (pid 153, procdesc fd 6)
dumpon exited (status 0, pid 45)
started disks (pid 159, procdesc fd 5)
hostid exited (status 0, pid 153)
disks exited (status 0, pid 159)
started swap (pid 207, procdesc fd 5)
swap exited (status 0, pid 207)
started fsck (pid 232, procdesc fd 5)
fsck exited (status 0, pid 232)
started root (pid 256, procdesc fd 5)
root exited (status 0, pid 256)
started mdconfig (pid 283, procdesc fd 5)
started serial (pid 284, procdesc fd 6)
serial exited (status 0, pid 284)
zfs exited (status 0, pid 48)
mdconfig exited (status 256, pid 283)
started mountcritlocal (pid 312, procdesc fd 5)
mountcritlocal exited (status 0, pid 312)
started kldxref (pid 338, procdesc fd 5)
started var_run (pid 339, procdesc fd 6)
started var (pid 340, procdesc fd 7)
started tmp (pid 341, procdesc fd 8)
var_run exited (status 0, pid 339)
var exited (status 0, pid 340)

the sources are positively current. I can provide an access to that system, to help debugging.

[bapt]

Handle cases when a user explicitly set /var/run a tmpfs in the fstab

[bapt]

more work on zombies
show resources for legacy scripts

[otis]

In D56835#1320998, @bapt wrote:

more work on zombies
show resources for legacy scripts

This works for me. thanks! No more zombies.

[vsasjason_gmail.com]

Will it help sshd to start without being indefinitely blocked by e.g. mysqld? That would be enough of a reason for me personally to switch to this thing immediately.

[michaelo]

In D56835#1321082, @vsasjason_gmail.com wrote:

Will it help sshd to start without being indefinitely blocked by e.g. mysqld? That would be enough of a reason for me personally to switch to this thing immediately.

Can you elaborate on this?

[vsasjason_gmail.com]

In D56835#1321115, @michaelo wrote:

In D56835#1321082, @vsasjason_gmail.com wrote:

Will it help sshd to start without being indefinitely blocked by e.g. mysqld? That would be enough of a reason for me personally to switch to this thing immediately.

Can you elaborate on this?

Sure:

$ rcorder /etc/rc.d/* /usr/local/etc/rc.d/* | egrep -n "sshd|usr"
29:/usr/local/etc/rc.d/microcode_update
108:/usr/local/etc/rc.d/git_daemon
114:/usr/local/etc/rc.d/xrdp
115:/usr/local/etc/rc.d/transmission
116:/usr/local/etc/rc.d/syncthing-relaysrv
117:/usr/local/etc/rc.d/syncthing-relaypoolsrv
118:/usr/local/etc/rc.d/syncthing-discosrv
119:/usr/local/etc/rc.d/syncthing
120:/usr/local/etc/rc.d/smartd
121:/usr/local/etc/rc.d/poudriered
122:/usr/local/etc/rc.d/miniupnpc
123:/usr/local/etc/rc.d/dbus
131:/etc/rc.d/sshd
133:/usr/local/etc/rc.d/lightdm
<CUT>

So whatever <131 potentially hangs, I've got no ssh into machine to fix/kill it as rc is sequential and depends are uncontrollable (because every port may install rc script and put it wherever maintainer decided). I, personally, suffered of this issue before; and the issue is probably as old as rc itself.

[bapt]

improve experience when dealing with legacy script:
better output of the rcctl command, and fix all bugs dealing with local_unbound
reported so far

[bapt]

In D56835#1321082, @vsasjason_gmail.com wrote:

Will it help sshd to start without being indefinitely blocked by e.g. mysqld? That would be enough of a reason for me personally to switch to this thing immediately.

yes