The feature needs some tests, see the existing ones in tests/sys/netlink.

After investigating the behavior of netlink send when the buffer is full, I found:
FreeBSD async send path behavior:

• Send buffer full: Deadlocks (this indicates the recv buffer is also full).
• Recv buffer full: Send doesn't depend on it, so it will never block. It just keeps filling the send buffer.

Linux send path behavior:

• Send buffer will never fill up because of sync.
• Recv buffer full: Send doesn't depend on it, so it will never block, but it will fail on the next recv.

Therefore, I think the FreeBSD sync send path behavior should be:

• Send buffer full: Deadlock.
• Recv buffer full: Same as linux

Is this the expected behavior? If so, I will add a test for it.

Thank you for working on this!
Added a bit of comments on the MR itself, also a bit more generic questions:

1. What is the proposed processing mechanism in sync case? Can we clearly state the logic in the description?

For example, if the caller sent multiple nettlink messages on a single send / write call - do we assume the caller should have allocated enough RX buffer for all of the replies? Or put it differently - what should happen when we don't have enough space in the receive buffer to (a) get all of the reply, (b) to send netlink header with error and (c) when we, say, processed and replied to 2 netlink messages out of 5?

2. If the problem is the requirement around association with the process, it doesn't necessarily require sync - the other option would be, for example, creating the kernel threads associated with the process in questions. See https://reviews.freebsd.org/D39180 for more details. Not necessarily suggesting this direction, more stating that there may be more than 1 solution to the problem.

In D57519#1322555, @melifaro wrote:

Thank you for working on this!
Added a bit of comments on the MR itself, also a bit more generic questions:

1. What is the proposed processing mechanism in sync case? Can we clearly state the logic in the description?

For example, if the caller sent multiple nettlink messages on a single send / write call - do we assume the caller should have allocated enough RX buffer for all of the replies? Or put it differently - what should happen when we don't have enough space in the receive buffer to (a) get all of the reply, (b) to send netlink header with error and (c) when we, say, processed and replied to 2 netlink messages out of 5?

Linux is sync. Its behavior is this send() will succeed and process all requests (5, for example). But it drops the replies that exceed the receive buffer(3 replies). And will return ENOBUFS in the next recv() to indicate we have dropped some replies.
In other words, the success of send() does not depend on whether the receive buffer is full. So I align with its behavior, the corresponding part is the nl_dropped_bytes in nl_soreceive().

2. If the problem is the requirement around association with the process, it doesn't necessarily require sync - the other option would be, for example, creating the kernel threads associated with the process in questions. See https://reviews.freebsd.org/D39180 for more details. Not necessarily suggesting this direction, more stating that there may be more than 1 solution to the problem.

Yes, my requirement is to associate with the process. I see your option drops the taskqueue and creates a kernel thread associated with the process, which I think can solve the problem too. So maybe both options are ok.
But sync might can bring some addition benefits:

• send() can return error, for example when nlmsghdr is invalid.
• async will deadlock when the send buffer is full. sync doesn't have send buffer, so we don't have this problem.

Do you see a use case for setting sync mode mid-session? If not, i’d suggest doing conversion in setsockopt handler, tearing down the async thread and returning an error if some messages are pending procesding.

• async will deadlock when the send buffer is full. sync doesn't have send buffer, so we don't have this problem.

Could you share the async deadlock reproducer? It shoukdn’t really happen.

In D57519#1322702, @melifaro wrote:

Do you see a use case for setting sync mode mid-session? If not, i’d suggest doing conversion in setsockopt handler, tearing down the async thread and returning an error if some messages are pending procesding.

I think you're right, it's weird to set sync mid-session. I will change it later

• async will deadlock when the send buffer is full. sync doesn't have send buffer, so we don't have this problem.

Could you share the async deadlock reproducer? It shoukdn’t really happen.

Sorry, may be deadlock is inappropriate terminology to use here.

	fd = fullsocket();

	/* Both buffers full: block. */
	timer_done = 0;
	ATF_REQUIRE(sigaction(SIGALRM, &sigact, NULL) == 0);
	ATF_REQUIRE(setitimer(ITIMER_REAL, &itv, NULL) == 0);
	ATF_REQUIRE(send(fd, &hdr, sizeof(hdr), 0) == -1);
	ATF_REQUIRE(errno == EINTR);
	ATF_REQUIRE(timer_done == 1);

When the send buffer full, I think send will block indefinitely, waiting for space in the send buffer.
However, send buffer depends on the recv buffer having space. The recv buffer is waiting for the user to call recv(). but user is blocked.