setsockopt: Improve SO_SETFIB handling

[markj]

Summary

• Serialize updates to so_fibnum with the socket lock.
• Make sure we can't change the FIB of a listening socket.

My goal here is to permit FIB-local listening sockets, wherein
multiple listening sockets listen on a port (using SO_REUSEPORT),
but each listening socket only accepts connections that come from
its FIB.

[markj]

In fact, this probably does not go far enough. I'm not sure when it's useful to change the fibnum of a socket after creation time, but it's dangerous in general since the fibnum is also inherited by the inpcb.

[melifaro]

In D47384#1080729, @markj wrote:

In fact, this probably does not go far enough. I'm not sure when it's useful to change the fibnum of a socket after creation time, but it's dangerous in general since the fibnum is also inherited by the inpcb.

What about multi-fib-aware applications? For example, nginx allows to specify a specific fib for each listening socket.

[markj]

In D47384#1080748, @melifaro wrote:

In D47384#1080729, @markj wrote:

In fact, this probably does not go far enough. I'm not sure when it's useful to change the fibnum of a socket after creation time, but it's dangerous in general since the fibnum is also inherited by the inpcb.

What about multi-fib-aware applications? For example, nginx allows to specify a specific fib for each listening socket.

Do you know offhand if it sets the FIB after calling listen()? I would hope/expect not.

[zlei]

In D47384#1080748, @melifaro wrote:

In D47384#1080729, @markj wrote:

In fact, this probably does not go far enough. I'm not sure when it's useful to change the fibnum of a socket after creation time, but it's dangerous in general since the fibnum is also inherited by the inpcb.

What about multi-fib-aware applications? For example, nginx allows to specify a specific fib for each listening socket.

From nginx doc https://nginx.org/en/docs/http/ngx_http_core_module.html,

setfib=number
this parameter (0.8.44) sets the associated routing table, FIB (the SO_SETFIB option) for the listening socket. This currently works only on FreeBSD.

and the usage: https://github.com/nginx/nginx/blob/master/src/core/ngx_connection.c#L817

nginx set SO_SETFIB option before listening ( the socket ).

For example, nginx allows to specify a specific fib for each listening socket.

Do you hint that one listening socket can have multiple / change the fib number ? That sounds not possible. Well it is absolutely OK to have multiple fibs for multiple different listening socket.

[markj]

In D47384#1081563, @zlei wrote:

In D47384#1080748, @melifaro wrote:

In D47384#1080729, @markj wrote:

In fact, this probably does not go far enough. I'm not sure when it's useful to change the fibnum of a socket after creation time, but it's dangerous in general since the fibnum is also inherited by the inpcb.

What about multi-fib-aware applications? For example, nginx allows to specify a specific fib for each listening socket.

From nginx doc https://nginx.org/en/docs/http/ngx_http_core_module.html,

setfib=number
this parameter (0.8.44) sets the associated routing table, FIB (the SO_SETFIB option) for the listening socket. This currently works only on FreeBSD.

and the usage: https://github.com/nginx/nginx/blob/master/src/core/ngx_connection.c#L817

nginx set SO_SETFIB option before listening ( the socket ).

Thanks. I think this means that my patch will not break nginx, though it should probably allow setting the FIB if the number isn't changing.

The problem I'm trying to deal with is that this SO_FIBNUM handler does not update the fibnum stored in the associated inpcb.

For example, nginx allows to specify a specific fib for each listening socket.

Do you hint that one listening socket can have multiple / change the fib number ? That sounds not possible. Well it is absolutely OK to have multiple fibs for multiple different listening socket.

One limitation here is that the fibnum is not used when deciding which listening socket to pass a new connection to. I would like to have a mode (disabled by default to avoid breaking compatibility) where a listening socket only accepts connections from the FIB it's associated with. This is the larger goal that I'm working towards with some of these patches.

[zlei]

Do you hint that one listening socket can have multiple / change the fib number ? That sounds not possible

Well, what if we allow changing the fib number of listening socket, so filtering new connections based on the new fib number ?

@markj The first part Serialize updates to so_fibnum with the socket lock. looks good to me.

As for the second part Make sure we can't change the FIB of a listening socket, I do not know if future plan will do the opposite. Maybe it should be discussed via net mailing list.

[markj]

In D47384#1082454, @zlei wrote:

Do you hint that one listening socket can have multiple / change the fib number ? That sounds not possible

Well, what if we allow changing the fib number of listening socket, so filtering new connections based on the new fib number ?

Yes, we could try to support that. I will look into it further.

@markj The first part Serialize updates to so_fibnum with the socket lock. looks good to me.

As for the second part Make sure we can't change the FIB of a listening socket, I do not know if future plan will do the opposite. Maybe it should be discussed via net mailing list.

Thank you for the feedback. I'll work on this some more and start a discussion.