What FreeBSD version (including revision) do you use for TEST PLAN script, e.g. what is your "stock FreeBSD"?

Are you sure that first command in the script is correct?

subnet=${1:-2.0}.0

It sets subnet to very different values like "2.0.0" with no arguments specified and to "1.0" for argument "1".

And you do you insist in doing things manually while moving IP (first route change, then reassign IP) instead of just allowing system do all housekeeping for you by reassigning IP first?

In tests/sys/netinet/fibs_test.sh there are some ATF tests that check this kind of functionality. They do it on separate FIBs so they won't upset the system running the tests, and they use tap(4) and/or epair(4) interfaces. You should be able to copy one of those to create a regression test for this problem.

In D14547#305342, @asomers wrote:

In tests/sys/netinet/fibs_test.sh there are some ATF tests that check this kind of functionality. They do it on separate FIBs so they won't upset the system running the tests, and they use tap(4) and/or epair(4) interfaces. You should be able to copy one of those to create a regression test for this problem.

How do I run the tests? Setting this in /usr/local/etc/kyua/kyua.conf doesn't seem to work:

test_suites.fibs_test.fibs = '1 2 3 4 5'

In D14547#305374, @rstone wrote:

In D14547#305342, @asomers wrote:

In tests/sys/netinet/fibs_test.sh there are some ATF tests that check this kind of functionality. They do it on separate FIBs so they won't upset the system running the tests, and they use tap(4) and/or epair(4) interfaces. You should be able to copy one of those to create a regression test for this problem.

How do I run the tests? Setting this in /usr/local/etc/kyua/kyua.conf doesn't seem to work:

test_suites.fibs_test.fibs = '1 2 3 4 5'

The fibs must actually exist. Set net.fibs=6 in /boot/loader.conf and reboot. Also, are you sure you need so many? I don't think there are any tests there that require more than two.

In D14547#305375, @asomers wrote:

The fibs must actually exist. Set net.fibs=6 in /boot/loader.conf and reboot. Also, are you sure you need so many? I don't think there are any tests there that require more than two.

I have done that. The problem is in kyua somewhere:

:
# sysctl net.fibs
net.fibs: 6
# kyua -c /usr/local/etc/kyua/kyua.conf test
fibs_test:arpresolve_checks_interface_fib -> skipped: Required configuration property 'fibs' not defined [0.002s]
fibs_test:default_route_with_multiple_fibs_on_same_subnet -> skipped: Required configuration property 'fibs' not defined [0.002s]
(and so on)
# cat /usr/local/etc/kyua/kyua.conf
-- $FreeBSD: head/devel/kyua/files/kyua.conf.in 364865 2014-08-14 20:21:56Z jmmv $
--
-- System-wide configuration file for kyua(1). See kyua.conf(5) for details
-- on the syntax.
--

syntax(2)


-- User to drop privileges to when invoking kyua(1) as root and a test case
-- requests to be run with non-root permissions.
unprivileged_user = 'tests'

-- An example to set a configuration property specific to FreeBSD.
--test_suites.FreeBSD.fstype = 'ffs'

test_suites.fibs_test.fibs = '1 2 3 4 5'

In D14547#305376, @rstone wrote:

In D14547#305375, @asomers wrote:

The fibs must actually exist. Set net.fibs=6 in /boot/loader.conf and reboot. Also, are you sure you need so many? I don't think there are any tests there that require more than two.

I have done that. The problem is in kyua somewhere:

:
# sysctl net.fibs
net.fibs: 6
# kyua -c /usr/local/etc/kyua/kyua.conf test
fibs_test:arpresolve_checks_interface_fib -> skipped: Required configuration property 'fibs' not defined [0.002s]
fibs_test:default_route_with_multiple_fibs_on_same_subnet -> skipped: Required configuration property 'fibs' not defined [0.002s]
(and so on)
# cat /usr/local/etc/kyua/kyua.conf
-- $FreeBSD: head/devel/kyua/files/kyua.conf.in 364865 2014-08-14 20:21:56Z jmmv $
--
-- System-wide configuration file for kyua(1). See kyua.conf(5) for details
-- on the syntax.
--

syntax(2)


-- User to drop privileges to when invoking kyua(1) as root and a test case
-- requests to be run with non-root permissions.
unprivileged_user = 'tests'

-- An example to set a configuration property specific to FreeBSD.
--test_suites.FreeBSD.fstype = 'ffs'

test_suites.fibs_test.fibs = '1 2 3 4 5'

Oh, I see. You specified the test suite wrong. You're confusing it with the test program. The correct line should be:

test_suites.FreeBSD.fibs = '1 2 3 4 5'

In D14547#305139, @eugen_grosbein.net wrote:

And you do you insist in doing things manually while moving IP (first route change, then reassign IP) instead of just allowing system do all housekeeping for you by reassigning IP first?

The route migration and the IP migration happened as a part of different failover processes IIRC. The subnet route migration happened in response to the route's interface losing link. The IP migration happened later in response to a different event (I can't remember off-hand what prompted it)

In any case, even if the use-case is weird, the current behaviour is clearly buggy and needs to be fixed in some way.

In D14547#305425, @rstone wrote:

In any case, even if the use-case is weird, the current behaviour is clearly buggy and needs to be fixed in some way.

Well, I guess so. But original configuration may be supposed to be "buggy" too (read: not supported). What if you use /16 mask for single interface only and assign IP from the same prefix to other interfaces using /32 mask?

In D14547#305453, @eugen_grosbein.net wrote:

In D14547#305425, @rstone wrote:

In any case, even if the use-case is weird, the current behaviour is clearly buggy and needs to be fixed in some way.

Well, I guess so. But original configuration may be supposed to be "buggy" too (read: not supported). What if you use /16 mask for single interface only and assign IP from the same prefix to other interfaces using /32 mask?

In that configuration I can't move the subnet route from one address to the other, which is exactly the behaviour that my customer expects.

In D14547#306213, @rstone wrote:

Well, I guess so. But original configuration may be supposed to be "buggy" too (read: not supported). What if you use /16 mask for single interface only and assign IP from the same prefix to other interfaces using /32 mask?

In that configuration I can't move the subnet route from one address to the other, which is exactly the behaviour that my customer expects.

Care to elaborate what are you trying to achieve by moving subnet route between interfaces?

In D14547#306224, @eugen_grosbein.net wrote:

Care to elaborate what are you trying to achieve by moving subnet route between interfaces?

My customer is using it as a lame form of failover. If the interface with address A goes down, we can fail over the subnet route to address B (on a different interface) and address B remains functional (address A remains down, of course. I did same it was a lame form of failover).

I have no ability to push back on the customer on this point. Their position is that this configuration was supported on previous versions and therefore must remain supported.

In D14547#306225, @rstone wrote:

In D14547#306224, @eugen_grosbein.net wrote:

Care to elaborate what are you trying to achieve by moving subnet route between interfaces?

My customer is using it as a lame form of failover. If the interface with address A goes down, we can fail over the subnet route to address B (on a different interface) and address B remains functional (address A remains down, of course. I did same it was a lame form of failover).

I have no ability to push back on the customer on this point. Their position is that this configuration was supported on previous versions and therefore must remain supported.

In which "previous version" was it "supported"?

Also, why don't you just use "ifconfig em0 x.y.z.A/24 -alias; ifconfig em1 x.y.z.B/24 alias" to perform the switch? It should just work.

I don't think that this conversation is going to go anywhere productive. Do you have specific technical objection to raise about this change?

In D14547#306227, @rstone wrote:

I don't think that this conversation is going to go anywhere productive.

Why? I'm trying to understand if such non-trivial amount of kernel code changes is justified by the problem.
Perhaps, there is simplier way to solve original problem.

Do you have specific technical objection to raise about this change?

I thought it should not be supported to assign same prefix to different network interfaces (for example, Cisco IOS won't allow that) but I might be wrong. You have a chance to describe real world setup when that's needed and cannot be solved with more consistent configuration.

In D14547#306226, @eugen_grosbein.net wrote:

Also, why don't you just use "ifconfig em0 x.y.z.A/24 -alias; ifconfig em1 x.y.z.B/24 alias" to perform the switch? It should just work.

This has a short timewindow, during which packets will be rejected. On high-traffic ports this will cause connection drops or losses.

In D14547#306289, @pi wrote:

In D14547#306226, @eugen_grosbein.net wrote:

Also, why don't you just use "ifconfig em0 x.y.z.A/24 -alias; ifconfig em1 x.y.z.B/24 alias" to perform the switch? It should just work.

This has a short timewindow, during which packets will be rejected. On high-traffic ports this will cause connection drops or losses.

We are talking about switch after link failure detected, aren't we? Drops and losses have already occured then, and for much longer period, so some extra does not really change the picture.

In D14547#306226, @eugen_grosbein.net wrote:

In D14547#306225, @rstone wrote:

In D14547#306224, @eugen_grosbein.net wrote:

Care to elaborate what are you trying to achieve by moving subnet route between interfaces?

My customer is using it as a lame form of failover. If the interface with address A goes down, we can fail over the subnet route to address B (on a different interface) and address B remains functional (address A remains down, of course. I did same it was a lame form of failover).

I have no ability to push back on the customer on this point. Their position is that this configuration was supported on previous versions and therefore must remain supported.

In which "previous version" was it "supported"?

This worked on older versions of our product, which were based roughly on FreeBSD 7.x

Also, why don't you just use "ifconfig em0 x.y.z.A/24 -alias; ifconfig em1 x.y.z.B/24 alias" to perform the switch? It should just work.

There is another feature of our product that does this (the configuration is managed by a daemon, not manually), but the customer has rejected it. The point is that their existing configuration used to provide failover, and now it does not, so that is a regression.