Expect *localsctp* to fail

[ngie]

Summary

The tests require ncat --sctp, which doesn't exist on FreeBSD, so expect
the tests to fail.

This will fix 2 deterministic failures that occur in the dtrace_test job..

MFC after: 2 weeks

Test Plan

Verified that the tests are expected to fail:

 90 atf_test_case tst_ipv4localsctp_ksh
 91 tst_ipv4localsctp_ksh_head()
 92 {
 93     atf_set 'descr' 'DTrace test common/ip/tst.ipv4localsctp.ksh'
 94 }   
 95 tst_ipv4localsctp_ksh_body()
 96 {
 97     atf_expect_fail "test is known to fail"
 98     atf_check -s exit:0 -o empty -e empty \
 99         "$(atf_get_srcdir)/../../dtest" "$(atf_get_srcdir)/tst.ipv4localsctp.ksh"
100 }
101 atf_test_case tst_localsctpstate_ksh
102 tst_localsctpstate_ksh_head()
103 {
104     atf_set 'descr' 'DTrace test common/ip/tst.localsctpstate.ksh'
105 }   
106 tst_localsctpstate_ksh_body()
107 {
108     atf_expect_fail "test is known to fail"
109     atf_check -s exit:0 -o empty -e empty \
110         "$(atf_get_srcdir)/../../dtest" "$(atf_get_srcdir)/tst.localsctpstate.ksh"
111 }

[lwhsu]

It looks these tests require ncat but not netcat. I'll check if the test image has this port installed.
Also, I think the test case requires external tools to execute would be better to be marked as SKIP (with a message states the lacking requirements) instead of being marked EXFAIL.

[lwhsu]

OK, I just check we have nmap in the test vm image and it does the work: https://github.com/freebsd/freebsd-ci/commit/689e19c8c71e1892f476c9ef9392f6b443d0ed15
The other failures are due to some perl stuff which I haven't had time on it: https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/lastCompletedBuild/testReport/common.ip/t_dtrace_contrib/tst_ipv4localsctp_ksh/

[lwhsu]

I'd prefer let's fix these test cases.

[tuexen]

In D20017#430241, @lwhsu wrote:

OK, I just check we have nmap in the test vm image and it does the work: https://github.com/freebsd/freebsd-ci/commit/689e19c8c71e1892f476c9ef9392f6b443d0ed15
The other failures are due to some perl stuff which I haven't had time on it: https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/lastCompletedBuild/testReport/common.ip/t_dtrace_contrib/tst_ipv4localsctp_ksh/

I'm debugging this now. Seems to be a kernel issue. Will let you know once I have understood and fixed the issue.

Is using ncat and issue?

[ngie]

In D20017#430242, @lwhsu wrote:

I'd prefer let's fix these test cases.

1. FreeBSD lacks ncat:

$ which nc
/usr/bin/nc
$ which ncat || echo not found
not found

2. nc under FreeBSD doesn't support SCTP.

These tests need to be written to implement an SCTP server of some kind. The perl APIs might be the right way to do this.

[ngie]

In D20017#430585, @tuexen wrote:

In D20017#430241, @lwhsu wrote:

OK, I just check we have nmap in the test vm image and it does the work: https://github.com/freebsd/freebsd-ci/commit/689e19c8c71e1892f476c9ef9392f6b443d0ed15
The other failures are due to some perl stuff which I haven't had time on it: https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/lastCompletedBuild/testReport/common.ip/t_dtrace_contrib/tst_ipv4localsctp_ksh/

I'm debugging this now. Seems to be a kernel issue. Will let you know once I have understood and fixed the issue.

Is using ncat and issue?

ncat isn't available on FreeBSD. The fact that there isn't a listening ncat service will mean that the test will not properly setup the necessary preconditions in order to test out the functionality.

[tuexen]

In D20017#430588, @ngie wrote:

In D20017#430585, @tuexen wrote:

In D20017#430241, @lwhsu wrote:

OK, I just check we have nmap in the test vm image and it does the work: https://github.com/freebsd/freebsd-ci/commit/689e19c8c71e1892f476c9ef9392f6b443d0ed15
The other failures are due to some perl stuff which I haven't had time on it: https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/lastCompletedBuild/testReport/common.ip/t_dtrace_contrib/tst_ipv4localsctp_ksh/

I'm debugging this now. Seems to be a kernel issue. Will let you know once I have understood and fixed the issue.

Is using ncat and issue?

ncat isn't available on FreeBSD. The fact that there isn't a listening ncat service will mean that the test will not properly setup the necessary preconditions in order to test out the functionality.

ncat is available in the ports tree. Is that an issue? The tests also use a ksh or ksh93, which is also available only in the ports tree...

[lwhsu]

ncat is from nmap port and has nothing to do with nc, their names are similar but they are different things.

It's fine to install tests requirements from ports, after all, kyua is installed from ports and it's an essential tool for running the tests.

For marking the tests, what I was saying is "skip the tests (and not report failure) when there is no nmap (ncat) installed, and have a message to users to install nmap."

[ngie]

In D20017#430611, @lwhsu wrote:

ncat is from nmap port and has nothing to do with nc, their names are similar but they are different things.

It's fine to install tests requirements from ports, after all, kyua is installed from ports and it's an essential tool for running the tests.

For marking the tests, what I was saying is "skip the tests (and not report failure) when there is no nmap (ncat) installed, and have a message to users to install nmap."

Oh, ok. I didn’t realize that nmap provided that capability.

I was trying to repro this and I didn’t have it installed beforehand; I’ll try it and see if it works.

As for checking for the programs, I’ll see about adding this support to the test generator; I don’t think it has require.progs support, yet.

[tuexen]

OK, I have figured out why the tests are failing. The problem is the discard server:

ncat --sctp --listen $local $sctpport &

Once an association is established, it figures out via select that stdin is readable, it reads 0 bytes from stdin and terminates.
For TCP one can get rid of that behaviour (using --no-shutdown / --recv-only), but none of this works for SCTP.

The perl connection setup figures out that the connection it tries to establish gets closed and reports an error.

So the best seems to be to use a perl based discard server... This avoids the ncat dependency and fixes the tests. Will open
a review once I have a patch ready. Then the tests should pass...

[ngie]

Abandoning review based on in-depth analysis from @tuexen.

[tuexen]

In D20017#430777, @ngie wrote:

Abandoning review based on in-depth analysis from @tuexen.

A proposed fix is in D20086.