Address suggestions:

• Change the warning message

Address suggestions:
In this first iteration, simply duplicate the code from the final statistics section into SIGINFO's.

• Avoid a potential square root of a negative number calculation (based on an alternative to D38480, this change should not part of this revision)

I have a branch where it unifies the code into a single summary() function, however it changes a lot more of the code that I would like/have tests for at the moment. Besides, the code ends up being almost identical to ping6.c's. Let's leave the unification for later in the ping/ping6 series.
A few cosmetic things discovered from this exercise are:

• ping's SIGINFO prints out to stderr (ping6 to stdout)
• ping/ping6 should just become ping when displayed to the user
• ping uses the expression "somebody's printing up packets!", I prefer ping6's "somebody's duplicating packets!" when a duplicate packet is received
• ping calls the population standard deviation "stddev", while ping6 calls it "std-dev". I'm OK either way, but it should be the same for both

• Remove pr_retip()

Address comments:

• Add single NOP case
• Remove unneeded int cast
• There is no need to decrease hlen, as cp is advancing to the next octet, (single NOP case added as proof). However, let's be pedantic (and true to the original implementation), and decrease it.

Update the "reference" implementation with the same tests added to the proposed fixes:

• Add max inner packet IHL without payload test
• Add IHL too short test
• Add quoted data too short test
• Add inner IHL too short test
• Add inner packet too short test

Yes, please!
In the meantime, I'll keep updating this "BASE" revision.
Changes:

• Add the ability doctor IHL fields
• Add the ability to remove inner packet load

Thank you @pauamma! Those are great suggestions. I took man page changes from pf.conf.5, so I consider it to be my "upstream".
How about a change there, and then pull in those changes here?

As a courtesy, tag @kp, as this code was mostly stolen from him.

In D38485#875773, @asomers wrote:

I like it. And yes I think you should add stddev to match the regular program output at termination.

In D38483#875772, @asomers wrote:

Philosophically, SIGQUIT is different than SIGINT. It's more like "I the programmer have determined that this program is behaving badly, and would like a core dump". But I could go either way, if people disagree with me here.

• Add a missing extra white space

The output was originally taken from a branch that has code that trims these extra spaces. Let's leave those cosmetic changes for last.

@markj already made a few suggestions by email. Let's move the discussion here.

In D38431#875438, @asomers wrote:

I don't understand why you need this revision. For what will it serve as a reference implementation?

• Remove WIP commit (it serves no purpose)
• Add tests to pr_pack() with options

The function pr_pack() prints out a packet. If the IP packet contains options, these are printed as well.
Commit 46d7b45a267b3d78c5054b210ff7b6c55bfca42b introduced an integer overflow bug, by changing hlen from int to uint8_t; while the fix is simple, let's test everything to avoid introducing regressions.

• Do not shadow opts var
• Disable net.inet.ip.process_options for RR options as well (this was discovered by randomizing the tests, as this sysctl was set by the previous tests)

Add tests for TCP/UDP unreachable packets:
Before tcpdump existed, ping provided some sort of packet sniffing capabilities. After ef9e6dc7eebe9830511602904d3ef5218d964080 this functionality was almost removed, to only display packets sent by us.
The TCP/UDP code remained, I do not know how to trigger it naturally, by doctoring the packet, however, we are able to test it.

I do knot fully understand what is the purpose of this script, if it is what I think it is, then this seems like a nice addition.
Anyhow, here are a few basic ShellCheck suggestions.

Abandoned in favor of D38053, which uses a native Python testing infrastructure. Thank you!

Updates:

• Use ToolsHelper.set_sysctl
• Use type hints
• Skip failing tests
• Remove tests of features not yet implemented

@ngie Thank you for your suggestions, I'll try to implement them!
I also plan to update this file, hopefully this week, I'll remove/skip the tests for D37930 (as it hasn't landed yet), and start submitting the proposed fixes for the utility itself. I don't think all of them will be able to make it to the upcoming release, so there isn't really a time constraint, but I would like to start receiving feedback on that as well. Admittedly, having nice tests usually facilitates it!

In D38053#867921, @melifaro wrote:

3. An atf_get_srcdir-equivalent to eventually read files relative to the source directory, however I think the idea is to disassociate from ATF later on.

This one slipped through the cracks, thanks for reminding! Will add in a day or two.

I thought of passing it via wrapper initially, but ended up with a bit more natural way - getting it from pytest itself.
I committed an example here.

In D38053#867961, @melifaro wrote:

4. In the meantime the kernel delayed object reclamation model issue is fixed, maybe adapt 80fc25025ffcb0d369fc0b6d4d272ad6fd3f53c3 ?

Committed in 20ea7f26e413 .

In D38053#867921, @melifaro wrote:

3. An atf_get_srcdir-equivalent to eventually read files relative to the source directory, however I think the idea is to disassociate from ATF later on.

This one slipped through the cracks, thanks for reminding! Will add in a day or two.

I thought of passing it via wrapper initially, but ended up with a bit more natural way - getting it from pytest itself.
I committed an example here.

Ha, very nice! I tend to prefer the f-string format (i.e., print(f" {k}: {v}"), is there a preference in style?

In D38053#867487, @ngie wrote:

In D38053#864580, @asomers wrote:

This looks like a good start. I'm not a fan of the methodize stuff, though. I think it makes the tests harder to read and harder to search. Is there a way to parameterize the tests without using methodize, even if it becomes slightly more verbose?

If you’re referring to “pytest.marks.parametrize”, it supports an optional id= parameter which better describes tests for humans.

In D38053#866669, @melifaro wrote:

In D38053#865570, @jlduran_gmail.com wrote:

Thank you! I'll wait for @asomers comments to update the revision.

I'll document here my small - not really important - wishlist:

1. Optional randomization of the test order, maybe something similar to pytest-random-order.

Nice one, can probably work out-of-the box, but the scope will be a single file, due to the way of ATF<>pytest interaction is implemented.

2. When debugging atf-python tests, the output could be a little bit less busy, maybe pytest_terminal_summary et al. instead of print() could be one answer.

I get the intent and I agree. Maybe you could come up with an example (or even a diff) on how do you see it?

Sure! Once all ping tests are added, I should be more acquainted with the testing framework and propose a revision.

3. An atf_get_srcdir-equivalent to eventually read files relative to the source directory, however I think the idea is to disassociate from ATF later on.

This one slipped through the cracks, thanks for reminding! Will add in a day or two.

4. In the meantime the kernel delayed object reclamation model issue is fixed, maybe adapt 80fc25025ffcb0d369fc0b6d4d272ad6fd3f53c3 ?

Yep, been on my list for quite some time, I'll do the change in a day or two.

Great!

Thank you! I'll wait for @asomers comments to update the revision.

Closed by 3099c99b392dd0ca738f9c16db7c1280f31d74a5.
Thank you!

In D38053#865288, @melifaro wrote:

In D38053#865132, @jlduran_gmail.com wrote:

Here are the last iteration changes:

• Use a dictionary for expectations — this keeps both ping and pinger tests inline. Comparing the expected with the actual subprocess.CompletedProcess was also not feasible. It also was suggested initially by @melifaro.

One thing I noticed is that atf-python tests are slower than atf-sh, this was somewhat expected, but when all tests run, the total time builds up. I believe the gains are really from the development perspective.

Q: what's the runtime? e.g. median time reported by kyua test?

Here are the last iteration changes:

Updates:

• Simple one-line ping parameter expectations are subprocess.CompletedProcesses. No need to have a custom class (ExpectedProcess) that does essentially the same thing.
• Incorporate the code from D37876. The pinger utility (now a function) returns a subprocess.CompletedProcess as well. Parametrization and cleanup are next.

In D38053#864674, @melifaro wrote:

Thank you for addressing the comments!
I’d still prefer to have the ids embedded explicitly via pytest.param, but I don’t insist on doing it.

Generally there may be somewhat conflicting ideas on what’s the “best” approach here, but personally I don’t think it’s a big deal here. This is a test code, it already is reasonably simple and allow for easy extension or debugging - that’s “good enough” and that’s what matters most (to me).

Address more suggestions:

• Embed the ids explicitly via pytest.param().

Address some of the suggestions:

Sorry, the tests are failing because of a silly mistake from my part: (kldload) kernel modules cannot be loaded from inside the jail, even if the user is root.
Besides the trivial comments that looks like debugging leftovers, this revision works for me.

In D38053#864580, @asomers wrote:

This looks like a good start. I'm not a fan of the methodize stuff, though. I think it makes the tests harder to read and harder to search. Is there a way to parameterize the tests without using methodize, even if it becomes slightly more verbose?

In D37930#861990, @jlduran_gmail.com wrote:

I believe the message "Request timeout for icmp_seq..." should appear when -A beeps.
Or maybe I am missing something?

😞 I believe ping -Ac1 192.0.2.1 should beep once, so there's a bug in -A?
Regardless, it would be tangential to this differential.

Thank you!
I am sorry to insist on the same issue, but I believe the message "Request timeout for icmp_seq..." should appear when -A beeps.
This seems to be the case for Darwin/macOS.
As stated in my first comment, I don't see any reason why not to use the code from Apple, it is simpler and achieves the desired functionality (without the case for -l). Or maybe I am missing something?

Also failing this test:

% ping -c1 192.0.2.1
PING 192.0.2.1 (192.0.2.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 0

I like this addition.

Here are the basic tests used:
https://github.com/jlduran/freebsd-src/blob/D37923/tests/examples/test_examples.py
And their results, both executed by root and by an unprivileged user named "user":
https://cirrus-ci.com/build/5416736921485312

In D37923#861709, @asomers wrote:

It also means that an unprivileged user won't be able to simply type "kyua test ..." and expect the unprivileged test cases to work.

I'll retry everything tomorrow, but in this regard, things are working as expected.
Unprivileged users:

• Can run unprivileged test cases
• root test cases are skipped
• VNET test cases are skipped entirely (because an unprivileged user cannot create jails)

Sorry about the parametrized test. It is wrong.

In my simplistic tests, this works well (on BaseTest, SingleVnetTestTemplate , and VnetTestTemplate):

@pytest.mark.require_user("root")
def test_root(self):
    assert subprocess.getoutput("id -un") == "root"

• Use the right path for the obsoleted file

@asomers I can drop the jailing part at your request, of course, although I personally prefer to have these tests jailed.
I am also playing with the native python interface, it'll take me a while, since I already had everything set up for atf-sh. I stopped short of getting started with IPv6 tests, and now I'm grateful, as I am currently lured by the possibility of using parameterized pytests.

In D37902#861460, @melifaro wrote:

In D37902#861452, @asomers wrote:

This looks great. I especially like the ability to parameterize test cases, which isn't easy with atf-sh-api or atf-c-api. What about cleanup? I think atf-python tests need to do that differently, right? It would be good to add an example for that.

Thank you!
Yes, the cleanup is different from the native pytest approach. The initial driving usecase was to enable cheap VNET-run tests, where the cleanup procedures are fixed - you need to remove the jail & interfaces. That part is done automatically by the framework.

While on this topic, the cleanup for the native pytest approach is done the "old" way. I have encountered a few races, where an epair interface remains after a massive test failure. In other words, I was contemplating the possibility of adapting 80fc25025ffcb0d369fc0b6d4d272ad6fd3f53c3, which destroys the epair assigned to the vnet from inside the jail. Obviously, this has nothing to do with this introductory tutorial.

Thank you!
From my perspective, this is an excellent intro to testing.

Couple more nits.
New run (this time as expected):
https://cirrus-ci.com/task/5756291799842816?logs=test_examples
I'll eventually add a simple example that consumes Scapy directly (instead of doing it via a shell script).

Thank you!
Certainly, much better than a wiki.
Is this the desired output:
https://cirrus-ci.com/task/6234303340740608 ?

@melifaro, do you take GitHub pull requests? This review was originally two commits, one for the style fixes, another for the actual "fix".
I believe it is safe to leave always with -vv?, as it matches the other ATF output.

• Add an ObsoleteFiles.inc entry (TODO: fix date upon commit)
• Rely on tests/sys/common/vnet.subr for VNET jail and tun interface creation
• Add a vnet_mktun() function to vnet.subr
• Imprison tests (test can now run simultaneously while using the same IP address)
• PEP 8 lint the script using black/flake8 (maximum-line-length 79 characters)

Address first round of suggestions.

In D37876#860561, @asomers wrote:

I assume you have a few extra test cases that you're planning to add to ping_test.sh?

Yes! Absolutely, I currently have more than 10 tests using this script.

And don't forget that you'll need to add injection.py to ObsoleteFiles.mk, or else rename pinger.py to injection.py.

Thank you, I was completely unaware of this step.
There it goes my ping-themed series of tests... I even had the jails named BRL, after the Ballistic Research Lab, :-(
I'll think it over, but it is likely I won't pollute ObsoleteFiles.mk out of sheer vanity and rename the script. However, it can do more than just inject packets.

In D37876#860539, @melifaro wrote:

Any thoughts of using native python interface instead of the shell wrappers?

That looks nice!, but do you mean using pytest instead of atf-sh(3), or instead of using subprocess.run to execute shell commands?
If the latter, given a separation between kernel and userland tests is likely desired, duplication of the tools/helpers is what we should do initially?
For example, instead of:

import subprocess

According to mdoc(7), the section name should be just "ENVIRONMENT".