Landed in a different fashion.

LGTM.
Good old days.

In D37971#863583, @ngie wrote:

I hate to be a wet blanket, but a lot of the code seems to go against design decisions made in pytest around fixtures, extendability, etc.

First of all, thank you for the feedback.
Some of the things intentionally or unintentionally are done in ATF way, which does not match pytest. If you could share a bit more detailed feedback, I'd try to make the framework changes to take it closer to the pytest.

In particular, just glancing at this commit, it seems to rely on J-Unit-like structure (used in unittest), which is not strictly adhered to in pytest.

Currently, the primary use case of python integration is development of the tests with the relatively complex setup/teardown procedures (e.g. custom vnet-based topologies). That is the part I feel is the least covered by the shell or c-based tests. Any suggestions on doing it in a more pytest way?

FWIW, I honestly think integration should be the other way around: ATF should integrate into pytest, not pytest should integrate into ATF. If things were done in that manner and we used the JUnit output format (supported natively in pytest), we could move away from Kyua to a framework that is less bespoke, has a ton less boilerplate than ATF, has better developer and user experience, and has better opensource mindshare than ATF/Kyua.
The main value ATF/Kyua provides (IMHO) is the ability to integrate in tests from NetBSD and a format to express legacy tests in, which gave FreeBSD a great head start in terms of CI/testability. Other than that, it's kind of a kludgy framework.

I agree with the ATF/Kyua assessment - I don't like the API or (lack of) functionality either. I agree that pytest can be a good candidate for the core testing framework, provided there is a consensus on that and the migration/support engineering resources are secured. I don't think we're here yet. Currently, shell and C test files correspond to around 90% of the all test files, with similar figures across kernel and userland. Out of the remaining python files, only 10% represent pytest tests; the rest are wrapped with the shell scripts.
Personally, I think that the most beneficial resource application at the moment is improving the python support & reducing the bar to adding python tests to drive adoption. I'd also like to note that the current state of the things is not contrary to the "pytest instead of ATF" idea. It should be notably easier to convert the tests to the "original" pytest format once desired than migrating the shell script mess.

In D38015#863418, @ae wrote:

we can't recover why would we ever have identical 4-tuples in the hash

One example JFYI, IPv6 link-local addresses can be the same for different interfaces. I think this isn't the case for now, since we keep them in embedded form, but it would work in proper way in the future and we should be able to handle several the same 4-tuple but with different scope zone id.

That's a good point. There indeed shouldn't be a problem with an embedded form. When we eliminate the embedded form, that shouldn't be a problem either: inpcbs assume unicast addresses and the only scope in unicast is link-local.
There is already the ifnet argument in the in6_pcblookup_hash_locked(), so converting it to the "normal" form will require and additional condition, checking inp scopeid with the ifp scopeid.

In D22012#862511, @roy_marples.name wrote:

In D22012#862327, @melifaro wrote:

Thank you for the clarification! Just to check: the relevant change Is commit @ 30 Aug, right? This looks like a regression, but the commit description looks a bit vague. I wasn't able to find any bug report or message in the ML. Did you / @roy_marples.name reported it? If that's a regression, maybe something can be change on the kernel side, so dhcp 10 is not exactly a blocker.

It is a regression in FreeBSD-14. I updated my VM to the latest FreeBSD-current and the issue is still there.
I didn't report it to any ML as I'm happier with the new code in dhcpcd-10.
In a nutshell it seems like the dhcpcd privileged process seems to be under capsicum control for *some* operations but not others and some in kernel message buffers seem to shrink as writev reports message too long.
The same code works fine on FreeBSD-13.

Happy to discuss this elsewhere?

Sure! I guess the best-suited platform for that would be our bugzilla - as I guess we need to get sort of a repro (or at least more detailed description) and go from here. Do you mind submitting something?

In D37902#861660, @melifaro wrote:

In D37902#861463, @asomers wrote:

A "test_case_cleanup" method would work. But it would be more Pythonic if it were a teardown method. Would that be possible?

Whoops. Sorry, I missed that comment. It landed as "cleanup_test_case" to mimic the generic "cleanup" method that already exists. That method was modelled after kuya.
I tried to make the framework abstracted from kuya/atf, but that part indeed slipped through the cracks. I'll come up with a separate renaming review next week if the time permits.
Meanwhile the per-test cleanups landed, with an example provided in this diff. Hope it does work for you :-)

Raised D37971 to rename the method.

.

In D22012#862257, @woodsb02 wrote:

Sorry for not providing an update on this sooner.
The summary is I was waiting on the next release of dhcpcd before proceeding with importing it into base.
The reason was a regression related to capsicum was noticed with FreeBSD-14 starting in August 2022 where the privilged process could no longer write to the routing table. This issue isn't occurring on FreeBSD-13, nor FreeBSD-14 from 2021.

Thank you for the clarification! Just to check: the relevant change Is commit @ 30 Aug, right? This looks like a regression, but the commit description looks a bit vague. I wasn't able to find any bug report or message in the ML. Did you / @roy_marples.name reported it? If that's a regression, maybe something can be change on the kernel side, so dhcp 10 is not exactly a blocker.

@roy_marples.name found and fixed the issue in dhcpcd, but only in what will become dhcpcd-10 (it's too invasive for dhcpcd-9 at this point which is the stable branch).
The dhcpcd-10 has been delayed a bit, since Roy has a few health issues that are his priority right now.
That said, I'm still hoping this can be merged into FreeBSD-14 before the code slush begins on 25th April 2023.

In D37923#861728, @jlduran_gmail.com wrote:

In D37923#861709, @asomers wrote:

It also means that an unprivileged user won't be able to simply type "kyua test ..." and expect the unprivileged test cases to work.

I'll retry everything tomorrow, but in this regard, things are working as expected.
Unprivileged users:

• Can run unprivileged test cases
• root test cases are skipped
• VNET test cases are skipped entirely (because an unprivileged user cannot create jails)

What is not working for me, is:

class TestVnetSimple(SingleVnetTestTemplate):
    @pytest.mark.parametrize(
        "user_tuple",
        [
            pytest.param(
                ["kldload if_epair", "module already loaded or in kernel"],
                marks=pytest.mark.require_user("root"),
                id="root",
            ),
            pytest.param(
                ["kldload if_epair", "Operation not permitted"],
                marks=pytest.mark.require_user("unprivileged"),
                id="unprivileged",
            ),
        ],
    )
    def test_parametrize_require_user(self, user_tuple):
        command, output = user_tuple
        assert output in subprocess.getoutput(command)

Specifically the test_parametrize_require_user[root] parameter. It gets an Operation not permitted error, as if unprivileged.
If you s/SingleVnetTestTemplate/BaseTest everything works as expected.

Thank you for testing & catching the issue!
I'll check this case and update the diff.

In D37923#861709, @asomers wrote:

I find this confusing. It's weird that Kyua handles some of the test case isolation, and Python handles other parts. It also means that an unprivileged user won't be able to simply type "kyua test ..." and expect the unprivileged test cases to work. It seems like either the vnet stuff needs to be moved up into Kyua itself, or the privilege dropping needs to be moved down out of the runner and into the test class. In the latter case, the test case would still have a "require.user root", but could use some other decorator or just the class setup to drop privileges.

Yep, that's a good observation! I'm happy to work towards making more "native" and less confusing.

Fix skipped tests.

I'm going to commit it as is for now and address the remaining issue in the follow-up commits.

In D37902#861463, @asomers wrote:

A "test_case_cleanup" method would work. But it would be more Pythonic if it were a teardown method. Would that be possible?

Whoops. Sorry, I missed that comment. It landed as "cleanup_test_case" to mimic the generic "cleanup" method that already exists. That method was modelled after kuya.
I tried to make the framework abstracted from kuya/atf, but that part indeed slipped through the cracks. I'll come up with a separate renaming review next week if the time permits.
Meanwhile the per-test cleanups landed, with an example provided in this diff. Hope it does work for you :-)

Uncomment unprivileged-user
Add example with the test cleanup

In D37902#861466, @jlduran_gmail.com wrote:

In D37902#861460, @melifaro wrote:

In D37902#861452, @asomers wrote:

This looks great. I especially like the ability to parameterize test cases, which isn't easy with atf-sh-api or atf-c-api. What about cleanup? I think atf-python tests need to do that differently, right? It would be good to add an example for that.

Thank you!
Yes, the cleanup is different from the native pytest approach. The initial driving usecase was to enable cheap VNET-run tests, where the cleanup procedures are fixed - you need to remove the jail & interfaces. That part is done automatically by the framework.

While on this topic, the cleanup for the native pytest approach is done the "old" way. I have encountered a few races, where an epair interface remains after a massive test failure. In other words, I was contemplating the possibility of adapting 80fc25025ffcb0d369fc0b6d4d272ad6fd3f53c3, which destroys the epair assigned to the vnet from inside the jail. Obviously, this has nothing to do with this introductory tutorial.

Yep, thanks for reminding! There is a known problem resulting from a corner case in the kernel delayed object reclamation model. I promised glebius@ to alter the mechanism as a workaround, but haven't done it yet. That's something that indeed needs to happen. I'll fix this in a day or two.

In D37902#861452, @asomers wrote:

This looks great. I especially like the ability to parameterize test cases, which isn't easy with atf-sh-api or atf-c-api. What about cleanup? I think atf-python tests need to do that differently, right? It would be good to add an example for that.

Thank you!
Yes, the cleanup is different from the native pytest approach. The initial driving usecase was to enable cheap VNET-run tests, where the cleanup procedures are fixed - you need to remove the jail & interfaces. That part is done automatically by the framework.
I'll add a hook that'll check if the test_name_cleanup() method exists and calls it to perform the cleanup, along with the example here. Does it look reasonable to you?

In D37902#861235, @jlduran_gmail.com wrote:

Couple more nits.
New run (this time as expected):
https://cirrus-ci.com/task/5756291799842816?logs=test_examples
I'll eventually add a simple example that consumes Scapy directly (instead of doing it via a shell script).

That would be awesome!
There are quite a bunch of scapy-based tests, that are indeed mostly shell-based.

Address comments.

In D37871#861097, @zlei wrote:

@melifaro @kp
I'd like to combine this (D37871) with D37872 , as they're simple and are same type of mistakes,
although cause different symptoms.

How shall I do ? Commit with two Differential Revision , or combine them into a new differential ?

I'd commit both separately, as it's easier to track and to reason about.

Thank you for the patch!

In D37894#860943, @jlduran_gmail.com wrote:

@melifaro, do you take GitHub pull requests? This review was originally two commits, one for the style fixes, another for the actual "fix".

Just in case - it's possible to create a review with multiple commits.

I believe it is safe to leave always with -vv?, as it matches the other ATF output.

I'm not sure I have the notifications configured, but let's try.

Thank you!
Will commit tomorrow

It would be nice to have the test script similar to the one in the testing section in our auto tests/.
Jails are complex beasts, so increasing the covered code scope would help detect regressions faster

In D37876#860550, @jlduran_gmail.com wrote:

In D37876#860539, @melifaro wrote:

Any thoughts of using native python interface instead of the shell wrappers?

That looks nice!, but do you mean using pytest instead of atf-sh(3), or instead of using subprocess.run to execute shell commands?

Pytest. There is a wrapper that automatically wraps pytest tests into kyua interface, allowing to run them in a standard system test way via kuya.

In D37874#860475, @zlei wrote:

This is a request by @melifaro in D37732 .

In D37732#858183, @melifaro wrote:

LGTM!
Do you think it would be possible to convert PR_IP to be static inline function, so that kind of erros are spotted by the compiler?

I'll take a look at that. Smart compilers help a lot ;)

Thank you!

Thank you for working on this!
Any thoughts of using native python interface instead of the shell wrappers?
Just in case, the examples can be found in tests/sys/net/routing/test_rtsock_multipath.py or tests/sys/netlink/test_rtnl_iface.py

If there are no objections, I will commit it on Monday, December 26.

In D36529#858749, @gbe wrote:

@melifaro thanks for the differential. I remember darkly, that there was PR a few month ago, that FreeBSD doesn't support "route print" like most operating systems do. Would it be possible to integrate that feature of that differential and netlink?

Generally I'm not opposed to the idea. There are a number of things to be sorted out, though. For example, what should be the column format, should it be machine-readable (as currently /sbin/route does not have any lib depenencies) and so on.
I guess the easiest would be to do execve /usr/sbin/netstat -[46]rn on route print inside route(8) if so desired. That should be pretty straight-forwarded to implement. I'm not sure I'll have cycles to do this ATM, as I'm full-hands in Netlink conversion. I'm happy to review such change if somebody else comes up with the patch.

In D37784#860268, @jhibbits wrote:

In D37784#860171, @melifaro wrote:

In D37784#860073, @jhibbits wrote:

In D37784#860049, @melifaro wrote:

Q: What blockers prevent moving the accessors outside of if_var.h?

Nothing, really. Just doing this piecemeal. I noticed in some of my conversions that keeping the if_t typedef in if_var.h necessitated pulling it in where it wasn't needed beyond getting if_t, so pulled it out. The current users of struct ifnet beyond a forward declaration already include if_var.h so there's no gain or loss there.

Got it, make sense. Do you have any tentative timeline in mind for the stage when these accessors can be moved outside if_var.h ?

Once all the drivers are converted over, so hoping to have it all done by end of January. I've got a work in progress, but tons of build failures. Seems drivers love to include in_var.h, in6_var.h, etc. Those will be hard to pull out, so will take some time.

Thank you for the clarification! No rush, just wanted to ensure we are on the same page w.r.t. reducing if_var.h header usage.

In D37757#860215, @melifaro wrote:

In D37757#860208, @zlei wrote:

struct route_cache {
  union {
   struct route ro;
   struct route_in ro_in;
   struct route_in6 ro_in6
  }
 struct mtx lock;
 struct rib_subscription *rs;
}

Does the lock protect the read part ?
If yes then it is apparent a contention when concurrent network IO occurs.

No, the update fucnction, which uses the lock, is inly called when it is needed to update the cache.
The read path is not affected.

Let me clarify it a bit further. The idea is change the parts of the stack that currently update nhop/lle cache in the struct route directly to use the new callback.
The hot/data/read part remains unaffected - no atomic or other ops are added.
When route cache is invalidated (route_cache_invalidate() clearing the pointer) datapath may read old nhop/lle pointer. It is safe, as both nhop and lles use epoch-backed garbage collection.

In D37757#860208, @zlei wrote:

struct route_cache {
  union {
   struct route ro;
   struct route_in ro_in;
   struct route_in6 ro_in6
  }
 struct mtx lock;
 struct rib_subscription *rs;
}

Does the lock protect the read part ?
If yes then it is apparent a contention when concurrent network IO occurs.

No, the update fucnction, which uses the lock, is inly called when it is needed to update the cache.
The read path is not affected.

In D37736#859376, @bapt wrote:

I manage to convert my code to use snl.h its ok for me.

Awesome!

Quick remarks, I plan to add snl_genl.h for genl specific case, would that be ok for you?

Of course.

Doesn't the routing specific part needs to be moved to a snl_route.h?

Yep, that makes sense. Moved to netlink_snl_route.h.