testing: change python ATF cleanup handlers to be more pythonic

[melifaro]

Summary

[Followup from D37902]

Python ATF interface integrates pytest tests into the test suite.
In pytest, test cleanups are named "teardown", which is different from the ATF "cleanup" routine.
Convert all such method names to be named "teardown" to provide a consistent pytest model.

[ngie]

I hate to be a wet blanket, but a lot of the code seems to go against design decisions made in pytest around fixtures, extendability, etc.
In particular, just glancing at this commit, it seems to rely on J-Unit-like structure (used in unittest), which is not strictly adhered to in pytest.
FWIW, I honestly think integration should be the other way around: ATF should integrate into pytest, not pytest should integrate into ATF. If things were done in that manner and we used the JUnit output format (supported natively in pytest), we could move away from Kyua to a framework that is less bespoke, has a ton less boilerplate than ATF, has better developer and user experience, and has better opensource mindshare than ATF/Kyua.
The main value ATF/Kyua provides (IMHO) is the ability to integrate in tests from NetBSD and a format to express legacy tests in, which gave FreeBSD a great head start in terms of CI/testability. Other than that, it's kind of a kludgy framework.

[asomers]

In D37971#863583, @ngie wrote:

I hate to be a wet blanket, but a lot of the code seems to go against design decisions made in pytest around fixtures, extendability, etc.
In particular, just glancing at this commit, it seems to rely on J-Unit-like structure (used in unittest), which is not strictly adhered to in pytest.
FWIW, I honestly think integration should be the other way around: ATF should integrate into pytest, not pytest should integrate into ATF. If things were done in that manner and we used the JUnit output format (supported natively in pytest), we could move away from Kyua to a framework that is less bespoke, has a ton less boilerplate than ATF, has better developer and user experience, and has better opensource mindshare than ATF/Kyua.
The main value ATF/Kyua provides (IMHO) is the ability to integrate in tests from NetBSD and a format to express legacy tests in, which gave FreeBSD a great head start in terms of CI/testability. Other than that, it's kind of a kludgy framework.

What are you saying? That Kyua should be able to run unmodified PyTest code?

[melifaro]

In D37971#863583, @ngie wrote:

I hate to be a wet blanket, but a lot of the code seems to go against design decisions made in pytest around fixtures, extendability, etc.

First of all, thank you for the feedback.
Some of the things intentionally or unintentionally are done in ATF way, which does not match pytest. If you could share a bit more detailed feedback, I'd try to make the framework changes to take it closer to the pytest.

In particular, just glancing at this commit, it seems to rely on J-Unit-like structure (used in unittest), which is not strictly adhered to in pytest.

Currently, the primary use case of python integration is development of the tests with the relatively complex setup/teardown procedures (e.g. custom vnet-based topologies). That is the part I feel is the least covered by the shell or c-based tests. Any suggestions on doing it in a more pytest way?

FWIW, I honestly think integration should be the other way around: ATF should integrate into pytest, not pytest should integrate into ATF. If things were done in that manner and we used the JUnit output format (supported natively in pytest), we could move away from Kyua to a framework that is less bespoke, has a ton less boilerplate than ATF, has better developer and user experience, and has better opensource mindshare than ATF/Kyua.
The main value ATF/Kyua provides (IMHO) is the ability to integrate in tests from NetBSD and a format to express legacy tests in, which gave FreeBSD a great head start in terms of CI/testability. Other than that, it's kind of a kludgy framework.

I agree with the ATF/Kyua assessment - I don't like the API or (lack of) functionality either. I agree that pytest can be a good candidate for the core testing framework, provided there is a consensus on that and the migration/support engineering resources are secured. I don't think we're here yet. Currently, shell and C test files correspond to around 90% of the all test files, with similar figures across kernel and userland. Out of the remaining python files, only 10% represent pytest tests; the rest are wrapped with the shell scripts.
Personally, I think that the most beneficial resource application at the moment is improving the python support & reducing the bar to adding python tests to drive adoption. I'd also like to note that the current state of the things is not contrary to the "pytest instead of ATF" idea. It should be notably easier to convert the tests to the "original" pytest format once desired than migrating the shell script mess.

What do you think?

P.S. the details of the percentage calculations are below. I know that some of the tests are third-party and not integrated in the framework, but It doesn't look that it change the results significantly.

• Userland+kernel test file stats: find . -type f -name \*.<c|sh|py> -ipath '*/tests/*' | wc -l
• Kernel test file stats: find tests/sys -type f -name \*.<c|sh|py> | wc -l
• Pytest file stats: find tests -name 'test*.py' | wc -l

Results:
kernel: 297(sh), 158 (c), 54(py)
kernel+user: 533(sh), 451(c), 76(py).
Pytest: 6 files

[ngie]

In D37971#863590, @asomers wrote:

In D37971#863583, @ngie wrote:

I hate to be a wet blanket, but a lot of the code seems to go against design decisions made in pytest around fixtures, extendability, etc.
In particular, just glancing at this commit, it seems to rely on J-Unit-like structure (used in unittest), which is not strictly adhered to in pytest.
FWIW, I honestly think integration should be the other way around: ATF should integrate into pytest, not pytest should integrate into ATF. If things were done in that manner and we used the JUnit output format (supported natively in pytest), we could move away from Kyua to a framework that is less bespoke, has a ton less boilerplate than ATF, has better developer and user experience, and has better opensource mindshare than ATF/Kyua.
The main value ATF/Kyua provides (IMHO) is the ability to integrate in tests from NetBSD and a format to express legacy tests in, which gave FreeBSD a great head start in terms of CI/testability. Other than that, it's kind of a kludgy framework.

What are you saying? That Kyua should be able to run unmodified PyTest code?

Yes. As a bonus, pytest supports the JUnit XML format out of the box and there's a pytest TAP plugin which can be leveraged for a quick integration PoC into kyua: https://pypi.org/project/pytest-tap/ .

[ngie]

In D37971#863634, @melifaro wrote:

In D37971#863583, @ngie wrote:

I hate to be a wet blanket, but a lot of the code seems to go against design decisions made in pytest around fixtures, extendability, etc.

First of all, thank you for the feedback.
Some of the things intentionally or unintentionally are done in ATF way, which does not match pytest. If you could share a bit more detailed feedback, I'd try to make the framework changes to take it closer to the pytest.

In particular, just glancing at this commit, it seems to rely on J-Unit-like structure (used in unittest), which is not strictly adhered to in pytest.

Currently, the primary use case of python integration is development of the tests with the relatively complex setup/teardown procedures (e.g. custom vnet-based topologies). That is the part I feel is the least covered by the shell or c-based tests. Any suggestions on doing it in a more pytest way?

FWIW, I honestly think integration should be the other way around: ATF should integrate into pytest, not pytest should integrate into ATF. If things were done in that manner and we used the JUnit output format (supported natively in pytest), we could move away from Kyua to a framework that is less bespoke, has a ton less boilerplate than ATF, has better developer and user experience, and has better opensource mindshare than ATF/Kyua.
The main value ATF/Kyua provides (IMHO) is the ability to integrate in tests from NetBSD and a format to express legacy tests in, which gave FreeBSD a great head start in terms of CI/testability. Other than that, it's kind of a kludgy framework.

I agree with the ATF/Kyua assessment - I don't like the API or (lack of) functionality either. I agree that pytest can be a good candidate for the core testing framework, provided there is a consensus on that and the migration/support engineering resources are secured. I don't think we're here yet. Currently, shell and C test files correspond to around 90% of the all test files, with similar figures across kernel and userland. Out of the remaining python files, only 10% represent pytest tests; the rest are wrapped with the shell scripts.
Personally, I think that the most beneficial resource application at the moment is improving the python support & reducing the bar to adding python tests to drive adoption. I'd also like to note that the current state of the things is not contrary to the "pytest instead of ATF" idea. It should be notably easier to convert the tests to the "original" pytest format once desired than migrating the shell script mess.

What do you think?

P.S. the details of the percentage calculations are below. I know that some of the tests are third-party and not integrated in the framework, but It doesn't look that it change the results significantly.

• Userland+kernel test file stats: find . -type f -name \*.<c|sh|py> -ipath '*/tests/*' | wc -l
• Kernel test file stats: find tests/sys -type f -name \*.<c|sh|py> | wc -l
• Pytest file stats: find tests -name 'test*.py' | wc -l

Results:
kernel: 297(sh), 158 (c), 54(py)
kernel+user: 533(sh), 451(c), 76(py).
Pytest: 6 files

pytest can run unittest expressed tests out of the box. Over time they could be migrated to a more pure pytest format (as needed), but at the very least one that use pytest for discovering, logging, etc.

I think the sh tests could be expressed using pytest as well. The C tests seem best suited for ATF and the C++ tests seem to be best suited for googletest.

I just revived some Macbooks that I thought were dead this past weekend and I have a hunch that I'll have a lot of dead time over the course of the week to develop for FreeBSD/OSS soon.

I'm going see what I can do to get back up to speed and help out with this effort.

[melifaro]

In D37971#866085, @ngie wrote:

In D37971#863634, @melifaro wrote:

In D37971#863583, @ngie wrote:

I hate to be a wet blanket, but a lot of the code seems to go against design decisions made in pytest around fixtures, extendability, etc.

First of all, thank you for the feedback.
Some of the things intentionally or unintentionally are done in ATF way, which does not match pytest. If you could share a bit more detailed feedback, I'd try to make the framework changes to take it closer to the pytest.

In particular, just glancing at this commit, it seems to rely on J-Unit-like structure (used in unittest), which is not strictly adhered to in pytest.

Currently, the primary use case of python integration is development of the tests with the relatively complex setup/teardown procedures (e.g. custom vnet-based topologies). That is the part I feel is the least covered by the shell or c-based tests. Any suggestions on doing it in a more pytest way?

FWIW, I honestly think integration should be the other way around: ATF should integrate into pytest, not pytest should integrate into ATF. If things were done in that manner and we used the JUnit output format (supported natively in pytest), we could move away from Kyua to a framework that is less bespoke, has a ton less boilerplate than ATF, has better developer and user experience, and has better opensource mindshare than ATF/Kyua.
The main value ATF/Kyua provides (IMHO) is the ability to integrate in tests from NetBSD and a format to express legacy tests in, which gave FreeBSD a great head start in terms of CI/testability. Other than that, it's kind of a kludgy framework.

I agree with the ATF/Kyua assessment - I don't like the API or (lack of) functionality either. I agree that pytest can be a good candidate for the core testing framework, provided there is a consensus on that and the migration/support engineering resources are secured. I don't think we're here yet. Currently, shell and C test files correspond to around 90% of the all test files, with similar figures across kernel and userland. Out of the remaining python files, only 10% represent pytest tests; the rest are wrapped with the shell scripts.
Personally, I think that the most beneficial resource application at the moment is improving the python support & reducing the bar to adding python tests to drive adoption. I'd also like to note that the current state of the things is not contrary to the "pytest instead of ATF" idea. It should be notably easier to convert the tests to the "original" pytest format once desired than migrating the shell script mess.

What do you think?

P.S. the details of the percentage calculations are below. I know that some of the tests are third-party and not integrated in the framework, but It doesn't look that it change the results significantly.

• Userland+kernel test file stats: find . -type f -name \*.<c|sh|py> -ipath '*/tests/*' | wc -l
• Kernel test file stats: find tests/sys -type f -name \*.<c|sh|py> | wc -l
• Pytest file stats: find tests -name 'test*.py' | wc -l

Results:
kernel: 297(sh), 158 (c), 54(py)
kernel+user: 533(sh), 451(c), 76(py).
Pytest: 6 files

pytest can run unittest expressed tests out of the box. Over time they could be migrated to a more pure pytest format (as needed), but at the very least one that use pytest for discovering, logging, etc.

I think the sh tests could be expressed using pytest as well. The C tests seem best suited for ATF and the C++ tests seem to be best suited for googletest.

If there is an implementation of ATF protocol for the test discovery/running for pytest, then both sh and C tests can be handled by the pytest.

I just revived some Macbooks that I thought were dead this past weekend and I have a hunch that I'll have a lot of dead time over the course of the week to develop for FreeBSD/OSS soon.

I'm going see what I can do to get back up to speed and help out with this effort.

It's worth to write something down to ensure we're on the same page.
What's the end state of the test suite you envision? Is it a combination of pytest/kyua/googletest?
If some of the tests (python, sh, potentially C) are moving away from kyua, what are the interfaces provided by the test runner? Are these interfaces to stay the same or be different? (The biggest difference between kyua and pytest, for example, is the explicit cleanup procedure, run in a separate process).