Page MenuHomeFreeBSD

Add basic tests generated via automation tool [1]
ClosedPublic

Authored by shivansh on Aug 15 2017, 2:29 PM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Nov 22, 9:41 AM
Unknown Object (File)
Thu, Nov 21, 6:31 AM
Unknown Object (File)
Wed, Nov 20, 5:39 AM
Unknown Object (File)
Wed, Nov 20, 5:38 AM
Unknown Object (File)
Thu, Nov 7, 5:55 PM
Unknown Object (File)
Oct 16 2024, 9:48 AM
Unknown Object (File)
Oct 2 2024, 5:47 AM
Unknown Object (File)
Oct 1 2024, 9:43 PM
Subscribers

Details

Summary

The set of introduced changes demonstrate a basic structure of tests
which are generated by an automation tool [1]. These tests lay the
guidelines on which future (generated) tests will be based upon.

Utilities covered -

  • chflags(1)
  • mkdir(1)
  • rcp(1)
  • rmdir(1)

The aim is to generate very basic amount of test coverage for all the
utilities in the base system. After all the utilities are covered, these
test scripts can then be manually updated.

A basic structure of testcases in a generated test -

  • invalid_usage : Verifies the behavior of a utility when an invalid usage is performed for a supported option.
  • ($option)_flag: Verifies the behavior of a utility when a valid usage is performed for a supported option - $option.
  • no_arguments : Verifies the behavior of a utility when it is used without any supported options.

Quoting @brooks on the purpose of having a minimal test coverage -

When replacing major toolchain components, it's quite easy to subtlety
break the startup code in layout or obscure-feature dependent ways. I'm
looking absolute minimum validation of startup. If that's a real test
great, but touching every non-contrib program would be a good start.
This project idea was borne out of frustration with our extremely poor
breadth of coverage while bringing up a new ABI out of tree.

[1]: https://github.com/shivansh/smoketestsuite

Test Plan
  • Execute make install from bin/($utility)/tests.
  • Execute kyua test from /usr/tests/bin/($utility). All testcases should succeed.

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Update etc/mtree/BSD.tests.dist

bin/chflags/tests/chflags_test.sh
40 ↗(On Diff #32088)

Hardcoding the usage string here just makes it harder to update chflags with new or changed options. It might be better merely to do atf_check -s exit:1 -e regex:"usage: chflags" chflags -f. That way we'll still get coverage of the test behavior, without duplicating human-readable stuff from chflags's source.

bin/mkdir/tests/mkdir_test.sh
43 ↗(On Diff #32088)

The man page merely specifies that the command will return >0, not 64. Plus, the use of EX_USAGE and related variables has fallen out of fashion. Probably better to do atf_check -s not-exit:0 -e inline:"$usage_output" mkdir -p

Address requested changes-

  • Avoid hardcoding usage string
  • Use "not-exit:0" for non-zero exit status

The test files weren't manually updated, relevant changes were first made in the tool and the tests generated afterwards.

These files look good to me. However, I'd recommend that you commit the code generation tool too. It can go into the tools directory, which is not automatically built.

This revision is now accepted and ready to land.Aug 28 2017, 3:24 PM

These look good to me.

This revision was automatically updated to reflect the committed changes.