Make capsicum test cases fine-grained
ClosedPublic
Actions

Authored by lwhsu on Jul 30 2020, 8:20 PM.

Details

Reviewers

emaste
ngie

Commits

rS366470: Make capsicum test cases fine-grained

Summary

Add a wrapping script to use ATF to run tests written with googletest
one by one. This helps locating and tracking the failing case in CI easier.

Before:

root@:/usr/tests/sys/capsicum # kyua test
bindat_connectat:bindat_connectat_1  ->  passed  [0.005s]
bindat_connectat:bindat_connectat_2  ->  passed  [0.004s]
bindat_connectat:bindat_connectat_3  ->  passed  [0.004s]
capsicum-test:main  ->  passed  [37.380s]
ioctls_test:cap_ioctls__listen_copy  ->  passed  [0.005s]

Results file id is usr_tests_sys_capsicum.20200730-195720-761226
Results saved to /root/.kyua/store/results.usr_tests_sys_capsicum.20200730-195720-761226.db

5/5 passed (0 failed)

After:

lwhsu@x1c:/usr/tests/sys/capsicum > kyua test
bindat_connectat:bindat_connectat_1  ->  passed  [0.005s]
bindat_connectat:bindat_connectat_2  ->  passed  [0.005s]
bindat_connectat:bindat_connectat_3  ->  passed  [0.005s]
functional:CapMode__AllowedMmapSyscallsForked  ->  passed  [0.116s]
functional:CapabilityPair__sendfile  ->  passed  [0.410s]
functional:Capability__BasicInterceptionForked  ->  passed  [0.347s]
functional:Capability__CapEnterForked  ->  passed  [0.368s]
functional:Capability__CapNewForked  ->  passed  [0.499s]
functional:Capability__DirOperationsForked  ->  passed  [0.122s]
functional:Capability__ExtendedAttributesForked  ->  passed  [0.328s]
functional:Capability__FileInSyncForked  ->  passed  [0.365s]
functional:Capability__InheritanceForked  ->  passed  [0.366s]
functional:Capability__MmapForked  ->  passed  [0.348s]
functional:Capability__NoBypassDAC  ->  passed  [0.322s]
functional:Capability__OpenAtDirectoryTraversalForked  ->  passed  [0.325s]
functional:Capability__OperationsForked  ->  passed  [0.296s]
functional:Capability__PipeUnseekable  ->  passed  [0.284s]
functional:Capability__SocketTransferForked  ->  passed  [4.297s]
functional:Capability__SyscallAt  ->  passed  [0.090s]
functional:Capmode__Abort  ->  passed  [0.099s]
functional:Capmode__AllowedAtSyscalls  ->  passed  [0.098s]
functional:Capmode__AllowedAtSyscallsCwd  ->  passed  [0.124s]
functional:Capmode__AllowedIdentifierSyscallsForked  ->  passed  [0.121s]
functional:Capmode__AllowedPipeSyscallsForked  ->  passed  [0.393s]
functional:Capmode__AllowedProfilSyscallForked  ->  passed  [0.418s]
functional:Capmode__AllowedResourceSyscallsForked  ->  passed  [0.373s]
functional:Capmode__AllowedSchedSyscallsForked  ->  passed  [0.322s]
functional:Capmode__AllowedTimerSyscallsForked  ->  passed  [0.133s]
functional:Capmode__NewThreadForked  ->  passed  [3.399s]
functional:Capmode__SelfKillForked  ->  passed  [0.301s]
functional:Fcntl__BasicForked  ->  passed  [0.349s]
functional:Fcntl__Commands  ->  passed  [0.373s]
functional:Fcntl__FLSubRights  ->  passed  [0.355s]
functional:Fcntl__OWNSubRights  ->  passed  [0.355s]
functional:Fcntl__PreserveSubRights  ->  passed  [0.338s]
functional:Fcntl__SubRightNormalFD  ->  passed  [0.319s]
functional:Fcntl__WriteLock  ->  passed  [0.317s]
functional:ForkedExecve_BasicFexecve___  ->  passed  [0.328s]
functional:ForkedExecve_FailWithoutCap___  ->  passed  [0.354s]
functional:ForkedExecve_InCapMode___  ->  passed  [0.333s]
functional:ForkedExecve_SucceedWithCap___  ->  passed  [0.368s]
functional:ForkedFexecveWithScript_CapModeScriptFail___  ->  passed  [0.347s]
functional:ForkedFexecve_ExecutePermissionCheck___  ->  passed  [0.344s]
functional:ForkedFexecve_ExecveFailure___  ->  passed  [0.328s]
functional:ForkedFexecve_SetuidIgnored___  ->  passed  [0.331s]
functional:ForkedOpenatTest_InCapabilityMode___  ->  passed  [0.365s]
functional:ForkedOpenatTest_WithFlagInCapabilityMode___  ->  passed  [0.331s]
functional:ForkedWithFiles_AllowedFileSyscalls___  ->  passed  [0.329s]
functional:ForkedWithFiles_AllowedMiscSyscalls___  ->  passed  [0.376s]
functional:ForkedWithFiles_AllowedSocketSyscalls___  ->  passed  [0.248s]
functional:ForkedWithFiles_DisallowedFileSyscalls___  ->  passed  [0.299s]
functional:ForkedWithFiles_DisallowedSocketSyscalls___  ->  passed  [0.209s]
functional:Ioctl__Basic  ->  passed  [0.356s]
functional:Ioctl__PreserveSubRights  ->  passed  [0.356s]
functional:Ioctl__SubRightNormalFD  ->  passed  [0.150s]
functional:Ioctl__SubRights  ->  passed  [0.103s]
functional:Ioctl__TooManySubRights  ->  passed  [0.115s]
functional:OpenatTest__WithCapability  ->  passed  [0.114s]
functional:OpenatTest__WithFlag  ->  passed  [0.108s]
functional:Openat__RelativeForked  ->  passed  [0.113s]
functional:Pdfork__Bagpuss  ->  passed  [6.539s]
functional:Pdfork__BagpussDaemon  ->  passed  [6.521s]
functional:Pdfork__DaemonUnrestrictedForked  ->  passed  [0.138s]
functional:Pdfork__FromThread  ->  passed  [2.238s]
functional:Pdfork__InvalidFlag  ->  passed  [0.116s]
functional:Pdfork__MissingRights  ->  passed  [0.112s]
functional:Pdfork__NonProcessDescriptor  ->  passed  [0.113s]
functional:Pdfork__OtherUserForked  ->  passed  [0.139s]
functional:Pdfork__PdkillForked  ->  passed  [1.441s]
functional:Pdfork__PdkillOtherSignal  ->  passed  [1.255s]
functional:Pdfork__PdkillSignalForked  ->  passed  [1.209s]
functional:Pdfork__Simple  ->  passed  [2.287s]
functional:Pdfork__TimeCheck  ->  passed  [0.117s]
functional:Pdfork__UseDescriptor  ->  passed  [0.125s]
functional:PipePdforkDaemon__Close  ->  passed  [0.234s]
functional:PipePdforkDaemon__NoPDSigchld  ->  passed  [0.415s]
functional:PipePdforkDaemon__Pdkill  ->  passed  [0.546s]
functional:PipePdfork__ChildExit  ->  passed  [0.467s]
functional:PipePdfork__Close  ->  passed  [0.483s]
functional:PipePdfork__CloseLast  ->  passed  [0.571s]
functional:PipePdfork__ModeBits  ->  passed  [0.493s]
functional:PipePdfork__MultipleRetrieveExitStatus  ->  passed  [0.519s]
functional:PipePdfork__NoSigchld  ->  passed  [0.409s]
functional:PipePdfork__PassProcessDescriptor  ->  passed  [3.463s]
functional:PipePdfork__Pdkill  ->  passed  [0.280s]
functional:PipePdfork__Poll  ->  passed  [0.411s]
functional:PipePdfork__PollMultiple  ->  passed  [1.421s]
functional:PipePdfork__WaitPdThenPid  ->  passed  [0.123s]
functional:PipePdfork__WaitPidThenPd  ->  passed  [0.116s]
functional:PipePdfork__WildcardWait  ->  passed  [0.123s]
functional:Poll__LotsOFileDescriptorsForked  ->  passed  [0.133s]
functional:PosixMqueue__CapModeForked  ->  passed  [0.427s]
functional:RenameAt__AbsDesignationSame  ->  passed  [0.410s]
functional:Rename__AbsDesignationSame  ->  passed  [0.411s]
functional:Select__LotsOFileDescriptorsForked  ->  passed  [0.288s]
functional:Socket__TCP  ->  passed  [1.439s]
functional:Socket__UDP  ->  passed  [0.105s]
functional:Socket__UnixDomain  ->  passed  [1.124s]
ioctls_test:cap_ioctls__listen_copy  ->  passed  [0.009s]

Results file id is usr_tests_sys_capsicum.20200730-195445-302259
Results saved to /home/lwhsu/.kyua/store/results.usr_tests_sys_capsicum.20200730-195445-302259.db

99/99 passed (0 failed)

Test Plan

cd /usr/tests/sys/capsicum/ && kyua test

Diff Detail

Repository

rS FreeBSD src repository - subversion

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

lwhsu created this revision.Jul 30 2020, 8:20 PM

Herald added subscribers: jonathan, imp, bdrewery. · View Herald TranscriptJul 30 2020, 8:20 PM

lwhsu requested review of this revision.Jul 30 2020, 8:20 PM

Harbormaster completed remote builds in B32664: Diff 75186.Jul 30 2020, 8:21 PM

lwhsu retitled this revision from Make capsicum cases fine-grained to Make capsicum test cases fine-grained.Jul 31 2020, 3:01 AM

NAK. I need to add Googletest support to kyua. Now that kyua has been forked, this is straightforward.

This revision now requires changes to proceed.Jul 31 2020, 6:59 AM

In D25896#573758, @ngie wrote:

NAK. I need to add Googletest support to kyua. Now that kyua has been forked, this is straightforward.

Is there an ETA of that? I still hope we can have this before that is implemented. I'm fine and happy to remove this after kyua supports googletest.

This change is quite non-intrusive - in my opinion we should add a comment (in share/mk/googletest.test.mk) explaining that this is a stop-gap/workaround, commit this for now, then revert it when we have googletest-integrated kyua in the tree.

Add comment in share/mk/googletest.test.mk to note this is a workaround

Harbormaster completed remote builds in B32684: Diff 75224.Jul 31 2020, 5:48 PM

In D25896#573765, @lwhsu wrote:

In D25896#573758, @ngie wrote:

NAK. I need to add Googletest support to kyua. Now that kyua has been forked, this is straightforward.

Is there an ETA of that? I still hope we can have this before that is implemented. I'm fine and happy to remove this after kyua supports googletest.

@ngie ping?

OK with me to commit this with the expectation we'll revert once kyua+googletest integration is ready

This revision was not accepted when it landed; it landed in state Needs Review.Oct 6 2020, 6:46 AM

Closed by commit rS366470: Make capsicum test cases fine-grained (authored by lwhsu). · Explain Why

This revision was automatically updated to reflect the committed changes.

lwhsu added a commit: rS366470: Make capsicum test cases fine-grained.

This significantly slows down running the capsicum tests: On QEMU AArch64 it takes about 360 seconds with the wrapper shell script and 50-60s without it. Can't we add the XFAIL/SKIP inside the C++ source code?
Jenkins reports the output of the failed test so we can see what went wrong.