hastd: Add full lifecycle test

[des]

Summary

Add a test case that creates and starts two nodes, has one assume the
primary role and the other the secondary role, writes to the device,
verifies that the data is synced to both nodes, then switches the roles,
and finally shuts both nodes down and compares their backing stores.

[ngie]

This relies specifically on IPv4. Can you use localhost instead of 127.0.0.1

sbin/hastd/tests/hastd_test.sh

143–146

This can be handled in the cleanup routine instead.

156–161

This just handles the cleanup once instead of doing it inline in the test.

[des]

sbin/hastd/tests/hastd_test.sh

143–146

Then we don't test that hastctl is capable of stopping hastd.

[ngie]

sbin/hastd/tests/hastd_test.sh

12

BTW: I'd give a ship-it for this change right now -- thank you :)!

143–146

Removing the files with rm -f (and without atf_check) masks issues if the files are missing. That seems less than ideal for the tests if your goal was to confirm that the services are no longer running if the pidfiles are gone.
At the very least leaving the files behind and cleaning them up in cleanup with emit warnings when kyua completes the test -- @jmmv filed an issue for no longer masking cleanup problems at test exit. It might be worth investing in that for a later release [if it's tunable via the config] so items like this can be caught and asserted on loudly.