Integrate indent tests added in r313544 into ATF/Kyua and the FreeBSD test suite

[ngie]

Summary

Integrate indent tests added in r313544 into ATF/Kyua

This change does the following:

• adds a test driver for the testcases added in r313544, patterned after bin/sh/tests/functional_test.sh . The driver calls indent as noted in r313544, with an exception: The $FreeBSD$ RCS keyword's expansion is reindented with indent, which means that the output differs from the expected output. Thus, all lines with $FreeBSD$ in them are deleted on the fly, both in the input file and the output file.
  
  The test inputs/outputs are copied to the kyua sandbox before the test is run as the pathing in some of the files relies on pathing normalized to the current directory (copying the files is the easiest way to resolve the issue).

• Introduces symmetry in the test inputs/outputs by adding the exit code to the files. This simplified the test driver notably by requiring less filename/test name manipulation.

Sponsored by: Dell EMC Isilon

Test Plan

$ (set -e; sudo make hier; cd usr.bin/indent/; make; sudo make install; cd tests; make check)
cd /usr/src/etc; PATH=/usr/obj/usr/src/tmp/legacy/usr/sbin:/usr/obj/usr/src/tmp/legacy/usr/bin:/usr/obj/usr/src/tmp/legacy/bin:/usr/obj/usr/src/tmp/usr/sbin:/usr/obj/usr/src/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin make LOCAL_MTREE= distrib-dirs
for file in /usr/share/doc/usd/10.exref /usr/share/doc/usd/11.edit /usr/share/doc/usd/12.vi /usr/share/doc/usd/13.viref; do  if [ -f /${file} ]; then  rm -f /${file};  fi;  done
mtree -deU -i -f /usr/src/etc/mtree/BSD.root.dist -p /
boot/kernel:
        type (dir, link)
mtree -deU -i -f /usr/src/etc/mtree/BSD.var.dist -p /var
mtree -deU -i -f /usr/src/etc/mtree/BSD.usr.dist -p /usr
mtree -deU -i -f /usr/src/etc/mtree/BSD.include.dist -p /usr/include
mtree -deU -i -f /usr/src/etc/mtree/BSD.debug.dist -p /usr/lib
mtree -deU -i -f /usr/src/etc/mtree/BSD.groff.dist -p /usr
mtree -deU -i -f /usr/src/etc/mtree/BSD.tests.dist -p /usr/tests
./usr.bin/indent missing (created)
mtree -deU -i -f /usr/src/etc/mtree/BSD.tests.dist -p /usr/lib/debug//usr/tests
mtree -deU -i -f /usr/src/etc/mtree/BSD.sendmail.dist -p /
install -l s usr/src/sys /sys
cd /usr/share/man;  for mandir in man*; do  install -l s ../$mandir  /usr/share/man/en.ISO8859-1/;  install -l s ../$mandir  /usr/share/man/en.UTF-8/;  done
cd /usr/share/openssl/man;  for mandir in man*; do  install -l s ../$mandir  /usr/share/openssl/man/en.ISO8859-1/;  done
set - `grep "^[a-zA-Z]" /usr/src/etc/man.alias`;  while [ $# -gt 0 ] ; do  install -l s "$2" "/usr/share/man/$1";  if [ "yes" != "no" ]; then  install -l s "$2"  "/usr/share/openssl/man/$1";  fi;  shift; shift;  done
set - `grep "^[a-zA-Z]" /usr/src/etc/nls.alias`;  while [ $# -gt 0 ] ; do  install -l s "$2" "/usr/share/nls/$1";  shift; shift;  done
===> tests (all)
install  -s -o root -g wheel -m 555   indent /usr/bin/indent
install  -o root -g wheel -m 444  indent.debug /usr/lib/debug/usr/bin/indent.debug
install  -o root -g wheel -m 444 indent.1.gz  /usr/share/man/man1/
===> tests (install)
install  -o root  -g wheel -m 555  functional_test  /usr/tests/usr.bin/indent/functional_test
install  -o root  -g wheel -m 444  Kyuafile  /usr/tests/usr.bin/indent/Kyuafile
install  -o root -g wheel  -m 444 /usr/src/usr.bin/indent/tests/comments.0 /usr/src/usr.bin/indent/tests/comments.0.stdout /usr/src/usr.bin/indent/tests/declarations.0 /usr/src/usr.bin/indent/tests/declarations.0.stdout /usr/src/usr.bin/indent/tests/elsecomment.0 /usr/src/usr.bin/indent/tests/elsecomment.0.stdout /usr/src/usr.bin/indent/tests/elsecomment.0.pro /usr/src/usr.bin/indent/tests/float.0 /usr/src/usr.bin/indent/tests/float.0.stdout /usr/src/usr.bin/indent/tests/label.0 /usr/src/usr.bin/indent/tests/label.0.stdout /usr/src/usr.bin/indent/tests/label.0.pro /usr/src/usr.bin/indent/tests/list_head.0 /usr/src/usr.bin/indent/tests/list_head.0.stdout /usr/src/usr.bin/indent/tests/nsac.0 /usr/src/usr.bin/indent/tests/nsac.0.stdout /usr/src/usr.bin/indent/tests/nsac.0.pro /usr/src/usr.bin/indent/tests/offsetof.0 /usr/src/usr.bin/indent/tests/offsetof.0.stdout /usr/src/usr.bin/indent/tests/sac.0 /usr/src/usr.bin/indent/tests/sac.0.stdout /usr/src/usr.bin/indent/tests/sac.0.pro /usr/src/usr.bin/indent/tests/struct.0 /usr/src/usr.bin/indent/tests/struct.0.stdout /usr/src/usr.bin/indent/tests/surplusbad.0 /usr/src/usr.bin/indent/tests/surplusbad.0.stdout /usr/src/usr.bin/indent/tests/surplusbad.0.pro /usr/src/usr.bin/indent/tests/types_from_file.0 /usr/src/usr.bin/indent/tests/types_from_file.0.stdout /usr/src/usr.bin/indent/tests/types_from_file.0.list /usr/src/usr.bin/indent/tests/types_from_file.0.pro /usr/src/usr.bin/indent/tests/wchar.0 /usr/src/usr.bin/indent/tests/wchar.0.stdout /usr/tests/usr.bin/indent/
functional_test:comments  ->  passed  [0.026s]
functional_test:declarations  ->  passed  [0.026s]
functional_test:elsecomment  ->  passed  [0.030s]
functional_test:float  ->  passed  [0.024s]
functional_test:label  ->  passed  [0.026s]
functional_test:list_head  ->  passed  [0.026s]
functional_test:nsac  ->  passed  [0.025s]
functional_test:offsetof  ->  passed  [0.027s]
functional_test:sac  ->  passed  [0.026s]
functional_test:struct  ->  passed  [0.026s]
functional_test:surplusbad  ->  passed  [0.026s]
functional_test:types_from_file  ->  passed  [0.029s]
functional_test:wchar  ->  passed  [0.028s]

Results file id is usr_tests_usr.bin_indent.20170303-192412-590088
Results saved to /home/ngie/.kyua/store/results.usr_tests_usr.bin_indent.20170303-192412-590088.db

13/13 passed (0 failed)

[ngie]

Mute warnings from arc lint by chmod -x'ing functional_test.sh

[ngie]

Fix Dell EMC company name in copyright

[ngie]

More copyright fudging...

[pstef]

The driver calls indent as noted in r313544, with an exception: The $FreeBSD$ RCS keyword's expansion is reindented with indent, which means that the output differs from the expected output.

I didn't predict that, sorry. The reformatting can be stopped from happening if we trick indent into thinking that the comments are box comments by adding a dash like this: /*- $FreeBSD */. The output would still differ since $test.0 and $test.0.stdout are two different files and $FreeBSD$ reflects that, but at least you'd have a simpler case to ignore (a single line).

functional_test:elsecomment -> failed: Test case body returned a non-ok exit code, but this is not allowed [0.028s]
functional_test:label -> failed: Test case body returned a non-ok exit code, but this is not allowed [0.027s]
functional_test:sac -> failed: Test case body returned a non-ok exit code, but this is not allowed [0.029s]
functional_test:types_from_file -> failed: Test case body returned a non-ok exit code, but this is not allowed [0.026s]

I don't understand any of that. Running the tests like this:

$ ../indent -P./elsecomment.pro < elsecomment.0 > elsecomment.run

yields exit code 0 for me.

[ngie]

In D9682#200303, @pstef wrote:

The driver calls indent as noted in r313544, with an exception: The $FreeBSD$ RCS keyword's expansion is reindented with indent, which means that the output differs from the expected output.

I didn't predict that, sorry. The reformatting can be stopped from happening if we trick indent into thinking that the comments are box comments by adding a dash like this: /*- $FreeBSD */. The output would still differ since $test.0 and $test.0.stdout are two different files and $FreeBSD$ reflects that, but at least you'd have a simpler case to ignore (a single line).

It's ok. Deleting the lines with sed is the easiest way to deal with the issue.

functional_test:elsecomment -> failed: Test case body returned a non-ok exit code, but this is not allowed [0.028s]
functional_test:label -> failed: Test case body returned a non-ok exit code, but this is not allowed [0.027s]
functional_test:sac -> failed: Test case body returned a non-ok exit code, but this is not allowed [0.029s]
functional_test:types_from_file -> failed: Test case body returned a non-ok exit code, but this is not allowed [0.026s]

I don't understand any of that.

It's the terse output for the tests. Running them with kyua debug divulges the actual error:

$ kyua debug -k /usr/tests/usr.bin/indent/Kyuafile functional_test:elsecomment
Executing command [ sed -e /\$FreeBSD.*\$/d /usr/tests/usr.bin/indent/elsecomment.0.stdout ]
Executing command [ /usr/bin/indent ]
Fail: stdout does not match golden output
--- output_file.parsed	2017-02-21 01:38:44.629238000 +0000
+++ /tmp/kyua.grjvii/2/work/check.HNplw3/stdout	2017-02-21 01:38:44.633813000 +0000
@@ -2,8 +2,7 @@
 void 
 t(void)
 {
-	if (0)
-	{
+	if (0) {
 
 	}			/* Old indent would remove the following
 				 * blank line */
@@ -12,8 +11,7 @@
 	 * test
 	 */
 
-	if (1)
-		;
+	if (1);
 	else			/* Old indent would get very confused here */
 	{
 
Files left in work directory after failure: input_file.parsed, output_file.parsed
functional_test:elsecomment  ->  failed: atf-check failed; see the output of the test for details

Running the tests like this:

$ ../indent -P./elsecomment.pro < elsecomment.0 > elsecomment.run

yields exit code 0 for me.

Yes. Part of the issue with the exit code being non-zero dealt with the fact that atf-check was returning a non-zero value when I called it as the second portion of the pipeline. Splitting up the sed operation and using a temporary file, then running indent on the temporary file, worked more intuitively in terms of illustrating the issue.

[ngie]

Use an intermediate file instead of calling indent like "sed ... | atf_check ... indent'

This makes the actual error more intuitive via "kyua test" when dealing with failing output

[pstef]

usr.bin/indent/tests/functional_test.sh

48 ↗

(On Diff #25454)

I'll test this later, but there should be no space after the -P.

[ngie]

usr.bin/indent/tests/functional_test.sh

48 ↗

(On Diff #25454)

Oh wild... indent(1) doesn't use getopt(3) >_>, so it needs to not have the space...

[ngie]

Remove the space between -P and the profile name

[pstef]

usr.bin/indent/tests/functional_test.sh

47 ↗

(On Diff #25774)

Based on our conversation in D9007, I was under the impression that it's OK for me to drop the exit status from the name of a profile file. But I think $tc here is elsecomment.0, so it tries to load elsecomment.0.pro and fails.

It can be handled here or the name of the file can be changed. I'm OK with either.

48 ↗

(On Diff #25774)

The profile flag should be added unconditionally, because it blocks loading ./.indent.pro or ~/.indent.pro.

[ngie]

Introduce symmetry proposed in D9007 to simplify the tests.

Also, as seen in testing, the tests require all of the files be located in the sandbox before
running the tests, otherwise test inputs/outputs, like the .list files won't be found and the
tests will fail.

[ngie]

If a profile isn't specified, use -npro to suppress implicit use of ~/.indent.pro