Page MenuHomeFreeBSD
Files F147297744

D12249.id32730.diff
No OneTemporary
Actions

D12249.id32730.diff
View Options

Index: tools/tools/smoketestsuite/LICENSE
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/LICENSE
@@ -0,0 +1,23 @@
+Copyright (c) 2017, Shivansh Rai
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Index: tools/tools/smoketestsuite/README.md
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/README.md
@@ -0,0 +1,59 @@
+# Smoke testing of base utilities (FreeBSD)
+
+Test generation tool made as a part of [Google Summer of Code '17 with FreeBSD](https://summerofcode.withgoogle.com/projects/#6426676740227072).
+**Refer the [FreeBSD wiki](https://wiki.freebsd.org/SummerOfCode2017/SmokeTestingOfBaseUtilities) for an overview and updates.**
+
+## Directory Structure
+```
+.
+├── deprecated_tests
+│   └── ........................:: Tests pertaining to initial test plan
+└── src ........................:: Automation tool pertaining to new test plan
+ ├── annotations
+ │   └── ....................:: Annotation files (generated/user-defined)
+ ├── generated_tests
+ │   └── ....................:: Generated atf-sh test scripts
+ ├── scripts
+ │ └── ....................:: Handy scripts
+ ├── generate_test.cpp ......:: Test generator
+ ├── read_annotations.cpp ...:: Annotation parser
+ └── utils.cpp ..............:: Index generator
+```
+
+## Automation tool
+An (in-progress) implementation of the automation tool briefly described [here](https://lists.freebsd.org/pipermail/soc-status/2017-July/001079.html).
+The following diagram summarizes how different components fit with the testcase-generator -
+
+![Automation-Tool](http://i.imgur.com/JhKM7h1.png)
+
+- - -
+
+## Dependencies
+* Boost C++ libraries : The tool was tested to work with the port `boost-all-1.64.0`.
+
+## Instructions
+
+Clone the repository via -
+```
+git clone git@github.com:shivansh/smoketestsuite.git ~/smoketestsuite
+```
+The location `~/smoketestsuite` is important! If using a different location, the scripts under [src/scripts](src/scripts) need to be updated accordingly (for the time being).
+
+### Populating groff scripts
+* The directory [src/groff](src/groff) should be populated with the relevant groff scripts before proceeding for test generation. These scripts are available in the FreeBSD source tree. For filtering the utilities section wise, [fetch_groff.sh](src/scripts/fetch_groff.sh) sets the variable `section` to a default value of **1**. This value can be changed at will. However, it should be noted that currently the tool is tested to successfully generate **section 1 utilities** and might probably fail for other section numbers.
+
+* The variable `src` in [fetch_groff.sh](src/scripts/fetch_groff.sh) should be updated to the local location of the FreeBSD source. The default value is `~/freebsd`.
+
+* For populating `src/groff`, execute from [src](src) -
+ ```
+ make fetch_groff
+ ```
+
+### Generating tests
+Execute the following commands inside [src](src) -
+```
+make clean
+make && make run
+```
+
+A few demo tests are located in [src/generated_tests](src/generated_tests).
Index: tools/tools/smoketestsuite/src/Makefile
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/Makefile
@@ -0,0 +1,34 @@
+# $FreeBSD$
+#
+# Makefile for building automation tool
+
+CC = c++
+CFLAGS = -I/usr/local/include -std=c++11
+LIBS = -L/usr/local/lib -lboost_system -lboost_thread
+OBJS = utils.o read_annotations.o add_testcase.o generate_test.o
+
+generate_tests : $(OBJS)
+ $(CC) $(CFLAGS) -o $(.TARGET) $(.ALLSRC) $(LIBS)
+
+$(OBJS) : $(.PREFIX).cpp $(.PREFIX).h
+ $(CC) $(CFLAGS) -c $(.PREFIX).cpp
+
+.PHONY: clean clean_tests run fetch_groff validate_changes
+
+clean:
+ rm -f generate_tests $(OBJS)
+
+clean_tests:
+ rm -f generated_tests/*
+
+run:
+ @echo Generating annotations...
+ sh scripts/generate_annot.sh
+ @echo Generating test files...
+ ./generate_tests
+
+fetch_groff:
+ sh scripts/fetch_groff.sh
+
+validate_changes:
+ sh scripts/validate.sh
Index: tools/tools/smoketestsuite/src/add_testcase.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/add_testcase.h
@@ -0,0 +1,40 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+namespace add_testcase {
+ void
+ add_known_testcase(std::string, std::string, std::string, \
+ std::string, std::ofstream&);
+
+ void
+ add_unknown_testcase(std::string, std::string, std::string, \
+ int, std::string&);
+
+ void
+ add_noargs_testcase(std::string, std::pair<std::string, int>, \
+ std::ofstream&);
+}
Index: tools/tools/smoketestsuite/src/add_testcase.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/add_testcase.cpp
@@ -0,0 +1,168 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <fstream>
+#include <iostream>
+#include "add_testcase.h"
+
+// Adds a test-case for an option with known usage.
+void
+add_testcase::add_known_testcase(std::string option,
+ std::string util_with_section,
+ std::string descr,
+ std::string output,
+ std::ofstream& test_script)
+{
+ std::string testcase_name;
+ std::string utility = util_with_section.substr(0,
+ util_with_section.length() - 3);
+
+ // Add testcase name.
+ test_script << "atf_test_case ";
+ if (!option.empty()) {
+ testcase_name = option;
+ testcase_name.append("_flag");
+ }
+ else
+ testcase_name = "no_arguments";
+ test_script << testcase_name + "\n";
+
+ // Add testcase description.
+ test_script << testcase_name
+ + "_head()\n{\n\tatf_set \"descr\" ";
+ if (!descr.empty())
+ test_script << descr;
+ else
+ test_script << "\"Verify the usage of option \'" + option + "\'\"";
+ test_script << "\n}\n\n";
+
+ // Add body of the testcase.
+ test_script << testcase_name + "_body()\n{"
+ + "\n\tatf_check -s exit:0 -o ";
+
+ // Match the usage output if generated.
+ if (!output.empty())
+ test_script << "inline:\"" + output + "\" ";
+ else
+ test_script << "empty ";
+ test_script << utility;
+ if (!option.empty())
+ test_script << " -" + option;
+ test_script << "\n}\n\n";
+}
+
+// Adds a test-case for an option with unknown usage.
+void
+add_testcase::add_unknown_testcase(std::string option,
+ std::string util_with_section,
+ std::string output,
+ int exitstatus,
+ std::string& testcase_buffer)
+{
+ std::string utility = util_with_section.substr(0,
+ util_with_section.length() - 3);
+
+ testcase_buffer.append("\n\tatf_check -s not-exit:0 -e ");
+
+ // Check if a usage message was produced.
+ if (!output.compare(0, 6, "usage:"))
+ testcase_buffer.append("match:\"$usage_output\" ");
+ else if (!output.empty())
+ testcase_buffer.append("inline:\"" + output + "\" ");
+ else
+ testcase_buffer.append("empty ");
+
+ testcase_buffer.append(utility);
+
+ if (!option.empty())
+ testcase_buffer.append(" -" + option);
+}
+
+// Adds a test-case for usage without any arguments.
+void
+add_testcase::add_noargs_testcase(std::string util_with_section,
+ std::pair<std::string, int> output,
+ std::ofstream& test_script)
+{
+ std::string descr;
+ std::string utility = util_with_section.substr(0,
+ util_with_section.length() - 3);
+
+ if (output.second) {
+ // An error was encountered.
+ test_script << std::string("atf_test_case no_arguments\n")
+ + "no_arguments_head()\n{\n\tatf_set \"descr\" ";
+ if (!output.first.empty()) {
+ // We expect a usage message to be generated in this case.
+ if (!output.first.compare(0, 6, "usage:")) {
+ descr = "\"Verify that " + util_with_section
+ + " fails and generates a valid usage"
+ + " message when no arguments are supplied\"";
+
+ test_script << descr + "\n}\n\nno_arguments_body()\n{"
+ + "\n\tatf_check -s not-exit:0 -e match:\"$usage_output\" "
+ + utility;
+ }
+ else {
+ descr = "\"Verify that " + util_with_section
+ + " fails and generates a valid output"
+ + " when no arguments are supplied\"";
+
+ test_script << descr + "\n}\n\nno_arguments_body()\n{"
+ + "\n\tatf_check -s not-exit:0 -e inline:\""
+ + output.first + "\" "
+ + utility;
+ }
+ }
+
+ else {
+ descr = "\"Verify that " + util_with_section
+ + "fails silently when no arguments are supplied\"" ;
+ test_script << descr + "\n}\n\nno_arguments_body()\n{"
+ + "\n\tatf_check -s not-exit:0 -e empty "
+ + utility;
+ }
+
+ test_script << "\n}\n\n";
+ }
+
+ else {
+ // The command ran successfully, hence we guessed
+ // a correct usage for the utility under test.
+ if (!output.first.empty())
+ descr = "\"Verify that " + util_with_section
+ + " executes successfully and produces a valid"
+ + " output when invoked without any arguments\"";
+ else
+ descr = "\"Verify that " + util_with_section
+ + " executes successfully and silently"
+ + " when invoked without any arguments\"";
+
+ add_testcase::add_known_testcase("", util_with_section, descr,
+ output.first, test_script);
+ }
+}
Index: tools/tools/smoketestsuite/src/annotations/date_test.annot
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/annotations/date_test.annot
@@ -0,0 +1,5 @@
+R_flag
+j_flag
+n_flag
+no_arguments
+u_flag
Index: tools/tools/smoketestsuite/src/generate_test.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generate_test.h
@@ -0,0 +1,33 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include "utils.h"
+
+namespace generate_test {
+ std::pair<std::string, int> exec(const char*);
+ void generate_test(std::string, std::string);
+}
Index: tools/tools/smoketestsuite/src/generate_test.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generate_test.cpp
@@ -0,0 +1,290 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <array>
+#include <boost/thread.hpp>
+#include <cstdlib>
+#include <dirent.h>
+#include <fstream>
+#include <iostream>
+#include <memory>
+#include <stdexcept>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <unordered_set>
+#include "generate_test.h"
+#include "add_testcase.h"
+#include "read_annotations.h"
+
+#define TIMEOUT 1 // threshold (seconds) for a function call to return.
+
+// Executes the passed argument "cmd" in a shell
+// and returns its output and the exit status.
+std::pair<std::string, int>
+generate_test::exec(const char* cmd)
+{
+ const int bufsize = 128;
+ std::array<char, bufsize> buffer;
+ std::string usage_output;
+ FILE* pipe = popen(cmd, "r");
+
+ if (!pipe)
+ throw std::runtime_error("popen() failed!");
+
+ try {
+ while (!feof(pipe))
+ if (std::fgets(buffer.data(), bufsize, pipe) != NULL)
+ usage_output += buffer.data();
+ }
+ catch(...) {
+ pclose(pipe);
+ throw "Unable to execute the command: " + std::string(cmd);
+ }
+
+ return std::make_pair<std::string, int>
+ ((std::string)usage_output, WEXITSTATUS(pclose(pipe)));
+}
+
+// Generate a test for the given utility.
+void
+generate_test::generate_test(std::string utility,
+ std::string section)
+{
+ std::list<utils::opt_rel*> ident_opt_list; // List of identified option relations.
+ std::vector<std::string> usage_messages; // Vector to store usage messages for comparison.
+ std::string command; // Command to be executed in shell.
+ std::string descr; // Testcase description.
+ std::string testcase_list; // List of testcases.
+ std::string testcase_buffer; // Buffer for (temporarily) holding testcase data.
+ std::string test_file; // atf-sh test name.
+ std::string util_with_section; // Section number appended to utility.
+ std::ofstream test_ofs; // Output stream for the atf-sh test.
+ std::ifstream license_ifs; // Input stream for license.
+ std::pair<std::string, int> output; // Return value type for `exec()`.
+ std::unordered_set<std::string> annot; // Hashset of utility specific annotations.
+ int temp;
+
+ // Read annotations and populate hash set "annot".
+ annotations::read_annotations(utility, annot);
+
+ util_with_section = utility + '(' + section + ')';
+
+ utils::opt_def f_opts;
+ ident_opt_list = f_opts.check_opts(utility);
+ test_file = "generated_tests/" + utility + "_test.sh";
+
+ // Add license in the generated test scripts.
+ test_ofs.open(test_file, std::ios::out);
+ license_ifs.open("license", std::ios::in);
+
+ test_ofs << license_ifs.rdbuf();
+ license_ifs.close();
+
+ // If a known option was encountered (i.e. `ident_opt_list` is
+ // populated), produce a testcase to check the validity of the
+ // result of that option. If no known option was encountered,
+ // produce testcases to verify the correct (generated) usage
+ // message when using the supported options incorrectly.
+
+ // Add testcases for known options.
+ if (!ident_opt_list.empty()) {
+ for (const auto &i : ident_opt_list) {
+ command = utility + " -" + i->value + " 2>&1";
+ output = generate_test::exec(command.c_str());
+ if (!output.first.compare(0, 6, "usage:")) {
+ add_testcase::add_known_testcase(i->value, util_with_section,
+ descr, output.first, test_ofs);
+ }
+ else {
+ // A usage message was produced, i.e. we
+ // failed to guess the correct usage.
+ add_testcase::add_unknown_testcase(i->value, util_with_section, output.first,
+ output.second, testcase_buffer);
+ }
+ testcase_list.append("\tatf_add_test_case " + i->value + "_flag\n");
+ }
+ }
+
+ // Add testcases for the options whose usage is not known (yet).
+ if (!f_opts.opt_list.empty()) {
+
+ // For the purpose of adding a "$usage_output" variable,
+ // we choose the option which produces one.
+ // TODO Avoid double executions of an option, i.e. one while
+ // selecting usage message and another while generating testcase.
+
+ if (f_opts.opt_list.size() == 1) {
+ // Utility supports a single option, check if it produces a usage message.
+ command = utility + " -" + f_opts.opt_list.front() + " 2>&1";
+ output = generate_test::exec(command.c_str());
+
+ if (output.second)
+ test_ofs << "usage_output=\'" + output.first + "\'\n\n";
+ }
+
+ else {
+ // Utility supports multiple options. In case the usage message
+ // is consistent for atleast "two" options, we reduce duplication
+ // by assigning a variable "usage_output" in the test script.
+ for (const auto &i : f_opts.opt_list) {
+ command = utility + " -" + i + " 2>&1";
+ output = generate_test::exec(command.c_str());
+
+ if (output.second && usage_messages.size() < 3)
+ usage_messages.push_back(output.first);
+ }
+
+ temp = usage_messages.size();
+ for (int j = 0; j < temp; j++) {
+ if (!(usage_messages.at(j)).compare(usage_messages.at((j+1) % temp))) {
+ test_ofs << "usage_output=\'"
+ + output.first.substr(0, 7 + utility.size())
+ + "\'\n\n";
+ break;
+ }
+ }
+ }
+
+ // Execute the utility with supported options
+ // and add (+ve)/(-ve) testcases accordingly.
+ for (const auto &i : f_opts.opt_list) {
+ // If the option is annotated, skip it.
+ if (annot.find(i) != annot.end())
+ continue;
+
+ command = utility + " -" + i + " 2>&1";
+ output = generate_test::exec(command.c_str());
+
+ if (output.second) {
+ // Non-zero exit status was encountered.
+ add_testcase::add_unknown_testcase(i, util_with_section, output.first,
+ output.second, testcase_buffer);
+ }
+ else {
+ // EXIT_SUCCESS was encountered. Hence,
+ // the guessed usage was correct.
+ add_testcase::add_known_testcase(i, util_with_section, "",
+ output.first, test_ofs);
+ testcase_list.append(std::string("\tatf_add_test_case ")
+ + i + "_flag\n");
+ }
+ }
+
+ testcase_list.append("\tatf_add_test_case invalid_usage\n");
+ test_ofs << std::string("atf_test_case invalid_usage\ninvalid_usage_head()\n")
+ + "{\n\tatf_set \"descr\" \"Verify that an invalid usage "
+ + "with a supported option produces a valid error message"
+ + "\"\n}\n\ninvalid_usage_body()\n{";
+
+ test_ofs << testcase_buffer + "\n}\n\n";
+ }
+
+ // Add a testcase under "no_arguments" for
+ // running the utility without any arguments.
+ if (annot.find("*") == annot.end()) {
+ command = utility + " 2>&1";
+ output = generate_test::exec(command.c_str());
+ add_testcase::add_noargs_testcase(util_with_section, output, test_ofs);
+ testcase_list.append("\tatf_add_test_case no_arguments\n");
+ }
+
+ test_ofs << "atf_init_test_cases()\n{\n" + testcase_list + "}\n";
+ test_ofs.close();
+}
+
+int
+main()
+{
+ std::ifstream groff_list;
+ std::list<std::pair<std::string, std::string>> utility_list;
+ std::string test_file; // atf-sh test name.
+ std::string util_name; // Utility name.
+ struct stat buffer;
+ struct dirent *ent;
+ DIR *dir;
+ char answer; // User input to determine overwriting of test files.
+ int flag = 0;
+
+ // For testing (or generating tests for only selected utilities),
+ // the utility_list can be populated above during declaration.
+ if (utility_list.empty()) {
+ if ((dir = opendir("groff"))) {
+ while ((ent = readdir(dir))) {
+ util_name = ent->d_name;
+ utility_list.push_back(std::make_pair<std::string, std::string>
+ (util_name.substr(0, util_name.length() - 2),
+ util_name.substr(util_name.length() - 1, 1)));
+ }
+ closedir(dir);
+ }
+ else {
+ fprintf(stderr, "Could not open directory: groff/");
+ return EXIT_FAILURE;
+ }
+ }
+
+ for (const auto &util : utility_list) {
+ test_file = "generated_tests/" + util.first + "_test.sh";
+
+ // Check if the test file already exists. In
+ // case it does, confirm before proceeding.
+ if (!flag && !stat(test_file.c_str(), &buffer)) {
+ std::cout << "Test file(s) already exists. Overwrite? [y/n] ";
+ std::cin >> answer;
+
+ switch (answer) {
+ case 'n':
+ case 'N':
+ fprintf(stderr, "Stopping execution\n");
+ flag = 1;
+ break;
+
+ default:
+ // TODO capture newline character
+ flag = 1;
+ break;
+ }
+ }
+
+ std::cout << "Generating test for: " + util.first
+ + '('+ util.second + ')' << " ...";
+
+ boost::thread api_caller(generate_test::generate_test, util.first, util.second);
+ if (api_caller.timed_join(boost::posix_time::seconds(TIMEOUT))) {
+ // API call successfully returned within TIMEOUT (seconds).
+ std::cout << "Successful\n";
+ }
+ else {
+ // API call timed out.
+ std::cout << "Failed!\n";
+ // Remove the incomplete test file.
+ remove(("generated_tests/" + util.first + "_test.sh").c_str());
+ }
+ }
+
+ return EXIT_SUCCESS;
+}
Index: tools/tools/smoketestsuite/src/generated_tests/README.md
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generated_tests/README.md
@@ -0,0 +1,20 @@
+# Generated tests
+**NOTE:** The generated tests are maintained at [shivansh/smoketests](https://github.com/shivansh/smoketests). The directory [generated_tests](src/generated_tests) only contains a few demos.
+The following table summarizes the type of test-cases produced for 3 (randomly chosen) utilities.
+
+| **Test** | **Positive test-cases** | **Negative test-cases** |
+----------------------------|:-------------------:|:-------------------:|
+[date_test.sh](src/generated_tests/date_test.sh) | ~~5~~ 0 | ~~2~~ 6
+[ln_test.sh](src/generated_tests/ln_test.sh) | 0 | 11
+[stdbuf_test.sh](src/generated_tests/stdbuf_test.sh)| 1 | ~~0~~ 3
+
+Some key-points worth noting :
+* [date_test.sh](src/generated_tests/date_test.sh) is time/timezone dependent, hence all the test-cases apart from `invalid_usage` will fail. The aim for using this utility is to demonstrate that the tool can generate positive test-cases along with negative ones. It also demonstrates the need to sanity check the tests.
+ - After adding functionality to generate annotations, the 5 timezone dependent testcases were removed.
+ - Adding functionality for long options increased the number of negative testcases by 4.
+
+* [ln_test.sh](src/generated_tests/ln_test.sh) demonstrates that if the utility produces a common usage message for every invalid usage, then the tool can define a variable containing this message and avoid duplication while generating the test-cases. Also, all the negative tests are collated under a single testcase named `invalid_usage`.
+
+* The groff script for `stdbuf(1)` is slightly different than others, hence
+the tool couldn’t generate any negative test-case in [stdbuf_test.sh](src/generated_tests/stdbuf_test.sh) as it couldn't pick up the supported options. This scenario will be taken into account next. The aim for using this utility is to demonstrate that the tool will always produce atleast one positive/negative testcase.
+ - Adding functionality for long options increased the number of negative testcases by 3.
Index: tools/tools/smoketestsuite/src/generated_tests/date_test.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generated_tests/date_test.sh
@@ -0,0 +1,66 @@
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+#
+
+atf_test_case invalid_usage
+invalid_usage_head()
+{
+ atf_set "descr" "Verify that an invalid usage with a supported option produces a valid error message"
+}
+
+invalid_usage_body()
+{
+ atf_check -s not-exit:0 -e inline:"date: option requires an argument -- d
+usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
+ [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
+" date -d
+ atf_check -s not-exit:0 -e inline:"date: option requires an argument -- f
+usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
+ [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
+" date -f
+ atf_check -s not-exit:0 -e inline:"date: option requires an argument -- r
+usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
+ [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
+" date -r
+ atf_check -s not-exit:0 -e inline:"date: option requires an argument -- r
+usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
+ [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
+" date -r
+ atf_check -s not-exit:0 -e inline:"date: option requires an argument -- t
+usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
+ [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
+" date -t
+ atf_check -s not-exit:0 -e inline:"date: option requires an argument -- v
+usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
+ [-f fmt date | [[[[[cc]yy]mm]dd]HH]MM[.ss]] [+format]
+" date -v
+}
+
+atf_init_test_cases()
+{
+ atf_add_test_case invalid_usage
+}
Index: tools/tools/smoketestsuite/src/generated_tests/ln_test.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generated_tests/ln_test.sh
@@ -0,0 +1,66 @@
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+#
+
+usage_output='usage: ln'
+
+atf_test_case invalid_usage
+invalid_usage_head()
+{
+ atf_set "descr" "Verify that an invalid usage with a supported option produces a valid error message"
+}
+
+invalid_usage_body()
+{
+ atf_check -s not-exit:0 -e match:"$usage_output" ln -F
+ atf_check -s not-exit:0 -e match:"$usage_output" ln -L
+ atf_check -s not-exit:0 -e match:"$usage_output" ln -P
+ atf_check -s not-exit:0 -e match:"$usage_output" ln -f
+ atf_check -s not-exit:0 -e match:"$usage_output" ln -h
+ atf_check -s not-exit:0 -e match:"$usage_output" ln -i
+ atf_check -s not-exit:0 -e match:"$usage_output" ln -n
+ atf_check -s not-exit:0 -e match:"$usage_output" ln -s
+ atf_check -s not-exit:0 -e match:"$usage_output" ln -v
+ atf_check -s not-exit:0 -e match:"$usage_output" ln -w
+}
+
+atf_test_case no_arguments
+no_arguments_head()
+{
+ atf_set "descr" "Verify that ln(1) fails and generates a valid usage message when no arguments are supplied"
+}
+
+no_arguments_body()
+{
+ atf_check -s not-exit:0 -e match:"$usage_output" ln
+}
+
+atf_init_test_cases()
+{
+ atf_add_test_case invalid_usage
+ atf_add_test_case no_arguments
+}
Index: tools/tools/smoketestsuite/src/generated_tests/mkdir_test.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generated_tests/mkdir_test.sh
@@ -0,0 +1,61 @@
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+#
+
+usage_output='usage: mkdir'
+
+atf_test_case invalid_usage
+invalid_usage_head()
+{
+ atf_set "descr" "Verify that an invalid usage with a supported option produces a valid error message"
+}
+
+invalid_usage_body()
+{
+ atf_check -s not-exit:0 -e inline:"mkdir: option requires an argument -- m
+usage: mkdir [-pv] [-m mode] directory_name ...
+" mkdir -m
+ atf_check -s not-exit:0 -e match:"$usage_output" mkdir -p
+ atf_check -s not-exit:0 -e match:"$usage_output" mkdir -v
+}
+
+atf_test_case no_arguments
+no_arguments_head()
+{
+ atf_set "descr" "Verify that mkdir(1) fails and generates a valid usage message when no arguments are supplied"
+}
+
+no_arguments_body()
+{
+ atf_check -s not-exit:0 -e match:"$usage_output" mkdir
+}
+
+atf_init_test_cases()
+{
+ atf_add_test_case invalid_usage
+ atf_add_test_case no_arguments
+}
Index: tools/tools/smoketestsuite/src/generated_tests/rmdir_test.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generated_tests/rmdir_test.sh
@@ -0,0 +1,58 @@
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+#
+
+usage_output='usage: rmdir'
+
+atf_test_case invalid_usage
+invalid_usage_head()
+{
+ atf_set "descr" "Verify that an invalid usage with a supported option produces a valid error message"
+}
+
+invalid_usage_body()
+{
+ atf_check -s not-exit:0 -e match:"$usage_output" rmdir -p
+ atf_check -s not-exit:0 -e match:"$usage_output" rmdir -v
+}
+
+atf_test_case no_arguments
+no_arguments_head()
+{
+ atf_set "descr" "Verify that rmdir(1) fails and generates a valid usage message when no arguments are supplied"
+}
+
+no_arguments_body()
+{
+ atf_check -s not-exit:0 -e match:"$usage_output" rmdir
+}
+
+atf_init_test_cases()
+{
+ atf_add_test_case invalid_usage
+ atf_add_test_case no_arguments
+}
Index: tools/tools/smoketestsuite/src/generated_tests/stdbuf_test.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generated_tests/stdbuf_test.sh
@@ -0,0 +1,63 @@
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+#
+
+atf_test_case invalid_usage
+invalid_usage_head()
+{
+ atf_set "descr" "Verify that an invalid usage with a supported option produces a valid error message"
+}
+
+invalid_usage_body()
+{
+ atf_check -s not-exit:0 -e inline:"stdbuf: option requires an argument -- e
+Usage: stdbuf [-e 0|L|<sz>] [-i 0|L|<sz>] [-o 0|L|<sz>] <cmd> [args ...]
+" stdbuf -e
+ atf_check -s not-exit:0 -e inline:"stdbuf: option requires an argument -- i
+Usage: stdbuf [-e 0|L|<sz>] [-i 0|L|<sz>] [-o 0|L|<sz>] <cmd> [args ...]
+" stdbuf -i
+ atf_check -s not-exit:0 -e inline:"stdbuf: option requires an argument -- o
+Usage: stdbuf [-e 0|L|<sz>] [-i 0|L|<sz>] [-o 0|L|<sz>] <cmd> [args ...]
+" stdbuf -o
+}
+
+atf_test_case no_arguments
+no_arguments_head()
+{
+ atf_set "descr" "Verify that stdbuf(1) executes successfully and silently when invoked without any arguments"
+}
+
+no_arguments_body()
+{
+ atf_check -s exit:0 -o empty stdbuf
+}
+
+atf_init_test_cases()
+{
+ atf_add_test_case invalid_usage
+ atf_add_test_case no_arguments
+}
Index: tools/tools/smoketestsuite/src/license
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/license
@@ -0,0 +1,28 @@
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+#
+
Index: tools/tools/smoketestsuite/src/read_annotations.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/read_annotations.h
@@ -0,0 +1,33 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <unordered_set>
+
+namespace annotations {
+ void read_annotations(std::string, \
+ std::unordered_set<std::string>&);
+}
Index: tools/tools/smoketestsuite/src/read_annotations.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/read_annotations.cpp
@@ -0,0 +1,58 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <dirent.h>
+#include <fstream>
+#include <iostream>
+#include <string>
+#include "read_annotations.h"
+
+// Read the annotation files and skip
+// generation of the respective tests.
+void
+annotations::read_annotations(std::string utility,
+ std::unordered_set<std::string>& annot)
+{
+ std::string line;
+ // TODO do this for all the annotation files
+ std::ifstream annot_fstream;
+ annot_fstream.open("annotations/" + utility + "_test.annot");
+
+ while (getline(annot_fstream, line)) {
+ // Add a unique identifier for no_arguments testcase
+ if (!line.compare(0, 12, "no_arguments"))
+ annot.insert("*");
+
+ // Add flag value for supported argument testcases
+ // Doing so we ignore the "invalid_usage" testcase
+ // as it is guaranteed to always succeed.
+ else if (!line.compare(2, 4, "flag"))
+ annot.insert(line.substr(0, 1));
+ }
+
+ annot_fstream.close();
+}
Index: tools/tools/smoketestsuite/src/scripts/README.md
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/scripts/README.md
@@ -0,0 +1,12 @@
+# Handy scripts
+
+Script Name | Functionality
+----------- | -------------
+[fetch_groff.sh](fetch_groff.sh) | Lists location of all section 1 utilities
+[fetch_utils.sh](fetch_utils.sh) | Saves all the base utilities in the src tree in **utils_list**
+[generate_annot.sh](generate_annot.sh) | Populates annotation files under [annotations](../annotations)
+[validate.sh](validate.sh) | Validates side-effects of newly introduced changes in the tool
+
+## TODO
+The variables for directory locations defined in these scripts are corresponding to my local machine.
+These will be updated very soon!
Index: tools/tools/smoketestsuite/src/scripts/fetch_groff.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/scripts/fetch_groff.sh
@@ -0,0 +1,47 @@
+#!/bin/sh
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# Script for listing location of groff scripts of utilities in section 1
+
+set -e
+
+src="$HOME/freebsd"
+dir_list="$HOME/smoketestsuite/src/scripts/utils_list"
+groff_src="$HOME/smoketestsuite/src/groff"
+section=1 # Section number for filtering the base utilities
+
+rm -rf "$groff_src" && mkdir "$groff_src"
+cd "$src"
+
+while IFS= read -r dir_entry
+do
+ for file in "$dir_entry"/*
+ do
+ case "$file" in
+ *."$section") cp "$file" "$HOME/smoketestsuite/src/groff/"
+ esac
+ done
+done< "$dir_list"
Index: tools/tools/smoketestsuite/src/scripts/fetch_utils.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/scripts/fetch_utils.sh
@@ -0,0 +1,40 @@
+#! /bin/sh
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# Script for listing all the base utilities
+
+set -e
+
+fetch_utils() {
+ cd "$path"
+ find . -name Makefile | xargs grep -l 'PROG\|PROG_CXX' \
+ | sed -e 's|/Makefile$||' | cut -c 3-
+}
+
+rm -f utils_list
+
+path="$HOME/source-codes/freebsd"
+(fetch_utils) >> utils_list
Index: tools/tools/smoketestsuite/src/scripts/generate_annot.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/scripts/generate_annot.sh
@@ -0,0 +1,92 @@
+#!/bin/sh
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+#
+# Script for generating annotations based on generated tests.
+
+pwd=$(pwd)
+suffix="_test"
+extension=".sh"
+update_required=0
+
+message="Following annotation files were updated. \n"
+
+printf "
++--------------------------------------------------+
+| Checking if any annotation file is to be updated |
++--------------------------------------------------+\n"
+
+for f in "generated_tests"/*
+do
+ annotations=""
+ file=$(basename "$f")
+ test=${file%$extension}
+ utility=${test%$suffix}
+ test_dir="/usr/tests/bin/$utility"
+
+ (
+ cd "$test_dir" || exit
+ report=$(kyua report)
+ i=2
+
+ while true
+ do
+ testcase=$(printf "%s" "$report" | awk 'NR=='"$i"' {print $1}')
+ status=$(printf "%s" "$report" | awk 'NR=='"$i"' {print $3}')
+ check=$(printf "%s" "$testcase" | cut -s -f1 -d":")
+
+ if [ "$check" != "$test" ]; then
+ if [ "$annotations" ]; then
+ if [ $update_required = 0 ]; then
+ printf $message
+ update_required=1
+ fi
+
+ annotations_file="$pwd/annotations/$test.annot"
+ # Append only the new annotations
+ printf "$annotations" > "$annotations_file.temp"
+ [ ! -e "$annotations_file" ] && touch "$annotations_file"
+ comm -13 "$annotations_file" "$annotations_file.temp" >> \
+ "$annotations_file" && printf "\t%s\n" "annotations/$test.annot"
+ rm -f "$annotations_file.temp"
+ fi
+
+ break
+ fi
+
+ if [ "$status" = "failed:" ]; then
+ testcase=${testcase#"$test:"}
+ annotations="$annotations$testcase\n"
+ fi
+
+ i=$((i+1))
+ done
+ )
+
+done
+
+printf "==============================================================================\n\n"
Index: tools/tools/smoketestsuite/src/scripts/utils_list
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/scripts/utils_list
@@ -0,0 +1,1266 @@
+crypto/openssl/crypto/idea
+crypto/openssl/crypto/objects
+crypto/openssl/crypto/srp
+crypto/openssl/crypto/bf
+crypto/openssl/crypto/aes
+crypto/openssl/crypto/seed
+crypto/openssl/crypto/modes
+crypto/openssl/crypto/pkcs7
+crypto/openssl/crypto/rc5
+crypto/openssl/crypto/ecdh
+crypto/openssl/crypto/des
+crypto/openssl/crypto/ts
+crypto/openssl/crypto/pqueue
+crypto/openssl/crypto/buffer
+crypto/openssl/crypto/md2
+crypto/openssl/crypto/pkcs12
+crypto/openssl/crypto/md4
+crypto/openssl/crypto/sha
+crypto/openssl/crypto
+crypto/openssl/crypto/mdc2
+crypto/openssl/crypto/x509
+crypto/openssl/crypto/err
+crypto/openssl/crypto/cast
+crypto/openssl/crypto/stack
+crypto/openssl/crypto/cms
+crypto/openssl/crypto/rc2
+crypto/openssl/crypto/rsa
+crypto/openssl/crypto/bio
+crypto/openssl/crypto/lhash
+crypto/openssl/crypto/whrlpool
+crypto/openssl/crypto/asn1
+crypto/openssl/crypto/engine
+crypto/openssl/crypto/ripemd
+crypto/openssl/crypto/ecdsa
+crypto/openssl/crypto/dso
+crypto/openssl/crypto/camellia
+crypto/openssl/crypto/x509v3
+crypto/openssl/crypto/txt_db
+crypto/openssl/crypto/jpake
+crypto/openssl/crypto/hmac
+crypto/openssl/crypto/store
+crypto/openssl/crypto/rc4
+crypto/openssl/crypto/rand
+crypto/openssl/crypto/ui
+crypto/openssl/crypto/bn
+crypto/openssl/crypto/dh
+crypto/openssl/crypto/dsa
+crypto/openssl/crypto/cmac
+crypto/openssl/crypto/md5
+crypto/openssl/crypto/ec
+crypto/openssl/apps
+crypto/openssl/ssl
+crypto/openssl
+crypto/openssl/engines/ccgost
+crypto/openssl/engines
+crypto/openssh/regress/misc/kexfuzz
+crypto/openssh/regress/unittests/kex
+crypto/openssh/regress/unittests/match
+crypto/openssh/regress/unittests/sshbuf
+crypto/openssh/regress/unittests/hostkeys
+crypto/openssh/regress/unittests/bitmap
+crypto/openssh/regress/unittests/sshkey
+crypto/openssh/regress/unittests/utf8
+rescue/rescue
+libexec/fingerd
+libexec/talkd
+libexec/mail.local
+libexec/dma/dmagent
+libexec/dma/dma-mbox-create
+libexec/rpc.sprayd
+libexec/revnetgroup
+libexec/rtld-elf/tests/target
+libexec/rtld-elf
+libexec/rpc.rwalld
+libexec/rpc.rquotad
+libexec/ftpd
+libexec/tftp-proxy
+libexec/rlogind
+libexec/comsat
+libexec/getty
+libexec/rshd
+libexec/ulog-helper
+libexec/bootpd
+libexec/bootpd/tools/bootpef
+libexec/bootpd/tools/bootptest
+libexec/bootpd/bootpgw
+libexec/tftpd
+libexec/atrun
+libexec/mknetid
+libexec/tcpd
+libexec/rpc.rusersd
+libexec/smrsh
+libexec/atf/atf-check
+libexec/atf/atf-sh
+libexec/telnetd
+libexec/ypxfr
+libexec/rbootd
+libexec/pppoed
+libexec/rpc.rstatd
+bin/cp
+bin/rm
+bin/stty
+bin/sh
+bin/ln
+bin/setfacl
+bin/pkill
+bin/ls
+bin/chio
+bin/cat
+bin/chflags
+bin/domainname
+bin/pwait
+bin/hostname
+bin/ed
+bin/dd
+bin/realpath
+bin/csh
+bin/sleep
+bin/kill
+bin/uuidgen
+bin/sync
+bin/date
+bin/expr
+bin/getfacl
+bin/ps
+bin/echo
+bin/rmail
+bin/rmdir
+bin/test
+bin/chmod
+bin/mv
+bin/mkdir
+bin/pax
+bin/kenv
+bin/pwd
+bin/rcp
+bin/df
+tests/sys/netinet
+tests/sys/kqueue/libkqueue
+tests/sys/mqueue
+tests/sys/kern/execve
+tests/sys/kern
+tests/sys/file
+tests/sys/pjdfstest/pjdfstest
+tests/sys/acl
+tests/sys/fs/tmpfs
+sbin/ffsinfo
+sbin/badsect
+sbin/etherswitchcfg
+sbin/mount_cd9660
+sbin/nandfs
+sbin/init
+sbin/adjkerntz
+sbin/reboot
+sbin/dump
+sbin/iscontrol
+sbin/kldunload
+sbin/gbde
+sbin/ccdconfig
+sbin/devfs
+sbin/pflogd
+sbin/mknod
+sbin/pfctl
+sbin/mksnap_ffs
+sbin/growfs
+sbin/ldconfig
+sbin/comcontrol
+sbin/hastctl
+sbin/dhclient
+sbin/recoverdisk
+sbin/nfsiod
+sbin/ipfw
+sbin/ddb
+sbin/restore
+sbin/sconfig
+sbin/kldstat
+sbin/mount_nullfs
+sbin/mdconfig
+sbin/umount
+sbin/fsirand
+sbin/decryptcore
+sbin/sysctl
+sbin/newfs
+sbin/fsck
+sbin/mdmfs
+sbin/kldload
+sbin/mount_nfs
+sbin/dmesg
+sbin/ping6
+sbin/newfs_msdos
+sbin/fdisk
+sbin/rtsol
+sbin/bsdlabel
+sbin/sunlabel
+sbin/nvmecontrol
+sbin/clri
+sbin/mount_msdosfs
+sbin/nos-tun
+sbin/fsck_ffs
+sbin/route
+sbin/swapon
+sbin/ggate/ggated
+sbin/ggate/ggatec
+sbin/ggate/ggatel
+sbin/rcorder
+sbin/mount_fusefs
+sbin/quotacheck
+sbin/spppcontrol
+sbin/routed
+sbin/routed/rtquery
+sbin/gvinum
+sbin/dumpfs
+sbin/ping
+sbin/fsdb
+sbin/setkey
+sbin/savecore
+sbin/atm/atmconfig
+sbin/mount
+sbin/natd
+sbin/hastd
+sbin/camcontrol
+sbin/dumpon
+sbin/mount_udf
+sbin/ifconfig
+sbin/zfsbootcfg
+sbin/shutdown
+sbin/devd
+sbin/tunefs
+sbin/conscontrol
+sbin/newfs_nandfs
+sbin/geom
+sbin/geom/core
+sbin/kldconfig
+sbin/md5
+sbin/mount_unionfs
+sbin/fsck_msdosfs
+sbin/ipf/ipmon
+sbin/ipf/ipnat
+sbin/ipf/ipfs
+sbin/ipf/ipresend
+sbin/ipf/ippool
+sbin/ipf/ipftest
+sbin/ipf/ipfstat
+sbin/ipf/ipf
+secure/libexec/ssh-keysign
+secure/libexec/sftp-server
+secure/libexec/ssh-pkcs11-helper
+secure/usr.bin/sftp
+secure/usr.bin/ssh-keyscan
+secure/usr.bin/openssl
+secure/usr.bin/ssh-agent
+secure/usr.bin/ssh-keygen
+secure/usr.bin/ssh-add
+secure/usr.bin/ssh
+secure/usr.bin/scp
+secure
+secure/usr.sbin/sshd
+contrib/tzcode/zic/zdump
+contrib/tzcode/zic/zic
+contrib/ofed/librdmacm/examples/build/cmatose
+contrib/ofed/librdmacm/examples/build/rping
+contrib/ofed/librdmacm/examples/build/udaddy
+contrib/ofed/librdmacm/examples/build/mckey
+contrib/ofed/usr.bin/smpquery
+contrib/ofed/usr.bin/perfquery
+contrib/ofed/usr.bin/ibsendtrap
+contrib/ofed/usr.bin/smpdump
+contrib/ofed/usr.bin/ibportstate
+contrib/ofed/usr.bin/vendstat
+contrib/ofed/usr.bin/ibtracert
+contrib/ofed/usr.bin/ibnetdiscover
+contrib/ofed/usr.bin/ibroute
+contrib/ofed/usr.bin/ibstat
+contrib/ofed/usr.bin/saquery
+contrib/ofed/usr.bin/ibsysstat
+contrib/ofed/usr.bin/sminfo
+contrib/ofed/usr.bin/osmtest
+contrib/ofed/usr.bin/ibaddr
+contrib/ofed/usr.bin/opensm
+contrib/ofed/usr.bin/ibping
+contrib/ofed/libibverbs/examples/build/ud_pingpong
+contrib/ofed/libibverbs/examples/build/device_list
+contrib/ofed/libibverbs/examples/build/devinfo
+contrib/ofed/libibverbs/examples/build/asyncwatch
+contrib/ofed/libibverbs/examples/build/rc_pingpong
+contrib/ofed/libibverbs/examples/build/uc_pingpong
+contrib/ofed/libibverbs/examples/build/srq_pingpong
+contrib/bmake
+contrib/ldns-host
+contrib/mdocml
+contrib/pjdfstest
+contrib/vis
+contrib/pnpinfo
+contrib/unvis
+contrib/elftoolchain/strings
+contrib/elftoolchain/ar
+contrib/elftoolchain/size
+contrib/elftoolchain/cxxfilt
+contrib/elftoolchain/addr2line
+contrib/elftoolchain/elfcopy
+contrib/elftoolchain/elfdump
+contrib/elftoolchain/nm
+contrib/elftoolchain/readelf
+contrib/elftoolchain/brandelf
+contrib/smbfs/smbutil
+contrib/mtree
+contrib/blacklist/bin
+contrib/blacklist/test
+share/examples/hwpmc
+share/examples/perfmon
+share/examples/ppi
+share/examples/FreeBSD_version
+share/examples/kld/cdev/test
+share/examples/kld/syscall/test
+share/examples/find_interface
+share/examples/scsi_target
+share/examples/libvgl
+share/examples/ses/getencstat
+share/examples/ses/setencstat
+share/examples/ses/sesd
+share/examples/ses/setobjstat
+usr.bin/tset
+usr.bin/usbhidaction
+usr.bin/ctlstat
+usr.bin/mesg
+usr.bin/tip/tip
+usr.bin/lam
+usr.bin/mkesdb
+usr.bin/systat
+usr.bin/ftp
+usr.bin/perror
+usr.bin/find
+usr.bin/renice
+usr.bin/tty
+usr.bin/printf
+usr.bin/mandoc
+usr.bin/xo
+usr.bin/strings
+usr.bin/opiekey
+usr.bin/apply
+usr.bin/expand
+usr.bin/timeout
+usr.bin/awk
+usr.bin/clang/llvm-lto2
+usr.bin/clang/lli
+usr.bin/clang/clang
+usr.bin/clang/llvm-xray
+usr.bin/clang/bugpoint
+usr.bin/clang/llvm-rtdyld
+usr.bin/clang/llvm-cxxdump
+usr.bin/clang/llc
+usr.bin/clang/llvm-profdata
+usr.bin/clang/llvm-lto
+usr.bin/clang/llvm-objdump
+usr.bin/clang/llvm-modextract
+usr.bin/clang/llvm-nm
+usr.bin/clang/llvm-symbolizer
+usr.bin/clang/llvm-tblgen
+usr.bin/clang/llvm-dis
+usr.bin/clang/llvm-diff
+usr.bin/clang/llvm-bcanalyzer
+usr.bin/clang/llvm-extract
+usr.bin/clang/llvm-link
+usr.bin/clang/llvm-mc
+usr.bin/clang/lldb
+usr.bin/clang/clang-tblgen
+usr.bin/clang/llvm-dwarfdump
+usr.bin/clang/clang-format
+usr.bin/clang/llvm-as
+usr.bin/clang/opt
+usr.bin/clang/llvm-cxxfilt
+usr.bin/clang/lld
+usr.bin/clang/llvm-cov
+usr.bin/clang/llvm-pdbdump
+usr.bin/clang/llvm-ar
+usr.bin/ar
+usr.bin/elf2aout
+usr.bin/uname
+usr.bin/rup
+usr.bin/unzip
+usr.bin/gcore
+usr.bin/compress
+usr.bin/rpcgen
+usr.bin/caesar
+usr.bin/tput
+usr.bin/du
+usr.bin/basename
+usr.bin/seq
+usr.bin/fstat
+usr.bin/smbutil
+usr.bin/iconv
+usr.bin/wall
+usr.bin/mt
+usr.bin/less
+usr.bin/dpv
+usr.bin/mkfifo
+usr.bin/primes
+usr.bin/pamtest
+usr.bin/head
+usr.bin/rpcinfo
+usr.bin/colldef
+usr.bin/chpass
+usr.bin/mktemp
+usr.bin/size
+usr.bin/grep
+usr.bin/showmount
+usr.bin/top
+usr.bin/gencat
+usr.bin/procstat
+usr.bin/logins
+usr.bin/mkcsmapper_static
+usr.bin/cxxfilt
+usr.bin/rsh
+usr.bin/bmake
+usr.bin/file2c
+usr.bin/rev
+usr.bin/catman
+usr.bin/col
+usr.bin/uudecode
+usr.bin/pr
+usr.bin/iscsictl
+usr.bin/column
+usr.bin/chkey
+usr.bin/mkstr
+usr.bin/yacc
+usr.bin/which
+usr.bin/nfsstat
+usr.bin/indent
+usr.bin/sockstat
+usr.bin/env
+usr.bin/newkey
+usr.bin/ident
+usr.bin/join
+usr.bin/c89
+usr.bin/w
+usr.bin/from
+usr.bin/rwall
+usr.bin/fortune/strfile
+usr.bin/fortune/unstr
+usr.bin/fortune/fortune
+usr.bin/yes
+usr.bin/leave
+usr.bin/mkesdb_static
+usr.bin/mkuzip
+usr.bin/nc
+usr.bin/fmt
+usr.bin/ul
+usr.bin/stat
+usr.bin/tcopy
+usr.bin/keylogout
+usr.bin/bzip2
+usr.bin/biff
+usr.bin/opieinfo
+usr.bin/truncate
+usr.bin/uuencode
+usr.bin/ee
+usr.bin/random
+usr.bin/bsdiff/bspatch
+usr.bin/bsdiff/bsdiff
+usr.bin/addr2line
+usr.bin/protect
+usr.bin/colrm
+usr.bin/vgrind
+usr.bin/m4
+usr.bin/unifdef
+usr.bin/locale
+usr.bin/csplit
+usr.bin/proccontrol
+usr.bin/su
+usr.bin/bc
+usr.bin/elfcopy
+usr.bin/touch
+usr.bin/c99
+usr.bin/resizewin
+usr.bin/rctl
+usr.bin/morse
+usr.bin/gzip
+usr.bin/elfdump
+usr.bin/ipcrm
+usr.bin/rwho
+usr.bin/sort
+usr.bin/logger
+usr.bin/bluetooth/rfcomm_sppd
+usr.bin/bluetooth/btsockstat
+usr.bin/bluetooth/bthost
+usr.bin/mkcsmapper
+usr.bin/opiepasswd
+usr.bin/at
+usr.bin/paste
+usr.bin/banner
+usr.bin/newgrp
+usr.bin/locate/bigram
+usr.bin/locate/code
+usr.bin/locate/locate
+usr.bin/sed
+usr.bin/nohup
+usr.bin/split
+usr.bin/killall
+usr.bin/ypcat
+usr.bin/vmstat
+usr.bin/printenv
+usr.bin/asa
+usr.bin/uniq
+usr.bin/ktrace
+usr.bin/nm
+usr.bin/compile_et
+usr.bin/revoke
+usr.bin/getaddrinfo
+usr.bin/readelf
+usr.bin/vacation
+usr.bin/false
+usr.bin/usbhidctl
+usr.bin/jot
+usr.bin/checknr
+usr.bin/localedef
+usr.bin/lessecho
+usr.bin/xzdec
+usr.bin/patch
+usr.bin/tar/tests
+usr.bin/tar
+usr.bin/tr
+usr.bin/colcrt
+usr.bin/users
+usr.bin/soelim
+usr.bin/tee
+usr.bin/vi
+usr.bin/hesinfo
+usr.bin/mail
+usr.bin/lastcomm
+usr.bin/rusers
+usr.bin/bzip2recover
+usr.bin/rs
+usr.bin/mklocale
+usr.bin/fold
+usr.bin/rlogin
+usr.bin/last
+usr.bin/diff
+usr.bin/dtc
+usr.bin/ministat
+usr.bin/mkimg
+usr.bin/sdiff
+usr.bin/file
+usr.bin/number
+usr.bin/what
+usr.bin/time
+usr.bin/ncal
+usr.bin/svn/svnadmin
+usr.bin/svn/svnserve
+usr.bin/svn/svnrdump
+usr.bin/svn/svnsync
+usr.bin/svn/svndumpfilter
+usr.bin/svn/svnfsfs
+usr.bin/svn/svnversion
+usr.bin/svn/svnlook
+usr.bin/svn/svn
+usr.bin/svn/svnmucc
+usr.bin/svn/svnbench
+usr.bin/limits
+usr.bin/ldd
+usr.bin/tail
+usr.bin/dirname
+usr.bin/getent
+usr.bin/who
+usr.bin/whois
+usr.bin/netstat
+usr.bin/nice
+usr.bin/vtfontcvt
+usr.bin/look
+usr.bin/minigzip
+usr.bin/cmp
+usr.bin/script
+usr.bin/ypmatch
+usr.bin/vis
+usr.bin/setchannel
+usr.bin/grdc
+usr.bin/xinstall
+usr.bin/stdbuf
+usr.bin/atm/sscop
+usr.bin/fetch
+usr.bin/cap_mkdb
+usr.bin/drill
+usr.bin/bsdcat/tests
+usr.bin/bsdcat
+usr.bin/tabs
+usr.bin/ipcs
+usr.bin/unvis
+usr.bin/fsync
+usr.bin/tsort
+usr.bin/comm
+usr.bin/wc
+usr.bin/truss
+usr.bin/ruptime
+usr.bin/true
+usr.bin/ctags
+usr.bin/hexdump
+usr.bin/tftp
+usr.bin/passwd
+usr.bin/ypwhich
+usr.bin/keylogin
+usr.bin/host
+usr.bin/brandelf
+usr.bin/gprof
+usr.bin/whereis
+usr.bin/numactl
+usr.bin/factor
+usr.bin/finger
+usr.bin/write
+usr.bin/lesskey
+usr.bin/telnet
+usr.bin/lzmainfo
+usr.bin/dc
+usr.bin/chat
+usr.bin/cpuset
+usr.bin/pom
+usr.bin/calendar
+usr.bin/xstr
+usr.bin/xz
+usr.bin/enigma
+usr.bin/units
+usr.bin/talk
+usr.bin/lex
+usr.bin/cut
+usr.bin/xargs
+usr.bin/getconf
+usr.bin/unexpand
+usr.bin/lockf
+usr.bin/ktrdump
+usr.bin/msgs
+usr.bin/pathchk
+usr.bin/login
+usr.bin/id
+usr.bin/getopt
+usr.bin/lsvfs
+usr.bin/kdump
+usr.bin/nl
+usr.bin/xlint/lint1
+usr.bin/xlint/lint2
+usr.bin/xlint/xlint
+usr.bin/cksum
+usr.bin/cpio/tests
+usr.bin/cpio
+usr.bin/quota
+usr.bin/logname
+usr.bin/lock
+
+usr.sbin/clear_locks
+usr.sbin/chroot
+usr.sbin/yppush
+usr.sbin/praliases
+usr.sbin/pwd_mkdb
+usr.sbin/tcpdchk
+usr.sbin/rpc.yppasswdd
+usr.sbin/rpc.umntall
+usr.sbin/iostat
+usr.sbin/gpioctl
+usr.sbin/ngctl
+usr.sbin/mount_smbfs
+usr.sbin/rtsold
+usr.sbin/bsdinstall/distextract
+usr.sbin/bsdinstall/partedit
+usr.sbin/bsdinstall/distfetch
+usr.sbin/wpa/wpa_passphrase
+usr.sbin/wpa/ndis_events
+usr.sbin/wpa/wpa_cli
+usr.sbin/wpa/wpa_supplicant
+usr.sbin/wpa/wpa_priv
+usr.sbin/wpa/hostapd_cli
+usr.sbin/wpa/hostapd
+usr.sbin/bhyve
+usr.sbin/mld6query
+usr.sbin/iscsid
+usr.sbin/tcpdmatch
+usr.sbin/vipw
+usr.sbin/getpmac
+usr.sbin/rpc.ypupdated
+usr.sbin/fdwrite
+usr.sbin/nmtree
+usr.sbin/ipfwpcap
+usr.sbin/praudit
+usr.sbin/chkgrp
+usr.sbin/authpf
+usr.sbin/chown
+usr.sbin/kldxref
+usr.sbin/unbound/checkconf
+usr.sbin/unbound/anchor
+usr.sbin/unbound/daemon
+usr.sbin/unbound/control
+usr.sbin/camdd
+usr.sbin/ac
+usr.sbin/prometheus_sysctl_exporter
+usr.sbin/cdcontrol
+usr.sbin/sendmail
+usr.sbin/kbdmap
+usr.sbin/accton
+usr.sbin/setpmac
+usr.sbin/kgzip
+usr.sbin/rpc.ypxfrd
+usr.sbin/newsyslog
+usr.sbin/usbconfig
+usr.sbin/crunch/crunchide
+usr.sbin/crunch/examples
+usr.sbin/crunch/crunchgen
+usr.sbin/makefs
+usr.sbin/timed/timed
+usr.sbin/timed/timedc
+usr.sbin/setfmac
+usr.sbin/ypbind
+usr.sbin/ip6addrctl
+usr.sbin/ckdist
+usr.sbin/ypserv
+usr.sbin/devinfo
+usr.sbin/watch
+usr.sbin/uefisign
+usr.sbin/hyperv/tools/vss
+usr.sbin/hyperv/tools/kvp
+usr.sbin/fmtree
+usr.sbin/zonectl
+usr.sbin/watchdogd
+usr.sbin/jls
+usr.sbin/pmcstudy
+usr.sbin/bootparamd/callbootd
+usr.sbin/bootparamd/bootparamd
+usr.sbin/usbdump
+usr.sbin/route6d
+usr.sbin/uhsoctl
+usr.sbin/fwcontrol
+usr.sbin/mlxcontrol
+usr.sbin/rtprio
+usr.sbin/rip6query
+usr.sbin/auditdistd
+usr.sbin/fstyp
+usr.sbin/cxgbetool
+usr.sbin/dconschat
+usr.sbin/traceroute
+usr.sbin/ndiscvt
+usr.sbin/sesutil
+usr.sbin/ctm/ctm_rmail
+usr.sbin/ctm/ctm
+usr.sbin/ctm/mkCTM
+usr.sbin/ctm/ctm_dequeue
+usr.sbin/ctm/ctm_smail
+usr.sbin/syslogd
+usr.sbin/daemon
+usr.sbin/cron/cron
+usr.sbin/cron/crontab
+usr.sbin/pciconf
+usr.sbin/ofwdump
+usr.sbin/rtadvd
+usr.sbin/devctl
+usr.sbin/ctladm
+usr.sbin/mailwrapper
+usr.sbin/pkg
+usr.sbin/getfmac
+usr.sbin/mpsutil
+usr.sbin/binmiscctl
+usr.sbin/powerd
+usr.sbin/vidcontrol
+usr.sbin/lpr/chkprintcap
+usr.sbin/lpr/filters.ru/koi2855
+usr.sbin/lpr/filters.ru/koi2alt
+usr.sbin/lpr/filters
+usr.sbin/lpr/lpr
+usr.sbin/lpr/lpc
+usr.sbin/lpr/lptest
+usr.sbin/lpr/lpq
+usr.sbin/lpr/pac
+usr.sbin/lpr/lprm
+usr.sbin/lpr/lpd
+usr.sbin/yppoll
+usr.sbin/btxld
+usr.sbin/nandtool
+usr.sbin/dumpcis
+usr.sbin/rtadvctl
+usr.sbin/kbdcontrol
+usr.sbin/nologin
+usr.sbin/ugidfw
+usr.sbin/rpcbind
+usr.sbin/auditd
+usr.sbin/pstat
+usr.sbin/smbmsg
+usr.sbin/bluetooth/btpand
+usr.sbin/bluetooth/bcmfw
+usr.sbin/bluetooth/l2control
+usr.sbin/bluetooth/bt3cfw
+usr.sbin/bluetooth/sdpcontrol
+usr.sbin/bluetooth/hcsecd
+usr.sbin/bluetooth/bthidd
+usr.sbin/bluetooth/hcseriald
+usr.sbin/bluetooth/rfcomm_pppd
+usr.sbin/bluetooth/ath3kfw
+usr.sbin/bluetooth/bthidcontrol
+usr.sbin/bluetooth/hccontrol
+usr.sbin/bluetooth/l2ping
+usr.sbin/bluetooth/sdpd
+usr.sbin/lmcconfig
+usr.sbin/editmap
+usr.sbin/nandsim
+usr.sbin/pmccontrol
+usr.sbin/blacklistd
+usr.sbin/fdread
+usr.sbin/boot0cfg
+usr.sbin/cpucontrol
+usr.sbin/fdcontrol
+usr.sbin/mixer
+usr.sbin/auditreduce
+usr.sbin/mptable
+usr.sbin/apmd
+usr.sbin/nfscbd
+usr.sbin/repquota
+usr.sbin/ypldap
+usr.sbin/lastlogin
+usr.sbin/bhyvectl
+usr.sbin/nfsuserd
+usr.sbin/pppctl
+usr.sbin/wake
+usr.sbin/flowctl
+usr.sbin/asf
+usr.sbin/eeprom
+usr.sbin/rwhod
+usr.sbin/arp
+usr.sbin/ancontrol
+usr.sbin/amd/fsinfo
+usr.sbin/amd/mk-amd-map
+usr.sbin/amd/amq
+usr.sbin/amd/pawd
+usr.sbin/amd/fixmount
+usr.sbin/amd/amd
+usr.sbin/amd/hlfsd
+usr.sbin/amd/wire-test
+usr.sbin/fifolog/fifolog_writer
+usr.sbin/fifolog/fifolog_create
+usr.sbin/fifolog/fifolog_reader
+usr.sbin/ypset
+usr.sbin/blacklistctl
+usr.sbin/wlandebug
+usr.sbin/gssd
+usr.sbin/mptutil
+usr.sbin/audit
+usr.sbin/keyserv
+usr.sbin/spray
+usr.sbin/jexec
+usr.sbin/pmcannotate
+usr.sbin/nfsrevoke
+usr.sbin/acpi/acpiconf
+usr.sbin/acpi/acpidb
+usr.sbin/acpi/iasl
+usr.sbin/acpi/acpidump
+usr.sbin/gstat
+usr.sbin/rrenumd
+usr.sbin/trpt
+usr.sbin/memcontrol
+usr.sbin/edquota
+usr.sbin/snapinfo
+usr.sbin/config
+usr.sbin/inetd
+usr.sbin/jail
+usr.sbin/efidp
+usr.sbin/portsnap/phttpget
+usr.sbin/portsnap/make_index
+usr.sbin/rpc.lockd
+usr.sbin/apm
+usr.sbin/utx
+usr.sbin/digictl
+usr.sbin/ftp-proxy
+usr.sbin/pnpinfo
+usr.sbin/rarpd
+usr.sbin/tcpdrop
+usr.sbin/yp_mkdb
+usr.sbin/zic/zdump
+usr.sbin/zic/zic
+usr.sbin/quot
+usr.sbin/ifmcstat
+usr.sbin/services_mkdb
+usr.sbin/ppp
+usr.sbin/kgmon
+usr.sbin/nscd
+usr.sbin/uathload
+usr.sbin/nfsdumpstate
+usr.sbin/lptcontrol
+usr.sbin/bsnmpd/gensnmptree
+usr.sbin/bsnmpd/tools/bsnmptools
+usr.sbin/bsnmpd/bsnmpd
+usr.sbin/moused
+usr.sbin/i2c
+usr.sbin/rpc.statd
+usr.sbin/traceroute6
+usr.sbin/sa
+usr.sbin/nghook
+usr.sbin/tzsetup
+usr.sbin/extattr
+usr.sbin/makemap
+usr.sbin/iovctl
+usr.sbin/procctl
+usr.sbin/ctld
+usr.sbin/ndp
+usr.sbin/fdformat
+usr.sbin/nvram
+usr.sbin/setfib
+usr.sbin/bhyveload
+usr.sbin/ntp/sntp
+usr.sbin/ntp/ntpdate
+usr.sbin/ntp/ntpd
+usr.sbin/ntp/ntptime
+usr.sbin/ntp/ntpq
+usr.sbin/ntp/ntpdc
+usr.sbin/ntp/ntp-keygen
+usr.sbin/diskinfo
+usr.sbin/mountd
+usr.sbin/nfsd
+usr.sbin/autofs
+usr.sbin/mtest
+usr.sbin/pw/tests
+usr.sbin/pw
+usr.sbin/efivar
+usr.sbin/mfiutil
+usr.sbin/extattrctl
+usr.sbin/tcpdump/tcpdump
+usr.sbin/pmcstat
+usr.sbin/mailstats
+usr.sbin/quotaon
+usr.sbin/rmt
+kerberos5/libexec/hprop
+kerberos5/libexec/kadmind
+kerberos5/libexec/kpasswdd
+kerberos5/libexec/digest-service
+kerberos5/libexec/ipropd-master
+kerberos5/libexec/hpropd
+kerberos5/libexec/kdc
+kerberos5/libexec/kfd
+kerberos5/libexec/ipropd-slave
+kerberos5/libexec/kcm
+kerberos5/libexec/kimpersonate
+kerberos5/libexec/kdigest
+kerberos5/usr.bin/kcc
+kerberos5/usr.bin/kpasswd
+kerberos5/usr.bin/verify_krb5_conf
+kerberos5/usr.bin/kadmin
+kerberos5/usr.bin/string2key
+kerberos5/usr.bin/kf
+kerberos5/usr.bin/kgetcred
+kerberos5/usr.bin/ksu
+kerberos5/usr.bin/kdestroy
+kerberos5/usr.bin/kinit
+kerberos5/usr.bin/hxtool
+kerberos5
+kerberos5/usr.sbin/ktutil
+kerberos5/usr.sbin/iprop-log
+kerberos5/usr.sbin/kstash
+kerberos5/tools/make-roken
+kerberos5/tools/asn1_compile
+kerberos5/tools/slc
+sys/crypto/rijndael
+sys/modules/aic7xxx/aicasm
+sys/contrib/xz-embedded/userspace
+sys/dev/patm/genrtab
+sys/dev/aic7xxx/aicasm
+sys/boot/i386/gptzfsboot
+sys/boot/i386/pmbr
+sys/boot/i386/mbr
+sys/boot/i386/cdboot
+sys/boot/i386/btx/btx
+sys/boot/i386/btx/btxldr
+sys/boot/i386/btx/lib
+sys/boot/i386/gptboot
+sys/boot/i386/zfsboot
+sys/boot/i386/kgzldr
+sys/boot/i386/boot0
+sys/boot/i386/loader
+sys/boot/i386/boot0sio
+sys/boot/i386/pxeldr
+sys/boot/powerpc/ps3
+sys/boot/powerpc/ofw
+sys/boot/powerpc/boot1.chrp
+sys/boot/powerpc/kboot
+sys/boot/powerpc/uboot
+sys/boot/zfs
+sys/boot/ficl
+sys/boot/mips/beri/loader
+sys/boot/efi/boot1
+sys/boot/efi/loader
+sys/boot/userboot/ficl
+sys/boot/userboot/test
+sys/boot/usb
+sys/boot/usb/tools
+sys/boot/sparc64/boot1
+sys/boot/sparc64/zfsloader
+sys/boot/sparc64/zfsboot
+sys/boot/sparc64/loader
+sys/teken/stress
+sys/teken/demo
+cddl/sbin/zfs
+cddl/sbin/zpool
+cddl/usr.bin/zstreamdump
+cddl/usr.bin/ctfconvert
+cddl/usr.bin/ctfmerge
+cddl/usr.bin/ctfdump
+cddl/usr.bin/zinject
+cddl/usr.bin/ztest
+cddl/usr.bin/zlook
+cddl/usr.sbin/zdb
+cddl/usr.sbin/zfsd
+cddl/usr.sbin/plockstat
+cddl/usr.sbin/zhack
+cddl/usr.sbin/lockstat
+cddl/usr.sbin/dtrace
+tools/KSE/rr
+tools/KSE/ksetest
+tools/bsdbox
+tools/tools/netrate/httpd
+tools/tools/netrate/http
+tools/tools/netrate/netblast
+tools/tools/netrate/tcpreceive
+tools/tools/netrate/tcpconnect
+tools/tools/netrate/tcpp
+tools/tools/netrate/netreceive
+tools/tools/netrate/netsend
+tools/tools/netrate/juggle
+tools/tools/pciroms
+tools/tools/bus_autoconf
+tools/tools/decioctl
+tools/tools/qrndtest
+tools/tools/ath/athregs
+tools/tools/ath/arcode
+tools/tools/ath/ath_prom_read
+tools/tools/ath/ath_ee_v14_print
+tools/tools/ath/athspectral
+tools/tools/ath/athratestats
+tools/tools/ath/athpoke
+tools/tools/ath/athdebug
+tools/tools/ath/athpow
+tools/tools/ath/athradar
+tools/tools/ath/athrd
+tools/tools/ath/athalq
+tools/tools/ath/athstats
+tools/tools/ath/athdecode
+tools/tools/ath/athaggrstats
+tools/tools/ath/athprom
+tools/tools/ath/athkey
+tools/tools/ath/ath_ee_9287_print
+tools/tools/ath/ath_ee_v4k_print
+tools/tools/ath/ath_ee_9300_print
+tools/tools/ath/athsurvey
+tools/tools/npe/npestats
+tools/tools/zfsboottest
+tools/tools/iwn/iwnstats
+tools/tools/fixwhite
+tools/tools/find-sb
+tools/tools/indent_wrapper
+tools/tools/vxge
+tools/tools/aac
+tools/tools/ioat
+tools/tools/wtap/wtap
+tools/tools/wtap/vis_map
+tools/tools/umastat
+tools/tools/tionxcl
+tools/tools/cxgbtool
+tools/tools/atsectl
+tools/tools/ifinfo
+tools/tools/gensnmpdef
+tools/tools/nxge
+tools/tools/usbtest
+tools/tools/dmardump
+tools/tools/vimage
+tools/tools/ether_reflect
+tools/tools/net80211/wlanwds
+tools/tools/net80211/wlantxtime
+tools/tools/net80211/w00t/redir
+tools/tools/net80211/w00t/expand
+tools/tools/net80211/w00t/assoc
+tools/tools/net80211/w00t/ap
+tools/tools/net80211/w00t/prga
+tools/tools/net80211/wesside/dics
+tools/tools/net80211/wesside/wesside
+tools/tools/net80211/wesside/udps
+tools/tools/net80211/wlanstats
+tools/tools/net80211/wlanwatch
+tools/tools/net80211/wlaninject
+tools/tools/net80211/stumbler
+tools/tools/cfi
+tools/tools/syscall_timing
+tools/tools/iwi
+tools/tools/gdb_regofs
+tools/tools/bootparttest
+tools/tools/netmap
+tools/tools/ncpus
+tools/tools/mfi
+tools/tools/tscdrift
+tools/tools/vt/mkkfont
+tools/tools/kttcp
+tools/tools/mwl/mwlstats
+tools/tools/mwl/mwldebug
+tools/tools/ipw
+tools/tools/pirtool
+tools/tools/ifpifa
+tools/tools/mcgrab
+tools/tools/mctest
+tools/tools/drm/radeon/mkregtable
+tools/test/iconv/tablegen
+tools/test/iconv/posix
+tools/test/iconv/refgen
+tools/test/iconv/gnu
+tools/test/net
+tools/test/ppsapi
+tools/test/ptrace
+tools/test/malloc
+tools/test/auxinfo
+tools/test/netfibs
+tools/test/upsdl
+tools/regression/posixsem2
+tools/regression/pthread/cv_cancel1
+tools/regression/pthread/mutex_isowned_np
+tools/regression/netinet/msocket_ifnet_remove
+tools/regression/netinet/tcpsocktimewait
+tools/regression/netinet/ipmulticast
+tools/regression/netinet/rawconnect
+tools/regression/netinet/udpzerobyte
+tools/regression/netinet/tcpfullwindowrst
+tools/regression/netinet/tcpconnect
+tools/regression/netinet/msocket
+tools/regression/netinet/tcpstream
+tools/regression/netinet/arphold
+tools/regression/netinet/ipdivert
+tools/regression/netinet/ipsockopt
+tools/regression/netinet/udpconnectjail
+tools/regression/netinet/tcpdrop
+tools/regression/netinet/ipbroadcast
+tools/regression/netinet/tcpsockclosebeforeaccept
+tools/regression/environ
+tools/regression/tls/ttls1
+tools/regression/tls/ttls4
+tools/regression/tls/ttls2
+tools/regression/kgssapi
+tools/regression/priv
+tools/regression/nfsmmap/test1
+tools/regression/nfsmmap/test2
+tools/regression/sysvsem
+tools/regression/netinet6/icmp6_filter
+tools/regression/netinet6/inet6_rth
+tools/regression/netinet6/ip6_sockets
+tools/regression/tmpfs
+tools/regression/usr.bin/env
+tools/regression/rpcsec_gss
+tools/regression/sysvmsg
+tools/regression/gaithrstress
+tools/regression/aio/aiop
+tools/regression/fsx
+tools/regression/sigqueue/sigqtest1
+tools/regression/sigqueue/sigqtest2
+tools/regression/include/stdatomic
+tools/regression/include/tgmath
+tools/regression/ufs/uprintf
+tools/regression/sockets/sigpipe
+tools/regression/sockets/unix_bindconnect
+tools/regression/sockets/udp_pingpong
+tools/regression/sockets/fstat
+tools/regression/sockets/kqueue
+tools/regression/sockets/unix_sorflush
+tools/regression/sockets/zerosend
+tools/regression/sockets/pr_atomic
+tools/regression/sockets/unix_gc
+tools/regression/sockets/unix_sendtorace
+tools/regression/sockets/accept_fd_leak
+tools/regression/sockets/listenclose
+tools/regression/sockets/accf_data_attach
+tools/regression/sockets/reconnect
+tools/regression/sockets/unix_cmsg
+tools/regression/sockets/so_setfib
+tools/regression/sockets/listen_backlog
+tools/regression/sockets/unix_socket
+tools/regression/sockets/socketpair
+tools/regression/sockets/rtsocket
+tools/regression/sockets/sendfile
+tools/regression/sockets/unix_close_race
+tools/regression/sockets/sblock
+tools/regression/sockets/shutdown
+tools/regression/sockets/listen_kqueue
+tools/regression/doat
+tools/regression/security/cap_test
+tools/regression/security/open_to_operation
+tools/regression/security/proc_to_proc
+tools/regression/security/access
+tools/regression/poll
+tools/regression/audit/audit_pipe_ioctl
+tools/regression/p1003_1b
+tools/regression/posixsem
+tools/regression/sysvshm
+tools/regression/mlock
+tools/regression/geom/MdLoad
+tools/regression/geom/ConfCmp
+tools/regression/ethernet/ethermulti
+tools/diag/localeck
+tools/diag/dumpvfscache
+release/picobsd/tinyware/sps
+release/picobsd/tinyware/aps
+release/picobsd/tinyware/vm
+release/picobsd/tinyware/view
+release/picobsd/tinyware/msg
+release/picobsd/tinyware/oinit
+release/picobsd/tinyware/msh
+release/picobsd/tinyware/ns
+release/picobsd/tinyware/passwd
+release/picobsd/tinyware/login
+release/picobsd/tinyware/simple_httpd
+release/picobsd/tinyware/help
+gnu/usr.bin/diff3
+gnu/usr.bin/grep
+gnu/usr.bin/groff/src/utils/indxbib
+gnu/usr.bin/groff/src/utils/addftinfo
+gnu/usr.bin/groff/src/utils/lookbib
+gnu/usr.bin/groff/src/utils/tfmtodit
+gnu/usr.bin/groff/src/utils/lkbib
+gnu/usr.bin/groff/src/utils/pfbtops
+gnu/usr.bin/groff/src/utils/hpftodit
+gnu/usr.bin/groff/src/devices/grops
+gnu/usr.bin/groff/src/devices/grolbp
+gnu/usr.bin/groff/src/devices/grohtml
+gnu/usr.bin/groff/src/devices/grolj4
+gnu/usr.bin/groff/src/devices/grotty
+gnu/usr.bin/groff/src/devices/grodvi
+gnu/usr.bin/groff/src/roff/groff
+gnu/usr.bin/groff/src/roff/troff
+gnu/usr.bin/groff/src/preproc/html
+gnu/usr.bin/groff/src/preproc/grn
+gnu/usr.bin/groff/src/preproc/tbl
+gnu/usr.bin/groff/src/preproc/pic
+gnu/usr.bin/groff/src/preproc/refer
+gnu/usr.bin/groff/src/preproc/eqn
+gnu/usr.bin/dialog
+gnu/usr.bin/cc/cc_tools
+gnu/usr.bin/cc/cc1plus
+gnu/usr.bin/cc/cc1
+gnu/usr.bin/cc/gcov
+gnu/usr.bin/cc/cpp
+gnu/usr.bin/cc/cc
+gnu/usr.bin/cc/c++
+gnu/usr.bin/diff
+gnu/usr.bin/dtc
+gnu/usr.bin/gdb/gdbserver
+gnu/usr.bin/gdb/gdb
+gnu/usr.bin/gdb/kgdb
+gnu/usr.bin/gdb/gdbtui
+gnu/usr.bin/binutils/objcopy
+gnu/usr.bin/binutils/ld
+gnu/usr.bin/binutils/objdump
+gnu/usr.bin/binutils/as
+gnu/usr.bin/gperf
+lib/ncurses/ncurses
+lib/libproc/tests
+lib/msun/tests
+lib/libblacklist
+lib/libc/tests/stdlib
+lib/libc/tests/net
+lib/libc/tests/net/getaddrinfo
+lib/libc/tests/gen
+lib/libc/tests/gen/posix_spawn
+lib/libc/tests/regex
+lib/libc/tests/db
+lib/libc/tests/hash
+lib/libc/tests/ssp
+lib/libc/tests/stdio
+lib/libc/db/test
+lib/libthr/tests
+lib/libarchive/tests
+lib/libxo/tests
+lib/atf/libatf-c++/tests/detail
+lib/atf/libatf-c/tests/detail
Index: tools/tools/smoketestsuite/src/scripts/validate.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/scripts/validate.sh
@@ -0,0 +1,56 @@
+#!/bin/sh
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# Script for validating side-effects of newly introduced changes.
+
+remote=origin
+
+git checkout testing
+if [ $? != 0 ]; then
+ git fetch && git checkout $remote/testing && git checkout -b testing
+fi
+
+git rebase master
+
+# Generate tests
+make
+echo 'y\n' | make run
+
+git status | grep -q generated_tests
+if [ $? != 0 ]; then
+ echo "
+ +----------------------------------+
+ | New changes have no side-effects |
+ +----------------------------------+
+ "
+ git checkout master
+else
+ echo "
+ +--------------------------------------------------+
+ | New changes introduced side-effects. Stopping... |
+ +--------------------------------------------------+
+ "
+fi
Index: tools/tools/smoketestsuite/src/utils.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/utils.h
@@ -0,0 +1,51 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <list>
+#include <unordered_map>
+
+namespace utils {
+
+ // Option relation which maps option names to
+ // a unique identifier in their description.
+ typedef struct opt_rel {
+ char type; // Option type: (s)short/(l)long.
+ std::string value; // Name of the option.
+ std::string keyword; // The keyword which should be looked up in the
+ // message (if) produced when using this option.
+ } opt_rel;
+
+ class opt_def {
+ public:
+ std::list<std::string> opt_list; // list of all the accepted options with unknown usage.
+ std::unordered_map<std::string, opt_rel> opt_map; // Map "option value" to "option definition".
+ std::unordered_map<std::string, opt_rel>::iterator opt_map_iter;
+
+ void insert_opts();
+ std::list<opt_rel*> check_opts(std::string);
+ };
+}
Index: tools/tools/smoketestsuite/src/utils.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/utils.cpp
@@ -0,0 +1,130 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <fstream>
+#include <iostream>
+#include <cstdlib>
+#include "utils.h"
+
+// Insert a list of user-defined option definitions
+// into a hashmap. These specific option definitions
+// are the ones which one can be easily tested.
+void
+utils::opt_def::insert_opts()
+{
+ // Option definitions.
+ opt_rel h_def; // '-h'
+ h_def.type = 's';
+ h_def.value = "h";
+ h_def.keyword = "help";
+
+ opt_rel v_def; // '-v'
+ v_def.type = 's';
+ v_def.value = "v";
+ v_def.keyword = "version";
+
+ // `opt_map` contains all the options
+ // which can be "easily" tested.
+ opt_map.insert(std::make_pair<std::string, opt_rel>
+ ("h", (opt_rel)h_def));
+ opt_map.insert(std::make_pair<std::string, opt_rel>
+ ("v", (opt_rel)v_def));
+};
+
+// For the utility under test, find the supported options
+// present in the hashmap generated by insert_opts()
+// and return them in a form of list of option relations.
+std::list<utils::opt_rel*>
+utils::opt_def::check_opts(std::string utility)
+{
+ std::string line; // An individual line in a man-page.
+ std::string opt_name; // Name of the option.
+ std::string opt_ident = ".It Fl"; // Identifier for an option in man page.
+ std::string buffer; // Option description extracted from man-page.
+ std::string opt_string; // Identified option names.
+ int opt_pos; // Starting index of the (identified) option.
+ int space_index; // First occurrence of space character
+ // in a multi-word option definition.
+ std::list<opt_rel*> ident_opt_list; // List of identified option relations (opt_rel's).
+
+ // Generate the hashmap opt_map.
+ insert_opts();
+
+ // TODO: Section number cannot be hardcoded.
+ std::ifstream infile("groff/" + utility + ".1");
+
+ // Search for all the options accepted by the
+ // utility and collect those present in `opt_map`.
+ while (std::getline(infile, line)) {
+ if ((opt_pos = line.find(opt_ident)) != std::string::npos) {
+ opt_pos += opt_ident.length() + 1; // Locate the position of option name.
+
+ if (opt_pos > line.length()) {
+ // This condition will trigger when a utility
+ // supports an empty argument, e.g. tset(issue #9)
+ continue;
+ }
+
+ // Check for long options ; While here, also sanitize
+ // multi-word option definitions in a man page to properly
+ // extract short options from option definitions such as:
+ // .It Fl r Ar seconds (taken from date(1)).
+ if ((space_index = line.find(" ", opt_pos + 1, 1))
+ != std::string::npos)
+ opt_name = line.substr(opt_pos, space_index - opt_pos);
+ else
+ opt_name = line.substr(opt_pos);
+
+ // Check if the identified option matches the identifier.
+ // `opt_list.back()` is the previously checked option, the
+ // description of which is now stored in `buffer`.
+ if (!opt_list.empty() &&
+ (opt_map_iter = opt_map.find(opt_list.back()))
+ != opt_map.end() &&
+ buffer.find((opt_map_iter->second).keyword) != std::string::npos) {
+ ident_opt_list.push_back(&(opt_map_iter->second));
+
+ // Since the usage of the option under test
+ // is known, we remove it from `opt_list`.
+ opt_list.pop_back();
+ }
+
+ // Update the list of valid options.
+ opt_list.push_back(opt_name);
+
+ // Empty the buffer for next option's description.
+ buffer.clear();
+ }
+ else {
+ // Collect the option description until next
+ // valid option definition is encountered.
+ buffer.append(line);
+ }
+ }
+
+ return ident_opt_list;
+}

File Metadata

Mime Type
text/plain
Expires
Tue, Mar 10, 6:56 PM (7 h, 8 m)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
29506982
Default Alt Text
D12249.id32730.diff (92 KB)

Event Timeline