Page MenuHomeFreeBSD
Files F153534773

D12249.id33240.diff
No OneTemporary
Actions

D12249.id33240.diff
View Options

Index: tools/tools/smoketestsuite/Makefile
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/Makefile
@@ -0,0 +1,46 @@
+# $FreeBSD$
+#
+# Makefile for building the test generation tool
+
+CC= c++
+CFLAGS=-I/usr/local/include -std=c++11
+LIBS= -L/usr/local/lib \
+ -lboost_system \
+ -lboost_filesystem \
+ -lboost_thread \
+ -lboost_chrono
+OBJS= utils.o \
+ read_annotations.o \
+ generate_license.o \
+ add_testcase.o \
+ generate_test.o
+
+generate_tests: $(OBJS)
+ $(CC) $(CFLAGS) -o $(.TARGET) $(.ALLSRC) $(LIBS)
+
+$(OBJS): $(.PREFIX).cpp $(.PREFIX).h
+ $(CC) $(CFLAGS) -c $(.PREFIX).cpp
+
+.PHONY: clean \
+ fetch_groff \
+ fetch_utils \
+ run
+
+clean:
+ rm -rf $(OBJS) \
+ generate_tests \
+ scripts/utils_list \
+ groff \
+ generated_tests
+
+fetch_groff:
+ sh scripts/fetch_groff.sh
+
+fetch_utils:
+ sh scripts/fetch_utils.sh
+
+run:
+ @echo Generating annotations...
+ sh scripts/generate_annot.sh
+ @echo Generating test files...
+ ./generate_tests
Index: tools/tools/smoketestsuite/README
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/README
@@ -0,0 +1,55 @@
+# Smoke testing of base utilities (FreeBSD)
+
+Test generation tool made as a part of [Google Summer of Code '17 with FreeBSD](https://summerofcode.withgoogle.com/projects/#6426676740227072).
+**Refer the [FreeBSD wiki](https://wiki.freebsd.org/SummerOfCode2017/SmokeTestingOfBaseUtilities) for an overview and updates.**
+
+## Directory Structure
+```
+.
+├── annotations
+│   └── ........................:: Annotation files (generated/user-defined)
+├── scripts
+│   └── ........................:: Handy scripts
+├── architecture.png ...........:: A brief architecture diagram
+├── add_testcase.cpp ...........:: Testcase generator
+├── generate_license.cpp .......:: Customized license generator
+├── generate_test.cpp ..........:: Test generator
+├── read_annotations.cpp .......:: Annotation parser
+└── utils.cpp ..................:: Index generator
+```
+
+## Automation tool
+An implementation of the automation tool briefly described [here](https://lists.freebsd.org/pipermail/soc-status/2017-July/001079.html).
+The following diagram summarizes how different components fit with the testcase-generator -
+
+![Automation-Tool](architecture.png)
+
+- - -
+
+## Dependencies
+* Boost C++ libraries : The tool was tested to work with the port `boost-all-1.64.0`.
+
+## Instructions
+Clone the repository at `$HOME/smoketestsuite`.
+The location `$HOME/smoketestsuite` is important! If using a different location, the scripts under [scripts](scripts) need to be updated accordingly (for the time being).
+
+### Populating groff scripts
+* The directory [groff](groff) should be populated with the relevant groff scripts before proceeding for test generation. These scripts are available in the FreeBSD source tree. For filtering the utilities section wise, [fetch_groff.sh](scripts/fetch_groff.sh) sets the variable `section` to a default value of **1**. This value can be changed at will. However, it should be noted that currently the tool is tested to successfully generate **section 1 utilities** and might probably fail for other section numbers.
+
+* The variable `src` in [fetch_groff.sh](scripts/fetch_groff.sh) should be updated to the location of the local FreeBSD source. The default value is `$HOME/freebsd`.
+
+* For populating the directory `groff`, execute the following from the project root -
+ ```
+ make fetch_utils
+ make fetch_groff
+ ```
+
+### Generating tests
+Execute the following from the project root -
+```
+make clean
+make && make run
+```
+
+When generating the tests via `make run`, it will so happen that the tool will start showing failures after generating tests for first 10 utilities. I wrote on this issue briefly sometime back (https://lists.freebsd.org/pipermail/soc-status/2017-September/001107.html) and this problem was largely solved by closing stdin (as suggested by @asomers). But utilities like ed(1) and pax(1) aren't affected by closing stdin, and hence the behavior which was described in the previous link is reproduced.
+A quick and dirty solution for this (for the time being) is to remove these groff scripts from the locally extracted ones: "ed.1", "pax.1", "passwd.1".
Index: tools/tools/smoketestsuite/add_testcase.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/add_testcase.h
@@ -0,0 +1,37 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+namespace add_testcase {
+ void add_known_testcase(std::string, std::string, std::string, \
+ std::string, std::ofstream&);
+
+ void add_unknown_testcase(std::string, std::string, std::string, \
+ int, std::string&);
+
+ void add_noargs_testcase(std::string, std::pair<std::string, int>, \
+ std::ofstream&);
+}
Index: tools/tools/smoketestsuite/add_testcase.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/add_testcase.cpp
@@ -0,0 +1,171 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <boost/algorithm/string.hpp>
+#include <fstream>
+#include <iostream>
+#include "add_testcase.h"
+
+// Adds a test-case for an option with known usage.
+void
+add_testcase::add_known_testcase(std::string option,
+ std::string util_with_section,
+ std::string descr,
+ std::string output,
+ std::ofstream& test_script)
+{
+ std::string testcase_name;
+ std::string utility = util_with_section.substr(0,
+ util_with_section.length() - 3);
+
+ // Add testcase name.
+ test_script << "atf_test_case ";
+ if (!option.empty()) {
+ testcase_name = option;
+ testcase_name.append("_flag");
+ }
+ else
+ testcase_name = "no_arguments";
+ test_script << testcase_name + "\n";
+
+ // Add testcase description.
+ test_script << testcase_name
+ + "_head()\n{\n\tatf_set \"descr\" ";
+ if (!descr.empty())
+ test_script << descr;
+ else
+ test_script << "\"Verify the usage of option \'" + option + "\'\"";
+ test_script << "\n}\n\n";
+
+ // Add body of the testcase.
+ test_script << testcase_name + "_body()\n{"
+ + "\n\tatf_check -s exit:0 -o ";
+
+ // Match the usage output if generated.
+ if (!output.empty())
+ test_script << "inline:\"" + output + "\" ";
+ else
+ test_script << "empty ";
+ test_script << utility;
+
+ if (!option.empty())
+ test_script << " -" + option;
+ test_script << "\n}\n\n";
+}
+
+// Adds a test-case for an option with unknown usage.
+void
+add_testcase::add_unknown_testcase(std::string option,
+ std::string util_with_section,
+ std::string output,
+ int exitstatus,
+ std::string& testcase_buffer)
+{
+ std::string utility = util_with_section.substr(0,
+ util_with_section.length() - 3);
+
+ testcase_buffer.append("\n\tatf_check -s not-exit:0 -e ");
+
+ // Check if a usage message was produced (case-insensitive match).
+ if (boost::iequals(output.substr(0, 6), "usage:"))
+ testcase_buffer.append("match:\"$usage_output\" ");
+ else if (!output.empty())
+ testcase_buffer.append("inline:\"" + output + "\" ");
+ else
+ testcase_buffer.append("empty ");
+
+ testcase_buffer.append(utility);
+
+ if (!option.empty())
+ testcase_buffer.append(" -" + option);
+}
+
+// Adds a test-case for usage without any arguments.
+void
+add_testcase::add_noargs_testcase(std::string util_with_section,
+ std::pair<std::string, int> output,
+ std::ofstream& test_script)
+{
+ std::string descr;
+ std::string utility = util_with_section.substr(0,
+ util_with_section.length() - 3);
+
+ if (output.second) {
+ // An error was encountered.
+ test_script << std::string("atf_test_case no_arguments\n")
+ + "no_arguments_head()\n{\n\tatf_set \"descr\" ";
+ if (!output.first.empty()) {
+ // We expect a usage message to be generated in this case
+ // (case-insensitive match).
+ if (boost::iequals(output.first.substr(0, 6), "usage:")) {
+ descr = "\"Verify that " + util_with_section
+ + " fails and generates a valid usage"
+ + " message when no arguments are supplied\"";
+
+ test_script << descr + "\n}\n\nno_arguments_body()\n{"
+ + "\n\tatf_check -s not-exit:0 -e match:\"$usage_output\" "
+ + utility;
+ }
+ else {
+ descr = "\"Verify that " + util_with_section
+ + " fails and generates a valid output"
+ + " when no arguments are supplied\"";
+
+ test_script << descr + "\n}\n\nno_arguments_body()\n{"
+ + "\n\tatf_check -s not-exit:0 -e inline:\""
+ + output.first + "\" "
+ + utility;
+ }
+ }
+
+ else {
+ descr = "\"Verify that " + util_with_section
+ + "fails silently when no arguments are supplied\"" ;
+ test_script << descr + "\n}\n\nno_arguments_body()\n{"
+ + "\n\tatf_check -s not-exit:0 -e empty "
+ + utility;
+ }
+
+ test_script << "\n}\n\n";
+ }
+
+ else {
+ // The command ran successfully, hence we guessed
+ // a correct usage for the utility under test.
+ if (!output.first.empty())
+ descr = "\"Verify that " + util_with_section
+ + " executes successfully and produces a valid"
+ + " output when invoked without any arguments\"";
+ else
+ descr = "\"Verify that " + util_with_section
+ + " executes successfully and silently"
+ + " when invoked without any arguments\"";
+
+ add_testcase::add_known_testcase("", util_with_section, descr,
+ output.first, test_script);
+ }
+}
Index: tools/tools/smoketestsuite/annotations/date_test.annot
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/annotations/date_test.annot
@@ -0,0 +1,5 @@
+R_flag
+j_flag
+n_flag
+no_arguments
+u_flag
Index: tools/tools/smoketestsuite/generate_license.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/generate_license.h
@@ -0,0 +1,30 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+namespace add_license {
+ std::string generate_license();
+}
Index: tools/tools/smoketestsuite/generate_license.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/generate_license.cpp
@@ -0,0 +1,71 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <iostream>
+#include "generate_license.h"
+
+std::string
+add_license::generate_license()
+{
+ std::string license;
+ std::string tester_name;
+
+ // Get the tester's name which will be added in the license.
+ std::cout << "Please enter the name to be added in the license: ";
+ std::cin >> tester_name;
+
+ license =
+ "#\n"
+ "# Copyright 2017 " + tester_name + "\n"
+ "# All rights reserved.\n"
+ "#\n"
+ "# Redistribution and use in source and binary forms, with or without\n"
+ "# modification, are permitted provided that the following conditions\n"
+ "# are met:\n"
+ "# 1. Redistributions of source code must retain the above copyright\n"
+ "# notice, this list of conditions and the following disclaimer.\n"
+ "# 2. Redistributions in binary form must reproduce the above copyright\n"
+ "# notice, this list of conditions and the following disclaimer in the\n"
+ "# documentation and/or other materials provided with the distribution.\n"
+ "#\n"
+ "# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND\n"
+ "# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n"
+ "# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n"
+ "# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE\n"
+ "# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n"
+ "# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS\n"
+ "# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)\n"
+ "# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\n"
+ "# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY\n"
+ "# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF\n"
+ "# SUCH DAMAGE.\n"
+ "#\n"
+ "# $FreeBSD$\n"
+ "#\n\n";
+
+ return license;
+}
Index: tools/tools/smoketestsuite/generate_test.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/generate_test.h
@@ -0,0 +1,33 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include "utils.h"
+
+namespace generate_test {
+ std::pair<std::string, int> exec(const char*);
+ void generate_test(std::string, std::string);
+}
Index: tools/tools/smoketestsuite/generate_test.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/generate_test.cpp
@@ -0,0 +1,288 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <array>
+#include <boost/filesystem.hpp>
+#include <boost/chrono.hpp>
+#include <boost/thread.hpp>
+#include <cstdlib>
+#include <dirent.h>
+#include <fstream>
+#include <iostream>
+#include <memory>
+#include <stdexcept>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <unordered_set>
+#include "add_testcase.h"
+#include "generate_license.h"
+#include "generate_test.h"
+#include "read_annotations.h"
+
+#define TIMEOUT 2 // threshold (seconds) for a function call to return.
+
+std::string license; // license file generated during runtime.
+
+// Executes the passed argument "cmd" in a shell
+// and returns its output and the exit status.
+std::pair<std::string, int>
+generate_test::exec(const char* cmd)
+{
+ const int bufsize = 128;
+ std::array<char, bufsize> buffer;
+ std::string usage_output;
+ FILE* pipe = popen(cmd, "r");
+
+ if (!pipe)
+ throw std::runtime_error("popen() failed!");
+
+ try {
+ while (!feof(pipe))
+ if (std::fgets(buffer.data(), bufsize, pipe) != NULL)
+ usage_output += buffer.data();
+ }
+ catch(...) {
+ pclose(pipe);
+ throw "Unable to execute the command: " + std::string(cmd);
+ }
+
+ return std::make_pair<std::string, int>
+ ((std::string)usage_output, WEXITSTATUS(pclose(pipe)));
+}
+
+// Generate a test for the given utility.
+void
+generate_test::generate_test(std::string utility,
+ std::string section)
+{
+ std::list<utils::opt_rel*> ident_opt_list; // List of identified option relations.
+ std::vector<std::string> usage_messages; // Vector to store usage messages for comparison.
+ std::string command; // Command to be executed in shell.
+ std::string descr; // Testcase description.
+ std::string testcase_list; // List of testcases.
+ std::string testcase_buffer; // Buffer for (temporarily) holding testcase data.
+ std::string test_file; // atf-sh test name.
+ std::string util_with_section; // Section number appended to utility.
+ std::ofstream test_ofs; // Output stream for the atf-sh test.
+ std::pair<std::string, int> output; // Return value type for `exec()`.
+ std::unordered_set<std::string> annot; // Hashset of utility specific annotations.
+ int temp;
+
+ // Read annotations and populate hash set "annot".
+ annotations::read_annotations(utility, annot);
+
+ util_with_section = utility + '(' + section + ')';
+
+ utils::opt_def f_opts;
+ ident_opt_list = f_opts.check_opts(utility);
+ test_file = "generated_tests/" + utility + "_test.sh";
+
+ // Add license in the generated test scripts.
+ test_ofs.open(test_file, std::ios::out);
+ test_ofs << license;
+
+ // If a known option was encountered (i.e. `ident_opt_list` is
+ // populated), produce a testcase to check the validity of the
+ // result of that option. If no known option was encountered,
+ // produce testcases to verify the correct (generated) usage
+ // message when using the supported options incorrectly.
+
+ // Add testcases for known options.
+ if (!ident_opt_list.empty()) {
+ for (const auto &i : ident_opt_list) {
+ command = utility + " -" + i->value + " 2>&1 </dev/null";
+ output = generate_test::exec(command.c_str());
+ if (!output.first.compare(0, 6, "usage:")) {
+ add_testcase::add_known_testcase(i->value, util_with_section,
+ descr, output.first, test_ofs);
+ }
+ else {
+ // A usage message was produced, i.e. we
+ // failed to guess the correct usage.
+ add_testcase::add_unknown_testcase(i->value, util_with_section, output.first,
+ output.second, testcase_buffer);
+ }
+ testcase_list.append("\tatf_add_test_case " + i->value + "_flag\n");
+ }
+ }
+
+ // Add testcases for the options whose usage is not known (yet).
+ if (!f_opts.opt_list.empty()) {
+
+ // For the purpose of adding a "$usage_output" variable,
+ // we choose the option which produces one.
+ // TODO Avoid double executions of an option, i.e. one while
+ // selecting usage message and another while generating testcase.
+
+ if (f_opts.opt_list.size() == 1) {
+ // Utility supports a single option, check if it produces a usage message.
+ command = utility + " -" + f_opts.opt_list.front() + " 2>&1 </dev/null";
+ output = generate_test::exec(command.c_str());
+
+ if (output.second)
+ test_ofs << "usage_output=\'" + output.first + "\'\n\n";
+ }
+
+ else {
+ // Utility supports multiple options. In case the usage message
+ // is consistent for atleast "two" options, we reduce duplication
+ // by assigning a variable "usage_output" in the test script.
+ for (const auto &i : f_opts.opt_list) {
+ command = utility + " -" + i + " 2>&1 </dev/null";
+ output = generate_test::exec(command.c_str());
+
+ if (output.second && usage_messages.size() < 3)
+ usage_messages.push_back(output.first);
+ }
+
+ temp = usage_messages.size();
+ for (int j = 0; j < temp; j++) {
+ if (!(usage_messages.at(j)).compare(usage_messages.at((j+1) % temp))) {
+ test_ofs << "usage_output=\'"
+ + output.first.substr(0, 7 + utility.size())
+ + "\'\n\n";
+ break;
+ }
+ }
+ }
+
+ // Execute the utility with supported options
+ // and add (+ve)/(-ve) testcases accordingly.
+ for (const auto &i : f_opts.opt_list) {
+ // If the option is annotated, skip it.
+ if (annot.find(i) != annot.end())
+ continue;
+
+ command = utility + " -" + i + " 2>&1 </dev/null";
+ output = generate_test::exec(command.c_str());
+
+ if (output.second) {
+ // Non-zero exit status was encountered.
+ add_testcase::add_unknown_testcase(i, util_with_section, output.first,
+ output.second, testcase_buffer);
+ }
+ else {
+ // EXIT_SUCCESS was encountered. Hence,
+ // the guessed usage was correct.
+ add_testcase::add_known_testcase(i, util_with_section, "",
+ output.first, test_ofs);
+ testcase_list.append(std::string("\tatf_add_test_case ")
+ + i + "_flag\n");
+ }
+ }
+
+ testcase_list.append("\tatf_add_test_case invalid_usage\n");
+ test_ofs << std::string("atf_test_case invalid_usage\ninvalid_usage_head()\n")
+ + "{\n\tatf_set \"descr\" \"Verify that an invalid usage "
+ + "with a supported option produces a valid error message"
+ + "\"\n}\n\ninvalid_usage_body()\n{";
+
+ test_ofs << testcase_buffer + "\n}\n\n";
+ }
+
+ // Add a testcase under "no_arguments" for
+ // running the utility without any arguments.
+ if (annot.find("*") == annot.end()) {
+ command = utility + " 2>&1 </dev/null";
+ output = generate_test::exec(command.c_str());
+ add_testcase::add_noargs_testcase(util_with_section, output, test_ofs);
+ testcase_list.append("\tatf_add_test_case no_arguments\n");
+ }
+
+ test_ofs << "atf_init_test_cases()\n{\n" + testcase_list + "}\n";
+ test_ofs.close();
+}
+
+int
+main()
+{
+ std::ifstream groff_list;
+ std::list<std::pair<std::string, std::string>> utility_list;
+ std::string test_file; // atf-sh test name.
+ std::string util_name; // Utility name.
+ const char *tests_dir = "generated_tests/";
+ struct stat sb;
+ struct dirent *ent;
+ DIR *groff_dir;
+ char answer; // User input to determine overwriting of test files.
+
+ // For testing (or generating tests for only selected utilities),
+ // the utility_list can be populated above during declaration.
+ if (utility_list.empty()) {
+ if ((groff_dir = opendir("groff"))) {
+ readdir(groff_dir); // Skip directory entry for "."
+ readdir(groff_dir); // Skip directory entry for ".."
+ while ((ent = readdir(groff_dir))) {
+ util_name = ent->d_name;
+ utility_list.push_back(std::make_pair<std::string, std::string>
+ (util_name.substr(0, util_name.length() - 2),
+ util_name.substr(util_name.length() - 1, 1)));
+ }
+ closedir(groff_dir);
+ }
+ else {
+ fprintf(stderr, "Could not open the directory: ./groff\nRefer to the "
+ "section \"Populating groff scripts\" in README!\n");
+ return EXIT_FAILURE;
+ }
+ }
+
+ // Check if the directory 'generated_tests' exists.
+ if (stat(tests_dir, &sb) || !S_ISDIR(sb.st_mode)) {
+ boost::filesystem::path dir(tests_dir);
+ if (boost::filesystem::create_directory(dir))
+ std::cout << "Directory created: " << tests_dir << std::endl;
+ }
+
+ // Generate a license to be added in the generated scripts.
+ license = add_license::generate_license();
+
+ for (const auto &util : utility_list) {
+ test_file = tests_dir + util.first + "_test.sh";
+ // TODO Check before overwriting existing test scripts.
+
+ // Enable line-buffering on stdout.
+ setlinebuf(stdout);
+
+ std::cout << "Generating test for: " + util.first
+ + '('+ util.second + ')' << " ...";
+
+ boost::thread api_caller(generate_test::generate_test, util.first, util.second);
+ if (api_caller.try_join_for(boost::chrono::seconds(TIMEOUT))) {
+ // API call successfully returned within TIMEOUT (seconds).
+ std::cout << "Successful\n";
+ }
+ else {
+ // API call timed out.
+ std::cout << "Failed!\n";
+ // Remove the incomplete test file.
+ remove(("generated_tests/" + util.first + "_test.sh").c_str());
+ }
+ }
+
+ return EXIT_SUCCESS;
+}
Index: tools/tools/smoketestsuite/read_annotations.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/read_annotations.h
@@ -0,0 +1,33 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <unordered_set>
+
+namespace annotations {
+ void read_annotations(std::string, \
+ std::unordered_set<std::string>&);
+}
Index: tools/tools/smoketestsuite/read_annotations.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/read_annotations.cpp
@@ -0,0 +1,58 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <dirent.h>
+#include <fstream>
+#include <iostream>
+#include <string>
+#include "read_annotations.h"
+
+// Read the annotation files and skip
+// generation of the respective tests.
+void
+annotations::read_annotations(std::string utility,
+ std::unordered_set<std::string>& annot)
+{
+ std::string line;
+ // TODO do this for all the annotation files
+ std::ifstream annot_fstream;
+ annot_fstream.open("annotations/" + utility + "_test.annot");
+
+ while (getline(annot_fstream, line)) {
+ // Add a unique identifier for no_arguments testcase
+ if (!line.compare(0, 12, "no_arguments"))
+ annot.insert("*");
+
+ // Add flag value for supported argument testcases
+ // Doing so we ignore the "invalid_usage" testcase
+ // as it is guaranteed to always succeed.
+ else if (!line.compare(2, 4, "flag"))
+ annot.insert(line.substr(0, 1));
+ }
+
+ annot_fstream.close();
+}
Index: tools/tools/smoketestsuite/scripts/README
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/scripts/README
@@ -0,0 +1,8 @@
+# Handy scripts
+
+Script Name | Functionality
+------------------|-----------------
+fetch_groff.sh | Lists location of all section 1 utilities
+fetch_utils.sh | Saves all the base utilities in the src tree in **utils_list**
+generate_annot.sh | Populates annotation files under [annotations](../annotations)
+validate.sh | Validates side-effects of newly introduced changes in the tool
Index: tools/tools/smoketestsuite/scripts/fetch_groff.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/scripts/fetch_groff.sh
@@ -0,0 +1,51 @@
+#!/bin/sh
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+
+# Script for listing location of groff scripts of utilities in section 1
+
+set -e
+
+src="$HOME/freebsd"
+pwd=$(pwd)
+dir_list="$pwd/scripts/utils_list"
+groff_src="$pwd/groff"
+section=1 # Section number for filtering the base utilities
+
+rm -rf "$groff_src" && mkdir "$groff_src"
+cd "$src"
+
+while IFS= read -r dir_entry
+do
+ for file in "$dir_entry"/*
+ do
+ # Check for only section 1 entries
+ case "$file" in
+ *.1) cp "$file" "$pwd/groff"
+ esac
+ done
+done< "$dir_list"
Index: tools/tools/smoketestsuite/scripts/fetch_utils.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/scripts/fetch_utils.sh
@@ -0,0 +1,44 @@
+#! /bin/sh
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+
+# Script for listing all the base utilities
+
+set -e
+pwd=$(pwd)
+dir="$pwd/scripts"
+src="$HOME/freebsd"
+
+fetch_utils() {
+ cd "$src"
+ find . -name Makefile | xargs grep -l 'PROG\|PROG_CXX' \
+ | sed -e 's|/Makefile$||' | cut -c 3-
+}
+
+rm -f utils_list
+
+(fetch_utils) >> "$dir/utils_list"
Index: tools/tools/smoketestsuite/scripts/generate_annot.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/scripts/generate_annot.sh
@@ -0,0 +1,92 @@
+#!/bin/sh
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+
+# Script for generating annotations based on generated tests.
+
+pwd=$(pwd)
+suffix="_test"
+extension=".sh"
+update_required=0
+
+message="Following annotation files were updated. \n"
+
+printf "
++--------------------------------------------------+
+| Checking if any annotation file is to be updated |
++--------------------------------------------------+\n"
+
+for f in "generated_tests"/*
+do
+ annotations=""
+ file=$(basename "$f")
+ test=${file%$extension}
+ utility=${test%$suffix}
+ test_dir="/usr/tests/bin/$utility"
+
+ (
+ cd "$test_dir" || exit
+ report=$(kyua report)
+ i=2
+
+ while true
+ do
+ testcase=$(printf "%s" "$report" | awk 'NR=='"$i"' {print $1}')
+ status=$(printf "%s" "$report" | awk 'NR=='"$i"' {print $3}')
+ check=$(printf "%s" "$testcase" | cut -s -f1 -d":")
+
+ if [ "$check" != "$test" ]; then
+ if [ "$annotations" ]; then
+ if [ $update_required = 0 ]; then
+ printf $message
+ update_required=1
+ fi
+
+ annotations_file="$pwd/annotations/$test.annot"
+ # Append only the new annotations
+ printf "$annotations" > "$annotations_file.temp"
+ [ ! -e "$annotations_file" ] && touch "$annotations_file"
+ comm -13 "$annotations_file" "$annotations_file.temp" >> \
+ "$annotations_file" && printf "\t%s\n" "annotations/$test.annot"
+ rm -f "$annotations_file.temp"
+ fi
+
+ break
+ fi
+
+ if [ "$status" = "failed:" ]; then
+ testcase=${testcase#"$test:"}
+ annotations="$annotations$testcase\n"
+ fi
+
+ i=$((i+1))
+ done
+ )
+
+done
+
+printf "==============================================================================\n\n"
Index: tools/tools/smoketestsuite/utils.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/utils.h
@@ -0,0 +1,50 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <list>
+#include <unordered_map>
+
+namespace utils {
+ // Option relation which maps option names to
+ // a unique identifier in their description.
+ typedef struct opt_rel {
+ char type; // Option type: (s)short/(l)long.
+ std::string value; // Name of the option.
+ std::string keyword; // The keyword which should be looked up in the
+ // message (if) produced when using this option.
+ } opt_rel;
+
+ class opt_def {
+ public:
+ std::list<std::string> opt_list; // list of all the accepted options with unknown usage.
+ std::unordered_map<std::string, opt_rel> opt_map; // Map "option value" to "option definition".
+ std::unordered_map<std::string, opt_rel>::iterator opt_map_iter;
+
+ void insert_opts();
+ std::list<opt_rel*> check_opts(std::string);
+ };
+}
Index: tools/tools/smoketestsuite/utils.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/utils.cpp
@@ -0,0 +1,130 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <fstream>
+#include <iostream>
+#include <cstdlib>
+#include "utils.h"
+
+// Insert a list of user-defined option definitions
+// into a hashmap. These specific option definitions
+// are the ones which one can be easily tested.
+void
+utils::opt_def::insert_opts()
+{
+ // Option definitions.
+ opt_rel h_def; // '-h'
+ h_def.type = 's';
+ h_def.value = "h";
+ h_def.keyword = "help";
+
+ opt_rel v_def; // '-v'
+ v_def.type = 's';
+ v_def.value = "v";
+ v_def.keyword = "version";
+
+ // `opt_map` contains all the options
+ // which can be "easily" tested.
+ opt_map.insert(std::make_pair<std::string, opt_rel>
+ ("h", (opt_rel)h_def));
+ opt_map.insert(std::make_pair<std::string, opt_rel>
+ ("v", (opt_rel)v_def));
+};
+
+// For the utility under test, find the supported options
+// present in the hashmap generated by insert_opts()
+// and return them in a form of list of option relations.
+std::list<utils::opt_rel*>
+utils::opt_def::check_opts(std::string utility)
+{
+ std::string line; // An individual line in a man-page.
+ std::string opt_name; // Name of the option.
+ std::string opt_ident = ".It Fl"; // Identifier for an option in man page.
+ std::string buffer; // Option description extracted from man-page.
+ std::string opt_string; // Identified option names.
+ int opt_pos; // Starting index of the (identified) option.
+ int space_index; // First occurrence of space character
+ // in a multi-word option definition.
+ std::list<opt_rel*> ident_opt_list; // List of identified option relations (opt_rel's).
+
+ // Generate the hashmap opt_map.
+ insert_opts();
+
+ // TODO: Section number cannot be hardcoded.
+ std::ifstream infile("groff/" + utility + ".1");
+
+ // Search for all the options accepted by the
+ // utility and collect those present in `opt_map`.
+ while (std::getline(infile, line)) {
+ if ((opt_pos = line.find(opt_ident)) != std::string::npos) {
+ opt_pos += opt_ident.length() + 1; // Locate the position of option name.
+
+ if (opt_pos > line.length()) {
+ // This condition will trigger when a utility
+ // supports an empty argument, e.g. tset(issue #9)
+ continue;
+ }
+
+ // Check for long options ; While here, also sanitize
+ // multi-word option definitions in a man page to properly
+ // extract short options from option definitions such as:
+ // .It Fl r Ar seconds (taken from date(1)).
+ if ((space_index = line.find(" ", opt_pos + 1, 1))
+ != std::string::npos)
+ opt_name = line.substr(opt_pos, space_index - opt_pos);
+ else
+ opt_name = line.substr(opt_pos);
+
+ // Check if the identified option matches the identifier.
+ // `opt_list.back()` is the previously checked option, the
+ // description of which is now stored in `buffer`.
+ if (!opt_list.empty() &&
+ (opt_map_iter = opt_map.find(opt_list.back()))
+ != opt_map.end() &&
+ buffer.find((opt_map_iter->second).keyword) != std::string::npos) {
+ ident_opt_list.push_back(&(opt_map_iter->second));
+
+ // Since the usage of the option under test
+ // is known, we remove it from `opt_list`.
+ opt_list.pop_back();
+ }
+
+ // Update the list of valid options.
+ opt_list.push_back(opt_name);
+
+ // Empty the buffer for next option's description.
+ buffer.clear();
+ }
+ else {
+ // Collect the option description until next
+ // valid option definition is encountered.
+ buffer.append(line);
+ }
+ }
+
+ return ident_opt_list;
+}

File Metadata

Mime Type
text/plain
Expires
Wed, Apr 22, 5:38 PM (24 m, 11 s)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
31991520
Default Alt Text
D12249.id33240.diff (48 KB)

Event Timeline