Page MenuHomeFreeBSD
Files F142164807

D12249.id33226.diff
No OneTemporary
Actions

D12249.id33226.diff
View Options

Index: tools/tools/smoketestsuite/LICENSE
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/LICENSE
@@ -0,0 +1,23 @@
+Copyright (c) 2017, Shivansh Rai
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Index: tools/tools/smoketestsuite/README.md
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/README.md
@@ -0,0 +1,53 @@
+# Smoke testing of base utilities (FreeBSD)
+
+Test generation tool made as a part of [Google Summer of Code '17 with FreeBSD](https://summerofcode.withgoogle.com/projects/#6426676740227072).
+**Refer the [FreeBSD wiki](https://wiki.freebsd.org/SummerOfCode2017/SmokeTestingOfBaseUtilities) for an overview and updates.**
+
+## Directory Structure
+```
+.
+└── src ........................:: Automation tool pertaining to new test plan
+ ├── annotations
+ │   └── ....................:: Annotation files (generated/user-defined)
+ ├── scripts
+ │ └── ....................:: Handy scripts
+ ├── add_testcase.cpp .......:: Testcase generator
+ ├── generate_license.cpp ...:: License generator
+ ├── generate_test.cpp ......:: Test generator
+ ├── read_annotations.cpp ...:: Annotation parser
+ └── utils.cpp ..............:: Index generator
+```
+
+## Automation tool
+An implementation of the automation tool briefly described [here](https://lists.freebsd.org/pipermail/soc-status/2017-July/001079.html).
+The following diagram summarizes how different components fit with the testcase-generator -
+
+![Automation-Tool](architecture.png)
+
+- - -
+
+## Dependencies
+* Boost C++ libraries : The tool was tested to work with the port `boost-all-1.64.0`.
+
+## Instructions
+
+Clone the repository at `$HOME/smoketestsuite`.
+The location `$HOME/smoketestsuite` is important! If using a different location, the scripts under [src/scripts](src/scripts) need to be updated accordingly (for the time being).
+
+### Populating groff scripts
+* The directory [src/groff](src/groff) should be populated with the relevant groff scripts before proceeding for test generation. These scripts are available in the FreeBSD source tree. For filtering the utilities section wise, [fetch_groff.sh](src/scripts/fetch_groff.sh) sets the variable `section` to a default value of **1**. This value can be changed at will. However, it should be noted that currently the tool is tested to successfully generate **section 1 utilities** and might probably fail for other section numbers.
+
+* The variable `src` in [fetch_groff.sh](src/scripts/fetch_groff.sh) should be updated to the location of the local FreeBSD source. The default value is `$HOME/freebsd`.
+
+* For populating `src/groff`, execute the following inside [src](src) -
+ ```
+ make fetch_utils
+ make fetch_groff
+ ```
+
+### Generating tests
+Execute the following inside [src](src) -
+```
+make clean
+make && make run
+```
Index: tools/tools/smoketestsuite/src/Makefile
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/Makefile
@@ -0,0 +1,43 @@
+# $FreeBSD$
+#
+# Makefile for building the automation tool
+
+CC= c++
+CFLAGS= -I/usr/local/include -std=c++11
+LIBS= -L/usr/local/lib \
+ -lboost_system \
+ -lboost_filesystem \
+ -lboost_thread \
+ -lboost_chrono
+OBJS= utils.o \
+ read_annotations.o \
+ generate_license.o \
+ add_testcase.o \
+ generate_test.o
+
+generate_tests: $(OBJS)
+ $(CC) $(CFLAGS) -o $(.TARGET) $(.ALLSRC) $(LIBS)
+
+$(OBJS): $(.PREFIX).cpp $(.PREFIX).h
+ $(CC) $(CFLAGS) -c $(.PREFIX).cpp
+
+.PHONY: clean \
+ clean_tests \
+ fetch_groff \
+ fetch_utils \
+ run
+
+clean:
+ rm -f $(OBJS) generate_tests scripts/utils_list
+
+fetch_groff:
+ sh scripts/fetch_groff.sh
+
+fetch_utils:
+ sh scripts/fetch_utils.sh
+
+run:
+ @echo Generating annotations...
+ sh scripts/generate_annot.sh
+ @echo Generating test files...
+ ./generate_tests
Index: tools/tools/smoketestsuite/src/add_testcase.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/add_testcase.h
@@ -0,0 +1,37 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+namespace add_testcase {
+ void add_known_testcase(std::string, std::string, std::string, \
+ std::string, std::ofstream&);
+
+ void add_unknown_testcase(std::string, std::string, std::string, \
+ int, std::string&);
+
+ void add_noargs_testcase(std::string, std::pair<std::string, int>, \
+ std::ofstream&);
+}
Index: tools/tools/smoketestsuite/src/add_testcase.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/add_testcase.cpp
@@ -0,0 +1,169 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <fstream>
+#include <iostream>
+#include "add_testcase.h"
+
+// Adds a test-case for an option with known usage.
+void
+add_testcase::add_known_testcase(std::string option,
+ std::string util_with_section,
+ std::string descr,
+ std::string output,
+ std::ofstream& test_script)
+{
+ std::string testcase_name;
+ std::string utility = util_with_section.substr(0,
+ util_with_section.length() - 3);
+
+ // Add testcase name.
+ test_script << "atf_test_case ";
+ if (!option.empty()) {
+ testcase_name = option;
+ testcase_name.append("_flag");
+ }
+ else
+ testcase_name = "no_arguments";
+ test_script << testcase_name + "\n";
+
+ // Add testcase description.
+ test_script << testcase_name
+ + "_head()\n{\n\tatf_set \"descr\" ";
+ if (!descr.empty())
+ test_script << descr;
+ else
+ test_script << "\"Verify the usage of option \'" + option + "\'\"";
+ test_script << "\n}\n\n";
+
+ // Add body of the testcase.
+ test_script << testcase_name + "_body()\n{"
+ + "\n\tatf_check -s exit:0 -o ";
+
+ // Match the usage output if generated.
+ if (!output.empty())
+ test_script << "inline:\"" + output + "\" ";
+ else
+ test_script << "empty ";
+ test_script << utility;
+
+ if (!option.empty())
+ test_script << " -" + option;
+ test_script << "\n}\n\n";
+}
+
+// Adds a test-case for an option with unknown usage.
+void
+add_testcase::add_unknown_testcase(std::string option,
+ std::string util_with_section,
+ std::string output,
+ int exitstatus,
+ std::string& testcase_buffer)
+{
+ std::string utility = util_with_section.substr(0,
+ util_with_section.length() - 3);
+
+ testcase_buffer.append("\n\tatf_check -s not-exit:0 -e ");
+
+ // Check if a usage message was produced.
+ if (!output.compare(0, 6, "usage:"))
+ testcase_buffer.append("match:\"$usage_output\" ");
+ else if (!output.empty())
+ testcase_buffer.append("inline:\"" + output + "\" ");
+ else
+ testcase_buffer.append("empty ");
+
+ testcase_buffer.append(utility);
+
+ if (!option.empty())
+ testcase_buffer.append(" -" + option);
+}
+
+// Adds a test-case for usage without any arguments.
+void
+add_testcase::add_noargs_testcase(std::string util_with_section,
+ std::pair<std::string, int> output,
+ std::ofstream& test_script)
+{
+ std::string descr;
+ std::string utility = util_with_section.substr(0,
+ util_with_section.length() - 3);
+
+ if (output.second) {
+ // An error was encountered.
+ test_script << std::string("atf_test_case no_arguments\n")
+ + "no_arguments_head()\n{\n\tatf_set \"descr\" ";
+ if (!output.first.empty()) {
+ // We expect a usage message to be generated in this case.
+ if (!output.first.compare(0, 6, "usage:")) {
+ descr = "\"Verify that " + util_with_section
+ + " fails and generates a valid usage"
+ + " message when no arguments are supplied\"";
+
+ test_script << descr + "\n}\n\nno_arguments_body()\n{"
+ + "\n\tatf_check -s not-exit:0 -e match:\"$usage_output\" "
+ + utility;
+ }
+ else {
+ descr = "\"Verify that " + util_with_section
+ + " fails and generates a valid output"
+ + " when no arguments are supplied\"";
+
+ test_script << descr + "\n}\n\nno_arguments_body()\n{"
+ + "\n\tatf_check -s not-exit:0 -e inline:\""
+ + output.first + "\" "
+ + utility;
+ }
+ }
+
+ else {
+ descr = "\"Verify that " + util_with_section
+ + "fails silently when no arguments are supplied\"" ;
+ test_script << descr + "\n}\n\nno_arguments_body()\n{"
+ + "\n\tatf_check -s not-exit:0 -e empty "
+ + utility;
+ }
+
+ test_script << "\n}\n\n";
+ }
+
+ else {
+ // The command ran successfully, hence we guessed
+ // a correct usage for the utility under test.
+ if (!output.first.empty())
+ descr = "\"Verify that " + util_with_section
+ + " executes successfully and produces a valid"
+ + " output when invoked without any arguments\"";
+ else
+ descr = "\"Verify that " + util_with_section
+ + " executes successfully and silently"
+ + " when invoked without any arguments\"";
+
+ add_testcase::add_known_testcase("", util_with_section, descr,
+ output.first, test_script);
+ }
+}
Index: tools/tools/smoketestsuite/src/annotations/date_test.annot
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/annotations/date_test.annot
@@ -0,0 +1,5 @@
+R_flag
+j_flag
+n_flag
+no_arguments
+u_flag
Index: tools/tools/smoketestsuite/src/generate_license.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generate_license.h
@@ -0,0 +1,30 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+namespace add_license {
+ std::string generate_license();
+}
Index: tools/tools/smoketestsuite/src/generate_license.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generate_license.cpp
@@ -0,0 +1,71 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <iostream>
+#include "generate_license.h"
+
+std::string
+add_license::generate_license()
+{
+ std::string license;
+ std::string tester_name;
+
+ // Get the tester's name which will be added in the license.
+ std::cout << "Please enter the name to be added in the license: ";
+ std::cin >> tester_name;
+
+ license =
+ "#\n"
+ "# Copyright 2017 " + tester_name + "\n"
+ "# All rights reserved.\n"
+ "#\n"
+ "# Redistribution and use in source and binary forms, with or without\n"
+ "# modification, are permitted provided that the following conditions\n"
+ "# are met:\n"
+ "# 1. Redistributions of source code must retain the above copyright\n"
+ "# notice, this list of conditions and the following disclaimer.\n"
+ "# 2. Redistributions in binary form must reproduce the above copyright\n"
+ "# notice, this list of conditions and the following disclaimer in the\n"
+ "# documentation and/or other materials provided with the distribution.\n"
+ "#\n"
+ "# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND\n"
+ "# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n"
+ "# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n"
+ "# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE\n"
+ "# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n"
+ "# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS\n"
+ "# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)\n"
+ "# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT\n"
+ "# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY\n"
+ "# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF\n"
+ "# SUCH DAMAGE.\n"
+ "#\n"
+ "# $FreeBSD$\n"
+ "#\n";
+
+ return license;
+}
Index: tools/tools/smoketestsuite/src/generate_test.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generate_test.h
@@ -0,0 +1,33 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include "utils.h"
+
+namespace generate_test {
+ std::pair<std::string, int> exec(const char*);
+ void generate_test(std::string, std::string);
+}
Index: tools/tools/smoketestsuite/src/generate_test.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/generate_test.cpp
@@ -0,0 +1,309 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <array>
+#include <boost/filesystem.hpp>
+#include <boost/chrono.hpp>
+#include <boost/thread.hpp>
+#include <cstdlib>
+#include <dirent.h>
+#include <fstream>
+#include <iostream>
+#include <memory>
+#include <stdexcept>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <unordered_set>
+#include "add_testcase.h"
+#include "generate_license.h"
+#include "generate_test.h"
+#include "read_annotations.h"
+
+#define TIMEOUT 1 // threshold (seconds) for a function call to return.
+
+std::string license; // license file generated during runtime.
+
+// Executes the passed argument "cmd" in a shell
+// and returns its output and the exit status.
+std::pair<std::string, int>
+generate_test::exec(const char* cmd)
+{
+ const int bufsize = 128;
+ std::array<char, bufsize> buffer;
+ std::string usage_output;
+ FILE* pipe = popen(cmd, "r");
+
+ if (!pipe)
+ throw std::runtime_error("popen() failed!");
+
+ try {
+ while (!feof(pipe))
+ if (std::fgets(buffer.data(), bufsize, pipe) != NULL)
+ usage_output += buffer.data();
+ }
+ catch(...) {
+ pclose(pipe);
+ throw "Unable to execute the command: " + std::string(cmd);
+ }
+
+ return std::make_pair<std::string, int>
+ ((std::string)usage_output, WEXITSTATUS(pclose(pipe)));
+}
+
+// Generate a test for the given utility.
+void
+generate_test::generate_test(std::string utility,
+ std::string section)
+{
+ std::list<utils::opt_rel*> ident_opt_list; // List of identified option relations.
+ std::vector<std::string> usage_messages; // Vector to store usage messages for comparison.
+ std::string command; // Command to be executed in shell.
+ std::string descr; // Testcase description.
+ std::string testcase_list; // List of testcases.
+ std::string testcase_buffer; // Buffer for (temporarily) holding testcase data.
+ std::string test_file; // atf-sh test name.
+ std::string util_with_section; // Section number appended to utility.
+ std::ofstream test_ofs; // Output stream for the atf-sh test.
+ std::pair<std::string, int> output; // Return value type for `exec()`.
+ std::unordered_set<std::string> annot; // Hashset of utility specific annotations.
+ int temp;
+
+ // Read annotations and populate hash set "annot".
+ annotations::read_annotations(utility, annot);
+
+ util_with_section = utility + '(' + section + ')';
+
+ utils::opt_def f_opts;
+ ident_opt_list = f_opts.check_opts(utility);
+ test_file = "generated_tests/" + utility + "_test.sh";
+
+ // Add license in the generated test scripts.
+ test_ofs.open(test_file, std::ios::out);
+ test_ofs << license;
+
+ // If a known option was encountered (i.e. `ident_opt_list` is
+ // populated), produce a testcase to check the validity of the
+ // result of that option. If no known option was encountered,
+ // produce testcases to verify the correct (generated) usage
+ // message when using the supported options incorrectly.
+
+ // Add testcases for known options.
+ if (!ident_opt_list.empty()) {
+ for (const auto &i : ident_opt_list) {
+ command = utility + " -" + i->value + " 2>&1 </dev/null";
+ output = generate_test::exec(command.c_str());
+ if (!output.first.compare(0, 6, "usage:")) {
+ add_testcase::add_known_testcase(i->value, util_with_section,
+ descr, output.first, test_ofs);
+ }
+ else {
+ // A usage message was produced, i.e. we
+ // failed to guess the correct usage.
+ add_testcase::add_unknown_testcase(i->value, util_with_section, output.first,
+ output.second, testcase_buffer);
+ }
+ testcase_list.append("\tatf_add_test_case " + i->value + "_flag\n");
+ }
+ }
+
+ // Add testcases for the options whose usage is not known (yet).
+ if (!f_opts.opt_list.empty()) {
+
+ // For the purpose of adding a "$usage_output" variable,
+ // we choose the option which produces one.
+ // TODO Avoid double executions of an option, i.e. one while
+ // selecting usage message and another while generating testcase.
+
+ if (f_opts.opt_list.size() == 1) {
+ // Utility supports a single option, check if it produces a usage message.
+ command = utility + " -" + f_opts.opt_list.front() + " 2>&1 </dev/null";
+ output = generate_test::exec(command.c_str());
+
+ if (output.second)
+ test_ofs << "usage_output=\'" + output.first + "\'\n\n";
+ }
+
+ else {
+ // Utility supports multiple options. In case the usage message
+ // is consistent for atleast "two" options, we reduce duplication
+ // by assigning a variable "usage_output" in the test script.
+ for (const auto &i : f_opts.opt_list) {
+ command = utility + " -" + i + " 2>&1 </dev/null";
+ output = generate_test::exec(command.c_str());
+
+ if (output.second && usage_messages.size() < 3)
+ usage_messages.push_back(output.first);
+ }
+
+ temp = usage_messages.size();
+ for (int j = 0; j < temp; j++) {
+ if (!(usage_messages.at(j)).compare(usage_messages.at((j+1) % temp))) {
+ test_ofs << "usage_output=\'"
+ + output.first.substr(0, 7 + utility.size())
+ + "\'\n\n";
+ break;
+ }
+ }
+ }
+
+ // Execute the utility with supported options
+ // and add (+ve)/(-ve) testcases accordingly.
+ for (const auto &i : f_opts.opt_list) {
+ // If the option is annotated, skip it.
+ if (annot.find(i) != annot.end())
+ continue;
+
+ command = utility + " -" + i + " 2>&1 </dev/null";
+ output = generate_test::exec(command.c_str());
+
+ if (output.second) {
+ // Non-zero exit status was encountered.
+ add_testcase::add_unknown_testcase(i, util_with_section, output.first,
+ output.second, testcase_buffer);
+ }
+ else {
+ // EXIT_SUCCESS was encountered. Hence,
+ // the guessed usage was correct.
+ add_testcase::add_known_testcase(i, util_with_section, "",
+ output.first, test_ofs);
+ testcase_list.append(std::string("\tatf_add_test_case ")
+ + i + "_flag\n");
+ }
+ }
+
+ testcase_list.append("\tatf_add_test_case invalid_usage\n");
+ test_ofs << std::string("atf_test_case invalid_usage\ninvalid_usage_head()\n")
+ + "{\n\tatf_set \"descr\" \"Verify that an invalid usage "
+ + "with a supported option produces a valid error message"
+ + "\"\n}\n\ninvalid_usage_body()\n{";
+
+ test_ofs << testcase_buffer + "\n}\n\n";
+ }
+
+ // Add a testcase under "no_arguments" for
+ // running the utility without any arguments.
+ if (annot.find("*") == annot.end()) {
+ command = utility + " 2>&1 </dev/null";
+ output = generate_test::exec(command.c_str());
+ add_testcase::add_noargs_testcase(util_with_section, output, test_ofs);
+ testcase_list.append("\tatf_add_test_case no_arguments\n");
+ }
+
+ test_ofs << "atf_init_test_cases()\n{\n" + testcase_list + "}\n";
+ test_ofs.close();
+}
+
+int
+main()
+{
+ std::ifstream groff_list;
+ std::list<std::pair<std::string, std::string>> utility_list;
+ std::string test_file; // atf-sh test name.
+ std::string util_name; // Utility name.
+ const char *tests_dir = "generated_tests/";
+ struct stat sb;
+ struct dirent *ent;
+ DIR *groff_dir;
+ char answer; // User input to determine overwriting of test files.
+ int flag = 0;
+
+ // For testing (or generating tests for only selected utilities),
+ // the utility_list can be populated above during declaration.
+ if (utility_list.empty()) {
+ if ((groff_dir = opendir("groff"))) {
+ while ((ent = readdir(groff_dir))) {
+ if (!strncmp(ent->d_name, ".", 1))
+ continue;
+
+ util_name = ent->d_name;
+ utility_list.push_back(std::make_pair<std::string, std::string>
+ (util_name.substr(0, util_name.length() - 2),
+ util_name.substr(util_name.length() - 1, 1)));
+ }
+ closedir(groff_dir);
+ }
+ else {
+ fprintf(stderr, "Could not open the directory: ./groff\nRefer to the "
+ "section \"Populating groff scripts\" in README!\n");
+ return EXIT_FAILURE;
+ }
+ }
+
+ // Check if the directory 'generated_tests' exists.
+ if (stat(tests_dir, &sb) || !S_ISDIR(sb.st_mode)) {
+ boost::filesystem::path dir(tests_dir);
+ if (boost::filesystem::create_directory(dir))
+ std::cout << "Directory created: " << tests_dir << std::endl;
+ }
+
+ // Generate a license to be added in the generated scripts.
+ license = add_license::generate_license();
+
+ for (const auto &util : utility_list) {
+ test_file = tests_dir + util.first + "_test.sh";
+
+ // Check if the test file already exists. In
+ // case it does, confirm before proceeding.
+ if (!flag && !stat(test_file.c_str(), &sb)) {
+ std::cout << "Test file(s) already exists. Overwrite? [y/n] ";
+ std::cin >> answer;
+
+ switch (answer) {
+ case 'n':
+ case 'N':
+ fprintf(stderr, "Stopping execution\n");
+ flag = 1;
+ break;
+
+ default:
+ // TODO capture newline character
+ flag = 1;
+ break;
+ }
+ }
+
+ // Enable line-buffering on stdout.
+ setlinebuf(stdout);
+
+ std::cout << "Generating test for: " + util.first
+ + '('+ util.second + ')' << " ...";
+
+ boost::thread api_caller(generate_test::generate_test, util.first, util.second);
+ if (api_caller.try_join_for(boost::chrono::seconds(TIMEOUT))) {
+ // API call successfully returned within TIMEOUT (seconds).
+ std::cout << "Successful\n";
+ }
+ else {
+ // API call timed out.
+ std::cout << "Failed!\n";
+ // Remove the incomplete test file.
+ remove(("generated_tests/" + util.first + "_test.sh").c_str());
+ }
+ }
+
+ return EXIT_SUCCESS;
+}
Index: tools/tools/smoketestsuite/src/read_annotations.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/read_annotations.h
@@ -0,0 +1,33 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <unordered_set>
+
+namespace annotations {
+ void read_annotations(std::string, \
+ std::unordered_set<std::string>&);
+}
Index: tools/tools/smoketestsuite/src/read_annotations.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/read_annotations.cpp
@@ -0,0 +1,58 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <dirent.h>
+#include <fstream>
+#include <iostream>
+#include <string>
+#include "read_annotations.h"
+
+// Read the annotation files and skip
+// generation of the respective tests.
+void
+annotations::read_annotations(std::string utility,
+ std::unordered_set<std::string>& annot)
+{
+ std::string line;
+ // TODO do this for all the annotation files
+ std::ifstream annot_fstream;
+ annot_fstream.open("annotations/" + utility + "_test.annot");
+
+ while (getline(annot_fstream, line)) {
+ // Add a unique identifier for no_arguments testcase
+ if (!line.compare(0, 12, "no_arguments"))
+ annot.insert("*");
+
+ // Add flag value for supported argument testcases
+ // Doing so we ignore the "invalid_usage" testcase
+ // as it is guaranteed to always succeed.
+ else if (!line.compare(2, 4, "flag"))
+ annot.insert(line.substr(0, 1));
+ }
+
+ annot_fstream.close();
+}
Index: tools/tools/smoketestsuite/src/scripts/README.md
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/scripts/README.md
@@ -0,0 +1,8 @@
+# Handy scripts
+
+Script Name | Functionality
+------------------|-----------------
+fetch_groff.sh | Lists location of all section 1 utilities
+fetch_utils.sh | Saves all the base utilities in the src tree in **utils_list**
+generate_annot.sh | Populates annotation files under [annotations](../annotations)
+validate.sh | Validates side-effects of newly introduced changes in the tool
Index: tools/tools/smoketestsuite/src/scripts/fetch_groff.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/scripts/fetch_groff.sh
@@ -0,0 +1,51 @@
+#!/bin/sh
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+
+# Script for listing location of groff scripts of utilities in section 1
+
+set -e
+
+src="$HOME/freebsd"
+pwd=$(pwd)
+dir_list="$pwd/scripts/utils_list"
+groff_src="$pwd/groff"
+section=1 # Section number for filtering the base utilities
+
+rm -rf "$groff_src" && mkdir "$groff_src"
+cd "$src"
+
+while IFS= read -r dir_entry
+do
+ for file in "$dir_entry"/*
+ do
+ # Check for only section 1 entries
+ case "$file" in
+ *.1) cp "$file" "$pwd/groff"
+ esac
+ done
+done< "$dir_list"
Index: tools/tools/smoketestsuite/src/scripts/fetch_utils.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/scripts/fetch_utils.sh
@@ -0,0 +1,45 @@
+#! /bin/sh
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+
+# Script for listing all the base utilities
+
+set -e
+src="$HOME/freebsd"
+pwd=$(pwd)
+dir="$pwd/scripts"
+
+fetch_utils() {
+ cd "$path"
+ find . -name Makefile | xargs grep -l 'PROG\|PROG_CXX' \
+ | sed -e 's|/Makefile$||' | cut -c 3-
+}
+
+rm -f utils_list
+
+path="$HOME/freebsd"
+(fetch_utils) >> "$dir/utils_list"
Index: tools/tools/smoketestsuite/src/scripts/generate_annot.sh
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/scripts/generate_annot.sh
@@ -0,0 +1,92 @@
+#!/bin/sh
+#
+# Copyright 2017 Shivansh Rai
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# $FreeBSD$
+
+# Script for generating annotations based on generated tests.
+
+pwd=$(pwd)
+suffix="_test"
+extension=".sh"
+update_required=0
+
+message="Following annotation files were updated. \n"
+
+printf "
++--------------------------------------------------+
+| Checking if any annotation file is to be updated |
++--------------------------------------------------+\n"
+
+for f in "generated_tests"/*
+do
+ annotations=""
+ file=$(basename "$f")
+ test=${file%$extension}
+ utility=${test%$suffix}
+ test_dir="/usr/tests/bin/$utility"
+
+ (
+ cd "$test_dir" || exit
+ report=$(kyua report)
+ i=2
+
+ while true
+ do
+ testcase=$(printf "%s" "$report" | awk 'NR=='"$i"' {print $1}')
+ status=$(printf "%s" "$report" | awk 'NR=='"$i"' {print $3}')
+ check=$(printf "%s" "$testcase" | cut -s -f1 -d":")
+
+ if [ "$check" != "$test" ]; then
+ if [ "$annotations" ]; then
+ if [ $update_required = 0 ]; then
+ printf $message
+ update_required=1
+ fi
+
+ annotations_file="$pwd/annotations/$test.annot"
+ # Append only the new annotations
+ printf "$annotations" > "$annotations_file.temp"
+ [ ! -e "$annotations_file" ] && touch "$annotations_file"
+ comm -13 "$annotations_file" "$annotations_file.temp" >> \
+ "$annotations_file" && printf "\t%s\n" "annotations/$test.annot"
+ rm -f "$annotations_file.temp"
+ fi
+
+ break
+ fi
+
+ if [ "$status" = "failed:" ]; then
+ testcase=${testcase#"$test:"}
+ annotations="$annotations$testcase\n"
+ fi
+
+ i=$((i+1))
+ done
+ )
+
+done
+
+printf "==============================================================================\n\n"
Index: tools/tools/smoketestsuite/src/utils.h
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/utils.h
@@ -0,0 +1,50 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <list>
+#include <unordered_map>
+
+namespace utils {
+ // Option relation which maps option names to
+ // a unique identifier in their description.
+ typedef struct opt_rel {
+ char type; // Option type: (s)short/(l)long.
+ std::string value; // Name of the option.
+ std::string keyword; // The keyword which should be looked up in the
+ // message (if) produced when using this option.
+ } opt_rel;
+
+ class opt_def {
+ public:
+ std::list<std::string> opt_list; // list of all the accepted options with unknown usage.
+ std::unordered_map<std::string, opt_rel> opt_map; // Map "option value" to "option definition".
+ std::unordered_map<std::string, opt_rel>::iterator opt_map_iter;
+
+ void insert_opts();
+ std::list<opt_rel*> check_opts(std::string);
+ };
+}
Index: tools/tools/smoketestsuite/src/utils.cpp
===================================================================
--- /dev/null
+++ tools/tools/smoketestsuite/src/utils.cpp
@@ -0,0 +1,130 @@
+//
+// Copyright 2017 Shivansh Rai
+// All rights reserved.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions
+// are met:
+// 1. Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+// 2. Redistributions in binary form must reproduce the above copyright
+// notice, this list of conditions and the following disclaimer in the
+// documentation and/or other materials provided with the distribution.
+//
+// THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+// ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
+// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+// OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+// LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+// OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+// SUCH DAMAGE.
+//
+// $FreeBSD$
+
+#include <fstream>
+#include <iostream>
+#include <cstdlib>
+#include "utils.h"
+
+// Insert a list of user-defined option definitions
+// into a hashmap. These specific option definitions
+// are the ones which one can be easily tested.
+void
+utils::opt_def::insert_opts()
+{
+ // Option definitions.
+ opt_rel h_def; // '-h'
+ h_def.type = 's';
+ h_def.value = "h";
+ h_def.keyword = "help";
+
+ opt_rel v_def; // '-v'
+ v_def.type = 's';
+ v_def.value = "v";
+ v_def.keyword = "version";
+
+ // `opt_map` contains all the options
+ // which can be "easily" tested.
+ opt_map.insert(std::make_pair<std::string, opt_rel>
+ ("h", (opt_rel)h_def));
+ opt_map.insert(std::make_pair<std::string, opt_rel>
+ ("v", (opt_rel)v_def));
+};
+
+// For the utility under test, find the supported options
+// present in the hashmap generated by insert_opts()
+// and return them in a form of list of option relations.
+std::list<utils::opt_rel*>
+utils::opt_def::check_opts(std::string utility)
+{
+ std::string line; // An individual line in a man-page.
+ std::string opt_name; // Name of the option.
+ std::string opt_ident = ".It Fl"; // Identifier for an option in man page.
+ std::string buffer; // Option description extracted from man-page.
+ std::string opt_string; // Identified option names.
+ int opt_pos; // Starting index of the (identified) option.
+ int space_index; // First occurrence of space character
+ // in a multi-word option definition.
+ std::list<opt_rel*> ident_opt_list; // List of identified option relations (opt_rel's).
+
+ // Generate the hashmap opt_map.
+ insert_opts();
+
+ // TODO: Section number cannot be hardcoded.
+ std::ifstream infile("groff/" + utility + ".1");
+
+ // Search for all the options accepted by the
+ // utility and collect those present in `opt_map`.
+ while (std::getline(infile, line)) {
+ if ((opt_pos = line.find(opt_ident)) != std::string::npos) {
+ opt_pos += opt_ident.length() + 1; // Locate the position of option name.
+
+ if (opt_pos > line.length()) {
+ // This condition will trigger when a utility
+ // supports an empty argument, e.g. tset(issue #9)
+ continue;
+ }
+
+ // Check for long options ; While here, also sanitize
+ // multi-word option definitions in a man page to properly
+ // extract short options from option definitions such as:
+ // .It Fl r Ar seconds (taken from date(1)).
+ if ((space_index = line.find(" ", opt_pos + 1, 1))
+ != std::string::npos)
+ opt_name = line.substr(opt_pos, space_index - opt_pos);
+ else
+ opt_name = line.substr(opt_pos);
+
+ // Check if the identified option matches the identifier.
+ // `opt_list.back()` is the previously checked option, the
+ // description of which is now stored in `buffer`.
+ if (!opt_list.empty() &&
+ (opt_map_iter = opt_map.find(opt_list.back()))
+ != opt_map.end() &&
+ buffer.find((opt_map_iter->second).keyword) != std::string::npos) {
+ ident_opt_list.push_back(&(opt_map_iter->second));
+
+ // Since the usage of the option under test
+ // is known, we remove it from `opt_list`.
+ opt_list.pop_back();
+ }
+
+ // Update the list of valid options.
+ opt_list.push_back(opt_name);
+
+ // Empty the buffer for next option's description.
+ buffer.clear();
+ }
+ else {
+ // Collect the option description until next
+ // valid option definition is encountered.
+ buffer.append(line);
+ }
+ }
+
+ return ident_opt_list;
+}

File Metadata

Mime Type
text/plain
Expires
Sat, Jan 17, 4:07 PM (18 h, 40 m)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
27696801
Default Alt Text
D12249.id33226.diff (49 KB)

Event Timeline