Below is the file 'TODO' from this revision. You can also download the file.
Automated Testing Framework (atf) Copyright (c) 2007 The NetBSD Foundation, Inc. All rights reserved. Things to do ============ This project is still in very early stages of development. The list below is very incomplete, but it details some items that have to be addressed. Basically they are here so that they are not forgotten. * Clean the environment before running a test case. E.g. LANG, TZ, LC_*... * Set the umask to a known value before running a test case. * Make atf-run account for the total executed test programs and how many of them failed. Possibly also collect their names. * Fix atf-run to show an error if a test program did not execute any test case. * Fix atf-run to show an error if a test program did not provide a result message for those test cases that failed or were skipped. * Raise errors for unknown tests in Atffiles, but be careful with globs. * Add a way to specify which bug/issue/whatever a given test case is stress-testing. This information is useful when detecting regressions. * Allow users to customize the build of atf by defining additional meta-data for test cases. At the moment this is possible because the meta-data is not sanity-checked, but I think it should be. Following the previous item, NetBSD could add a 'netbsd.pr' variable and then use this data when generating reports to add direct links to the appropriate PRs. * An atf_check_equal function for shell tests can be useful. * Add a class, similar in idea to boost::noncopyable, to easily mark some classes as not copyable. Or, failing to do that, any other way to do the same. * Test cases should be safe to closing stdin, stdout and stderr, to some extent. Once implemented, make t_file_handle.cpp and t_pipe.cpp use stdout again, instead of stderr. * Add format and versioning tags to the outputs that have to be machine- parseable. For example: the output of test programs. * Add the ability to specify the work directory for the tests. * Implement the atf-report tool, which takes the output of atf-run and converts it to some nice format. Ideas for these output formats include: proplib, a home-grown XML format, and HTML pages. * Make sure that the tests in tests/atf have, at the very least, the same coverage as the ones in tests/bootstrap. * Add manual pages for the tools and the library interfaces. * Document the code. * Add a way to add an upper-bound limit for test run times. Otherwise a test that enters an infinite loop can stall the whole process. * Possibly add a way to automatically gain or drop privileges when require.user is set. Things to we will not do ======================== This is a list of things that will *not* be addressed anytime soon in the project. Of course most of them would be nice to have in the future, but they will not block releases nor drive development. We can obviously change our mind in the future and add move some of these to the above list. * Native Win32 support: we are not talking about building atf with tools such as Cygwin or MinGW, which should already be possible. Native Win32 support means modifying the code to use Win32 library calls and easily build out of the box (i.e. the GNU Autotools are not useful in that case). * Extreme efficiency: even though we will focus on using the most suitable data structures in each situation, we will not attempt to get extreme efficiency by adding hacks that make the code uglier. Testing is a task that requires a lot of resources on its own, and some tests will be specially intensive, so there is no point in micro-optimizing atf if the global gains are negligible. * Extreme portability: the initial goal of this project was to provide a testing framework for the NetBSD Operating System. Given that this system has a very nice build framework, we can be sure that atf will be built and used with sane tools (basically a decent C++ compiler and a POSIX-compliant shell interpreter). We will definitely not aim for portability to broken systems by tweaking our code to not follow the standards. -- EOF