Meet the New Test Framework
In early December, we pushed a new test framework to illumos. While facilitating the addition of unit tests, this framework also served as the landing zone for a port of the ZFS Test Suite, which previously ran under STF (Solaris Test Framework). As doing this port represented an incremental investment on top of simply fixing the tests that were broken, we needed to be certain jettisoning STF was worth it. Below are some of the points that informed our decision.
Tests Live with the Source
We knew from the beginning that we wanted the tests to live in the same repository as the source. This delivers us from the horrors of spending three hours running the suite only to return to a gigantic pile of failures and the realization that you ran your tests on OS bits that were too old or in some cases, too new! It also allows developers making changes in the OS to easily update the relevant tests. In order to realize this goal, we needed to wire the build mechanics of STF into illumos.
One of the eccentricities of STF is that the framework itself is responsible for building any of the tests that run under it, and indeed the framework creates even its own package via a proto area. This meant additional software that would need to be installed on a build machine as part of the build process itself. At this point, it was clear that a creating a new framework would be easier than unraveling the old one to make it fit into the illumos build process.
We now build these tests with traditional Makefiles, which build more quickly than the STF method, with the added bonus of lint cleanliness for the (admittedly few) .c files.
In the course of testing, it’s quite common to need to run tests at different privilege levels. For example, one test may need to verify a ‘regular’ user may not modify a file with given permissions while another verifies that root must be able to do so. The ZFS Test Suite has two solutions to this dilemma. The first solution is built into STF – it’s a setuid root binary called stf_gosu which execs the arguments after calling setuid(0). The other solution is a required package called SUNWstc-runwattr, which is essentially a setuid root copy of ppriv(1).
Even if ‘only’ running on a lab machine, this was clearly a suboptimal solution. With the new framework, each test may run as a specific user, and that access is controlled via the sudoers(4) file. This provides the flexibility to run tests at varying privilege levels, while not leaving a root login for every local user.
Ease of Configuration
STF finds tests to execute by recursing through the directory structure starting at the current working directory and consuming the contents of ‘stf_description’ files delivered with the test packages. This makes it easy to execute all the tests under a given directory. Cherry picking tests from different directories is significantly more awkward.
The new framework provides an ini style configuration file with which users can specify any number of arbitrary tests desired, and run them in any order desired. This has the pleasant side effect of allowing all the tests to live in the illumos gate, while any number of downstream distros may maintain configuration files specific to relevant features or desired test runtimes. And for developers, this provides the ability to run only the set of tests important to them in the course of bug fixing or feature development.
So… Now What?
While a large number of tests were ported in the initial push, there are still some ZFS tests left to port over from STF. This is a great way to make a contribution to illumos in general and ZFS specifically. There are also numerous other tests from the STC (Solaris Test Collection) that might provide valuable test coverage that’s currently missing in illumos. Have a look; perhaps something there will strike your fancy or perhaps even inspire you to write tests of your own in another area.
Perhaps you work in one of the many distributions other than OI where ZFS can be found today. If you’re interested in using these tests, and/or contributing tests you’ve written for ZFS on your platform back to illumos then feel free to use this repo as a sort of staging area where you can create a pull request with a slightly lower barrier to entry than is required to make an illumos RTI.