OpenZFS Developer Summit 2014: OpenZFS on illumos

The OpenZFS project is growing! The second annual OpenZFS developer summit concluded just under two months ago, and overall I thought it went very well.

The OpenZFS project is growing! The second annual OpenZFS developer summit concluded just under two months ago, and overall I thought it went very well. There was roughly 70 attendees, twice as many as the previous year, and the talks given were very engaging and interesting.

I gave a short talk about ZFS on the illumos platform, and also touched briefly on some of my subjective opinions coming from a ZFS on Linux background. While a video recording of my talk is available, I thought it'd be good to also present some of that in written format, which might be a more approachable format for those who weren't able to attend the conference.

So here goes... OpenZFS History As most readers will already know, ZFS was originally designed and developed at Sun Microsystems starting around 2001. It was later released as open source in 2005 via OpenSolaris. In 2010 illumos was born as a fork of OpenSolaris, and in 2013 the OpenZFS was created. As a result of this lineage, OpenZFS really feels "at home" within the illumos project; and what I mean by that, is it has a very polished integration with the illumos operating system and the code base as a whole. Just to state a couple explicit examples of this:

  • It's very easy to boot and run ZFS as the root filesystem (even in the face of on-disk feature changes) as it and the boot loader both live and change within the same repository.
  • The debugging tools on illumos (e.g. mdb) have knowledge of ZFS-specific structures and have ZFS-specific features to allow efficient debugging and development.

But, as OpenZFS is growing, its integration on other operating systems is continually getting better.

OpenZFS on illumos Development Process

The development model for OpenZFS on illumos follows the same conventions as all other illumos development. Access to commit to the repository is granted to "advocates", and these advocates rely on the community (possibly themselves as well) to review any proposed changes, and ensure the changes are correct and are of high quality.

In practice, each proposed change must reviewed and approved by a subject matter expert (e.g. for OpenZFS, this is typically Matt Ahrens or George Wilson) prior to an advocate committing the change. Since the illumos project doesn't have any specific releases, this implies that each commit to the repository must be of high quality as each commit is effectively a "release." There isn't the option of merging code known to be defective, with the intention of fixing the defect in a follow up commit prior to the "next release." 

OpenZFS on illumos Collaboration

Since OpenZFS on illumos is the "upstream" platform for OpenZFS, it's important that there is open communication and collaboration between the illumos developers and developers of the other platforms. As changes from one platform are merged into illumos, all other platforms will (ideally) pull these changes into their own version of OpenZFS which is mutually beneficial for a number of reasons:

  1. Less time spent on merge conflicts when pulling changes in between OpenZFS platforms. If each platform is active in pushing to illumos and pulling from illumos, then we'll all be using as close to the same source code as possible.
  2. Less duplication of effort. There's been a number of times where I've personally spent effort investigating or solving a specific problem, only to find out that it's already been solved on another OpenZFS platform. If each platform was more actively kept in sync with each other, this situation would be far less likely.
  3. More varied testing. If all platforms are running the same code, then we all benefit from the variety of workloads each platform brings to the table; and as a result, we all test each others' code.
  4. Proliferation of new OpenZFS features. Currently, most new ZFS features are first developed on, and committed to illumos. Thus, if other platforms are active in pushing changes into illumos, these new features are inherently developed around and incorporate the most recent changes in each platform. This makes it much easier to pull the new features into the downstream platforms, allowing all OpenZFS consumers to benefit from the new features as soon as possible.

In order to achieve this goal, the process of building and testing changes to the OpenZFS on illumos code base needs to be simplified and streamlined. Work is on-going on this front, and my hope is that the necessary improvements will happen over the next year, and we'll be able to sing a different tune for next year's developer summit.

Developer/Debugging Tools on illumos

One of the nice things about OpenZFS development on illumos is that I get to make use of the outstanding developer tools available on the platform. Having spent much of the last few years working within the Linux kernel, I've been implicitly trained to rely almost entirely on the source code and stack traces for debugging most production and development problems. Now that I've jumped to illumos, I can see first hand what I've been missing with tools like dtrace, kmdb, and especially mdb. In my short time working on the illumos platform, I've found mdb to be easily one of the best parts about the jump from ZFS on Linux. Certain tasks that would taken hours, or have been prohibitively difficult, on Linux can be done with ease using mdb's dcmds and pipelines. For example, while working on the ZFS on Linux port I would often be puzzled by what I found when looking at the ARC kstats. To try and make some sense of it, I wanted to know exactly what was contained in the ARC and what was contained in the dbuf cache; this eventually led to the implementation of the "dbufs" proc handler on the Linux port. On illumos, there's already a mdb dcmd to do exactly that:


And better yet, I can print the full contents for each buffer if that level of detail is needed:


Or filter out the specific dbufs of interesting:


But what if one wants to filter on something that isn't supported by the ::dbufs command? Well, here's a more general approach, filtering out only the buffers with a reference count of 3, using ::grep:


Additionally, the ability to get stack traces detailing where objects were allocated using kmem_flags=0x1 is hugely beneficial. Using it, things like the ::refcount dcmd can not only provide information about the current number of holds, but it can also provide the full kernel stack trace of the thread that took the hold when the hold was taken. Likewise, the ::whatis dcmd also makes use of this information and displays it to the user. Imagine a case where you have a "random" pointer to an object (e.g. a pointer to an arc_buf_t), and wanted to know where this object came from. If using kmem_flags=0x1 on illumos, this is as simple as the following:


I could go on and on about the cool things mdb allows, but this post is already long enough, so I'll cut myself off here. It would be great to see an mdb port to Linux (or even a more feature rich version of crash with OpenZFS support)!