Delphix intern exclusive: OpenZFS Hackathon

Written by Joe Stein (Brown University '16)

Written by Joe Stein (Brown University '16)

It all began at 9 AM on a Wednesday. We sat huddled around an elliptical wooden conference table on the fourth floor of the Delphix San Francisco office, preparing for our briefing. The occasion: a three-day hackathon. But it wasn't just any hackathon.

Its first notable distinction was its participants, a mix of interns working on projects all over the Delphix ecosystem -- from the OS to the Java stack to QA to Operations -- and grizzled veterans of this hackathon's codebase. Second was its structure. While hackathons are often characterized as sleepless multi-day coding frenzies fuelled by enough caffeine to kill a horse, this one was reasonably paced over three working days, which happily reflects the Delphix work culture.

And finally, the goal: to improve ZFS. Let me add some context here: ZFS is a file system, and it's kind of scary. My intern project is in ZFS, and I'm still often intimidated by its monolithic size and vast complexity (one legendary part of the system sits in one 15,000-line file). Not only that, but most of its code is at the operating system level, meaning that the slightest misstep won't just crash your program -- it will crash your entire computer.

My point is that ZFS is quite formidable, and it almost certainly wouldn't have been possible for us participants to accomplish what we did (more on that later, it's impressive) without the expert guidance of five Delphix full-timers on the Storage team. These guys are absolute wizards of ZFS/systems programming, have many years of experience, and regularly perform what are pretty close to miracles on the kernel.

It's especially worth noting that they dedicated much of three days of their time to do pro-am collaboration with us (which must have sort of felt like a three-legged race), and four of the five traveled in from the South Bay, Las Vegas, and Atlanta (yes, Georgia) for the hackathon. All of which they did willingly and happily. I could easily extend this digression/well-deserved hero worship to the end of this post, but that's not what you want to read. So I'll detail our descent into the dark labyrinthine bowels of ZFS.

We began with an overview of ZFS: what it is, how it's different from other filesystems, and how we use it at Delphix. We then visited the various layers of ZFS, all the way from the command line to the disk. (Aside: later, some of our participants tried to reconstruct the many, many initialisms of said layers as an exercise in fun/futility.)

Next was a whirlwind walkthrough of the code paths of the most essential functions of ZFS, reading and writing -- a filesystem's first function is to store data, after all! "Deep dives" of the internal mechanisms and structures of other parts of ZFS followed.

George Wilson, Sr. Staff Engineer, teaching interns about the workings of ZFS
George Wilson, Sr. Staff Engineer, teaching interns about the workings of ZFS

Last, but certainly not least, were the demonstrations of DTrace and MDB, two powerful and somewhat arcane debugging tools. Our valiant mentors taught us to construct obscure one-liners that probed the depths of the ZFS machine, to poke and prod parts of the running system, and finally, to sleuth our way to the cause when we inevitably brought our machines to their knees or worse.

I can't speak for the other interns, but at this point, my mind was fully saturated with ZFS material. Thus, Phase 2 of the hackathon began: after the veterans presented ideas, we split off into groups and went to work. Another day of ideation and execution flew by. In observance of the hackathon ethos, pizza was heartily enjoyed by all, thanks to one of our project managers. I was definitely hacking, although I think more as in "hacking my virtual machine to pieces" than "hacking up a piece of software." It was a blast.