With Networking for Systems Administrators production wrapping up, OpenZFS Mastery is starting to lurch forward. Here’s a tidbit.

ZFS was designed by highly experienced Unix engineers who spent decades writing scripts to parse df(1) and du(1) and fstat(1) and lsof(1) output and were determined to quit relying on sed(1) and awk(1) for even the simplest operations. The ZFS commands are not only designed to work together, each offers “customizable output designed to be piped directly into other commands. They can even produce JSON for convenient automation, as we’ll illustrate with examples throughout,

OpenZFS Mastery is open for sponsorship at https://sponsor.mwl.io.

Have some OpenZFS Mastery.

“ZFS reports that one of the seventy-nine drives in our array is failing? Great! We can replace it in a convenient maintenance window before it causes problems. Uh… which drive is it?”

If you have a large storage array, you need proper physical and logical labeling. Keep a spreadsheet of each drive’s physical location and the information presented to the operating system, and also document this within ZFS itself. It’s easier than it sounds. Proper preparations during installation let you zero right in on a failed disk—even a disk at a remote facility. Jude runs a lot of very dense storage arrays in locations all over the world, and uses this scheme to keep hard drive maintenance from overwhelming him. He even records warranty information, expiration dates, model numbers, and more within ZFS so his monitoring system can flag sixty days before the warranty runs out.

It’s a couple days before I wanted to announce it, but if the new OpenZFS book is of interest you might look at https://sponsor.mwl.io.

I skipped last week due to Excessive System Administration, but here’s a snippet from OpenZFS Mastery.

Sponsorships for OpenZFS Mastery will not open until I ship out gifts for the last sponsorship. I don’t take money for a new thing until the previous new thing is complete. Stupid ethics!

Here’s a tidbit from the backers-only special edition of Networking for System Administrators.

All those shrooms were one interconnected, internally networked living thing.

Networked. Sort of like computers.

Except mushroom networks are efficient. The nodes eliminate interoperability issues by requiring identical hardware and a fully patched ShroomOS. The Internet would be infinitely better if every node ran BeOS or, better still, Fifth Edition Bell Laboratories UNIX on VAX. But the Humungous Fungus infects trees and shrubs much like the Internet infects washing machines and democracy. Most of an a. ostoyae colony is buried underground. Most of the Internet is buried in concrete. The Humungous Fungus wins on points because it is not only edible, it is a delicacy. No matter how thoroughly you stir-fry your laptop or how much black bean sauce you add, it remains unpalatable.

If you need a tie breaker? Fungi are at peace. They don’t even try to support printers.

If you didn’t back the book before now, this bit is all you will ever see.

“Trust” seems to be a recurrent theme of the OpenZFS book. Here we discuss so-called hardware RAID.

All RAID is software RAID. Your hardware RAID controller runs a custom operating system to perform RAID tasks, and in the process obscures the storage hardware from the operating system. This made sense back in the early days of widespread commercial computing, when consumer operating systems could not be trusted. Spend three seconds contemplating OS-level software RAID on Windows 3.1, and you’ll understand why hardware RAID became ubiquitous. The computing ecosystem has changed. Our operating systems have improved. Our hardware is billions of times more powerful.

ZFS is designed for direct access to the hardware. It deliberately stores critical metadata on multiple disks. It watches those disks for errors, and makes decisions based on those errors. A hardware RAID device hides all of this worrisome detail from the operating system, eliminating ZFS’ ability to heal itself. Hardware RAID presents no competing abilities.

The Sinclair ZX81 was as good as computers got.

Last week’s episode was called on account of covid. My brain is only lightly friend today, so here’s a bit of OpenZFS Mastery.

Storage is a sysadmin’s natural enemy.

You might think it’s management, but some folks touch computers for fun. Similarly, not all sysadmins have users other than themselves. Most computer hardware is fairly robust, barring encounters with free-range pile drivers. CPUs and memory rarely catch fire. Packets go missing? No biggie, just retransmit. But no matter how big your system or how many users it has or who you answer to, storage is an unending annoyance.

A computer’s filesystem dictates much of its performance and behavior. We rebuild entire systems because a major filesystem was configured incorrectly, or the filesystem chosen wasn’t suitable for the task, or because subtle filesystem corruption infiltrated our data and we can’t trust even the basic programs shipped with the operating system.

OpenZFS on Linux is best described as a “trip.” No, not a journey, a fall.

I’m at EuroBSDCon in Croatia teaching TLS and SMTP, so here’s a snippet from my TLS tutorial.

Let’s say I create a public key pair. I keep one key of the pair. The other key I give to my chief goon, Vizzini, before I dispatch him out into the world. Nobody else has either of these keys. My goon and I can use these keys to exchange messages that can be read only with the other key in the pair. I use my half to encrypt my messages to my goon. Hopefully I remember what I said, because once the message is encrypted, my key cannot decrypt my message. Fortunately, “Start a war and frame Guilder for it” is short enough that even I can remember it. Only the other key in the pair can decrypt that message. I mail my message. Anyone who snoops on that message sees only indecipherable gibberish.

When my message reaches my goons, they use their key to decrypt it. They can then use their key to encrypt a response, like “How about we kidnap the princess?” and send the encrypted message back to me. Only my half of the key can decrypt this message.

So far, so good.

Yes, the tutorial is based on TLS Mastery, which features the most apropos cover art of any book I have written.

The Networking for System Administrators Kickstarter is running full steam ahead, so I’m making words on OpenZFS Mastery. It’s a second edition of FreeBSD Mastery: ZFS and FreeBSD Mastery: Advanced ZFS, but a single large volume also covering Linux.

ZFS support in Linux distributions varies. The broader Linux community questions the compatibility between the GPL and the CDDL and in response, distributions have added and removed installer ZFS support. Intellectual property attorneys who have considered CDDL/GPL compatibility often say “it’s an interesting question,” which is lawyerese for “$500 an hour and the hearing’s gonna be lit.” This doesn’t matter to users, but operating system vendors are risk-adverse. The result? OpenZFS considers Linux a top-tier platform. Most Linux distributions do not consider ZFS a top-tier filesystem. Check your Linux documentation for installation and upgrade instructions.

We have chosen to believe that some Linux will adopt ZFS as a preferred filesystem and integrate all its features the way FreeBSD and Illumos have. Until then, Linux users often compromise to make their lives easier.

License wars are the worst.

Before you ask, I won’t open sponsorships on OpenZFS Mastery until I ship the N4SA2e sponsor gifts.

I can’t believe I’ve done 100 of these silly things. Anyway, here’s a tidbit from my next FreeBSD Journal Letters column.

Don’t give me that look. All computers increase pain.

The purpose of a system is what it does, and a computer does pain.

We cope with the pain by adding more computers.

The first Rule of System Administration declares “computers were a mistake” and examining the mental health impact of the teetering cataclysm of systems inside your computer proves it, let alone the systems in your car. Don’t get me started on your car. Requiring everyone in the United States to borrow, purchase, or otherwise “acquire” a multi-ton kinetic energy weapon was bad enough, but add in separate computers for the brakes and the accelerator and the climate control and congratulations! You’ve built a Rolling Debacle. When your drive-by-wire steering catches a SIGABRT the only question is who gets debacled, you or bystanders. I would say “innocent bystanders” but nowadays everybody uses a computer so nobody is innocent. Even folks who want to remain unsoiled are compelled to use a website or an app because the guilty make themselves feel better by dragging everyone down to their sewer. (“Come to my party! It’ll be great! Never mind the floaters!”)

Yes, I have a list of the Rules of System Administration. No, it’s not published. They’re scattered throughout the books, articles, columns, and whatever. One will get written on a bathroom wall at EuroBSDCon 2025.

I’m swamped doing math for the networking book kickstarter, trying to compensate for the recent political stupidity that’s kicking small business when it’s down, so here’s a tidbit from an older work.

Dennis Ritchie, co-creator of Unix, declared it so. Who are you to argue with someone who can write a complete operating system without using a glass teletype?1

Many younger sysadmins naively hoist their pennants to defend overblown, overwrought, overdesigned text editors like ex, vi, or even the impossibly bloated nvi. A few are so lost as to devote themselves to turgid editors meant for mere users, such as vim and Emacs. This way lies not only appalling sysadmin skills, but an absence of moral fiber. As a sysadmin, you must have enough brain power to remember what you typed, to hold your own context in your head, and to truly commune with the machine on a deep and personal level.

1 You know, a glass teletype. That toy that kids keep calling a “monitor,” even though we all know monitors are reference speakers used in audio production.

Why read this today? Because this crossed my feed…