The Backup Wrap-Up

Vinicius "Vinny" Grippa, the co-author of O'Reilly's Learning MySQL (now in its second edition) talks MySQL and MongoDB, as well as that all-important topic of how to back them up! We first learn a little bit about Percona, where Vinny works, as they consult in the database space. We then discussed a hot topic from Curtis, which is this idea of companies that say they don't want an IT department. We then discuss the book, Learning MySQL 2nd edition, and Vinny's top 3 performance suggestions for MySQL, including a discussion about the differences between MyISAM and InnoDB tables. We then discuss the typical ways people back up MySQL and MongoDB, followed by a discussion of two free tools that Percona makes available: Xtrabackup and PerconaBackup for MongoDB. It is a fascinating discussion you won't want to miss.

This week we celebrate Data Privacy Day, which is an international event that occurs every year on 28 January. According to its website, "The purpose of Data Privacy Day is to raise awareness and promote privacy and data protection best practices. It is currently observed in the United States, Canada, Nigeria, Israel and 47 European countries." Prasanna and Curtis discuss the latest in privacy practices and regulations, drawing on Prasanna's new experience protecting the privacy of Zoom's customers. We talk about the difference between PII and Personal Data, different regulations around the world, and some new tech features you can use to protect your privacy. We also discuss a few gotchas out there, such as Verizon's new system that they require you to opt out of! Happy Data Privacy Day, everyone!

Things got a little tense on this week's podcast when James Strong (@strongjz, Co-Author of O'Reilly's Networking & Kubernetes) hinted at DR being a thing of the past with K8s. Mr. Backups was having none of that. No blows were thrown, mostly because it was all online, but it was a really good conversation that K8s and DR enthusiasts alike will find interesting. We also cover the new book, Networking and Kubernetes, by James Strong and Vallery Lancey, including why the decided to write it, and what it covers. We talk about monitoring K8s networking, and James uses at least 25 acronyms that may be new to some listeners. Don't worry: we make him explain all of them.

Bob Plankers, resiliency specialist from VMware, joins us on this week's podcast, as we examine last year's major Facebook outage that took out Facebook, WhatsApp, and Instagram all at once. We discuss what we believe happened, just how bad it got, and our thoughts as to what we can learn from this huge outage. This isn't schadenfreude, and we acknowledge that we are Monday morning quarterbacking. It's an attempt to LEARN from the misfortunes of others – not to take joy in them. Solid discussion with @plankers.

On this first business day of 2022, let’s take a look back at the year that was. It was, of course, another year of COVID. In fact, Curtis contracted COVID right at the end, despite being boosted. This is also the year of the OVH fire that we talked about for three episodes:

https://www.backupcentral.com/datacenter-manager-dan-frith-discusses-the-ovh-fire-restore-it-all-podcast-105/

https://www.backupcentral.com/how-do-you-prove-your-backup-service-is-real-restore-it-all-podcast-106/

https://www.backupcentral.com/ovhs-backup-service-didnt-work-restore-it-all-podcast-107/

We think our most interesting episode of the year goes to Paul VanDyke from Kodiak Island, who deleted his whole environment and then tested his backups: https://www.backupcentral.com/it-admin-deletes-entire-datacenter-then-tests-his-backups-restore-it-all-podcast-135/

We even had a barbecue episode. No backups, just beef and BBQ. https://www.backupcentral.com/no-backups-just-beef-bbq-restore-it-all-bonus-episode/

Prasanna said he learned a lot about tape this year from these three episodes:

https://www.backupcentral.com/tape-drive-designer-schools-mr-backup-on-tape-restore-it-all-podcast-111/

https://www.backupcentral.com/deep-dive-into-why-tape-still-has-a-future-in-storage-restore-it-all-podcast-129/

https://www.backupcentral.com/fujifilm-tape-evangelist-explains-past-present-future-of-tape-lto-restore-it-all-podcast-132/

We think our best “get” was Peter Krogh, who talked about how he coined the term “3-2-1 Rule” while writing the first edition of The DAM Book: Digital Asset Management for Photographers.

https://www.backupcentral.com/peter-krogh-who-coined-the-3-2-1-rule-on-our-podcast-restore-it-all-podcast-131/

This led to a brief discussion about solar flares, where I mention a talk I watched by Intel. This is a link to that talk:

https://techfieldday.com/event/eicd16/

We also had two Druva competitors on this year, Veeam and HYCU.

https://www.backupcentral.com/dave-russell-answers-our-questions-about-veeam-restore-it-all-podcast-104/

https://www.backupcentral.com/veeam-reps-explain-defense-against-conti-ransomware-restore-it-all-podcast-127/

https://www.backupcentral.com/hycu-vp-explains-their-service-to-mr-backup-2/

The big winner of the year was ransomware. We talked about it a lot:

https://www.backupcentral.com/ransomware-victim-tells-his-story-restore-it-all-podcast-96/

https://www.backupcentral.com/is-entity-level-encryption-the-answer-to-exfiltration-ransomware-restore-it-all-podast-119/

https://www.backupcentral.com/protecting-your-network-from-ransomware-restore-it-all-podcast-122/

https://www.backupcentral.com/restoring-quickly-from-a-ransomare-attack-with-a-long-dwell-time-restore-it-all-podcast-123/

https://www.backupcentral.com/veeam-reps-explain-defense-against-conti-ransomware-restore-it-all-podcast-127/

Curtis this everyone should be looking into an intelligent DDI (DNS, DHCP, IP management) system that will spot (and stop) ransomware when it tries to reach out to its command and control servers.

https://www.backupcentral.com/stop-ransomware-in-its-tracks-with-dns-dhcp-ipam-restore-it-all-podcast-87/

You also need to monitor your bandwidth to look for exfiltration:

https://www.backupcentral.com/securing-speeding-up-network-traffic/

https://www.backupcentral.com/stop-ransomware-attacks-in-seconds-restore-it-all-podcast-126/

Finally, we talked a little about the book, and the upcoming 2022.

https://www.backupcentral.com/why-you-need-a-copy-of-modern-data-protection-restore-it-all-podcast-110/

Happy New Year, everyone! Here’s to a better 2022!

This week’s guest tells the most incredible story we’ve ever had on the podcast. We’ve had ransomware restores, disaster recoveries after a hurricane, but we’ve never had someone who deleted their entire computing environment and then restored it using their backups. (Backups that had never been tested to this degree, BTW.)

Paul VanDyke is the IT Supervisor at the Kodiak Island Borough in Alaska, which is the second largest island in the US and has to satisfy its backup and DR needs while staying on the island. Cloud resources are not a possibility due to bandwidth concerns, so he’s doing things “old school.” We first talk about the kinds of things they are protecting from, including tsunamis, fires, and strong winds. They are primarily based on tape, and for DR they store copies of all backups in a nearby safe. We discussed ways they could improve their resilience, such as shipping some tapes to a location on the mainland.

But the highlight of this episode is the story of when Paul intentionally destroyed his entire environment and then tested his backup system! He learned many valuable lessons, starting with “don’t ever do that again!” Luckily, his test was successful, albeit not without some challenges. He wiped the storage arrays on five servers: two domain controllers, an email server, a file server, and an application server and then restored them. (He had his reasons for doing it this way, which he goes into in the podcast.)

One big thing he learned was how restores are often slower than backups. So he prioritized critical apps (e.g. email, fileserver, logins) and got them up by Monday morning. Then it took him a few more days to get the application server up and running due to a more complicated restore. We have a really good discussion on how Paul could have done things better, including a really good idea that Prasanna came up with it. Curtis also tells a similar story about the first time he “tested” backups when he actually needed them, versus doing it in advance.

We cover a number of topics and questions on this podcast:

What was an Exabyte Mammoth (M2) tape drive?

What is a helical scan tape drive?

What is multiplexing?

Why can restores be slower than backups?

What happens when you rebuild a RAID array?

Should you have a post-mortem after a large incident?

How important is recovery testing?

How important is it to set expectations in IT, especially when it comes to recovery times?

The founder of rsync.net, John Kozubik, joins us on the podcast this week. It's a unique offering: a ZFS filesystem running in a private cloud – accessible only via SSH – that is designed just for sending your backup data to. They support anything that can run over SSH. Use rsync, scp, etc. to copy your data unencrypted, or something like restic, duplicity, or borg, if you want your backups to be encrypted. (All backups are encrypted in flight, of course, because they are all over SSH.).

The servers are completely locked down except for the SSH port, so they're about as secure as they can be for what they are. You can configure ssh to behave the way you want it (e.g. passphrase, MFA, etc.), and the ZFS filesystem automatically creates daily snapshots of the backups you send there. (More complicated schedules can also be created.)

You pay by the gigabyte ($.025/GB/mth) for the size of the ZFS filesystem and its associated snapshots, but they urge you to NOT over-provision. Provisioning is easy and non-disruptive, so only add storage when you need it. For an extra fee ($.017/GB/mth), they can also replicate your backups to another region.

It's a no-nonsense offering that seems to be unique out there – especially when you add the ZFS features. Check out the website and rsync.net, and you'll see they aren't spending any money on being flashy. They just want to build a rock-solid ZFS syncing destination that is separate from any cloud provides.

This week, we talk to Nick Craigwood, the creator and principal developer of rclone, a very popular open-source tool for copying data to and from cloud providers. Rclone is downloaded roughly 250,000 times each month, and has over 30,000 stars on GitHub. There are six core developers, and a great community of users and other developers at rclone.org.

We talk a little bit about Nick’s development philosophy, which is that he doesn’t mind adding features - as long as they don’t break backwards compatibility. Then we talk about how rclone works, and what it’s like to sync a filesystem to an object store – including support for multi-part uploads and downloads. We also talk about rclone’s encryption support, while Nick was “relaxing” on holiday. We then talked about how rclone can be used to minimize the risk of backing up to any one cloud provider, preventing things like what happened during the OVH fire earlier in 2021. We also discuss some strategies, such as backing up directly to two different clouds, versus backing up to one, then syncing to another – and how CloudFlare’s R2 might figure into things. Finally, we talk about Nick’s plans for rclone’s future, such as making their web UI better to increase usability for many more people – while not sacrificing the command line. Join us for a fascinating episode, the first one where we’re talking to the creator of the tool in question.

Don’t forget the drawing for a free e-book version of Modern Data Protection. All you have to do to be eligible is sign up for my newsletter at https://www.backupcentral.com/subscribe-to-our-newsletter/

Fujifilm's tape evangelist, Rich Gadomski, joins us for an interesting discussion on tape and LTO. We talk about the different subtrates that have been used over the years, and how that changed things. We then talk about LTO-9 and what that brings to market. We also talk about how tape has seen a bit of a resurgence in interest in the backup market due to the advent of ransomware. Always fun to talk to someone that can talk at this depth on such things.

The term "3-2-1 rule" comes up on almost every episode, and we have the guy that coined it with us on the podcast! How exciting! Peter Krogh coined the term fifteen years ago. He is now Chief Product Officer at Tandem Vault, but this week he is talking to us.

He first talks about how he coined the term “3-2-1 Rule” while writing the first edition of The DAM Book: Digital Asset Management for Photographers, now in it’s 3rd edition. He didn’t invent the idea of three copies and offsite backup, but he did distill it down to what we now refer to as the 3-2-1 rule. (Three copies on two media types, one of which is offsite.) We’ve played with it a bit over the years, but that is the core idea.)

He explains how digital photographers were some of the first to need significant amounts of storage -- and to have the need to protect that storage so they don’t lose everything. Hard drives were too small to hold your whole collection, so what do you do?

Like a lot of folks in this space, his love for good backups goes back to a moment when he thought he lost it all. Curtis then tells his very similar story of how his company almost lost the company’s purchasing database, which also launched his career in backups. Peter then explains the incredible importance of metadata, and the huge importance it plays in the overall value of an image.

Then we get into the nitty-gritty of what the person who coined the term “3-2-1 rule” was thinking for each of the numbers. And interestingly enough, Mr. Backup had a slightly different understanding of the 2! Peter feels that the “2” refers to different media types. (This led to a very interesting discussion about how you do what he’s asking for in today’s cloud world.) One idea he talked about is that if you have two hard drives on the same network, they’re still subject to many of the same risks, which isn’t really keeping in line with the original idea of the 2.

We then talk about those that believe that RAID is backup, and follow that with a discussion about how SaaS services aren’t backing up your data – unless they specifically say they do so in your contract. Then we get into a discussion of Peter’s company, Tandem Vault, and how they have designed the next generation of Digital Asset Management and delivered it as a SaaS offering.