This is a response to Tom Hollingsworth's (@networkingnerd) video "Disaster Recovery is a Security Function," found here: https://gestaltit.com/tomversations/tom/disaster-recovery-is-a-security-function-tomversations-episode-25/.

I respectfully disagree w/Tom's assertions in his video, and decided to use this as the first episode I'm going to publish a video version of. You can listen to the podcast on all the usual podcast channels, or watch the video version on youtube here: https://youtu.be/ym_ibNWVjgA

Tom said that backup and security are very closely related, and suggested that if we reported to the same team, we could perhaps accomplish more together. While I understand the point he is making, I disagree with it, and Prasanna and I discuss it on this episode. We believe Tom's opinion comes from an outdated concept of how security works in backup systems; we haven't worked like that in quite some time. I explain how modern backup systems work from a security perspective, then talk about the idea of backup folks reporting to security folks. I think it's a bad idea for several reasons.

This week's episode is about an incident that happened at the University of Kyoto, Japan, where they lost 77 TB of research data forever. What can we learn from what happened to them? First we discuss the concept of "we can't afford backup," that seems to be prevalent in a lot of universities and research institutions. We then ask and answer the question of whether or not it is every OK to not backup data, along with whose responsibility is it? We pause the recording for what appears at first to be a spam call, but you'll have to listen to hear that. We talk about what happened there, including a letter from HPE apologizing for what happened. Kudos to HPE for that. We also discuss a story from my very first week on the job in 1993; it's not pretty.

Vinicius "Vinny" Grippa, the co-author of O'Reilly's Learning MySQL (now in its second edition) talks MySQL and MongoDB, as well as that all-important topic of how to back them up! We first learn a little bit about Percona, where Vinny works, as they consult in the database space. We then discussed a hot topic from Curtis, which is this idea of companies that say they don't want an IT department. We then discuss the book, Learning MySQL 2nd edition, and Vinny's top 3 performance suggestions for MySQL, including a discussion about the differences between MyISAM and InnoDB tables. We then discuss the typical ways people back up MySQL and MongoDB, followed by a discussion of two free tools that Percona makes available: Xtrabackup and PerconaBackup for MongoDB. It is a fascinating discussion you won't want to miss.

This week we celebrate Data Privacy Day, which is an international event that occurs every year on 28 January. According to its website, "The purpose of Data Privacy Day is to raise awareness and promote privacy and data protection best practices. It is currently observed in the United States, Canada, Nigeria, Israel and 47 European countries." Prasanna and Curtis discuss the latest in privacy practices and regulations, drawing on Prasanna's new experience protecting the privacy of Zoom's customers. We talk about the difference between PII and Personal Data, different regulations around the world, and some new tech features you can use to protect your privacy. We also discuss a few gotchas out there, such as Verizon's new system that they require you to opt out of! Happy Data Privacy Day, everyone!

Things got a little tense on this week's podcast when James Strong (@strongjz, Co-Author of O'Reilly's Networking & Kubernetes) hinted at DR being a thing of the past with K8s. Mr. Backups was having none of that. No blows were thrown, mostly because it was all online, but it was a really good conversation that K8s and DR enthusiasts alike will find interesting. We also cover the new book, Networking and Kubernetes, by James Strong and Vallery Lancey, including why the decided to write it, and what it covers. We talk about monitoring K8s networking, and James uses at least 25 acronyms that may be new to some listeners. Don't worry: we make him explain all of them.

Bob Plankers, resiliency specialist from VMware, joins us on this week's podcast, as we examine last year's major Facebook outage that took out Facebook, WhatsApp, and Instagram all at once. We discuss what we believe happened, just how bad it got, and our thoughts as to what we can learn from this huge outage. This isn't schadenfreude, and we acknowledge that we are Monday morning quarterbacking. It's an attempt to LEARN from the misfortunes of others – not to take joy in them. Solid discussion with @plankers.

On this first business day of 2022, let’s take a look back at the year that was. It was, of course, another year of COVID. In fact, Curtis contracted COVID right at the end, despite being boosted. This is also the year of the OVH fire that we talked about for three episodes:

https://www.backupcentral.com/datacenter-manager-dan-frith-discusses-the-ovh-fire-restore-it-all-podcast-105/

https://www.backupcentral.com/how-do-you-prove-your-backup-service-is-real-restore-it-all-podcast-106/

https://www.backupcentral.com/ovhs-backup-service-didnt-work-restore-it-all-podcast-107/

We think our most interesting episode of the year goes to Paul VanDyke from Kodiak Island, who deleted his whole environment and then tested his backups: https://www.backupcentral.com/it-admin-deletes-entire-datacenter-then-tests-his-backups-restore-it-all-podcast-135/

We even had a barbecue episode. No backups, just beef and BBQ. https://www.backupcentral.com/no-backups-just-beef-bbq-restore-it-all-bonus-episode/

Prasanna said he learned a lot about tape this year from these three episodes:

https://www.backupcentral.com/tape-drive-designer-schools-mr-backup-on-tape-restore-it-all-podcast-111/

https://www.backupcentral.com/deep-dive-into-why-tape-still-has-a-future-in-storage-restore-it-all-podcast-129/

https://www.backupcentral.com/fujifilm-tape-evangelist-explains-past-present-future-of-tape-lto-restore-it-all-podcast-132/

We think our best “get” was Peter Krogh, who talked about how he coined the term “3-2-1 Rule” while writing the first edition of The DAM Book: Digital Asset Management for Photographers.

https://www.backupcentral.com/peter-krogh-who-coined-the-3-2-1-rule-on-our-podcast-restore-it-all-podcast-131/

This led to a brief discussion about solar flares, where I mention a talk I watched by Intel. This is a link to that talk:

https://techfieldday.com/event/eicd16/

We also had two Druva competitors on this year, Veeam and HYCU.

https://www.backupcentral.com/dave-russell-answers-our-questions-about-veeam-restore-it-all-podcast-104/

https://www.backupcentral.com/veeam-reps-explain-defense-against-conti-ransomware-restore-it-all-podcast-127/

https://www.backupcentral.com/hycu-vp-explains-their-service-to-mr-backup-2/

The big winner of the year was ransomware. We talked about it a lot:

https://www.backupcentral.com/ransomware-victim-tells-his-story-restore-it-all-podcast-96/

https://www.backupcentral.com/is-entity-level-encryption-the-answer-to-exfiltration-ransomware-restore-it-all-podast-119/

https://www.backupcentral.com/protecting-your-network-from-ransomware-restore-it-all-podcast-122/

https://www.backupcentral.com/restoring-quickly-from-a-ransomare-attack-with-a-long-dwell-time-restore-it-all-podcast-123/

https://www.backupcentral.com/veeam-reps-explain-defense-against-conti-ransomware-restore-it-all-podcast-127/

Curtis this everyone should be looking into an intelligent DDI (DNS, DHCP, IP management) system that will spot (and stop) ransomware when it tries to reach out to its command and control servers.

https://www.backupcentral.com/stop-ransomware-in-its-tracks-with-dns-dhcp-ipam-restore-it-all-podcast-87/

You also need to monitor your bandwidth to look for exfiltration:

https://www.backupcentral.com/securing-speeding-up-network-traffic/

https://www.backupcentral.com/stop-ransomware-attacks-in-seconds-restore-it-all-podcast-126/

Finally, we talked a little about the book, and the upcoming 2022.

https://www.backupcentral.com/why-you-need-a-copy-of-modern-data-protection-restore-it-all-podcast-110/

Happy New Year, everyone! Here’s to a better 2022!

This week’s guest tells the most incredible story we’ve ever had on the podcast. We’ve had ransomware restores, disaster recoveries after a hurricane, but we’ve never had someone who deleted their entire computing environment and then restored it using their backups. (Backups that had never been tested to this degree, BTW.)

Paul VanDyke is the IT Supervisor at the Kodiak Island Borough in Alaska, which is the second largest island in the US and has to satisfy its backup and DR needs while staying on the island. Cloud resources are not a possibility due to bandwidth concerns, so he’s doing things “old school.” We first talk about the kinds of things they are protecting from, including tsunamis, fires, and strong winds. They are primarily based on tape, and for DR they store copies of all backups in a nearby safe. We discussed ways they could improve their resilience, such as shipping some tapes to a location on the mainland.

But the highlight of this episode is the story of when Paul intentionally destroyed his entire environment and then tested his backup system! He learned many valuable lessons, starting with “don’t ever do that again!” Luckily, his test was successful, albeit not without some challenges. He wiped the storage arrays on five servers: two domain controllers, an email server, a file server, and an application server and then restored them. (He had his reasons for doing it this way, which he goes into in the podcast.)

One big thing he learned was how restores are often slower than backups. So he prioritized critical apps (e.g. email, fileserver, logins) and got them up by Monday morning. Then it took him a few more days to get the application server up and running due to a more complicated restore. We have a really good discussion on how Paul could have done things better, including a really good idea that Prasanna came up with it. Curtis also tells a similar story about the first time he “tested” backups when he actually needed them, versus doing it in advance.

We cover a number of topics and questions on this podcast:

What was an Exabyte Mammoth (M2) tape drive?

What is a helical scan tape drive?

What is multiplexing?

Why can restores be slower than backups?

What happens when you rebuild a RAID array?

Should you have a post-mortem after a large incident?

How important is recovery testing?

How important is it to set expectations in IT, especially when it comes to recovery times?

The founder of rsync.net, John Kozubik, joins us on the podcast this week. It's a unique offering: a ZFS filesystem running in a private cloud – accessible only via SSH – that is designed just for sending your backup data to. They support anything that can run over SSH. Use rsync, scp, etc. to copy your data unencrypted, or something like restic, duplicity, or borg, if you want your backups to be encrypted. (All backups are encrypted in flight, of course, because they are all over SSH.).

The servers are completely locked down except for the SSH port, so they're about as secure as they can be for what they are. You can configure ssh to behave the way you want it (e.g. passphrase, MFA, etc.), and the ZFS filesystem automatically creates daily snapshots of the backups you send there. (More complicated schedules can also be created.)

You pay by the gigabyte ($.025/GB/mth) for the size of the ZFS filesystem and its associated snapshots, but they urge you to NOT over-provision. Provisioning is easy and non-disruptive, so only add storage when you need it. For an extra fee ($.017/GB/mth), they can also replicate your backups to another region.

It's a no-nonsense offering that seems to be unique out there – especially when you add the ZFS features. Check out the website and rsync.net, and you'll see they aren't spending any money on being flashy. They just want to build a rock-solid ZFS syncing destination that is separate from any cloud provides.

This week, we talk to Nick Craigwood, the creator and principal developer of rclone, a very popular open-source tool for copying data to and from cloud providers. Rclone is downloaded roughly 250,000 times each month, and has over 30,000 stars on GitHub. There are six core developers, and a great community of users and other developers at rclone.org.

We talk a little bit about Nick’s development philosophy, which is that he doesn’t mind adding features - as long as they don’t break backwards compatibility. Then we talk about how rclone works, and what it’s like to sync a filesystem to an object store – including support for multi-part uploads and downloads. We also talk about rclone’s encryption support, while Nick was “relaxing” on holiday. We then talked about how rclone can be used to minimize the risk of backing up to any one cloud provider, preventing things like what happened during the OVH fire earlier in 2021. We also discuss some strategies, such as backing up directly to two different clouds, versus backing up to one, then syncing to another – and how CloudFlare’s R2 might figure into things. Finally, we talk about Nick’s plans for rclone’s future, such as making their web UI better to increase usability for many more people – while not sacrificing the command line. Join us for a fascinating episode, the first one where we’re talking to the creator of the tool in question.

Don’t forget the drawing for a free e-book version of Modern Data Protection. All you have to do to be eligible is sign up for my newsletter at https://www.backupcentral.com/subscribe-to-our-newsletter/