The Backup Wrap-Up

This week we are joined by John "Ricky" Martin, Director of Strategy at NetApp (and former owner of a tape recovery business), to talk about his paper that declares that backup is fundamentally evil and done in an unintelligent way. Mr. Backup wasn't sure how this one was going to go, and there were at least one or two arguments along the way. No blows were thrown, though. We definitely talk about what a tape recovery business is, and what it was like to do that. We also talk about tape backup, full backups, multiplexing, tape handling, and other elements of how backup is still done today by many people. It's a fun episode where you should learn a lot.

We have none other than Snorkel42 from Reddit on the podcast today. He has 20 years experience in InfoSec, and is a prolific writer on Reddit under the handle Snorkel42. (Check out his posts here: https://www.reddit.com/user/snorkel42/). (We will not be using his given name during the recording.).

He thinks Okta managed to turn a mole hill into a mountain by incorrectly handling the hack that happened in January – that we just learned about last week. That's right, we just found out about a hack that actually happened in January!

We dive deep into what happened, what it means, and how the worst problem of all is how Okta responded to it. Our expert says he no longer trusts Okta, and gives advice to customers on what to do next.

This is a very timely episode that you will really enjoy – unless you're an Okta customer or employee.

Gary Williams tells a great story about earlier in his career that taught him the value of testing backups and updating documentation. He explains how he thought his backups were fine, until a "new guy" came onto the scene and dared to ask the question, "When was the last time you tested your backups?" As Gary explains, sometimes new people have the best perspective. They let him do the first test, and .... it failed spectacularly! It all came down to the documentation they were so proud of. Hear Gary's story and learn from his mistake – one that defined his career. (Mr. Backup also tells the story that defined his career as well!)

When you back up your SaaS apps (because you know you are supposed to), should you back them up to a SaaS service or on on-premises backup system? After defining what SaaS is and isn't, Prasanna and Curtis discuss this important question. First they look at how sizing a SaaS system for backup is different than when you do it in a datacenter, and how that creates challenges for backup design. Does it make sense to use on-premises backup to backup a cloud resource like SaaS?

Learn from others' mistakes by reviewing last year's worst ransomware attacks with Mr. Backup and Prasanna Malaiyandi. Listen to them review the 10 worst attacks from 2021, then discuss lessons learned: Colonial Pipeline, BrennTag, Acer, JBS, Quanta, NBA,AXA, CNA, CD Projekt, and Kaseya. Then they discuss the trends they see, and the lessons we can all learn from these horrible attacks.

W. Curtis Preston (Mr. Backup) and Prasanna Malaiyandi weigh in on a reddit thread that started with a simple question from a user. He has MSPs trying to sell him a solution to back up Microsoft 365, and he's wondering if that's even necessary. As usual on reddit, there are many opinions. Mr. Backup debates the various points being made by the anit-backup crowd, including an article arguing the same point. We start with an overview of why it's important, then we take on the various arguments used to support not backing it up. We talk about how retention policies are not backup, and why. Even retention lock doesn't help, and neither do lagged copies of Exchange Online. We also talk about how the e-discovery tool is NOT a restore tool and will not give you what you're looking for. Finally, we talk about the idea that backup tools can't do DR for Exchange online, and how that is used to bash them as well. Great discussion on this one.

On this week's episode of Backup Central's Restore it All, Mr. Backup himself becomes the guest, while Prasanna Malaiyandi takes over as host. W. Curtis Preston explains the backup configuration of the website behind the Restore it All podcast, and how bit rot caused him to have to restore part of it. We talk about bit rot, the 3-2-1 rule, off-site backups, backups stored in S3 and Google Drive, and what it's like to restore part of a MySQL database. Luckily, the folks at LiquidWeb were very helpful. Watch Curtis explain how practices what he preaches over at BackupCentral.com.

If you want to watch the video version of this episode, it's here: https://www.youtube.com/watch?v=I3285etiYBs

This is a response to Tom Hollingsworth's (@networkingnerd) video "Disaster Recovery is a Security Function," found here: https://gestaltit.com/tomversations/tom/disaster-recovery-is-a-security-function-tomversations-episode-25/.

I respectfully disagree w/Tom's assertions in his video, and decided to use this as the first episode I'm going to publish a video version of. You can listen to the podcast on all the usual podcast channels, or watch the video version on youtube here: https://youtu.be/ym_ibNWVjgA

Tom said that backup and security are very closely related, and suggested that if we reported to the same team, we could perhaps accomplish more together. While I understand the point he is making, I disagree with it, and Prasanna and I discuss it on this episode. We believe Tom's opinion comes from an outdated concept of how security works in backup systems; we haven't worked like that in quite some time. I explain how modern backup systems work from a security perspective, then talk about the idea of backup folks reporting to security folks. I think it's a bad idea for several reasons.

This week's episode is about an incident that happened at the University of Kyoto, Japan, where they lost 77 TB of research data forever. What can we learn from what happened to them? First we discuss the concept of "we can't afford backup," that seems to be prevalent in a lot of universities and research institutions. We then ask and answer the question of whether or not it is every OK to not backup data, along with whose responsibility is it? We pause the recording for what appears at first to be a spam call, but you'll have to listen to hear that. We talk about what happened there, including a letter from HPE apologizing for what happened. Kudos to HPE for that. We also discuss a story from my very first week on the job in 1993; it's not pretty.