Business Tech Playbook

#6 – Backups

1 year ago
Transcript
Robbz

This is the business tech. Playbook your source for it. Help for your business. BJ it's summertime, and I'm going to start a patio pond. What have you been doing outside for fun?

BJ

It'd be easier to kind of let you know what I haven't been doing outside for fun? Well, we're actually going camping this weekend, so I'm looking forward to that. In our backyard, we've got 22 tomato plants planted, a bunch of cucumbers and squash, about a dozen berry plants, dozen grapes, five ducks, five chickens, and just shy of 45 trees. So we spent a lot of time outside.

Robbz

Well, we need to spend some time inside and tell these fine people about backups. I'm your host, Rob Z Olsen.

BJ

And I'm your co host, William Pote. I oftentimes go by BJ because my dad was Bill and I'm the junior. My name is BJ own Etop Technology, and we're going to tell you about backups today on our Business Tech Playbook podcast.

Robbz

Now, before we dive in, I want to take a moment at the beginning of the podcast instead of reminding you at the end if you've liked our content, our content is here for you, the business owner, the manager, the small business CFO, and you're trying to make business decisions. We're here to help you demystify some technology and help you with your business tech. So if you've enjoyed the podcast, you like this. You know a business friend that would benefit from some of our tips, tricks, and help, and most of all, clarification in this crazy alphabet soup world. Share us. Share it to your friends. Subscribe. You'll find our website businesstechplaybook.com. At the bottom of the website. You can join our discord. This is actually where we would have our questions, chat and where we record the podcast live. If you want to tune in, we're doing these publishing them about every two weeks so far and would love to have you on board to add feedback or suggestions. Please share with a friend.

BJ

And we're trying to record functionally every Thursday, so that way we are a little bit ahead on our podcast schedule. And if you happen to want to jump into a live podcast, go ahead and let us know in the Discord Channel and we will make sure you get invited for the right time.

Robbz

It's also in the show notes, so check those out every episode.

BJ

Magic. Magic. It's like you have information underneath the thing.

Robbz

The thing.

BJ

Love it.

Robbz

The thing. Yeah. Hopefully we don't use the word thing as we describe our path today.

BJ

Well, it is just a thing to a lot of people, but to us, it's our life.

Robbz

It may just happen at a time. So we have a bit of a schedule here on how to tackle backups. So to get into questions, we'll go with the easy ones. Why do you backup BJ well, because.

BJ

It'S the only way to get out of a parking spot. Oh, sorry. Okay, I apologize. Being an, it what happens when everything goes wrong. That's why you back up. We functionally know that there are going to be problems, people are going to delete things. 95 or 98% of our restores aren't disaster recovery. In fact, it's probably more like 99 and a half percent of the data we restore isn't a disaster. It's literally because somebody deleted a file or dragged it. More realistically, they dragged a file and forgot where they put it and then three weeks later they need this critical information and need a restore. This is why you do backups. So that way you don't have to go and recreate the wheel every single time something gets deleted or moved legitimately.

Robbz

Just before I started the podcast, I was working with one of our clients, it's a law office. And they unacidentally deleted the entire case file for something that they had to go to the courtroom for in 2 hours. They were panicking. What were we going to do? I mean, every shred of the case file just deleted by accident. Because we had good backups, I was able to recover it within minutes. They had all their case files back. They thought that they were going to just panic, have to request a delay in the trial, and now they're good to go. Those backups really did save the day. And those incremental small I lost a file can start by saving you an hour of work by recreating the document or years of work, if that's something that you guys live and breathe in daily. So it's not just, hey, a tornado hit my building or my servers lit on fire. It's even the incremental work that really helps out.

BJ

We always plan for the disaster, but it always gets used for the simple. So the kind of running joke in our office is that Godzilla steps on your building. How do we restore everything to pregodzilla?

Robbz

And instead it was just Kathleen just spilling coffee on files or Bob, our.

BJ

Fictional CFO had their mouse battery died in the middle of a file move and then suddenly they can't figure out where the file is. We have seen like entire folders just disappear because somebody accidentally clicked and dragged and it's years of work gone. It's astonishing to me the things that we've had to restore that have nothing to do with Godzilla stepping on a building.

Robbz

You get ready to be a superhero, you put on your cape, you wear the big old e on your jersey and you just don't have to use it. Instead you saved 8 hours here, 4 hours there. Find a case file here. And that is the one thing. It is a thankless job if you can help that person recover something that they deleted by mistake. I have never had a customer more thankful in my life.

BJ

It's true. We may talk about it in this podcast, but we had a client whose building burned down and we had them running in our data center within a day. It was a really incredible chance for us to Dave the day. We didn't just have backups, but we were able to get them back up and running fully as an organization to the point where they could actually sell the company within sight of a couple of months. If we hadn't done our job properly, they wouldn't have had a company to sell other than the amount of pain that was. It was a lot of work for everyone. For them, for us, it was rough, but at the end of the day, it was one of the most interesting, cool things I've ever gotten to do for a client. When you actually walk through a disaster scenario and it all works, most businesses plan for the disaster, and most of the time it's just kathy deleting a file. Right?

Robbz

Most businesses, if they go through an actual dramatic deal like hardware failures, something happens to their building. They're missing all of those invoices for the past. Let's pretend it's just one year of invoices. Businesses don't recover from that. Can you ask yourself truly that if one year of your build invoices that you're trying to get money off of just go poof. How you're supposed to do taxes, how you're supposed to get money on that, it just all disappears? You can do your best and try to look through tickets and walk through, but then you're not running the business. You don't live after things like that, so that's why you back up. And what do you back up, DJ?

BJ

Anything you don't want to lose, which is generally everything for us. We approach it in several different layers. We look at it from what's in the building, and then we look at it from what's outside the building. We generally back up everything. So the one area we don't typically back up is endpoints. But we don't directly back them up because endpoints are becoming more and more disposable. Not from the you just throw it away. But if you have a problem, you literally reinstall Windows. And if you're using the right tools, all your stuff is there when the computer resets or you reinstall Windows. We backup servers. If it's a critical endpoint, we will back it up, and then we backup Office 365. Those are our key things that we back up personally.

Robbz

So to explain a little bit more in detail, an endpoint for those listening is someone's workstation. It's like a desktop, a laptop, the workstation that they live in. Generally speaking, when you have an employee and they're doing their work, they're living inside of your system that probably lives on the server, lives in some sort of Web application, even lives in their email, which all get backed up on the server. Anything that they have on their desktop. We a lot of times, unless we've had the conversation with the customer, assume that that's just a desktop. They haven't finished and published those documents. And when they do, they're going to put it on the server, going to put it on the OneDrive, they're going to put it out to a place that's being backed up that isn't their workstation.

BJ

That's what's supposed to happen, right? The desktop and the documents on your personal workstation or computer are supposed to be kind of a scratch place where it's stuff in progress. Realistically, even the work in progress should be saved in the central repository of the server. Whether that's local in the cloud, it doesn't matter. We treat it kind of the same way. But what happens is people aren't careful. They keep way too much stuff locally and as a result, we require all of our users to have OneDrive. And they call it Kfm. So here's some text soup, but it's known folder move. Again, not super clear, but effectively it keeps the desktop documents and pictures synchronized with Office 365. So if that computer gets thrown away, we log them back in, make sure that those folders are synced, and within about 20 minutes all of their stuff's back.

Robbz

Right?

BJ

So we expect that people aren't careful and so we try to have some contingency plans around that.

Robbz

As a rule of thumb at least, that I've had in the past with customers that I've had to do training on to make sure that they're saving their work. Stuff like engineers, stuff like artists, stuff that people have to put in a lot of time, effort, whether it's a document, a CAD file picture or movie that they're editing. My rule of thumb is if they're actively working on it and you go to break, you should save that project. That project should be saved, maybe uploaded to the server or OneDrive. That way if something happens, you didn't lose all of the work since you started the day to your break. So anytime you go on a break, you go on lunch, you leave the project, you should be saving that document, project, file, whatever it is, to the location that's going to be backed up at. That's always the training that I've done.

BJ

For years and to some extent share sorry, OneDrive with that known folder move has kind of broken me of that. The reason being is because almost all of those files are in a kind of a continuous sync state. So they have autosave on and so that autosave, like anytime you make any change, it's live, updated, that document is effectively in the cloud instantly. I just don't save anything anymore, right, personally. But I also know how we have things set up and I know that I could throw my laptop in the trash, log into another laptop and be functional within about 20 minutes.

Robbz

And you're also not doing a specialized program or something like a CAD drafter that they're having to do something that isn't incrementally changed. So getting yourself into the habit. So you do it on everything, even though it has that was it KBM?

BJ

You said Kfm kfm folder move even.

Robbz

Though it has the Kfm ability where it can auto save, and you don't have to click the save button for it to save, and it incrementally saves as you go. It's nice to get in that habit because maybe you're in a program that doesn't support that yet.

BJ

Hope you are, but no, absolutely. That is a good point, especially for those that are using a lot more production level. Almost all of the tools I use on a day to day basis are completely web based. It's much less of a risk of loss. But again, I live and breathe backups, and so I'm aware of that, and it's a risk I'm willing to take personally, but it's not a risk your team should be taking with your data.

Robbz

So a couple categories that we're going to go through. We're going to tell you about the different types of backups, and then we're going to tell you the different locations of those backups, if that makes any sense. So for the first category, number one, the full backup. This full backup creates a copy of all the data in a system or device, including files, folders, or applications. It provides a complete snapshot at a specific point in time. This is traditionally called a full one time backup. So if I was to, for instance, log into your computer or your server or whatever device I'm looking for, I would do a full backup. I would have a complete snapshot at I'm looking at my clock right now, 04:29 P.m., and everything that happened prior to 04:29 P.m. Would be saved on that one snapshot that has been saved at that one period of time. So if someone deleted something before 01:29 P.m., I'm not going to have it. If something changed, something after, I'm not capturing it. It's just 04:29 p.m. Captured in time capsule.

BJ

The best part about backups is you can have several of the different pieces that we're talking about all in play together, and it depends on the tool that you're using. Right. For us, we do what they call image backups. We're backing up. I'm trying not to dive into too deep of a technical concept.

Robbz

We got five more points to go through as well.

BJ

Yeah, I know. All of our backups start with a full. We always have to start with a full, because if you don't have that initial full backup, then at the end of the day, you have nothing to restore from.

Robbz

It's called setting a baseline for those in the industry.

BJ

Exactly. Well, from there, typically the way we run is we do an initial full backup and then we start doing what they call an incremental backup. And you can do forward incrementals and reverse incrementals. I'm not really going to go into it because it doesn't really matter, but if you hear that terminology. Effectively a forward incremental. It merges all of your snapshots into the current one. And a reverse incremental makes a chain off of your initial full. So it just depends on where your full backup lives. It changes how fast things restore. Basically, if you have any questions about it, start diving into it with your actual It provider. Realistically, it's going to be a little bit of whatever they're most comfortable with to meet your company's needs.

Robbz

I had someone try to explain to me incremental backup. So you have full backup. So let's say you made a Lego house and you now have a complete second Lego house. That is the copy of the first one. Well, as the incremental backups capture, they grab pieces where it sees sections of the house changed. And as you get more and more, you'll have a set of sections of the house, but it's not completely built. And the moment that you have to go back and put that house together, it takes time to take those sections and put it back to create the incremental house. That was the funniest, super simple oversimplification of incremental backups that I was told. And I know it's missing a lot of intricacies.

BJ

I'm going to just use numbers because it should hopefully be easier to understand. But let's say you have 100, I'm going to say 100 gigs. It doesn't really matter the amount, but just let's say your server is 100 gigs of data. When you take that full backup, you're doing the initial transfer. It's going to take depending on how fast your server is, how many other people are using it, it'll take a while. Let's say it takes 5 hours. Part of why you run an incremental is because every time you run that backup, it could take 5 hours. And that's potentially a big waste of time. But if you're taking an incremental, it's doing basically the difference between that first backup and the second backup, doing a quick comparison and only backing up the three gigs that changed. So you had a 3% difference of data throughout the day. It allows you to take, you could take maybe ten incrementals in a day. So that way you could go back to different points in the day rather than just doing one per night. Again, it goes back to what's your business need, how much data can you afford to lose and how long can you afford to be down.

Robbz

Number three that we have on the list is a differential backup. Now, similar to incremental backups, differential backups save the changes made since the last backup. However, unlike incremental backups, differentials don't distinguish between different backups. They save changes since the last full backup, which can make the restoration faster, but requires more storage, better pieces. But again, it costs more storage.

BJ

Back to what I said the last time is it goes back to how much data can you afford to lose and how long can you afford to be down when it comes to backups? Those are always the conversations we end up having. It's not something I can dictate to the client. I can't tell them how long they can afford to be down.

Robbz

Right.

BJ

That's something that has to be business driven for us. We're a pretty small company and because of how we do things, we have pretty high resilience rating. But at the same time, if we're down for 4 hours, it's not the end of the world. It's frustrating, but it's explainable. It's not even really going to hurt our income. Too bad to happen because we have people distributed throughout the country, we have our tools distributed through the cloud. There's a very low chance of every single person being fully down inside of our organization because of how we've built things for our clients. But if we have 100 million dollar a year manufacturer downtime is extremely costly for them. They're going to spend a lot more time and money on making sure that their backups are right on the money.

Robbz

Now number four in the list, which we see in only the most needy customers, the customers that want to make sure that every Iota gets saved as it happens and they spend the most money and resources to get backups done. Is mirror backup like full backup? It does take full backup copies, but it's known as synchronization or real time backups. These mirror backups make an exact copy of the selected files of files or folders in real time. Any changes made to the document are immediately reflected in the backup, ensuring that the full copy is up to date again. As Kathy saves that Word doc, it is making a full backup of every change as it happens. This takes an incredible amount of CPU power in the background to get this done and we just don't see it happen that often. There are not a lot of business cases where having absolutely every change backed up in real time is cost effective.

BJ

The way we see that the most is honest. As funny as it sounds, it's a combination of so we don't do mirror backups traditionally. It's just not a good fit for our client base.

Robbz

Very expensive.

BJ

If you're using a program called Veeam, they're calling it CDP. I know it's another acronym, but it's continuous data protection where effectively the backup copy is reaching into the live production data constantly and it's just streaming the data over. Because of how they do it, they make it functional. You still need to have fairly fast storage. Like they make it work really amazingly.

Robbz

And generally to do other place that.

BJ

We do this sorry, what?

Robbz

I apologize. Generally they also do this with independent hardware. Instead of having your server do this and other tasks, you have to have other equipment. You have to have its own engine to get this taken care of. It's that heavy.

BJ

Well with veeam. Specifically, if you're doing the continuous data protection, you're running it inside of what they call a runbook. And that runbook is monitoring the live server for a failover scenario. Like if it doesn't respond within 3 minutes or whatever your predefined timeline is, it can automatically fail over to whatever was active 3 minutes ago. And so you might have two, three, four or 5 minutes of downtime. And in some companies like that, 5 minutes of downtime could be several tens of thousands of dollars. So what's your business need? But then the funny part is, to that same point, Kfm or OneDrive's known folder move keeps data synchronized.

Robbz

Yes, it does.

BJ

So it's basically keeping a live copy of all of our documentation on our machines to the point where I could literally I can work on it on my computer, log into my OneDrive on the browser, and my data is there.

Robbz

Right number five on the list that we have. And this is generally done for virtualization and we'll explain that in a bit. Snapshots snapshots are a captured state of the system at a specific moment, creating a point in time copy of the data, similar to full backup. But these are generally used in virtualized environments, meaning that it will virtualize it to the point of it being booted up. These snapshots, you can flip back and forth and make copies of complete systems. And it's more of a tool that we use for virtualizing servers. So not to make your head explode, we'll have one server, one physical box that's either in a server hosting location or at the customer's physical site. That one server box is specked and large enough to be five virtual servers inside the physical box. So you'll have one for your backup, you'll have one for your domain controller so it can run Windows logins and other security measures. You'll have one for your files, just for your file share. You might have one for your specific application, all running inside this box. So snapshots are how we take the moment in time and can reel back or transfer data between that's effectively all we do.

BJ

So all of our environments are virtualized internally at Etop, both our environments and our client environments because it really means that you're not reliant on the hardware. So if the hardware dies, you can literally restore it to just about anything because you're backing up a file. And so because you're backing up that file at that point in time, it's extremely easy to bring back online. Well, it's simple, it's not necessarily easy. And because that's how we do things, it's part of why we're able to have very good uptime and resilience with our clients.

Robbz

BJ and I definitely had days back in yesteryear, that when we wanted to upgrade a server or replace a server, that it was a planned deal, sometimes took a couple of days to finish everything. Now, because we can virtualize we can make that snapshot backup. We can simply wait until the end of day 05:00 on business day, create that snapshot, move it to the new piece of hardware, and we're relatively done within an hour. In some situations where that was normally all hands on deck, let's make sure stuff is moving. Reinstallation it was a big pain point. In most It technicians lives well, and.

BJ

A server migration still takes time. But we try and build parallel environments and then schedule a cutover. So we'll take a backup of the existing environment, we'll schedule a cutover, do the cutover side by side, and then if something goes wrong, we still have a backup.

Robbz

We go right back to the original image.

BJ

It just makes life so much simpler and so much less downtime. And so we're able to do so many more things just during business hours because of how we build stuff. I don't like working after hours.

Robbz

So now that we've explained the five most common ways of how to backup, how this works traditionally, at least for our customers, we'll explain to you how we handle it is customers have a file I'm going to pick on Kathy on the front desk. She deleted in case folder. What's going to happen? I need that case folder. They call us up and we have the incremental backups running, and we can see that at 08:00, noon, 04:00, 08:00 P.m., those are all taking backups of all of those files. And we can go back in, recover those, and we'll go into a detail towards the end here on the tool to do that. But for the end customer, they're getting four times a day where they're getting actual backups. The problem with that is you have to ask yourself, do you want to spend more money to capture more? Do you think that having between 08:00 A.m. And Noon or Noon and 04:00 P.m. That you're going to lose data in between and you want to capture those changes? Most of our customers taking those four snapshots a day is more than enough. They're comfortable with the loss of 2 hours worth of work. Potentially, if something breaks right, you got to ask yourself, how much money are you willing to pay for the comfort of data? Other businesses, they do three total backups a week because they aren't as reliant on changes. They just need access to the data and don't need to change it. Machining shops are common with this. They don't do a lot of changes. They just need that data to be there when they need to do a new run. Printing companies are the same way. There's different businesses for different needs, whereas a law office probably needs that four times a day. Maybe some even have a use case for more.

BJ

Well, the conversation with backups again, I hate to beat a dead horse, but it's always going to be, how much data can you afford to lose? And how long can you afford to be down? Those are always going to be the metrics that you measure a backup off of in my mind. So if your building gets stepped on by Godzilla, how long can you afford to be down for your phones, for your files, your applications? How long can your users afford to be down? If you can afford two days, great. That changes the conversation. If you can only afford to be down half a day, if you're going to be losing customers because you're not responding to emails after 2 hours as a $20 million a year law firm. Okay, well, that might be a place where you spend a little more time and money on making sure that your backups are up to date than a seven user machine shop, because they just don't have the need. They can drop a file on their machine and let it run for two weeks, and they're going to run the same thing for two weeks. They still need to be up and running, but if their building gets stepped on, they're out of business because all their machines are gone.

Robbz

All right, so just to recap on the different methods, full backup, otherwise known as your starting point, your baseline backup. Then you have incremental backups, capturing the changes along the way, differential backups, which go even deeper into that and grab more mirror backups, which we don't see a ton of, but capture every single change as they happen, and snapshot backups, where we take a snapshot in time and generally use those for virtualization. Then we have the I'm going to change the order on US BJ for the where these backups are saved. We'll start with tape backups. Now, this sounds very rudimetric, but it's still a thing today. Remember the floppy disks that people had back in the day? We no longer use them. Well, they still use magnetic tape drives to store data. This hurts me and pains me in every way, but I still see businesses.

BJ

Doing to this day, tape backups are still an extraordinarily viable situation for high data companies. I hear what you're saying. It's not nearly as sexy as Director Cloud or whatever, but at the end of the day, what do you think Amazon glacier? Is it's tape backup because it's cheap?

Robbz

I hate to say it, but yes, they have found ways of making tape backup cheap. I don't like it. They have some proven metrics. I've been in situations where they've used tape backups and the tapes don't validate. They haven't ran correctly. There's newer, modern ways that they've taken care of it. But how do I explain tape backups and why I hate them? Remember when we used to burn whale oil? You've done history class. I didn't live through it, but they used to burn whale oil.

BJ

We're going way back with whale oil.

Robbz

We're going way back with whale oil. They used to burn whale oil. Whale oils. Was used for our main source of oil in the North America many, many years ago. You wanted to burn a candle. You wanted to have extra oil on the side. Whales is where it came from. It didn't work as good as petrol oils. Right? Petrol oils came out. Everybody started using it. But for some reason these people hang on to whale oil because they figured out a way to make it cheaper. I really don't know. They kept with it. That's my comparison. Tape backups is whale oil. We have petroleum based oils. It does better. But they figured out how to make whale oils so cheap they can't be ignored.

BJ

Sorry, I was not expecting the whale oil analogy.

Robbz

You're welcome.

BJ

And it really threw me off my.

Robbz

Yeah, I'm here for you.

BJ

I'm laughing about that all the evening.

Robbz

It doesn't do the job. It smells awful, but yet you're still using it because it's cheap. That's the whole thing.

BJ

So here's the thing though, is like there's a lot of actual really strong upsides because it's got a really areas that it actually really excels is if you need to write once, right?

Robbz

It probably tastes better.

BJ

It definitely tastes better than, okay, give me your upside.

Robbz

Because you can tell tape versus anything else is a debatable topic.

BJ

And I'm being an advocate when I'm somebody who doesn't use it, right.

Robbz

I hate you.

BJ

But that's fine.

Robbz

It's good as we establish the baseline.

BJ

As long as we're all on the same page here. So tape backups have a really valid place in certain situations. Typically it's going to be more of a compliance based organization where they need to keep tapes are great because you can take a backup and keep it off site. Tapes are great because there are reasons you can do it. Tapes tend to be awful because people don't take them off site. They fail, they're mechanical. But if you have an extremely high data density and you need to keep copies of that off site where you have like a right once media, it is a valid situation. Is it something we do? No, we don't.

Robbz

God bless you.

BJ

But are there good ways to do it? Yes, there are, but it's a pretty specific use case. But I had to be a bit of an advocate there just because there.

Robbz

Is and you're right, they still made it work. Some people put in the engineering Internet's the same way, right? DSL through the phone lines was at one time the best way of getting internet. It beat dial up. Then you moved to cable modem, then you moved to fiber. Right? And we got this lightning fast internet connection. But guess what? There's still people because it's cheap and the lines are already there, developing ways of making that old platform work. I don't like the old platform. I like the new platform for many reasons. But for some reason they've taken that old DSL line and figured out new ways to engineer to make it get close to fiber. They're still holding on. It still technically works, but I don't like it.

BJ

The funny part is that it's actually not even that inexpensive. So it's not that cheap to go to tape anymore. But again, there are very specific use cases where it has a really good place. Is it for most people? No, it really isn't. It's going to be cheap, it's going to be failure prone. And it's going to require more of somebody that's in technology to make it, to manage it now. So it's going to generally be a lot more expensive.

Robbz

If you're listening to this and your company uses a tape drive and you haven't had that service in 20 years, I guarantee you your tape drives are awful. They're probably not validated. They're not doing the job for you. There is modern tape drives where they've taken the time they've validated it. It's under a service contract. They're working. If you have old tape drives, get on something else, it will fail. People have messaged me, is this anything like a VCR tape? Some of them were compared to the old VCR tape style, magnetic tape film. And after 35 years it goes away. Magnetization of the tape fails. That's why VHS right now, they put out a big thing that saying VHS haven't been printed in so many years. You should make copies that are digital, don't have old tape drives, have something current. If you have tape drives completely.

BJ

And I was actually about to walk down that road. And so one of the valid reasons you would do tape, for example, I'm looking at one of the current generation tape cartridges. It'll hold 45 terabytes of compressed data and it'll write it a gigabyte per second.

Robbz

And how much is that?

BJ

$115? Yeah.

Robbz

That's insanity. How cheap that is for the amount that they put on it is insane. And that's why they're still around.

BJ

Well, and so here's the thing. Here's the second part of that. So let's say you're only on 100 megabit. Let's say you're on a smaller internet connection like that DSO, or Starlink or anything under like a 500 megabit connection, which is a very substantial connection. Try transferring 45 terabytes of data inside your building to your data center for.

Robbz

A backup and time it go put on the clock.

BJ

It will take you months at least, at least weeks. You want to know how fast you can take a 45 terabyte drive to a granite mountain? That's not the name of it. They have storage facilities that are made for tape drives. Somebody comes to your office, picks up your tape, your 45 terabyte tape that you wrote to it in 4 hours and they drive it to the office. If you were trying to push that across your internet connection, you'd be at it for weeks.

Robbz

And some of your internet connections that you have out there even business have caps. Once you've reached the data cap, which some in some areas is a half a terabyte, once you've reached that data cap, they're going to throttle your whole business.

BJ

So sometimes the fastest data transfer you can get is literally carrying it to where you need it to go. Well, it's a reason companies like Amazon and Google and Microsoft have these things called Snowballs, where it's like this 100 terabyte portable disk unit that you take to your data center. You copy all the data locally across your local network to this portable unit, and then you ship it via FedEx overnight back to Amazon. Where they pull it back into their data center. Yes, because it's going to be faster to transfer it that way than it is to literally send it across the better Internet. So it goes back to there are still really strong use cases for this. In the right situations, it's not going to be most people, but there are use cases for it.

Robbz

So, number one, tape backup. Number two, disk to disk backup. So you would have some sort of physical box in your office, and you'd have a second disk. Disk A in your server would be the one that you're running. There are all the virtual machines on that we spoke about before. Disk B is strictly just for copies. That is where all of the backup lives, is on a completely separate drive. If you want to even do an external drive, a disk to disk image, that works as well. But this is just letting you know that you can save it not just on tape, but on an actual disk that you would save on your local server.

BJ

At this point, I would like to clarify that. Raid is not backup.

Robbz

It is not.

BJ

Raid is redundancy. Please do not say I have a Raid and expect to have a backup. Because it is not a backup. It's redundancy. You can lose a disk, and oftentimes you can swap a new disk in and keep running.

Robbz

All right. But it is not to recap on that. To explain how disks work. In the past, we have spinning actual mechanisms in a computer that run much like a record player. You have a spinning disk, you have a needle, and it reads the disk back and forth as the disk spins. Anytime that you have something mechanical in your computer, it will fail eventually. It's just a matter of how long. So we have these servers that were expected to run 24 hours a day, seven days a week in a box with a spinning optical drive. They're going to fail. So what they would do is create something called a Raid. And you can set up raids in a lot of different configurations, but essentially, it would be hard drives that would image itself onto the other. So if one, let's say, a Raid, you have four hard drives in there to make a Raid, you would have one of those drives fail. All you do is take a blank drive, plug it into its place like a Nintendo cartridge, and it would begin to write data onto that extra hard drive. It's not a backup, it's just a way that we had to do in the past for mechanical hard drives to not fail and lose all of our data.

BJ

Well, and again, it goes back to its redundancy. We still have a lot of raids in almost every single one of our actually every single one of our servers is going to have some kind of Raid system in it. But we then also run a backup server that we then keep all of the data synchronized to, right? So for us we would probably fall mostly into some combination of this. We don't keep our backup disks in the same server because to me it's too big of a risk. So we run something called a BDR. So. A backup and disaster recovery server.

Robbz

We're going to get to that one.

BJ

We're going to get there.

Robbz

That's on the list.

BJ

Slow me down.

Robbz

That's on the list.

BJ

I won't explain things too soon. All right. All right.

Robbz

So number one, tape backup. Number two, disk to disk in some form. Then number three is a cloud backup. So yes, you have your server. Either it's your server on site or the server in some other location. You can also use cloud services to copy that onto someone else's hardware. You can pay for different services from some company. Veeam is one that's mentioned I've used a lot of different products in the past. Data used to be one of them that I used to use. There's a lot of different cloud backup solutions to take your data and incrementally do all the methods that we talked about before. Full backups, incremental, differential, mirror any type and push it to the cloud, purchase another server and have it live in another place. That is the last one besides hybrid. So when we talk about these solutions, tape disk to disk cloud, we are not saying pick one. We are saying what works for you and what's your expectation. Maybe you want to do them all. So maybe you have disk to disk on your server at your location. Maybe that pushes out to the cloud, makes another copy in case, heaven forbid, your whole building gets hit by a tornado. And even the cloud backup has some sort of tape combination that you pay a guy to get the whole reel and keep it another copy off site or somewhere in between. Maybe you pick two out of the three. However it works for you, the conversation has to be when you go down, how long is it going to take to get you back up and how long do you want to capture between times? That 08:00 p.m.. The noon. What are you looking at for your business purpose and that's where you can go ahead and talk about BDR.

BJ

Oh, goodness. One of the tools that we use is a program called Veeam. Veeam is pretty much magic when it comes to backups, in my opinion. The downside with Veeam is it's an absolute beast to manage and it's crazy flexible. So you can have Veeam running on a server and have it push take local copies so it'll keep your local backup on the BDR. So it's a backup and disaster recovery server where you could boot your client environment or boot your production environment on a failover server. So that's one way. You can have Veeam pushed a tape, you can have Veeam pushed to a cloud provider, you can have Veeam pushed to your Amazon s three bucket. You can have Veeam do really just about anything you want, but you have to know what you want and you have to know where you want it to go. While Veeam is an incredibly powerful tool, it's extremely flexible. It's very much you own the whole like you own everything about it. Which is also why there are times where it makes a lot of sense to go with like a fully all in provider that's going to own they'll make sure that your backups are running. They provide the software, they provide the storage. Personally, we function off of having a provider manage a lot of our backups for us because it's so much work to do it properly, in my opinion. Also why we pay a lot of attention to other providers out there and are constantly reevaluating what our posture looks like so that way we can make changes if something better is out there. In my mind, when it comes to backups, I strongly believe in a three two one methodology where you have your data is in three locations on two different types of media. And I forgot what the one was.

Robbz

Three two, one method states you should have three copies of your data, two different media, and one copy off site for disaster. So you have three different copies at least. So you have your full backup, your incremental backups, and maybe a snapshot. You have two different types of meters. How I would discon tape and one hauled off site just to be safe.

BJ

Well, make sure Matthew, the front office person, is printing out a copy of everything because that is a different media. I'm joking. Don't do that. Yeah. Realistically, if you follow that type of methodology, that's awesome. You might even follow a three two two, where you have two different locations off site, so if somebody gets breached, you have two copies. I personally lean a little bit more that way. I like having more copies off site. It does tend to be more expensive, but it helps us all sleep better at night.

Robbz

All right, so the last part that we have actually, I'm going to bump this so tape disk to disk cloud and the hybrid theory. And we talked about the three, two, one steps. Now our favorite, just a single tool, if you're going to take one tool away from this podcast is using the Microsoft built in previous file version. Can you go in a little detail before we go too much farther on that one?

BJ

I'm glad you asked. So previous versions is one, it's something magical that Microsoft built in. A lot of other tools are starting to do this as well. But if you have a Windows file server, you literally can go into the server and enable Windows previous versions on the different file locations. We set ours up to three to four times a day. Typically previous versions are amazing because it's something that's included inside of Microsoft Windows server. You can go into the server operating system, enable it, and it typically is about a 10% space premium. And it takes these little micro snapshots three, four, five times a day. Depending on what your need is. You can literally teach your team how to right click and restore from the Windows File Explorer. It is truly magical. We use it a lot. That's where most of our data restores come from. And the only time we wouldn't use it oh, sorry, the only time we wouldn't use it is yeah, there isn't.

Robbz

A time we wouldn't use it.

BJ

If it's a disaster scenario, why wouldn't we always use it? We always use it.

Robbz

We always use it.

BJ

It's such a small file penalty for such a strong restorability. Like it makes our life so much easier.

Robbz

So most of our backups that we have, what, go back, what, close to 30 days, sometimes a couple of weeks, depending on the customer's space requirements. And it's as easy as them right clicking the folder, going to properties, and then there's a tab for previous versions. They don't have to call up their It guy, they don't have to do anything. They can see every previous version for the 08:00, noon, 04:00 and 08:00 P.m. And they can restore their own files without ever having to pick up the phone and call us. It's extremely easy and efficient and this takes how much space generally on their server.

BJ

So I figure it takes typically between five and 10% of the total consumed storage. So let's say they have 200 gigs available and they're using 100. Figure it's going to use between five and ten gigs and kind of hidden storage costs. But a five to 10% penalty for extremely easy self service restorability is pennies on the dollar. We're typically running something called deduplication in the background that's saving them so much space on top of that that they will never know the difference. And it just makes their life, it makes everyone so much happier.

Robbz

All right, last two bits, just a recap on what we've done. Tape disk to disk, cloud, hybrid backup. And then we talked about Microsoft's built in previous file versions. I want to take just a moment and tell you that you can do these backups on your own. You don't have to have a tool. You can have your It guy or pay a guy to do this for you, but it's very inefficient. And I've never seen a person that you can pay a normal wage to do this correctly without human error as good as you can pay for that BDR platform. We use veeam. Both BJ and I, the past, have used other platforms. We recommend veeam. We've stood by it. But there are other reputable companies you can look at. There's a chronos data barracuda. VMware. I highly recommend doing homework, researching, talking with your tech expert, seeing if they're using one. That middleman platform takes care of these steps, or at least you should find a vendor that can handle all three of these steps. The three, two, one policy should be handled by a single vendor, in my.

BJ

Opinion, and it should be automatic.

Robbz

You shouldn't have to worry about it.

BJ

And it should be tested now, because if you don't have an automatic backup, nobody's doing it. My favorite is when somebody's like I'm rotating external hard drives around. So what happens is somebody plugs in an external hard drive, nobody's checking to make sure this thing is alive. And so I've seen several situations where people had a backup drive riding around in the back of their car or in somebody's purse or in the safe somewhere. But the last time they had a functional backup was months ago. So in my mind, that has to be automatic and it has to be tested regularly.

Robbz

I had a gentleman that just has just a simple shop. He's a mechanic. All he has is his place where he bills out customers and then keeps the history of the repairs. It's a tool called napatrax. Napa, the auto parts place, gives it free when you sign up for an account and you purchase parts from them. That does not have, at the time of me talking with him, any type of cloud backup, no validating nothing. It was just a free application. Here, you use it and you worry about your own backups. So the guy at Napa, not a technician, told him, hey, here's a flash drive. Plug it in your computer, and every day put a copy to that flash drive. So he would take that flash drive and he would keep it in the safe, and he would do it every day, once a day as he closed the shop, he did not test that flash drive. The Napa Tracks program did not have any sort of self validation. And when his computer crashed and he needed that backup, that flash drive had a bunch of junk data on it that was never tested once, and he almost lost his business because of it. That was his entire bible of his business, where he had everything kept for everyone don't just trust that someone's doing that. Test it and validate it. And these companies, the reputable ones at least, I know Veeam definitely does validations. And I came from data they did at the time. They did validations.

BJ

Let's be clear. You have to set Veeam up to do validations, right?

Robbz

It's a feature that you would ask for. They're capable. They're not going to do it automatically. You have to request these things. Have them do validations where they actually spin up the backup, make sure it works and the data is there a lot of times that takes a human to do. And even if you have a company doing that, I'd sit on the calendar that every 90 days you have a human. Test them. Not just trust the tool. You physically test your own backups in some shape, way, or form. Your backups are only as good as them being validated.

BJ

125,000,000%. There's kind of a running joke in our space that is an untested backup is as good as not having one.

Robbz

Now we only have so much time.

BJ

On your testing schedule.

Robbz

We only have so much time for your listening ears. There's more to get to on backup. Probably have a part two. I know that we definitely have to do another episode on software as a service backup. Right now we're just talking about companies flat files. We'll get that to another time. But the takeaways from this is, what are you backing up? Is it the stuff that you need to do if something goes wrong, how fast is it going to recover? And if you're going to leave with one valid point, please test your backups. Talk to your vendor, talk to your It person, even do it yourself. Test those backups. You'll thank me later.

BJ

It always goes back to how long can you afford to be down and how much can you afford to lose? As long as you keep those things in kind of the forefront of your mind and are realistic with it, you can have a really good conversation with your It provider or your in house person, your in house person, your external person. At the end of the day, they want to take care of you as long as you understand those two questions. The rest of this can be walked through. And it's just a fairly simple technical exercise to make happen.

Robbz

Don't be afraid to walk into that technical meeting with that vendor or your internal person or managed service provider like us and say, if something goes wrong, I want to be up in 2 hours. Okay, here's what that's going to take is they'll come back with then recognize.

BJ

That if you have a two hour requirement, they're probably going to hit you with a much bigger number than if you had like an eight hour requirement.

Robbz

Bingo. Then you'll literally have it on paper. This is the dollars it takes to get that request. What's the business need.

BJ

Well, and I think what we should talk about in one of the future ones, Robbie, is business continuity and disaster recovery, because this whole backup conversation flows into those two pieces.

Robbz

Backup part one.

BJ

Oh, man. This could be like, 17 parts. People are going to be like, oh, my God, they're talking about backups again.

Robbz

Oh, no.

BJ

We promise we'll come up with other new and interesting things to talk about over the next couple of weeks. We're definitely going to feed into kind of that whole business continuity, disaster recovery because it's such a big part of your business risk. Mitigation.

Robbz

You got questions? Go to businesstechplaybook.com. Go to the bottom of the website. You'll find discord in our show notes. You can find information to contact us. If you got questions or you are experienced concerns with your own backup policy, we're happy to answer them. Thanks, BJ. And I do want to tell people that no whales were harmed in the making of this podcast.

BJ

No whales were arm to the making of this podcast. Oh, man. Seriously? Well, have a great one, and we'll talk to you soon.

Robbz

I hate tape drives.

Episode Notes

For more episodes got to http://businesstechplaybook.com

Find more on LinkedIn: https://www.linkedin.com/in/william-pote-75a87233

This podcast is provided by the team at Etop Technology: https://etoptechnology.com/

Special thanks to Giga for the intro/outro sounds: https://soundcloud.com/gigamusicofficial