Backups and Utilizing Them

Welcome to Building Better Developers, the Developer podcast, where we work on getting better step by step professionally and personally. Let's get started. Hello and welcome. We are continuing our season when we're looking at mistakes made, errors, things like that, and how they may have been sometimes devastating at the time, but actually turned out to be great steps towards success or lessons learned that were very valuable down the road. Last episode we talked about making changes that were sweeping when you didn't really intend to do so. Mention the idea of backups. In this episode, we're going to talk about backups and utilizing those because they don't always back up properly. There are some things that can go on with that. I've had two examples in my life and I'm probably going to go ahead and roll both of them together into this one because although they're similar, there's some differences as well. The first one goes back to college. We had a, we could have gone in and used the school network and stuff like that and tried to build everything on their systems, but it was easier for us to do some of that, but also to utilize a couple of desktops. One in particular, because my roommate was a partner on the team, we had a desktop in our room. It was easy for both of us to have access to it. We rarely had to work on it at the same time, so it was perfect. We're just like, you know what? We can just crank through some stuff on that. We did so. We got fairly far into this thing and we got to a demo day basically. At this point, it was basically, we were getting close to the end for our thing and so we needed to go, needed a computer to go show stuff off. We could have used one again at the school. We had some issues getting data moved from one to the other. This is back in the world of floppy disks. We would copy stuff onto a disk and it just didn't, the other system wasn't reading it right and so we just sort of finally said, forget it. We'll give up. We'll just bring the desktop in and use that for our demo. We did and it worked fine. That should have been a clue that maybe there were some issues because at this point, we had everything on one machine or for the most part had everything on one machine and we did have some printouts and stuff like that and we also had backups. We had stuff that was copied off fairly regularly onto floppy disks, but we didn't really check those floppy disks. We were just copying off to those and saying, okay, cool. We backed it up. We're good. Flash forward not much longer. I think it was a few weeks and the desktop had some major issues. It just stopped. It started getting really flaky and stopped basically working the way we expected it to work. And so we take it in and find out that the system was slowly degrading in a way that the way it wrote data was the idea of like, if you think about something that's like slowing down, then its standard speed is now at a lower pace and so now it becomes a little harder and harder for things to understand and for them to translate. For example, if you take like this recording and you start to slow it down and you knock a percent off of the speed every day or increase the speed 1% every day, it's going to be okay for a while, but there's going to be some point where it gets so slow or so fast that it's unintelligible. That's basically what happened from how they describe it to me with the system that we were using. So what happened is we had a drive that was writing this thing out to these floppy disks and it could recognize them because it was at the same speed. But even then, from one day to the next, and maybe it wasn't a day, but a few days later, the speed was changed to a point where it couldn't read its own backups. So rule number one, test your backups on a regular basis because if you don't, they may not be backing up to the level that you think they are. And this will be, we'll see this again, I saw this many, many years later in a similar situation. We lost the ability to move our code off of that computer. And I forget all the details about it, but it was basically one of these things where, yeah, we had the code, but the hard drive was pretty much dying in the way it was. It was dying in sort of the same fashion. So we were not able to recover the data from the hard drive. The saving grace, I guess, of this was we had printouts of code. And this is tens of thousands of lines of code. But we had printouts, so we could go back and basically just read the printout and retype it into a new computer, which we got everything changed over, went to a new computer, retyped it. It took us months, basically. Not a lot, probably about two months, I think it was, to get back to where we were. Now we did have a bonus that's an upside, is there were several design decisions that we had made that we were not thrilled with at this point. By the time what the code was at the end. So we actually took this opportunity to re-architect, redesign several of the pieces. And so what we had at the end was actually pretty solid. It was, it ran a lot faster, a lot less memory, all kinds of other things that made it a better application. So there was a silver lining to this dark cloud. Now the other thing is, is that I learned fairly quickly there that you back up, you back up to multiple places. It's not just, don't trust just one backup. Back up to multiple areas. Like for example, these days, back up to the cloud, but also occasionally back it off to a different cloud provider or go to like, if you actually have a something that's a CD burner or something like that, or if you have a thumb drive or some sort of external driver or something, back up to a couple of different locations and test your backups on a regular basis. Because if you don't, you may think you're covered when you're not. Which leads me to the more recent situation is I use, I'm pretty much a Mac, Apple type person and their time machine backup stuff is awesome. It is, it has saved me many, many, many times. I will probably mention again in some future episodes. And the way you do it is you basically hook your machine up to, you know, you connect to a drive, whether it's wireless or you plug it in or whatever. And that back, it will, the software will in the background regularly back your your changes up to that drive. And there is a way that you can go in what they call time machine and you can actually go back through these various backup times and go find files that were from that given time and pull them forward. Now it is not a perfect solution. Recently, what happened is I had a machine that was starting to have a few issues and I didn't realize it until it basically just decided to reboot itself and then couldn't come back up one day. And one of the things that I did not realize is that this background backup process had stopped approximately three or four months before the last successful backup was from January. And then this thing happened in, I don't know, March, April, something like that, maybe May. So there was a solid period of time where there were changes going on, but there were no backups being taken. So when I grabbed everything, did what I could to pull stuff off of it, and it ended up being something that they thought the hard drive, actually the initial assessment was the main hard drive was dead, was basically there were some it was corrupted sectors. You were going to have to replace the entire hard drive. That ended up not being the case. There was some more like driver controller type thing that was having issues that that was able to replace. So had I not gone through some of the processes I did, I probably would have still had my hard drive and it would have been a non-issue. But I jumped ahead a little bit, tried to troubleshoot it, walked through the examples, they said, I said, all right, you got to do this, this and this. I said, OK, let's get it done. And what ended up happening is we ended up having to basically redo that, that hard drive. And they did end up through all of it, I think, replacing the hard hard drive anyway. So it's like, all right, at least I've got now a fresh SSD drive. The bad news is I had for that machine, my most recent backup was months old. Now, this goes back to the lesson I learned earlier is that you do back up stuff in multiple places. So it was not a complete loss. While I had that backup was basically dead, was was old. And I also had cloud services like I use Dropbox. I had all my code sits out on other machines that get repositories. And there's a couple of other shared drives and things like that that I had. So including I have a second machine that had more or less current, although it was a laptop that I hadn't used as frequently, it was more or less up to date. So. The loss was not complete and it was still was not perfect. It would have been great to have been able to pull those files from that day or something like that, as opposed to months behind. But because I had multiple backups, multiple locations and used those on a regular basis, I was pretty confident I was able to not exactly do it without skipping a beat, but was able to get back up to speed and up to date fairly quickly. The biggest problem actually was more around just moving to back and forth to machines and doing just a normal kind of stuff you need to do when you're installing software or updating stuff, things of that nature. So my far from decades back mistake in the lessons learned there did protect me from a much more, it probably would have been a much more painful loss this time around. Now with that, the other thing is the, I had used time machine before to pull files here and there. I had not ever tried to use time machine to move data from one, essentially one machine to another. In this case, it was one hard drive to another, but since it was a reinstalled operating system because of how everything was set up, it was the same machine, but as far as the backup was concerned, it was two different machines. That was a bit of a challenge. And it's, I think something that we need to consider as well. What I ran into is I had space issues basically, as I was moving from one machine to another. And the way I, you know, you had to, I had to pull stuff off of the backup onto a drive and then do some manipulation of that to get it in the right place. So I essentially had to have a good deal more space available than I had on my backup drive. That is something I think we run into depending on how you set up your backup space and systems. If you're on the cloud, you're probably okay because the cloud can just grow and grow and grow and grow. But otherwise, like if you're using a thumb drive or something like that, if you've got a thumb drive where you're at, I don't know, 90% of its capacity, particularly if you have an external drive, which I think a lot of people do these days where, you know, maybe your main machine is, I don't know, one, two terabyte drive and you've got a six terabyte backup drive. If that backup drive is basically full, then you're going to have a hard time pulling all that information over to a machine to work with it. And you may not have to, but in this case, because of the way it was set up, I had a lot of data sitting on that drive that I basically had to pull off of. I had to copy it off of that drive in order to work with it and do the restoration, the restoring that I needed to. Not to mention the fact that there's a lot of data to move across the, you know, the wireless, wirelessly move that data across to start rebuilding it on my machine. So it does go back to regularly where you can do verification of your backups. A great way to do it if it depends on what your machine cycle is, as far as like how often do you get a new machine or move to a new machine? Or I used to do it when I was on windows all the time. I would regularly have backups and because the machine would just get loaded with crap and slow down, I would just format the drive, reinstall the operating system and start all over again and then pull all my backup data off. And it, it always helped. It was like, it would move faster for a while. I'd have more free space, all that kind of good stuff. Now, since I've moved, I don't, you know, since I've changed to different machines, my, I don't do that, you know, probably annually, which is what I used to do. Now I'm more on a multi-year cycle. So there can be a lot of stuff that builds up. And so what you want to do is maybe check it on a regular basis. If you've got a desktop and a laptop, that's a great way to do it is just periodically verify that whatever you do on one machine, you can do on the other. Or if you don't, if you don't want to move that stuff over, for example, I have some business accounting stuff that I have only on really one machine. I don't want to move it somewhere else because it's going to make, you know, cause issues and synchronization problems, stuff like that. But I do periodically, like if I'm going to travel or something, I'll copy stuff over and move from like maybe my desktop being my primary to my laptop being my primary for awhile. And that's actually, if you're using laptops, that's actually a great use of them is switch to that and do it where the laptop is not plugged in. So you can exercise the battery, keep it useful, all that kind of stuff. If you have a machine that's just sitting, then it's, it can cause issues. You may need it. And then it's not working at that point. So don't be afraid to do that. The move from one machine to another. If you have a work versus a home machine, you may do some work there. If you've got a, an external drive that you use to move stuff back and forth, doesn't hurt to back up that external drive to have a second one. Um, although with that, you're probably going to be, you know, you should back that stuff up, but also with that, you should be able to periodically plug that into other machines and verify that, yep, other guys, you know, other machines can read it, can write to it and do all the things that they need to do run into that on a regular basis, where maybe you've got, uh, like a windows machine where you've got permissions set up a certain way, you store stuff on a drive. You go to another machine and it doesn't, because the way permissions are set up, it's not able to read that data properly. So there are a lot of little things that can happen. That's why when we talk about disaster recovery, there is also, it's not just the idea of, of backing stuff up and having backup systems. It's also testing your DR plan is going through and like in data centers, they'll build out a disaster recovery plan and they'll just go throw the power, just cut the power off to everything and test the plan and say, okay, does this work? You take as close as you can to a, a complete disaster and run through a cycle. Now, depending on what you do, that may or may not feel like something that matters that much. And if you have your stuff backed up eight different ways and you're constantly moving to different machines and you're particularly these days, like you may be accessing it from your phone or your tablet, and then also you've got a desktop or a laptop, maybe at work, maybe at home. Then that may build that sort of testing into your backup and data distribution process by itself without having to do anything extra. But if you don't, or if you found yourself getting in a rut where you were only working on one machine for a while, then maybe it's time to back up and say, I need to like change gear a little bit. I need to change maybe like just how I, you know, how I work, change my point of view where I'm at, what office or room that I'm in. And particularly these days with the remote stuff, there may be things that you're working remotely and that you have files on your home machines and they're not getting backed up properly to the work network and things that normally would be covered because your, your IT group at your business takes care of all that stuff. If you're not connecting right, you may not get all of it. You may not get all of it. Or if you're not logging in right, if you've got like a VPN connection and you don't always connect through the VPN, you know, you're doing your work without being connected, then you may be missing some stuff. You may be missing updates as well. And that's its own little issue when you have software that, you know, there may be some things that you need to be updating or suddenly the files that you're creating are not compatible with the versions other people are running on. All kinds of little gotchas in the world of disaster recovery. So, you know, bottom line here goes back to number one, back stuff up, back it up to multiple locations and test whatever your personal disaster recovery plan is. Test that on a somewhat regular basis. I would say it should not be too hard to do that annually. Depending on how valuable and how dynamic the files are that you're working with, you may want to do it as much as quarterly because it goes back to how much. What happens if you lose data from now to the last time that you did a backup, to the last time that you tested and verified a backup, because that would be probably your, your worst case scenario. And you want to make sure that that's something that's something you can, you can survive that you can put up with, that you can bear that pain versus something where you say, Oh no, if I had lost the last six months of stuff that I've done, because that's the last time I checked my backups, that would be devastating. Make sure you don't get into that situation after the fact. If you look at it and you say, Ooh, that would be bad if I lost my machine today. Then go make sure your backups to validate your backups today or as soon as possible. Trust me, I've been through the pain and it's always self-inflicted. So that's always the worst kind. That being said, hopefully this is not inflicting pain on you to have gone through this, even though it was a little longer than a couple of the others. But I think it's one of those that it's like, Hey, everybody makes mistakes and let's just plan for those and have some good backups and a better strategy to handle them. And we are not done going through all of these. So we're going to continue as the season moves on, got all sorts of mistakes and errors and missteps to cover and lessons learned from them. But we'll wrap this one up and so go out there and have yourself a great day, a great week, and we will talk to you. Thank you for listening to building better developers, the developer nor podcast. You can subscribe on Apple podcasts, Stitcher, Amazon, anywhere that you can find podcasts. We are there. And remember just a little bit of effort every day ends up adding into great momentum and great success. Hi, this is Rob from building better developers, the developer nor podcast. We're excited to be on Alexa. Now you can enable us by simply saying Alexa enable building better developers and we will be there ready for you. Every time you want to listen to your now favorite podcast, whether we are your favorite podcast or not, we would love to hear from you. So please leave a review on Amazon. Now back to the show.