When Coffee Hits the Fan - Developer Disaster Recovery

Welcome to Building Better Developers, the Developer podcast, where we work on getting better step by step, professionally and personally. Let's get started. Hello and welcome back. We are continuing our season when we're talking about the developer journey. And today we're going to talk about when the coffee hits the fan, basically, when you're in an environment and it is not a very environment that's conducive to getting work done. Recently there was a company that we're not going to talk about, but you may have heard about it on the news, depending on how far back this is that you're in the future, you're listening to this. They had some issues and it caused a lot of updates to not update properly and a lot of machines and a lot of systems to be unfunctional for a while. I'll even say non-functional because unfunctional is not really a word. Before we get into this one though, I want to introduce myself. My name is Rob Broadhead. I am one of the founders of Develop-a-Nor, Building Better Developers, also a founder of RB Consulting, where we tackle technology fatigue and sprawl. And we find ways through simplification, automation and integration to take your big nasty ugly things of technology and streamline them and get it down to something that works well, and is easy to maintain, upgrade, scale, all that good stuff. Somebody that else that is pretty much one of a kind, so he's not going to scale very well and you're not going to get too many upgrades, at least not until they go into the million dollar man, all that kind of stuff. Michael on the other side, go ahead and introduce yourself. Hey everyone, my name is Michael Milosz. I'm one of the co-founders of Develop-a-Nor and I'm a founder of Envision QA, where we help small and mid-sized companies look at their software stack, see what technologies they have and help them either integrate better systems or build something that suits more to their needs. So let's talk about, I think we've talked before about being like a road warrior kind of thing as a developer and like some of the things where you have, it's useful to have like a lot of different plugs and a lot of different, you know, ways to integrate your system to whatever it is you're on, to make sure that your phone can plug in, to make sure that your laptop can plug in. But these days, most people have a laptop. So that allows you to be, you don't have to actually be plugged into power. I mean, you do at some point, but you know, you can go for a while without power. And sometimes you can go without, you know, most people need the internet, but the bonus is and one of the things to prepare yourself, sort of your own little disaster recovery is have like an external drive where you've got some of your, you know, your primary source code that you're working on right now, if that's feasible or, or make sure that you've brought it local. Even if you're doing, if you're using like GitHub or one of those or Git or some distributed version control system like that, as long as you've brought stuff local, you can continue to make branches. You can commit, you can do all that stuff. And then eventually you can sync back and get all of that stuff to like, you know, line back up. So you don't have, you can fake it. You can also bring, you know, through things we've talked about, like, you know, Docker and Kubernetes and stuff like that. You can have an entire system spun up, you know, virtual servers and all this kind of stuff and it, it can take time. It can take an investment to make sure that you have that environment available, but when everything goes wrong, it is useful. I will, before I pass it on to Michael and get your feedback and thoughts, I will share one little story that I have that just sort of like, as we were thinking about and discussing the topic for this is this is many years ago. It's probably now, shoot, I don't even know. I guess it's probably 15 years ago. It's been a while, been a day or two, as they would say. And we were, we were working, this was back when everybody basically, you know, nobody worked remote. Everybody worked in an office and we had a nice big office and we're in a big office building and we had deadlines and we were cranking through stuff and the power went out. Whole building, power went out, lost all the internet. Now we had laptops and they were MacBooks, a little Apple ad. They were basically for you. So they even then had serious battery power. So you could, you could code and be a developer for hours on those suckers before they ran out of power. The problem was we had to connect to the internet. This is where it's useful to have a device. Especially now it's your phone that is a hotspot. And we literally are, because our manager was a big iPhone guy and this is when iPhones had just come out. They'd just gotten to the point where you could actually, I think it was the first version where it was ability to have a hotspot. Your iPhone could be its own hotspot. And he walked out of his office, set his phone on top of a shelf there and then said everybody, this is the wireless, here's the password. And there's like, I don't know, five, 10 of us that we're working on using his phone to connect out to the internet. It was like, you know, 0.0 G or whatever. I don't remember what it was not 4G or 5G. I can't remember what it was. It was not terribly fast, but it got the job done for as far as like, you know, having to deal with emails and do it. You know, you're not going to download, you know, gigabytes and gigabytes of data, but you probably don't need that. And if you do, you should plan a little bit better so you don't have to in those kinds of situations is think through like how self-sufficient is your system. If you get in a situation where you really need to get something done, but you don't have your normal work environment. If you're a consultant, if you're a side hustler, this is the kind of thing that can come up fairly often. And it does for me where you're out, you're not anywhere near your office. You're living life and somebody calls with a critical issue and you've got to find a way to help them out. I mean, you could, you can always drive back home, but it may take eight hours and they're maybe down all that time, particularly, you know, if you're working 24 seven support, things like that, there's all kinds of things that come up. So, you know, look at like your phone. You may be able to do a lot from that these days because there's, you know, especially if it's a smartphone, you may have a tablet, you know, like an iPad or something like that. You may have like a Chromebook or your regular laptop that you can just throw in a backpack that you carry with you, which yes, I'm a geek. I do that way too often. People know me as a guy that's like, I'll sit down at the piano bar and crack out the laptop every so often and be like, all right, I got work to do. Those are some of the things to think about. And it's as always, you want to think about those before the disaster happens as opposed to when it happens. And on that, no, no relation to disasters happening. I will pass it over to you and get your thoughts on it, Michael. Thanks, Rob. Yeah. So it's free. Your story and how you're talking about the disaster recovery and especially we don't want to name names, but you know, the screen of death kind of thing. But as developers, we run into other situations. It's not even just the internet goes down or power goes down. We run into similar issues as to what happened in the news. We could take an update to our IDE. Oops, our IDE is now broken. We could take an update to the operating system, the operating system shot. So building a model or setting up your environment in a way that makes you as basically environment agnostic as possible is key, at least to me. Over the years and why I say that is over the years, I've gone through different environments, be it Windows, Linux, Unix, Mac, whatever. All of them have the same problems. A system update gets pushed down. Things get shot. AS 400, a wrong patch gets installed. You're bricked. You now have to restore everything and you're down for hours. As developers, we are susceptible to little things like that. We could go from say Java 2 to Java 8 and now the code suddenly doesn't work. We have lots of little things that can break our environment. So we have to be very careful and very mindful of what it is that we do to our machines that could impact our work and not just the work, but also our customers. Because when we push that software out, we want to make sure that the customer is not impacted by something that we did wrong. So the story I'm going to tell is back at a company we were at, but before you came on, when I first came on board, we had these very ancient, archaic machines that barely had four gigs of RAM. And we were trying to write code on this. Very minimal system requirements. We kept writing out disk space. So this was before Docker and really before VMs to an extent. VMware had just started out and our network guy had been using Citrix. So he started spinning up some, playing around with VM and started spinning up some VM environments. And what I essentially did was I created a base environment image. And that was the only image that was allowed to go out to any of the developers. So all we had to do was we basically just dropped our image, reload the image, and it pointed to a virtual drive that had all their files in a separate location. So they never lost their files. We would always once a quarter review any system updates, security patches, whatever. We did it in one silo, made sure it worked, and then we pushed it out to everyone. We never went down. Prior to that, we had hard drives going out. We had monitors going out. We had more hardware issues than you could throw a skunk at. It was crazy. Now in today's world, we have Docker, we have containers, we have things that we can set up and now all you need is a virtual disk space or a virtual environment. You can do it online. You could do it on your computers. You can essentially do it anywhere. And then you essentially, again, treat your environment as a silo. And if you need to change something, test it there. If you break it there, you're not impacting your development. So that's just one approach I would take to saving yourself the headaches of potential disaster from updates. First off, I have to start with the visual image of throwing a skunk at your problem. I was just like, got to watch out. Pepe Le Pew may get a little bad when you do that. That is actually a perfect example of the kind of disaster recovery environments that we would love to have. This is something that now it's much more feasible to do in the world of cloud development and all that kind of stuff. It is very easy to set up. And I have done this. I actually did this for one of the seasons. I can't remember how far back we did. I went through the entire thing. I did an entire project and I used Amazon's Cloud9. I think it was even before Amazon bought Cloud9, but it was basically working on a completely virtual ID. So I just logged in. It didn't matter where I was, what machine I was on. I logged in. It logged me into my machine that was out there. Yes, I needed to have internet connectivity. But other than that, all my code was there. All of the everything I'd done, it was everything I needed. My entire environment was there. It was very similar to what Michael just described, although the one he described was a bit more powerful because of Citrix and it was closer and some things like that. But the nice thing about that was, when I came in, I used that environment all the time and I almost never had to touch my actual laptop. Didn't care what was on it. Didn't care anything about it. As long as I could get to the browser for that, the Citrix window, I could go open up the session. And yes, we did have problems as we grew a little bit because people would step on each other's sessions. We had to have the right sleep. You had about six or seven development environments we had set up. But then you knew where to go. If I'm focused on this, I'm going to go to this machine. If I'm focused on that, I'm going to go to that machine. If I need to go fix the database, I got to go to this one. Those kinds of things allow you to be completely free. Now you don't have to worry about, oh crap, I left that on my machine. So utilize the cloud for that kind of stuff. Don't be afraid to use things like definitely GitHub and places like that for your version control so you can access your source code from anywhere. Also I make heavy use of Dropbox and have for a long time and there are other tools like that that are commercially available. If you want to do your own personal cloud, there are plenty of options for that where you basically go get a multi-terabyte drive and then you can access it from anywhere. As long as you want to take care of this security and things like that. These things will be, they are more than worth their weight in gold when you need them. And if you get yourself using them all the time, you will find more and more, for example, that you don't need a high-end laptop when you go upgrade your development laptop or your development machine. You just need a way to get to that environment. Maybe you just need a browser. So now suddenly, maybe it is like a little Chromebook, works fine for you because you really don't, that thing doesn't matter. All that matters is you have a decent display, a decent keyboard, and you can browse to wherever your cloud environment is. And this is becoming more and more common, particularly if you're in the world of like Salesforce and NetSuite and HubSpot and all these things that are, basically you can live your professional life on the site or in this, you know, software as a service solution. And you don't have to worry about it being on your machine and sharing stuff out and all the things that can be involved in that. Closing thoughts? Yeah, in addition to what we've talked about up to this point, the other thing that this type of approach is good for is if you are looking at taking on new customers or you have a customer that needs some software written and you're not sure if either your software is going to work with theirs, this is where you can build those virtual environments locally, test them and make sure they work before you actually impact the production environment. If you need more RAM, you know ahead of time, go buy the RAM, put it in the cost. Don't just push your software out and then have 1000 users turn around and say, hey, only 10 people can get in because we don't have enough switches. These are things that you can test, you should be testing and the beauty of it is, it is not that expensive to build your own virtual, virtualization environment at home or on the web. It is also not that expensive to give us some feedback, to get something in there, throw some comments out there. Give us a little love, a little like, hey, we're spending some time here, we're coming into your, I know it is a little bit intrusive coming into your ear holes as we are, whether you're audio or your eye holes if you're watching through YouTube, but we love that feedback. We'd love to hear from you, we'd love to hear where would you like us to go next as we're talking through our developer journey? And yes, we are not done. We're only about roughly halfway maybe through the season and we've got plenty of topics ahead. Even we don't know what those topics are yet though, because we're sort of like we're journeying as we go through some of these things. And that's worked out pretty good, we think so far, hopefully you agree, but obviously your feedback will help and we can either use your name or we can keep you anonymous if you want to, you know, if you don't want to have anything to do with us, we get that. However, come on back, hang out with us for a little bit and see where we go with this. Feedback is always welcome. And as always, go out there and have yourself a great day, a great week, and we will talk to you next time. Thank you for listening to Building Better Developers, the Develop-a-Nor podcast. You can subscribe on Apple Podcasts, Stitcher, Amazon, anywhere that you can find podcasts. We are there. And remember, just a little bit of effort every day ends up adding into great momentum and great success.