Detailed Notes
Welcome back to another episode of Building Better Developers, where we dive into essential aspects of software development. Today, we’re delving into “Managing Hardware Resource Issues in Software Development: Best Practices and Real-World Examples.” In this episode, we’ll explore the often-overlooked yet critical area of hardware resource challenges and their impact on software performance. From memory and storage constraints to processing limitations, we’ll navigate through common issues developers face and provide insights into effective debugging strategies and optimization techniques. Let’s embark on this journey to understand and address hardware resource issues in software development.
Read more: https://develpreneur.com/managing-hardware-resource-issues-in-software-development
Stay Connected: Join the Developreneur Community We invite you to join our community and share your coding journey with us. Whether you’re a seasoned developer or just starting, there’s always room to learn and grow together. Contact us at [email protected] with your questions, feedback, or suggestions for future episodes. Together, let’s continue exploring the exciting world of software development.
Additional Resources * Troubleshooting Tips for Common EC2 Problems (https://develpreneur.com/troubleshooting-tips-common-ec2-problems/) * Testing Challenges – Teach More Than Quality (https://develpreneur.com/testing-challenges-teach-more-than-quality/) * Software Architecture Patterns and Anti-Patterns Overview (https://develpreneur.com/software-architecture-patterns-and-anti-patterns-overview/)
Transcript Text
[Music] all right uh let's talk about what do we want to do for this episode you have a I was like we just came up out of nowhere last time so did you have have you had any thoughts today of or this week of like what would be a good episode topic well the kind of to go with the example you were giving previously before we recorded last time uh have you run into situations where not only is your code your issue but the hardware is the issue like you're not actually configuring the hardware for your application oh gez I've been like that was yesterday I was that was half my day it wasn't it was it's frustrating so it's I've put together um Apache front ends to Jango now a couple of times and it's it's always a little bit different because it like if you do a dbn versus um and I forget what the core like but yeah if you use like aptg versus yum versus Mac and I haven't even tried Windows nor do I you know I did like decades back and don't want to ever touch it again they're all different they're all like close but not quite and so you have to like go find the right library and the right download and the right all this stuff and I was working on a machine that I thought was one I got and realized oh wait it's different but it's like it's a whole thing that I'm setting up because I'm setting up Apache which I got that you know that was fine but I'm setting up Apache I'm setting up SSH or you know SSL stuff I'm using um certbot so I'm grabbing aert and I'm trying to make sure that I've got like that hooked up so that it grabs the right certific certificate and then making sure I've got the files in the right place that it handles you know Port you know HTTP as well as https and it's connecting to everything and that it's all routing right and it's just a bunch of stuff that's like very sensitive got to be the right got to be the exact right name names have got to match right and everything and I wasn't able to do it in one sitting I got something done got pulled away got something else done pulled away and so it just took forever because every time I sat down I had to one had to remember where I was at and then two had to like go track this down and there's you know three or four different log files I'm looking for so short story long yes I have definitely run into that couple of times configuration is if I could never if I could do software and never configure anything ever again in my life even my own software I think I would be happy because I I get bit by my own configuration settings on a regular basis where it's like I built it like an app I was testing today I was like why is that not working it's like oh yeah we've got this configuration setting and we didn't set it right it's like we adjusted it all right let's set it back and then boom everything works again and seeing on the flip side of that with the testing stuff when you want to do like low testing or high-end regression testing you need to run your test multi-threaded well in order to do that you need a certain amount of CPU and you need a certain amount of ram in order to be able to keep up with that load it's not necessarily always about memory leaks or things like that if you are trying to push high volume you need to have the resources to push it and that was one of these things I was running down today is we tested all of our production code in QA at high volume for generating reports and I had it all working everything worked for every use case we had now it's in production and what I didn't know is that we've been kicking up the memory in production uh we're up to like 16 gigs and we're still running out of memory with our largest uh with one of our largest vendors that has over 2 million records come to find out we're only running for uh four core processor with 16 gigs of RAM so I actually uh one of the cool things about spring boot uh and Eclipse you can actually configure your profiler for the specific cores and memory you want to run so I've got a 12 core laptop with 64 gigs of RAM so I can actually do pretty much whatever I want locally without even needing to do containers or I could stand up a container whatever but what essentially I ran into is I clicked run locally I mean I'm running all of our applications locally we all microservices distributed but I have it all locally clicked run ran fine use case worked report guide generated no problem I'm like all right what the hell's wrong and then I started to getting into it I'm like well let me go look at production finally got uh back into production because I found out AWS for some reason defaulted to a different region for me so while I'm able to log in I had not like I had no servers I had no I'm like where the hell is all my stuff that was a fun hour uh I have lost that hour myself a couple of times anyway once I got in there though I walked into the deployment and looked at the setup and found out okay we've got four cores and 16 Ram so I went into to jvm and I configured it to 4 gigs of RAM and it was cool cuz I found out with the jvm you could um with any Clips you can do like available chords or available system I I forget the exact name it's not the same as you would for Maven it's a different uh property uh VM but you can actually set the ches so then I used uh visual VM uh yeah visual VM to uh Prof the memory cuz we used that before in the past it's a cool little free profiler and I spun it up I watched things go and before I had this nice flat line um with my uh 12 cores with all my memory it barely got above uh like four gigs of RAM under four cores and 16 gigs of RAM or I did I did four cores and I did four gigs of RAM to start with so I did four for and immediately I watched the spike go from this to this and within a like 30 seconds I'm out of memory so I'm like okay that was interesting so then I flipped it back to 12 core still ran out of memory but I was able to get it up to um six gigs of RAM and it worked so I slowly then started rolling back the processors and essentially I got it down to eight and eight so if you have eight cores with 8 gigs of RAM it worked now my boss is like well we need to go look at the code we have a memory leag I don't think so I think trying to process two million records in memory at once because we're sorting them we're doing manipulations with that's a lot of processing power it's not just memory we're trying to do stuff with that if we were just holding it in memory sure I would say we have a memory leak but no we're constantly like resorting the list for um building uh spreadsheets with this I don't think it's necessarily A Memory leak as it's a processor Le we need more processing power anyway this was kind of this discussion I thought would be good for the podcast which we're now like way I think so I think we can well we're gonna yeah I think we can go into some of this stuff is just some like common odity type things uh that we've that we've run into and uh I've got some similar types of experiences that I can talk about a little bit and sort of like what uh some of it's like thresholds and stuff like that and um sort of tackling those a little bit and then I'll toss it over to you and then you can you can probably rehash some of that so those you guys that are listening sorry you may hear this twice but we don't remember what we said so it'll be a little bit different anyway so there may be something more ingenious on that second side around well hello and welcome back we are I'm sorry I didn't mean to smack too close to the mic but that happens sometimes uh we are back with building better developers yet another episode where we're talking about stuff that we've run into uh this episode we're going to go a little bit uh some of it's very current for Michael some of it's not so much for me but some of the things that I've hit in the past and it's really it gets into like debugging stuff when you get away from it's like not just code it's hardware issues and and it literally is like memory and resource issues and things like that which you may not run into as much depending on what your your applications are because these days there's a lot of stuff that is very much plays well within the the Norms but if you start doing big data if you start doing data processing if you've got a an app that's got a lot of users or a lot of processing a lot of data any of that a lot of stuff going on things can happen so we're going to talk about that first I'm going to introduce myself my name is Rob briad I'm one of the founders of developer nor I'm also a founder of RB Consulting all things about simplifying integrating and and just automating your software your solutions to make your business run smoother on the other side this time I'm not even going to mention his name I'm going to let them do the whole introduction so go for it sir hey everyone my name is Michael MOS I'm another co-founder of developer erer I'm also the founder of Envision QA where we build custom software for small to midsize businesses and and Healthcare clinicians we also uh offered consulting services and testing services for your needs awesome now we're get right into one of those cases where you may need some of those testing services in particularly it is with like the the big Behemoth of stuff that we can sometimes run into in the software world is the hardware stuff the resources that are related to like processing and the ones related to well and storage and memory now the fun ones are when storage and memory cross lines and this is where you see some really neat stuff and these are the kinds of things that it's just we're going to give you some like they're not I don't know if they're necessarily red flags but there's some some things that are maybe some notes that if you are in a testing situation or in a situation where hey it works on my machine or usually and now it doesn't and the code should be the same when you get a sense like that doesn't make sense it's it's not a different system here's some things to go look at maybe there's something going on that's a little deeper than that where you're going to want to have some sort of you know some more detailed logging particularly from the the operating system side the platform side to see what is actually going on now one of the like the mother of all break your application things that happens is when you trigger when you when you get past physical memory and it starts to cash stuff because what happens is now you've got physical memory that's just like really it's like boom boom boom boom boom boom boom no it's gotten better because SSD drives and some of that stuff is very fast and and there's a lot of intelligence buil around those things often so that you can so it's going to keep the uh the most recent data like up front so if you have something that if you're caching your memory the system memory into like a SSD drive or really high-speed drive you may not notice the difference as much but definitely if you're old school and you go to one of the old you know uh dis driven type hard drives this the processing speed goes from microc seconds to multisecond and it literally I don't know how many times it was stuff where it's like you know I've just pick a number like we have let's say we have two gig of of memory on our machine and everything was awesome until we had it grow Beyond two gig because as soon as it did that it had to C it had to swap stuff out and now you have the swap cost so now it's it's pushing stuff out to that hard drive and then having to pull stuff back and that's adding some processing time and then it just sort of like feeds on top of itself now in the real world what you will often see is that that really spirals out of control because what you have is you have people are pulling reports and they're expecting reports to come let's say you know pick a note every 10 seconds they should be able to get the report back and see their data and what happens is people get impatient so when it triggers that and so now it goes from 10 to say 15 seconds people hit report again and so there are times that have S like if if you don't have intelligence around that then it'll be you know I just created eight instances of trying to run this report that is doing table scans and if you don't know what that is go look it up and realize that's something that you should never do but maybe doing table scans and the next thing you know it's say your time has gone from 10 seconds to 10 hours and it like everybody freaks out everybody panics for somewhat I mean it's somewhat reasonable but it's also it's like hey just chill if we all we need to do is make sure that we don't hit that limit or or sometimes it's just a matter of like hey let's clean up our queries a little bit or what we cash or what we store now sometimes that's harder than others sometimes the easiest way is to just like slap a bunch of hardware on it and say boom we're going to get more memory we're going to get more processors and we'll take care of it that way but honestly it is safer better and more reliable to fix it in the software in the code itself to do the software fix now sometimes that means rewriting and so I'm going to give one more ex uh one more thing that I've run into the past before I toss it back over to that guy I didn't name earlier and that is sometimes you don't want you want to change and this goes into like being a better developer you want to change your approach or your format your what you're doing completely the framework it could even be the foundation of what you're doing and a good example of this for me is I was working for a company many years ago and they were doing these very large a files which is basically just a lot of fixed length data that's coming across we're talking hundreds of thousands of records and I was hitting this problem is that we I originally had like a nice little I think it was a Java application that would just pull this thing in spin through it really fast and kick it back out and did what it needed oh storing it out to a database it just stored it out to the database cool and it was doing all the manipulations and everything well once it got to a certain size what happened is it was sucking too much of the file in and then it ran out of memory and then we started running into problems now I switched it first thing I did I was like okay let's try to do this differently so it was like pulling a line at a time I was like okay we're just going to process We're Not Gon to have the whole file in we're just G have a pointer we're going to move through still ended up with problems because what we were having to do is actually read data and then refer to other data and so there was a lot of relational things that we had to deal with within the data so that to me was a clue that what if we try a relational database and what I ended up finding out was that if I use and this was just using uh my SQL I think it was my squel may have been postgress it was one of those where I used their import their it was a it was a store procedure that allowed you to read from a file and so what I did is I would read from the file create a create a temporary table shove all that stuff in the table do a bunch of manipulations with it and then peel it back out I think I was even creating indexes on the fly so I would pull it in index certain things basically do a big You Know M some Mass updates and then rep it back out and I went from hours to less than 30 seconds to run through this stuff and it was because the tool that I had originally thought I was going to use and so and worked fine to a certain extent and then it went it was like it was limited after that ain't going to work and I I ended up rewriting it it went a lot better and that is why more of the story is make sure that you don't get stuck in a single technology a single solution or just insist that whatever you're writing is the way to do it until you've you know you have verified that so that's my little nugget for the day let's toss this back over to you so I'm going to take it a different direction so you talked more single system based you know talking about writing your code doing the database side of things moving it to the data moving the file import to the data datase so it it's kind of a paradigm shift or moving the load essentially to a different location now in today's world we hear a lot about distributive systems we hear a lot about micro uh applications Microsystems and as we break things down smaller into their smaller components and we break these out into different applications apis deployments um you know S3 with Amazon you know message cues you know there's a whole lot of different systems out there that we have to integrate with and it can actually mask a lot of the problems that we potentially have either with our code with the configurations we have that for these systems that are deployed and it's rather interesting so it this I'm going to tell you a story of a situation I was in a couple years ago and the application I inherited uh when I became a manager of this team I found out that we had essentially our core application was on fire it literally went down once a day or two to three times a day this is a health care application this is actually pulling Healthcare information in from uh external systems into our system so this could actually impact patients lives so we needed to come up with a way one we had to figure out why it was going down so much and the very first thing we had to look at was is it our code well the system that was crashing was called Rap City and we found out that it it's essentially just a message CU that's pulling information in and passing it over to us yes it did some things on its side uh but it was essentially being disused as a pass through so as we were looking at that we literally had just taken out all of the logic and just moved message in message out so we would just pass it over it still kept crashing so we literally just moved it down to one process so it did one thing we still were essentially going down because of volume so the other thing we looked at was okay what is our Network threshold are we just essentially having too much information coming in and it's a big pipe over here and a small pipe over here and that was part of it so we did uh in increase the output flow through the network so we had more um data going out than we did initially and then we essentially found out that we were running on a machine that had two core processes maybe four gigs of RAM and it was nowhere near enough to handle the amount of data that we were consuming the other thing that no one happened to think about was how much disc space we had because of the large volume that was coming in we were actually running out a disk space on top of the memory issues on top of the CPU issues so it's not always your applications it could essentially be that the system that this was built for has aged out to the point that you need to upgrade the hardware that's the backbone of the system so that was one story The Second Story is now in a more distributed environment with AWS we have lots of applications all over the place and with the microsystem approach you typically want to rate your application to be very small to consume a very small amount of resources so it'll run on a very small build or um AWS uh like ec2 instance or Windows instance something like that so the smaller instance that you can deployed to the less money it's going to cost to keep that application up and running the problem with that attent is you could scale very small your application works initially but then again you could run into that situation like we did with the rap city system where you're the load of the application or the application is essentially outgrown it's Hardware it's outgrown the container that it's being deployed to recently I ran into a situation where we to address this they were essentially just kicking up the memory of our uh container thinking that oh we needed more memory to keep working you know to keep things happy well at 16 gigs we're still going down for one particular transaction that goes through doing some research like uh we've talked about before so this is one of those situations where the customer does one thing it breaks we as developers do something the same thing literally the same thing and it works so now it's like okay what is different between the two environments well as developers we typically have these supercomputers or very beefy machines that can do just about anything in that we wanted to do but again we're deploying to containers that are meant to be very small take up a l you know a lot less resources so they cost less money ideally what I found out was we underpowered our container to the point that we were doing so many transactions so much work behind the scenes that we essentially just see more CPU not more memory so this is one of those situations where you need to understand what the CPU is used for versus the amount of memory yes we had a lot of uh data in memory but we're doing a lot of transactions with that so you actually need more CPU power than memory because if you try to keep all that in memory you're going to get an imbalance and things are going to crash however the potentially you could still have a memory leak so what you're going to want to do is you're going to want to get things like profilers to to go monitor your code as they're running and then look for things like Heap Dums or objects in memory or memory spikes when you're doing transactions within your code so you can not just check the hardware and the physical Hardware but you still can check your code to make sure that you don't have runaway code you don't have like an infinite Loop that's just eating up memory and you never know what happens so pass back I'll give you two little quick ones too that on top both that you reminded me as you going through this one is hard drive space it used to be like I don't know in the back in the day back in the old days you used to always find codes somewhere along the way that was saying hey is there enough hard drive space like I remember going to like this is back like the old 386 dos days where like every little app that you did one of the first things it would do would be do you have enough hard drive space on it and that still exists but not to the same level in particular server applications there's a lot of times that they don't think about it because they don't care they're like hey I'm just going to run but things like log files in particular can two things actually log files I've s several times that those got huge and those sucked up the all of the data for a I actually have a little miniature same thing it had a little VM app or little you know small ec2 app and it blew up because the law got too big and it and it crashed the whole system and I've also seen it when you've got upload files if you have the ability for users to upload stuff if they can upload huge images or videos or if that's part of it then you need to make sure that that is a constant check that you have in some way form or fashion to make sure that you have enough space if You' start actually if you store locally but also if you temporarily store it locally to make sure that those are getting cleaned out because I've seen some situations where it's it's using S3 or something like that so it's moving stuff off but sometimes it doesn't move it off fast enough or clean enough and then the next thing you know you've blown that up the other thing that you can run into I just forgot but it's so it's log files and I now forgot the other one that have run into oh it's I'm sorry the other one is the debugger itself I have run into situations where if you put something into debug mode then it causes issues it in itself because it's eating up extra processing and extra time and so I have actually seen situations where it goes in the application goes into production and it breaks because somebody left the debug flag on or it's not useful because they've got it in debug mode so when you push it to production clean it up you know don't like shut that stuff off like make sure that you've got your settings such that it's not like just dumping a whole bunch of debug information to a drive somewhere if you don't need it so don't be afraid to like have some Flags or settings or something so that it only does that when you want it to which is basically you as a developer you want to see all that stuff nobody else does uh final thoughts on that yeah the other things we kind of talked about them but we didn't really address the resource things but in this distributed world we live in we have things like databases we have inmemory databases we have message cues we have S3 for external data sources as developers as we write our applications we're unfortunately we're side kind of silad we kind of see our picture our problem that we're solving and we know we have these resources we use these resources but we don't consciously take into account what is my code change going to do to the rest of the ecosystem so typically you kind of throw it over the wall and let someone else deal with it and you don't think about it but sometimes you really need to do you need to understand how that ecosystem works so that as you're writing your code if something does happen you have a better idea where to go within the structure to debug the problem also you don't want to necessarily fill up someone's database by leaving those log files on and have you know make someone else's day miserable because they're not necessarily in your department you know you don't want to really make your sis up guys mad so one other bonus that I just thought of while you were going through that is don't assume unless you can don't assume that you are always your app is always going to run in a connected state in particular one of the things that early on when I was doing I would do some development stuff and I would have some things break periodically and I'd be like wow like that website I'm bringing up looks horrible and what it was is because I was just instead of having the some of those like say like CSS files and things like that that we like to grab local they were on some you know CDN somewhere else and they were pulling it down and so if I wasn't connected nor were those and so you would suddenly have like keepy you know you'll have Library and stuff like that that they're not available because you're normally pulling those off the internet this also and this even if it's if you're a web application so you assume that the only P way people are going to get to it is if you know the internet exists and it's working don't be afraid to pull and I would say don't be afraid but just like the smart person the wise man pulls those things locally because if that CDN goes down or that site that you're borrowing that from or you're connecting to or that you're relying on goes down you are down so the more you can bring that stuff in and make it you know so if your sight's up it's up and if it's not it's not and you have less other places that you're relying on then you're going to be it's just going to be that much better however if you rely on us do that just like Double Down On It rely on us check us out like subscribe six different ways subscribe on your phone on your laptop on your watch what whatever it is because we're going to give you good stuff and if we don't that's your fault because you didn't tell us at info@ developer.com or developer.com fill out a contact form and let us know what is the good stuff that you want us to talk about how can we help you out because we're happy to help help you out this way to just like as you can see we are an endless well of full of just mistakes and issues and all kinds of problems you can run into but we might have a couple that you know be more than happy for you to suggest something so that we can make sure that it's much more applicable to you yes you right there that you're the one I'm talking to not the no not the one next to you don't look behind you you you are the one that can help us out and then we get to talk to you specifically and say Here's your problem and we may even use your name if you want us to or we'll use a really cool fake name if you want us to do that those are just some of the services we have to offer that being said it is time to wrap this one up we got to go we got to run so do you so go out there have yourself a great day a great week and we will talk to you next time deep breath cleansing sigh all right uh any bonus material you want to throw out on this one I not necessarily bonus but just kind of like a historical thing so I just read an article that I guess Doss is now open source so you can now go get Doss and reinstalled again and play around with it I remember back in the day I mean we're talking true Doss Dr Doss before Windows 31 if you wanted to install a mouse you had to configure your system to move the mouse into upper memory you had lower memory you actually had to figure out okay how many bits how many btes do I need to put this here and there what do I need so for those of you that are maybe the last decade in software you didn't have to deal with this these are we literally had to look and count how many bits and bites we could actually put in certain places in memory now today it's a lot easier however still with jvms with you know compilers things like that you still need to be conscious of this maybe not as bad but still you know Hardware is expensive even though AWS makes life a lot simpler with the resource uh sharing it does just be mindful that you know if you have something that can probably run on you know half a you know half a CPU let maybe a 100 Megs of memory run it that way you know don't you know don't kill yourself um but just be conscious of that you know don't spin up something that's 8 gigs 8 core for something that you don't need yeah um that's funny because I remember we had a had a PC Junior and got a we're going to get a mouse for it and the mouse came with a memory stamp baning card so that you could jack it up to 128 K not mag but K of ram because it needed it it had that you had to have that to be able to load the driers so they're like you know what we're just going to bundle the memory with it and I can't remember I want to say that we ended up got it and just used it for the memory as opposed to the mouse the mouse was there but like almost nothing used the mouse cuz it was on command yeah yeah I like I think I had like one like it was PC Magazine or something like that I don't remember if that was in one that like had code periodically so there was a basic app that I that like here's a code it was like 500 lines of code to do a little like game of some sort and it included code to access the the mouse and be able to do some stuff so that was the first thing where I was like I remember like sitting there and copying it off cuz you couldn't copy and paste I had to like sit there and type all of it reading from from the thing run the whole thing and I was like oh look I'm using a mouse and now it's you know gone way beyond that but I mean even before that my first computer was the Commodore Vic 20 so if we wanted memory we had to buy a 64k cartridge that plugged into the back of our keyboard and we had the tape drive so the only way you could load uh stuff into memory was to essentially play it from a cassette tape record it to a cassette tape and play it that way one last thing in the video game realm uh when the Sega Saturn came out the Sega Saturn came out with actual memory cards that you had actually you could plug into the systems um memory card so that you could actually play larger games so they kind of had an external Jack that you could uh plug in extra memory so you could play more games without having to open up the system and install new hardware so that was kind of a cool little tidbit there and some of those things haven't changed you still you know if you get depending on what you get you can get all these little like memory plug PL and stuff like that and and storage plugins so they're always going to find ways to like charge you a few extra bucks based on your your stuff oh by the way just totally random because we mentioned Games bonus stuff apparently ps5s are readily available because I was I had to go byy Best Buy to replace something the other day and they had a wall just full of them and so after what has been like two years and maybe it happened a month or two ago but I know for the longest time you couldn't find a PS5 hardly anywhere it's like if you found one you need to buy it right there because like you weren't going to see one again and I know that's I think that's how you got one I both my kids that have them that's what they did they were just like one was in a Walmart and it was like just happened to fin it's like well I wasn't going to buy a PS5 today but I saw it had to buy it uh but now apparently they're readily available so if that's something you want go out and grab yourself one before they become scarce again which probably will don't have to worry until they come out with the ps6 in which case that'll take you three or four years to be able to get your first one of those it may take you five or six years to afford it cuz um it sounds like the price Tag's going to be up there with a high-end gaming rig yeah that's that's what they keep doing it's like you know because I think the I think the PS2 when it originally came out was like 150 bucks or something like maybe1 199 and then the PS4 when it came out I think was like 299 the PS5 I think was like five or six the low version it's like they it's like everybody else they're gonna like they're gonna throw some extra stuff but they're going to charge you more we will not we're going to charge you the same amount every time you come here we are going to wrap this one up because we're going to write in the sunset or something like that and go have a good time as always feel free to like leave us notes in the comments uh check out the show notes those are like we you know got some AI stuff and stuff like that we usually get really cool show notes lots of links there check us out here so you can check that out uh on the site at developer.com or you can just leave some comments here and say hey dudes can you send me something and we'll shoot you an email of that episode stuff or we may throw other stuff at you it depends because we do have things floating around that we could send to you just a little teaser there no promises but you never know you never know what might end up in your mail if you send us something because we are very appreciative for you guys out there listening to us and you know being there and putting up with our ugly mugs for the last little bit of time that you did that being said we're not going to go get facial surgery so you're still going to be able to see our same ugly mugs next time around and you guys have a great day and we will talk to you then take care everyone uh wh that is not the button I there we go stop recording I freaked out [Music]
Transcript Segments
[Music]
all right uh let's talk about what do we
want to do for this episode you have a I
was like we just came up out of nowhere
last time so did you have have you had
any thoughts today of or this week of
like what would be a good episode
topic well the kind of to go with the
example you were giving previously
before we recorded last time uh have you
run into situations where not only is
your code your issue but the hardware is
the issue like you're not actually
configuring the hardware for your
application oh gez I've been like that
was yesterday I was that was half my day
it wasn't it was it's frustrating so
it's I've put together um Apache front
ends to Jango now a couple of times and
it's it's always a little bit different
because it like if you do a dbn
versus um and I forget what the core
like but yeah if you use like aptg
versus yum versus Mac and I haven't even
tried Windows nor do I you know I did
like decades back and don't want to ever
touch it again they're all different
they're all like close but not
quite and so you have to like go find
the right library and the right download
and the right all this stuff and I was
working on a machine that I thought was
one I got and realized oh wait it's
different but it's like it's a whole
thing that I'm setting up because I'm
setting up Apache which I got that you
know that was fine but I'm setting up
Apache I'm setting up SSH or you know
SSL stuff I'm using um certbot so I'm
grabbing aert and I'm trying to make
sure that I've got like that hooked up
so that it grabs the right certific
certificate and then making sure I've
got the files in the right place that it
handles you know Port you know HTTP as
well as https and it's connecting to
everything and that it's all routing
right and it's just a bunch of stuff
that's like very sensitive got to be the
right got to be the exact right name
names have got to match right and
everything and I wasn't able to do it in
one
sitting I got something done got pulled
away got something else done pulled away
and so it just took forever because
every time I sat down I had to one had
to remember where I was at and then two
had to like go track this down and
there's you know three or four different
log files I'm looking for
so short story long yes I have
definitely run into that couple of times
configuration
is if I could never if I could do
software and never configure anything
ever again in my life even my own
software I think I would be happy
because I I get bit by my own
configuration settings on a regular
basis where it's like I built it like an
app I was testing today I was like why
is that not working it's like oh yeah
we've got this configuration setting and
we didn't set it right it's like we
adjusted it all right let's set it back
and then boom everything works again and
seeing on the flip side of that with the
testing stuff when you want to do like
low testing or high-end regression
testing you need to run your test
multi-threaded well in order to do that
you need a certain amount of CPU and you
need a certain amount of ram in order to
be able to keep up with that load it's
not necessarily always about memory
leaks or things like that if you are
trying to push high volume you need to
have the resources to push it and that
was one of these things I was running
down today is
we tested all of our production code in
QA at high volume for generating
reports and I had it all working
everything worked for every use case we
had now it's in production and what I
didn't know is that we've been kicking
up the memory in
production uh we're up to like 16 gigs
and we're still running out of memory
with our largest uh with one of our
largest vendors that has over 2 million
records come to find out we're only
running for uh four core processor with
16 gigs of RAM so I actually uh one of
the cool things about spring boot uh and
Eclipse you can actually configure your
profiler for the specific cores and
memory you want to run so I've got a 12
core laptop with 64 gigs of RAM so I can
actually do pretty much whatever I want
locally without even needing to do
containers or I could stand up a
container whatever but what essentially
I ran into is I clicked run locally I
mean I'm running all of our applications
locally we all microservices distributed
but I have it all
locally clicked run ran
fine use case worked report guide
generated no problem I'm like all right
what the hell's wrong and then I started
to getting into it I'm like well let me
go look at
production finally got uh back into
production because I found out AWS for
some reason defaulted to a different
region for
me so while I'm able to log in I had not
like I had no servers I had no I'm like
where the hell is all my
stuff that was a fun hour uh I have lost
that hour myself a couple of times
anyway once I got in there though I
walked into the deployment and looked at
the setup and found out okay we've got
four cores and 16 Ram so I went into to
jvm and I configured it to 4 gigs of RAM
and it was cool cuz I found out with the
jvm you could um with any Clips you can
do like available chords or available
system I I forget the exact name it's
not the same as you would for Maven it's
a
different uh property uh VM but you can
actually set the ches so then I used uh
visual
VM uh yeah visual VM to uh Prof the
memory cuz we used that before in the
past it's a cool little free profiler
and I spun it up I watched things go and
before I had this nice flat line um with
my uh 12 cores with all my memory it
barely got above uh like four gigs of
RAM under four cores and 16 gigs of RAM
or I did I did four cores and I did four
gigs of RAM to start with so I did four
for and immediately I watched the spike
go from this to this and within a like
30 seconds I'm out of memory so I'm like
okay that was interesting so then I
flipped it back to 12 core still ran out
of memory but I was able to get it up to
um six gigs of RAM and it worked so I
slowly then started rolling back the
processors and essentially I got it down
to eight and eight so if you have eight
cores with 8 gigs of RAM it worked now
my boss is like well we need to go look
at the code we have a memory leag I
don't think so I think trying to process
two million records in memory at once
because we're sorting them we're doing
manipulations with that's a lot of
processing power it's not just memory
we're trying to do stuff with that if we
were just holding it in memory sure I
would say we have a memory leak but no
we're constantly like resorting the list
for um building uh spreadsheets with
this I don't think it's necessarily A
Memory leak as it's a processor Le we
need more processing power anyway this
was kind of this discussion I thought
would be good for the
podcast which we're now like way I think
so I think we
can well we're gonna yeah I think we can
go into some of this stuff is just some
like
common odity type things uh that we've
that we've run into and uh I've got some
similar types of experiences that I can
talk about a little bit and sort of like
what uh some of it's like thresholds and
stuff like
that and um sort of tackling those a
little bit and then I'll toss it over to
you and then you can you can probably
rehash some of that so those you guys
that are listening sorry you may hear
this twice but we don't remember what we
said so it'll be a little bit different
anyway so there may be something more
ingenious on that second side around
well hello and welcome back we are I'm
sorry I didn't mean to smack too close
to the mic but that happens sometimes uh
we are back with building better
developers yet another episode where
we're talking about stuff that we've run
into uh this episode we're going to go a
little bit uh some of it's very current
for Michael some of it's not so much for
me but some of the things that I've hit
in the past and it's really it gets into
like debugging stuff when you get away
from it's like not just code it's
hardware issues and and it literally is
like memory and resource issues and
things like that which
you may not run into as much depending
on what your your applications are
because these days there's a lot of
stuff that is very much plays well
within the the Norms but if you start
doing big data if you start doing data
processing if you've got a an app that's
got a lot of users or a lot of
processing a lot of data any of that a
lot of stuff going on things can happen
so we're going to talk about that first
I'm going to introduce myself my name is
Rob briad I'm one of the founders of
developer nor I'm also a founder of RB
Consulting all things about simplifying
integrating and and just automating your
software your solutions to make your
business run smoother on the other side
this time I'm not even going to mention
his name I'm going to let them do the
whole introduction so go for it sir hey
everyone my name is Michael MOS I'm
another co-founder of developer erer I'm
also the founder of Envision QA where we
build custom software for small to
midsize businesses and and Healthcare
clinicians we also uh offered consulting
services and testing services for your
needs awesome now we're get right into
one of those cases where you may need
some of those testing
services in particularly it is with like
the the big Behemoth of stuff that we
can sometimes run into in the software
world is the hardware stuff the
resources that are related to like
processing and the ones related to well
and storage and memory now the fun ones
are when storage and memory cross lines
and this is where you see some really
neat stuff and these are the kinds of
things that it's just we're going to
give you some like they're not I don't
know if they're necessarily red flags
but there's some some things that are
maybe some notes that if you are in a
testing situation or in a situation
where hey it works on my machine or
usually and now it doesn't and the code
should be the same when you get a sense
like that doesn't make sense it's it's
not a different system here's some
things to go look at maybe there's
something going on that's a little
deeper than that where you're going to
want to have some sort of you know some
more detailed logging particularly from
the the operating system side the
platform side to see what is actually
going on now one of the like the mother
of all break your application things
that happens is when you trigger when
you when you get past physical memory
and it starts to cash stuff because what
happens is now you've got physical
memory that's just like really it's like
boom boom boom boom boom boom boom no
it's gotten better because SSD drives
and some of that stuff is very fast and
and there's a lot of intelligence buil
around those things often so that you
can so it's going to keep the uh the
most recent data like up front so if you
have something that if you're caching
your memory the system memory into like
a SSD drive or really high-speed drive
you may not notice the difference as
much but definitely if you're old school
and you go to one of the old you know uh
dis driven type hard drives this the
processing speed goes from microc
seconds to multisecond
and it literally I don't know how many
times it was stuff where it's like you
know I've just pick a number like we
have let's say we have two gig of of
memory on our machine and everything was
awesome until we had it grow Beyond two
gig because as soon as it did that it
had to C it had to swap stuff out and
now you have the swap cost so now it's
it's pushing stuff out to that hard
drive and then having to pull stuff back
and that's adding some processing time
and then it just sort of like feeds on
top of itself now in the real world
what you will often see is that that
really spirals out of control because
what you have is you have people are
pulling reports and they're expecting
reports to come let's say you know pick
a note every 10 seconds they should be
able to get the report back and see
their data and what happens is people
get impatient so when it triggers that
and so now it goes from 10 to say 15
seconds people hit report again and so
there are times that have S like if if
you don't have intelligence around that
then it'll be you know I just created
eight instances of trying to run this
report that is doing table scans and if
you don't know what that is go look it
up and realize that's something that you
should never do but maybe doing table
scans and the next thing you know it's
say your time has gone from 10 seconds
to 10
hours and it like everybody freaks out
everybody panics for somewhat I mean
it's somewhat reasonable but it's also
it's like hey just chill if we all we
need to do is make sure that we don't
hit that limit or or sometimes it's just
a matter of like hey let's clean up our
queries a little bit or what we cash or
what we store now sometimes that's
harder than others sometimes the easiest
way is to just like slap a bunch of
hardware on it and say boom we're going
to get more memory we're going to get
more processors and we'll take care of
it that way but honestly it is safer
better and more reliable to fix it in
the software in the code itself to do
the software fix now sometimes that
means rewriting and so I'm going to give
one more ex uh one more thing that I've
run into the past before I toss it back
over to that guy I didn't name earlier
and that is sometimes you don't want you
want to change and this goes into like
being a better developer you want to
change your approach or your format your
what you're doing completely the
framework it could even be the
foundation of what you're doing and a
good example of this for me is I was
working for a company many years ago and
they were doing these very large a files
which is basically just a lot of fixed
length data that's coming across we're
talking hundreds of thousands of records
and I was hitting this problem is that
we I originally had like a nice little I
think it was a Java application that
would just pull this thing in spin
through it really fast and kick it back
out and did what it needed oh storing it
out to a database it just stored it out
to the database cool and it was doing
all the manipulations and everything
well once it got to a certain
size what happened is it was sucking too
much of the file in and then it ran out
of memory and then we started running
into problems now I switched it first
thing I did I was like okay let's try to
do this differently so it was like
pulling a line at a time I was like okay
we're just going to process We're Not
Gon to have the whole file in we're just
G have a pointer we're going to move
through still ended up with problems
because what we were having to do is
actually read data and then refer to
other data and so there was a lot of
relational things that we had to deal
with within the data so that to me was a
clue that what if we try a relational
database
and what I ended up finding out was that
if I use and this was just using uh my
SQL I think it was my squel may have
been postgress it was one of those where
I used their import their it was a it
was a store procedure that allowed you
to read from a file and so what I did is
I would read from the file create a
create a temporary table shove all that
stuff in the table do a bunch of
manipulations with it and then peel it
back out I think I was even creating
indexes on the fly so I would pull it in
index certain things
basically do a big You Know M some Mass
updates and then rep it back out and I
went from hours to less than 30 seconds
to run through this stuff and it was
because the tool that I had originally
thought I was going to use and so and
worked fine to a certain extent and then
it went it was like it was limited after
that ain't going to work and I I ended
up rewriting it it went a lot better and
that is why more of the story is make
sure that you don't get stuck in a
single technology a single solution or
just insist that whatever you're writing
is the way to do it until you've you
know you have verified that so that's my
little nugget for the day let's toss
this back over to
you so I'm going to take it a different
direction so you talked more single
system based you know talking about
writing your code doing the database
side of things moving it to the data
moving the file import to the data
datase so it it's kind of a paradigm
shift or moving the load essentially to
a different
location now in today's world we hear a
lot about distributive systems we hear a
lot about micro uh applications
Microsystems and as we break things down
smaller into their smaller components
and we break these out into different
applications apis deployments um you
know S3 with Amazon you know message
cues you know there's a whole lot of
different systems out there that we have
to integrate with and it can actually
mask a lot of the problems that we
potentially have either with our code
with the configurations we have that for
these systems that are
deployed and it's rather interesting so
it this I'm going
to tell you a story of a situation I was
in a couple years ago and the
application I inherited uh when I became
a manager of this team I found out that
we had essentially our core application
was on fire it literally went down once
a day or two to three times a day this
is a health care application this is
actually pulling Healthcare information
in from uh external systems into our
system so this could actually impact
patients lives so we needed to come up
with a way one we had to figure out why
it was going down so much and the very
first thing we had to look at was is it
our code well the system that was
crashing was called Rap City and we
found out that it it's essentially just
a message CU that's pulling information
in and passing it over to us yes it did
some things on its side uh but it was
essentially being disused as a pass
through so as we were looking at that we
literally had just taken out all of the
logic and just moved message in message
out so we would just pass it over it
still kept
crashing so we literally just moved it
down to one process so it did one thing
we still were essentially going down
because of volume so the other thing we
looked at was okay what is our Network
threshold are we just essentially having
too much information coming in and it's
a big pipe over here and a small pipe
over here and that was part of it so we
did uh in increase the output flow
through the network so we had more um
data going out than we did initially and
then we essentially found out that we
were running on a machine that had two
core processes maybe four gigs of RAM
and it was nowhere near enough to handle
the amount of data that we were
consuming the other thing that no one
happened to think about was how much
disc space we had because of the large
volume that was coming in we were
actually running out a disk space on top
of the memory issues on top of the CPU
issues so it's not always your
applications it could essentially be
that the system that this was built for
has aged out to the point that you need
to upgrade the hardware that's the
backbone of the system so that was one
story The Second Story is now in a more
distributed environment with AWS we have
lots of applications all over the place
and with the microsystem approach you
typically want to rate your application
to be very small to consume a very small
amount of resources so it'll run on a
very small build or um
AWS uh like ec2 instance or Windows
instance something like that so the
smaller instance that you can deployed
to the less money it's going to cost to
keep that application up and running the
problem with that attent is you could
scale very small your application works
initially but then again you could run
into that situation like we did with the
rap city system where you're the load of
the application or the application is
essentially outgrown it's Hardware it's
outgrown the container that it's being
deployed
to recently I ran into a situation where
we to address this they were essentially
just kicking up the memory of our uh
container thinking that oh we needed
more memory to keep working you know to
keep things happy well at 16 gigs we're
still going down for one particular
transaction that goes
through doing some
research like uh we've talked about
before so this is one of those
situations where the customer does one
thing it breaks we as developers do
something the same thing literally the
same thing and it works so now it's like
okay what is different between the two
environments well as developers we
typically have these supercomputers or
very beefy machines that can do just
about anything in that we wanted to do
but again we're deploying to containers
that are meant to be very small take up
a l you know a lot less resources so
they cost less
money ideally what I found out was we
underpowered our container to the point
that we were doing so many transactions
so much work behind the scenes that we
essentially just see more CPU not more
memory so this is one of those
situations where you need to understand
what the CPU is used for versus the
amount of memory yes we had a lot of uh
data in memory but we're doing a lot of
transactions with that so you actually
need more CPU power than memory because
if you try to keep all that in memory
you're going to get an imbalance and
things are going to
crash however the potentially you could
still have a memory leak so what you're
going to want to do is you're going to
want to get things like profilers to to
go monitor your code as they're running
and then look for things like Heap Dums
or objects in memory or memory spikes
when you're doing transactions within
your code so you can not just check the
hardware and the physical Hardware but
you still can check your code to make
sure that you don't have runaway code
you don't have like an infinite Loop
that's just eating up memory and you
never know what
happens so pass back I'll give you two
little quick ones too that on top both
that you reminded me as you going
through this one is hard drive space it
used to be like I don't know in the back
in the day back in the old days you used
to always find codes somewhere along the
way that was saying hey is there enough
hard drive space like I remember going
to like this is back like the old 386
dos days where like every little app
that you did one of the first things it
would do would be do you have enough
hard drive space on it and that still
exists but not to the same level in
particular server applications there's a
lot of times that they don't think about
it because they don't care they're like
hey I'm just going to run but things
like log files in particular can two
things actually log files I've s several
times that those got huge and those
sucked up the all of the data for a I
actually have a little miniature same
thing it had a little VM app or little
you know small ec2 app and it blew up
because the law got too
big and it and it crashed the whole
system and I've also seen it when you've
got upload files if you have the ability
for users to upload stuff if they can
upload huge images or videos or if
that's part of it then you need to make
sure that that is a
constant check that you have in some way
form or fashion to make sure that you
have enough space if You' start actually
if you store locally but also if you
temporarily store it locally to make
sure that those are getting cleaned out
because I've seen some situations where
it's it's using S3 or something like
that so it's moving stuff off but
sometimes it doesn't move it off fast
enough or clean enough and then the next
thing you know you've blown that
up the other thing that you can run into
I just forgot but it's so it's log
files
and I now forgot the other one that have
run
into oh it's I'm sorry the other one is
the debugger itself I have run into
situations where if you put something
into debug
mode then it causes issues it in itself
because it's eating up extra processing
and extra time and so I have actually
seen situations where it goes in the
application goes into production and it
breaks because somebody left the debug
flag on or it's not useful because
they've got it in debug mode so when you
push it to
production clean it up you know don't
like shut that stuff off like make sure
that you've got your settings such that
it's not like just dumping a whole bunch
of debug information to a drive
somewhere if you don't need it so don't
be afraid to like have some Flags or
settings or something so that it only
does that when you want it to which is
basically you as a developer you want to
see all that stuff nobody else does
uh final thoughts on
that yeah the other things we kind of
talked about them but we didn't really
address the resource things but in this
distributed world we live in we have
things like databases we have inmemory
databases we have message cues we have
S3 for external data
sources as developers as we write our
applications we're unfortunately we're
side kind of silad we kind of see our
picture our problem that we're solving
and we know we have these resources we
use these resources but we don't
consciously take into account what is my
code change going to do to the rest of
the ecosystem so typically you kind of
throw it over the wall and let someone
else deal with it and you don't think
about it but sometimes you really need
to do you need to understand how that
ecosystem works so that as you're
writing your code if something does
happen you have a better idea where to
go within the structure to debug the
problem also you don't want to
necessarily fill up someone's database
by leaving those log files on and have
you know make someone else's day
miserable because they're not
necessarily in your department you know
you don't want to really make your sis
up guys
mad so one other bonus that I just
thought of while you were going through
that is don't assume unless you can
don't assume that you are always your
app is always going to run in a
connected state in particular one of the
things that
early on when I was doing I would do
some development stuff and I would have
some things break periodically and I'd
be like wow like that website I'm
bringing up looks horrible and what it
was is because I was just instead of
having the some of those like say like
CSS files and things like that that we
like to grab local they were on some you
know CDN somewhere else and they were
pulling it down and so if I wasn't
connected nor were those and so you
would suddenly have like keepy you know
you'll have Library and stuff like that
that they're not available because
you're normally pulling those off the
internet this also and this even if it's
if you're a web application so you
assume that the only P way people are
going to get to it is if you know the
internet exists and it's
working don't be afraid to pull and I
would say don't be afraid but just like
the smart person the wise man pulls
those things locally because if that CDN
goes down or that site that you're
borrowing that from or you're connecting
to or that you're relying on goes down
you are down so the more you can bring
that stuff in and make it you know so if
your sight's up it's up and if it's not
it's not and you have less other places
that you're relying on then you're going
to be it's just going to be that much
better however if you rely on us do that
just like Double Down On It rely on us
check us out like subscribe six
different ways subscribe on your phone
on your laptop on your watch what
whatever it is because we're going to
give you good stuff and if we don't
that's your fault because you didn't
tell us at info@ developer.com or
developer.com fill out a contact form
and let us know what is the good stuff
that you want us to talk about how can
we help you out because we're happy to
help help you out this way to just like
as you can see we are an endless well of
full of just mistakes and issues and all
kinds of problems you can run into but
we might have a couple that you know be
more than happy for you to suggest
something so that we can make sure that
it's much more applicable to you yes you
right there that you're the one I'm
talking to not the no not the one next
to you don't look behind you you you are
the one that can help us out and then we
get to talk to you specifically and say
Here's your problem and we may even use
your name if you want us to or we'll use
a really cool fake name if you want us
to do that those are just some of the
services we have to offer that being
said it is time to wrap this one up we
got to go we got to run so do you so go
out there have yourself a great day a
great week and we will talk to you next
time deep
breath cleansing sigh all right uh any
bonus material you want to throw out on
this one I not necessarily bonus but
just kind of like a historical thing
so I just read an article that I guess
Doss is now open source so you can now
go get Doss and reinstalled again and
play around with
it I remember back in the day I mean
we're talking true Doss Dr Doss before
Windows 31 if you wanted to install a
mouse you had to configure your system
to move the mouse into upper memory you
had lower memory you actually had to
figure out okay how many bits how many
btes do I need to put this here and
there what do I need so for those of you
that are maybe the last decade in
software you didn't have to deal with
this these are we literally had to look
and count how many bits and bites we
could actually put in certain places in
memory now today it's a lot easier
however still with jvms with you know
compilers things like that you still
need to be conscious of this maybe not
as bad but still you know Hardware is
expensive even though AWS makes life a
lot simpler with the resource uh sharing
it does just be mindful that you know if
you have something that can probably run
on you know half a you know half a CPU
let maybe a 100 Megs of memory run it
that way you know don't you know don't
kill yourself um but just be conscious
of that you know don't spin up something
that's 8 gigs 8 core for something that
you don't
need yeah um that's funny because I
remember we had a had a PC Junior
and got a we're going to get a mouse for
it and the mouse came with a memory
stamp baning card so that you could jack
it up to 128 K not mag but K of ram
because it needed it it had that you had
to have that to be able to load the
driers so they're like you know what
we're just going to bundle the memory
with it and I can't remember I want to
say that we ended up got it and just
used it for the memory as opposed to the
mouse the mouse was there but like
almost nothing used the mouse cuz it was
on command yeah yeah I like I think I
had like one like it was PC Magazine or
something like that I don't remember if
that was in one that like had code
periodically so there was a basic app
that I that like here's a code it was
like 500 lines of code to do a little
like game of some sort and it included
code to access the the mouse and be able
to do some stuff so that was the first
thing where I was like I remember like
sitting there and copying it off cuz you
couldn't copy and paste I had to like
sit there and type all of it reading
from from the thing run the whole thing
and I was like oh look I'm using a mouse
and now it's you know gone way beyond
that but I mean even before that my
first computer was the Commodore Vic 20
so if we wanted memory we had to buy a
64k cartridge that plugged into the back
of our keyboard and we had the tape
drive so the only way you could load uh
stuff into memory was to essentially
play it from a cassette tape record it
to a cassette tape and play it that way
one last thing in the video game realm
uh when the Sega Saturn came out the
Sega Saturn came out with actual memory
cards that you had actually you could
plug into the systems um memory card so
that you could actually play larger
games so they kind of had an external
Jack that you could uh plug in extra
memory so you could play more games
without having to open up the system and
install new hardware so that was kind of
a cool little tidbit there and some of
those things haven't changed you still
you know if you get depending on what
you get you can get all these little
like memory plug PL and stuff like that
and and storage plugins so they're
always going to find ways to like charge
you a few extra bucks based on your your
stuff oh by the way just totally random
because we mentioned Games bonus stuff
apparently ps5s are readily available
because I was I had to go byy Best Buy
to replace something the other day and
they had a wall just full of them and so
after what has been like two years and
maybe it happened a month or two ago but
I know for the longest time you couldn't
find a PS5 hardly anywhere it's like if
you found one you need to buy it right
there because like you weren't going to
see one again and I know that's I think
that's how you got one I both my kids
that have them that's what they did they
were just like one was in a Walmart and
it was like just happened to fin it's
like well I wasn't going to buy a PS5
today but I saw it had to buy it uh but
now apparently they're readily available
so if that's something you want go out
and grab yourself one before they become
scarce again which probably will don't
have to worry until they come out with
the ps6 in which case that'll take you
three or four years to be able to get
your first one of those it may take you
five or six years to afford it cuz um it
sounds like the price Tag's going to be
up there with a high-end gaming rig yeah
that's that's what they keep doing it's
like you know because I think the I
think the PS2 when it originally came
out was like 150 bucks or something like
maybe1 199 and then the PS4 when it came
out I think was like 299 the PS5 I think
was like five or six the low version
it's like they it's like everybody else
they're gonna like they're gonna throw
some extra stuff but they're going to
charge you more we will not we're going
to charge you the same amount every time
you come here we are going to wrap this
one up because we're going to write in
the sunset or something like that and go
have a good time as always feel free to
like leave us notes in the comments uh
check out the show notes those are like
we you know got some AI stuff and stuff
like that we usually get really cool
show notes lots of links there check us
out here so you can check that out uh on
the site at developer.com or you can
just leave some comments here and say
hey dudes can you send me something and
we'll shoot you an email of that episode
stuff or we may throw other stuff at you
it depends because we do have things
floating around that we could send to
you just a little teaser there no
promises but you never know you never
know what might end up in your mail if
you send us something because we are
very appreciative for you guys out there
listening to us and you know being there
and putting up with our ugly mugs for
the last little bit of time that you did
that being said we're not going to go
get facial surgery so you're still going
to be able to see our same ugly mugs
next time around and you guys have a
great day and we will talk to you then
take care
everyone uh
wh that is not the button I there we go
stop recording I freaked out
[Music]