📺 Develpreneur YouTube Episode

Video + transcript

Managing Hardware Resource Issues in Software Development

2024-06-13 •Youtube

Detailed Notes

Welcome back to another episode of Building Better Developers, where we dive into essential aspects of software development. Today, we’re delving into “Managing Hardware Resource Issues in Software Development: Best Practices and Real-World Examples.” In this episode, we’ll explore the often-overlooked yet critical area of hardware resource challenges and their impact on software performance. From memory and storage constraints to processing limitations, we’ll navigate through common issues developers face and provide insights into effective debugging strategies and optimization techniques. Let’s embark on this journey to understand and address hardware resource issues in software development.

Read more: https://develpreneur.com/managing-hardware-resource-issues-in-software-development

Stay Connected: Join the Developreneur Community We invite you to join our community and share your coding journey with us. Whether you’re a seasoned developer or just starting, there’s always room to learn and grow together. Contact us at [email protected] with your questions, feedback, or suggestions for future episodes. Together, let’s continue exploring the exciting world of software development.

Additional Resources * Troubleshooting Tips for Common EC2 Problems (https://develpreneur.com/troubleshooting-tips-common-ec2-problems/) * Testing Challenges – Teach More Than Quality (https://develpreneur.com/testing-challenges-teach-more-than-quality/) * Software Architecture Patterns and Anti-Patterns Overview (https://develpreneur.com/software-architecture-patterns-and-anti-patterns-overview/)

Transcript Text
[Music]
all right uh let's talk about what do we
want to do for this episode you have a I
was like we just came up out of nowhere
last time so did you have have you had
any thoughts today of or this week of
like what would be a good episode
topic well the kind of to go with the
example you were giving previously
before we recorded last time uh have you
run into situations where not only is
your code your issue but the hardware is
the issue like you're not actually
configuring the hardware for your
application oh gez I've been like that
was yesterday I was that was half my day
it wasn't it was it's frustrating so
it's I've put together um Apache front
ends to Jango now a couple of times and
it's it's always a little bit different
because it like if you do a dbn
versus um and I forget what the core
like but yeah if you use like aptg
versus yum versus Mac and I haven't even
tried Windows nor do I you know I did
like decades back and don't want to ever
touch it again they're all different
they're all like close but not
quite and so you have to like go find
the right library and the right download
and the right all this stuff and I was
working on a machine that I thought was
one I got and realized oh wait it's
different but it's like it's a whole
thing that I'm setting up because I'm
setting up Apache which I got that you
know that was fine but I'm setting up
Apache I'm setting up SSH or you know
SSL stuff I'm using um certbot so I'm
grabbing aert and I'm trying to make
sure that I've got like that hooked up
so that it grabs the right certific
certificate and then making sure I've
got the files in the right place that it
handles you know Port you know HTTP as
well as https and it's connecting to
everything and that it's all routing
right and it's just a bunch of stuff
that's like very sensitive got to be the
right got to be the exact right name
names have got to match right and
everything and I wasn't able to do it in
one
sitting I got something done got pulled
away got something else done pulled away
and so it just took forever because
every time I sat down I had to one had
to remember where I was at and then two
had to like go track this down and
there's you know three or four different
log files I'm looking for
so short story long yes I have
definitely run into that couple of times
configuration
is if I could never if I could do
software and never configure anything
ever again in my life even my own
software I think I would be happy
because I I get bit by my own
configuration settings on a regular
basis where it's like I built it like an
app I was testing today I was like why
is that not working it's like oh yeah
we've got this configuration setting and
we didn't set it right it's like we
adjusted it all right let's set it back
and then boom everything works again and
seeing on the flip side of that with the
testing stuff when you want to do like
low testing or high-end regression
testing you need to run your test
multi-threaded well in order to do that
you need a certain amount of CPU and you
need a certain amount of ram in order to
be able to keep up with that load it's
not necessarily always about memory
leaks or things like that if you are
trying to push high volume you need to
have the resources to push it and that
was one of these things I was running
down today is
we tested all of our production code in
QA at high volume for generating
reports and I had it all working
everything worked for every use case we
had now it's in production and what I
didn't know is that we've been kicking
up the memory in
production uh we're up to like 16 gigs
and we're still running out of memory
with our largest uh with one of our
largest vendors that has over 2 million
records come to find out we're only
running for uh four core processor with
16 gigs of RAM so I actually uh one of
the cool things about spring boot uh and
Eclipse you can actually configure your
profiler for the specific cores and
memory you want to run so I've got a 12
core laptop with 64 gigs of RAM so I can
actually do pretty much whatever I want
locally without even needing to do
containers or I could stand up a
container whatever but what essentially
I ran into is I clicked run locally I
mean I'm running all of our applications
locally we all microservices distributed
but I have it all
locally clicked run ran
fine use case worked report guide
generated no problem I'm like all right
what the hell's wrong and then I started
to getting into it I'm like well let me
go look at
production finally got uh back into
production because I found out AWS for
some reason defaulted to a different
region for
me so while I'm able to log in I had not
like I had no servers I had no I'm like
where the hell is all my
stuff that was a fun hour uh I have lost
that hour myself a couple of times
anyway once I got in there though I
walked into the deployment and looked at
the setup and found out okay we've got
four cores and 16 Ram so I went into to
jvm and I configured it to 4 gigs of RAM
and it was cool cuz I found out with the
jvm you could um with any Clips you can
do like available chords or available
system I I forget the exact name it's
not the same as you would for Maven it's
a
different uh property uh VM but you can
actually set the ches so then I used uh
visual
VM uh yeah visual VM to uh Prof the
memory cuz we used that before in the
past it's a cool little free profiler
and I spun it up I watched things go and
before I had this nice flat line um with
my uh 12 cores with all my memory it
barely got above uh like four gigs of
RAM under four cores and 16 gigs of RAM
or I did I did four cores and I did four
gigs of RAM to start with so I did four
for and immediately I watched the spike
go from this to this and within a like
30 seconds I'm out of memory so I'm like
okay that was interesting so then I
flipped it back to 12 core still ran out
of memory but I was able to get it up to
um six gigs of RAM and it worked so I
slowly then started rolling back the
processors and essentially I got it down
to eight and eight so if you have eight
cores with 8 gigs of RAM it worked now
my boss is like well we need to go look
at the code we have a memory leag I
don't think so I think trying to process
two million records in memory at once
because we're sorting them we're doing
manipulations with that's a lot of
processing power it's not just memory
we're trying to do stuff with that if we
were just holding it in memory sure I
would say we have a memory leak but no
we're constantly like resorting the list
for um building uh spreadsheets with
this I don't think it's necessarily A
Memory leak as it's a processor Le we
need more processing power anyway this
was kind of this discussion I thought
would be good for the
podcast which we're now like way I think
so I think we
can well we're gonna yeah I think we can
go into some of this stuff is just some
like
common odity type things uh that we've
that we've run into and uh I've got some
similar types of experiences that I can
talk about a little bit and sort of like
what uh some of it's like thresholds and
stuff like
that and um sort of tackling those a
little bit and then I'll toss it over to
you and then you can you can probably
rehash some of that so those you guys
that are listening sorry you may hear
this twice but we don't remember what we
said so it'll be a little bit different
anyway so there may be something more
ingenious on that second side around
well hello and welcome back we are I'm
sorry I didn't mean to smack too close
to the mic but that happens sometimes uh
we are back with building better
developers yet another episode where
we're talking about stuff that we've run
into uh this episode we're going to go a
little bit uh some of it's very current
for Michael some of it's not so much for
me but some of the things that I've hit
in the past and it's really it gets into
like debugging stuff when you get away
from it's like not just code it's
hardware issues and and it literally is
like memory and resource issues and
things like that which
you may not run into as much depending
on what your your applications are
because these days there's a lot of
stuff that is very much plays well
within the the Norms but if you start
doing big data if you start doing data
processing if you've got a an app that's
got a lot of users or a lot of
processing a lot of data any of that a
lot of stuff going on things can happen
so we're going to talk about that first
I'm going to introduce myself my name is
Rob briad I'm one of the founders of
developer nor I'm also a founder of RB
Consulting all things about simplifying
integrating and and just automating your
software your solutions to make your
business run smoother on the other side
this time I'm not even going to mention
his name I'm going to let them do the
whole introduction so go for it sir hey
everyone my name is Michael MOS I'm
another co-founder of developer erer I'm
also the founder of Envision QA where we
build custom software for small to
midsize businesses and and Healthcare
clinicians we also uh offered consulting
services and testing services for your
needs awesome now we're get right into
one of those cases where you may need
some of those testing
services in particularly it is with like
the the big Behemoth of stuff that we
can sometimes run into in the software
world is the hardware stuff the
resources that are related to like
processing and the ones related to well
and storage and memory now the fun ones
are when storage and memory cross lines
and this is where you see some really
neat stuff and these are the kinds of
things that it's just we're going to
give you some like they're not I don't
know if they're necessarily red flags
but there's some some things that are
maybe some notes that if you are in a
testing situation or in a situation
where hey it works on my machine or
usually and now it doesn't and the code
should be the same when you get a sense
like that doesn't make sense it's it's
not a different system here's some
things to go look at maybe there's
something going on that's a little
deeper than that where you're going to
want to have some sort of you know some
more detailed logging particularly from
the the operating system side the
platform side to see what is actually
going on now one of the like the mother
of all break your application things
that happens is when you trigger when
you when you get past physical memory
and it starts to cash stuff because what
happens is now you've got physical
memory that's just like really it's like
boom boom boom boom boom boom boom no
it's gotten better because SSD drives
and some of that stuff is very fast and
and there's a lot of intelligence buil
around those things often so that you
can so it's going to keep the uh the
most recent data like up front so if you
have something that if you're caching
your memory the system memory into like
a SSD drive or really high-speed drive
you may not notice the difference as
much but definitely if you're old school
and you go to one of the old you know uh
dis driven type hard drives this the
processing speed goes from microc
seconds to multisecond
and it literally I don't know how many
times it was stuff where it's like you
know I've just pick a number like we
have let's say we have two gig of of
memory on our machine and everything was
awesome until we had it grow Beyond two
gig because as soon as it did that it
had to C it had to swap stuff out and
now you have the swap cost so now it's
it's pushing stuff out to that hard
drive and then having to pull stuff back
and that's adding some processing time
and then it just sort of like feeds on
top of itself now in the real world
what you will often see is that that
really spirals out of control because
what you have is you have people are
pulling reports and they're expecting
reports to come let's say you know pick
a note every 10 seconds they should be
able to get the report back and see
their data and what happens is people
get impatient so when it triggers that
and so now it goes from 10 to say 15
seconds people hit report again and so
there are times that have S like if if
you don't have intelligence around that
then it'll be you know I just created
eight instances of trying to run this
report that is doing table scans and if
you don't know what that is go look it
up and realize that's something that you
should never do but maybe doing table
scans and the next thing you know it's
say your time has gone from 10 seconds
to 10
hours and it like everybody freaks out
everybody panics for somewhat I mean
it's somewhat reasonable but it's also
it's like hey just chill if we all we
need to do is make sure that we don't
hit that limit or or sometimes it's just
a matter of like hey let's clean up our
queries a little bit or what we cash or
what we store now sometimes that's
harder than others sometimes the easiest
way is to just like slap a bunch of
hardware on it and say boom we're going
to get more memory we're going to get
more processors and we'll take care of
it that way but honestly it is safer
better and more reliable to fix it in
the software in the code itself to do
the software fix now sometimes that
means rewriting and so I'm going to give
one more ex uh one more thing that I've
run into the past before I toss it back
over to that guy I didn't name earlier
and that is sometimes you don't want you
want to change and this goes into like
being a better developer you want to
change your approach or your format your
what you're doing completely the
framework it could even be the
foundation of what you're doing and a
good example of this for me is I was
working for a company many years ago and
they were doing these very large a files
which is basically just a lot of fixed
length data that's coming across we're
talking hundreds of thousands of records
and I was hitting this problem is that
we I originally had like a nice little I
think it was a Java application that
would just pull this thing in spin
through it really fast and kick it back
out and did what it needed oh storing it
out to a database it just stored it out
to the database cool and it was doing
all the manipulations and everything
well once it got to a certain
size what happened is it was sucking too
much of the file in and then it ran out
of memory and then we started running
into problems now I switched it first
thing I did I was like okay let's try to
do this differently so it was like
pulling a line at a time I was like okay
we're just going to process We're Not
Gon to have the whole file in we're just
G have a pointer we're going to move
through still ended up with problems
because what we were having to do is
actually read data and then refer to
other data and so there was a lot of
relational things that we had to deal
with within the data so that to me was a
clue that what if we try a relational
database
and what I ended up finding out was that
if I use and this was just using uh my
SQL I think it was my squel may have
been postgress it was one of those where
I used their import their it was a it
was a store procedure that allowed you
to read from a file and so what I did is
I would read from the file create a
create a temporary table shove all that
stuff in the table do a bunch of
manipulations with it and then peel it
back out I think I was even creating
indexes on the fly so I would pull it in
index certain things
basically do a big You Know M some Mass
updates and then rep it back out and I
went from hours to less than 30 seconds
to run through this stuff and it was
because the tool that I had originally
thought I was going to use and so and
worked fine to a certain extent and then
it went it was like it was limited after
that ain't going to work and I I ended
up rewriting it it went a lot better and
that is why more of the story is make
sure that you don't get stuck in a
single technology a single solution or
just insist that whatever you're writing
is the way to do it until you've you
know you have verified that so that's my
little nugget for the day let's toss
this back over to
you so I'm going to take it a different
direction so you talked more single
system based you know talking about
writing your code doing the database
side of things moving it to the data
moving the file import to the data
datase so it it's kind of a paradigm
shift or moving the load essentially to
a different
location now in today's world we hear a
lot about distributive systems we hear a
lot about micro uh applications
Microsystems and as we break things down
smaller into their smaller components
and we break these out into different
applications apis deployments um you
know S3 with Amazon you know message
cues you know there's a whole lot of
different systems out there that we have
to integrate with and it can actually
mask a lot of the problems that we
potentially have either with our code
with the configurations we have that for
these systems that are
deployed and it's rather interesting so
it this I'm going
to tell you a story of a situation I was
in a couple years ago and the
application I inherited uh when I became
a manager of this team I found out that
we had essentially our core application
was on fire it literally went down once
a day or two to three times a day this
is a health care application this is
actually pulling Healthcare information
in from uh external systems into our
system so this could actually impact
patients lives so we needed to come up
with a way one we had to figure out why
it was going down so much and the very
first thing we had to look at was is it
our code well the system that was
crashing was called Rap City and we
found out that it it's essentially just
a message CU that's pulling information
in and passing it over to us yes it did
some things on its side uh but it was
essentially being disused as a pass
through so as we were looking at that we
literally had just taken out all of the
logic and just moved message in message
out so we would just pass it over it
still kept
crashing so we literally just moved it
down to one process so it did one thing
we still were essentially going down
because of volume so the other thing we
looked at was okay what is our Network
threshold are we just essentially having
too much information coming in and it's
a big pipe over here and a small pipe
over here and that was part of it so we
did uh in increase the output flow
through the network so we had more um
data going out than we did initially and
then we essentially found out that we
were running on a machine that had two
core processes maybe four gigs of RAM
and it was nowhere near enough to handle
the amount of data that we were
consuming the other thing that no one
happened to think about was how much
disc space we had because of the large
volume that was coming in we were
actually running out a disk space on top
of the memory issues on top of the CPU
issues so it's not always your
applications it could essentially be
that the system that this was built for
has aged out to the point that you need
to upgrade the hardware that's the
backbone of the system so that was one
story The Second Story is now in a more
distributed environment with AWS we have
lots of applications all over the place
and with the microsystem approach you
typically want to rate your application
to be very small to consume a very small
amount of resources so it'll run on a
very small build or um
AWS uh like ec2 instance or Windows
instance something like that so the
smaller instance that you can deployed
to the less money it's going to cost to
keep that application up and running the
problem with that attent is you could
scale very small your application works
initially but then again you could run
into that situation like we did with the
rap city system where you're the load of
the application or the application is
essentially outgrown it's Hardware it's
outgrown the container that it's being
deployed
to recently I ran into a situation where
we to address this they were essentially
just kicking up the memory of our uh
container thinking that oh we needed
more memory to keep working you know to
keep things happy well at 16 gigs we're
still going down for one particular
transaction that goes
through doing some
research like uh we've talked about
before so this is one of those
situations where the customer does one
thing it breaks we as developers do
something the same thing literally the
same thing and it works so now it's like
okay what is different between the two
environments well as developers we
typically have these supercomputers or
very beefy machines that can do just
about anything in that we wanted to do
but again we're deploying to containers
that are meant to be very small take up
a l you know a lot less resources so
they cost less
money ideally what I found out was we
underpowered our container to the point
that we were doing so many transactions
so much work behind the scenes that we
essentially just see more CPU not more
memory so this is one of those
situations where you need to understand
what the CPU is used for versus the
amount of memory yes we had a lot of uh
data in memory but we're doing a lot of
transactions with that so you actually
need more CPU power than memory because
if you try to keep all that in memory
you're going to get an imbalance and
things are going to
crash however the potentially you could
still have a memory leak so what you're
going to want to do is you're going to
want to get things like profilers to to
go monitor your code as they're running
and then look for things like Heap Dums
or objects in memory or memory spikes
when you're doing transactions within
your code so you can not just check the
hardware and the physical Hardware but
you still can check your code to make
sure that you don't have runaway code
you don't have like an infinite Loop
that's just eating up memory and you
never know what
happens so pass back I'll give you two
little quick ones too that on top both
that you reminded me as you going
through this one is hard drive space it
used to be like I don't know in the back
in the day back in the old days you used
to always find codes somewhere along the
way that was saying hey is there enough
hard drive space like I remember going
to like this is back like the old 386
dos days where like every little app
that you did one of the first things it
would do would be do you have enough
hard drive space on it and that still
exists but not to the same level in
particular server applications there's a
lot of times that they don't think about
it because they don't care they're like
hey I'm just going to run but things
like log files in particular can two
things actually log files I've s several
times that those got huge and those
sucked up the all of the data for a I
actually have a little miniature same
thing it had a little VM app or little
you know small ec2 app and it blew up
because the law got too
big and it and it crashed the whole
system and I've also seen it when you've
got upload files if you have the ability
for users to upload stuff if they can
upload huge images or videos or if
that's part of it then you need to make
sure that that is a
constant check that you have in some way
form or fashion to make sure that you
have enough space if You' start actually
if you store locally but also if you
temporarily store it locally to make
sure that those are getting cleaned out
because I've seen some situations where
it's it's using S3 or something like
that so it's moving stuff off but
sometimes it doesn't move it off fast
enough or clean enough and then the next
thing you know you've blown that
up the other thing that you can run into
I just forgot but it's so it's log
files
and I now forgot the other one that have
run
into oh it's I'm sorry the other one is
the debugger itself I have run into
situations where if you put something
into debug
mode then it causes issues it in itself
because it's eating up extra processing
and extra time and so I have actually
seen situations where it goes in the
application goes into production and it
breaks because somebody left the debug
flag on or it's not useful because
they've got it in debug mode so when you
push it to
production clean it up you know don't
like shut that stuff off like make sure
that you've got your settings such that
it's not like just dumping a whole bunch
of debug information to a drive
somewhere if you don't need it so don't
be afraid to like have some Flags or
settings or something so that it only
does that when you want it to which is
basically you as a developer you want to
see all that stuff nobody else does
uh final thoughts on
that yeah the other things we kind of
talked about them but we didn't really
address the resource things but in this
distributed world we live in we have
things like databases we have inmemory
databases we have message cues we have
S3 for external data
sources as developers as we write our
applications we're unfortunately we're
side kind of silad we kind of see our
picture our problem that we're solving
and we know we have these resources we
use these resources but we don't
consciously take into account what is my
code change going to do to the rest of
the ecosystem so typically you kind of
throw it over the wall and let someone
else deal with it and you don't think
about it but sometimes you really need
to do you need to understand how that
ecosystem works so that as you're
writing your code if something does
happen you have a better idea where to
go within the structure to debug the
problem also you don't want to
necessarily fill up someone's database
by leaving those log files on and have
you know make someone else's day
miserable because they're not
necessarily in your department you know
you don't want to really make your sis
up guys
mad so one other bonus that I just
thought of while you were going through
that is don't assume unless you can
don't assume that you are always your
app is always going to run in a
connected state in particular one of the
things that
early on when I was doing I would do
some development stuff and I would have
some things break periodically and I'd
be like wow like that website I'm
bringing up looks horrible and what it
was is because I was just instead of
having the some of those like say like
CSS files and things like that that we
like to grab local they were on some you
know CDN somewhere else and they were
pulling it down and so if I wasn't
connected nor were those and so you
would suddenly have like keepy you know
you'll have Library and stuff like that
that they're not available because
you're normally pulling those off the
internet this also and this even if it's
if you're a web application so you
assume that the only P way people are
going to get to it is if you know the
internet exists and it's
working don't be afraid to pull and I
would say don't be afraid but just like
the smart person the wise man pulls
those things locally because if that CDN
goes down or that site that you're
borrowing that from or you're connecting
to or that you're relying on goes down
you are down so the more you can bring
that stuff in and make it you know so if
your sight's up it's up and if it's not
it's not and you have less other places
that you're relying on then you're going
to be it's just going to be that much
better however if you rely on us do that
just like Double Down On It rely on us
check us out like subscribe six
different ways subscribe on your phone
on your laptop on your watch what
whatever it is because we're going to
give you good stuff and if we don't
that's your fault because you didn't
tell us at info@ developer.com or
developer.com fill out a contact form
and let us know what is the good stuff
that you want us to talk about how can
we help you out because we're happy to
help help you out this way to just like
as you can see we are an endless well of
full of just mistakes and issues and all
kinds of problems you can run into but
we might have a couple that you know be
more than happy for you to suggest
something so that we can make sure that
it's much more applicable to you yes you
right there that you're the one I'm
talking to not the no not the one next
to you don't look behind you you you are
the one that can help us out and then we
get to talk to you specifically and say
Here's your problem and we may even use
your name if you want us to or we'll use
a really cool fake name if you want us
to do that those are just some of the
services we have to offer that being
said it is time to wrap this one up we
got to go we got to run so do you so go
out there have yourself a great day a
great week and we will talk to you next
time deep
breath cleansing sigh all right uh any
bonus material you want to throw out on
this one I not necessarily bonus but
just kind of like a historical thing
so I just read an article that I guess
Doss is now open source so you can now
go get Doss and reinstalled again and
play around with
it I remember back in the day I mean
we're talking true Doss Dr Doss before
Windows 31 if you wanted to install a
mouse you had to configure your system
to move the mouse into upper memory you
had lower memory you actually had to
figure out okay how many bits how many
btes do I need to put this here and
there what do I need so for those of you
that are maybe the last decade in
software you didn't have to deal with
this these are we literally had to look
and count how many bits and bites we
could actually put in certain places in
memory now today it's a lot easier
however still with jvms with you know
compilers things like that you still
need to be conscious of this maybe not
as bad but still you know Hardware is
expensive even though AWS makes life a
lot simpler with the resource uh sharing
it does just be mindful that you know if
you have something that can probably run
on you know half a you know half a CPU
let maybe a 100 Megs of memory run it
that way you know don't you know don't
kill yourself um but just be conscious
of that you know don't spin up something
that's 8 gigs 8 core for something that
you don't
need yeah um that's funny because I
remember we had a had a PC Junior
and got a we're going to get a mouse for
it and the mouse came with a memory
stamp baning card so that you could jack
it up to 128 K not mag but K of ram
because it needed it it had that you had
to have that to be able to load the
driers so they're like you know what
we're just going to bundle the memory
with it and I can't remember I want to
say that we ended up got it and just
used it for the memory as opposed to the
mouse the mouse was there but like
almost nothing used the mouse cuz it was
on command yeah yeah I like I think I
had like one like it was PC Magazine or
something like that I don't remember if
that was in one that like had code
periodically so there was a basic app
that I that like here's a code it was
like 500 lines of code to do a little
like game of some sort and it included
code to access the the mouse and be able
to do some stuff so that was the first
thing where I was like I remember like
sitting there and copying it off cuz you
couldn't copy and paste I had to like
sit there and type all of it reading
from from the thing run the whole thing
and I was like oh look I'm using a mouse
and now it's you know gone way beyond
that but I mean even before that my
first computer was the Commodore Vic 20
so if we wanted memory we had to buy a
64k cartridge that plugged into the back
of our keyboard and we had the tape
drive so the only way you could load uh
stuff into memory was to essentially
play it from a cassette tape record it
to a cassette tape and play it that way
one last thing in the video game realm
uh when the Sega Saturn came out the
Sega Saturn came out with actual memory
cards that you had actually you could
plug into the systems um memory card so
that you could actually play larger
games so they kind of had an external
Jack that you could uh plug in extra
memory so you could play more games
without having to open up the system and
install new hardware so that was kind of
a cool little tidbit there and some of
those things haven't changed you still
you know if you get depending on what
you get you can get all these little
like memory plug PL and stuff like that
and and storage plugins so they're
always going to find ways to like charge
you a few extra bucks based on your your
stuff oh by the way just totally random
because we mentioned Games bonus stuff
apparently ps5s are readily available
because I was I had to go byy Best Buy
to replace something the other day and
they had a wall just full of them and so
after what has been like two years and
maybe it happened a month or two ago but
I know for the longest time you couldn't
find a PS5 hardly anywhere it's like if
you found one you need to buy it right
there because like you weren't going to
see one again and I know that's I think
that's how you got one I both my kids
that have them that's what they did they
were just like one was in a Walmart and
it was like just happened to fin it's
like well I wasn't going to buy a PS5
today but I saw it had to buy it uh but
now apparently they're readily available
so if that's something you want go out
and grab yourself one before they become
scarce again which probably will don't
have to worry until they come out with
the ps6 in which case that'll take you
three or four years to be able to get
your first one of those it may take you
five or six years to afford it cuz um it
sounds like the price Tag's going to be
up there with a high-end gaming rig yeah
that's that's what they keep doing it's
like you know because I think the I
think the PS2 when it originally came
out was like 150 bucks or something like
maybe1 199 and then the PS4 when it came
out I think was like 299 the PS5 I think
was like five or six the low version
it's like they it's like everybody else
they're gonna like they're gonna throw
some extra stuff but they're going to
charge you more we will not we're going
to charge you the same amount every time
you come here we are going to wrap this
one up because we're going to write in
the sunset or something like that and go
have a good time as always feel free to
like leave us notes in the comments uh
check out the show notes those are like
we you know got some AI stuff and stuff
like that we usually get really cool
show notes lots of links there check us
out here so you can check that out uh on
the site at developer.com or you can
just leave some comments here and say
hey dudes can you send me something and
we'll shoot you an email of that episode
stuff or we may throw other stuff at you
it depends because we do have things
floating around that we could send to
you just a little teaser there no
promises but you never know you never
know what might end up in your mail if
you send us something because we are
very appreciative for you guys out there
listening to us and you know being there
and putting up with our ugly mugs for
the last little bit of time that you did
that being said we're not going to go
get facial surgery so you're still going
to be able to see our same ugly mugs
next time around and you guys have a
great day and we will talk to you then
take care
everyone uh
wh that is not the button I there we go
stop recording I freaked out
[Music]
Transcript Segments
1.35

[Music]

28.16

all right uh let's talk about what do we

30.759

want to do for this episode you have a I

32.96

was like we just came up out of nowhere

35.28

last time so did you have have you had

37.28

any thoughts today of or this week of

39.92

like what would be a good episode

45.48

topic well the kind of to go with the

49.44

example you were giving previously

52.359

before we recorded last time uh have you

56.359

run into situations where not only is

59.399

your code your issue but the hardware is

62.039

the issue like you're not actually

63.68

configuring the hardware for your

68.2

application oh gez I've been like that

70.439

was yesterday I was that was half my day

74.08

it wasn't it was it's frustrating so

77.119

it's I've put together um Apache front

81.6

ends to Jango now a couple of times and

85.96

it's it's always a little bit different

88.4

because it like if you do a dbn

92.32

versus um and I forget what the core

95.56

like but yeah if you use like aptg

97.479

versus yum versus Mac and I haven't even

100.479

tried Windows nor do I you know I did

102.52

like decades back and don't want to ever

104.719

touch it again they're all different

107.399

they're all like close but not

110.28

quite and so you have to like go find

112.479

the right library and the right download

114.119

and the right all this stuff and I was

116.64

working on a machine that I thought was

118.079

one I got and realized oh wait it's

120.039

different but it's like it's a whole

122.2

thing that I'm setting up because I'm

123.399

setting up Apache which I got that you

125.399

know that was fine but I'm setting up

127.039

Apache I'm setting up SSH or you know

129.399

SSL stuff I'm using um certbot so I'm

133.08

grabbing aert and I'm trying to make

134.48

sure that I've got like that hooked up

136

so that it grabs the right certific

138.64

certificate and then making sure I've

140.519

got the files in the right place that it

142.08

handles you know Port you know HTTP as

144.599

well as https and it's connecting to

146.8

everything and that it's all routing

148.92

right and it's just a bunch of stuff

150.76

that's like very sensitive got to be the

153.56

right got to be the exact right name

155.28

names have got to match right and

156.48

everything and I wasn't able to do it in

158.48

one

159.2

sitting I got something done got pulled

161.64

away got something else done pulled away

163.84

and so it just took forever because

165.2

every time I sat down I had to one had

168.159

to remember where I was at and then two

170.2

had to like go track this down and

172.239

there's you know three or four different

173.56

log files I'm looking for

175.4

so short story long yes I have

178.76

definitely run into that couple of times

181.48

configuration

182.92

is if I could never if I could do

185.4

software and never configure anything

187.319

ever again in my life even my own

189.959

software I think I would be happy

191.44

because I I get bit by my own

193.76

configuration settings on a regular

196

basis where it's like I built it like an

197.92

app I was testing today I was like why

199.959

is that not working it's like oh yeah

201.48

we've got this configuration setting and

203.08

we didn't set it right it's like we

204.4

adjusted it all right let's set it back

206.56

and then boom everything works again and

208.599

seeing on the flip side of that with the

210.28

testing stuff when you want to do like

213.08

low testing or high-end regression

216

testing you need to run your test

218.12

multi-threaded well in order to do that

220

you need a certain amount of CPU and you

222.12

need a certain amount of ram in order to

225.12

be able to keep up with that load it's

227.72

not necessarily always about memory

229.48

leaks or things like that if you are

231.519

trying to push high volume you need to

233.4

have the resources to push it and that

235.92

was one of these things I was running

237.959

down today is

240.76

we tested all of our production code in

243.879

QA at high volume for generating

247.799

reports and I had it all working

251.4

everything worked for every use case we

254.799

had now it's in production and what I

258

didn't know is that we've been kicking

259.759

up the memory in

261.24

production uh we're up to like 16 gigs

264.68

and we're still running out of memory

266.199

with our largest uh with one of our

269.44

largest vendors that has over 2 million

272.759

records come to find out we're only

276

running for uh four core processor with

279.479

16 gigs of RAM so I actually uh one of

283.12

the cool things about spring boot uh and

285.479

Eclipse you can actually configure your

287.44

profiler for the specific cores and

290.199

memory you want to run so I've got a 12

292.96

core laptop with 64 gigs of RAM so I can

295.6

actually do pretty much whatever I want

297.44

locally without even needing to do

299.72

containers or I could stand up a

301.479

container whatever but what essentially

304.639

I ran into is I clicked run locally I

306.88

mean I'm running all of our applications

309.16

locally we all microservices distributed

311.639

but I have it all

313.08

locally clicked run ran

316.8

fine use case worked report guide

320.16

generated no problem I'm like all right

321.88

what the hell's wrong and then I started

324.56

to getting into it I'm like well let me

326.319

go look at

327.6

production finally got uh back into

330.639

production because I found out AWS for

332.479

some reason defaulted to a different

334.44

region for

335.759

me so while I'm able to log in I had not

339.6

like I had no servers I had no I'm like

341.72

where the hell is all my

343.28

stuff that was a fun hour uh I have lost

347.68

that hour myself a couple of times

350.72

anyway once I got in there though I

352.28

walked into the deployment and looked at

354.44

the setup and found out okay we've got

356.16

four cores and 16 Ram so I went into to

359.84

jvm and I configured it to 4 gigs of RAM

363.6

and it was cool cuz I found out with the

365.919

jvm you could um with any Clips you can

369.12

do like available chords or available

371.319

system I I forget the exact name it's

373.919

not the same as you would for Maven it's

376.68

a

377.599

different uh property uh VM but you can

381.68

actually set the ches so then I used uh

384.599

visual

386.039

VM uh yeah visual VM to uh Prof the

389.919

memory cuz we used that before in the

391.72

past it's a cool little free profiler

395.12

and I spun it up I watched things go and

399.199

before I had this nice flat line um with

402.88

my uh 12 cores with all my memory it

405.199

barely got above uh like four gigs of

409.16

RAM under four cores and 16 gigs of RAM

414.4

or I did I did four cores and I did four

417.96

gigs of RAM to start with so I did four

420.08

for and immediately I watched the spike

423.599

go from this to this and within a like

427.919

30 seconds I'm out of memory so I'm like

431

okay that was interesting so then I

432.52

flipped it back to 12 core still ran out

435.28

of memory but I was able to get it up to

439.24

um six gigs of RAM and it worked so I

444.599

slowly then started rolling back the

446.68

processors and essentially I got it down

448.8

to eight and eight so if you have eight

450.56

cores with 8 gigs of RAM it worked now

453.96

my boss is like well we need to go look

455.84

at the code we have a memory leag I

458.68

don't think so I think trying to process

461.24

two million records in memory at once

463.759

because we're sorting them we're doing

466.759

manipulations with that's a lot of

468.479

processing power it's not just memory

470.12

we're trying to do stuff with that if we

472.159

were just holding it in memory sure I

474.56

would say we have a memory leak but no

476.72

we're constantly like resorting the list

479.599

for um building uh spreadsheets with

483.199

this I don't think it's necessarily A

486.24

Memory leak as it's a processor Le we

488.36

need more processing power anyway this

491.52

was kind of this discussion I thought

493.44

would be good for the

495.84

podcast which we're now like way I think

498.24

so I think we

500.84

can well we're gonna yeah I think we can

503.12

go into some of this stuff is just some

504.639

like

506.599

common odity type things uh that we've

510.12

that we've run into and uh I've got some

512.64

similar types of experiences that I can

514.32

talk about a little bit and sort of like

516.039

what uh some of it's like thresholds and

518.36

stuff like

519.599

that and um sort of tackling those a

522.519

little bit and then I'll toss it over to

523.839

you and then you can you can probably

526

rehash some of that so those you guys

527.32

that are listening sorry you may hear

528.64

this twice but we don't remember what we

531.16

said so it'll be a little bit different

532.519

anyway so there may be something more

533.839

ingenious on that second side around

536.44

well hello and welcome back we are I'm

539

sorry I didn't mean to smack too close

540.44

to the mic but that happens sometimes uh

543.279

we are back with building better

544.68

developers yet another episode where

546.519

we're talking about stuff that we've run

548.24

into uh this episode we're going to go a

550.959

little bit uh some of it's very current

553.24

for Michael some of it's not so much for

555.12

me but some of the things that I've hit

556.44

in the past and it's really it gets into

560.399

like debugging stuff when you get away

562.279

from it's like not just code it's

564.04

hardware issues and and it literally is

566.76

like memory and resource issues and

568.56

things like that which

570.16

you may not run into as much depending

572.72

on what your your applications are

574.72

because these days there's a lot of

576.2

stuff that is very much plays well

578.72

within the the Norms but if you start

581.32

doing big data if you start doing data

583.399

processing if you've got a an app that's

586.24

got a lot of users or a lot of

587.76

processing a lot of data any of that a

589.88

lot of stuff going on things can happen

593.44

so we're going to talk about that first

595.68

I'm going to introduce myself my name is

597

Rob briad I'm one of the founders of

599.399

developer nor I'm also a founder of RB

601.959

Consulting all things about simplifying

604.8

integrating and and just automating your

608.24

software your solutions to make your

610.399

business run smoother on the other side

613.36

this time I'm not even going to mention

614.76

his name I'm going to let them do the

616.24

whole introduction so go for it sir hey

619.64

everyone my name is Michael MOS I'm

621.16

another co-founder of developer erer I'm

623.44

also the founder of Envision QA where we

625.839

build custom software for small to

627.839

midsize businesses and and Healthcare

630.24

clinicians we also uh offered consulting

633.48

services and testing services for your

637.16

needs awesome now we're get right into

641

one of those cases where you may need

642.36

some of those testing

643.88

services in particularly it is with like

647.839

the the big Behemoth of stuff that we

650.6

can sometimes run into in the software

652.839

world is the hardware stuff the

655.76

resources that are related to like

657.92

processing and the ones related to well

660.32

and storage and memory now the fun ones

663.76

are when storage and memory cross lines

668.24

and this is where you see some really

670.959

neat stuff and these are the kinds of

674.36

things that it's just we're going to

675.399

give you some like they're not I don't

676.88

know if they're necessarily red flags

678.079

but there's some some things that are

679.76

maybe some notes that if you are in a

682.639

testing situation or in a situation

685.079

where hey it works on my machine or

688.279

usually and now it doesn't and the code

691.48

should be the same when you get a sense

693.2

like that doesn't make sense it's it's

695.399

not a different system here's some

698.36

things to go look at maybe there's

700.72

something going on that's a little

702

deeper than that where you're going to

703.2

want to have some sort of you know some

705.279

more detailed logging particularly from

708.44

the the operating system side the

711.56

platform side to see what is actually

715.04

going on now one of the like the mother

719.399

of all break your application things

722.2

that happens is when you trigger when

725.56

you when you get past physical memory

730.079

and it starts to cash stuff because what

733.04

happens is now you've got physical

734.48

memory that's just like really it's like

736.519

boom boom boom boom boom boom boom no

738.279

it's gotten better because SSD drives

740.24

and some of that stuff is very fast and

742.12

and there's a lot of intelligence buil

745.079

around those things often so that you

746.92

can so it's going to keep the uh the

749.56

most recent data like up front so if you

751.8

have something that if you're caching

754

your memory the system memory into like

757.48

a SSD drive or really high-speed drive

759.72

you may not notice the difference as

761.76

much but definitely if you're old school

765.16

and you go to one of the old you know uh

768.279

dis driven type hard drives this the

772.36

processing speed goes from microc

774.959

seconds to multisecond

777.279

and it literally I don't know how many

779.76

times it was stuff where it's like you

781.04

know I've just pick a number like we

783

have let's say we have two gig of of

784.6

memory on our machine and everything was

787.279

awesome until we had it grow Beyond two

790.36

gig because as soon as it did that it

792.32

had to C it had to swap stuff out and

794.92

now you have the swap cost so now it's

797.76

it's pushing stuff out to that hard

799.36

drive and then having to pull stuff back

801.399

and that's adding some processing time

803.92

and then it just sort of like feeds on

806.399

top of itself now in the real world

809.8

what you will often see is that that

812.399

really spirals out of control because

815.56

what you have is you have people are

817.079

pulling reports and they're expecting

818.48

reports to come let's say you know pick

820.36

a note every 10 seconds they should be

822.56

able to get the report back and see

824.04

their data and what happens is people

826.079

get impatient so when it triggers that

828.32

and so now it goes from 10 to say 15

830.519

seconds people hit report again and so

833.36

there are times that have S like if if

834.92

you don't have intelligence around that

836.959

then it'll be you know I just created

839.68

eight instances of trying to run this

841.399

report that is doing table scans and if

843.92

you don't know what that is go look it

846.36

up and realize that's something that you

847.6

should never do but maybe doing table

850.12

scans and the next thing you know it's

851.36

say your time has gone from 10 seconds

854.44

to 10

855.6

hours and it like everybody freaks out

859.04

everybody panics for somewhat I mean

861.639

it's somewhat reasonable but it's also

863.199

it's like hey just chill if we all we

865.44

need to do is make sure that we don't

868

hit that limit or or sometimes it's just

870.04

a matter of like hey let's clean up our

871.68

queries a little bit or what we cash or

873.68

what we store now sometimes that's

875.959

harder than others sometimes the easiest

878.12

way is to just like slap a bunch of

879.68

hardware on it and say boom we're going

881.16

to get more memory we're going to get

882.199

more processors and we'll take care of

883.6

it that way but honestly it is safer

886.959

better and more reliable to fix it in

890.079

the software in the code itself to do

892.12

the software fix now sometimes that

895.48

means rewriting and so I'm going to give

897.04

one more ex uh one more thing that I've

899.759

run into the past before I toss it back

902.68

over to that guy I didn't name earlier

904.88

and that is sometimes you don't want you

908.199

want to change and this goes into like

910.12

being a better developer you want to

911.8

change your approach or your format your

915.44

what you're doing completely the

917.68

framework it could even be the

919.04

foundation of what you're doing and a

920.44

good example of this for me is I was

922.519

working for a company many years ago and

924.12

they were doing these very large a files

927.839

which is basically just a lot of fixed

930

length data that's coming across we're

932.68

talking hundreds of thousands of records

936.279

and I was hitting this problem is that

938.04

we I originally had like a nice little I

941.04

think it was a Java application that

942.36

would just pull this thing in spin

944.16

through it really fast and kick it back

945.839

out and did what it needed oh storing it

947.8

out to a database it just stored it out

949.279

to the database cool and it was doing

951.44

all the manipulations and everything

953.319

well once it got to a certain

955.12

size what happened is it was sucking too

957.759

much of the file in and then it ran out

959.639

of memory and then we started running

961.639

into problems now I switched it first

964.639

thing I did I was like okay let's try to

966.12

do this differently so it was like

967.24

pulling a line at a time I was like okay

969.12

we're just going to process We're Not

970.12

Gon to have the whole file in we're just

971.48

G have a pointer we're going to move

972.56

through still ended up with problems

974.839

because what we were having to do is

976.36

actually read data and then refer to

978.68

other data and so there was a lot of

980.6

relational things that we had to deal

982.56

with within the data so that to me was a

985.279

clue that what if we try a relational

987.639

database

989.48

and what I ended up finding out was that

992.079

if I use and this was just using uh my

994.56

SQL I think it was my squel may have

996.12

been postgress it was one of those where

998.36

I used their import their it was a it

1001.639

was a store procedure that allowed you

1003.48

to read from a file and so what I did is

1006.48

I would read from the file create a

1008.72

create a temporary table shove all that

1010.36

stuff in the table do a bunch of

1012.279

manipulations with it and then peel it

1014.079

back out I think I was even creating

1015.639

indexes on the fly so I would pull it in

1017.88

index certain things

1020.399

basically do a big You Know M some Mass

1022.44

updates and then rep it back out and I

1025.039

went from hours to less than 30 seconds

1029.079

to run through this stuff and it was

1031.039

because the tool that I had originally

1033.199

thought I was going to use and so and

1035.559

worked fine to a certain extent and then

1037.839

it went it was like it was limited after

1040.199

that ain't going to work and I I ended

1042.559

up rewriting it it went a lot better and

1044.839

that is why more of the story is make

1047.76

sure that you don't get stuck in a

1049.84

single technology a single solution or

1053.12

just insist that whatever you're writing

1055.2

is the way to do it until you've you

1057.799

know you have verified that so that's my

1060.24

little nugget for the day let's toss

1062.44

this back over to

1064

you so I'm going to take it a different

1066.44

direction so you talked more single

1069.2

system based you know talking about

1071.039

writing your code doing the database

1072.88

side of things moving it to the data

1076.96

moving the file import to the data

1078.799

datase so it it's kind of a paradigm

1081.679

shift or moving the load essentially to

1085.24

a different

1086.76

location now in today's world we hear a

1090.32

lot about distributive systems we hear a

1092.919

lot about micro uh applications

1096.559

Microsystems and as we break things down

1099.08

smaller into their smaller components

1101.4

and we break these out into different

1103.12

applications apis deployments um you

1106.679

know S3 with Amazon you know message

1109.6

cues you know there's a whole lot of

1112.159

different systems out there that we have

1113.919

to integrate with and it can actually

1116.159

mask a lot of the problems that we

1118.64

potentially have either with our code

1121.48

with the configurations we have that for

1123.48

these systems that are

1125.039

deployed and it's rather interesting so

1127.96

it this I'm going

1130.2

to tell you a story of a situation I was

1135.24

in a couple years ago and the

1139.48

application I inherited uh when I became

1142.12

a manager of this team I found out that

1145.039

we had essentially our core application

1148.44

was on fire it literally went down once

1153.039

a day or two to three times a day this

1155.64

is a health care application this is

1157.48

actually pulling Healthcare information

1159.159

in from uh external systems into our

1162.08

system so this could actually impact

1164

patients lives so we needed to come up

1167.4

with a way one we had to figure out why

1170.08

it was going down so much and the very

1173.28

first thing we had to look at was is it

1177.12

our code well the system that was

1180.36

crashing was called Rap City and we

1183.32

found out that it it's essentially just

1185.039

a message CU that's pulling information

1186.919

in and passing it over to us yes it did

1189.32

some things on its side uh but it was

1192

essentially being disused as a pass

1194.08

through so as we were looking at that we

1196.52

literally had just taken out all of the

1199.6

logic and just moved message in message

1202.28

out so we would just pass it over it

1204.36

still kept

1205.88

crashing so we literally just moved it

1208.559

down to one process so it did one thing

1211.039

we still were essentially going down

1213.88

because of volume so the other thing we

1215.84

looked at was okay what is our Network

1218.44

threshold are we just essentially having

1220.679

too much information coming in and it's

1222.96

a big pipe over here and a small pipe

1224.64

over here and that was part of it so we

1227.36

did uh in increase the output flow

1230.28

through the network so we had more um

1232.679

data going out than we did initially and

1236.039

then we essentially found out that we

1237.6

were running on a machine that had two

1241.52

core processes maybe four gigs of RAM

1244.72

and it was nowhere near enough to handle

1246.36

the amount of data that we were

1248.36

consuming the other thing that no one

1251.12

happened to think about was how much

1252.96

disc space we had because of the large

1255.76

volume that was coming in we were

1257.679

actually running out a disk space on top

1260.08

of the memory issues on top of the CPU

1262.799

issues so it's not always your

1266.44

applications it could essentially be

1269.4

that the system that this was built for

1272.559

has aged out to the point that you need

1274.84

to upgrade the hardware that's the

1277.039

backbone of the system so that was one

1281.039

story The Second Story is now in a more

1285.679

distributed environment with AWS we have

1288.919

lots of applications all over the place

1291.799

and with the microsystem approach you

1294.159

typically want to rate your application

1296.039

to be very small to consume a very small

1298.279

amount of resources so it'll run on a

1300.72

very small build or um

1304.559

AWS uh like ec2 instance or Windows

1307.84

instance something like that so the

1310.4

smaller instance that you can deployed

1312.84

to the less money it's going to cost to

1314.799

keep that application up and running the

1317.279

problem with that attent is you could

1320.32

scale very small your application works

1323.52

initially but then again you could run

1325.559

into that situation like we did with the

1327.559

rap city system where you're the load of

1331.159

the application or the application is

1332.679

essentially outgrown it's Hardware it's

1335.52

outgrown the container that it's being

1336.96

deployed

1339.08

to recently I ran into a situation where

1343.559

we to address this they were essentially

1346.2

just kicking up the memory of our uh

1350

container thinking that oh we needed

1352.279

more memory to keep working you know to

1354.799

keep things happy well at 16 gigs we're

1357.88

still going down for one particular

1359.84

transaction that goes

1361.96

through doing some

1364.64

research like uh we've talked about

1366.799

before so this is one of those

1367.919

situations where the customer does one

1369.88

thing it breaks we as developers do

1372.64

something the same thing literally the

1375.08

same thing and it works so now it's like

1378.2

okay what is different between the two

1380.6

environments well as developers we

1383.12

typically have these supercomputers or

1385

very beefy machines that can do just

1386.88

about anything in that we wanted to do

1389.88

but again we're deploying to containers

1391.919

that are meant to be very small take up

1394.32

a l you know a lot less resources so

1396.279

they cost less

1397.4

money ideally what I found out was we

1401.6

underpowered our container to the point

1403.44

that we were doing so many transactions

1405.159

so much work behind the scenes that we

1407.12

essentially just see more CPU not more

1409.159

memory so this is one of those

1410.64

situations where you need to understand

1412.679

what the CPU is used for versus the

1415.08

amount of memory yes we had a lot of uh

1417.64

data in memory but we're doing a lot of

1419.88

transactions with that so you actually

1421.36

need more CPU power than memory because

1424.08

if you try to keep all that in memory

1426.039

you're going to get an imbalance and

1427.36

things are going to

1428.799

crash however the potentially you could

1431.6

still have a memory leak so what you're

1434.2

going to want to do is you're going to

1435.679

want to get things like profilers to to

1438.559

go monitor your code as they're running

1441.799

and then look for things like Heap Dums

1443.679

or objects in memory or memory spikes

1446.24

when you're doing transactions within

1447.76

your code so you can not just check the

1451.2

hardware and the physical Hardware but

1454.2

you still can check your code to make

1456.76

sure that you don't have runaway code

1458.32

you don't have like an infinite Loop

1460.279

that's just eating up memory and you

1462.2

never know what

1463.919

happens so pass back I'll give you two

1467.84

little quick ones too that on top both

1470.32

that you reminded me as you going

1472.039

through this one is hard drive space it

1475.679

used to be like I don't know in the back

1478.279

in the day back in the old days you used

1480.76

to always find codes somewhere along the

1482.32

way that was saying hey is there enough

1483.919

hard drive space like I remember going

1485.679

to like this is back like the old 386

1489.679

dos days where like every little app

1492.44

that you did one of the first things it

1494.039

would do would be do you have enough

1495.559

hard drive space on it and that still

1498

exists but not to the same level in

1501.12

particular server applications there's a

1503.799

lot of times that they don't think about

1505.919

it because they don't care they're like

1508.36

hey I'm just going to run but things

1510.52

like log files in particular can two

1514.48

things actually log files I've s several

1516.84

times that those got huge and those

1519.48

sucked up the all of the data for a I

1522.12

actually have a little miniature same

1524

thing it had a little VM app or little

1525.96

you know small ec2 app and it blew up

1529

because the law got too

1530.52

big and it and it crashed the whole

1533.08

system and I've also seen it when you've

1535.6

got upload files if you have the ability

1538.12

for users to upload stuff if they can

1540.76

upload huge images or videos or if

1544.72

that's part of it then you need to make

1546.799

sure that that is a

1548.48

constant check that you have in some way

1551.039

form or fashion to make sure that you

1552.52

have enough space if You' start actually

1555.24

if you store locally but also if you

1557.6

temporarily store it locally to make

1560.12

sure that those are getting cleaned out

1561.96

because I've seen some situations where

1563.799

it's it's using S3 or something like

1567.039

that so it's moving stuff off but

1569.96

sometimes it doesn't move it off fast

1571.76

enough or clean enough and then the next

1573.76

thing you know you've blown that

1575.799

up the other thing that you can run into

1579.559

I just forgot but it's so it's log

1583.279

files

1585.48

and I now forgot the other one that have

1588.6

run

1589.799

into oh it's I'm sorry the other one is

1592.159

the debugger itself I have run into

1594.48

situations where if you put something

1596.48

into debug

1597.919

mode then it causes issues it in itself

1601.84

because it's eating up extra processing

1604

and extra time and so I have actually

1606.44

seen situations where it goes in the

1610.039

application goes into production and it

1612.48

breaks because somebody left the debug

1614.76

flag on or it's not useful because

1618.399

they've got it in debug mode so when you

1620.72

push it to

1622.48

production clean it up you know don't

1625.2

like shut that stuff off like make sure

1627.2

that you've got your settings such that

1629.32

it's not like just dumping a whole bunch

1632.48

of debug information to a drive

1634.36

somewhere if you don't need it so don't

1636.6

be afraid to like have some Flags or

1638.279

settings or something so that it only

1641.52

does that when you want it to which is

1643.36

basically you as a developer you want to

1644.799

see all that stuff nobody else does

1648.52

uh final thoughts on

1650.2

that yeah the other things we kind of

1653.44

talked about them but we didn't really

1654.96

address the resource things but in this

1657.799

distributed world we live in we have

1660.32

things like databases we have inmemory

1662.6

databases we have message cues we have

1665.039

S3 for external data

1667.519

sources as developers as we write our

1670

applications we're unfortunately we're

1672.679

side kind of silad we kind of see our

1675

picture our problem that we're solving

1677.559

and we know we have these resources we

1680.279

use these resources but we don't

1683.08

consciously take into account what is my

1686

code change going to do to the rest of

1688.399

the ecosystem so typically you kind of

1691.64

throw it over the wall and let someone

1694.039

else deal with it and you don't think

1696.24

about it but sometimes you really need

1698.32

to do you need to understand how that

1699.84

ecosystem works so that as you're

1701.76

writing your code if something does

1704.399

happen you have a better idea where to

1706.799

go within the structure to debug the

1709.36

problem also you don't want to

1711.72

necessarily fill up someone's database

1713.799

by leaving those log files on and have

1716.12

you know make someone else's day

1717.6

miserable because they're not

1719.08

necessarily in your department you know

1720.48

you don't want to really make your sis

1722.519

up guys

1723.679

mad so one other bonus that I just

1726.08

thought of while you were going through

1727.12

that is don't assume unless you can

1731.48

don't assume that you are always your

1733.399

app is always going to run in a

1734.64

connected state in particular one of the

1737.519

things that

1738.84

early on when I was doing I would do

1741.12

some development stuff and I would have

1742.96

some things break periodically and I'd

1744.679

be like wow like that website I'm

1746.08

bringing up looks horrible and what it

1748.519

was is because I was just instead of

1751.76

having the some of those like say like

1753.799

CSS files and things like that that we

1755.799

like to grab local they were on some you

1759.08

know CDN somewhere else and they were

1761.2

pulling it down and so if I wasn't

1763.12

connected nor were those and so you

1765.72

would suddenly have like keepy you know

1767.24

you'll have Library and stuff like that

1769.679

that they're not available because

1771.399

you're normally pulling those off the

1773.559

internet this also and this even if it's

1777.279

if you're a web application so you

1779.32

assume that the only P way people are

1780.84

going to get to it is if you know the

1783.48

internet exists and it's

1785.279

working don't be afraid to pull and I

1787.919

would say don't be afraid but just like

1789.44

the smart person the wise man pulls

1792.08

those things locally because if that CDN

1795.039

goes down or that site that you're

1796.64

borrowing that from or you're connecting

1798.36

to or that you're relying on goes down

1800.6

you are down so the more you can bring

1802.799

that stuff in and make it you know so if

1805.08

your sight's up it's up and if it's not

1806.96

it's not and you have less other places

1810.64

that you're relying on then you're going

1811.96

to be it's just going to be that much

1814.08

better however if you rely on us do that

1818.76

just like Double Down On It rely on us

1820.88

check us out like subscribe six

1823.399

different ways subscribe on your phone

1825.64

on your laptop on your watch what

1828

whatever it is because we're going to

1829.76

give you good stuff and if we don't

1832.44

that's your fault because you didn't

1833.96

tell us at info@ developer.com or

1836.2

developer.com fill out a contact form

1839.12

and let us know what is the good stuff

1841.88

that you want us to talk about how can

1843.44

we help you out because we're happy to

1845.76

help help you out this way to just like

1848.36

as you can see we are an endless well of

1851.48

full of just mistakes and issues and all

1854.2

kinds of problems you can run into but

1856.84

we might have a couple that you know be

1858.6

more than happy for you to suggest

1860

something so that we can make sure that

1861.88

it's much more applicable to you yes you

1863.84

right there that you're the one I'm

1865.2

talking to not the no not the one next

1867.08

to you don't look behind you you you are

1869.32

the one that can help us out and then we

1871.519

get to talk to you specifically and say

1873.039

Here's your problem and we may even use

1874.88

your name if you want us to or we'll use

1876.44

a really cool fake name if you want us

1877.799

to do that those are just some of the

1879.48

services we have to offer that being

1882.08

said it is time to wrap this one up we

1885.279

got to go we got to run so do you so go

1887.639

out there have yourself a great day a

1889.279

great week and we will talk to you next

1893.519

time deep

1895.48

breath cleansing sigh all right uh any

1899.32

bonus material you want to throw out on

1900.639

this one I not necessarily bonus but

1903.6

just kind of like a historical thing

1906.48

so I just read an article that I guess

1909.399

Doss is now open source so you can now

1912.159

go get Doss and reinstalled again and

1914.88

play around with

1916.84

it I remember back in the day I mean

1920.44

we're talking true Doss Dr Doss before

1925.039

Windows 31 if you wanted to install a

1928.2

mouse you had to configure your system

1931.84

to move the mouse into upper memory you

1933.919

had lower memory you actually had to

1935.799

figure out okay how many bits how many

1938.6

btes do I need to put this here and

1941.36

there what do I need so for those of you

1944.08

that are maybe the last decade in

1946.919

software you didn't have to deal with

1948.6

this these are we literally had to look

1951.639

and count how many bits and bites we

1954.44

could actually put in certain places in

1956.44

memory now today it's a lot easier

1959.2

however still with jvms with you know

1962.039

compilers things like that you still

1963.6

need to be conscious of this maybe not

1965.919

as bad but still you know Hardware is

1970.279

expensive even though AWS makes life a

1973.24

lot simpler with the resource uh sharing

1976.2

it does just be mindful that you know if

1979.399

you have something that can probably run

1982.159

on you know half a you know half a CPU

1986.24

let maybe a 100 Megs of memory run it

1990.08

that way you know don't you know don't

1992.919

kill yourself um but just be conscious

1996.039

of that you know don't spin up something

1997.96

that's 8 gigs 8 core for something that

2000.639

you don't

2002.36

need yeah um that's funny because I

2005.639

remember we had a had a PC Junior

2008.44

and got a we're going to get a mouse for

2010.96

it and the mouse came with a memory

2013.519

stamp baning card so that you could jack

2015.559

it up to 128 K not mag but K of ram

2022.12

because it needed it it had that you had

2024.12

to have that to be able to load the

2025.799

driers so they're like you know what

2026.919

we're just going to bundle the memory

2029.24

with it and I can't remember I want to

2031.519

say that we ended up got it and just

2034

used it for the memory as opposed to the

2036.2

mouse the mouse was there but like

2037.84

almost nothing used the mouse cuz it was

2040.36

on command yeah yeah I like I think I

2042.24

had like one like it was PC Magazine or

2044.76

something like that I don't remember if

2045.919

that was in one that like had code

2047.76

periodically so there was a basic app

2050.24

that I that like here's a code it was

2051.919

like 500 lines of code to do a little

2053.96

like game of some sort and it included

2057.159

code to access the the mouse and be able

2059.879

to do some stuff so that was the first

2061.359

thing where I was like I remember like

2062.919

sitting there and copying it off cuz you

2064.679

couldn't copy and paste I had to like

2066.119

sit there and type all of it reading

2067.52

from from the thing run the whole thing

2069.44

and I was like oh look I'm using a mouse

2071.359

and now it's you know gone way beyond

2073.76

that but I mean even before that my

2075.879

first computer was the Commodore Vic 20

2077.76

so if we wanted memory we had to buy a

2079.599

64k cartridge that plugged into the back

2082.399

of our keyboard and we had the tape

2084.399

drive so the only way you could load uh

2086.919

stuff into memory was to essentially

2089.079

play it from a cassette tape record it

2091.359

to a cassette tape and play it that way

2093.96

one last thing in the video game realm

2097.44

uh when the Sega Saturn came out the

2099.359

Sega Saturn came out with actual memory

2101.32

cards that you had actually you could

2103.4

plug into the systems um memory card so

2107.28

that you could actually play larger

2108.96

games so they kind of had an external

2111.599

Jack that you could uh plug in extra

2115.2

memory so you could play more games

2117.079

without having to open up the system and

2118.96

install new hardware so that was kind of

2120.599

a cool little tidbit there and some of

2122.96

those things haven't changed you still

2124.359

you know if you get depending on what

2125.64

you get you can get all these little

2126.96

like memory plug PL and stuff like that

2128.839

and and storage plugins so they're

2131.32

always going to find ways to like charge

2132.8

you a few extra bucks based on your your

2135.04

stuff oh by the way just totally random

2137.88

because we mentioned Games bonus stuff

2140.359

apparently ps5s are readily available

2142.64

because I was I had to go byy Best Buy

2144.119

to replace something the other day and

2145.8

they had a wall just full of them and so

2148.68

after what has been like two years and

2151

maybe it happened a month or two ago but

2152.88

I know for the longest time you couldn't

2154.359

find a PS5 hardly anywhere it's like if

2156.16

you found one you need to buy it right

2157.76

there because like you weren't going to

2158.96

see one again and I know that's I think

2160.839

that's how you got one I both my kids

2162.92

that have them that's what they did they

2164.079

were just like one was in a Walmart and

2166.52

it was like just happened to fin it's

2167.68

like well I wasn't going to buy a PS5

2169.079

today but I saw it had to buy it uh but

2171.359

now apparently they're readily available

2172.76

so if that's something you want go out

2174.44

and grab yourself one before they become

2175.96

scarce again which probably will don't

2178.599

have to worry until they come out with

2179.52

the ps6 in which case that'll take you

2181.2

three or four years to be able to get

2182.52

your first one of those it may take you

2184.599

five or six years to afford it cuz um it

2187.64

sounds like the price Tag's going to be

2189.119

up there with a high-end gaming rig yeah

2192.56

that's that's what they keep doing it's

2193.96

like you know because I think the I

2195.4

think the PS2 when it originally came

2196.88

out was like 150 bucks or something like

2198.96

maybe1 199 and then the PS4 when it came

2201.52

out I think was like 299 the PS5 I think

2204.76

was like five or six the low version

2206.64

it's like they it's like everybody else

2208.319

they're gonna like they're gonna throw

2209.44

some extra stuff but they're going to

2210.48

charge you more we will not we're going

2212.839

to charge you the same amount every time

2214.52

you come here we are going to wrap this

2216.4

one up because we're going to write in

2217.839

the sunset or something like that and go

2220.72

have a good time as always feel free to

2222.839

like leave us notes in the comments uh

2225.8

check out the show notes those are like

2227.48

we you know got some AI stuff and stuff

2229.48

like that we usually get really cool

2231.119

show notes lots of links there check us

2233.64

out here so you can check that out uh on

2236.28

the site at developer.com or you can

2238.24

just leave some comments here and say

2239.76

hey dudes can you send me something and

2241.359

we'll shoot you an email of that episode

2243.52

stuff or we may throw other stuff at you

2245.24

it depends because we do have things

2246.64

floating around that we could send to

2248.119

you just a little teaser there no

2250.119

promises but you never know you never

2251.88

know what might end up in your mail if

2253.839

you send us something because we are

2255.28

very appreciative for you guys out there

2257.04

listening to us and you know being there

2259.52

and putting up with our ugly mugs for

2261.319

the last little bit of time that you did

2263.04

that being said we're not going to go

2264.4

get facial surgery so you're still going

2265.88

to be able to see our same ugly mugs

2268.04

next time around and you guys have a

2270.56

great day and we will talk to you then

2273.28

take care

2276.16

everyone uh

2278.839

wh that is not the button I there we go

2282.2

stop recording I freaked out

2285.31

[Music]