Detailed Notes
We wrap up the series of tutorials as we build out a backup system that drops archive files into Amazon's S3 and related storage. This is our final tutorial in the series as we work on deploying and scheduling our script.
You can learn more through our online classes at https://school.develpreneur.com and register for free. Registration will add you to our email list, and you will periodically receive coupons for courses and notifications of the latest releases.
Transcript Text
[Music] well hello and welcome back we are wrapping up essentially this time our little backup script and we're going to walk through a couple last minute cleanup things and deploy and we're to talk about deploying a python script and a couple little things we can do to help ourselves out in the future now one of the things that has sort of come together is that we have got um all of our little functions we built now one of the things that we pulled over this time around is uh we're going to call it backup database oops I don't need that and uh we saw this in the prior one where it's just a uh we just didn't pull it into our backup script I don't think and so what we've also done it's just because we've got a you know a lot of useful little things we've now tweaked our script a little bit so that it will allow us to we can use a um a bucket parameter and if we do that by itself it's going to list the buckets that we have uh connected for our account if we do files and give it a bucket name then it's going to list the files for that bucket and then we've got of course our other pieces we've also added dump which gives us our data we give it a database name and an output file so it would be you know my database and you my database.sql and that one the dump is just going to dump the file we could like we did with our ZIP and store we could actually upload our file so if we wanted uh let's see backup data base let's just do that we'll just uh oh and we can clean that we bring that over this way a little bit so what we could do here is we could say hey I'm going to back up the database and then I'm going to take take the out file whatever output file is and I'm going to send it to my bucket name but then that means I've got to give it a another bucket name so let's not uh we'll save that for another time uh actually what we could do is yeah we'll leave it for now but basically we could add another bucket name and then send that out or uh there's a couple different ways we could do that for our backup database if we wanted to keep it simple as number of parameters then what we could do is generate based on the database name we could send a database in a bucket and just do database.sql for output so we don't have to send that file name in anymore and then we could send it out to a bucket but that's you know neither here nor there that's some of the stuff that you can do now so for example if we do Python 3 Let's just show one real quick uh bucket project is what we call this and we're going to just call it buckets if we do that then it's going to list hey here's all of the buckets that we have available if I want to take one of those I don't even know what this thing is let's see what this is uh that one is going to be uh files and then it's got a couple different files there so it's got some text files setting out there obviously it's been sitting around for a while too so that is our backup script now what we can do let's open this up a little bit more so I'm going to go and I'm going to copy this script out uh this is called bucket project and I'm going to copy it out to I'm going to SCP it there's a couple ways you can do it but I'm going to SCP this script out to My Demo server and I'm going to do I'm going go log into that and so now if you look I've got my bucket project there now what I'm going to do is I'm going to try to run that and this has got a thing called shorty so let's start with um let's do this let's do make uh do I have a let me see if I have a backup folder I do not so let's do this I'm just going to do yeah I might do that so I'm going to make something I'm going to make a backup folder out on the opt and do something out there and so let's just go ahead and chod 777 that and I'm going to own it I'm to change that oops I need to pseudo that so now this user owns that backup so if I look I can see that on backup I'm owning it I've got all the rights I need so now I'm going to go to backup oh and I'm actually going to go ahead and copy my script out here just because just to keep it sort of clean so I'm going to take the uh bucket project to here and then I'm going to move that to uh backup. py so now I've got that so now I can try to do Python 3 back up if I don't give it anything then it's going to say hey I don't have boto 3 so I need to do pip 3 install boto 3 we're going to look at this in just a second it's going to take it and we're going to go ahead and we're going to install that and we're going to try to run it let's see what it does now this one is going to probably does not have Okay so that's l list out of range so I don't have it any further able to do that so let's just do buckets I need to give it something and so now I don't have my um I don't have my AWS installed so I'm going to need to do that so I had to go through and install my client on this guy so now when we run it we can see here's my list of buckets as we've seen before now one of things we can do is that we can do a pip three freeze and we can actually see all of now this one's got a whole bunch of crap but we can see all of the packages that we have set up and so we can see for example boto3 photo core all that kind of stuff so what we can do is we can take this and if we do that into requirements. text then one of the things we can do for an install is uh to help us out is we can do a pip 3 install with a-r give it the requirements file and it will try to install all of that stuff and now here it's already satisfied because I've already done it uh even though it looks like it's picking a couple interesting little versions along the way not sure why it's doing that but that's okay uh the bottom line is is that makes it easier so we don't have to like we just did run through an install or run through run the project or the application and then find out that oh I'm missing all these requirements we can just run it once it's going to pick up whatever we're missing and then we're off and running otherwise it can get a little bit tedious particularly when we're doing things like uh like a Jango application or something we've got a lot of different things thrown in there or there's some other things if you're doing flask and some other uh python projects that can get a little bit complicated so now what we want to do is one we're going to wait for this to be done installing dependencies so we can get through this little thing apologies if that took a second but that's okay I'll be right back when this is done okay that took a long time I apologize for that um let me clean up a couple things and so now what we've got is we've got our little backup script so what we would want to do in this case is let's take uh let's just back up the database so if we wanted to do something complex I guess we could have it like schedule we can just do our very basic uh backup dump I think it's lower I think it's uppercase and we're just going to call it shorty. SQL oops uh so I don't have my I don't have my do my config so and I probably need to clean up that call in there three two one so let me let's make sure I don't have my user do I have Rob here somewhere I don't okay so now I should be able to dump uh let's see let's make sure I've got it oh I did not try one more time nope that's why I don't have that user set so if I set up my user properly let me make sure I've got him okay I do one second I need to change my my conf 3 2 1 okay so now that I've got it now it will run now one of the things we can do is we can actually scoll ripped this so let's go to uh we're going to make scripts and we're going to do a backup. shell three 2 one and so we're going to come in here and we're going to create a little script that's just going to do Python 3 our backup and we're going to call this uh shorty and we're going to give it this it's going to give us our name so now right it's going to give us a date so what we can do here is if we chod plus X the backup script and then we do um SLB backup then if we go to our opt backup what we see now is we've got our name with us with the date in that and we could also do it with time and such like so that we can actually now we could create a Cron job that runs us for example every night at midnight and now we've backed up our database now the other thing we can do is if we want to back up folders we could do the same thing so I could do uh let me go back to my home and let me go to my scripts and what I can also do well where did I put that oops put that in the wrong place so I'm going to put that to scripts and so now if I go to scripts I could also say I'm going to back up my when I do this I also want let me do this there we we go so the same thing I can go back and I can do opt backup backup. py and I can do uh what did I call that I call that uh where is where is the see OH ZIP and store so archive I called that archive and so then I can take switch uh let's see so I'm going to take my home folder and I'm just going to call it uh home let's do this demo 3 home. zip and I think I also need does that give me the bucket let's see if I do arive oh because it gives me a hardcoded bucket name right now so I'm going to change it well I'll leave that for now so what I want to do is give it an extra um give it something extra so I can give it the right bucket but if I do that and let me just run that real quick um so if I wh whoa let's just do this where did I go there we go so if I take that and run it then it's going to take a second I'm assuming and then it's going to store my file and that's probably where it's taking a second I don't know how big this thing is I'm actually creating I probably should have looked at that well what I'm running into is I've got something that's getting me some sort of weird link because I'm getting a huge file which normally I would not get so if I just take this um see if this will work it's still giving me a huge so something is wrong with my script if I look at archive okay make archive directory let's do this real quick let's make sure we're getting this right so the backup name that's the directory let's make sure it's getting this right so if I just do let's do this um scripts so let's run that I'm going to Archive I'm going to do scripts there we go so something about that slash it does not like so instead I'm going to go so if I do do it from here let's see if I do it from here is that going to be huge uh apparently it is so so there's something about the number of files and how it's pulling it in and honestly that's okay um we can always fix that yeah see this thing is like blowing up super big so that's something we can chase down another time it's somehow grabbing the wrong folder uh yeah it just says dot so not sure why it's running that uh why it's running a problem but the bonus is you can come in you can actually set up your uh you can set up your script and you can use your dates you can actually pull off some folders and or files and the next thing you know you've got something that's pretty nice and handy it is a backup script but also one that you can push stuff out to your buckets if you want to out on out into your S3 buckets it's Tak me a little bit to get through this one because of a couple little bugs apologies for that hopefully it's not an overly long or overly short recording but now we've got our back backup script we will come back next time around and we'll have some new stuff to talk through uh whether it's python or something else we'll just sort of see where we're going if you have any questions shoot us an email at info@ develop andur go out there and have yourself a great day a great week and we will talk to you next time well hello this is Rob from develop andur building better developers and we have a mastermind class that will be starting or actually several Mastermind classes starting in 2024 check us out at technology Mastermind 204. we've got all the information you need including some pricing early bird discounts up to 50% off but those are things that you're going to have to act soon there is an application process so go ahead and sign up today thanks a [Music] lot
Transcript Segments
[Music]
well hello and welcome back we are
wrapping up essentially this time our
little backup script and we're going to
walk through a couple last minute
cleanup things and deploy and we're to
talk about deploying a python script and
a couple little things we can do to help
ourselves out in the future now one of
the things that has sort of come
together is that we have got
um all of our little functions we built
now one of the things that we pulled
over this time around is uh we're going
to call it backup database oops I don't
need
that and uh we saw this in the prior one
where it's just a uh we just didn't pull
it into our backup script I don't think
and so what we've also
done it's just because we've got a you
know a lot of useful little things we've
now tweaked our script a little bit so
that it will allow us to we can use a um
a bucket
parameter and if we do that by itself
it's going to list the buckets that we
have uh connected for our account if we
do files and give it a bucket name then
it's going to list the files for that
bucket and then we've got of course our
other pieces we've also added dump which
gives us our data we give it a database
name and an output file so it would be
you know my database and you my
database.sql
and that one the dump is just going to
dump the file we could like we did with
our ZIP and store we could actually
upload our
file so if we
wanted uh let's see backup data base
let's just do that we'll just uh oh and
we can clean that we bring that over
this way a little bit so what we could
do here is we could say hey I'm going to
back up the database and then I'm going
to take take the out
file whatever output file is and I'm
going to send it to my bucket name but
then that means I've got to give it a
another bucket name so let's
not uh we'll save that for another time
uh actually what we could do
is yeah we'll leave it for now but
basically we could add another bucket
name and then send that out
or uh there's a couple different ways we
could do that for our backup database if
we wanted to keep it simple as number of
parameters then what we could do is
generate based on the database name we
could send a database in a bucket and
just do database.sql for output so we
don't have to send that file name in
anymore and then we could send it out to
a bucket but that's you know neither
here nor there that's some of the stuff
that you can do now so for example if we
do Python 3 Let's just show one real
quick uh bucket project is what we call
this and we're going to just call it
buckets if we do that then it's going to
list hey here's all of the buckets that
we have available if I want to take one
of those I don't even know what this
thing is let's see what this is uh that
one is going to be uh
files and then it's got a couple
different files there so it's got some
text files setting out
there obviously it's been sitting around
for a while too so that is our backup
script now what we can do let's open
this up a little bit more so I'm going
to go and I'm going to copy this script
out uh this is called bucket project and
I'm going to copy it out
to I'm going to SCP it there's a couple
ways you can do it but I'm going to SCP
this script out
to My Demo server and I'm going to do
I'm going go log into
that and so now if you look I've got my
bucket project there now what I'm going
to do is I'm going to try to run
that and this has got a thing called
shorty so let's start
with um let's do this let's do make uh
do I have a let me see if I have a
backup folder I do not so let's do this
I'm just going to do yeah I might do
that so I'm going to make something I'm
going to make a backup folder out on the
opt
and do something out
there and so let's just go ahead and
chod 777
that and I'm going to own it I'm to
change
that oops I need to pseudo
that so now this user owns that backup
so if I look I can see that on backup
I'm owning it I've got all the rights I
need so
now I'm going to go to backup
oh and I'm actually going to go ahead
and copy my script out here just because
just to keep it sort of clean so I'm
going to take
the uh bucket
project to here and then I'm going to
move that
to uh backup.
py so now I've got that so now I can try
to do Python 3 back up if I don't give
it anything then it's going to say hey I
don't have boto 3 so I need to do pip 3
install boto 3 we're going to look at
this in just a
second it's going to take it and we're
going to go ahead and we're going to
install that and we're going to try to
run it let's see what it
does now this one is going to probably
does not have Okay so that's l list out
of range so I don't have it any further
able to do that so let's just do buckets
I need to give it
something and so now I don't have my um
I don't have my AWS
installed so I'm going to need to do
that so I had to go through and install
my client on this guy so now when we run
it we can see here's my list of buckets
as we've seen before now one of things
we can do is that we can do a pip three
freeze and we can actually see all of
now this one's got a whole bunch of crap
but we can see all of the packages that
we have set up and so we can see for
example
boto3 photo core all that kind of stuff
so what we can do is we can take
this and if we do that into
requirements.
text
then one of the things we can do for an
install is uh to help us out is we can
do a pip 3
install with a-r give it the
requirements file and it will try to
install all of that stuff and now here
it's already satisfied because I've
already done it uh even though it looks
like it's picking a couple interesting
little versions along the way not sure
why it's doing
that but that's okay uh the bottom line
is is that makes it easier so we don't
have to like we just did run through an
install or run through run the project
or the application and then find out
that oh I'm missing all these
requirements we can just run it once
it's going to pick up whatever we're
missing and then we're off and running
otherwise it can get a little bit
tedious particularly when we're doing
things like uh like a Jango application
or something we've got a lot of
different things thrown in there or
there's some other things if you're
doing flask and some other uh python
projects that can get a little bit
complicated so now what we want to do is
one we're going to wait for this
to be done installing dependencies so we
can get through this little thing
apologies if that took a second but
that's okay I'll be right back when this
is done okay that took a long time I
apologize for that um let me clean up a
couple
things and so now what we've got is
we've got our little backup script so
what we would want to do in this case is
let's take uh let's just back up the
database so if we wanted to
do something complex I guess we could
have it like schedule we can just do our
very basic uh backup dump I think it's
lower I think it's uppercase and we're
just going to call it shorty.
SQL oops uh so I don't have
my I don't have my do my
config
so
and I probably need to clean up that
call in
there
three
two one so let me let's make sure I
don't have my user do I have Rob here
somewhere I don't okay so now I should
be able to
dump uh let's
see let's make sure I've got
it oh I did
not try one more
time nope that's why I don't have that
user set so if I set up my user properly
let me make sure I've got
him okay I do one second I need to
change my my
conf
3 2 1 okay so now that I've got it now
it will run now one of the things we can
do is we can actually scoll ripped this
so let's go
to uh we're going to make
scripts and we're going to do a
backup.
shell
three 2 one and so we're going to come
in here and we're going to create a
little script that's just going to do
Python 3 our backup
and we're going to call
this uh
shorty and we're going to give it this
it's going to give us our name so now
right it's going to give us a date so
what we can do here is if we chod plus X
the backup
script and then we do
um SLB
backup then if we go to our opt backup
what we see now is we've got our name
with us with the date in that and we
could also do it with time and such like
so that we can actually now we could
create a Cron job that runs us for
example every night at midnight and now
we've backed up our database now the
other thing we can do is if we want to
back up folders we could do the same
thing so I could do uh let me go back to
my home and let me go to my scripts and
what I can also do
well where did I put
that oops put that in the wrong place so
I'm going to put that to
scripts and so now if I go to scripts I
could also say I'm going to back up
my when I do this I also
want let me do this there we we go so
the same thing I can go back and I can
do opt
backup backup.
py and I can do uh what did I call that
I call
that uh where is where is the
see OH ZIP and store so
archive I called that
archive and so then I can take switch uh
let's see so I'm going to take my home
folder and I'm just going to call it uh
home let's do this demo 3 home.
zip and I think I also
need does that give me the bucket let's
see if I do
arive oh because it gives me a hardcoded
bucket name right now so I'm going to
change it well I'll leave that for now
so what I want to do is give it an extra
um give it something extra so I can give
it the right bucket but if I do
that and let me just run that real quick
um so if I wh
whoa let's just do this where did I
go there we go so if I take that and run
it
then it's going to take a
second I'm
assuming and then it's going to store my
file and that's probably where it's
taking a second I don't know how big
this thing is I'm actually creating I
probably should have looked at
that well what I'm running into is I've
got something that's getting me some
sort of weird link because I'm getting a
huge file which normally I would not get
so if I just
take
this
um see if this will
work it's still giving me a huge so
something is wrong with my
script if I look at
archive okay make archive directory
let's do this real
quick let's make sure we're getting this
right so the backup name that's the
directory let's make sure it's getting
this right so if I just
do let's do this
um
scripts so let's run that I'm going to
Archive I'm going to do
scripts there we go so something about
that slash it does not like so
instead I'm going to go so if I do do it
from here let's see if I do it from
here is that going to be
huge uh apparently it is so so there's
something about the number of files and
how it's pulling it in and honestly
that's okay um we can always fix that
yeah see this thing is like blowing up
super
big so that's something we can chase
down another time it's somehow grabbing
the wrong
folder uh yeah it just says dot so not
sure why it's running that uh why it's
running a problem but the bonus is you
can come in you can actually set up
your uh you can set up your script and
you
can use your dates you can actually pull
off some folders and or files and the
next thing you know you've got something
that's pretty nice and handy it is a
backup script but also one that you can
push stuff out to your buckets if you
want to out on out into your S3
buckets it's Tak me a little bit to get
through this one because of a couple
little bugs apologies for that hopefully
it's not an overly long or overly short
recording but now we've got our back
backup script we will come back next
time around and we'll have some new
stuff to talk through uh whether it's
python or something else we'll just sort
of see where we're going if you have any
questions shoot us an email at info@
develop andur go out there and have
yourself a great day a great week and we
will talk to you next
time well hello this is Rob from develop
andur building better developers and we
have a mastermind class that will be
starting or actually several Mastermind
classes starting in 2024 check us out at
technology Mastermind
204. we've got all the information you
need including some pricing early bird
discounts up to 50% off but those are
things that you're going to have to act
soon there is an application process so
go ahead and sign up today thanks a
[Music]
lot