Detailed Notes
We continue the series of tutorials as we build out a backup system that drops archive files into Amazon's S3 and related storage. This episode uses the Boto3 library and AWS command line interface in a practical sense as we store and retrieve files to S3.
You can learn more through our online classes at https://school.develpreneur.com and register for free. Registration will add you to our email list, and you will periodically receive coupons for courses and notifications of the latest releases.
Transcript Text
[Music] well hello and welcome back we are continuing our little uh backup project we're looking at uh in particular doing some Amazon AWS S3 stuff and uh dealing with some of their buckets and storing files and also going to do some packaging of files and things like that this time we have um we've already set up stuff we've we've got our S3 uh using B3 using that Library we've got that set up got it configured we created a bucket so now uh and we've also been able to list a bucket uh the buckets that we have so now what we're going to do is we're going to do some uh file manipulation so what we're going to do is we're going to add uh the ability to upload a file and then also to bring one back down and take a look at what's in there so what we're going to do is uh we have uh here we're going to do upload a file to S3 and it's actually very straightforward uh because what we're going to do is we're going to give it a file which is basically whatever it is to our local file and then we're going to give it a bucket name and we're going to load it up there so like we've done before we're going to create a uh just come in and get an instance of the Bodo 3 client and for the that's going to for the S3 uh if you ever do and we'll probably do some as I said some other stuff where we'll get into some of the other um AWS services and features because there are definitely a lot of things you can do with this uh as always you know if you want to take a look at what AWS offers look at aws.amazon.com and just look at the I think they're up to 150 200 Services something like that lots of stuff out there I dig grass so what we're going to do is we're going to grab the file name uh we're just going to do that based off of the path here uh we're going to get the the base name for the file whoops and then all we have to do stop doing my don't give me that uh basically we're going to just tell it upload a file so we're going to say where it is we're going to tell it what the bucket name is and then what is the file name that we're going to save it as and then we're going to do a little thing out here and it's a straightforward exception so if we look at uh let's just look here so we're going to actually upload we're going to upload this bucket project. py and uh we call this upload file to whoop to S3 so what we're going to do here is if we give it upload then we're going to give it the uh the file or the bucket name in this case for upload uh let's do it like this way equals 4 let's see well yeah we'll leave it like that we're going to have two usages so the first one is going to be upload the second one's going to be download let's make this a little less confusing so for upload we're going to do the name and then we're going to do the Target and so the name is the file name and then second is going to be the target bucket and let me make sure we've got that right upload to S3 if we look at that okay so the file and then the bucket yeah so that's what we're going to do so right now we're going to do an upload with that so if we come in let's see I save it and it's going to start out well let's comment these little helper little things out it's going to go with that uh let's see oh this is going to be uh let's do this if the length of arguments equals three then it's going to be operation name and Target operation name Target upload there we go that's going to be operation be one okay so there we should be good so let's do so if we do Python 3 uh what do we call this bucket project and it's going to be upload and we're going to upload bucket project and we're going to upload it into our little bucket that we created which is this guy right here so if we do that uh operation is not defined uh let's see interest interesting why did it say it did that uh did I miss something uh see one two oh it is going to be four that's what I made I'm sorry I miscounted because for one of them is one of the arguments is the uh the name itself so that's the name of the script and then we have three oh we have two after that I'm sorry it's python oh wait let's look at this what did I just mess up I think I just miscounted stuff so operation equals equal name operation is not Define oh because it wasn't equal to three it's actually equal to four which is correct right 1 two 3 one 2 3 4 there we go okay yes so script action uh Source bucket so that's four and so I need to do that here and actually I don't need that because now I know it's equal to four okay sorry I was overthinking that a little bit uh so now if we run it then uh we uploaded it to our uh up to that bucket and so what we're going to see is that uh let's go we're going to list our buckets and we're going to list bucket contents so uh we actually not going to bother with listing buckets and we're just going to do it so now we can run it without any let's do this Python 3 bucket we're going to run it and then we're going to see uh right here oh I guess I I should probably show you how we did this I forgot so listing buckets is also very straightforward we're going to come in we're going to get a client as we've done before uh then we go like here it says hello oh that's sorry that's list buckets I want list bucket content which is what we're calling down here and we're just giving it a bucket so then we come up to list bucket contents same thing we get a we create a client uh we grab the bucket and say hey this is the bucket we're going to use and then for each object in buckets we're just going to print out here what's the file key which happens to be the name and then I'm throwing the object itself which is basically a bucket name and a key right now so we could do which we saw here here's where we can just grab the key so if we want the file name then we just grab the key and we're off and running if we wanted to make that more complicated then we could uh but here uh when we do our oops that's downloaded when we upload we're just using that file name so if we wanted to do something more complicated like let's say file name plus let's call it uh let's call it archived if we do that and now we save our file uh then if we run it did I run oh I did yeah so now if we run just this we're going to see that uh here sorry it's like not the here let me do it this way let me just quit because then I'll make it a little easier let's see if I do this let's just quit after this so we can see it and let's like clean this up a little bit this so now we're going to see that we have bucket the file key was bucket project and then bucket project archived and so we just tacked on uh here we just tacked on that string so you can do whatever you want to set that key uh typically it may be a file name but depending on what you do it may be something much more complicated than that uh for example very often if it's a file related to a uh like for an application where you're going to upload files for users then maybe it's going to be like the user ID or something like that followed by you know maybe you want to do a dash and then um you the name of the file or something like that or some user you know entered name of the file there are uh some uh there are some restrictions a little bit about how the names go but uh you can always look those up on the site generally speaking if you grab the file name you should be safe so now what we want to do is uh we want to download a file so we want to come in and we can either here let's see we'll do it simply this time what we're going to do is we're going to download the bucket contents uh so this time we're just going to do uh a name and a Target except for this is going to be download and the name is going to be the bucket name correct make sure I got that right yeah bucket name and then the target is going to be the locate directory now what I'm going to do here we're not going to get uh we're going to keep this sort of simple because what we're going to do is we're going to come down again we create our client uh grab our bucket and we're going to create the local directory in case we need to uh because we're going to shove all these files into a location and they're going to walk through each object and then we are going to use the local directory thing plus the key so it's going to be plus the file name and then we're going to try to download it and then just download it in there so it's really straightforward again it's like download file and upload file S3 is really easy to work with it's almost like just you copying stuff around anyway so now if I take uh download and I'm going to download My Bucket name where did my bucket name go so I'm going take that and I'm just going to put it in Temp and let's see if it creates Temp and throws those files on there uh so I did oh that was that list I didn't really need that any oh it's not going to run it right now because I have that and that so I don't need that um yeah I'm just going to leave it like this for now we're keeping it really simple in our script um so now what we're going to see is so I downloaded that I downloaded the initial one and the archived one into the temp and if we look at temp we're going to see there there's the two files now the interesting thing is you can see where because we have the name different size a little different but if we go look at uh we'll C it well let's look it over here if we go look here in temp look that looks very familiar because that is the file that we uploaded so I think we'll stop there for this uh this time we're go in and uh we've got some you know really simple stuff we can go we can send files up to S3 and we can bring them back down and we'll just continue on looking at some of the different features and utilities that we want to use uh as always Sho us an email at info@ developer.com if you have any questions other than that go out there and have yourself a great day a great week and we will talk to you next [Music] time well hello this is Rob from develop andur building better developers and we have a mastermind class that will be starting or actually several Mastermind classes starting in 2024 check us out at technology Mastermind 2024 we've got all the information you need including some pricing early bird discounts up to 50% off but those are things that you're going to have to act soon there is an application process so go ahead and sign up today thanks a lot
Transcript Segments
[Music]
well hello and welcome back we are
continuing our little uh backup project
we're looking at uh in particular doing
some Amazon AWS S3 stuff and uh dealing
with some of their buckets and storing
files
and also going to do some packaging of
files and things like that this time we
have um we've already set up stuff we've
we've got our S3 uh using B3 using that
Library we've got that set up got it
configured we created a bucket so now uh
and we've also been able to list a
bucket uh the buckets that we have so
now what we're going to do is we're
going to do some uh file manipulation so
what we're going to do is we're going to
add uh the ability to upload a file and
then also to bring one back down and
take a look at what's in there so what
we're going to do is uh we
have uh here we're going to do upload a
file to
S3 and it's actually very
straightforward uh because what we're
going to do is we're going to give it a
file which is basically whatever it is
to our local file and then we're going
to give it a bucket name and we're going
to load it up there so like we've done
before we're going to create a uh just
come in and get an instance of the Bodo
3 client and for the that's going to for
the S3 uh if you ever do and we'll
probably do some as I said some other
stuff where we'll get into some of the
other um AWS services and features
because there are definitely a lot of
things you can do with this uh as always
you know if you want to take a look at
what AWS offers look at aws.amazon.com
and just look at the I think they're up
to 150 200 Services something like that
lots of stuff out
there I dig grass so what we're going to
do is we're going to grab the file name
uh we're just going to do that based off
of the path here uh we're going to get
the the base name for the file whoops
and then all we have to do stop doing my
don't give me that uh basically we're
going to just tell it upload a file so
we're going to say where it is we're
going to tell it what the bucket name is
and then what is the file name that
we're going to save it as and then we're
going to do a little thing out here and
it's a straightforward exception so if
we look at uh let's just look here so
we're going to actually upload we're
going to upload this bucket project. py
and uh we call this upload file to whoop
to S3 so what we're going to do here is
if we give it upload then we're going to
give it the uh the file or the bucket
name in this case for upload uh let's do
it like this
way equals
4 let's see well yeah we'll leave it
like that
we're going to have two
usages so the first one is going to be
upload the second one's going to be
download let's make this a little less
confusing so for upload we're going to
do the name and then we're going to do
the Target and so the name is the file
name and then second is going to be the
target
bucket and let me make sure we've got
that right upload to S3 if we look at
that okay so the file and then the
bucket yeah so that's what we're going
to do so right now we're going to do an
upload with that so if we come
in let's see I save
it and it's going to start out
well let's comment these little helper
little things
out it's going to go with
that uh let's see oh this is going to
be
uh let's do
this if the length of arguments equals
three then it's going to be operation
name and Target operation name Target
upload there we go that's going to be
operation be one okay so there we should
be good so let's do
so if we do Python
3 uh what do we call this bucket
project and it's going to be
upload and we're going to upload bucket
project and we're going to upload it
into our little bucket that we created
which is this guy right
here so if we do
that uh operation is not defined uh
let's see interest interesting why did
it
say it did
that
uh did I miss something uh see one
two oh it is going to
be
four
that's what I made I'm sorry I
miscounted because for one of them is
one of the arguments is the
uh the name itself so that's the name of
the script and then we have three oh we
have two after that I'm sorry it's
python oh wait let's look at
this what did I just mess up I think I
just miscounted stuff so operation
equals equal name operation is not
Define oh because it wasn't equal to
three it's actually equal to four which
is correct right
1 two
3 one 2 3 4 there we go okay yes so
script action uh Source bucket so that's
four and so I need to do that
here
and actually I don't need that because
now I know it's equal to four okay sorry
I was overthinking that a little bit uh
so now if we run
it then uh we uploaded it to our uh up
to that bucket and so what we're going
to see is that uh let's
go we're going to list our buckets and
we're going to list bucket contents so
uh we actually not going to bother with
listing
buckets and we're just going to do it so
now we can run it without any let's do
this Python 3 bucket we're going to run
it and then we're going to
see uh right here oh I guess I I should
probably show you how we did this I
forgot so listing buckets is also very
straightforward we're going to come in
we're going to get a client as we've
done
before uh then we go like here it says
hello oh that's sorry that's list
buckets I want list bucket content which
is what we're calling down here and
we're just giving it a bucket so then we
come up to list bucket contents same
thing we get a we create a client uh we
grab the bucket and say hey this is the
bucket we're going to use and then for
each object in buckets we're just going
to print out here what's the file key
which happens to be the name and then
I'm throwing the object itself which is
basically a bucket name and a key right
now so we could do which we saw here
here's where we can just grab the key so
if we want the file name then we just
grab the key and we're off and running
if we wanted to make that more
complicated then we
could uh but here uh when we do our oops
that's downloaded when we upload we're
just using that file name so if we
wanted to do something more
complicated like let's say file name
plus let's call
it uh let's call it
archived if we do that and now we save
our
file
uh
then if we run it did I run oh I did
yeah so now if we run just this we're
going to see
that uh
here sorry it's like not the here let me
do it this way let me just quit because
then I'll make it a little easier let's
see if I do
this let's just quit after
this so we can see it and let's like
clean this up a little bit this so now
we're going to see that we have bucket
the file key was bucket project and then
bucket project
archived and so we just tacked
on uh here we just tacked on that string
so you can do whatever you want to set
that key uh typically it may be a file
name but depending on what you do it may
be something much more complicated than
that uh for example very often if it's a
file related to a uh like for an
application where you're going to upload
files for users then maybe it's going to
be like the user ID or something like
that followed by you know maybe you want
to do a dash and then um you the name of
the file or something like that or some
user you know entered name of the file
there are uh
some uh there are some restrictions a
little bit about how the names go but uh
you can always look those up on the site
generally speaking if you grab the file
name you should be safe so now what we
want to
do is uh we want to download a file so
we want to come in and we can
either here let's see we'll do it simply
this time what we're going to do is
we're going to download the bucket
contents uh
so this time we're just going to do uh a
name and a Target except for this is
going to be
download and the name is going to be the
bucket name correct make sure I got that
right yeah bucket name and then the
target is going to be the locate
directory now what I'm going to do here
we're not going to
get
uh we're going to keep this sort of
simple because what we're going to do is
we're going to come down again we create
our client uh grab our bucket and we're
going to create the local directory in
case we need to uh because we're going
to shove all these files into a location
and they're going to walk through each
object and then we are going to use the
local directory thing plus the key so
it's going to be plus the file name and
then we're going to try to download it
and then just download it in there so
it's really straightforward again it's
like download file and upload file S3 is
really easy to work with it's almost
like just you copying stuff around
anyway so now if I take uh
download and I'm going to download My
Bucket name where did my bucket name
go so I'm going take
that and I'm just going to put it in
Temp and let's see if it creates Temp
and throws those files on there uh so I
did oh that was that list I didn't
really need that any oh it's not going
to run it right now because I have that
and that so I don't need
that
um yeah I'm just going to leave it like
this for now we're keeping it really
simple in our
script um so now what we're going to see
is so I downloaded that I downloaded the
initial one and the archived one into
the temp and if we look at temp we're
going to see there there's the two files
now the interesting thing is you can see
where because we have the name different
size a little different but if we go
look
at uh we'll C it well let's look it over
here if we go look here in
temp look that looks very familiar
because that is the file that we
uploaded so I think we'll stop there for
this uh this time we're go in and uh
we've got some you know really simple
stuff we can go we can send files up to
S3 and we can bring them back down and
we'll just continue on looking at some
of the different features and utilities
that we want to use uh as always Sho us
an email at info@ developer.com if you
have any questions other than that go
out there and have yourself a great day
a great week and we will talk to you
next
[Music]
time
well hello this is Rob from develop
andur building better developers and we
have a mastermind class that will be
starting or actually several Mastermind
classes starting in 2024 check us out at
technology Mastermind
2024 we've got all the information you
need including some pricing early bird
discounts up to 50% off but those are
things that you're going to have to act
soon there is an application process so
go ahead and sign up today thanks a lot