SAP-C02 Amazon AWS Certified Solutions Architect Professional – New Domain 5 – Continuous Improvement for Existing Solutions part 3

7. Understanding Deletion Policies in CloudFormation

Hey everyone and welcome back to the Knowledge Portal video series. And in today’s lecture, we’ll be speaking about the Deletion Policy attribute of the Cloud Formation. Now, this is a quite important topic as far as exams are concerned, so make sure you pay close attention on this. So in cloud formation, basically when we delete the stack, all the resources which created through that stack will also be deleted. Now, in certain cases we want that certain resource should not be deleted or snapshot must be taken before the resources are terminated. Now, let me give you one of the examples. So in the earlier lecture we had created an EC two instance with cloud formation. So this is the stack which is KP Labs Hyphen sample.

Now, if I go ahead and click on Delete stack, what would happen is it would delete all the resource which were created in this stack. So basically the instance now you see, the instance state is now shutting down and after this it will be deleted. Now, in many of the cases, specifically when you have RDS or EBS volumes, you don’t want them to be deleted or certain times you want that before they get terminated, a snapshot should be taken. So those aspects can be controlled by the Deletion Policy attribute in cloud formation. So as discussed, Deletion Policy attribute basically allows us to preserve or back up a resource when the stack is deleted. Now, in order to do that, we must specify a Deletion Policy attribute for each resource that we want to control.

So in this example, we are creating an S three bucket. You see, the S three bucket is the resource. And within the deletion policy. I have a deletion policy of retain. So whenever I create a stack, it will create a bucket. And when I delete the stack, although the stack will be removed, but the bucket will not be deleted. And the reason is because there is a Deletion Policy of type retain. So retain mean do not delete the resource. So there are two options when we use the Deletion Policy attribute. One is retain and second is snapshot. So as the name itself suggests, whenever we use the Deletion Policy of Retain, cloud formation keeps the resource without deleting it.

So even if we delete the Cloud formation stack, it will not delete the underlying resource which were created. And second is of type snapshot where cloud permission creates the snapshot of the resource before deleting it. Now, this can be associated with, let’s assume EBS instance or RDS or even Elastic Cache Cluster. So let’s assume that you are deleting a stack of RDS and it has a Deletion Policy with snapshot altribute. So what Cloud formation will do is before it deletes the RDS, it will take the snapshot and after snapshot, then it will go ahead and terminate the RDS instance. So this is about the theoretical part. Let’s go ahead and look into the practical aspect. So again, I have a clean slate of cloud formations stack.

So let’s do one thing. Let’s go ahead and create a new stack from a design. So I’ll go to my designer and let’s select S three and I’ll go ahead and create a bucket. So automatically it gave us the template of the bucket. Now, if you will see over here, within the deletion policy, it is giving us the blank output. So let’s do one thing. Let’s copy this up and I’ll paste it in my Atom editor. And within the deletion policy, I’ll put Retain. Perfect. So this seems to be quite good. Let’s do one thing. Let’s create a new text file. I’ll copy this, a template, and I’ll save it as S three dot JSON. Perfect. So we have our S three dot JSON which is created. Let’s go to Cloud Formation. Now let’s go to Cloud Formation.

I’ll click on Create and use that. And basically I’ll upload the S three JSON file. Click on next. Let’s give it a stack name. I’ll say KP Labs Dell Policy. I’ll click on Next. Next. And I’ll click on Create. So basically, this will go ahead and create a S three bucket. The S three bucket which has been created, it has a deletion policy of Retain. So once the S Three bucket is created, after that, even if we delete this specific cloud formation stack, it will not delete the S Three bucket. So let’s just wait for a while for the status to get completed. Perfect. So the creation is completed. So let’s do one thing. Let’s open up our S three. And this is the S Three bucket which got created by the cloud formation. So now let’s do one thing. So this is Kplab’s Hyphen Dell Policy.

You see, the bucket name also has Kplab’s Hyphen Dell Policy. So perfect. So now let’s do one thing. Let’s quickly verify the template. The template has a deletion policy of retail. So in order to test it, let’s go ahead and delete the stack and see whether our S Three bucket gets terminated or not. Perfect. So the stack is now removed. Now, if you go ahead and refresh the page, you see our S Three bucket is still present. And this is what the deletion policy attribute is all about. So as far as exam is concerned, do remember about what deletion policy attributes are all about. Do remember the two options which we can specify while putting the deletion policy attribute which is Retain and snapshot.

8. S3 – Multi-Part Uploads

Hey everyone, and welcome back to the Knowledge Pool video series. Now, today we are going to talk about a quite important topic, specifically for the users who downloads a lot, and that is multipart uploads and downloads. So I’ll tell you a story long time back around seven, eight years back in India, we had a speed of 256 kbps, the Indian Internet speed. And that was considered as a lot. And for the people who likes to take a lot of risk, like me, we try to download one GB file directly from browser. So one of the problem that happens is if your internet disconnects, then you have to download the entire file again. And that used to be a big challenge.

I still remember one time I was downloading a one DB file with 256 kbps connection and at 99% the rain gods decided to show up and the Internet disconnected. And at 99% the entire file went away. So I had to redownload the entire file again and it used to take like a day to download. So that used to be a quite big pain. So that is for download, even for uploading. So whenever you are uploading a large file and if your internet disconnects, then the entire uploads begin from the start. And this was a big pain. So, in order to save our computer’s lives, there has to be a good technology which allows better solution. And this is where the multi part upload and download comes into picture.

So multipart, as the name suggests, basically tells that there are multiple parts of a single file. So multipart upload is a way in which we upload an entire file in form of small individual chunks to the storage device. So let’s assume that we have this original object over here. So what we do is we divide this object into multiple small, small parts and then we upload these small, small parts to the S three. So even if your internet disconnects, what we have to do is we only have to again re upload that small chunk of data to the storage device. We don’t really have to begin from the start. So this original object is divided into twelve parts and you have three, six, nine and twelve.

So let’s assume you uploaded the nine parts and while you were uploading the 10th part, your internet disconnected. So after your internet starts back again, you don’t really have to begin uploading from start. You can resume your upload from the 10th file and complete the rest three. Now, after you upload all of these two S three, the S three will combine all of them together to form the original object again. So this is called as the multipart page technology. Now, one important thing to remember is when we are uploading data via multi part, we need to specify the part number and its position in the uploaded object. Now this is very important because this will help AWS reconstruct the data. So we have to tell AWS S three that this is let’s assume this is part one, this is part two, this is part three.

This is 45678, 910 1112. So when the S three receives all the parts, it needs to reconstruct them and it will reconstruct based on the number that we assign. This is the part number that we assign to each and every object that we upload in S three. So this is the basic about multipart upload. Let’s do one thing, let’s create a practical and see on how exactly it would really work. So I am in my EC two instance. So I have an EC Two instance which is running, and I’m connected to the EC Two instance. Now, if I do an AWS S three, LS, there are four files or four S Three buckets which are present right now. So, what we’ll be doing is we will upload a file to one of these S Three buckets via multipart uploads. So let’s do one thing since we need a good amount of file size.

So recommended way of multipart upload is if you have a file size which is greater than 100 MB, then you should be doing a multipart uploads. So let’s create a file. So I’ll do a DD. I’ll say input file is dev view random output file would be, say, bigfile TXT, block size is one MD and count is let’s assume 150 Oops. So what this will do is basically this will create a file of around 150 MB. So, as we have already discussed, it is recommended to have a file size of 100 greater than 100 MB if you are doing a multipart upload. Now you have a bigfile TXT. So if you do a quick du hyphen sh on bigfile TXT, it is of size 150 MB. Now, there are two ways, as we already discussed. There are two ways. One is you directly upload the file without multipart upload.

And during that case, if Internet breaks, then you have to upload everything from beginning. And the second way is you divide the file and you upload the files in multiple chunks. So, I have a nice little script called as multihyphon part PY. And this script basically uploads this file in multiple parts. So, let’s try this out. So, I’ll run multi part PY, I’ll specify the big file TXT. So this is the file that we want to upload and we’ll specify the bucket where we have to upload it, which is KP Labs iPhone Failover. So when we press Enter, what will happen is this Python file, you see, it will divide the file into multiple chunks and at a time it will upload the individual chunk. So this is chunk number zero. So once this chunk gets uploaded, then you have the chunk number one.

Let’s wait. Perfect. So now, what you have seen, the entire file got divided into two chunks. And first the upload was of first chunk, then upload was of second chunk. Now, you can definitely divide into much more smaller parts. So since there are two chunks, you can assume it this one to be 75 MB, this one to be 75 MB. But you can actually reduce it more further depending upon your requirements. Now, if you just want to quickly verify I’ll do. AWS. S three LS KP labs. Hyphen failed over. And you see, I have a big file TXT of size 157 MB. Now, this is the same file size. This is the same file size. So although it got uploaded in chunks in S three, it got combined together, the chunks got combined together, and you get a single big file. So this is it, about the multipart upload. I will be uploading this script in our forum section, so you can go ahead and download, and you can try it out in your labs. So this is it, about this lecture.

9. Multi-Part Upload – Practical

Hey everyone and welcome back. Now, in the earlier video of multipart upload where we had a demo by running a script, all the script is good for demo but it might not be a long time viable solution because there are lot of changes behind the scenes and script might not work after six months or one year. So what I decided was we’ll record one more video. This time will go with the AWS S Three API low level based CLI commands and look into how exactly the multipart upload works. So generally one important thing to remember is that even if you do an AWS S Three CP, by default, if the file is huge, it will do a multi part uploading. So that is the default thing that AWS S Three CP does. However, it does not really fit all the use cases.

So one use case that I can give you is that let’s suppose that one file is coming from server A and second part of the file you want to upload it from the server two. So now both the files you upload from a different server and in s three you want to merge both of those files. So that cannot be done with the AWS S Three CP command. So for those kind of use cases you need to go through a low level CLI commands and AWS S Three API is the right one that can allow us to do that. Now, if you look into the command, there are various available commands. For this you have complete multipart upload. In fact, if I do multipart you will see you have complete, you have abort, you have create and if you go bit down you have list also.

So all of these are the multipart upload commands that you can use to create a multi part upload. And if you do it through this way, you will actually come to know how exactly things work behind the scenes. Great. So for our demo purpose, what I have done is so for a sample I have MP4 file. Now this MP4 file have taken it as a random file from our TerraForm course. So if I quickly open this you would see that it is running and basically we will be using this file for our demo. Now, if you look into the properties, it is close to around 38. 7 MB, quite a good size for our sample multi part upload demo. Now, along with that, I already have a bucket which is created. Now the bucket name is Kplabs Hyphen multi PARTYPhone Demo and this is where we’ll be doing all the practicals from.

So I’m in my command prompt and if I do a LS, you see I have a terraformed up MP4 file. So since this is a little big file, let’s assume this is a big file. What we really need to do as a first step in multipart upload is to split a larger file into smaller chunks of data and this can be done with various commands including the built in Linux split command. So let me clear the screen and I’ll do LS, iPhone LH. So the current size of this file is 39 MB. So what I want is I want this file to be splitted into three parts. So the best way to do it through the built in Linux command is by running the split command. So this split basically allows us to split the file. Now, the file size that we’ll be giving is 18 MB and the file name is TerraForm and I’ll say part Hyphen.

So now if you do an LS again, you will see that it has divided this terraformed up MP4 into three files. One is part AA, you have part BB and part CC. Part AA is 18 MB, part AB is 18 MB and Part AC is 2. 8 MB. So these are the three parts of the files. So we will be uploading each one of these parts via the s three API based calls. So what I have done is I have already documented the commands and I’ll also be putting it just right below the video. So you can go ahead and you can download it after you complete watching the video. So the first command after you do the split is AWS s three API. You have create multipart upload and you are defining the bucket and you are also defining the key.

So in my case, the key would be TerraForm MP4 and let me copy this command and I’ll paste it wire. So once I press Enter, you would see that it basically has given me an upload ID. Now this upload ID is very important, do copy this upload ID and store it somewhere because this is going to be quite important in the upcoming command. So I’ll say upload ID and I’ll paste the upload ID here. Now, the next thing that you need to do is that you have to start uploading the parts. Now, we have already divided the file into three parts. So we’ll have to upload each one of these parts. So the command to do that is AWS s three API. Now, instead of create multipart upload, you have an upload part.

The bucket remains the same, the key also needs to be same which is TerraForm MP4 and additional thing is you have to define the part number. Now the part number is one. So this is going to be the first part and with body you will have to give the part name which is part AA. So if you will quickly see over here, this is the first part, part AA and then you have to give the upload ID. Now upload ID means the one that we figured it out from the previous command. So let me copy this up first. I’ll copy this and I’ll paste it over here and the next thing that I need is I have to copy the upload ID. And I’ll paste the upload ID. So it is uploading the file and after it has completed uploading the file, it will give you an etag as the output.

Now. Just copy this Etag. It will be important as well. So I’ll say for part one, the e tag is this one. Now, I’ll run the same command multiple times because the earlier command that we had run, it had part number one and the body was part AA. Now we know that we had splitted TerraForm MP4 into three files. So basically I have to run the same command three times. So this time it would be part number two and the part name would be part AB. So again, it is uploading the file. Once it has uploaded, it has given me the Etag, I’ll copy the etag and I’ll save this here. Along with that, we’ll have to do this again one more time because we have three parts. I’ll do the same thing.

This time it would be part C and part AC. Perfect. And again, this has given us the e tag and now we have the three e tags which are ready. So if you quickly do a clear screen. Now, if I quickly show you, if I try and refresh this, you will see that there are no files which are available over here. The reason why there are no files is because you have uploaded three parts, you have not concatenated them. So you will have to merge them so that merging is remaining. And this is the reason why you are not seeing the file over here. So in order to do that, the next thing that you will have to do is you have to do the concatenation part. So before we do that, we can do an AWS S three API list part call, which basically will show me all the parts which are uploaded.

So if you quickly run this command by specifying the upload ID, also I copy the upload ID. And if I press Enter all right, so we made a little mistake here. It should be TerraForm MP4. And here you will see that there are three files. One is part one, you have part number two and you have part number three. So all of them are present. All you have to do is you have to merge them back. Because these are multipart, you have to merge all the three parts into one part and that one part will be visible in the S three bucket. So let me clear the screen again and let me create a file call as file JSON. So within the file JSON, basically you have to specify all the e tags and the associate part name. So there is a specific format that you need to follow and I already have that format available.

Again, I’ll be uploading this format and you can use it at your end. Now, within this format, you see that there are part numbers and each part number has an associated Etag. So this etag comes from the Etag that we had stored earlier. So let me copy the etag associated with part one and I’ll paste it over here. Great. I’ll copy the e tag associated with part two and same with part three. So, once you have this directly, copy this up and paste it within the file dot JSON and save it. Now, the last thing that you have to do is you have to complete the multi part upload. So you have already done uploading of individual parts, so you have to complete the multi part upload. So for that we have a sample syntax of AWS s three API. This time it is complete the multipart upload.

Now we are specifying the multipart upload file as file JSON. So this is the file JSON which contains details related to part and its associate etag. So this allows s three to know which is part one, which is part two, which is part three, so that it can do the concatenation. Now, once it comes to know that, you have to basically specify the upload ID here. So you have multiplied upload file file JSON, you specify the bucket name, you specify the key which is terraformot MP4, and you specify the upload ID. So this is the only thing which is required. So, since our bucket name is kplabs multipart demo, I’ll use this, let me copy this up, I’ll clear the screen, I’ll run this command and I’ll put the upload ID also.

So once you have entered, you see that it is giving the output saying that the key is TerraForm dot MP4 and it is also giving the exact location to the TerraForm dot MP4 file. So now if you will go to the S three management console and if you press refresh, you will see that you have TerraForm MP4 file present. So, in order to quickly verify everything is working as expected, what I’ll do is I’ll download the file and we’ll try running it just to make sure that everything is working correctly. So let me open this up and if you will see that it is working perfectly from the last to end. So this means that our multi part upload has been completed successfully. So this is the low level details into how multipart really works.

So whenever you do an AWS s three CP, it does all of these things behind the scenes. However, this is something which gives you a good information related to how things work in a lower level. So this is it. About this video. I hope this has been informative for you and I look forward to seeing you in the next video.