Intro to Kinesis Analytics As we start our journey into the analysis domain of big data. Let’s start off with Kinesis Analytics. It’s another system for querying streams of data continuously, very similar in spirit to Spark Streaming, but it is specific to AWS Kinesis. So conceptually it’s pretty simple. Kinesis Data Analytics can basically receive data from either a Kinesis data stream or from a Kinesis data firehose stream. And just like Spark Streaming, you can set up windows of time that you can look back on and aggregate…

[Exercise] Elastic MapReduce, Part 1 For our next hands on activity, we’re going to build a product recommendation system for Cadabra. com. And the good news is that we’ve already built out most of this system way back in Exercise One. So we already have an EC Two instance that is Populating server logs that get consumed by Kinesis data firehose, which in turn dumps that data into an Amazon S Three bucket. And this has already been set up. And you might recall that we already put 500,000 rows…

Hue, Splunk, and Flume There are several other technologies that are associated with EMR that you need to at least know what they are. You’re not going to need a whole lot of depth on these, so I’m just going to spend one slide on each one. The important things here are what each technology does and how it integrates with AWS. Those are the key points. Let’s start with Hue. Hue stands for Hadoop User Experience, and basically this is the front end for your entire cluster. It’s like…

Pig on EMR Let’s briefly talk about Apache pig. That is also an important part of the Hadoop ecosystem that comes preinstalled on Amazon EMR. So Pig Arose is sort of an alternative interface to MapReduce. So it recognizes that writing code for Mappers and reducers using MapReduce takes a long time. And pig introduced instead a language called Pig Latin. This is basically a scripting language that lets you use SQL style syntax to define your map and reduce. So instead of writing Java code for a MapReduce code,…

EMR Promises; Intro to Hadoop Let’s talk about some of the guarantees EMR makes to you. What is that? EMR charges you by the hour in addition to EC two charges. So you are paying for every hour that your cluster is running, whether you’re using it or not. So it is not a serverless kind of thing. You are provisioning a fixed amount of servers and you are paying for those servers for as long as they’re running. Like we talked about, you can use transient clusters to actually…

Glue Costs and Anti-Patterns Let’s talk about Glue’s cost model real quick. Again, it is serverless, so you’re just paying for the capacity that you’re consuming. You will be billed by the minute for your crawler jobs and ETL jobs. So you’re just billed by CPU time, basically, for any crawlers that are actually extracting information from your schema, and any ETL jobs that are transforming that data on demand. The first million objects stored and accessed are free for the Glue data cap log, and any development endpoints for…

[Exercise] AWS Lambda So now that we know what AWS Lambda is all about, we can finally finish up our order history app requirement here. We’re just going to fill in this last missing piece here. Already. We’ve created our fake server logs on an EC Two instance. We’ve put those into a kinesis data stream using the Kinesis agent, and we have a DynamoDB table set up to receive that data that an app could then talk to to actually retrieve customer order information from an end user. The…

What is AWS Lambda? Let’s talk about AWS lambda. Lambda is a serverless data processing tool that you can use. Let’s talk about what that means. So what is lambda? Basically, it’s a way to run little snippets of code in the cloud. So if you have a little bit of code in pretty much any language you can imagine, lambda can run that for you without you having to worry about what servers to run it on. So you don’t have to go and provision a bunch of EC…

DynamoDB Security So now let’s talk about DynamoDB security. Basically, you have VPC endpoints to access DynamoDB without the Internet. You can get access fully controlled by IAM. You can get encryption at rest using KS, encryption in transit using SSL and TLS. So it’s basically as secure as any AWS service. You get backup and restore feature available, so you can do a point in time. Restore? Just like RDS, there’s no performance impact. We can do backup and restores. You can define global tables, which are multiregional, fully…

The alert log Part 2 Hello everyone and welcome back. I hope that you are doing fine. In the previous lecture we took an overview about the alert look, today we will navigate and we will see the alert logs. So I open my virtual machine. I will go to to home. Then I will go to other location computer one app Oracle. Then I will go to Diag. Then I will go to the product RDBMS. Then the oracle and oracle again. So this is the directory. Actually you…