Amazon Kinesis Streams

bogotobogo.com site search:

What is Kinesis?

Amazon Kinesis services make it easy to work with real-time streaming data in the AWS cloud.

Using AWS Lambda with Amazon Kinesis

Custom app writes records to the stream.
AWS Lambda polls the stream and, when it detects new records in the stream, invokes Lambda function.
AWS Lambda executes the Lambda function by assuming the execution role we specified at the time we created the Lambda function.

Streams constructs

Ref : Visualizing Web Traffic Using Amazon Kinesis Streams.

We will be familiarized with streams constructs; specifically streams, data producers, and data consumers.

The tutorial uses a sample application based upon a common use case of real-time data analytics.

We'll create data-processing applications, known as Amazon Kinesis Streams applications.

What Is Amazon Kinesis Streams?

A typical Amazon Kinesis Streams application reads data from an Amazon Kinesis stream as data records. These applications can use the Amazon Kinesis Client Library, and they can run on Amazon EC2 instances.

The processed records can be sent to dashboards, used to generate alerts, dynamically change pricing and advertising strategies, or send data to a variety of other AWS services.

Start the Sample Application

Start the sample application using a AWS CloudFormation template provided by AWS.

The sample application has a stream writer that randomly generates records and sends them to an Amazon Kinesis stream, a data consumer that counts the number of HTTPS requests to a resource, and a web application that displays the outputs of the stream processing data as a continuously updated graph.

Open the AWS CloudFormation template.

On the "Select Template page", the URL for the template is provided. Choose Next.
"Specify Details" page:

On the "Review" page, select I acknowledge that this template might cause AWS CloudFormation to create IAM resources, and then choose "Create".

Amazon Kinesis Stream

AWS CloudFormation automatically creates the stream for the sample application.

To view the stack details:

Select the "KinesisDataVisSample" stack.
On the Outputs tab, choose the link in URL.

The form of the URL should be similar to the following: http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com.
It takes about 10 minutes for the application stack to be created and for meaningful data to show up in the data analysis graph. The real-time data analysis graph appears on a separate page, titled "Streams Data Visualization Sample". It displays the number of requests sent by the referring URL over a 10 second span, and the chart is updated every 1 second. The span of the graph is the last 2 minutes.

Stream Details

To view the stream details:

Open the Amazon Kinesis console at https://console.aws.amazon.com/kinesis.
Select the stream whose name has the following form: KinesisDataVisSampleApp-KinesisStream-[randomString].

Choose the name of the stream to view the stream details.
The graphs display the activity of the data producer putting records into the stream and the data consumer getting data from the stream.

Please check Kinesis Data Firehose with Lambda and ElasticSearch to learn how Kinesis Firehose captures streaming data and transforms the data, and then sends it to ElasticSearch service.