AWS Kinesis

Amazon Kinesis is a fully managed platform for streaming data on Amazon Web Services. In this guide we will show you how you can setup integrations to let your data (such as measurements and events) flow in and out of Kinesis.

Goal

The goal is to set up an inbound and outbound integration between Amazon Kinesis and Blockbax.

Prerequisites

• Blockbax project (with administrator access to it)
• Amazon Kinesis Data Stream
• Access via the AWS Management Console

Steps to stream data from AWS to Blockbax

Several steps in both AWS and Blockbax are needed to get the data from AWS to Blockbax.

Create a Blockbax user and the required policy in AWS IAM

1. Login to AWS Management Console and go to IAM
2. Click on ‘Create user’, give the user the desired name and click ‘Next’
3. When setting the permission options, choose ‘Attach policies directly’ and ‘Create policy’. You will be forwarded to a separate tab to create the policy. Choose JSON and copy and paste the snippet below. Replace arn:aws:kinesis:eu-west-1:123456789012:stream/ExampleStream with the correct ARN (including the ID of your AWS account)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReadStream",
            "Effect": "Allow",
            "Action": [
                "kinesis:DescribeStream",
                "kinesis:GetRecords",
                "kinesis:GetShardIterator",
                "kinesis:ListShards"
            ],
            "Resource":
               "arn:aws:kinesis:eu-west-1:123456789012:stream/ExampleStream"
        }
    ]
}

4. Click ‘Next’, give your policy the desired name and click ‘Create policy’
5. Then go back to the tab where you were creating the user
6. Refresh the policy list and select your just created policy
7. Click ‘Next’ and ‘Create user’

Create an access key in AWS IAM

1. Click on the user you created in the previous step
2. Navigate to the tab ‘Security credentials’ and the section ‘Access keys’
3. Click ‘Create access key’, ‘Other’, ‘Next’ and ‘Create access key’
4. Copy your ‘Access key’ and ‘Secret access key’, you need it in the following step

Create an inbound connector in Blockbax

1. Go to settings in the project
2. Navigate to the tab ‘Inbound connectors’ and click on the green ‘+’ in the top right
3. In the setting ‘Protocol’ select the option ‘AWS Kinesis consume’
4. Choose a suitable name for the inbound connector
5. The ‘access key ID’ and ‘Secret access key’ fields have to be filled in with the key that was created earlier
6. The rest of the required fields can be filled in by copying the information your AWS Kinesis overview page

   

7. Write a payload conversion script to convert the incoming payload to the correct Blockbax format. More information can be found here
8. Click the ‘Create connector’ and validate if the ‘Successfully (re)connected inbound connector, will start processing measurements.’ log shows up. This can take up to 30 seconds. If you encounter any issues feel free to contact us

Steps to stream data from Blockbax to AWS

1. Follow the steps from Create a Blockbax user and the required policy in AWS IAM, but instead use the policy JSON below. Replace arn:aws:kinesis:eu-west-1:123456789012:stream/ExampleStream with the correct ARN (including the ID of your AWS account).

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "WriteStream",
            "Effect": "Allow",
            "Action": [
                "kinesis:ListShards",
                "kinesis:DescribeStream",
                "kinesis:DescribeStreamSummary",
                "kinesis:ListTagsForStream",
                "kinesis:PutRecord",
                "kinesis:PutRecords"
            ],
            "Resource":
               "arn:aws:kinesis:eu-west-1:123456789012:stream/ExampleStream"
        }
    ]
}

Create an outbound connector in Blockbax

This can be enabled on request. Please contact us if you want us to set this up for you.