Amazon Kinesis

Docs

Amazon Kinesis

Inbound connectors

Inbound connectors are custom integrations to ingest measurements into the Blockbax Platform. They allow you to extract information from a custom payload and to ingest measurements in the format that Blockbax expects. They also allow you to log information during this conversion. A payload conversion script is defined as a plain JavaScript function. A dedicated page about payload conversion describes how to develop such a function.

Note that we do not process payloads with a size over 200 KiB. When this happens an error log will be output.

Configuration options

The following configuration options are available:

Field	Description
State	Active or Inactive
Name	A descriptive name to recognize and find your inbound connector
Auto-create subjects	Automatically creates a subjects based on the incoming payload (see below)
Region	The AWS region where the Kinesis stream is located
Stream name	The name of the stream
Consumer group ID	The consumer group ID that the client will use
Initial position in stream	The place in the stream to start (only has effect when the broker does not recognize the consumer group id)
Access key ID	The ID of the access key that the client will connect with
Secret access key	The secret of the corresponding access key

Auto-create subjects

Measurements are mapped to subjects and metrics by so-called ingestion IDs. By default, an ingestion ID is derived from the subjects’ external ID and the metrics’ external ID (e.g. MyCar$Location). You can also override this default with a custom ingestion ID. If a measurement with an unknown ingestion ID is encountered, Blockbax tries to find a single metric with an external ID equal to the part after the dollar sign. If this is successful, a subject will be created with an external ID equal to the part before the dollar sign. For example, for the ingestion ID MyCar$Location a subject with external ID MyCar will be created if the metric with external ID Location can be linked to exactly one subject type.

Outbound connectors

Please contact us if you require this.

Connection guide

Here you can find the information to help you get started quickly with this integration.

Stream measurements from AWS to Blockbax

Several steps in both AWS and Blockbax are needed to get the data from AWS to Blockbax:

Create a user with the required policy in AWS IAM via a role or attached directly (replace “arn:aws:kinesis:eu-west-1:123456789012:stream/ExampleStream” with the ARN of your Kinesis stream)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ReadStream",
            "Effect": "Allow",
            "Action": [
                "kinesis:DescribeStream",
                "kinesis:GetRecords",
                "kinesis:GetShardIterator",
                "kinesis:ListShards"
            ],
            "Resource":
              "arn:aws:kinesis:eu-west-1:123456789012:stream/ExampleStream"
        }
    ]
}

Create an access key for this user. Copy the access key and secret, you need it in the following step.
Create an inbound connector for Amazon Kinesis in Blockbax and validate if an info log appears that it is able to connect successfully. This can take up to 30 seconds.

Make sure your data is properly time ordered and has proper sharding if applicable, meaning data of one asset (i.e. subject) is always consumed from the same shard. Contact us if you have question about this.

Stream data from Blockbax to AWS

Please contact us if you require this.