Payload conversion

The payload conversion is a script that is used to transform an arbitrary received payload to measurements that are ingested in the Blockbax Platform. A payload conversion has an expected payload format and is defined as a plain JavaScript function.

Payload formats

Currently, four different payload formats are supported by Inbound Connectors. Payloads must always be sent to the inbound connector as bytes, but the format specifies how these bytes are decoded and passed to the payload conversion function.

Payload conversion script

The payload conversion script is a user-defined script that is used to transform the decoded payload to measurements that are to be ingested by the Blockbax Platform. Additionally, log messages can be generated to provide feedback to the user from the payload conversion script.

The payload conversion is defined as a JavaScript (ECMAScript 2022 compliant) function with required signature convertPayload(payload, context) { ... }. The web client provides a convenient editor with auto-completion and method signatures on hover. An example script is shown below:

1
2
3
4
5
6
7

function convertPayload(payload, context) {
  const ingestionIdPrefix = payload.id + "$";
  const timestamp = date(payload.timestamp);
  for (const [key, value] of Object.entries(payload.data)) {
    context.addMeasurement(ingestionIdPrefix + key, value, timestamp);
  }
}

This script takes a JSON type payload and uses a top level property for prefixing the ingestion ids. The timestamp for all measurements contained by the payload is parsed from the top level object as well. The script then iterates over all key value pairs in the data field to ingest them as measurements.

Payload parameter

The payload parameter is the decoded payload sent to the inbound connector. This parameter can be either a JavaScript object, a string or an ArrayBuffer. The type of the payload depends on the specified payload format. The payload object is immutable and will throw a TypeError on modification.

Context parameter

The context parameter is an object used to pass measurements and logs to the Blockbax Platform. The context parameter cannot be modified. Its functionalities are described below:

Optionally when the MQTT subscribe connector is selected, it will also have the following functionality:

Library functions

Several library functions are available when writing your conversion function to make writing a conversion easier and more concise. The available functions are described below.

Limits

Every payload received by an inbound connector is processed separately. A payload is a single message as received by the inbound connector, for example one HTTP request, one MQTT message, one Kafka record, one Azure Event Hub event or one Amazon Kinesis record. Messages that are read from a stream in batches are still processed one by one. The following limits apply to each payload, for both default and custom inbound connectors:

For default inbound connectors, which expect the default format, the measurement limit applies to the number of measurements in the payload. For custom inbound connectors, it applies to the measurements created by the payload conversion script, i.e. the number of times context.addMeasurement is called while converting one payload. What counts is how many measurements the script creates, not how much data the payload contains. For example, if a payload contains a list of 600 sensor readings and the script creates a measurement for each of them, the payload is rejected. A large payload from which the script creates only a few measurements is accepted, as long as it does not exceed the size limit.

Make sure that the system sending data to Blockbax keeps each message within these limits. This is especially relevant if it groups data before sending it, for example when it collects readings and sends them together in one message, or when it sends everything it held back during a connection loss at once. In that case, make sure it splits the data over multiple messages.

Testing payload conversion scripts

Payload Conversion scripts can be tested in the Blockbax Web Client. Depending on the payload type a test payload can be provided, and the conversion script is then executed for that payload. For the JSON, CBOR, String and Avro types, a string can be provided as payload. For the CBOR and Bytes type, a hex string can be provided. The test payload can be saved with the conversion script. This allows doing basic regression testing when making changes to the script and verifying that the output measurements are correct for that payload. When the conversion script is executed for the payload the web client will show the measurements and logs that were produced by the payload conversion. Please note that this is only a visual representation of the outcome of the payload conversion, no actual measurements are ingested. Any problems with the script or resulting measurements are shown as log messages.