Docs

Databricks

Databricks has no dedicated Blockbax connector. Instead, Databricks Structured Streaming can read from and write to the same Kafka topic used by a Blockbax outbound or inbound connector.

Connection guide

Stream measurements from Databricks to Blockbax

  1. Create a Kafka inbound connector in Blockbax to consume measurements from a topic.
  2. In Databricks, write to that topic with the Structured Streaming Kafka connector (writeStream.format("kafka")), producing one message per measurement in the payload format configured on the inbound connector.
  3. Validate if an info log appears that it is able to connect successfully. This can take up to 30 seconds.

Stream measurements from Blockbax to Databricks

  1. Create a Kafka outbound connector in Blockbax to stream measurements and events to a topic.
  2. In Databricks, read from that topic with the Structured Streaming Kafka connector (spark.readStream.format("kafka")).
  3. Parse the measurement or event payload from each Kafka message.