Redpanda Connect (Benthos)

Integration guide​

This guide shows the steps to use the QuestDB Kafka connector to read JSON data from Kafka topics and write them as rows into a QuestDB table. For Confluent users, please check the instructions in the Confluent Docker images.

Prerequisites​

You will need the following:

Download Redpanda Connect​

The QuestDB output component was added to Redpanda Connect in version X.X.X

To download the latest version of Redpanda Connect, follow the installation instructions in the official documentation.

Configure Redpanda Connect​

One of Redpanda Connect's strengths is the ability to configure an entire data pipeline in a single yaml file. We will create a simple configuration to demonstrate the QuestDB connector's capabilities by using a straightforward input source.

Create this file and name it config.yaml in your current directory

input:
  stdin: {}

output:
  questdb:
    address: localhost:9000
    table: redpanda_connect_demo
    doubles:
      - price
    designated_timestamp_field: timestamp

This configuration will read lines from stdin and publish them to your running QuestDB instance

Run Redpanda Connect and publish messages​

Run the following command to send some messages over Influx Line Protocol to QuestDB through Redpanda Connect

echo '{"symbol": "AAPL", "price": 225.83, "timestamp": 1727294094}
{"symbol": "MSFT", "price": 431.78, "timestamp": 1727294142}` | rpk connect run config.yaml

Verify the integration​

Navigate to the QuestDB Web Console at http://localhost:9000 and run the following query to see your data:

SELECT *
FROM redpanda_connect_demo