Kafka
Apache Kafka is a distributed streaming platform. It allows you to create publishers, which create data streams, and consumers, which subscribe to and ingest the data streams produced by publishers.
You can use HeavyDB KafkaImporter C++ program to consume a topic created by running Kafka shell scripts from the command line. Follow the procedure below to use a Kafka producer to send data, and a Kafka consumer to store the data, in HeavyDB.
Create a sample topic for your Kafka producer.
- 1.Run the
kafka-topics.sh
script with the following arguments:bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1--partitions 1 --topic matstream - 2.Create a file named
myfile
that consists of comma-separated data. For example:michael,1andrew,2ralph,3sandhya,4 - 3.Use
heavysql
to create a table to store the stream.create table stream1(name text, id int);
Load your file into the Kafka producer.
- 1.Create and start a producer using the following command.cat myfile | bin/kafka-console-producer.sh --broker-list localhost:9097--topic matstream
Load the data to HeavyDB using the Kafka console consumer and the
KafkaImporter
program.- 1.Pull the data from Kafka into the
KafkaImporter
program./home/heavyai/build/bin/KafkaImporter stream1 heavyai -p HyperInteractive -u heavyai --port 6274 --batch 1 --brokers localhost:6283--topic matstream --group-id 1Field Delimiter: ,Line Delimiter: \nNull String: \NInsert Batch Size: 11 Rows Inserted, 0 rows skipped.2 Rows Inserted, 0 rows skipped.3 Rows Inserted, 0 rows skipped.4 Rows Inserted, 0 rows skipped. - 2.Verify that the data arrived using
heavysql
.heavysql> select * from stream1;name|idmichael|1andrew|2ralph|3sandhya|4