Kafka

​Apache Kafka is a distributed streaming platform. It allows you to create publishers, which create data streams, and consumers, which subscribe to and ingest the data streams produced by publishers.
You can use OmniSciDB KafkaImporter C++ program to consume a topic created by running Kafka shell scripts from the command line. Follow the procedure below to use a Kafka producer to send data, and a Kafka consumer to store the data, in OmniSciDB.
This example assumes you have already installed and configured Apache Kafka. See the Kafka website.
Creating a Topic
Create a sample topic for your Kafka producer.
1.Run the kafka-topics.sh script with the following arguments:
1
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1
2
--partitions 1 --topic matstream
Copied!
2.Create a file named myfile that consists of comma-separated data. For example:
1
michael,1
2
andrew,2
3
ralph,3
4
sandhya,4
Copied!
3.Use omnisql to create a table to store the stream.
1
create table stream1(name text, id int);
Copied!
Using the Producer
Load your file into the Kafka producer.
1.Create and start a producer using the following command.
1
cat myfile | bin/kafka-console-producer.sh --broker-list localhost:9097
2
--topic matstream
Copied!
Using the Consumer
Load the data to OmniSciDB using the Kafka console consumer and the KafkaImporter program.
1.Pull the data from Kafka into the KafkaImporter program.
1
/home/omnisci/build/bin/KafkaImporter stream1 omnisci -p HyperInteractive -u omnisci --port 6274 --batch 1 --brokers localhost:6283  
2
--topic matstream --group-id 1
3
​
4
Field Delimiter: ,
5
Line Delimiter: \n
6
Null String: \N
7
Insert Batch Size: 1
8
1 Rows Inserted, 0 rows skipped.
9
2 Rows Inserted, 0 rows skipped.
10
3 Rows Inserted, 0 rows skipped.
11
4 Rows Inserted, 0 rows skipped.
Copied!
2.Verify that the data arrived using omnisql.
1
omnisql> select * from stream1;
2
name|id
3
michael|1
4
andrew|2
5
ralph|3
6
sandhya|4
Copied!

Loading and Exporting Data - Previous

Supported Data Sources

Using Immerse Data Manager

Last modified 11mo ago

Export as PDF

Copy link

Contents

Creating a Topic

Using the Producer

Using the Consumer