NODE JS Kafka Producer/Subscriber

What is Kafka:

Kafka is a publish/subscribe based messaging system. In other words its an event streaming platform by which we can write/read events streams. There are lot of libraries in node js which we can use to write producer/subscriber for our application. 'kafkajs' is one of them.

First step is to install kafkajs in our node js application

npm install kafkajs

import kafkajs in your app.js file

const { Kafka } = require('kafkajs')

Now define kafka configuration like kafka listener/server. In my case it is 'YOURDESKTOP-TEST.mshome.net'

const kafka = new Kafka({
  brokers: ['YOURDESKTOP-TEST.mshome.net:9092']
});

Now create producer instance using kafka client

const producer = kafka.producer();

Now to produce a message to defined topic using producer instance.

const sendMessage = async () => {
    await producer.connect()
    await producer.send({
      topic: 'test',
      messages: [
        { key: 'name', value: 'Ashish K' }
      ],
    })
    await producer.disconnect()
  }
sendMessage();

We are done with producer. Now it's time to write subscriber. Subscribers read messages from Kafka topics by subscribing to topics.

const receiveMessage = async () => {
  await consumer.connect()
  await consumer.subscribe({ topic: 'test' })

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log({
        value: message.value.toString(),
      })
    },
  })
};
receiveMessage();