I have always been interested in learning more about consumer applications like building a chat room, this weekend, I managed to build a chat room using Kafka and Python flask just in a few hours.

The technology stack is very simple. It is primarily using Pytho Flask, very limited HTML and JavaScript. I had to use Redit to get the flask SSE working and Kafka as the message queue system, that both of those two components requires close to minimal set up.

The architecture is very straightforward too. When the client entered a message and clicked submit, a JavaScript function will be trigger, pass on the username and message to a python flask endpoint. There must be a JavaScript library to directly connect to Kafka but I feel it is cleaner to keep JavaScript separated from the backend like database directly, rather to hit the middleware API endpoints. That will for sure free the clients from having visibility into any backend structure from the security perspective, also created some flexibility of any modification to the backend without impacting any front end work. Then the Flask Sendmsg endpoint will use the python kafka library to produce a message into Kafka.

As you can tell from a consumer who has been subscribing to the topic, all the messages are now in the queue. The idea is whenever there is a new message, there is a mechanism to detect and then “broadcast” or “publish” to all the “subscribers”, or all corresponding clients who happened to be in the same chat room.

At this moment, I don’t know how to create a long running process within Flask so I created a small Python consumer running in another script. It listens to the topic and push any new message to another Flask endpoint called “broadcast”.

The broadcast endpoints does nothing more than taking in the new message, but it will publish it via server side event. There are plenty of benefits of using server side event comparing with polling or sometimes even a open websocket.

Now, all the clients have an event listener to the SSE stream. The any received new message from the stream will trigger the newly received message be appended to the chat history.

More work can be done around the Kafka part as the user’s chat history could be retrieved from a database or we use Kafka as the database directly. And the offset will determine to how long a history that we will go back.

Anyway, it is a pretty fun project overall. 🙂