Banish your Apache Kafka FOMO forever with deKaf
Never miss a moment of your Kafka experience with deKaf, an open-source, web-based metrics visualization tool.
Let’s say you’re a software engineer. (If you’re reading this Medium article, the likelihood is pretty high.) You and your team have built something amazing — so amazing, in fact, that your traffic is increasing. A lot. Suddenly, you’ve got tons of users — and more data than you know what to do with. You want to use that data effectively, and you want to act on it as quickly as possible to give your users the most responsive experience possible.
It sounds like you could use event streaming.
The rise of event streaming
Event streaming is one of the most crucial technologies to rise to prominence in the last decade. It’s dramatically changed the way companies analyze and use data by giving them the capability to respond to events in real time. Event streams allow for immediate action on data — they deliver data on an ongoing basis, maintaining a consistent connection between producer and consumer. In a world of instant gratification and on-demand services, companies are finding that event streaming is pivotal to their operations.
And among event streaming solutions, Apache Kafka remains at the front of the pack. Developed at LinkedIn and used by more than 80% of all Fortune 100 companies, Kafka is a powerful, fast, and scalable way to capture, transmit, and store data in real time. If you’ve ever used Uber, Spotify, Slack, or Netflix — and chances are you have — you’ve witnessed the power of Kafka in action.
Okay, so what’s the catch?
Despite the amazing things Kafka can do, it can be hard to actually keep track of what’s going on inside your Kafka instance. And if your company is relying on Kafka for maintenance, real-time data insights and analysis, or a better customer experience, that could be a huge problem for you and your engineering team.
deKaf: a new metrics visualization solution
The solution? deKaf, a new, open-source, web-based monitoring tool for Kafka. deKaf offers an intuitive, user-friendly GUI to easily view all of the key metrics of your Kafka instance in one place, with dynamic visualizations powered by D3. (And it’s coffee-themed. Who doesn’t love coffee?)
How does it work?
Using deKaf is easy. Once you’ve signed in or created an account, you’ll be asked to provide the port where your Kafka server is running, as well as a few details about the Kafka topics you’d like to monitor. You can even choose to test your Kafka instance with random data, to make sure it’s functioning properly.
On the Metrics Overview page, you can easily select which category you’d like to monitor: Topic, Messages, Consumer, or Producer. deKaf will connect to your Kafka instance and render your metrics dynamically via D3, updating regularly to make sure you have access to the latest data.
At launch, deKaf supports the following metrics:
Topic:
- Topics
- Partitions
- Messages per partition
Messages:
- Latest message
- Partition of latest message
- Total messages in consumer
Consumer:
- Total messages received
- Message quantity over time
- Message size
Producer:
- Total messages sent
- Message quantity over time
- Message size
We’re excited to help you power up your Kafka clusters! Stay tuned for more updates on our website and at GitHub — planned future features include additional metrics, improved visualizations, and the ability to view personalized session history. In the meantime, we hope deKaf helps you never miss a beat.
If you’re interested in contributing to deKaf, or if you have questions or feedback, we’d love to hear from you! Check out our repo on GitHub, or feel free to reach out to any of us directly.
The deKaf team is:
Achille Perducat | GitHub | LinkedIn
Billy Hepfinger | GitHub | LinkedIn
Jake Song | GitHub | LinkedIn
Mike Feldman | GitHub | LinkedIn
Noah Mattingly | GitHub | LinkedIn