Apache Kafka is a powerful data streaming platform, but it comes with many limitations. Operating Kafka at petabyte scale requires significant amounts of operational overhead and manual intervention.
In this white paper, we present some of the challenges DevOps teams face implementing Kafka as a data streaming platform. We’ll discuss the fundamental challenges with Kafka’s design, the impact that its constraints have on high-throughput applications, and the difficulty in optimizing Kafka around these constraints.
Our most important concerns in scaling Kafka are:
![]()
|
Scaling downstream consumers efficiently |
![]()
|
Redistributing work between consumers |
![]()
|
Handling unexpected interactions between downstream consumers and live data |
The information in this eBook is based on this webinar presented by Engineering Manager, Muaz Siddiqui.
This white paper assumes a working knowledge of Kafka 2.0.X or higher.
You can find the webinar here.