Apache Kafka is a powerful data streaming platform, but it comes with many limitations. Operating Kafka at petabyte scale requires significant amounts of operational overhead and manual intervention.
In this white paper, we present some of the challenges DevOps teams face implementing Kafka as a data streaming platform. We’ll discuss the fundamental challenges with Kafka’s design, the impact that its constraints have on high-throughput applications, and the difficulty in optimizing Kafka around these constraints.
Our most important concerns in scaling Kafka are: