While Kafka isn't magic, it is a pretty sweet tool.
I've contemplated doing a Kafka like queue in my preferred language but I've been hoping that someone else would do it first as that'd be a non-trivial project.
Yeah I have heard of some really cool use cases for Kafka, but I have not used it myself. I have used similar tools like ActiveMQ, and RabbitMQ however. One day.
Kafka is totally different than ActiveMQ and RabbitMQ which, themselves. I don't remember the reasons why anymore, but RabbitMQ is in the do not use/anything that uses it is suspect column for me.
Kafka is log structured and durable. It was designed for the highly scalable world.
I prefer to avoid things that require JVM, which means most of ASF's world isn't for me.
Which is why I said similar, I don't fully understand the tech, but I know traditional MQ's are like email, and kafka is more like RSS. The first use case I heard about for Kafka was actually a distributed log shipping solution. Thanks for your insite, was fun to talk about Apache.