Consuming messages from Kafka - decoupling consumption and processing

Common approach for consuming messages from Kafka is to use single thread both for fetching and processing of messages. KafkaConsumer is not thread-safe, so this approach fits in well. But in some use cases, this might not be most optimal way to do it. Another approach is to decouple consumption and processing, so we can achieve processing parallelization with single consumer and get the most out of multithreaded execution environment. While this can boost your application efficiency, it's not trivial to implement. In this talk we'll explore challenges we face when trying to optimize message processing from Kafka.