Kafka

Fundamentals

(With an emphasis on fun...and another emphasis on mental)

Consumer Groups

• Each consumer in a group can be assigned [0 -> p] partitions

• Each partition can only be assigned to a single consumer at a time

• partition -> consumer assignment is configurable

• Provides automatic load balancing of partitions and consumers in a group

Useful Commands

Produce Messages with Keys

./kafka-console-producer.sh --broker-list localhost:9092 \
  --topic my_topic \
  --property "parse.key=true" --property "key.separator=:" < input.txt

Get Earliest Offset in Topic

./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic my_topic --time -2

Additional Reading

• The Kafka Protocol
• https://github.com/wurstmeister/kafka-docker/wiki/Connectivity