Kafka

Consumes messages from a Kafka topic.

Configuration

source:
  kind: kafka
  brokers: "localhost:9092"
  topic: events
  group_id: skippr-consumer

Field	Default	Description
brokers	(required)	Kafka bootstrap servers
topic	(required)	Topic to consume
group_id	auto-generated	Consumer group ID
auto_offset_reset	earliest	earliest or latest
security_protocol		Security protocol
sasl_mechanism		SASL mechanism
sasl_username / sasl_password		SASL credentials
mode	stream	stream or batch
idle_timeout_seconds	5	Batch mode idle timeout

Namespace: kafka.{topic}

CLI

skippr connect source kafka \
  --brokers localhost:9092 \
  --topic events \
  --group-id skippr-consumer

Flag	Description
--brokers	Kafka bootstrap servers
--topic	Topic to consume
--group-id	Consumer group ID
--mode	Consumption mode

CDC

Kafka supports real-time Change Data Capture via Debezium envelope parsing. Add cdc_enabled: true to the source config and configure a pipeline cdc: block.

See CDC Sources -- Kafka for prerequisites, configuration, and resume behavior.