Refactor kafka to pure Go (franz-go), fix DBC stubs, update Dockerfile

This commit is contained in:
Chris Rai
2026-01-31 00:05:47 -05:00
parent fbb820d7b3
commit b5bec57dfa
776 changed files with 18945 additions and 2052 deletions

83
pkg/kafka/README.md Normal file
View File

@@ -0,0 +1,83 @@
# Kafka Package
Pure Go Kafka client using [franz-go](https://github.com/twmb/franz-go).
## Why franz-go?
- Pure Go - no CGO, no librdkafka dependency
- Builds to static binaries, works with scratch/distroless images
- Well-maintained, performant, full Kafka protocol support
## Usage
### Producer
```go
producer, err := kafka.NewProducer(ctx)
if err != nil {
return err
}
defer producer.Close()
// Sync produce
err = producer.Produce("my-topic", "key", myPayload, nil)
// Async produce
err = producer.ProduceToChannel("my-topic", "key", myPayload)
```
### Consumer
```go
consumer, err := kafka.NewConsumer("my-service")
if err != nil {
return err
}
defer consumer.Stop()
// With handler
consumer.Consume([]string{"topic1", "topic2"}, func(key, value []byte) error {
// process message
return nil
})
// Or with channel
events := make(chan *kafka.Message)
go consumer.ConsumeToChannel([]string{"topic1"}, events)
for msg := range events {
// process msg
}
```
### Admin
```go
// Ensure topics exist (creates if missing)
kafka.EnsureTopicsExist([]string{"topic1", "topic2"})
```
## Configuration
All config via environment variables:
| Variable | Default | Description |
|----------|---------|-------------|
| `KAFKA_HOSTS` | `localhost:9092` | Broker addresses (comma-separated) |
| `KAFKA_SECURITY_PROTOCOL` | `plaintext` | `plaintext`, `ssl`, `sasl_plaintext`, `sasl_ssl` |
| `KAFKA_SASL_MECHANISMS` | `` | `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` |
| `KAFKA_SASL_USERNAME` | `` | SASL username |
| `KAFKA_SASL_PASSWORD` | `` | SASL password |
| `KAFKA_SESSION_TIMEOUT_MS` | `45000` | Consumer session timeout |
| `KAFKA_HEARTBEAT_INTERVAL_MS` | `3000` | Consumer heartbeat interval |
| `KAFKA_ENABLE_AUTO_COMMIT` | `true` | Auto-commit offsets |
| `KAFKA_AUTO_COMMIT_INTERVAL_MS` | `5000` | Auto-commit interval |
| `KAFKA_TIMEOUT` | `10000` | Poll timeout (ms) |
| `KAFKA_TOPIC_PARTITIONS` | `1` | Default partitions for new topics |
| `KAFKA_TOPIC_REPLICATION_FACTOR` | `1` | Default replication factor |
## Migration from confluent-kafka-go
The interfaces are compatible. Main changes:
- `*kafka.Message` → `*kafka.Message` (our wrapper type)
- `kafka.TopicPartition` → `kafka.TopicPartition` (our type)
- No more CGO build tags or librdkafka deps