The Postmortem of a 20-Minute Kafka 3.8 Outage That Delayed 1M Order Messages

The provided code snippet outlines a Java program that demonstrates how to properly configure and use the Kafka producer API for sending messages, especially in scenarios where broker failover or rolling upgrades might occur. The original problematic configuration is contrasted with corrected settings to ensure message delivery reliability.

Problematic Configuration (Original Code)

• Retries: Set to 20, which may not be sufficient during a broker failover window.
• Retry Backoff: Set to 100ms, too short for the Kafka brokers' startup time in newer versions.
• Delivery Timeout: Not explicitly set, leading to potential message loss if delivery is delayed beyond the default timeout.
• Request Timeout: Default or insufficient value (not specified), which might cause issues with slow broker responses.

Corrected Configuration

1. Retries: Set to Integer.MAX_VALUE for maximum retries during failover windows.
2. Retry Backoff: Increased to 500ms, matching the startup time of Kafka brokers in version 3.8 and later.
3. Delivery Timeout: Set to 2 minutes (120 seconds) to cover rolling upgrade scenarios where broker availability might be temporarily compromised.
4. Request Timeout: Increased

Read the full article at DEV Community

---

Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.