Skip to content

kafka.mdx

Latest commit

 

History

History
94 lines (53 loc) · 4.27 KB

kafka.mdx

File metadata and controls

94 lines (53 loc) · 4.27 KB
title Kafka sink
sidebarTitle Kafka sink
description Stream Postgres changes directly to Apache Kafka with Sequin's Kafka sink. Configure message ordering and partitioning for reliable change data capture.

The Kafka sink sends messages to a Kafka topic.

This is the reference for the Kafka sink. See the [quickstart](/quickstart/kafka) for a step-by-step walkthrough or the [how-to guide](/how-to/stream-postgres-to-kafka) for an explanation of how to use the Kafka sink.

Configuration

  • Hosts

    A comma-separated list of host:port pairs for your Kafka brokers (e.g., localhost:9092,localhost:9093).

  • Topic

    The name of the Kafka topic where messages will be sent. Maximum length is 255 characters.

  • Username (optional)

    Username for SASL authentication.

  • Password (optional)

    Password for SASL authentication.

  • SASL Mechanism (optional)

    The SASL mechanism to use for authentication. Can be PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512. This is required if you specify a username and password.

  • TLS

    Enable TLS encryption for connections to Kafka. Currently, Sequin only supports TLS with verify none. If you need a different TLS configuration, please let us know.

  • Compression (optional)

    Choose a compression algorithm to reduce the size of messages sent to Kafka. Options include:

    • None (default): No compression
    • Gzip: Good compression ratio with wide compatibility, uses Erlang's built-in :zlib module

    Compression is applied to batches of messages before sending to Kafka, which can significantly reduce network bandwidth and storage requirements. When using compression, be mindful of the max_batch_size setting to ensure compressed batches don't exceed Kafka's max.message.bytes limit.

Advanced configuration

  • Max Ack Pending

    Sets the maximum number of messages that can be in-flight at any time. This helps control message flow and prevents overwhelming the broker.

Message format

Sequin sends messages to Kafka as JSON. You can find the shape of the messages in the messages reference.

Retry behavior

If Sequin is unable to deliver a message to Kafka, it will retry the message indefinitely. Sequin will exponentially back off the retry interval, with a maximum backoff of roughly 3 minutes.

Message size limits

The maximum message size in Kafka is configured at the broker level. The default maximum message size in Kafka is 1MB. If you need to handle larger messages, you'll need to adjust the message.max.bytes setting on your Kafka brokers.

If Kafka rejects a message due to size limitations, Sequin will cancel the message's delivery.

If you want to see logging/alerting for this situation, please [upvote the corresponding issue](#596).

Message ordering and partitioning

Kafka ensures message ordering within individual partitions. When messages are sent to the same partition, they are guaranteed to be delivered in the order they were produced. Partitioning plays a critical role in maintaining this order.

Sequin leverages Kafka's partitioning model by using the hash of a record's primary key(s) as the default partition key. This strategy ensures that all messages related to the same record are routed to the same partition, preserving the order of updates for that record.

You can override the default partition key by setting the message_key field in your routing function.

Security

Sequin supports the following security features for Kafka connections:

  • TLS encryption
  • SASL authentication, with support for PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512

Make sure your Kafka brokers are properly configured to accept the security settings you enable in Sequin.

Debugging

You can view the status of your Kafka sink in the Sequin web console.

On the "Messages" tab, you can see which messages are in-flight to Kafka, which messages Sequin is unable to deliver, and recently delivered messages.

Messages that Sequin is unable to deliver will have a "Deliver count" greater than 1. You can click on a message to see more details, including the last error response received from Kafka.