Skip to content

Add Amazon DynamoDB sink connector #3731

Description

@ethanlin01x

Description

Add a native Rust sink connector for Amazon DynamoDB, allowing Iggy stream messages to be written directly into DynamoDB tables.

This is tracked in the connector ecosystem roadmap: #2753

Affected area / component

Connectors

Proposed solution

Crate: aws-sdk-dynamodb
(official AWS SDK, async/Tokio compatible)

Key design decisions:

  1. Batchingbatch_write_item has a hard limit of 25 items per request. The consume() handler will split incoming message batches into chunks of ≤25 before sending.

  2. Idempotency — The connector will support a configurable partition_key_field (and optional sort_key_field) mapped from message payload. batch_write_item uses PutRequest which overwrites
    existing items on key conflict, making repeated delivery idempotent as long as the primary key is deterministic.

  3. UnprocessedItems handlingbatch_write_item can return partial failures via UnprocessedItems without throwing an error. The connector will retry unprocessed items with exponential backoff rather than silently dropping them.

  4. Throughput errorsProvisionedThroughputExceededException will be handled with retry logic compatible with both Provisioned and On-Demand capacity modes.

  5. Item size limit — DynamoDB enforces a 400KB per-item limit. Oversized messages will be logged and skipped rather than failing the entire batch.

Alternatives considered

No response

Contribution

  • I'm willing to submit a pull request to implement this feature

Good first issue

  • I think this could be a good first issue for a new contributor

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions