Description
Add a native Rust sink connector for Amazon DynamoDB, allowing Iggy stream messages to be written directly into DynamoDB tables.
This is tracked in the connector ecosystem roadmap: #2753
Affected area / component
Connectors
Proposed solution
Crate: aws-sdk-dynamodb
(official AWS SDK, async/Tokio compatible)
Key design decisions:
-
Batching — batch_write_item has a hard limit of 25 items per request. The consume() handler will split incoming message batches into chunks of ≤25 before sending.
-
Idempotency — The connector will support a configurable partition_key_field (and optional sort_key_field) mapped from message payload. batch_write_item uses PutRequest which overwrites
existing items on key conflict, making repeated delivery idempotent as long as the primary key is deterministic.
-
UnprocessedItems handling — batch_write_item can return partial failures via UnprocessedItems without throwing an error. The connector will retry unprocessed items with exponential backoff rather than silently dropping them.
-
Throughput errors — ProvisionedThroughputExceededException will be handled with retry logic compatible with both Provisioned and On-Demand capacity modes.
-
Item size limit — DynamoDB enforces a 400KB per-item limit. Oversized messages will be logged and skipped rather than failing the entire batch.
Alternatives considered
No response
Contribution
Good first issue
Description
Add a native Rust sink connector for Amazon DynamoDB, allowing Iggy stream messages to be written directly into DynamoDB tables.
This is tracked in the connector ecosystem roadmap: #2753
Affected area / component
Connectors
Proposed solution
Crate:
aws-sdk-dynamodb(official AWS SDK, async/Tokio compatible)
Key design decisions:
Batching —
batch_write_itemhas a hard limit of 25 items per request. Theconsume()handler will split incoming message batches into chunks of ≤25 before sending.Idempotency — The connector will support a configurable
partition_key_field(and optionalsort_key_field) mapped from message payload.batch_write_itemusesPutRequestwhich overwritesexisting items on key conflict, making repeated delivery idempotent as long as the primary key is deterministic.
UnprocessedItems handling —
batch_write_itemcan return partial failures viaUnprocessedItemswithout throwing an error. The connector will retry unprocessed items with exponential backoff rather than silently dropping them.Throughput errors —
ProvisionedThroughputExceededExceptionwill be handled with retry logic compatible with both Provisioned and On-Demand capacity modes.Item size limit — DynamoDB enforces a 400KB per-item limit. Oversized messages will be logged and skipped rather than failing the entire batch.
Alternatives considered
No response
Contribution
Good first issue