Mammoth

Mammoth is a self-hosted PostgreSQL change-event relay focused on reliable delivery of database change events to downstream systems. It turns PostgreSQL logical replication streams into observable, recoverable delivery pipelines.

This repository contains the Mammoth Data Plane, the open-source runtime in the broader Mammoth ecosystem:

Mammoth
├── Mammoth Data Plane
├── Mammoth Control Plane
├── Mammoth Control Agent
└── Mammoth Extensions

PostgreSQL
      ↓
CDC Ecosystem source adapter
      ↓
CDC::Core::TransactionEnvelope
      ↓
Mammoth Data Plane
      ↓
Webhook fanout

Mammoth is intentionally boring infrastructure. It uses YAML configuration, JSON Schema validation, local SQLite operational state, and the CDC Ecosystem's shared vocabulary so operators can inspect, recover, and reason about delivery.

Why Mammoth?

Mammoth is designed for applications where PostgreSQL is the system of record, including Rails applications that need reliable reactions to database state changes without coupling every workflow to request handling.

Applications increasingly need to react to changes in their primary database:

• update search indexes after data changes
• synchronize external systems
• trigger integrations and asynchronous workflows
• build analytics pipelines
• maintain audit and event histories

Many systems start with polling, application callbacks, or database triggers. As applications grow, these approaches can become increasingly difficult to operate, recover from failures, and reason about.

Mammoth uses PostgreSQL logical replication as the authoritative change stream and provides a reliable delivery layer between PostgreSQL and downstream consumers.

The goal is simple: make database change delivery explicit, observable, and recoverable.

What makes Mammoth different?

Mammoth deliberately focuses on the delivery boundary between PostgreSQL and downstream consumers rather than trying to become a general-purpose streaming platform.

It treats change delivery as an operational concern:

• transactional change-event handling
• durable delivery state
• explicit checkpoint management
• replayable failures
• observable runtime behavior

Unlike application callbacks, Mammoth does not require workflows to execute inside request handling. Unlike broad streaming platforms, it provides a focused PostgreSQL-native model for teams that need reliable change delivery without adopting a larger streaming infrastructure stack.

Mammoth ecosystem

The Data Plane is one component of the broader Mammoth ecosystem. The companion Control Plane, Control Agent, and Extensions are developed separately and are not part of this open-source repository or its MIT distribution.

See the Mammoth ecosystem overview for public scope, integration boundaries, and roadmap status. Planned capabilities are directional and do not represent a release commitment.

Reference Implementation

The Spherical Mammoth repository is the production-shaped, cloud-native reference environment for deploying, validating, and operating the Mammoth Data Plane.

It demonstrates how Mammoth integrates into a complete application environment:

Application services
        ↓
PostgreSQL transactional state
        ↓
PostgreSQL logical replication
        ↓
Mammoth Data Plane
        ↓
Webhook delivery
        ↓
Independent downstream consumers

Spherical Mammoth provides:

• a realistic SaaS-style application workflow
• PostgreSQL-backed business state
• Ory Kratos identity lifecycle integration
• polyglot downstream consumers
• Docker Compose and Kubernetes deployment paths
• Helm-based deployment
• OpenTofu-managed cloud infrastructure examples
• CDC validation and failure scenarios

The repository is intentionally separate from Mammoth OSS. Mammoth provides the CDC relay data plane; Spherical Mammoth demonstrates how that data plane can be deployed, exercised, and operated in a cloud-native environment.

See:

• Spherical Mammoth
• Cloud-native deployment guide

Supported PostgreSQL versions

Mammoth supports PostgreSQL 14 through PostgreSQL 18, inclusive. These versions are currently maintained by the PostgreSQL community and covered by Mammoth's logical-replication E2E compatibility matrix.

New PostgreSQL majors are unsupported until explicitly tested and documented. EOL versions may be removed from the supported range in a subsequent Mammoth minor release with release-note notice.

PostgreSQL 19 is a development release and is not supported.

Documentation

Documentation site:

https://kanutocd.github.io/mammoth/

API documentation:

https://kanutocd.github.io/mammoth/Mammoth.html

Start Here

The recommended first-run experience is the webhooks-quickstart stack. One Compose command starts a demo application, logical-replication-enabled PostgreSQL, Mammoth, a signed webhook receiver with visible retries and payload-policy masking, and Mammoth health endpoints:

cd webhooks-quickstart
docker compose up --build --wait

Released tags default to their matching Mammoth image. When running main with Unreleased configuration features, follow the quickstart's local-image instructions.

Once the flow is visible, follow its ADAPTING.md guide to connect Mammoth to an existing PostgreSQL application. Its optional monitoring profile adds seeded traffic, a provisioned Grafana dashboard and alert rules, and a curated Prometheus query library; see the observability guide.

v1 Release Scope

Mammoth 1.x includes:

• operator CLI for validation, bootstrap, status, delivery, observability, and dead-letter workflows
• YAML configuration loading
• JSON Schema-backed configuration validation
• SQLite operational memory bootstrap
• checkpoint persistence
• dead-letter persistence
• delivered-envelope ledger persistence
• webhook delivery sink
• webhook fanout to multiple destinations
• fanout route filters by schema, table, and operation
• per-destination payload removal and masking policies
• per-destination enable/disable and retry policy controls
• delivery worker with retry, delivered-ledger, and DLQ handling
• configurable debug, info, warn, and error JSON logging to stdout
• contiguous delivery watermark for checkpoint and PostgreSQL acknowledgement
• source-owned transport LSN preservation independent of payload commit_lsn
• fail-closed PostgreSQL slot and checkpoint continuity preflight
• fail-closed publication replica-identity preflight for UPDATE and DELETE
• dead-letter inspection and filtered replay commands
• CDC-core event serialization boundary
• CDC Ecosystem source-adapter integration boundary
• Docker image support
• public Helm chart support
• unit and e2e test tasks
• health, PostgreSQL slot readiness, and retained-WAL metrics endpoints
• canonical CDC dispatch counters through a CDC::Core::Observer
• explicit extension registries for state, destination, and runtime adapters
• node identity and local capability reporting
• lifecycle hooks, configuration providers, and reusable local command objects

Feature Examples

The runnable examples are organized around production behaviors and failure modes, not isolated API snippets.

Example	v1 capability demonstrated
live_postgres_webhook	End-to-end PostgreSQL logical replication into webhook delivery.
transaction_webhook	TransactionEnvelope preservation through the concurrent runtime.
webhook_fanout	Routed multi-destination fanout, destination payload masking, environment-backed headers, signing, and independent retry policies.
ordering	Ordered and throughput-oriented transaction scheduling.
checkpoint_recovery	Durable restart recovery, replay suppression, checkpointing, and acknowledgement.
slot_invalidation_recovery	Fail-closed slot invalidation and explicit operator reconciliation.
composite_replica_identity	Composite, non-id replica identity across INSERT, UPDATE, and DELETE.
postgres_observability	Slot readiness and Prometheus metrics correlated with PostgreSQL catalogs.
schema_evolution	Consumer-first additive schema evolution without implying DDL delivery.
destination_idempotency	Atomic destination-side duplicate suppression across isolated relay ledgers.
failing_webhook_retry	Retry exhaustion and durable dead-letter persistence.
operational_state	Inspectable checkpoints, delivered ledgers, and dead-letters.
kubernetes_helm	Single-consumer Kubernetes deployment using the public Helm chart.

See examples/README.md for the complete index, boundary notes, and commands.

v1 Compatibility

Mammoth 1.x treats its validated configuration, serialized webhook envelopes, documented CLI command behavior, and forward operational-state migrations as supported contracts. Compatible minor releases may add optional configuration or payload fields, but do not remove or reinterpret existing fields.

Human-readable CLI formatting and PostgreSQL-derived row columns are not frozen: scripts should rely on documented exit behavior, while receivers must tolerate additive fields and coordinate source schema changes. See docs/COMPATIBILITY.md for the complete promise and major-version boundaries. See docs/WEBHOOK-PAYLOADS.md for the canonical event and transaction JSON contracts, column-change semantics, and event-ID behavior.

Boundary

Mammoth begins at CDC Core work items and ends at webhook fanout delivery.

Mammoth does not own pgoutput protocol parsing, value decoding, source normalization, or core dispatch vocabulary. Those responsibilities belong to upstream CDC Ecosystem components. Mammoth selects and composes a delivery runtime while delegating scheduling mechanics to the runtime layer. The runtime registry wraps the selected adapter with configured batch accumulation; Application only streams core work and coordinates lifecycle flush and shutdown calls.

For the live PostgreSQL stream, pgoutput-source-adapter incrementally owns Begin/Commit buffering and emits exact CDC::Core::ChangeEvent or CDC::Core::TransactionEnvelope work items. Mammoth only composes the transport, parser, decoder, and source adapter, then forwards the resulting core work to delivery. Mammoth's publication preflight supplies ordered, catalog-derived replica-identity columns to the adapter, which owns composite and non-id key extraction.

At the downstream boundary, Mammoth::DeliveryProcessor implements CDC::Core::Processor and returns CDC::Core::ProcessorResult. Inline and concurrent runtimes notify a CDC::Core::Observer; Mammoth's default observer maps canonical started, succeeded, failed, and skipped notifications to Prometheus counters.

Mammoth::ReplicationConsumer accepts only CDC Core event and transaction envelope types.

deliver-sample input passes through PersistedPayloadDeserializer before entering the live delivery path, ensuring stored hashes do not masquerade as CDC Core work.

Dead-letter replay is a separate destination boundary: Mammoth sends the exact prepared JSON persisted for that destination and does not reconstruct CDC work or reapply the current payload policy.

Extensions

Mammoth Data Plane exposes small adapter registries for future extensions:

• operational state adapters
• destination adapters
• runtime adapters
• lifecycle hooks
• configuration providers
• local command objects

See docs/EXTENSIONS.md.

Configuration

Mammoth configuration is YAML-backed and IDE-friendly.

# yaml-language-server: $schema=./mammoth.schema.json

Validate configuration:

bundle exec ./exe/mammoth validate config/mammoth.example.yml

Fanout destinations can be routed and tuned independently:

destinations:
  - name: audit_webhook
    type: webhook
    enabled: true
    url: https://audit.example.com/cdc
    timeout_seconds: 5
    route:
      schemas: [public]
      tables: [orders]
      operations: [insert, update]
    payload_policy:
      rules:
        - columns: [customer_email]
          action: mask
    retry:
      max_attempts: 3
      schedule_seconds: [1, 10]

See docs/PAYLOAD-POLICIES.md for deterministic data minimization, retry and dead-letter behavior, and policy boundaries.

CLI

bundle exec ./exe/mammoth version
bundle exec ./exe/mammoth validate config/mammoth.example.yml
bundle exec ./exe/mammoth bootstrap config/mammoth.example.yml
bundle exec ./exe/mammoth status config/mammoth.example.yml
bundle exec ./exe/mammoth start config/mammoth.example.yml
bundle exec ./exe/mammoth observability config/mammoth.example.yml

Logging

Mammoth emits one JSON object per line to standard output. Set logging.level to debug, info, warn, or error; info is the recommended normal setting, while debug adds per-work and WAL acknowledgement records.

logging:
  level: info

In Docker Compose, follow the application, replication, delivery, retry, dead-letter, and checkpoint activity with:

docker compose logs -f mammoth

Operational identifiers are included, but payload bodies, configured headers, credentials, signing secrets, and exception messages are excluded. See docs/CONFIGURATION.md for the event and level details.

Reconstruct and deliver a single persisted event JSON file through Mammoth's core delivery path:

bundle exec ./exe/mammoth deliver-sample \
  examples/postgres_webhook/config/mammoth.yml \
  examples/postgres_webhook/events/order_insert.json

SQLite Operational State

Mammoth stores operational memory in SQLite:

• schema_migrations
• checkpoints
• dead_letters
• delivered_envelopes

SQLite is the built-in default behind operational_state.adapter. Bootstrap, status, observability, and dead-letter commands consume the adapter contract rather than opening SQLite directly.

Performance

A reference benchmark run on Ruby 4.0.5 produced:

Benchmark	Result
Event payload projection with explicit ID	56,265 events/sec
Transaction payload projection with explicit IDs	55,612 events/sec
Mask payload-policy projection	72,403 events/sec
Concurrent delivery, 25 workers	3,825 events/sec
Concurrent delivery, 50 workers	7,447 events/sec
Signed webhook delivery with 10 ms receiver latency	83.5 requests/sec
SQLite delivered-ledger writes	1,630 writes/sec
Dead-letter replay mechanics	1,030 entries/sec

Mammoth processed 1,862 transactions/sec against a calculated workload ceiling of 2,000 transactions/sec, while maintaining 28.4 ms P95 latency. This indicates low scheduling overhead for I/O-bound downstream delivery.

These are local, single-trial reference measurements, not universal capacity claims or comparisons with other CDC platforms. The benchmarks isolate specific Mammoth components and may exclude PostgreSQL transport, decoding, checkpoint coordination, network latency, or destination processing.

The reference snapshot records the exact commands, environment, raw output, and detailed interpretation. Re-run the scripts in benchmark/ on your own hardware when choosing runtime.concurrency, destination count, SQLite storage, scrape frequency, and DLQ replay expectations.

Create a publishable benchmark snapshot with:

bundle exec ruby benchmark/snapshot.rb

E2E

bundle exec rake test:e2e
# or
script/test-e2e

The e2e task uses a real HTTP receiver, real SQLite database, and real filesystem paths. Set MAMMOTH_E2E_POSTGRES_URL to include the real PostgreSQL logical-replication scenarios:

MAMMOTH_E2E_POSTGRES_URL=postgres://postgres:postgres@127.0.0.1:5432/mammoth_e2e \
  bundle exec rake test:e2e

The PostgreSQL fixture must have wal_level=logical; its test role must be able to create publications and logical replication slots, terminate replication backends, and change max_slot_wal_keep_size.

Kubernetes

The public Helm chart lives under:

charts/mammoth

Install example:

helm install mammoth charts/mammoth

The chart uses one replica and Recreate strategy to respect PostgreSQL's logical replication slot constraint: one slot, one active subscriber.

Production operators should also monitor retained WAL and slot readiness, configure PostgreSQL retention guardrails, and alert on database disk and catalog health. DDL and sequence state are not replicated; coordinate schema changes with webhook consumers and synchronize sequences externally when building a writable database copy. See docs/POSTGRESQL.md and docs/TROUBLESHOOTING.md.

License

Mammoth OSS Data Plane is licensed under the MIT License.