Skip to content

feat: Table partitioning for high-volume participant/event/audit tables #567

Description

@joelpeace48-cell

Background & current state

High-volume tables (participants, indexed events, audit logs) grow unbounded across many campaigns. Large monolithic tables degrade query latency and bloat indexes. No partitioning exists.

Goal

Partition high-volume tables (by campaign_id and/or time range) so hot queries use partition pruning and old data can be archived/dropped cheaply.

Technical design

  • Postgres declarative partitioning: PARTITION BY LIST (campaign_id) or RANGE (created_at) for events/audit; automated partition creation for new ranges.
  • Migrations to convert existing tables (create partitioned table, backfill, swap) with zero/low downtime.
  • Validate plans with EXPLAIN ANALYZE (pruning confirmed).

Edge cases

  • Existing data migration → online backfill + swap; document downtime.
  • Cross-partition queries → ensure indexes support them.
  • Partition explosion (too many campaigns) → use hash sub-partitioning or time-based for events.
  • Retention: drop old time partitions instead of DELETE.

Task breakdown

  • Choose partition keys per table; design migration.
  • Migrations (create/backfill/swap) + auto-partition creation.
  • Retention via partition drop.
  • EXPLAIN-based pruning tests.

Acceptance criteria

  • Hot queries use partition pruning (EXPLAIN confirms).
  • Migration round-trips without data loss.

Testing & verification

  • EXPLAIN ANALYZE assertions in tests; migration up/down round-trip.

Out of scope

  • Replicas (NEW-034).

Dependencies / related

  • Complements NEW-024 (events), NEW-034.

Difficulty: medium · Effort: M · infra + performance

Metadata

Metadata

Assignees

Labels

Stellar WaveIssues in the Stellar wave programarea: backendBackend API (Node/Express)difficulty: mediumModerate complexityenhancementNew feature or requestinfraDeployment, docker, runtimeperformancePerf and scalability

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions