You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Reduce stored push document size and rollup scan cost across sink backends #1691
Discussion issue, applies to every sink backend (fs, mongo, and the PostgreSQL sink proposed in #1687).
Every push row stores the complete diff twice. getDiff writes it to both steps[].logs (via step.log(diff)) and steps[].content (via step.setContent(diff)), and writeAudit persists the whole action document as-is in all backends. Large pushes therefore produce multi-megabyte rows, and everything that touches push documents pays for it:
The repo activity rollup scans all push rows on every repos page. Indexing can make the scan cheap on PostgreSQL, but it remains O(number of pushes) on every backend, and on mongo/fs it reads the full documents.
List endpoints and exports ship or filter over fat documents.
Storage grows with diff size rather than with metadata size.
Ideas worth weighing, none of which are backend-specific:
Store the diff once, not twice: drop the duplicate from steps[].logs or make setContent and the log reference one value.
Move diff content out of the push document into its own store or table, loaded only by the push-detail view.
A retention or truncation policy for step logs and diff content on old, settled pushes.
Maintain the repo activity rollup incrementally (a counters document or table updated on push transitions) instead of recomputing it by scanning all pushes.
Filing as a design discussion before any implementation; the immediate, index-level mitigations for the PostgreSQL sink are tracked separately.
Activity