Maintenance inception

Documentation/config/repack.adoc

@@ -64,3 +64,13 @@ repack.midxNewLayerThreshold::
 When the tip layer has fewer packs than this threshold, those packs are
 excluded from the geometric repack entirely, and are thus left
 unmodified. Must be at least 1. Defaults to 8.
+
+repack.aggregate::
+	If set to true, linkgit:git-repack[1] will spawn a background
+	linkgit:git-pack-aggregate[1] for the duration of its main
+	pack-objects run.  The aggregator rolls up small packs and
+	loose objects that arrive after pack-objects has enumerated
+	its inputs, preventing them from accumulating in
+	`objects/pack/` and slowing other Git operations on busy
+	servers.  Defaults to false.  Can be overridden on the
+	command line with `--aggregate` or `--no-aggregate`.

Documentation/git-pack-aggregate.adoc

@@ -0,0 +1,115 @@
+git-pack-aggregate(1)
+=====================
+
+NAME
+----
+git-pack-aggregate - Periodically roll up loose objects and small packs
+into a single pack without doing any delta compression or search
+
+SYNOPSIS
+--------
+[verse]
+'git pack-aggregate' (--once | --loop) [--interval=<seconds>]
+                   [--min-loose=<n>] [--min-packs=<n>]
+                   [--exclude-pack-file=<path>]
+                   [--exclude-loose-file=<path>]
+                   [--parent-pipe-fd=<n>]
+
+DESCRIPTION
+-----------
+
+`git pack-aggregate` watches `$GIT_DIR/objects` for loose objects and
+small packfiles that have accumulated since the last cycle, and
+periodically rolls them up into a single new pack.  Each cycle has two
+steps:
+
+1. Bundle local loose objects (minus any listed in
+   `--exclude-loose-file`) into a new pack and unlink the loose copies.
+2. Aggregate small local packs (minus any listed in
+   `--exclude-pack-file`, or those referenced by the multi-pack-index,
+   or those carrying a `.keep`, `.promisor`, `.mtimes`, or `.bitmap`
+   sidecar) into another new pack and unlink the source packs.  The
+   pack written in step 1 is naturally a candidate here and will
+   normally be folded into the step-2 output.
+
+No delta search is performed, no bitmaps or commit-graphs are updated,
+and no objects are recompressed; the existing on-disk representation
+of each object is copied through unchanged.  Both new packs are
+written with `pack-objects --window=0 --mark-bad-deltas`, so each one
+ships with a `.baddeltas` sidecar (see linkgit:gitformat-pack[5]).  A
+subsequent thorough repack (linkgit:git-repack[1]) honors that marker
+and reconsiders intra-pack deltas at that time.
+
+The purpose of this command is to keep `objects/` from accumulating
+large numbers of loose objects or small packs while a long-running
+repack is in flight, so unrelated `git` operations are not slowed down
+by readdir cost in a sprawling object directory.  Used together with
+`repack.aggregate` (see linkgit:git-repack[1]), `git repack` will spawn
+this command for the duration of its own work.
+
+`git pack-aggregate` takes no locks of its own.  It mirrors
+linkgit:git-repack[1] in this regard: callers that need serialization
+must arrange it themselves (for example by running under `git gc`).
+The exclusion files supplied via `--exclude-pack-file` and
+`--exclude-loose-file` are the only mechanism by which a concurrent
+`pack-objects` or `git repack` declares "do not touch these inputs".
+
+OPTIONS
+-------
+
+--once::
+	Run a single cycle and exit.  Exactly one of `--once` or
+	`--loop` is required.
+
+--loop::
+	Run cycles forever, sleeping `--interval` seconds between
+	cycles.  Stops on `SIGTERM`, `SIGHUP`, or `SIGINT`.
+
+--interval=<seconds>::
+	Number of seconds to sleep between the end of one cycle and the
+	start of the next.  Defaults to 60.  Only meaningful with
+	`--loop`.
+
+--min-loose=<n>::
+	Skip step 1 if fewer than `<n>` loose objects (after applying
+	`--exclude-loose-file`) are present.  Defaults to 5.
+
+--min-packs=<n>::
+	Skip step 2 if fewer than `<n>` aggregatable packs (after
+	applying all exclusions) are present.  Defaults to 5.
+
+--exclude-pack-file=<path>::
+	Read a list of pack basenames (one per line) from `<path>` and
+	never touch any of those packs.  Lines may name a basename with
+	or without a `.pack` or `.idx` suffix; the suffix is stripped.
+	Blank lines and lines beginning with `#` are ignored.  When
+	`git pack-aggregate` is spawned by a long-running `git repack`,
+	this file is populated by that `pack-objects` itself via
+	`--emit-input-packs`, and may contain a conservative superset
+	of the packs it will actually consume (such as in `--geometric`
+	mode).
+
+--exclude-loose-file=<path>::
+	Read a list of loose object IDs (one per line) from `<path>`
+	and never pack or unlink any of those loose objects.  Blank
+	lines and lines beginning with `#` are ignored.  When `git
+	pack-aggregate` is spawned by a long-running `git repack`, this
+	file is populated by that `pack-objects` itself via
+	`--emit-input-loose`.
+
+--parent-pipe-fd=<n>::
+	An inherited pipe file descriptor whose write end the parent
+	process holds open.  When the parent exits the pipe is closed
+	and `git pack-aggregate` exits at the next cycle boundary,
+	without waiting out the remainder of `--interval`.  This is an
+	internal plumbing option used by `git repack` to keep its
+	companion aggregator from outliving it.
+
+SEE ALSO
+--------
+linkgit:git-repack[1], linkgit:git-pack-objects[1],
+linkgit:gitformat-pack[5]
+
+GIT
+---
+Part of the linkgit:git[1] suite

Documentation/git-pack-objects.adoc

@@ -143,6 +143,14 @@ options which imply `--revs`.
 	have an mtime older than `<approxidate>`. If unspecified (and
 	given `--cruft`), then no objects are eliminated.
 
+--mark-bad-deltas::
+	Write a `.baddeltas` marker file alongside each output pack.  The
+	marker signals that objects within the pack have not been fully
+	delta-searched against other objects within the same pack and
+	that future repacking should consider them.  Any deltas that do
+	exist within this pack can still be reused, however.
+	Incompatible with `--stdout`.
+
 --window=<n>::
 --depth=<n>::
 	These two options affect how the objects contained in

Documentation/git-repack.adoc

@@ -12,6 +12,7 @@ SYNOPSIS
 'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [-b] [-m]
 	[--window=<n>] [--depth=<n>] [--threads=<n>] [--keep-pack=<pack-name>]
 	[--write-midx[=<mode>]] [--name-hash-version=<n>] [--path-walk]
+	[--[no-]aggregate]
 
 DESCRIPTION
 -----------
@@ -300,6 +301,16 @@ created for any new pack(s) without disturbing the existing chain.
 	Pass the `--path-walk` option to the underlying `git pack-objects`
 	process. See linkgit:git-pack-objects[1] for full details.
 
+--aggregate::
+--no-aggregate::
+	Spawn a background linkgit:git-pack-aggregate[1] for the
+	duration of the main pack-objects run.  The aggregator rolls
+	up small packs and loose objects that arrive after
+	pack-objects has enumerated its inputs, preventing them from
+	accumulating in `objects/pack/` and slowing other Git
+	operations on busy servers.  Overrides the
+	`repack.aggregate` configuration variable.  Off by default.
+
 CONFIGURATION
 -------------
 
@@ -324,6 +335,7 @@ SEE ALSO
 --------
 linkgit:git-pack-objects[1]
 linkgit:git-prune-packed[1]
+linkgit:git-pack-aggregate[1]
 
 GIT
 ---

Documentation/gitformat-pack.adoc

@@ -12,6 +12,7 @@ SYNOPSIS
 $GIT_DIR/objects/pack/pack-*.{pack,idx}
 $GIT_DIR/objects/pack/pack-*.rev
 $GIT_DIR/objects/pack/pack-*.mtimes
+$GIT_DIR/objects/pack/pack-*.baddeltas
 $GIT_DIR/objects/pack/multi-pack-index
 
 DESCRIPTION
@@ -357,6 +358,32 @@ All 4-byte numbers are in network byte order.
     and a checksum of all of the above (each having length according
     to the specified hash function).
 
+== pack-*.baddeltas files
+
+The optional `.baddeltas` file is an empty marker sitting alongside a
+`pack-*.pack` (and its `.idx`).  It signals to `git pack-objects` that
+the delta layout of the pack should not be trusted: even when two
+objects appear together in the same pack and neither is stored as a
+delta, the next packing run should still call out to its delta search
+routine for the pair instead of assuming a prior pack-objects already
+considered (and rejected) the pair.
+
+This is intended for producers that intentionally skip delta search
+when writing a pack (for example, processes that bulk-import objects
+or aggregate multiple existing packs without recomputing deltas).
+Without this marker, the same-pack delta skip in `git pack-objects`
+would silently inherit those producers' lack of delta search into
+future repacks.
+
+The contents of the file are currently ignored.  Producers should
+write an empty file; consumers must tolerate (and ignore) any
+content.
+
+The marker only affects whether `git pack-objects` will attempt to
+compute new deltas for object pairs that share the marked pack.  It
+does not disable reuse of existing on-disk deltas, nor does it affect
+multi-pack-index, bitmap, or pack reuse for transfer.
+
 == multi-pack-index (MIDX) files have the following format:
 
 The multi-pack-index files refer to multiple pack-files and loose objects.

Documentation/meson.build

@@ -99,6 +99,7 @@ manpages = {
   'git-name-rev.adoc' : 1,
   'git-notes.adoc' : 1,
   'git-p4.adoc' : 1,
+  'git-pack-aggregate.adoc' : 1,
   'git-pack-objects.adoc' : 1,
   'git-pack-refs.adoc' : 1,
   'git-patch-id.adoc' : 1,

Makefile

@@ -1450,6 +1450,7 @@ BUILTIN_OBJS += builtin/multi-pack-index.o
 BUILTIN_OBJS += builtin/mv.o
 BUILTIN_OBJS += builtin/name-rev.o
 BUILTIN_OBJS += builtin/notes.o
+BUILTIN_OBJS += builtin/pack-aggregate.o
 BUILTIN_OBJS += builtin/pack-objects.o
 ifndef WITH_BREAKING_CHANGES
 BUILTIN_OBJS += builtin/pack-redundant.o

builtin.h

@@ -224,6 +224,7 @@ int cmd_multi_pack_index(int argc, const char **argv, const char *prefix, struct
 int cmd_mv(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_name_rev(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_notes(int argc, const char **argv, const char *prefix, struct repository *repo);
+int cmd_pack_aggregate(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_pack_objects(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_pack_redundant(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_patch_id(int argc, const char **argv, const char *prefix, struct repository *repo);