Maintenance inception

[newren]

During the time between when a repack starts and when it completes,
large repositories can gain thousands of packfiles and tens of thousands
of loose objects that are not part of that repack. Not only does this
mean that the repository immediately needs maintenance again, but git
operations slow down dramatically during the repack -- sometimes to the
point that the service is effectively considered offline.

This series proposes a simple (optional, off by default) solution:

• Doing maintenance during maintenance.

or, more specifically:

• Doing (cheap mere aggregation) maintenance (of new objects and packs)
  during (far more thorough) maintenance (of older objects and packs).

=== Concerns/Objection ===

The description above probably raises several serious concerns; let me
attempt to address them (feedback on whether I've missed any, or
insufficiently addressed some very welcome):

Objection 1: What does "cheap mere aggregation" mean?

It means omitting all of the following:

• delta compression or even delta searching
• reachability checks, and anything related to them
  • commit-graph updates
  • bitmap generation
  • cruft pack generation or updates
• multi-pack index creation or updates

Objection 2: Aggregating packs without delta searches causes bad deltas

Yes, but this series adds a .baddeltas marker file (similar to .keep,
.mtimes, .bitmap, .promisor that already exist) to notify future
pack-objects invocations that two objects within that pack should be
checked for deltas, so the effect does not persist.

(On the flip side, see Trade-off 1)

Objection 3: Repack or its subprocess can fail if packfiles disappear

repack calls pack-objects, which enumerates the existing packs, and
only repacks those (or a subset of those if doing a geometric repack),
and then writes a new pack. The remainder of repack (e.g. multi-pack
index creation, bitmap generation) -- at least up until the
odb_reprepare() call immediately before deleting redundant packs --
only works on this subset of packs. Thus, if we have pack-objects
tell us the packs it is working on and we avoid touching any packs:

• in an existing multi-pack-index
• in the list of packs that repack's pack-objects is working on
• that is written by repack's pack-objects process
• that are covered by other sidecars (mainly '.keep' and '.mtimes', but
  we throw in '.promisor' and '.bitmap' for good measure)

Then we should be safe; this does require modifying pack-objects to:

• let us know exactly what packs it is working on
• create its new pack with a (temporary, see below) .keep file

(We also have pack-objects notify us which loose objects exist at that
time, even though that is not needed for correctness, but does allow
us to generally avoid putting a loose object into two new packs.)

Objection 4: The .keep created by pack-objects might persist

It is removed at the end of the repack process.

Further, if for some reason the repack process is killed or dies, the
.keep file contains a marker making it clear the keep was meant to be
temporary and has the corresponding PID of the repack process, meaning
that if that PID no longer exists, the .keep file can be cleaned up.
See also the next point.

Objection 5: Will this "cheap" maintenance actually be cheap?

There's an unnecessary reachability walk today which makes this kind
of "cheap" maintenance somewhat expensive. Patch 3 removes it and
makes a typical case 500x faster and take 50x less memory.

That's fast enough that I not only run "cheap" maintenance during
"real" maintenance, I also run a "cheap" maintenance step before "real"
maintenance. The reasons for this inserted preparatory step are:

• it allows other git operations to be faster while waiting for
  the "real" maintenance to complete
• it reduces the number of packfiles and loose objects that need
  to be tracked for exclusion
• the preparatory step can look for leftover ".keep" files from a
  previous repack process that died early and remove them

=== Important tradeoffs ===

Trade-off 1: need to repeat some delta checks

Two objects being in the same pack that aren't a delta against each
other usually means a previous process has checked and determined
those two objects are not a good delta. Any such objects placed into
a .baddeltas pack will lose that information and we'll have to
re-check for deltas.

Trade-off 2: mtime delay for expiring objects

When we repack loose objects into a (non-cruft) pack, that effectively
updates their mtime when they could have been ready to expire. If
these objects are unreachable, then the "real" maintenance's
reachability check will determine that and either unpack these objects
or move them to a cruft pack. Either way, they'll be stamped with the
pack's mtime they were removed from. For large repositories where we
essentially continuously run maintenance all day long, this can only
add a delay equal to how long a maintenance run takes and is thus
considered inconsequential.

=== Series overview ===

Patch 1: Add support for a ".baddeltas" marker for packs, in try_delta()
Patch 2: Support --mark-bad-deltas option in pack-objects
Patch 3: Accelerate pack-objects when delta search is off
Patch 4: Add --emit-input-{packs,loose} plumbing options to pack-objects

This patch is critical, because we need to know what packs the
regular repacking will be working with so we avoid them.

Trying to enumerate the packs outside of repack/pack-objects risks a
race with newly pushed packs, and in large repositories which push
multiple times per second, losing that race is quite likely.

Patch 5: New pack-aggregate builtin for cheap object/pack aggregation
Patch 6: Allow repack to spawn pack-aggregate while doing regular repacking