feat(core): support sort manifest entries by partition

[baiyangtx]

Purpose

Manifest entries are currently written in arrival order, which scatters entries belonging
to the same partition across multiple manifest files. This leads to:

• Inefficient partition-pruning during reads — the reader must scan many manifest files
  to find all entries for a single partition.
• Suboptimal data locality when compaction reorganizes files.

This PR introduces manifest entry sorting by partition, so that entries for the same
partition are clustered together within manifest files.

Changes

New configuration options:

Option	Default	Description
manifest.merge.sorted	true	Sort entries by partition during manifest full compaction
manifest.merge.sort-on-commit	false	Sort entries by partition during manifest full merge in commit
manifest.delta.sorted	true	Sort entries by partition when writing manifest delta

Core implementation:

• ManifestFileMerger: Introduces mergeSortedByPartition() and mergeUnsorted().
  When manifest.merge.sorted is enabled, entries are collected into a
  BinaryExternalSortBuffer (with spill-to-disk support), then written in
  partition-major order: (partition, bucket, level).
• ManifestFileMerger.createManifestEntryComparator(): Comparator used for
  sorting delta manifests, falling back to pure-Java comparison when codegen
  is unavailable.
• FileStoreCommitImpl: Wires sort parameters into all three paths —
  commit manifest merge, delta file writing, and manifest compaction.

Tests

# Manifest merge with sorting
mvn -pl paimon-core -am -DfailIfNoTests=false \
    -Dtest=ManifestFileMetaTest test

# No-partition edge case
mvn -pl paimon-core -am -DfailIfNoTests=false \
    -Dtest=NoPartitionManifestFileMetaTest test

# Spark compact procedure
mvn -pl paimon-spark/paimon-spark-ut -am -DfailIfNoTests=false \
    -Dtest=CompactProcedureTestBase test

[Aitozi]

LGTM +1, Please resolve the conflict

[Aitozi]

CC @JingsongLi for another look

[JingsongLi]

Hi @baiyangtx @Aitozi , please take a look to #7842 , I didn't look closely, maybe the abilities of these two PR implementations are the same.

[baiyangtx]

Hi @baiyangtx @Aitozi , please take a look to #7842 , I didn't look closely, maybe the abilities of these two PR implementations are the same.

Thanks @discivigour for working on this too. My PR #7866 takes a different approach — sorting inline during the existing full compaction path, rather than a separate post-commit sort pipeline. Key differences:

• No new compact type — sorting happens inside ManifestFileMerger.tryFullCompaction, reusing the existing full compaction trigger and lifecycle. No additional manifest scans or rewrite passes.
• Memory-safe with external sort — uses BinaryExternalSortBuffer (already battle-tested across Paimon). When entries exceed manifest.merge.sort.buffer, it automatically spills to disk. No risk of OOM regardless of manifest count.
• Finer-grained config — separate toggles for merge compaction, commit-time merge, and delta writing, instead of a single switch.

Happy to align with the community's direction, but I believe the inline approach is simpler and less invasive.