Shallow and concurrent git clones by alt-romes · Pull Request #10254 · haskell/cabal

alt-romes · 2024-08-12T10:40:39Z

Please read Github PR Conventions and then fill in one of these two templates.

Template Α: This PR modifies behaviour or interface

Include the following checklist in your PR:

Patches conform to the coding conventions.
Any changes that could be relevant to users have been recorded in the changelog.
The documentation has been updated, if necessary.
Manual QA notes have been included.
Tests have been added. (Ask for help if you don’t know how to write them! Ask for an exemption if tests are too complex for too little coverage!)

Manual QA: Create or use a package with multiple source-repository-package dependencies, cabal clean, then cabal build. You should see two repositories start being cloned at the same time, and then see other repositories being cloned concurrently as jobs are finished. The reason why only two repositories get cloned in parallel is that the limit of asynchronous downloads is == 2, to mimic asyncFetchPackages.

Questions: The maximum two-job cap on cloning concurrency seems too low, I think it would be best if it could be configured (as a follow up MR?)

mpickering · 2024-08-16T11:09:39Z

This change also affects cabal get -s, that needs documenting if it's intentional.

alt-romes · 2024-11-09T22:07:27Z

This is ready

ulysses4ever

Thanks!

alt-romes · 2024-11-11T10:42:53Z

I've addressed your review @ulysses4ever. Merging this now!

ulysses4ever · 2024-11-16T19:57:13Z

@alt-romes can you resolve the conflicts?

Cloning the entire repository for the purpose of compiling packages specified in source-repository-packages is wasted effort. To read and compile the package, we need only the HEAD of the repository, thus a shallow clone is sufficient. Note that this doesn't change the behaviour of `cabal get -s` which still does a full clone (--depth=1 is only used in vcsSyncRepo, not in vcsCloneRepo) Fixes haskell#7264

Cloning/synchronising VCS repos can be unnecessarily slow if done serially. By synchronizing the repos concurrently we make much better use of time. Introduces rerunConcurrentlyIfChanged, a Rebuild monad function that runs, from multiple actions, the actions that need rebuilding concurrently.

9999years · 2024-11-22T18:14:22Z

Hi, I'm working on the VCS test suite and (while running it in verbose mode) I noticed this:

Running: git -c 'protocol.file.allow=always' clone '--depth=1' --no-checkout /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-89726/src /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-89726/dest1
Cloning into '/private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-89726/dest1'...
warning: --depth is ignored in local clones; use file:// instead.

As a result, I'm not sure this feature is actually being tested.

When I patched the test suite to use the file:// URLs as suggested by the warning, I found a failure:

Running: git --version
Running: git -c 'protocol.file.allow=always' init
Initialized empty Git repository in /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/src/.git/
Running: git -c 'protocol.file.allow=always' add file/C
Running: git -c 'protocol.file.allow=always' -c 'user.name=A' -c 'user.email=a@example.com' commit --all '--message=a patch' '--author=A <a@example.com>'
[master (root-commit) 569c51c] a patch
 1 file changed, 1 insertion(+)
 create mode 100644 file/C
Running: git -c 'protocol.file.allow=always' log '--format=%H' -1
Running: git -c 'protocol.file.allow=always' tag --force --no-sign tag_XFUS

--------------------------------------------------------------------------------

Running: git -c 'protocol.file.allow=always' clone '--depth=1' --no-checkout 'file:///private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/src' /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest1
Cloning into '/private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest1'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (4/4), done.
Running: git -c 'protocol.file.allow=always' submodule deinit --force --all
Running: git -c 'protocol.file.allow=always' fetch origin tag_XFUS
From file:///private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/src
 * tag               tag_XFUS   -> FETCH_HEAD
Running: /Users/wiggles/.nix-profile/bin/git -c 'protocol.file.allow=always' reset --hard FETCH_HEAD --
HEAD is now at 569c51c a patch
Running: git -c 'protocol.file.allow=always' clean -ffxdq
Running: git -c 'protocol.file.allow=always' clone '--depth=1' --no-checkout 'file:///private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/src' /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest2 --reference /private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest1
Cloning into '/private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest2'...
fatal: reference repository '/private/var/folders/z5/fclwwdms3r1gq4k4p3pkvvc00000gn/T/vcstest-60316/dest1' is shallow

9999years · 2024-11-22T22:43:55Z

I have a number of PRs to make the VCS tests easier to work with open right now; I'll see if I can work on this when I get them merged:

jgm · 2026-05-31T09:39:52Z

Hi, I'm running into an issue on GitHub CI that seems to have to do with this behavior change.

Cloning into '/home/runner/work/pandoc/pandoc/dist-newstyle/src/commonmar_-15584dbd45cc637d4aa9b7fab57416f28e34dae8d2ac18a8dcf5092e89b5c646'...
From https://github.com/jgm/commonmark-hs
 * branch            204e4bb2e91bac82203468de207d8e2f79700dc0 -> FETCH_HEAD
HEAD is now at 204e4bb Math extension: bail out on pathological nestings.
Cloning into '/home/runner/work/pandoc/pandoc/dist-newstyle/src/commonmar_-15584dbd45cc637d4aa9b7fab57416f28e34dae8d2ac18a8dcf5092e89b5c646-2'...
fatal: reference repository '/home/runner/work/pandoc/pandoc/dist-newstyle/src/commonmar_-15584dbd45cc637d4aa9b7fab57416f28e34dae8d2ac18a8dcf5092e89b5c646' is shallow

This arises when cabal.project contains two source-repository-package stanzas for the same repository, but different directories.

I have found no good way to work around the issue.

philderbeast · 2026-05-31T11:09:16Z

Thanks @jgm, I think I found the pandoc CI run with this problem but can't find the source. Could you please raise an issue for this and include the source-repository-package stanza involved?

Update: I may have found it, if the commit that first sees the problem in CI is jgm/pandoc@af3af04?

source-repository-package
  type: git
  location: https://github.com/jgm/commonmark-hs.git
  subdir: commonmark
  tag: 48b95bab66401d94f80b7c84c5c33ab9e2cd39ea

source-repository-package
  type: git
  location: https://github.com/jgm/commonmark-hs.git
  subdir: commonmark-extensions
  tag: 204e4bb2e91bac82203468de207d8e2f79700dc0

ulysses4ever · 2026-05-31T12:12:02Z

This looks like severe regression. We need a fresh ticket

jgm · 2026-05-31T14:49:33Z

OK, I can be a bit more precise about this now.

You are right about the commit that first sees the problem.

I tried to fix this in a subsequent commit by using the same tag for both subdirectories, and that does seem to have worked: https://github.com/jgm/pandoc/actions/runs/26708083903 . So perhaps the problem is not as severe as I had thought.

ulysses4ever · 2026-05-31T16:56:48Z

Good to know that switching to the same commit across different srp's fixes it!

I have a standalone reproducer with CI exercising the failure now: https://github.com/ulysses4ever/cabal-srp-shallow-reference-reproducer

ulysses4ever · 2026-05-31T22:13:06Z

draft fix:

[WIP] Avoid sharing shallow SRP references #11921

alt-romes force-pushed the wip/romes/cabal-shallow-mr branch 3 times, most recently from 844e1bb to fd373ad Compare August 12, 2024 11:07

alt-romes changed the title ~~Wip/romes/cabal shallow mr~~ Shallow and concurrent git clones Aug 12, 2024

alt-romes force-pushed the wip/romes/cabal-shallow-mr branch from fd373ad to eb4f43c Compare August 16, 2024 13:51

alt-romes force-pushed the wip/romes/cabal-shallow-mr branch 7 times, most recently from 0416a52 to 85b2a5c Compare November 8, 2024 10:35

alt-romes requested review from fgaz, geekosaur and ulysses4ever November 9, 2024 22:07

geekosaur approved these changes Nov 9, 2024

View reviewed changes

ulysses4ever approved these changes Nov 10, 2024

View reviewed changes

alt-romes force-pushed the wip/romes/cabal-shallow-mr branch from 85b2a5c to 0d9629a Compare November 11, 2024 10:42

alt-romes self-assigned this Nov 11, 2024

alt-romes added the merge me Tell Mergify Bot to merge label Nov 11, 2024

mergify Bot added the ready and waiting Mergify is waiting out the cooldown period label Nov 11, 2024

alt-romes mentioned this pull request Nov 11, 2024

source-repository-package generates fat clones #7264

Closed

ulysses4ever linked an issue Nov 11, 2024 that may be closed by this pull request

source-repository-package generates fat clones #7264

Closed

mergify Bot added the merge delay passed Applied (usually by Mergify) when PR approved and received no updates for 2 days label Nov 13, 2024

ulysses4ever mentioned this pull request Nov 16, 2024

synchronize source repos concurrently #9844

Closed

5 tasks

alt-romes force-pushed the wip/romes/cabal-shallow-mr branch from 0d9629a to c89ab54 Compare November 16, 2024 21:19

alt-romes added merge me Tell Mergify Bot to merge and removed merge me Tell Mergify Bot to merge labels Nov 16, 2024

mergify Bot merged commit c3a9dd7 into haskell:master Nov 17, 2024

9999years mentioned this pull request Nov 22, 2024

VCS: Fix weird tag creation #10586

Merged

2 tasks

9999years mentioned this pull request Dec 3, 2024

New git invocation changes the behaviour of source-repository-package #10605

Open

ffaf1 mentioned this pull request Jan 29, 2025

cabal run creates dist-newstyle/cache/compiler in the current folder #10686

Closed

Uh oh!

Conversation

alt-romes commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mpickering commented Aug 16, 2024

Uh oh!

alt-romes commented Nov 9, 2024

Uh oh!

ulysses4ever left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alt-romes commented Nov 11, 2024

Uh oh!

ulysses4ever commented Nov 16, 2024

Uh oh!

9999years commented Nov 22, 2024

Uh oh!

9999years commented Nov 22, 2024

Uh oh!

jgm commented May 31, 2026

Uh oh!

philderbeast commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ulysses4ever commented May 31, 2026

Uh oh!

jgm commented May 31, 2026

Uh oh!

ulysses4ever commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ulysses4ever commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

alt-romes commented Aug 12, 2024 •

edited

Loading

philderbeast commented May 31, 2026 •

edited

Loading

ulysses4ever commented May 31, 2026 •

edited

Loading

ulysses4ever commented May 31, 2026 •

edited

Loading