Add function to checkout the jobs repository by ColinKinloch · Pull Request #75 · collabora/gitlab-runner-rs

ColinKinloch · 2025-01-24T17:25:34Z

This branch adds crate::clone_git_repository to clone git repos generally and Job::clone_git_repository to clone specifically defined by the gitlab job.

clone_git_repository is heavily based on PrepareFetch::fetch_only

Since the gitoxide PrepareFetch and PrepareCheckout structs delete the target path when they are dropped I opted to checkout the repository to a temporary directory, then move it's contents to the path provided to the target directory.

This adds a rather unwieldy GitCheckoutError enum. gitoxides Error types triggers the result_large_err clippy warning.

This requires rust version 1.73.0 or later.

Fixes #21

andrewshadura

This looks okay, but I haven’t gone deeply into details. I trust you tested this code well? (It would be great if it also included some automated tests.)

refi64 · 2025-05-30T15:00:48Z

 }
+
+// TODO: Should clone_git_repository be async
+//   gitoxide doesn't currently have async implemented for http backend


I don't think this is viable without either async or cloning in a separate thread (which would make cancellation really tricky, though you seem to have some interrupt-related stuff wired up already); blocking an entire I/O thread for what might be a very long clone process is not workable. Should this maybe just be using libgit2 or something else instead?

fwiw i'd prefer to stay with gitoxide to avoid more C dependences in the core of gitlab-runner. But yes it should be done in a dedicated thread if there is no async api

I've updated it to use tokio::task::spawn_blocking which runs it on a different thread.
I used a CancellationToken and tokio::select to trigger gitoxide to interrupt, CancellationToken is already used in the gitlab-runner-rs API so hopefully is okay.
gitoxide re-exports Progress from the prodash crate to allow the caller to monitor progress. Is that something we should do?

refi64 · 2025-05-30T15:10:22Z

+     *  * Clones to BuildDir
+     *  * Checks out git_info.sha and fetches all in git_info.refspecs
+     *  * Hardcoded username "gitlab-ci-token"
+     *  * credHelperCommand?


this is a bit tricky to understand out-of-context, might want to add a few more details for some of these points

refi64 · 2025-05-30T15:14:20Z

+    }
+
+    let checkout_progress = progress.add_child("checkout".to_string());
+    let (checkout, _outcome) = fetch.fetch_then_checkout(checkout_progress, &should_interrupt)?;


thought: if the goal is just to be able to read files from the repo, do we need to do an entire checkout to begin with? couldn't it clone a bare repo and then load files as-needed from there? though the apis might be harder to make for that...but it would likely also be a lot faster.

Fwiw i wouldn't mind this being a potential future update ; I'm not sure how much faster it would be ("likely be a lot of faster" is always a dangerous statement in the land of optimisation). I guess it primarily depend on what work would be saved, which i guess is primarily unpacking the pack file.

Regardless; the time would be comparable to a normal checking in a normal gitlab runner (or faster as iirc gix is faster at this point then git).

Copilot

Pull Request Overview

This PR implements Git repository cloning functionality for the GitLab runner by adding two main functions: clone_git_repository for general Git repository cloning and Job::clone_git_repository for job-specific cloning. The implementation leverages the gitoxide library and includes comprehensive error handling through a new GitCheckoutError enum.

Key changes:

Adds comprehensive Git cloning functionality with support for refspecs, depth control, and SHA-specific checkout
Implements proper error handling for Git operations with detailed error categorization
Provides both async and sync interfaces for Git operations with cancellation support

Reviewed Changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
gitlab-runner/src/run.rs	Fixes error message for build directory creation
gitlab-runner/src/lib.rs	Adds core Git cloning functions and comprehensive error handling
gitlab-runner/src/job.rs	Implements job-specific Git repository cloning method
gitlab-runner/src/client.rs	Defines Git information structures and error types
gitlab-runner/examples/demo-runner.rs	Adds example commands for testing Git functionality
gitlab-runner/Cargo.toml	Updates dependencies and adds gitoxide library

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-30T14:55:01Z

+    if let Some(depth) = depth.and_then(NonZeroU32::new) {
+        fetch = fetch.with_shallow(remote::fetch::Shallow::DepthAtRemote(depth));


Using NonZeroU32::new will return None for depth value 0, but depth 0 should be valid for unlimited depth. Consider using a separate check or handling depth 0 explicitly.

Suggested change

if let Some(depth) = depth.and_then(NonZeroU32::new) {

fetch = fetch.with_shallow(remote::fetch::Shallow::DepthAtRemote(depth));

match depth {

Some(0) => {

// Depth 0 means unlimited depth; do not set shallow clone.

}

Some(d) => {

if let Some(nonzero_depth) = NonZeroU32::new(d) {

fetch = fetch.with_shallow(remote::fetch::Shallow::DepthAtRemote(nonzero_depth));

}

}

None => {

// No depth specified; do not set shallow clone.

}

Copilot · 2025-09-30T14:55:02Z

+
+    index.write(Default::default())?;
+
+    Ok(repo_dir.keep())


[nitpick] The function returns the temporary directory path, but the calling code is expected to move its contents manually. Consider returning the final destination path or handling the move operation within this function to improve the API design.

Copilot · 2025-09-30T14:55:02Z

+        clone_git_repository(
+            self.build_dir(),
+            &self.response.git_info.repo_url,
+            Some(&self.response.git_info.sha),


[nitpick] The SHA is always passed as Some(), but the generic clone_git_repository function expects Option<&str>. Consider whether this should handle cases where SHA might be empty or invalid.

Suggested change

Some(&self.response.git_info.sha),

if self.response.git_info.sha.is_empty() {

None

} else {

Some(&self.response.git_info.sha)

},

Copilot · 2025-09-30T14:55:02Z

+                    std::fs::rename(repo_path, path)
+                        .map_err(|e| outputln!("Failed to move repo path: {}", e.to_string()))?;


Using std::fs::rename to move a directory may fail across filesystem boundaries. Consider using a recursive copy operation or std::fs::create_dir_all followed by moving individual files.

* `clone_git_repository` to clone arbitrary repo * `Job::clone_git_repository` to clone the jobs repo

Also throw an error if the provided HEAD ref cannot be parsed.

Otherwise, clippy gets quite upset because the error type is so large (>200 bytes).

This is set if the runner isn't allowed to reuse an existing repository by re-running a fetch there, which is unrelated to the ability to clone itself.

It's a bit confusing that this automatically considers the corresponding CI variable, but some other settings (such as the git strategy allowing repo reuse) do not.

No real reason for them to be public, and it frees us from handling a lot of edge cases and allows for some significant (future) code cleanup.

If this is private, and only used by Job, it makes sense for it to be in the same place.

Given that the example didn't even use the cancellation token properly, it's better for cancellation to be handled internally.

The `sha` field is never going to contain a non-SHA, among other things, so this code can be wittled down quite a bit now.

Much of this can be replaced with helpers from the crate itself.

In these cases, we just return None to signal back to the caller that the clone couldn't be performed for a non-error reason.

refi64 · 2026-05-08T21:46:01Z

Rebased this and added a bunch of extra fix-up commits on top. The new changes are split up the changes quite aggressively to try to make them easier to review (we can squash as needed in the end anyway). minimal-direct-dependencies is passing locally, I'll figure out why it's failing later on...

ColinKinloch force-pushed the git branch 2 times, most recently from 587d8dd to ff98e93 Compare January 28, 2025 10:58

ColinKinloch mentioned this pull request Jan 28, 2025

Checkout git repo on start of job collabora/lava-gitlab-runner#81

Draft

ColinKinloch force-pushed the git branch 4 times, most recently from 082f212 to 7428588 Compare January 28, 2025 17:03

ColinKinloch force-pushed the git branch 3 times, most recently from dbcc18f to c0f9c71 Compare February 10, 2025 18:02

ColinKinloch marked this pull request as ready for review February 10, 2025 18:03

andrewshadura approved these changes Mar 21, 2025

View reviewed changes

refi64 requested changes May 30, 2025

View reviewed changes

ColinKinloch force-pushed the git branch 5 times, most recently from 6b04326 to 6b020a2 Compare July 28, 2025 17:51

sjoerdsimons requested a review from Copilot September 30, 2025 14:53

Copilot AI reviewed Sep 30, 2025

View reviewed changes

ColinKinloch and others added 10 commits April 30, 2026 16:49

Add the ability to checkout git repositories

7f0a79a

* `clone_git_repository` to clone arbitrary repo * `Job::clone_git_repository` to clone the jobs repo

Offload gitoxide to a tokio blocking thread

1a5b99f

Support git clone by commit id SHA

97661b4

Also throw an error if the provided HEAD ref cannot be parsed.

typo: "Failed to create build dir"

d2f42e6

Make GitCheckoutError's inner errors boxed

15d04eb

Otherwise, clippy gets quite upset because the error type is so large (>200 bytes).

Clarify comment on git credentials helper

8db3b8d

Fix incorrect interpretation of allow_git_fetch

06c67bf

This is set if the runner isn't allowed to reuse an existing repository by re-running a fetch there, which is unrelated to the ability to clone itself.

Add comment clarifying meaning of the depth field

94def56

It's a bit confusing that this automatically considers the corresponding CI variable, but some other settings (such as the git strategy allowing repo reuse) do not.

Make git checkout APIs private

c04d569

No real reason for them to be public, and it frees us from handling a lot of edge cases and allows for some significant (future) code cleanup.

Relocate git clone code to job.rs

10a90e2

If this is private, and only used by Job, it makes sense for it to be in the same place.

refi64 added 4 commits May 8, 2026 16:41

Make clone_git_repository natively cancel-safe

42c0d1a

Given that the example didn't even use the cancellation token properly, it's better for cancellation to be handled internally.

Reduce scope of git cloning code to only handle GitLab's inputs

b93e102

The `sha` field is never going to contain a non-SHA, among other things, so this code can be wittled down quite a bit now.

Use more gitoxide helpers for the cloning process

af309c5

Much of this can be replaced with helpers from the crate itself.

Add support for avoiding clones when GIT_STRATEGY=none/empty

cc08849

In these cases, we just return None to signal back to the caller that the clone couldn't be performed for a non-error reason.

refi64 force-pushed the git branch from 6b020a2 to 2e9e579 Compare May 8, 2026 21:43

refi64 added 2 commits May 8, 2026 16:44

mock: Add the ability to customize more GitInfo fields

b297f5a

Add tests for clone_git_repository

bd12498

refi64 force-pushed the git branch from 2e9e579 to bd12498 Compare May 8, 2026 21:45

		if let Some(depth) = depth.and_then(NonZeroU32::new) {
		fetch = fetch.with_shallow(remote::fetch::Shallow::DepthAtRemote(depth));

-    if let Some(depth) = depth.and_then(NonZeroU32::new) {
-        fetch = fetch.with_shallow(remote::fetch::Shallow::DepthAtRemote(depth));
+    match depth {
+        Some(0) => {
+            // Depth 0 means unlimited depth; do not set shallow clone.
+        }
+        Some(d) => {
+            if let Some(nonzero_depth) = NonZeroU32::new(d) {
+                fetch = fetch.with_shallow(remote::fetch::Shallow::DepthAtRemote(nonzero_depth));
+            }
+        }
+        None => {
+            // No depth specified; do not set shallow clone.
+        }

		std::fs::rename(repo_path, path)
		.map_err(\|e\| outputln!("Failed to move repo path: {}", e.to_string()))?;

Conversation

ColinKinloch commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrewshadura left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

refi64 May 30, 2025

Choose a reason for hiding this comment

Uh oh!

sjoerdsimons Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

ColinKinloch Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

refi64 May 30, 2025

Choose a reason for hiding this comment

Uh oh!

refi64 May 30, 2025

Choose a reason for hiding this comment

Uh oh!

sjoerdsimons Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

refi64 commented May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

ColinKinloch commented Jan 24, 2025 •

edited

Loading

andrewshadura left a comment •

edited

Loading