Merge pull request #69 from mxstack/feature/smart-threading-pushurl

Improve smart threading for HTTPS URLs with pushurl

Author: jensens

CHANGES.md

@@ -16,6 +16,8 @@
   [jensens]
 - Chore: Updated test fixture data versions to resolve Dependabot security alerts. Updated urllib3 from 1.26.9 to 2.5.0 and requests from 2.28.0 to 2.32.4 in test data files. These are test fixtures only and were never actual dependencies or security risks. Resolves GitHub Dependabot alerts #1-7.
   [jensens]
+- Performance: Smart threading now processes HTTPS URLs with `pushurl` in parallel. When a package defines both an HTTPS `url` and a `pushurl` (typically SSH), the HTTPS URL is assumed to be read-only/public and won't prompt for credentials, making it safe for parallel processing. This improves checkout performance for the common pattern of public repos with separate push URLs.
+  [jensens]
 - Fix: Add 'synchronize' event to pull_request workflow triggers. This ensures CI runs when PRs are updated with new commits (e.g., after rebasing or pushing new changes), not just when opened or reopened.
   [jensens]
 - Chore: Optimize GitHub Actions to prevent duplicate workflow runs on pull requests. Restrict `push` trigger to only run on `main` branch, so PRs only trigger via `pull_request` event. This reduces CI resource usage by 50% for PR workflows.

README.md

@@ -94,8 +94,10 @@ The **main section** must be called `[settings]`, even if kept empty.
 
 When `smart-threading` is enabled (default), mxdev uses a two-phase approach to prevent credential prompts from overlapping:
 
-1. **Phase 1**: HTTPS packages are processed serially (one at a time) to ensure clean, visible credential prompts
-2. **Phase 2**: Remaining packages (SSH, local) are processed in parallel for speed
+1. **Phase 1**: HTTPS packages **without `pushurl`** are processed serially (one at a time) to ensure clean, visible credential prompts
+2. **Phase 2**: Remaining packages (SSH, local, HTTPS with `pushurl`) are processed in parallel for speed
+
+**Optimization**: HTTPS URLs with `pushurl` defined are assumed to be read-only/public and processed in parallel, since the `pushurl` indicates authenticated write access is separate.
 
 This solves the problem where parallel git operations would cause multiple credential prompts to overlap, making it confusing which package needs credentials.
 

src/mxdev/vcs/common.py

@@ -175,6 +175,9 @@ def __init__(
     def _separate_https_packages(self, packages: list[str]) -> tuple[list[str], list[str]]:
         """Separate HTTPS packages from others for smart threading.
 
+        HTTPS packages WITH pushurl are safe for parallel processing
+        (pushurl implies the https url is read-only/public).
+
         Returns (https_packages, other_packages)
         """
         https_packages = []
@@ -186,9 +189,13 @@ def _separate_https_packages(self, packages: list[str]) -> tuple[list[str], list
                 continue
             source = self.sources[name]
             url = source.get("url", "")
-            if url.startswith("https://"):
+            has_pushurl = "pushurl" in source
+
+            if url.startswith("https://") and not has_pushurl:
+                # HTTPS without pushurl: may need credentials, process serially
                 https_packages.append(name)
             else:
+                # SSH, fs, or HTTPS with pushurl: safe for parallel
                 other_packages.append(name)
 
         return https_packages, other_packages

tests/test_git_additional.py

@@ -963,3 +963,47 @@ def mock_run_git_side_effect(commands, **kwargs):
             with patch.object(wc, "git_version", return_value=(2, 30, 0)):
                 with pytest.raises(GitError, match="git checkout"):
                     wc.git_switch_branch("", "")
+
+
+def test_smart_threading_separates_https_with_pushurl():
+    """Test smart threading correctly separates HTTPS packages based on pushurl.
+
+    HTTPS URLs with pushurl should go to parallel queue (other_packages).
+    HTTPS URLs without pushurl should go to serial queue (https_packages).
+    """
+    from mxdev.vcs.common import WorkingCopies
+
+    sources = {
+        "https-no-pushurl": {
+            "name": "https-no-pushurl",
+            "url": "https://github.com/org/repo1.git",
+            "path": "/tmp/repo1",
+        },
+        "https-with-pushurl": {
+            "name": "https-with-pushurl",
+            "url": "https://github.com/org/repo2.git",
+            "path": "/tmp/repo2",
+            "pushurl": "git@github.com:org/repo2.git",
+        },
+        "ssh-url": {
+            "name": "ssh-url",
+            "url": "git@github.com:org/repo3.git",
+            "path": "/tmp/repo3",
+        },
+        "fs-url": {
+            "name": "fs-url",
+            "url": "/local/path/repo4",
+            "path": "/tmp/repo4",
+        },
+    }
+
+    wc = WorkingCopies(sources=sources, threads=4, smart_threading=True)
+    packages = ["https-no-pushurl", "https-with-pushurl", "ssh-url", "fs-url"]
+
+    https_pkgs, other_pkgs = wc._separate_https_packages(packages)
+
+    # Only HTTPS without pushurl should be in serial queue
+    assert https_pkgs == ["https-no-pushurl"]
+
+    # HTTPS with pushurl, SSH, and fs should be in parallel queue
+    assert set(other_pkgs) == {"https-with-pushurl", "ssh-url", "fs-url"}