Skip to content

DESIGN.md

Latest commit

 

History

History
365 lines (285 loc) · 13.3 KB

DESIGN.md

File metadata and controls

365 lines (285 loc) · 13.3 KB
Raw

pwforge — Patchwork/Forge Bidirectional Sync Bridge

What it does

pwforge is a bridge that synchronizes patch workflows between mailing lists (tracked by Patchwork) and code forges (currently GitHub). It converts mailing list patch series into pull requests and vice versa, forwarding comments, reviews, and CI results in both directions.

Problem statement

Many open source projects use mailing list patch workflows (via git send-email) while contributors increasingly expect GitHub-style pull request workflows. Patchwork tracks patches submitted to mailing lists but has no forge integration. pwforge bridges this gap so that:

  • A patch series submitted to a mailing list automatically becomes a GitHub pull request.
  • A pull request opened on GitHub automatically becomes a patch series on the mailing list.
  • Comments, reviews, and CI results flow between both systems.

Architecture overview

pwforge runs as a standalone HTTP server that receives webhooks from both Patchwork and GitHub. It processes events asynchronously through two internal queues (one per source) and performs the appropriate sync operations.

                    ┌───────────────────┐
  mailing list ───► │ Patchwork ingress │──► Patchwork
                    └───────────────────┘       │
                                                │ webhook
                                                ▼
                                            ┌────────┐
                                            │pwforge │
                                            └────────┘
                                                ▲
                                                │ webhook
                                                │
                                             GitHub

Multi-project support

pwforge auto-discovers projects from the Patchwork API. For each Patchwork project, it parses the web_url or scm_url field to determine the corresponding forge repository. Optional per-project overrides can be specified in the config file via [[project]] TOML array sections.

Each project gets its own:

  • Forge client instance (with project-specific owner/repo)
  • Bare git mirror (at mirror-path/<link_name>.git)
  • MLToForge and ForgeToML sync handlers
  • Mailing list address (from Patchwork's list_email field)

The project map is cached with a configurable TTL and refreshed lazily on the next webhook.

Mailing list to forge sync

When Patchwork fires a series-completed webhook:

  1. pwforge fetches the series metadata from the Patchwork API.
  2. If the series already has a linked PR (stored in series metadata), it skips. If this is a respin of a previous series (detected via patchwork's previous_series pointer), it force-pushes to the same branch and posts an update comment on the existing PR.
  3. Otherwise, it downloads the series mbox, creates a temporary git worktree from the repo's default branch, applies the patches with git am, pushes to a branch named {prefix}/{series_id}/{slug}, and creates a pull request.
  4. The PR body includes the cover letter content (or single patch commit message), a link back to the Patchwork series, and a <!-- pwforge --> HTML comment marker.
  5. The PR number and branch name are stored in the series metadata on Patchwork for future reference.

When Patchwork fires a patch-comment-created or cover-comment-created webhook:

  1. pwforge checks if the comment originated from itself (via the X-PWForge-Event email header stored in patchwork's comment headers). If so, it skips to prevent loops.
  2. It finds the series associated with the comment, looks up the linked PR from the series metadata, and posts the comment on the GitHub PR.

Content handling

  • The PR body is built from the cover letter or single-patch commit message. Trailers from the submitter (Signed-off-by, etc.) are stripped. Diffstat sections are removed.
  • Branch names are generated as {prefix}/{hex_id}/{slug} where slug is a sanitized, truncated version of the series name.

Forge to mailing list sync

When GitHub fires a pull_request webhook (opened or synchronized):

  1. pwforge checks if the PR's branch starts with the configured prefix (e.g. pwforge/). If so, this PR was created by pwforge itself and is skipped to prevent loops.
  2. It fetches the PR ref, creates a worktree, and generates a patch series via git send-email. The patches include:
    • A cover letter (if more than one commit) with the PR title and sanitized body.
    • Reply-To: header pointing to the mailing list.
    • X-PWForge-PR: header with the PR URL.
    • X-PWForge-Branch: header with the branch name.
    • For respins (PR synchronize events): version numbering (-v2, -v3, etc.) and --range-diff against the previous version.
    • --in-reply-to pointing to the v1 cover letter for threading.
  3. The PR body is sanitized: HTML comments are stripped, AI-generated sections (CodeRabbit, Copilot summaries) are removed.

When GitHub fires an issue_comment webhook:

  1. If the comment contains the <!-- pwforge --> marker, it was posted by pwforge and is skipped.
  2. The comment is sent as an email reply to the series cover letter (or first patch) via git send-email, with the commenter as the From address when their email is known.

When GitHub fires a pull_request_review webhook:

  1. The review body and all inline comments are collected.
  2. A single email is sent with the review state, body text, and all inline comments (each with file path and diff context).

When GitHub fires check_suite (completed) or check_run (created) webhooks:

  1. CI check results are posted on each patch in the series via the Patchwork checks API.
  2. A summary email is sent to the mailing list with all check run names, statuses, and URLs.

Loop prevention

Three mechanisms prevent infinite sync loops:

  1. Branch prefix check: PRs on branches starting with the configured prefix (default pwforge/) are recognized as pwforge-created and skipped during forge-to-ML sync.
  2. HTML comment marker: All GitHub comments and PR bodies posted by pwforge contain <!-- pwforge -->. Comments containing this marker are skipped during ML-to-forge sync.
  3. Email header check: All emails sent by pwforge include an X-PWForge-Event header. When patchwork ingests such an email and fires a comment webhook, pwforge detects the header in the comment's stored headers and skips it.

Forge abstraction

The forge implementation is behind an interface so that other forges (GitLab, Gitea, etc.) can be added without changing the core sync logic.

Forge interface

Each forge implementation provides:

  • BaseBranch(): Default branch of the repository.
  • RepoURL(): Clone URL (used for git push).
  • WriteCredentials(path): Write auth credentials for git operations.
  • MetaKeyPR(), MetaKeyBranch(): Metadata key names used to store forge references in Patchwork series metadata.
  • PRRef(n): Full URL for a PR (used in metadata).
  • PRRefSpec(n): Git refspec to fetch a PR.
  • CreatePR(title, body, head, base): Create a pull request.
  • PostComment(n, body): Post a comment on a PR.
  • RepoKey(): Normalized repository identifier for webhook routing.

Registration

Each forge package registers itself via init():

models.RegisterForge("github", New, newWebhookParser, resolveRepo,
    models.WithSetupHandler(SetupHandler))

This provides:

  • A constructor for per-project forge instances.
  • A webhook parser factory (created once, shared across projects).
  • A URL resolver to extract owner/repo from patchwork project URLs.
  • An optional setup handler for interactive app registration.

Webhook parsing

Webhook parsing is decoupled from per-project forge instances. A single parser is created at startup from the global config (API client + webhook secret). It verifies signatures, parses event payloads, and populates a ForgeEvent struct that includes a RepoKey field for routing to the correct project.

For GitHub, the parser extracts repository.full_name from every event payload and uses the go-github library's typed event structs. Review and check suite events make additional API calls to fetch inline comments and individual check runs.

GitHub implementation details

Authentication

Two authentication modes are supported:

  1. Personal Access Token (PAT): Simple bearer token. Configured via token in the [github] config section.
  2. GitHub App: JWT-based auth via the ghinstallation library. Configured via app-id, private-key-file. The installation-id is auto-discovered at startup by listing the app's installations via the App JWT API.

GitHub App manifest flow

pwforge includes a /setup endpoint that implements the GitHub App manifest flow for one-click app registration:

  1. Admin visits /setup on the pwforge instance.
  2. A form auto-submits a JSON manifest to GitHub with the correct permissions (contents:read, pull_requests:write, issues:write, checks:read) and webhook URL.
  3. The user confirms the app name on GitHub.
  4. GitHub redirects back with a temporary code.
  5. pwforge exchanges the code via POST /app-manifests/{code}/ conversions and receives the app ID, private key (PEM), and webhook secret.
  6. Credentials are written to the config directory on disk.
  7. The user installs the app on their repositories and restarts pwforge.

Fork-based workflow

When fork-owner and fork-repo are configured (different from owner/repo), pwforge pushes branches to the fork and creates PRs against the upstream repository. The PR head reference is prefixed with fork-owner: as required by the GitHub API.

Git operations

All git operations use a bare mirror clone stored at the configured mirror-path. The mirror is created on first use and updated via git fetch --all --prune before each operation.

Credential isolation

  • Git operations use temporary credential files written before each operation and deleted immediately after.
  • GIT_CONFIG_GLOBAL=/dev/null and GIT_CONFIG_SYSTEM=/dev/null are set to prevent user/system git config from interfering.
  • GIT_TERMINAL_PROMPT=0 prevents interactive auth prompts.
  • SMTP passwords are stored in the mirror's git config (not logged).

Email sending

All outgoing email is sent via git send-email, which handles MIME encoding, header formatting, and SMTP delivery. Two modes:

  1. Patch series (SendPatches): Uses git send-email with --cover-letter, --range-diff, version numbering, and custom headers. The cover letter description is written to a temporary file.
  2. Standalone emails (SendEmail): For comments, reviews, and CI results. Writes headers and body to a temporary .eml file and sends via git send-email --from --subject.

Worktree management

Each sync operation creates a temporary worktree from the mirror, performs the work (apply patches, generate emails), and cleans up. This prevents interference between concurrent operations and avoids state leaks.

Patchwork API client

pwforge uses Patchwork's REST API v1.4 for:

  • Fetching series, patches, covers, and comments.
  • Updating series metadata (PR links, branch names).
  • Creating CI checks on patches.
  • Querying series by metadata (for finding linked PRs).
  • Listing projects (for auto-discovery).
  • Downloading series mbox (with retry on empty response).

The client uses DisableKeepAlives: true on the HTTP transport to avoid connection reuse issues.

Configuration

pwforge is configured via TOML files loaded from (in order):

  1. /etc/pwforge.toml
  2. pwforge.toml (current directory)
  3. $PWFORGE_TOML (if set)

Settings from later files override earlier ones. CLI flags take precedence over any configuration file. Run pwforge --print-config to generate a default configuration file.

listen = ":8080"
base-url = "https://patches.example.com/pwforge"
forge = "github"
queue-size = 10

[patchwork]
url = "https://patches.example.com"
token = "<api-token>"
# optional: restrict to a single project
#project = "my-project"
cache-ttl = 600

[github]
# either PAT:
#token = "ghp_xxx"
# or App (auto-discovered via /setup):
app-id = 123456
private-key-file = "/etc/pwforge/github-app.pem"
webhook-secret = "<secret>"

[smtp]
host = "localhost"
port = 25
encryption = "none"
from = "Pwforge <pwforge@patches.example.com>"

[sync]
ml-to-forge = true
forge-to-ml = true

[git]
mirror-path = "/var/cache/pwforge"
branch-prefix = "pwforge"
subject-prefix = "PATCH"

# optional per-project overrides
[[project]]
name = "grout"
owner = "DPDK"
repo = "grout"
subject-prefix = "PATCH grout"

Local testbed

A local testbed (make local or ./local/run.sh) creates a tmux session with:

  • Patchwork HTTP server (pw http, localhost:8888)
  • Patchwork SMTP ingress (pw ingress, localhost:2525)
  • pwforge binary (auto-restart via inotifywait on rebuild)
  • GitHub webhook forwarding via gh webhook forward
  • Mail client (aerc, neomutt, or mutt)
  • Git working directory for patch submission

The testbed builds patchwork-next from a local checkout (default ../patchwork-next, configurable via $PATCHWORK_NEXT_DIR), initializes a SQLite database with a test project and users via pw admin commands, and creates a GitHub repo (or reuses an existing one).