cmdrunner: release process handle in _pidAlive to avoid pidfd leak

[texasich]

Description

_pidAlive in internal/cmdrunner/process_posix.go calls os.FindProcess(pid) and never releases the returned *os.Process. On Linux with Go 1.23+ that call now opens a pidfd under the hood (os.pidfdFind → pidfd_open), so every invocation leaks one file descriptor.

pidWait polls once per second per plugin, so the leak scales linearly with plugin count × uptime. In the downstream Nomad report (hashicorp/nomad#27847) a client with a few hundred allocations saw the host-side FD count cycle between 20k and 130k, eventually tripping EMFILE and breaking CNI config loads, disk-stats collection, pipe2, and docker socket dials.

The fix is a one-liner: defer proc.Release() right after the FindProcess call, so the pidfd is closed on every return path. The Windows implementation already does the equivalent with defer syscall.CloseHandle(h) in process_windows.go, so this just brings the POSIX side in line.

Before:

pidfd_open(686713, 0) = 90771
pidfd_open(690846, 0) = 90772
pidfd_open(123738, 0) = 90774
...

(one new FD per poll iteration per plugin, never closed)

Related Issue

Downstream report: hashicorp/nomad#27847

How Has This Been Tested?

• go build ./... and go vet ./... clean on both host (Windows) and GOOS=linux
• go test ./internal/cmdrunner/... passes
• Reviewed against the Windows _pidAlive which has always done the equivalent cleanup via defer syscall.CloseHandle(h), so behavior is symmetric across platforms

Caching the *os.Process on the runner would avoid the repeated FindProcess entirely and is probably the better long-term change, but it's a larger refactor touching Runner lifecycle. Keeping this PR to the minimal, backport-friendly fix; happy to follow up with the caching variant in a separate PR if preferred.

[hashicorp-cla-app]

All committers have signed the CLA.

[tgross]

Thanks for the PR @texasich! LGTM once the lint is addressed!

I've reproduced the circumstances described in hashicorp/nomad#27847 with both the current tip of Noamd and a version of Nomad using this PR for go-plugin (via a replace directive in my go.mod). I run a single minimal job and then restart the agent. (Note this requires running not-in--dev mode.)

Using this little script:

#!/usr/bin/env bash

while :
do
    ls /proc/$1/fd/ | wc -l
    sleep 1
done

We see that before the patch, the number of open file handles increases after restart Nomad. After the patch, it does not.

[tgross]

This is triggering a very dumb lint. Can we either swallow the return value or put a //nolint:errcheck directive here?

[texasich]

Thanks @tgross — appreciate you running the repro with the replace directive. Pushed 8382575 swapping the bare defer proc.Release() for defer func() { _ = proc.Release() }() so errcheck stays quiet.

[tgross]

LGTM

[texasich]

Quick CI update — lint is clean on 8382575. The remaining failure is TestClient_TLS in the top-level package, which looks like a flake. The error tls: first record does not look like a TLS handshake at client_test.go:1064 is the classic TLS-handshake race pattern, and master CI is green on this test. The diff between attempts on our side is just defer proc.Release() → defer func() { _ = proc.Release() }(), which is semantically identical — proc.Release() releases the local handle without touching the running process, so there's no path from cmdrunner internals to the top-level TLS test.

When you get a moment, could you re-run the go-test job? If it fails the same way a second time I'll dig deeper.

[ritikrajdev]

Thanks for checking this out. Looks Good to me as well.