Abandon Script

[jimmyp]

Why

Tentacle needs to release the script-isolation mutex even when a script can't be cancelled. Philips is running into situations where CrowdStrike and Rapid7 contention freezes the script causing cancel commands to not be able to complete.

This PR adds an AbandonScript command that causes Tentacle to stop monitoring the process and release the mutex, write a line into the script log and accept new work. If we can kill the OS process we do, but if we can't we stop waiting on it and leave it running.

What changed

• AbandonScript RPC on IScriptServiceV2 / IAsyncScriptServiceV2 / IAsyncClientScriptServiceV2: fires the abandon token and returns the current status of the script
• RunningScript logs the abandonment and returns -48 (AbandonedExitCode); the isolation-mutex using unwinds, releasing it.
• SilentProcessRunner best-effort kills the process on abandon, then stops waiting.
• CompleteScriptAsync tolerates a workspace.Delete failure for an abandoned script (the still running script can hold file handles etc)
• TentacleClient.ExecuteScript now has an optional abandonAfterCancellationPendingFor paramete once cancellation has been pending that long it abandons the script.
• Non-Kubernetes Tentacles advertise the AbandonScriptAsync capability.
• For simulating frozen scripts I had to add TentacleDebugDisableProcessKill_UNSAFE_FOR_PRODUCTION which makes Hitman a no-op instead of killing the script process.

Before and after

Cancel only helps if the process actually dies. When it doesn't, the mutex stays held and every later deployment to that Tentacle queues forever. Today the only fix is an RDP onto the machine to kill the script or a reboot of the machine. Abandon changes that. After the cancel timeout the server abandons the script and the Tentacle releases the mutex, so work flows again.

Before

flowchart TD
    A["CancelScript fires the cancel token"] --> B["cancel.Register runs DoOurBestToCleanUp:<br/>Hitman.TryKillProcessAndChildrenRecursively, then process.Close()"]
    B --> C{"What happens to the OS process?"}
    C -->|"killed cleanly"| D["process.WaitForExit() returns"]
    C -->|"child dies, reparented grandchild holds the pipes"| E["process.Close() releases the redirected pipes"]
    E --> D
    C -->|"hung script won't die (CrowdStrike + Rapid7)"| F["process.WaitForExit() blocks forever<br/>isolation mutex never released; every later deploy queues"]
    D --> G["drain stdout/stderr, SafelyGetExitCode, return the process exit code"]

Loading

After

flowchart TD
    A["CancelScript fires the cancel token"] --> B["cancel.Register runs DoOurBestToCleanUp:<br/>Hitman.TryKillProcessAndChildrenRecursively, then process.Close()"]
    B --> C{"What happens to the OS process?"}
    C -->|"killed cleanly"| D["waitForExit task completes"]
    C -->|"child dies, reparented grandchild holds the pipes"| E["process.Close() releases the redirected pipes"]
    E --> D
    C -->|"hung script won't die"| W["await Task.WhenAny(waitForExit, WaitForAbandon(abandon)) stays pending"]
    D --> N["WhenAny returns, abandon not set: drain, return the process exit code"]
    W --> AB["AbandonScript fires the abandon token<br/>(server escalates after abandonAfterCancellationPendingFor)"]
    AB --> X["WhenAny returns via WaitForAbandon: DoOurBestToCleanUp once more (best-effort),<br/>log 'Tentacle has abandoned this script', return AbandonedExitCode (-48)"]
    X --> R["isolation mutex released, work flows again;<br/>OS process left running if it still won't die"]

Loading

Risks & mitigations

Machines may start collecting stuck script processes that aren't monitored by tentacle. Any real mitigation of this is beyond our control which is why we have added product documentation around this feature to make users aware of their responisibilities.

Workspace files may leak on Windows. However the workspace gets cleaned up on Tentacle restart or by out-of-band cleanup mechanisms.

When abandoning scripts that aren't fully frozen, but maybe just slow to respond there's a race between the script completing naturally and abandonment, the script can finish between the time we call abandon and abandon completing. If the script exits in this gap, we dont make any guarantees we will return the correct error code. This is a natural race condition for the user clicking cancel button and the script completing anyway, so there is no real justification to mitigate this

During CI builds we start scripts and simulate them hanging, which could result in stray processes hanging around on build agents. We make best attempts to clean these up

For Kubernetes agent owners

K8s agents do NOT advertise abandon. The capability set only includes AbandonScriptAsync for non-Kubernetes Tentacles. K8s agents continue to advertise their own capability set without abandon support.

Why: K8s scripts run in pods. A hung pod is recoverable by deleting it, which the server can do through the existing K8s lifecycle. The Philips-style hang doesn't translate to k8s pods.

How to review

• Is the abandon behaviour right? (mutex released, OS process killed if we can and left alone if we can't, log message honest)
• Given this is a significant change to Tentacle, is there anything else we can do to add safety nets or reduce risk?

[JIM_BOT.EXE v2.13]

[jimmyp]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

I'm Claude, leaving this review on Jim's behalf.

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[jimmyp]

Need to address these comments

[jimmyp]

Rebased on top of the new foundational PR #1236 (async migration of SilentProcessRunner). The diff is now focused on the abandon feature itself — the async-migration plumbing has moved to the base PR. Previous head was preserved at git tag claude-safety-2026-05-25-pre-split.

[jimmyp]

Rebased on top of #1236's Pattern 1 refactor. The abandon-feature commits now build on a cleaner async migration (single bridges instead of multiple in-method GetAwaiter calls). Previous head preserved at tag claude-safety-2026-05-26-pre-pattern1-rebase.

[jimmyp]

I'm not sure what we are testing here. Explain it to me

[jimmyp]

remove this comment

[jimmyp]

Where are the other test cases. Do we need more coverage here?

[jimmyp]

Dont we need a test for our new capability here?

[jimmyp]

This should be a second test

[jimmyp]

What is this asserting?

[jimmyp]

This doesnt make sense. Explain it again

[jimmyp]

I'm not sure this is right

[jimmyp]

This should be start script not execute script

[jimmyp]

Fix all comments in the test as you go. All tests except ExecuteScript_WhenCancellationStaysPending_EscalatesToAbandon_AndReleasesMutex should use start and not execute

[jimmyp]

We should have tests for both start and execute paths

[jimmyp]

Also assert the process is still running in all of these tests so we know we're actually setting the test up right

[jimmyp]

Change this to make explicit calls to start, cancel and abandon. We shouldnt need TentacleDebugDisableProcessKill_UNSAFE_FOR_PRODUCTION either

[jimmyp]

Add these counts back

[jimmyp]

Lets create ScriptServiceVersion.IsV2 to centralise this and remove the comment below