Grandchild processes ignoring SIGTERM are not killed

[oleg-vinted]

Repro: https://gist.github.com/oleg-vinted/97ee727e8d0b1050f40fc3ec9f940281

Even with the process group change, in some cases processes can still be orphaned.

In that repro, we have: overman -> service -> stubborn-process -> sleep process hierarchy.

What I think is happening:

• On Ctrl-C, SIGINT is sent only to overman, because its children are running in a different process group.
• Overman sends SIGTERM to the child process group correctly, impacting service, stubborn-process, sleep.
• service and sleep terminate upon receiving SIGTERM.
• stubborn-process ignores SIGTERM and spawns another instance of sleep and keeps running.
• overman sees that its immediate child (service) has exited and removes its PID from the running list: through here, here and here.
• overman thinks its children have exited, so it does not send SIGKILL to the process group, even though SIGKILL is still necessary because stubborn-process is still running (Edit: not true, sending a SIGKILL to the process group doesn't work at this point. We need some other way to detect if children have exited.)

[dombarnes]

Sounds like this could be similar to what I'm seeing happen with vite on a Rails project. After hitting Ctrl C, the other items from my Procfile terminal, but vite takes up to 30s(?) to terminate, but if I run the command natively (yarn vite dev or bin/vite dev) it terminates immediately.

[oleg-vinted]

Not sure if this is the same issue. Could it be that Vite handles SIGTERM and SIGINT differently? If you run Vite natively, on Ctrl-C, its process receives SIGINT. If you run Vite through Foreman/Overman, Vite process receives SIGTERM.

[dentarg]

@oleg-vinted Re: ddollar#780 (comment), does the other foreman variants handle what you have reported here? If so, how do they do it?

[oleg-vinted]

The other variants work fine, and as far as I remember (haven't looked into this since July, please double check) they wait for the stdout pipe to be closed on the child process's end, and this indicates to the process manager that all the subprocesses have exited.

In other words, it waits for both (1) immediate child PID to exit, (2) the stdout pipe to close on the other end.

(1) can occur before (2), and (2) is the missing part in overman.

[dentarg]

Maybe use https://github.com/enkessler/childprocess? Or look at it for inspiration (thinking of enkessler/childprocess#175)