Failed slurm jobs leave dependencies in queue

[ltalirz]

By the way, I notice that if an early task exits with an error, the dependent slurm jobs are not cleaned up but remain stuck in slurm - not sure whether this is intended behavior

     90464 main_queu                  merge_results  PENDING       0:00    1       0  UNLIMITED (DependencyNeverSatisfied)
     90463 main_queu                            cte  PENDING       0:00    8       0  UNLIMITED (DependencyNeverSatisfied)
     90462 main_queu                      viscosity  PENDING       0:00    8       0  UNLIMITED (DependencyNeverSatisfied)
     90461 main_queu                      structure  PENDING       0:00    1       0  UNLIMITED (DependencyNeverSatisfied)

Originally posted by @ltalirz in #62

The question here is: what do we need to do to fix this? (we don't want to pollute the slurm queue)

[jan-janssen]

@ltalirz I just returned from my holidays and try to slowly catch up on open issues. Have you tried adding --kill-on-invalid-dep=yes ? https://slurm.schedmd.com/sbatch.html#OPT_kill-on-invalid-dep

[ltalirz]

Ah thanks, I haven't tried that. Would I need to use a slurm template for this with executorlib?

If executorlib does not have a built-in cleanup mechanism, it would make sense to me if executorlib would be setting this flag by default

[jan-janssen]

If executorlib does not have a built-in cleanup mechanism, it would make sense to me if executorlib would be setting this flag by default

Executorlib is an interface to job schedulers like SLURM. As job schedulers commonly do not allow to access the job queue from a given job allocation, what you are asking for would require a daemon process which watches the job scheduler for orphan jobs. Technically this is possible, but the much easier option would be to enable this feature directly in SLURM, still that might require recompiling SLURM with different build options.

[ltalirz]

I'm not quite following, perhaps I did not express myself clearly.

I am simply wondering whether executorlib should by default inject

#SBATCH --kill-on-invalid-dep=yes

into the sbatch scripts it generates?