Skip to content

Wrap Augustus rerun extraction in timeout-safe error handling#1170

Merged
hyphaltip merged 10 commits into
masterfrom
enable_augustustimeout_busco
Jun 12, 2026
Merged

Wrap Augustus rerun extraction in timeout-safe error handling#1170
hyphaltip merged 10 commits into
masterfrom
enable_augustustimeout_busco

Conversation

@hyphaltip

@hyphaltip hyphaltip commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator

_augustus_rerun called _extract() without any error handling, meaning a timed-out or truncated Augustus output file in the second pass would crash the entire run. The first-pass extraction already had a try/except guard; the second pass was missing it.

Change

  • _augustus_rerun extraction loop (funannotate-BUSCO2.py ~L1502): wrapped _extract(mainout, plain_name) and _extract(mainout, plain_name, aa=False) in the same try/except Exception block used by the first-pass loop, logging a warning and skipping the offending file rather than propagating the exception.
try:
    self._extract(self.mainout, plain_name)
    self._extract(self.mainout, plain_name, aa=False)
except Exception as e:
    _logger.warning(
        'Failed to extract proteins from %s (possibly truncated by timeout), skipping: %s'
        % (plain_name, e)
    )

This makes timeout resilience consistent across both Augustus passes.

hyphaltip and others added 4 commits June 6, 2026 18:22
Individual Augustus jobs in funannotate-BUSCO2.py had no timeout, so a
hung job would block the entire BUSCO training run indefinitely. Adds:
- --augustus_timeout (default 300s): kills stuck Augustus jobs using
  SIGKILL on the process group (start_new_session=True is thread-safe;
  avoids orphaning the augustus child under shell=True)
- per-job elapsed time tracking across all worker threads with a summary
  log after the Augustus pass (mean, median, max, timeout count) so users
  can empirically tune the timeout value
- try/except around _extract calls to survive any truncated output files
  left by killed jobs, losing at most one gene per timeout

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Passes the per-job Augustus timeout through to funannotate-BUSCO2.py via
--augustus_timeout. Default is 300s, matching the BUSCO2 script default.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses issue #1169 where Augustus can hang indefinitely during BUSCO-based training by introducing a per-job timeout that kills long-running Augustus subprocesses and allows the overall run to continue.

Changes:

  • Added --augustus_timeout to funannotate-BUSCO2.py and enforced per-job timeouts in threaded Augustus execution (with timing/timeout summary logging).
  • Forwarded a new funannotate predict CLI option (--augustus_BUSCO_timeout) into the BUSCO2 wrapper as --augustus_timeout.
  • Documented the new behavior and flags in CHANGES.md.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
funannotate/predict.py Adds a new CLI flag and forwards it to the BUSCO2 subprocess invocation.
funannotate/aux_scripts/funannotate-BUSCO2.py Implements per-job Augustus timeouts, kill logic, and post-run timing/timeout logging.
CHANGES.md Documents the new timeout feature and related CLI options.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread funannotate/aux_scripts/funannotate-BUSCO2.py
Comment thread funannotate/aux_scripts/funannotate-BUSCO2.py
Comment thread CHANGES.md Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
hyphaltip and others added 2 commits June 6, 2026 19:26
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI changed the title Enable augustustimeout busco Wrap Augustus rerun extraction in timeout-safe error handling Jun 7, 2026
@hyphaltip hyphaltip merged commit 5c0f466 into master Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants