Skip to content

[13.x] Job expiration based on job timeout#60751

Open
SjorsO wants to merge 4 commits into
laravel:13.xfrom
SjorsO:retry-at-per-job
Open

[13.x] Job expiration based on job timeout#60751
SjorsO wants to merge 4 commits into
laravel:13.xfrom
SjorsO:retry-at-per-job

Conversation

@SjorsO

@SjorsO SjorsO commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

This PR adds an expire_after_timeout option to the Redis and database queue drivers. When enabled, jobs expire 10 seconds after their timeout, instead of naively based on retry_after.

Quick recap on how job timeouts, expiration, and retries work:

  • Jobs with $tries = 1 fail instantly when they time out
  • Jobs with $tries > 1 get killed when they time out. The queue starts the next attempt when the previous attempt has expired.

Currently, jobs always expire retry_after seconds after they're started, which causes two problems:

  • retry_after must be longer than your longest-running job. If it's shorter, the job gets retried while it's still running, so it ends up running twice at the same time.
  • I've worked on multiple projects where retry_after was set to 1 hour. That meant a short job that timed out would sit for a full hour before retrying. Even worse, nothing gets logged when this happens, so the job essentially disappears for an hour (which is not fun to debug).

Enabling expire_after_timeout fixes both these problems, and in most cases it removes the need for retry_after entirely. retry_after is now only used for jobs that run without a timeout (e.g. $timeout = 0).

retry_after and the new expire_after_timeout are only used by the Redis and database queue drivers:

  • The Redis driver currently stores now + retry_after as the job expiration. With expire_after_timeout enabled it stores now + timeout + 10
  • The database driver uses a reserved_at column that gets set to now when a worker picks up the job. With expire_after_timeout enabled it's now + timeout + 10 - retry_after. This works nicely with the existing query that finds expired jobs via reserved_at <= now - retry_after (it does make the reserved_at name misleading, but I think that's fine)

This PR is backwards compatible and the feature is disabled by default, nothing changes for existing applications when this is merged. A small follow-up PR for Horizon is needed to enable this flag there too.


if (! is_null($timeout)) {
$reservationOffset = (int) $timeout + 10 - $this->retryAfter;
}

@SjorsO SjorsO Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For both the Redis and database drivers:

  • If a timeout is set on the job, that is used
  • If no timeout is set on the job, the timeout on the worker is used
  • If no timeout on the worker or the job, then the retry_after is used (same as now)

"--backoff={$options->backoff}",
"--memory={$options->memory}",
"--sleep={$options->sleep}",
"--timeout={$options->timeout}",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for queue:listen and queue:work --once. For those commands the --timeout option currently isn't used (because pcntl isn't used to kill the worker like it is in queue:work). After this PR the timeout still isn't used for anything except for the job expiration when expire_after_timeout is enabled.


$this->raiseWorkerStartingEvent($connectionName, $queue, $options);

$this->shareWorkerTimeout($connectionName, $options);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For sharing the timeout set on the worker (e.g. queue:work --timeout=300). This timeout is used when a job doesn't define a timeout of it's own.

@jackbayliss

jackbayliss commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

I've also hit this and ended up creating a new queue and connection, just for one job. So, this PR is very much appreciated 🫶 at least puts some light on it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants