shutdown Celery workers do not have an opportunity to reset their task status

[MoralCode]

We observed in production that Flower was showing 0 core tasks running for an extended period of time, yet the collection_status operations table had ~40 core tasks listed as Collecting (with UUIDs).

confirming (by searching the repo url for those repos in flower) that these tasks were not currently running and hadnt already completed, we concluded that they were stale/the DB's picture of collection status had gotten out of sync with flower.

Changing a few of these tasks from Collecting with a UUID, to Pending (and nulling out the UUID) seems to have partially cleared the block on core workers.

When asked to identify bugs that could cause this out of sync-ness/stale tasks, Generative AI identified 4 issues. Three of these were deemed (by me) to be not relevant and/or intended behavior, but the one that was promising seemed to be:

No on_revoke / task revocation handler

celery_app.py Lines 1-10
from celery.signals import worker_process_init, worker_process_shutdown
There's no task_revoked or task_prerun/task_postrun signal handler that would reset COLLECTING to ERROR/PENDING when tasks are revoked during worker shutdown. The on_failure on CoreRepoCollectionTask is only triggered by exceptions raised inside the task function — not by SIGKILL, OOM kills, or Celery task revocation.

This seems like a very plausible error chain:

1. a celery worker gets shutdown or recreated for some reason
2. we have no error handler to catch this when that workers tasks are revoked
3. we subsequently never change the status of those now-revoked tasks in the DB
4. even if/when the new workers come back, the collection monitor doesnt pick up any more tasks because it thinks core collection is already running at full beans

[MoralCode]

I'm realizing that this may not have much of a meaningful impact - i think the primary value of this is to allow the "terminate" button in flower (which i suspect issues a revocation of an incomplete celery task) to function

https://docs.celeryq.dev/en/stable/userguide/signals.html#task-revoked seems to suggest this is mostly for revocations from within celery itself (or this flower button)