Skip to content

Exponential-backoff retry is hand-rolled in three modules; host one util they can all reach #7095

Description

@aglinxinyuan

Task Summary

Exponential-backoff retry is hand-rolled once per module, so there is no single util a reviewer can point a contributor at.

Site Module Kind Knobs
Utils.retry amber Future, waits on a Timer attempts, base backoff ms, timer, onRetry
LakeFSStorageClient.retryWithBackoff common/workflow-core blocking maxAttempts, initialDelayMillis, injectable sleep
FileService.awaitDependency file-service blocking maxAttempts, initialDelayMillis, injectable sleep, description

All three do the same thing: run the operation, on failure wait, double the wait, give up after N attempts, surface the last failure as the cause.

They cannot share code today because of the module graph and one dependency:

amber  ->  common/workflow-core        (amber sees workflow-core, not the reverse)
file-service  ->  common/workflow-core
com.twitter:util-core                  declared only in amber/build.sbt

So Utils (in amber) is invisible to the two blocking copies, and the Future/Timer variant cannot move down without dragging util-core with it.

Proposal:

  1. Host the blocking variant in common/workflow-core -- it needs nothing but Thread.sleep, so no new dependency -- with the injectable-sleep seam both existing copies already use for tests.
  2. Migrate LakeFSStorageClient.retryWithBackoff and FileService.awaitDependency onto it, keeping their current messages and interrupt handling (both restore the interrupt status and fail fast; that behavior must survive).
  3. Leave the Future/Timer variant as Utils.retry in amber, cross-referenced from the blocking one, unless util-core also moves down later.

Related: #7088 introduced the async Utils.retry and pointed region termination at it; this issue is the part that PR deliberately left out to stay a fix.

Task Type

  • Refactor / Cleanup

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions