Skip to content

fix(rest): release fetched FileScanTask state to prevent unbounded memory growth - #17429

Open
Solaris-star wants to merge 1 commit into
apache:mainfrom
Solaris-star:fix/17427-release-fetched-plan-tasks
Open

fix(rest): release fetched FileScanTask state to prevent unbounded memory growth#17429
Solaris-star wants to merge 1 commit into
apache:mainfrom
Solaris-star:fix/17427-release-fetched-plan-tasks

Conversation

@Solaris-star

Copy link
Copy Markdown

Summary

Fixes #17427

The reference REST server-side scan planning implementation retains all planned FileScanTask objects in the singleton InMemoryPlanningState even after clients have successfully fetched every plan task. Neither fileScanTasksForPlanTask nor nextPlanTask removes the fetched entries, and the state is only released via cancelPlan. Per the REST Catalog OpenAPI spec, cancellation is not required after all plan tasks have been fetched, so a successful fetch lifecycle must release state without an explicit cancel request.

Fix

Add two methods to InMemoryPlanningState:

  • releasePlanTask(planTaskKey) — removes a single fetched plan task's FileScanTask list and next-task link
  • releaseAsyncPlanForTask(planTaskKey) — removes the async planning state when the last plan task in a chain is fetched (extracts the planId from the key format {planId}-{tableId}-{sequence})

Call both from CatalogHandlers.fetchScanTasks after building the response:

IN_MEMORY_PLANNING_STATE.releasePlanTask(planTask);
if (nextPlanTasks.isEmpty()) {
  IN_MEMORY_PLANNING_STATE.releaseAsyncPlanForTask(planTask);
}

Tests

Added TestInMemoryPlanningState with 4 unit tests:

  • releasePlanTask removes both file scan tasks and next-task link
  • releaseAsyncPlanForTask removes async planning state
  • malformed keys are handled gracefully (no exception, no side effects)
  • releasing unknown keys is a no-op

Note: I could not run the full Gradle test suite locally (partial clone conflicts with the multi-module build), but the changes are minimal Map.remove() calls following existing patterns in the codebase. CI should validate.

…mory growth

The reference REST server-side scan planning implementation retains all
planned FileScanTask objects in the singleton InMemoryPlanningState even
after clients have successfully fetched every plan task. Neither
fileScanTasksForPlanTask nor nextPlanTask removes the fetched entries,
and the state is only released via cancelPlan. Per the REST Catalog
OpenAPI spec, cancellation is not required after all plan tasks have been
fetched, so a successful fetch lifecycle must release state without an
explicit cancel request (apache#17427).

Add releasePlanTask() to remove a single fetched plan task's FileScanTask
list and next-task link, and releaseAsyncPlanForTask() to remove the async
planning state when the last plan task in a chain is fetched. Call both
from CatalogHandlers.fetchScanTasks after building the response.

Adds TestInMemoryPlanningState with unit tests covering:
- releasePlanTask removes both file scan tasks and next-task link
- releaseAsyncPlanForTask removes async planning state
- malformed keys are handled gracefully
- releasing unknown keys is a no-op

Note: could not run the full Gradle test suite locally (partial clone
conflicts with the multi-module build), but the changes are minimal
Map.remove() calls following existing patterns in the codebase.
@github-actions github-actions Bot added the core label Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] REST server-side scan planning retains fetched FileScanTasks indefinitely

1 participant