Apache Iceberg version
1.9.2
Query engine
Trino
Please describe the bug 🐞
Hi Iceberg community,
I am reporting a failure pattern from an Iceberg production workflow and asking for guidance on safe snapshot expiration policy and operational guardrails.
Summary
We are seeing intermittent read-path failures caused by missing Puffin delete files.
At the workflow level, this appears as a pipeline health-check timeout, but the underlying data-plane failure is a Spark read error from Iceberg metadata/file resolution.
Observed failure path
A workflow controller starts or validates a set of dependent Spark reader jobs.
One or more reader jobs attempt to scan an Iceberg table.
The read fails with NotFoundException for a deletes.puffin file.
Because these reader jobs are required for workflow health, the overall validation step times out and fails.
Important context
Reader jobs are standard current-state table reads (not intentionally pinned to old snapshots).
We run snapshot maintenance with:
retention window = 14 days
retainLast = 4000 snapshots
We also have multiple operational workflows acting on the same physical table storage location, and maintenance/read workloads can overlap in time.
Why we are asking
Even with 14 days and retainLast 4000, we still observed missing Puffin-file references at read time.
We want to understand whether this is mainly:
policy sizing (retention/keep count too aggressive),
maintenance/read concurrency timing,
multi-actor coordination on shared table storage,
or known caveats around snapshot/file lifecycle for delete vectors.
How we interpret retention window vs retainLast
retention window: protects snapshots newer than X days.
retainLast: protects the newest N snapshots regardless of age.
effective history kept is whichever rule keeps more snapshots for the table churn profile.
That said, we are seeing that these parameters alone may not prevent missing-file failures when maintenance overlaps with active reads or when multiple actors can affect the same physical table path.
Questions for maintainers
What is the recommended policy framework for delete-vector tables to avoid missing Puffin file failures?
How should retention window be sized relative to realistic reader lag, retries, and restart windows?
How should retainLast be sized relative to snapshot churn rate?
Is it recommended to enforce strict maintenance windows with no active readers?
In shared-storage or multi-workflow environments, is a single-owner model for maintenance strongly recommended?
Are there known pitfalls when combining expireSnapshots and orphan-file cleanup that can surface as missing Puffin files during active reads?
Are there preflight/postflight checks you recommend around expiration to detect unsafe conditions early?
Operational policy we are considering
Increase retention and retainLast beyond current 14 days and 4000 snapshots.
Run snapshot expiration only in tightly controlled maintenance windows.
Ensure no out-of-band physical file deletions.
Centralize maintenance ownership per physical table location.
Add preflight and postflight read validation around maintenance runs.
If there is existing documentation or prior issue discussion for this exact Puffin missing-file pattern, please point us to it. We are happy to help improve docs once the recommended guardrails are clear.
Thanks.
Willingness to contribute
Apache Iceberg version
1.9.2
Query engine
Trino
Please describe the bug 🐞
Hi Iceberg community,
I am reporting a failure pattern from an Iceberg production workflow and asking for guidance on safe snapshot expiration policy and operational guardrails.
Summary
We are seeing intermittent read-path failures caused by missing Puffin delete files.
At the workflow level, this appears as a pipeline health-check timeout, but the underlying data-plane failure is a Spark read error from Iceberg metadata/file resolution.
Observed failure path
A workflow controller starts or validates a set of dependent Spark reader jobs.
One or more reader jobs attempt to scan an Iceberg table.
The read fails with NotFoundException for a deletes.puffin file.
Because these reader jobs are required for workflow health, the overall validation step times out and fails.
Important context
Reader jobs are standard current-state table reads (not intentionally pinned to old snapshots).
We run snapshot maintenance with:
retention window = 14 days
retainLast = 4000 snapshots
We also have multiple operational workflows acting on the same physical table storage location, and maintenance/read workloads can overlap in time.
Why we are asking
Even with 14 days and retainLast 4000, we still observed missing Puffin-file references at read time.
We want to understand whether this is mainly:
policy sizing (retention/keep count too aggressive),
maintenance/read concurrency timing,
multi-actor coordination on shared table storage,
or known caveats around snapshot/file lifecycle for delete vectors.
How we interpret retention window vs retainLast
retention window: protects snapshots newer than X days.
retainLast: protects the newest N snapshots regardless of age.
effective history kept is whichever rule keeps more snapshots for the table churn profile.
That said, we are seeing that these parameters alone may not prevent missing-file failures when maintenance overlaps with active reads or when multiple actors can affect the same physical table path.
Questions for maintainers
What is the recommended policy framework for delete-vector tables to avoid missing Puffin file failures?
How should retention window be sized relative to realistic reader lag, retries, and restart windows?
How should retainLast be sized relative to snapshot churn rate?
Is it recommended to enforce strict maintenance windows with no active readers?
In shared-storage or multi-workflow environments, is a single-owner model for maintenance strongly recommended?
Are there known pitfalls when combining expireSnapshots and orphan-file cleanup that can surface as missing Puffin files during active reads?
Are there preflight/postflight checks you recommend around expiration to detect unsafe conditions early?
Operational policy we are considering
Increase retention and retainLast beyond current 14 days and 4000 snapshots.
Run snapshot expiration only in tightly controlled maintenance windows.
Ensure no out-of-band physical file deletions.
Centralize maintenance ownership per physical table location.
Add preflight and postflight read validation around maintenance runs.
If there is existing documentation or prior issue discussion for this exact Puffin missing-file pattern, please point us to it. We are happy to help improve docs once the recommended guardrails are clear.
Thanks.
Willingness to contribute