Search before asking
Fluss version
main (development)
Please describe the bug 🐞
Description
For an Iceberg-enabled Fluss table, Admin#getTableInfo (and consumers such as catalogs or UIs) exposes cluster-level Iceberg catalog options as table properties by adding a table. prefix.
In the reproduced setup, the Fluss cluster is configured with:
datalake.format: iceberg
datalake.iceberg.type: glue
datalake.iceberg.warehouse: s3://<bucket>/fluss/iceberg
an Iceberg-enabled table is reported with:
table.datalake.iceberg.warehouse=s3://<bucket>/fluss/iceberg
However, an existing Iceberg table may be stored at an explicit location such as s3://<bucket>/fluss/native_implicit_day_pk_demo_v2. The reported warehouse still points to s3://<bucket>/fluss/iceberg, even though the table is stored somewhere else. Because Fluss exposes this value as a table property, users can easily mistake it for the table's actual location.
How to reproduce
- Use a Fluss cluster configured with
datalake.iceberg.warehouse: s3://<bucket>/fluss/iceberg.
- Create
native_implicit_day_pk_demo_v2 in the same Glue catalog with the explicit table location s3://<bucket>/fluss/native_implicit_day_pk_demo_v2.
- Create or bind the matching Fluss table with
table.datalake.enabled=true.
- Call
Admin#getTableInfo or inspect the table through a catalog/UI.
- Observe that
table.datalake.iceberg.warehouse contains the cluster-level default warehouse even though the Iceberg table uses a different location.
Expected behavior
TableInfo.properties should not make inherited cluster-level catalog configuration look like persisted table properties. Inherited/resolved catalog options should be distinguishable from stored table properties, and a warehouse value should not be presented as the actual location of an existing Iceberg table.
Current behavior and impact
LakeStorageUtils#generateDefaultTableLakeOptions prefixes cluster options with table., and TableRegistration#toTableInfo overlays them when returning TableInfo.
This is misleading for users and metadata consumers. It also means the reported table properties can change when cluster configuration changes, without any ALTER TABLE, and the problem is visible when binding existing Iceberg tables with custom locations.
Related history
Solution
Separate persisted table properties from inherited/resolved lake catalog options.
Possible approaches:
- expose resolved lake catalog options through a dedicated metadata structure instead of merging them into
TableInfo.properties; or
- retain the resolved options but mark their source explicitly and keep persisted table properties separate.
Add a regression test with a cluster warehouse and an existing Iceberg table whose explicit location differs from that warehouse.
Are you willing to submit a PR?
Search before asking
Fluss version
main (development)
Please describe the bug 🐞
Description
For an Iceberg-enabled Fluss table,
Admin#getTableInfo(and consumers such as catalogs or UIs) exposes cluster-level Iceberg catalog options as table properties by adding atable.prefix.In the reproduced setup, the Fluss cluster is configured with:
an Iceberg-enabled table is reported with:
However, an existing Iceberg table may be stored at an explicit location such as
s3://<bucket>/fluss/native_implicit_day_pk_demo_v2. The reported warehouse still points tos3://<bucket>/fluss/iceberg, even though the table is stored somewhere else. Because Fluss exposes this value as a table property, users can easily mistake it for the table's actual location.How to reproduce
datalake.iceberg.warehouse: s3://<bucket>/fluss/iceberg.native_implicit_day_pk_demo_v2in the same Glue catalog with the explicit table locations3://<bucket>/fluss/native_implicit_day_pk_demo_v2.table.datalake.enabled=true.Admin#getTableInfoor inspect the table through a catalog/UI.table.datalake.iceberg.warehousecontains the cluster-level default warehouse even though the Iceberg table uses a different location.Expected behavior
TableInfo.propertiesshould not make inherited cluster-level catalog configuration look like persisted table properties. Inherited/resolved catalog options should be distinguishable from stored table properties, and a warehouse value should not be presented as the actual location of an existing Iceberg table.Current behavior and impact
LakeStorageUtils#generateDefaultTableLakeOptionsprefixes cluster options withtable., andTableRegistration#toTableInfooverlays them when returningTableInfo.This is misleading for users and metadata consumers. It also means the reported table properties can change when cluster configuration changes, without any
ALTER TABLE, and the problem is visible when binding existing Iceberg tables with custom locations.Related history
describeLakeStorage.Solution
Separate persisted table properties from inherited/resolved lake catalog options.
Possible approaches:
TableInfo.properties; orAdd a regression test with a cluster warehouse and an existing Iceberg table whose explicit location differs from that warehouse.
Are you willing to submit a PR?