[rust] Limit scan fails to read Arrow log batches written before ADD COLUMN

[slfan1989]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.9.0 (latest release)

Please describe the bug 🐞

The Rust LimitBatchScanner log path does not handle schema evolution when reading Arrow log batches.

decode_log_batch creates a single ArrowReadContext using the current table schema and uses it to decode every returned LogRecordBatch. However, each log batch carries its own schema_id and may have been written using an older schema.

As a result, after an ADD COLUMN, a limit scan that includes batches written before the schema change fails because the decoder expects the newly added Arrow field to exist in the old batch.

Steps to reproduce

1. Create an Arrow log table with the following schema:

   id INT,
   name STRING

2. Append and flush one or more records.
3. Add a new column:

ALTER TABLE ... ADD COLUMN age BIGINT;

4. Create a limit scanner using the latest table metadata.
5. Scan a bucket containing records written before the schema change.

A minimal unit-level reproduction is to:

1. Build an Arrow log batch using the old two-column schema.
2. Call decode_log_batch with a TableInfo containing the new three-column schema.

Solution

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!