feat: support HDFS object store by hfutatzhanghb · Pull Request #5472 · lance-format/lance

hfutatzhanghb · 2025-12-15T10:04:53Z

What changed

add an optional hdfs feature for lance-io backed by OpenDAL
register hdfs:// with the object store registry
map HDFS URLs, storage options, and supported environment variables into OpenDAL configuration
add unit coverage for HDFS path and configuration handling

Why

This allows Lance datasets to be accessed through HDFS URLs, including deployments using HDFS nameservices.

Validation

cargo fmt --all -- --check
cargo clippy -p lance-io --all-targets --features hdfs --locked -- -D warnings
cargo clippy --all --tests --benches -- -D warnings
cargo test -p lance-io --features hdfs --locked object_store::providers::hdfs::tests --no-fail-fast
cargo +1.91.0 check -p lance-io --features hdfs --locked

chatgpt-codex-connector · 2025-12-15T10:04:58Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

hfutatzhanghb · 2025-12-15T10:05:51Z

Hi, @jackye1995 @wojiaodoubao @majin1102 . Could you please help review this PR when you have free time? Thanks a lot.

Xuanwo · 2025-12-15T10:13:55Z


 [features]
-default = ["aws", "azure", "gcp"]
+default = ["aws", "azure", "gcp", "hdfs"]


I think we shouldn't enable HDFS by default. It introduces many new dependencies in Lance and also requires users to have a Java setup. Without it, Lance will fail to start.

Xuanwo · 2025-12-15T10:15:23Z

+                }
+            } else {
+                // Fall back to system username
+                config_map.insert("user".to_string(), whoami::username());


I prefer having users fill this out instead of using whoami.

Xuanwo · 2025-12-15T10:16:32Z

                                    <copyTo>${project.build.directory}/classes/nativelib</copyTo>
                                    <copyWithPlatformDir>true</copyWithPlatformDir>
+                                    <environmentVariables>
+                                        <CARGO_FEATURES>hdfs</CARGO_FEATURES>


I feel the same. It doesn’t seem like a good idea to enable hdfs by default.

@Xuanwo Thanks for reviewing. Got it, will push an new version based on your advice laterly.

github-actions · 2026-05-16T02:19:42Z

Thank you for your contribution. This PR has been inactive for a while, so we're closing it to free up bandwidth. Feel free to reopen it if you still find it useful.

codecov · 2026-06-09T16:42:41Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

hfutatzhanghb · 2026-06-10T01:54:42Z

@Xuanwo @BubbleCal @jiaoew1991 Hi， could you please review this PR when free? Thanks very much. We have use HDFS as backend storage for a long time.

hfutatzhanghb · 2026-06-11T02:33:52Z

@yanghua Hi， could you please help review this PR when have free time? Thanks very much!

yanghua · 2026-06-11T03:29:01Z

@claude review

hfutatzhanghb · 2026-06-11T05:38:43Z

@claude review
😂， No output ? @claude

yanghua · 2026-06-11T06:05:25Z

It seems the bot does not work. I could take a look at this PR later.

hfutatzhanghb · 2026-06-11T08:44:48Z

It seems the bot does not work. I could take a look at this PR later.

Thanks！

hfutatzhanghb · 2026-06-16T03:33:34Z

@hfutatzhanghb, fix the conflicts?

have fixed and pushed. Let's wait pipeline ok. Thanks for your reviewing!!!

yanghua · 2026-06-18T02:06:00Z

Will take a look today.

yanghua

+1 cc @Xuanwo

Xuanwo

Thanks for adding HDFS support. I think this needs a few correctness fixes before merge.

The main blocker is that this PR registers hdfs:// as writable, but commit routing still falls through to UnsafeCommitHandler. That means concurrent HDFS writers can overwrite manifest versions without real create-if-absent / conflict detection semantics.

There are also two HDFS identity/listing issues:

HDFS is advertised as list_is_lexically_ordered=true, but the backend does not guarantee sorted listings before Lance uses listing order for manifest discovery.
store_prefix is derived from the URI authority even when hdfs_name_node / HDFS_NAME_NODE overrides the actual NameNode, so different effective clusters can be treated as the same datastore.

Please route HDFS to a safe commit handler or explicitly reject HDFS writes without one, default HDFS listings to unordered unless sorted in the provider, and derive store_prefix from the resolved effective NameNode.

hfutatzhanghb · 2026-06-19T10:03:35Z

Thanks for adding HDFS support. I think this needs a few correctness fixes before merge.

The main blocker is that this PR registers hdfs:// as writable, but commit routing still falls through to UnsafeCommitHandler. That means concurrent HDFS writers can overwrite manifest versions without real create-if-absent / conflict detection semantics.

There are also two HDFS identity/listing issues:

HDFS is advertised as list_is_lexically_ordered=true, but the backend does not guarantee sorted listings before Lance uses listing order for manifest discovery.

store_prefix is derived from the URI authority even when hdfs_name_node / HDFS_NAME_NODE overrides the actual NameNode, so different effective clusters can be treated as the same datastore.

Please route HDFS to a safe commit handler or explicitly reject HDFS writes without one, default HDFS listings to unordered unless sorted in the provider, and derive store_prefix from the resolved effective NameNode.

@Xuanwo Thanks for reviewing.

For UnsafeCommitHandler, we have already implements a HdfsRenameCommitHandler and planed to push in another PR after this mergeed. Now have pushed in this PR.
For indentity/listing issues: Will fix immediately.

The hdfs feature requires Hadoop native libraries (libhdfs.so) which are not available on CI runners. Exclude it from ALL_FEATURES computation, following the same pattern as protoc and slow_tests. Also update Cargo.lock to include hdfs-related dependencies (hdrs, hdfs-sys, java-locator, opendal-service-hdfs) so --locked builds don't fail when hdfs is accidentally enabled.

Add .filter(|v| !v.is_empty()) to hdfs_name_node and hdfs_user storage options lookups for consistency with the kerberos and atomic_write_dir fields.

This test was a duplicate of test_hdfs_store_paths in the unit test file, both exercise extract_path logic with the same URL patterns.

hfutatzhanghb · 2026-06-19T16:18:59Z

@Xuanwo @yanghua To provide better rename semantics for the Lance HDFS feature, we also optimized the libhdfs rename operation in apache/hadoop#8411.

Xuanwo · 2026-06-19T19:00:20Z

    "num-traits",
    "std",
 ] }
+hdrs = "0.3.2"


We should not depend on this

Hi, @Xuanwo . Do you mean we should avoid using hdrs directly and implement the HDFS commit rename through OpenDAL instead? My concern is preserving the rename-if-destination-does-not-exist semantics required for commit conflict detection.

OpenDAL’s HDFS rename currently overwrites the destination by deleting it first. The manifest commit requires an atomic rename-if-destination-does-not-exist operation for conflict detection. Would you prefer adding conditional rename support to OpenDAL first, or is there another OpenDAL API intended for this semantic?

github-actions Bot added enhancement New feature or request A-java Java bindings + JNI labels Dec 15, 2025

Xuanwo reviewed Dec 15, 2025

View reviewed changes

hfutatzhanghb force-pushed the dev-hdfs-support branch from 84a92c5 to c6087b6 Compare January 13, 2026 08:36

github-actions Bot added the Stale label May 6, 2026

github-actions Bot closed this May 16, 2026

BubbleCal reopened this May 28, 2026

github-actions Bot removed the Stale label May 29, 2026

hfutatzhanghb force-pushed the dev-hdfs-support branch from c6087b6 to 467c486 Compare June 9, 2026 13:22

github-actions Bot added the A-encoding Encoding, IO, file reader/writer label Jun 9, 2026

hfutatzhanghb force-pushed the dev-hdfs-support branch from 467c486 to e646087 Compare June 9, 2026 14:51

hfutatzhanghb changed the title ~~feat: lance supports hdfs scheme~~ feat: support HDFS object storage Jun 9, 2026

github-actions Bot added A-deps Dependency updates A-ci CI / build workflows labels Jun 9, 2026

hfutatzhanghb force-pushed the dev-hdfs-support branch from 07a46bc to a48caed Compare June 10, 2026 02:32

github-actions Bot added A-docs Documentation A-python Python bindings labels Jun 10, 2026

hfutatzhanghb requested a review from Xuanwo June 11, 2026 06:55

hfutatzhanghb force-pushed the dev-hdfs-support branch from 257563f to 6c924e5 Compare June 13, 2026 08:03

hfutatzhanghb changed the title ~~feat: support HDFS object storage~~ feat: support HDFS object store Jun 13, 2026

hfutatzhanghb force-pushed the dev-hdfs-support branch from 563d129 to 8b4178b Compare June 16, 2026 03:26

yanghua approved these changes Jun 18, 2026

View reviewed changes

Xuanwo requested changes Jun 19, 2026

View reviewed changes

zhanghaobo@kanzhun.com added 17 commits June 19, 2026 23:52

feat: lance supports hdfs scheme

70fe27d

fix: improve optional HDFS handling

2cbc753

fix java build failed

fc3d878

docs: explain Java HDFS builds

dd5d0a8

docs: use mvn in Java build examples

3c34728

docs: document HDFS object store setup

f1f3ba1

docs: move HDFS configuration after TOS

330e010

optimize feature hdfs

af19ff3

fix: filter empty string values in HDFS storage options

aee5b6b

Add .filter(|v| !v.is_empty()) to hdfs_name_node and hdfs_user storage options lookups for consistency with the kerberos and atomic_write_dir fields.

refactor: remove duplicate test_hdfs_url_parsing integration test

4155f24

This test was a duplicate of test_hdfs_store_paths in the unit test file, both exercise extract_path logic with the same URL patterns.

docs: explain HDFS integration test opt-in

2486617

test: remove ineffective HDFS config validation

1df96ea

test: explicitly ignore HDFS integration tests

cc0cc65

fix: set list_is_lexically_ordered to false for HDFS

535ab49

fix: derive store_prefix from resolved effective NameNode for HDFS

988ad4b

feat: add safe HDFS rename commit handler via libhdfs

87a6af7

hfutatzhanghb force-pushed the dev-hdfs-support branch from 74d4f33 to 87a6af7 Compare June 19, 2026 15:53

zhanghaobo@kanzhun.com added 2 commits June 20, 2026 00:05

style: fix cargo fmt issues in hdfs commit handler

d87b8d5

test: add unit tests for HDFS commit handler helper functions

1cc2215

zhanghaobo@kanzhun.com added 2 commits June 20, 2026 00:20

fix: move StorageOptionsAccessor import into test module

3de1e1b

chore: sync python/Cargo.lock with workspace hdrs dep

3ac5474

Xuanwo reviewed Jun 19, 2026

View reviewed changes

Conversation

hfutatzhanghb commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Why

Validation

Uh oh!

chatgpt-codex-connector Bot commented Dec 15, 2025

Uh oh!

hfutatzhanghb commented Dec 15, 2025

Uh oh!

Xuanwo Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Xuanwo Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Xuanwo Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 16, 2026

Uh oh!

codecov Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hfutatzhanghb commented Jun 10, 2026

Uh oh!

hfutatzhanghb commented Jun 11, 2026

Uh oh!

yanghua commented Jun 11, 2026

Uh oh!

hfutatzhanghb commented Jun 11, 2026

Uh oh!

yanghua commented Jun 11, 2026

Uh oh!

hfutatzhanghb commented Jun 11, 2026

Uh oh!

hfutatzhanghb commented Jun 16, 2026

Uh oh!

yanghua commented Jun 18, 2026

Uh oh!

yanghua left a comment

Choose a reason for hiding this comment

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hfutatzhanghb commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Xuanwo Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb Jun 20, 2026

Choose a reason for hiding this comment

Uh oh!

hfutatzhanghb Jun 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hfutatzhanghb commented Dec 15, 2025 •

edited

Loading

codecov Bot commented Jun 9, 2026 •

edited

Loading

hfutatzhanghb commented Jun 19, 2026 •

edited

Loading

hfutatzhanghb commented Jun 19, 2026 •

edited

Loading