Skip to content

feat(storage/opendal): route oss:// URLs through the S3 backend#2332

Open
plusplusjiajia wants to merge 2 commits into
apache:mainfrom
plusplusjiajia:feat/oss-routes-through-s3
Open

feat(storage/opendal): route oss:// URLs through the S3 backend#2332
plusplusjiajia wants to merge 2 commits into
apache:mainfrom
plusplusjiajia:feat/oss-routes-through-s3

Conversation

@plusplusjiajia
Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A

What changes are included in this PR?

Aliyun OSS exposes an S3-compatible API, and the other two Iceberg implementations already read oss:// tables through their S3 code path with canonical s3.* properties:

iceberg-rust routes oss:// to opendal's native Oss service instead, which reads a separate oss.* namespace and can't be driven by the same credentials. Align with the rest of the ecosystem.

Copy link
Copy Markdown
Collaborator

@CTTY CTTY left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @plusplusjiajia thanks for the contribution! Logic wise this makes sense, but I'm concerned that this will confuse users that OSS storage has two modes. Do you think we should just keep one?

Comment thread crates/storage/opendal/src/resolving.rs Outdated
#[cfg(feature = "opendal-oss")]
// OSS is S3-API-compatible; route through S3 so `s3.*` props
// work for OSS-backed tables (mirrors pyiceberg/Java S3FileIO).
#[cfg(feature = "opendal-s3")]
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[cfg(feature = "opendal-s3")]
#[cfg(all(feature = "opendal-oss", feature = "opendal-s3"))]

Only having s3 feature flag will introduce oss related code to non-oss users

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CTTY Fixed.

Comment thread crates/storage/opendal/src/resolving.rs Outdated
}
// Fallback: builds without `opendal-s3` but with `opendal-oss`
// still use the native OSS service (which consumes `oss.*` keys).
#[cfg(all(feature = "opendal-oss", not(feature = "opendal-s3")))]
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not very familiar with OSS storage so not sure about it: Is the fallback actually necessary? I'm thinking if using OSS exactly like S3 is the common case, maybe we can just use OpenDalS3Storage directly?

Copy link
Copy Markdown
Member Author

@plusplusjiajia plusplusjiajia May 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CTTY Agreed, removed it. OSS is S3-compatible and the Java/Python Iceberg implementations both route OSS through S3FileIO without a native fallback.

@github-actions
Copy link
Copy Markdown
Contributor

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label May 16, 2026
@plusplusjiajia plusplusjiajia force-pushed the feat/oss-routes-through-s3 branch from db91b0b to 3260943 Compare May 17, 2026 10:08
@Xuanwo
Copy link
Copy Markdown
Member

Xuanwo commented May 17, 2026

iceberg-rust routes oss:// to opendal's native Oss service instead, which reads a separate oss.* namespace and can't be driven by the same credentials. Align with the rest of the ecosystem.

Instead of trying to route OSS to S3, I suggest allowing OSS to parse options from the S3 config.

OpenDAL's OSS support is natively implemented, which means it works with Aliyun OSS's RAM out of the box. Switching to S3 services prevents users from working with Aliyun's OIDC or assume roles.

@plusplusjiajia
Copy link
Copy Markdown
Member Author

Hi @plusplusjiajia thanks for the contribution! Logic wise this makes sense, but I'm concerned that this will confuse users that OSS storage has two modes. Do you think we should just keep one?

Good point. I've removed the native OSS fallback entirely — OSS now only routes through S3, matching what pyiceberg does (apache/iceberg-python#1392).

@plusplusjiajia plusplusjiajia force-pushed the feat/oss-routes-through-s3 branch from 3260943 to b73b446 Compare May 17, 2026 10:46
@github-actions github-actions Bot removed the stale label May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants