Add blog post: The Arrow C Data Interface: Zero-Copy Between Rust and the JVM in DataFusion Comet#195
Conversation
Signed-off-by: Amogh Ramesh <ramogh2404@gmail.com>
3d736cc to
50042ff
Compare
|
This @Amogh-2404. I was really assuming that the blog post would draw on all the lessons learned by the contributors that actually put in the hours implementing and optimizing Comet's use of FFI over the past two years rather than just a generic AI generated blog post. I guess I should have made that clear in the issue. Let's see what other contributors think. |
|
Yeah, that's fair @andygrove. I'll leave it to you and the others on whether any version of this worth keeping. |
One thing you could do is use AI to research the releated PRs in datafusion-comet and then try and summarize the lessions /et from there. Then at least it wouldn't be generic, but it would still probably need review by the authors themselves |
|
Thanks, that makes sense. I’ll take another pass from the actual Comet PRs and see if there’s a real post there. If not, I’ll close this one. |
This adds a tutorial blog post on the Arrow C Data Interface, using DataFusion Comet's real Rust↔JVM handoff as the worked example. It addresses apache/datafusion-comet#3678.
Closes apache/datafusion-comet#3678
content/blog/2026-06-06-arrow-c-data-interface.mdcontent/images/arrow-c-data-interface/The post separates the two questions the interface answers — memory layout vs. ownership — and traces a single
Int32column[1, 2, NULL, 4]field by field across the boundary. The Comet section (move_to_spark,from_spark,executePlan,CometPlainVector, thearrow_ffi_safecopy fallbacks) is pinned to source at a fixed commit (e79183ee).Happy to adjust scope, framing, or length before merge — flag anything you'd like covered or cut.