zenoh-flat transition (integration branch)#482
Draft
milyin wants to merge 3 commits into
Draft
Conversation
Tracking document for the rebuild of zenoh-java on the generated zenoh-flat-jni bindings. Constituent PRs target this branch; it merges to main as a whole when the transition completes (removing this file).
* initial code generation
* templates in separate files in prebindgen-ext
* mangle harness
* dependency on zenoh-flat-jni added
* switch to keyexprValidate, package support
* zenoh-jni reexported by zenoh-flat-jni
* No load generator
* output only declared in prebindgen-ext
* better error reporting
* reporting fix
* report unresolved descendants from required roots
BFS from each required-but-unresolved type through struct fields, enum
variants, generic args, and impl Fn args, surfacing transitively
unresolved entries that propagate_required cannot reach (because subs
edges don't exist past an unresolved parent). Without this, a missing
declaration in build.rs for a type only referenced as a field of another
unresolved struct went unreported.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* refactored, test added
* ptr class added
* qualify non-fqn types
* filter undeclared fns out of JNINative/JNIOrphaned emit
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* drop JNIOrphaned bucket; reuse declared_functions trait method
.method() is a namespace declaration — every #[prebindgen] fn must
be assigned to a typed handle, data class, enum, or kotlin_package.
There is no third "orphan" bucket, so JNIOrphaned was always emitted
empty and the duplicate declared_function_names helper just repeated
the PrebindgenExt::declared_functions trait method.
- delete write_jni_orphaned, jni_orphaned_class_name, the stale
generated JNIOrphaned.kt file
- rename render_jni_orphaned_source -> render_jni_package_source,
drop the now-redundant include_only_promoted / wrap_in_object /
declared / class_name params
- drop declared_function_names; render_jni_native_source now takes
&HashSet<syn::Ident> from declared_functions() directly
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* missing functions added to autogeneration
* into in build.rs
* universal post-emit qualification via VisitMut visitor
Replace the five scattered self.qualify_emitted_type() calls (build_input_fn,
build_output_fn, opaque_handle_input, build_handle_destructor_items,
option_input) with a single AST visitor pass applied once at write time.
The previous design qualified only at specific emit sites — each new emit
site had to remember the call. emit_into_dispatcher missed it, so the
impl-Into dispatcher body emitted bare KeyExpr/ZKeyExpr references that
failed to resolve in the binding crate's scope.
Architecture:
- New PrebindgenExt::post_process_item(item: &mut syn::Item) trait hook
(default no-op), invoked by write_rust on every emitted item.
- JniExt implements it via qualify_item, which runs QualifyEmittedTypes —
a syn::visit_mut::VisitMut that walks the full AST (signatures, generic
args, type ascriptions in let bindings, *mut/*const in cast expressions)
and prefixes every bare single-segment Type::Path matching a registered
source-module name with source_module.
- option_input no longer takes &JniExt; the body uses the bare t1 token
and the post-emit visitor qualifies it later.
Also fixes the pre-existing TypeEntry field-name drift in
final_invariant_reports_unresolved_field_of_unresolved_struct.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* transition to zenoh-flat keyexpr for zenoh-java
* ZKeyExpr improved
* closeable data classes
* universa close for data classes
* throwable data class
* config in zenoh-flat
* scouting api
* JNIScout removed
* whatami from jni
* value type
* zenoh id is a wrapper
* &str as input parameter
* logging api
* structured methods
* buildrs api renamr
* scouting package
* packaging fixes
* set intersection level
* zenoh-flat as mods
* qos enums ported
* enum validation in prebindgen
* wrap enums
* bytearray in zenoh-flat
* zenoh-java zbytes
* jni synchronized
* enum return fix
* take and ptr wraper in callback
* manually implemented serialization
* publisher ported
* attachment into
* encoding in zenoh-flat
* encoding constants
* renaming, fromstring added
* zencoding
* subscriber ported
* querier/queriable ported
* query and reply methods ported
* query structure
* renaming
* get and corresponding structures
* full timestamp representation in zenoh-flat
* old log init removed
* liveliness migrated, codegenerator fixed
* value type return fix
* sesstion migrated
* valur type handling fix
* niche fix
* wrappers directly in Session and. Liveliness
* bytes form zenoh_id to string added
* zenoh-jni removed
* jni generator and zenoh-flat moved out
* wrapper lock function
* blob type
* vlaue blob zzenohid
* missing api surface added
* switch to wrapped functions
* single into
* gtom struct from callback
* recursive call
* flatten callbacks
* create java object directly
* flatten input
* no array for locks
* removed expanded code
* expand constructor support
* types expanded
* use accessors
* sample attachment accessor
* nested output
* vec unfold
* comment added
* naming shortened
* converter, renamings
* process error as normal type
* zerror declared
* keyexpr default constructor
* accessor_fun
* full expand
* recursive expamd demo
* new api migration
* zbytes support callback error interface
* expand callbacks
* output kotlin one file per package
* flattened structure
* named callback parameters
* regenrate kotlin code
* generated files reformatted
* cargo fmt
* callback formatting
* no throws in generated code
* hello and query unwrap
* tests fixed
* query close in reply operations
* reply unwrap
* multiline long calls
* sample attributes added
* typed interface callback
* generated code insignificant change
* derfaults in error
* proxy interface
* accelerated onerror handler
* unfold encoding for acceleration
* renamings in builder
* refactor: Remove local zenoh-flat-jni subproject, consume as Maven dependency
- Remove :zenoh-flat-jni from gradle subprojects (moved to separate repo)
- Replace local source dependency with Maven: org.eclipse.zenoh:zenoh-flat-jni:1.9.0
- Remove Cargo build orchestration (buildZenohFlatJni, buildZenohFlatJNI, BuildMode)
- Remove build paths for zenoh-flat-jni libraries (now bundled in Maven JAR)
- Remove Cargo-related Gradle plugins and tasks
- Native libraries are now obtained from the zenoh-flat-jni Maven artifact
- Kotlin sources are now imported from the Maven JAR as a dependency
- Supports optional local development via: includeBuild("../zenoh-flat-jni")
- Simplifies zenoh-java build: Gradle now only compiles Kotlin, not Rust
Migration notes:
- zenoh-flat-jni is now at https://github.com/ZettaScaleLabs/zenoh-flat-jni
- Versions will be synchronized with Zenoh releases (1.9.0, etc.)
- For local development, clone both repos and use composite builds if needed
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* build: Enable Gradle composite build for local zenoh-flat-jni development
- Uncomment includeBuild for ../zenoh-flat-jni since it's now in the PREBINDGEN workspace
- zenoh-flat-jni is at /Users/milyin/PREBINDGEN/zenoh-flat-jni
- Gradle will prefer local build over Maven artifact when composite build is enabled
- Allows coordinated development of zenoh-java and zenoh-flat-jni
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* chore: Remove zenoh-flat-jni directory from zenoh-java
The zenoh-flat-jni project has been moved to:
- GitHub: https://github.com/ZettaScaleLabs/zenoh-flat-jni
- Local workspace: /Users/milyin/PREBINDGEN/zenoh-flat-jni
zenoh-java now consumes zenoh-flat-jni via:
1. Maven artifact (org.eclipse.zenoh:zenoh-flat-jni:1.9.0) in production
2. Gradle composite build (../zenoh-flat-jni) for local development
Removing the local copy avoids duplication and confusion.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* api change ported
* serialize fixes
* loading in zenoh-flat-jni
* import predefined encodings as consts
* Use typed zenoh-flat-jni key expression overloads
* Install CI lint components for the pinned Rust toolchain
* Run flat JNI clippy without a tracked lockfile
* Preserve consuming Rust API semantics
* Limit rustfmt CI to the JNI crate
* Remove stale local task lock
* Use generated JNI receiver methods
* Pin receiver-style JNI bindings in CI
* Track de-prefixed zenoh-flat type names in JNI bindings
zenoh-flat's consolidated API dropped the Z prefix from its Rust types
(ZSample -> Sample, ...), and the generated callback interface names follow
the type name mechanically: ZSampleCallback -> SampleCallback etc. Rename the
four callback references in FlatCallbacks.kt (signatures are unchanged) and
follow the encoding id accessor rename (id -> getId).
Pin CI to the updated zenoh-flat-jni (receiver-style bindings regenerated
against prebindgen main and zenoh-flat main) and to zenoh-flat main.
Verified locally: jvmTest 101 tests, 0 failures against the composite build.
* Use derived JNI factory and getter names
zenoh-flat-jni now derives all regular method names from the flat crate's
identifiers (explicit .name() only where a clash forces it), so the internal
calls follow: JniKeyExpr.newTryFrom/newAutocanonize/newJoin/newConcat,
JniConfig.newFrom*, JniEncoding.newFromString/newWithSchema,
JniZBytes.newFromVec, zSession.getPeersZid/getRoutersZid. SDK public API
unchanged. CI pin bumped to the matching zenoh-flat-jni commit.
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…put override (#484) * Make Encoding and ZenohId.toString pure JVM values An encoding IS its (id, schema) pair — Zenoh's own representation; the textual form is derived from a fixed table. With the id<->name table already JVM-side (the generated ENCODING_* constants) and the companion zenoh-flat-jni change crossing Encoding by value in both directions, the SDK Encoding becomes a plain immutable value: no native handle, no native calls, no lazy caches, no locking, nothing to close. This also removes a native memory leak: received Sample/Query/ReplyError previously retained a cloned Encoding handle with no close path. - Encoding.kt: immutable (id, schema); from()/toString()/withSchema() implement Zenoh's exact conversion rules (incl. the custom-encoding name-preserving withSchema and the render-only CUSTOM<->"" table row); equality on (id, schema), matching Zenoh core. - ZenohId.toString: the little-endian lowercase-hex rule in pure Kotlin. - Callback/fromParts leaves follow the new (encId, encSchema) decomposition; outbound sites pass .id/.schema directly. - Correspondence tests (EncodingCorrespondenceTest, ZenohIdCorrespondenceTest) verify the pure implementations against the native ones across the whole predefined id range, parse/render edge shapes, and random ids — the contract for any JVM-side reimplementation of zenoh-flat API. One test caught and fixed a real divergence (custom-encoding withSchema). Pairs with ZettaScaleLabs/zenoh-flat-jni#4 (CI pin bumped). jvmTest: 105 tests, 0 failures. * Publisher default encoding at declare + PinnedEncoding per-put override Two encoding fast paths over the dual-arm expansion (prebindgen#80, zenoh-flat#3, zenoh-flat-jni#4): - The publisher's default encoding is set NATIVELY once at declare time (PublisherOptions.encoding -> declarePublisher's encoding arm); plain put(payload) passes absent (sel -1), so no encoding data crosses per call. Previously the JVM tier re-sent (id, schema) on every put. - PinnedEncoding (Encoding.pinned()) preallocates the native form; passing it in PutOptions/ReplyOptions/GetOptions crosses only a borrowed handle (native clone = Arc bump) — no schema-string traffic in hot publish loops. close() releases the handle (falls back to the plain (id, schema) arm); a finalizer backstops leaks. A PinnedEncoding equals its plain counterpart. - All outbound sites route through one Encoding?.forWire() helper emitting the selector tuple (-1 absent / 0 value / 1 pinned handle). Tests: PinnedEncodingTest (publisher-default applies natively end-to-end; pinned override reused across puts; post-close fallback), correspondence tests updated to the dual-arm newWithSchema. jvmTest: 107 tests, 0 failures. CI pins: zenoh-flat-jni cbbca40, zenoh-flat d792547. * Move encoding/zid logic to the shared bindings tier; drop pinning zenoh-flat-jni is the shared tier reused by both zenoh-java and zenoh-kotlin, so the pure-JVM conversion logic moves there (EncodingCodec, ZenohId.zidString — next to the generated constants they are built from). io.zenoh.bytes.Encoding shrinks to a thin facade: the established constants and API delegating to the shared codec, zero logic. Drop PinnedEncoding and the selector-tuple wire helper: with the publisher's default encoding set natively at declare time, the hot publish loop crosses no encoding data at all, and per-call overrides crossing (id, schema) cost only a short string decode — not worth a resource class in the public API. Encoding params revert to the plain (present, id, schema) crossing. jvmTest: 106 tests, 0 failures (correspondence suites now exercise the shared codec; PublisherEncodingTest verifies the native declare-time default + per-put override end-to-end). CI pins: zenoh-flat-jni 9bc155b, zenoh-flat main (6d22091, #3 merged).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Umbrella PR for rebuilding zenoh-java on top of the generated JNI/Kotlin bindings (
zenoh-flat-jni, produced by prebindgen from zenoh-flat), replacing the hand-writtenzenoh-jnilayer.This PR stays draft while the transition is in progress. Individual changes land as reviewable PRs targeting the
zenoh-flat-transitionbranch; this PR accumulates them and merges tomainas a whole when the transition is complete. SeeZENOH_FLAT_TRANSITION.mdon the branch for the architecture and the living PR list.Constituent PRs
Merge order
For each constituent PR: upstream companion PRs (prebindgen → zenoh-flat → zenoh-flat-jni) merge first; the zenoh-java PR pins the exact upstream commits in CI and merges here after its checks are green.