Skip to content

Maven tweaks for usability #4039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
headius wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Maven tweaks for usability #4039

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
979881e
Move JNI-based Parser to jni subpackage
headius Mar 24, 2026
5782d74
Generate templated sources under main/java-templates
headius Mar 24, 2026
1687481
Upload an archive of Java sources for TruffleRuby to copy
headius Mar 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/documentation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ jobs:
- name: Check C coverage
run: doxygen Doxyfile
- name: Check Java coverage
run: javadoc -Xdoclint:all,-missing -d doc/java -sourcepath java/api/target/generated-sources/java/:java/api/src/main/java org.ruby_lang.prism
run: javadoc -Xdoclint:all,-missing -d doc/java -sourcepath java/api/src/main/java-templates/:java/api/src/main/java org.ruby_lang.prism
- name: Generate Rust documentation
run: |
bundle exec rake cargo:build
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/github-pages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
- name: Build with Doxygen
run: doxygen Doxyfile
- name: Build with JavaDoc
run: javadoc -Xdoclint:all,-missing -d doc/java -sourcepath java/api/target/generated-sources/java/:java/api/src/main/java org.ruby_lang.prism
run: javadoc -Xdoclint:all,-missing -d doc/java -sourcepath java/api/src/main/java-templates/:java/api/src/main/java org.ruby_lang.prism
- name: Build with rustdoc
run: |
bundle exec rake cargo:build
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/java-wasm-bindings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,4 +55,4 @@ jobs:
- uses: actions/upload-artifact@v7
with:
name: prism.wasm
path: java/wasm/src/test/resources/prism.wasm
path: java/wasm/src/main/wasm/prism.wasm
9 changes: 9 additions & 0 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,15 @@ jobs:
bundler-cache: true
- name: Run Java Loader test
run: PRISM_SERIALIZE_ONLY_SEMANTICS_FIELDS=1 PRISM_JAVA_BACKEND=truffleruby bundle exec rake compile
- name: Archive sources for TruffleRuby
run: |
jar --create --file java/prism-truffleruby-sources.jar -C java/api/src/main/java `ls -r java/api/src/main/java`
jar --update --file java/prism-truffleruby-sources.jar -C java/api/src/main/java-templates `ls -r java/api/src/main/java-templates`
jar --update --file java/prism-truffleruby-sources.jar -C java/native/src/main/java `ls -r java/native/src/main/java`
Comment on lines +204 to +206
Copy link
Copy Markdown
Member

@eregon eregon Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I appreciate the thought and effort but it's better to directly copy files from my Prism checkout, that way it also works if e.g. I have some local changes in Prism I'm testing in TruffleRuby.
So this can be removed.

- uses: actions/upload-artifact@v7
with:
name: truffleruby-sources
path: java/prism-truffleruby-sources.jar

lex-ruby:
runs-on: ubuntu-latest
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ out.svg
/java/org/ruby_lang/prism/AbstractNodeVisitor.java
/java/org/ruby_lang/prism/Loader.java
/java/org/ruby_lang/prism/Nodes.java
/java/wasm/src/test/resources/prism.wasm
/java/wasm/src/main/wasm/prism.wasm
/lib/prism/compiler.rb
/lib/prism/dispatcher.rb
/lib/prism/dot_visitor.rb
Expand Down
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ all: shared static
shared: build/libprism.$(SOEXT)
static: build/libprism.a
wasm: javascript/src/prism.wasm
java-wasm: java/wasm/src/test/resources/prism.wasm
java-wasm: java/wasm/src/main/wasm/prism.wasm

build/libprism.$(SOEXT): $(SHARED_OBJECTS)
$(ECHO) "linking $@ with $(CC)"
Expand All @@ -51,7 +51,7 @@ javascript/src/prism.wasm: Makefile $(SOURCES) $(HEADERS)
-Oz -g0 -flto -fdata-sections -ffunction-sections \
-o $@ $(SOURCES)

java/wasm/src/test/resources/prism.wasm: Makefile $(SOURCES) $(HEADERS)
java/wasm/src/main/wasm/prism.wasm: Makefile $(SOURCES) $(HEADERS)
$(ECHO) "building $@"
$(Q) $(MAKEDIRS) $(@D)
$(Q) $(WASI_SDK_PATH)/bin/clang \
Expand Down
6 changes: 3 additions & 3 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ A lot of code in prism's repository is templated from a single configuration fil
* `include/prism/node_new.h` - for defining the functions that create the nodes in C
* `javascript/src/deserialize.js` - for defining how to deserialize the nodes in JavaScript
* `javascript/src/nodes.js` - for defining the nodes in JavaScript
* `java/api/target/generated-sources/java/org/ruby_lang/prism/AbstractNodeVisitor.java` - for defining the visitor interface for the nodes in Java
* `java/api/target/generated-sources/java/org/ruby_lang/prism/Loader.java` - for defining how to deserialize the nodes in Java
* `java/api/target/generated-sources/java/org/ruby_lang/prism/Nodes.java` - for defining the nodes in Java
* `java/api/src/main/java-templates/org/ruby_lang/prism/AbstractNodeVisitor.java` - for defining the visitor interface for the nodes in Java
* `java/api/src/main/java-templates/org/ruby_lang/prism/Loader.java` - for defining how to deserialize the nodes in Java
* `java/api/src/main/java-templates/org/ruby_lang/prism/Nodes.java` - for defining the nodes in Java
* `lib/prism/compiler.rb` - for defining the compiler for the nodes in Ruby
* `lib/prism/dispatcher.rb` - for defining the dispatch visitors for the nodes in Ruby
* `lib/prism/dot_visitor.rb` - for defining the dot visitor for the nodes in Ruby
Expand Down
3 changes: 3 additions & 0 deletions java/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
api/src/main/java-templates
api/target
native/target
wasm/src/main/wasm
wasm/target
target
.idea
4 changes: 2 additions & 2 deletions java/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Sources under `api` are generated from templates in `../templates`. Those source
$ PRISM_EXCLUDE_PRETTYPRINT=1 PRISM_SERIALIZE_ONLY_SEMANTICS_FIELDS=1 bundle exec rake templates
```

The files go under `api/target/generated-sources/java` and will be removed with `mvn clean`.
The files are generated under `api/src/main/java-templates` and will not be removed with `mvn clean`.

### WASM build of Prism

Expand All @@ -28,7 +28,7 @@ The `wasm` project needs a WASM build of Prism to be generated with the followin
$ make java-wasm WASI_SDK_PATH=<path to WASI sdk>
```

The files go under `wasm/target/generated-sources/wasm` and will be removed with `mvn clean`.
The build is generated under `wasm/src/main/wasm` and will not be removed with `mvn clean`.

### Build and install

Expand Down
2 changes: 1 addition & 1 deletion java/api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
</goals>
<configuration>
<sources>
<source>target/generated-sources/java</source>
<source>src/main/java-templates</source>
</sources>
</configuration>
</execution>
Expand Down
2 changes: 1 addition & 1 deletion ...main/java/org/ruby_lang/prism/Parser.java → .../java/org/ruby_lang/prism/jni/Parser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package org.ruby_lang.prism;
package org.ruby_lang.prism.jni;

public abstract class Parser {
Copy link
Copy Markdown
Member

@eregon eregon Mar 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class, which is the only class under java/native doesn't really do anything currently.
Notably the corresponding C code is not here, it's in https://github.com/truffleruby/truffleruby/blob/master/src/main/c/yarp_bindings/src/yarp_bindings.c.

Maybe we should actually remove this class? Or import the C file here?
If the C file would get compiled in CI that could be nice, as it'd catch cases that it needs to be updated, e.g. with the recent changes in the C API.

OTOH this is an area where usages might want different things (e.g. how the reading is done, maybe from a file/using mmap/etc), and notably truffleruby compiles that .c file together with libprism.a.

I'd tend to say the simplest is to just remove this file, but then maybe there is an issue because java/native is empty?

Copy link
Copy Markdown
Contributor Author

@headius headius Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logical change would be to put the JNI C code in here as well and use Maven to build it. But there's the larger issue of building, packaging, and releasing that JNI library.

I have looked briefly at the available Maven tooling for building and releasing native libraries but never attempted to use them:

https://www.mojohaus.org/maven-native/native-maven-plugin/

Unfortunately the source links for this plugin appear to be dead. I'm not sure what is considered "standard" for building and releasing JNI libraries at this point.

Copy link
Copy Markdown
Contributor Author

@headius headius Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, the link from the mojohouse page was incorrect. The plugins are here:

https://github.com/mojohaus/maven-native

Copy link
Copy Markdown
Contributor Author

@headius headius Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking into other projects that publish native libraries, I have found no standard tooling to do this.

lwjgl builds and publishes an extensive set of native binaries, but they also have an enormous set of ant build files to do it. MacOS example: https://github.com/LWJGL/lwjgl3/blob/4ef1eebe4af235b2934a165e82aeefcaf8d9b893/config/macos/build.xml#L140

Of course JRuby's JFFI also publishes many native libraries, but we have always built them by hand and committed them to the repository due to the challenges of doing this automatically.

Copy link
Copy Markdown
Member

@eregon eregon Mar 30, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the simplest for now would be to remove this file then, we can always add it back later if useful. It's like 3 lines of code and not useful on its own.

It doesn't need to be solved in this PR, but if it removes an extra Maven module and/or simplifies things it seems useful to remove it earlier than later.

Copy link
Copy Markdown
Member

@eregon eregon Mar 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW TruffleRuby and more specifically mx has its own way to build & package this stuff but it's not reusable here as I'd think we wouldn't want a dependency on mx.


Expand Down
2 changes: 1 addition & 1 deletion java/wasm/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@
</goals>
<configuration>
<name>org.ruby_lang.prism.wasm.PrismParser</name>
<wasmFile>src/test/resources/prism.wasm</wasmFile>
<wasmFile>src/main/wasm/prism.wasm</wasmFile>
</configuration>
</execution>
</executions>
Expand Down
2 changes: 1 addition & 1 deletion java/wasm/src/main/java-templates/org/ruby_lang/prism/wasm/WasmResource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package org.ruby_lang.prism.wasm;

public final class WasmResource {
public static final String absoluteFile = "file://${project.basedir}/src/test/resources/prism.wasm";
public static final String absoluteFile = "file://${project.basedir}/src/main/wasm/prism.wasm";

private WasmResource() {}
}
1 change: 0 additions & 1 deletion java/wasm/src/test/resources/.gitignore
View file
Open in desktop

This file was deleted.

0 ...y_lang/prism/AbstractNodeVisitor.java.erb → ...y_lang/prism/AbstractNodeVisitor.java.erb
View file
Open in desktop
File renamed without changes.
0 .../java/org/ruby_lang/prism/Loader.java.erb → ...lates/org/ruby_lang/prism/Loader.java.erb
View file
Open in desktop
File renamed without changes.
0 ...s/java/org/ruby_lang/prism/Nodes.java.erb → ...plates/org/ruby_lang/prism/Nodes.java.erb
View file
Open in desktop
File renamed without changes.
6 changes: 3 additions & 3 deletions templates/template.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -684,9 +684,9 @@ def locals
"javascript/src/deserialize.js",
"javascript/src/nodes.js",
"javascript/src/visitor.js",
"java/api/target/generated-sources/java/org/ruby_lang/prism/Loader.java",
"java/api/target/generated-sources/java/org/ruby_lang/prism/Nodes.java",
"java/api/target/generated-sources/java/org/ruby_lang/prism/AbstractNodeVisitor.java",
"java/api/src/main/java-templates/org/ruby_lang/prism/Loader.java",
"java/api/src/main/java-templates/org/ruby_lang/prism/Nodes.java",
"java/api/src/main/java-templates/org/ruby_lang/prism/AbstractNodeVisitor.java",
"lib/prism/compiler.rb",
"lib/prism/dispatcher.rb",
"lib/prism/dot_visitor.rb",
Expand Down