Find examples and references for non-deterministic compilation

[jensdietrich]

Initial experiments show that we dont observe non-deterministic compilation in our dataset. More research is needed to find examples and perhaps construct a different dataset.

Start wth adding examples and references as comments for this issues.

[jensdietrich]

Xiong et al. Towards build verifiability for Java-based systems [ICSE-SIP'22] mentions some examples in table 2 -- RC2 category. Some are external systems, but Kubernetes-client-project-5.4.1 and docker-maven-plugin-0.36.1 are mentioned. The issues are line number and method ordering. The paper also mentions issues with constant pool entries in internal programs. I have asked one of the authors I know (Filipe Cogo) for more info.

[jensdietrich]

This example illustrates how different callsites (invoke statements) could be generated depending on the order of libraries in the classpath (those would be different versions of the same lib, changing a method signature). For maven this should be deterministic though, so it is not clear how this could occur. Perhaps other build systems do it differently, or special plugins can change the default maven behaviour.

[jensdietrich]

Here is another example: https://bugs.openjdk.org/browse/JDK-8264306. The change made to fix this is to replace a hashmap by a linked hashmap with a predictable iteration order.

[jensdietrich]

Here is the repo from Xiong et al. Towards build verifiability for Java-based systems [ICSE-SIP'22] : https://github.com/jvm-repo-rebuild/reproducible-central/ . The readme has a list of (non)reproducible artifacts. This has then detailed diffs down to the class level and build instructions, but not the actual jar files. I will try to get them next. We may just use this for EQ1 but it would be good if could also get or construct the actual jars.

[wtwhite]

Evidence of apparent determinism in our own runs

On wtwhite-vuw-vm, the 3 jcompile runs 16, 17 and 18 all produced identical results according to the CRC-32s of .class files in the built jars (I first reported this yesterday morning):

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ ./get_crcs.sh 16
--snip--
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ ./get_crcs.sh 17
--snip--
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ ./get_crcs.sh 18
--snip--
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ cd runs
wtwhite@wtwhite-vuw-vm:~/code/jcompile/runs$ ( cd 16 && md5sum jars/*/*.crc > crc.md5 )
wtwhite@wtwhite-vuw-vm:~/code/jcompile/runs$ ( cd 17 && md5sum jars/*/*.crc > crc.md5 )
wtwhite@wtwhite-vuw-vm:~/code/jcompile/runs$ ( cd 18 && md5sum jars/*/*.crc > crc.md5 )
wtwhite@wtwhite-vuw-vm:~/code/jcompile/runs$ md5sum */crc.md5
0b3e974949543422f3f6525d9f9ffb93  16/crc.md5
0b3e974949543422f3f6525d9f9ffb93  17/crc.md5
0b3e974949543422f3f6525d9f9ffb93  18/crc.md5

As expected, the jars themselves are different (likely due to timestamps and/or different file orders):

wtwhite@wtwhite-vuw-vm:~/code/jcompile/runs$ ( cd 16 && md5sum jars/*/*.jar > jar.md5 )
wtwhite@wtwhite-vuw-vm:~/code/jcompile/runs$ ( cd 17 && md5sum jars/*/*.jar > jar.md5 )
wtwhite@wtwhite-vuw-vm:~/code/jcompile/runs$ ( cd 18 && md5sum jars/*/*.jar > jar.md5 )
wtwhite@wtwhite-vuw-vm:~/code/jcompile/runs$ md5sum */jar.md5
42e41ac9ccafa872ebef44053d928884  16/jar.md5
80731890f9e86e34bf0edf8390f19562  17/jar.md5
1f669e900de44af49ca1388eb750f71d  18/jar.md5

Today I found the results from a run on piccolo are identical (which would usually be a good thing):

[whitewa@piccolo ~/code/jcompile]$ ./get_crcs.sh 7_make_-j_4_nohup_with_checkstyle
--snip--
[whitewa@piccolo ~/code/jcompile]$ ( cd runs/7_make_-j_4_nohup_with_checkstyle/ && md5sum jars/*/*.crc > crc.md5 )
[whitewa@piccolo ~/code/jcompile]$ md5sum runs/7_make_-j_4_nohup_with_checkstyle/crc.md5 
0b3e974949543422f3f6525d9f9ffb93  runs/7_make_-j_4_nohup_with_checkstyle/crc.md5

[jensdietrich]

@wtwhite how about the antrl generated sources ? I guess they are identical too ?

[jensdietrich]

I will talk to Filipe, one on the authors of "Towards .." paper tomorrow. Perhaps we should switch to this for the dataset. @wtwhite pls have a look at their repo ( https://github.com/jvm-repo-rebuild/reproducible-central/) to get a feel for feasibility / resources needed. They do have build scripts, and we could focus on the artifacts with already known problems.

[wtwhite]

@wtwhite how about the antrl generated sources ? I guess they are identical too ?

@jensdietrich Yes. The runs in my earlier comment included the checkstyle project, and to be sure I just now checked that a checkstyle jar contains classes built by antlr -- based on the fact that there's no JavadocParser.java in the source tree initially, but there is a JavadocParser.g4 file, and after building there is a JavadocParser.java under generated-sources and a JavadocParser.class under target that gets included in the jar, I think the jars definitely include the results of compiling antlr-generated sources:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ STOP_BEFORE=docker-stop ./docker-build-project.sh eclipse-temurin:11.0.19_7-jdk openjdk-11.0.19 checkstyle checkstyle-10.12.4.jar checkstyle-10.12.4 jars
--snip--
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ find worktrees/pid2231354-checkstyle-10.12.4/src -name 'JavadocParser.*'
worktrees/pid2231354-checkstyle-10.12.4/src/main/resources/com/puppycrawl/tools/checkstyle/grammar/javadoc/JavadocParser.g4
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ find worktrees/pid2231354-checkstyle-10.12.4/target -name 'JavadocParser.*'
worktrees/pid2231354-checkstyle-10.12.4/target/classes/com/puppycrawl/tools/checkstyle/grammar/javadoc/JavadocParser.class
worktrees/pid2231354-checkstyle-10.12.4/target/classes/com/puppycrawl/tools/checkstyle/grammar/javadoc/JavadocParser.g4
worktrees/pid2231354-checkstyle-10.12.4/target/generated-sources/antlr/com/puppycrawl/tools/checkstyle/grammar/javadoc/JavadocParser.interp
worktrees/pid2231354-checkstyle-10.12.4/target/generated-sources/antlr/com/puppycrawl/tools/checkstyle/grammar/javadoc/JavadocParser.java
worktrees/pid2231354-checkstyle-10.12.4/target/generated-sources/antlr/JavadocParser.tokens
worktrees/pid2231354-checkstyle-10.12.4/target/generated-classes/jacoco/com/puppycrawl/tools/checkstyle/grammar/javadoc/JavadocParser.class
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ unzip -l jars/openjdk-11.0.19/checkstyle-10.12.4.jar | grep -F JavadocParser.class
   205889  2023-11-08 23:31   com/puppycrawl/tools/checkstyle/grammar/javadoc/JavadocParser.class

[wtwhite]

Here is another example: https://bugs.openjdk.org/browse/JDK-8264306. The change made to fix this is to replace a hashmap by a linked hashmap with a predictable iteration order.

A long shot, but there might be other, similar issues we could exploit since it seems (I'm guessing based on pathnames) that javac is still creating quite a few plain HashMaps:

wtwhite@wtwhite-vuw-vm:~/code$ git clone git@github.com:openjdk/jdk.git
--snip--
wtwhite@wtwhite-vuw-vm:~/code$ cd jdk/src/jdk.compiler/
wtwhite@wtwhite-vuw-vm:~/code/jdk/src/jdk.compiler$ git grep -n 'new LinkedHashMap'|wc -l
48
wtwhite@wtwhite-vuw-vm:~/code/jdk/src/jdk.compiler$ git grep -n 'new HashMap'|wc -l
117

[wtwhite]

(OTOH, from the sound of "cmp-baseline builds" at https://bugs.openjdk.org/browse/JDK-8264306 it sounds like they're already quite focused on detecting and eliminating this.)

[wtwhite]

I will talk to Filipe, one on the authors of "Towards .." paper tomorrow. Perhaps we should switch to this for the dataset. @wtwhite pls have a look at their repo ( https://github.com/jvm-repo-rebuild/reproducible-central/) to get a feel for feasibility / resources needed. They do have build scripts, and we could focus on the artifacts with already known problems.

First quick look:

Based on the format of their diffoscope output for docker-maven-plugin, there might be 40 projects with bytecode-level differences:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ git grep -l 'javap -verbose -constants' '**/*.diffoscope'
content/ch/qos/reload4j/reload4j-1.2.18.5.diffoscope
content/ch/qos/reload4j/reload4j-1.2.19.diffoscope
content/com/google/guava/guava-parent-32.0.0-jre.diffoscope
content/com/io7m/jade/com.io7m.jade-1.0.2.diffoscope
content/de/gematik/pki/gemlibpki/gemLibPki-0.11.0.diffoscope
content/de/gematik/pki/gemlibpki/gemLibPki-0.9.2.diffoscope
content/de/gematik/pki/gemlibpki/gemLibPki-0.9.3.diffoscope
content/eu/europa/ted/eforms/efx-toolkit-java/efx-toolkit-java-2.0.0-alpha.1.diffoscope
content/io/fabric8/docker-maven-plugin/docker-maven-plugin-0.36.0.diffoscope
content/io/fabric8/docker-maven-plugin/docker-maven-plugin-0.36.1.diffoscope
content/io/fabric8/kubernetes-client/kubernetes-client-project-5.11.1.diffoscope
content/io/fabric8/kubernetes-client/kubernetes-client-project-5.4.0.diffoscope
content/io/fabric8/kubernetes-client/kubernetes-client-project-6.0.0-RC1.diffoscope
content/io/opentelemetry/instrumentation/opentelemetry-instrumentation-api-1.14.0-alpha.diffoscope
content/io/opentelemetry/instrumentation/opentelemetry-instrumentation-api-1.17.0-alpha.diffoscope
content/net/bytebuddy/byte-buddy-parent-1.12.13.diffoscope
content/org/apache/dubbo/dubbo-parent-3.0.5.diffoscope
content/org/apache/helix/helix-1.0.3.diffoscope
content/org/apache/helix/helix-1.0.4.diffoscope
content/org/apache/maven/plugins/maven-project-info-reports-plugin/maven-project-info-reports-plugin-3.2.1.diffoscope
content/org/apache/maven/plugins/maven-wrapper-plugin/maven-wrapper-plugin-3.0.2.diffoscope
content/org/apache/shiro/shiro-root-1.10.0.diffoscope
content/org/apache/shiro/shiro-root-1.11.0.diffoscope
content/org/apache/shiro/shiro-root-2.0.0-alpha-3.diffoscope
content/org/apache/sling/org.apache.sling.feature.cpconverter/org.apache.sling.feature.cpconverter-1.1.10.diffoscope
content/org/apache/ws/commons/axiom/axiom-1.4.0.diffoscope
content/org/codehaus/mojo/jaxb2-maven-plugin/jaxb2-maven-plugin-3.1.0.diffoscope
content/org/eclipse/transformer/transformer-maven-plugin-0.5.0.diffoscope
content/org/glassfish/main/glassfish-main-aggregator-7.0.5.diffoscope
content/org/glassfish/main/glassfish-main-aggregator-7.0.8.diffoscope
content/org/glassfish/main/glassfish-main-aggregator-7.0.9.diffoscope
content/org/jrivard/xmlchai/xmlchai-0.1.0.diffoscope
content/org/junit/junit5/junit-bom-5.8.2.diffoscope
content/org/mockito/mockito-core/mockito-core-4.4.0.diffoscope
content/org/mockito/mockito-core/mockito-core-5.5.0.diffoscope
content/org/owasp/dependency-check/dependency-check-parent-6.5.0.diffoscope
content/org/owasp/dependency-check/dependency-check-parent-7.0.4.diffoscope
content/org/owasp/dependency-check/dependency-check-parent-7.1.0.diffoscope
content/org/slf4j/slf4j-parent-1.7.36.diffoscope
content/org/slf4j/slf4j-parent-2.0.5.diffoscope

Their dataset overlaps ours at least a little, e.g., they build commons-net:3.9.0 under JDK 98 and find 2 of the 7 output artifacts to be non-reproducible -- but these 2 artifacts are -ftp.jar and -examples.jar files, and they differ only in ZIP timestamps and META-INF header lines.

Will keep looking.

[wtwhite]

Learnt/realised a couple of things:

1. https://github.com/jvm-repo-rebuild/reproducible-central is not managed by the authors of "Towards Build Verifiability for Java-based Systems" (Xiong et al.) -- its data and tools (rebuild.sh, etc.) are by a separate group specifically interested in comparing with Maven Central repo, not across multiple local builds. The paper authors just reuse a subset of its projects. This is actually clear from p. 5 of the paper:

   Reproducible Central [5] is part of Reproducible Build efforts, which rebuilds open source Java-based systems and compare the deliverable packages with the stored ones in Maven Central.

2. The actual artifact comparisons (producing log lines like [INFO] Checking against reference build from central...) are not done by anything at this git repo any special docker images it pulls/creates, but rather by maven-artifact-plugin:compare, which appears to be a standard plugin.