EQ1 nondeterministic compilation

[wtwhite]

• Try to reproduce the 23 public issues from "Towards Build Verifiability for Java-based Systems" (Xiong et al.) ("Huawei group")
  • P6 Constant pool
    • Already attempted in "JAVA_HOME is not correctly set" when building docker-maven-plugin:0.36.0 from https://github.com/jvm-repo-rebuild/reproducible-central #32 (comment): fails to see variation across 100 local builds in same docker container (see ref 19 there)
  • P5 LineNumber table
    • Unlikely to be different as it is just version 0.36.1 of the same artifact and the same class file (HelpMojo.class) is affected
  • P10 Method order (kubernetes-client-project-5.4.1)
    • This is the key one: Check for method order nondeterminism on kubernetes-client-project-5.4.1 #34
• Likely outcome: Fails.
  • Evidence in repo is that Maven Central jar is the reference, not multiple local builds
  • Maven Central artifact only records compiler major version number (Build-Jdk-Spec: 11 in META-INF/MANIFEST.MF) 😞
• Interpretation: Huawei group consider "nondeterminism" to mean variation between "equivalent" build environments (same major version of javac), not run-to-run variation
• Control experiment: Look for pairs records in EQ2 with different results where the major compiler version is the same but minor/patch version is different)
  • Look for unequal bytecode pairs in EQ2 from the same compiler major version #36
• Future work:
  • Is there any nondeterminism in javac?
  • Perhaps add "semantic-preserving mutations" like reordering constant pool

[jensdietrich]

thanks @wtwhite for the good summary of our discussion -- @behnazh-w do you have contacts to anybody from the compiler team to ask whether javac is non-deterministic, and how we could find actual examples (we could of course always synthesise some examples).

[behnazh-w]

@behnazh-w do you have contacts to anybody from the compiler team to ask whether javac is non-deterministic, and how we could find actual examples (we could of course always synthesise some examples).

@jensdietrich I'll update you soon.

[behnazh-w]

@jensdietrich @wtwhite I got this reply:

It should not produce different results when all properties (machine, os, jdk version, input, files environment, ..., system time(?)) are the same. But, I learned to never say never. Which Maven artifacts are you building?

[wtwhite]

Thanks @behnazh-w, that's very helpful.

Which Maven artifacts are you building?

I haven't yet been able to demonstrate "strong" (run-to-run) compiler nondeterminism even when trying examples marked as such by others (1, 2), so I don't have any Maven artifacts to suggest -- rather, we are looking for such examples.

[behnazh-w]

Another confirmation for javac producing determinitic output:

generally we strive to keep javac's output deterministic on a given platform for the same set of sources. This is not to say that the output cannot change if the JDK is upgraded, etc. For example, javac internally will use collections akin to LinkedHashSet as opposed to HashSet so that the in-practice iteration order over not-fundamentally-ordered set will be the same.

[jensdietrich]

This makes sense and is very much what we expected, I came across an issue illustrating this approach, link must be somewhere in this thread.

…

On Thu, 16 Nov 2023 at 4:15 AM, Behnaz Hassanshahi ***@***.***> wrote: Another confirmation for javac producing determinitic output: generally we strive to keep javac's output deterministic on a given platform for the same set of sources. This is not to say that the output cannot change if the JDK is upgraded, etc. For example, javac internally will use collections akin to LinkedHashSet as opposed to HashSet so that the in-practice iteration order over not-fundamentally-ordered set will be the same. — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASSV5WOOG4VQA6ZPU547T3YEVZL5AVCNFSM6AAAAAA7IS46F2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJTGY3TOOJRGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[behnazh-w]

Hm, couldn't find the link in the thread.

[behnazh-w]

Also, this script from junit might be useful for checking reproducibility? https://github.com/junit-team/junit5/blob/main/gradle/scripts/checkBuildReproducibility.sh

And take a look at the conversation in this thread: junit-team/junit-framework#2217

[wtwhite]

Hm, couldn't find the link in the thread.

@behnazh-w I think this is the link (there are a few issues on this overall topic): #30 (comment).

[behnazh-w]

Also, looks like there have been several fixes to the JDK and the JDK build in the name of “reproducible builds”. However, many of these are specific to the JDK or the jlink tool for creating run-time images.

For Java projects, the Java team suggest to also look at the jar tool or docs builds with javadoc (but I assume docs is disabled in your build scripts?). Here are some issues that may be relevant:

• https://bugs.openjdk.org/browse/JDK-8272984
• https://bugs.openjdk.org/browse/JDK-8276766
• https://bugs.openjdk.org/browse/JDK-8286712
• https://bugs.openjdk.org/browse/JDK-8231640

[wtwhite]

Also, this script from junit might be useful for checking reproducibility? https://github.com/junit-team/junit5/blob/main/gradle/scripts/checkBuildReproducibility.sh

And take a look at the conversation in this thread: junit-team/junit5#2217

@behnazh-w Thanks for that, however that PR and discussion looks to be purely about differences in timestamps and metadata included in built jars, while for this paper we have restricted our scope to only the content of .class files.

For Java projects, the Java team suggest to also look at the jar tool or docs builds with javadoc (but I assume docs is disabled in your build scripts?). Here are some issues that may be relevant:

• https://bugs.openjdk.org/browse/JDK-8272984
• https://bugs.openjdk.org/browse/JDK-8276766
• https://bugs.openjdk.org/browse/JDK-8286712

Good to know, however these 3 issues also relate to content outside of the resulting .class files (not 100% sure about jmods in https://bugs.openjdk.org/browse/JDK-8276766, but I think so), so we can't take advantage of them for EQ1.

• https://bugs.openjdk.org/browse/JDK-8231640

Writing out properties in nondeterministic order could conceivably result in nondeterminism under older JDKs for projects that generate code, though I don't know how common that would be. I haven't seen nondeterminism from my attempts to build checkstyle, which contains ANTLR-generated source code.

I could search for open source projects (like checkstyle) that use known code-generating dependencies like ANTLR based on Maven dependencies, but I don't know a good way to refine that search to include only projects that read property files and use that for code generation -- so the only strategy I can think of would be to try to build large numbers of open source projects, a large number of times each (to detect nondeterminism). Feels like a long shot to me -- WDYT?