Check for method order nondeterminism on `kubernetes-client-project-5.4.1`

[wtwhite]

This is the main TODO of #33 -- P10 from the "Towards Build Verifiability for Java-based Systems", Xiong et. al paper.

[wtwhite]

Building with rebuild.sh from https://github.com/jvm-repo-rebuild/reproducible-central mostly works, failing in an unimportant step after successful compilation, but many dependencies seem to be built locally and are detected as different from their Maven Central repo versions:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ time ./rebuild.sh content/io/fabric8/kubernetes-client/kubernetes-client-5.4.1.buildspec 
--snip--
[INFO] --- maven-artifact-plugin:3.5.0:compare (default-cli) @ kubernetes-openshift-uberjar ---
[INFO] Saved aggregate info on build to /var/maven/app/uberjar/target/kubernetes-openshift-uberjar-5.4.1.buildinfo
[INFO] Aggregate buildinfo copied to /var/maven/app/target/kubernetes-client-project-5.4.1.buildinfo
[INFO] Checking against reference build from central...
Downloading from central: https://repo.maven.apache.org/maven2/io/fabric8/kubernetes-openshift-uberjar/5.4.1/kubernetes-openshift-uberjar-5.4.1.buildinfo
[INFO] Reference buildinfo file not found: it will be generated from downloaded reference artifacts
Downloading from central: https://repo.maven.apache.org/maven2/io/fabric8/kubernetes-openshift-uberjar/5.4.1/kubernetes-openshift-uberjar-5.4.1.pom
Downloaded from central: https://repo.maven.apache.org/maven2/io/fabric8/kubernetes-openshift-uberjar/5.4.1/kubernetes-openshift-uberjar-5.4.1.pom (16 kB at 34 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/io/fabric8/kubernetes-openshift-uberjar/5.4.1/kubernetes-openshift-uberjar-5.4.1.jar
Downloaded from central: https://repo.maven.apache.org/maven2/io/fabric8/kubernetes-openshift-uberjar/5.4.1/kubernetes-openshift-uberjar-5.4.1.jar (17 MB at 6.8 MB/s)
Downloading from central: https://repo.maven.apache.org/maven2/io/fabric8/kubernetes-openshift-uberjar/5.4.1/kubernetes-openshift-uberjar-5.4.1-versioned.jar
Downloaded from central: https://repo.maven.apache.org/maven2/io/fabric8/kubernetes-openshift-uberjar/5.4.1/kubernetes-openshift-uberjar-5.4.1-versioned.jar (18 MB at 8.9 MB/s)
Downloading from central: https://repo.maven.apache.org/maven2/io/fabric8/kubernetes-openshift-uberjar/5.4.1/kubernetes-openshift-uberjar-5.4.1-sources.jar
Downloaded from central: https://repo.maven.apache.org/maven2/io/fabric8/kubernetes-openshift-uberjar/5.4.1/kubernetes-openshift-uberjar-5.4.1-sources.jar (3.8 kB at 5.8 kB/s)
[INFO] Reference build java.version: 1.8 (from MANIFEST.MF Build-Jdk-Spec)
[INFO] Reference build os.name: Unix (from pom.properties newline)
[INFO] Minimal buildinfo generated from downloaded artifacts: /var/maven/app/target/reference/kubernetes-openshift-uberjar-5.4.1.buildinfo
[ERROR] size mismatch kubernetes-model-common-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-common-5.4.1.jar kubernetes-model-generator/kubernetes-model-common/target/kubernetes-model-common-5.4.1.jar
[ERROR] size mismatch model-annotator-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/model-annotator-5.4.1.jar model-annotator/target/model-annotator-5.4.1.jar
[ERROR] size mismatch kubernetes-model-core-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-core-5.4.1.jar kubernetes-model-generator/kubernetes-model-core/target/kubernetes-model-core-5.4.1.jar
[ERROR] size mismatch kubernetes-model-core-5.4.1-sources.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-core-5.4.1-sources.jar kubernetes-model-generator/kubernetes-model-core/target/kubernetes-model-core-5.4.1-sources.jar
[ERROR] size mismatch kubernetes-model-rbac-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-rbac-5.4.1.jar kubernetes-model-generator/kubernetes-model-rbac/target/kubernetes-model-rbac-5.4.1.jar
[ERROR] size mismatch kubernetes-model-admissionregistration-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-admissionregistration-5.4.1.jar kubernetes-model-generator/kubernetes-model-admissionregistration/target/kubernetes-model-admissionregistration-5.4.1.jar
[ERROR] size mismatch kubernetes-model-apps-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-apps-5.4.1.jar kubernetes-model-generator/kubernetes-model-apps/target/kubernetes-model-apps-5.4.1.jar
[ERROR] size mismatch kubernetes-model-autoscaling-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-autoscaling-5.4.1.jar kubernetes-model-generator/kubernetes-model-autoscaling/target/kubernetes-model-autoscaling-5.4.1.jar
[ERROR] size mismatch kubernetes-model-apiextensions-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-apiextensions-5.4.1.jar kubernetes-model-generator/kubernetes-model-apiextensions/target/kubernetes-model-apiextensions-5.4.1.jar
[ERROR] size mismatch kubernetes-model-batch-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-batch-5.4.1.jar kubernetes-model-generator/kubernetes-model-batch/target/kubernetes-model-batch-5.4.1.jar
[ERROR] size mismatch kubernetes-model-certificates-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-certificates-5.4.1.jar kubernetes-model-generator/kubernetes-model-certificates/target/kubernetes-model-certificates-5.4.1.jar
[ERROR] size mismatch kubernetes-model-coordination-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-coordination-5.4.1.jar kubernetes-model-generator/kubernetes-model-coordination/target/kubernetes-model-coordination-5.4.1.jar
[ERROR] size mismatch kubernetes-model-discovery-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-discovery-5.4.1.jar kubernetes-model-generator/kubernetes-model-discovery/target/kubernetes-model-discovery-5.4.1.jar
[ERROR] size mismatch kubernetes-model-events-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-events-5.4.1.jar kubernetes-model-generator/kubernetes-model-events/target/kubernetes-model-events-5.4.1.jar
[ERROR] size mismatch kubernetes-model-extensions-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-extensions-5.4.1.jar kubernetes-model-generator/kubernetes-model-extensions/target/kubernetes-model-extensions-5.4.1.jar
[ERROR] size mismatch kubernetes-model-networking-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-networking-5.4.1.jar kubernetes-model-generator/kubernetes-model-networking/target/kubernetes-model-networking-5.4.1.jar
[ERROR] size mismatch kubernetes-model-metrics-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-metrics-5.4.1.jar kubernetes-model-generator/kubernetes-model-metrics/target/kubernetes-model-metrics-5.4.1.jar
[ERROR] size mismatch kubernetes-model-policy-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-policy-5.4.1.jar kubernetes-model-generator/kubernetes-model-policy/target/kubernetes-model-policy-5.4.1.jar
[ERROR] size mismatch kubernetes-model-scheduling-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-scheduling-5.4.1.jar kubernetes-model-generator/kubernetes-model-scheduling/target/kubernetes-model-scheduling-5.4.1.jar
[ERROR] size mismatch kubernetes-model-storageclass-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-storageclass-5.4.1.jar kubernetes-model-generator/kubernetes-model-storageclass/target/kubernetes-model-storageclass-5.4.1.jar
[ERROR] size mismatch openshift-model-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/openshift-model-5.4.1.jar kubernetes-model-generator/openshift-model/target/openshift-model-5.4.1.jar
[ERROR] size mismatch openshift-model-5.4.1-sources.jar: investigate with diffoscope target/reference/io.fabric8/openshift-model-5.4.1-sources.jar kubernetes-model-generator/openshift-model/target/openshift-model-5.4.1-sources.jar
[ERROR] size mismatch kubernetes-model-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-5.4.1.jar kubernetes-model-generator/kubernetes-model/target/kubernetes-model-5.4.1.jar
[ERROR] size mismatch kubernetes-model-jsonschema2pojo-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-jsonschema2pojo-5.4.1.jar kubernetes-model-generator/kubernetes-model-jsonschema2pojo/target/kubernetes-model-jsonschema2pojo-5.4.1.jar
[ERROR] size mismatch kubernetes-model-flowcontrol-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-flowcontrol-5.4.1.jar kubernetes-model-generator/kubernetes-model-flowcontrol/target/kubernetes-model-flowcontrol-5.4.1.jar
[ERROR] size mismatch kubernetes-model-node-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-model-node-5.4.1.jar kubernetes-model-generator/kubernetes-model-node/target/kubernetes-model-node-5.4.1.jar
[ERROR] size mismatch openshift-model-operator-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/openshift-model-operator-5.4.1.jar kubernetes-model-generator/openshift-model-operator/target/openshift-model-operator-5.4.1.jar
[ERROR] size mismatch openshift-model-operator-5.4.1-sources.jar: investigate with diffoscope target/reference/io.fabric8/openshift-model-operator-5.4.1-sources.jar kubernetes-model-generator/openshift-model-operator/target/openshift-model-operator-5.4.1-sources.jar
[ERROR] size mismatch openshift-model-operatorhub-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/openshift-model-operatorhub-5.4.1.jar kubernetes-model-generator/openshift-model-operatorhub/target/openshift-model-operatorhub-5.4.1.jar
[ERROR] size mismatch openshift-model-monitoring-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/openshift-model-monitoring-5.4.1.jar kubernetes-model-generator/openshift-model-monitoring/target/openshift-model-monitoring-5.4.1.jar
[ERROR] size mismatch openshift-model-console-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/openshift-model-console-5.4.1.jar kubernetes-model-generator/openshift-model-console/target/openshift-model-console-5.4.1.jar
[ERROR] size mismatch kubernetes-client-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-client-5.4.1.jar kubernetes-client/target/kubernetes-client-5.4.1.jar
[ERROR] size mismatch kubernetes-client-5.4.1-bundle.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-client-5.4.1-bundle.jar kubernetes-client/target/kubernetes-client-5.4.1-bundle.jar
[ERROR] size mismatch kubernetes-server-mock-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-server-mock-5.4.1.jar kubernetes-server-mock/target/kubernetes-server-mock-5.4.1.jar
[ERROR] size mismatch openshift-client-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/openshift-client-5.4.1.jar openshift-client/target/openshift-client-5.4.1.jar
[ERROR] size mismatch openshift-client-5.4.1-bundle.jar: investigate with diffoscope target/reference/io.fabric8/openshift-client-5.4.1-bundle.jar openshift-client/target/openshift-client-5.4.1-bundle.jar
[ERROR] size mismatch knative-model-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/knative-model-5.4.1.jar extensions/knative/model/target/knative-model-5.4.1.jar
[ERROR] size mismatch knative-client-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/knative-client-5.4.1.jar extensions/knative/client/target/knative-client-5.4.1.jar
[ERROR] sha512 mismatch knative-mock-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/knative-mock-5.4.1.jar extensions/knative/mock/target/knative-mock-5.4.1.jar
[ERROR] size mismatch tekton-model-v1alpha1-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/tekton-model-v1alpha1-5.4.1.jar extensions/tekton/model-v1alpha1/target/tekton-model-v1alpha1-5.4.1.jar
[ERROR] sha512 mismatch tekton-model-v1beta1-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/tekton-model-v1beta1-5.4.1.jar extensions/tekton/model-v1beta1/target/tekton-model-v1beta1-5.4.1.jar
[ERROR] size mismatch tekton-model-triggers-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/tekton-model-triggers-5.4.1.jar extensions/tekton/model-triggers/target/tekton-model-triggers-5.4.1.jar
[ERROR] size mismatch tekton-client-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/tekton-client-5.4.1.jar extensions/tekton/client/target/tekton-client-5.4.1.jar
[ERROR] size mismatch tekton-mock-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/tekton-mock-5.4.1.jar extensions/tekton/mock/target/tekton-mock-5.4.1.jar
[ERROR] size mismatch servicecatalog-model-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/servicecatalog-model-5.4.1.jar extensions/service-catalog/model/target/servicecatalog-model-5.4.1.jar
[ERROR] size mismatch servicecatalog-client-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/servicecatalog-client-5.4.1.jar extensions/service-catalog/client/target/servicecatalog-client-5.4.1.jar
[ERROR] size mismatch servicecatalog-server-mock-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/servicecatalog-server-mock-5.4.1.jar extensions/service-catalog/mock/target/servicecatalog-server-mock-5.4.1.jar
[ERROR] size mismatch service-catalog-examples-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/service-catalog-examples-5.4.1.jar extensions/service-catalog/examples/target/service-catalog-examples-5.4.1.jar
[ERROR] size mismatch volumesnapshot-model-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/volumesnapshot-model-5.4.1.jar extensions/volumesnapshot/model/target/volumesnapshot-model-5.4.1.jar
[ERROR] size mismatch volumesnapshot-client-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/volumesnapshot-client-5.4.1.jar extensions/volumesnapshot/client/target/volumesnapshot-client-5.4.1.jar
[ERROR] size mismatch volumesnapshot-server-mock-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/volumesnapshot-server-mock-5.4.1.jar extensions/volumesnapshot/mock/target/volumesnapshot-server-mock-5.4.1.jar
[ERROR] sha512 mismatch volumesnapshot-examples-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/volumesnapshot-examples-5.4.1.jar extensions/volumesnapshot/examples/target/volumesnapshot-examples-5.4.1.jar
[ERROR] size mismatch chaosmesh-model-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/chaosmesh-model-5.4.1.jar extensions/chaosmesh/model/target/chaosmesh-model-5.4.1.jar
[ERROR] size mismatch chaosmesh-client-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/chaosmesh-client-5.4.1.jar extensions/chaosmesh/client/target/chaosmesh-client-5.4.1.jar
[ERROR] size mismatch chaosmesh-server-mock-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/chaosmesh-server-mock-5.4.1.jar extensions/chaosmesh/mock/target/chaosmesh-server-mock-5.4.1.jar
[ERROR] sha512 mismatch camel-k-model-v1-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/camel-k-model-v1-5.4.1.jar extensions/camel-k/model-v1/target/camel-k-model-v1-5.4.1.jar
[ERROR] size mismatch camel-k-model-v1alpha1-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/camel-k-model-v1alpha1-5.4.1.jar extensions/camel-k/model-v1alpha1/target/camel-k-model-v1alpha1-5.4.1.jar
[ERROR] size mismatch camel-k-client-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/camel-k-client-5.4.1.jar extensions/camel-k/client/target/camel-k-client-5.4.1.jar
[ERROR] size mismatch camel-k-mock-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/camel-k-mock-5.4.1.jar extensions/camel-k/mock/target/camel-k-mock-5.4.1.jar
[ERROR] size mismatch openshift-server-mock-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/openshift-server-mock-5.4.1.jar openshift-server-mock/target/openshift-server-mock-5.4.1.jar
[ERROR] size mismatch kubernetes-examples-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-examples-5.4.1.jar kubernetes-examples/target/kubernetes-examples-5.4.1.jar
[ERROR] size mismatch kubernetes-karaf-itests-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8.kubernetes/kubernetes-karaf-itests-5.4.1.jar platforms/karaf/itests/target/kubernetes-karaf-itests-5.4.1.jar
[ERROR] size mismatch crd-generator-api-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/crd-generator-api-5.4.1.jar crd-generator/api/target/crd-generator-api-5.4.1.jar
[ERROR] size mismatch crd-generator-apt-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/crd-generator-apt-5.4.1.jar crd-generator/apt/target/crd-generator-apt-5.4.1.jar
[ERROR] size mismatch kubernetes-openshift-uberjar-5.4.1.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-openshift-uberjar-5.4.1.jar uberjar/target/kubernetes-openshift-uberjar-5.4.1.jar
[ERROR] size mismatch kubernetes-openshift-uberjar-5.4.1-versioned.jar: investigate with diffoscope target/reference/io.fabric8/kubernetes-openshift-uberjar-5.4.1-versioned.jar uberjar/target/kubernetes-openshift-uberjar-5.4.1-versioned.jar
[ERROR] Reproducible Build output summary: 186 files ok, 66 different
[ERROR] see diff target/reference/kubernetes-openshift-uberjar-5.4.1.buildinfo uberjar/target/kubernetes-openshift-uberjar-5.4.1.buildinfo
[ERROR] see also https://maven.apache.org/guides/mini/guide-reproducible-builds.html
[INFO] Reproducible Build output comparison saved to /var/maven/app/uberjar/target/kubernetes-openshift-uberjar-5.4.1.buildcompare
[INFO] Aggregate buildcompare copied to /var/maven/app/target/kubernetes-client-project-5.4.1.buildcompare
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Fabric8 :: Kubernetes :: Project 5.4.1:
[INFO] 
[INFO] Fabric8 :: Kubernetes :: Project ................... SUCCESS [01:47 min]
[INFO] Fabric8 :: Kubernetes Model Parent ................. SUCCESS [ 24.812 s]
[INFO] Fabric8 :: Kubernetes Model :: Common .............. SUCCESS [  4.568 s]
[INFO] Fabric8 :: Kubernetes :: Model Annotator ........... SUCCESS [  2.752 s]
[INFO] Fabric8 :: Kubernetes Model :: Core ................ SUCCESS [ 19.325 s]
[INFO] Fabric8 :: Kubernetes Model :: RBAC ................ SUCCESS [  4.838 s]
[INFO] Fabric8 :: Kubernetes Model :: Admission Registration, Authentication and Authorization SUCCESS [  7.984 s]
[INFO] Fabric8 :: Kubernetes Model :: Apps ................ SUCCESS [  6.687 s]
[INFO] Fabric8 :: Kubernetes Model :: Autoscaling ......... SUCCESS [  8.396 s]
[INFO] Fabric8 :: Kubernetes Model :: API Extensions ...... SUCCESS [ 10.739 s]
[INFO] Fabric8 :: Kubernetes Model :: Batch ............... SUCCESS [  5.933 s]
[INFO] Fabric8 :: Kubernetes Model :: Certificates ........ SUCCESS [  5.937 s]
[INFO] Fabric8 :: Kubernetes Model :: Coordination ........ SUCCESS [  4.700 s]
[INFO] Fabric8 :: Kubernetes Model :: Discovery ........... SUCCESS [  6.053 s]
[INFO] Fabric8 :: Kubernetes Model :: Events .............. SUCCESS [  4.971 s]
[INFO] Fabric8 :: Kubernetes Model :: Extensions .......... SUCCESS [  7.907 s]
[INFO] Fabric8 :: Kubernetes Model :: Networking .......... SUCCESS [  7.368 s]
[INFO] Fabric8 :: Kubernetes Model :: Metrics ............. SUCCESS [  4.254 s]
[INFO] Fabric8 :: Kubernetes Model :: Policy .............. SUCCESS [  7.248 s]
[INFO] Fabric8 :: Kubernetes Model :: Scheduling .......... SUCCESS [  4.525 s]
[INFO] Fabric8 :: Kubernetes Model :: Storage Class ....... SUCCESS [  6.212 s]
[INFO] Fabric8 :: OpenShift Model ......................... SUCCESS [ 29.322 s]
[INFO] Fabric8 :: Kubernetes Model ........................ SUCCESS [  1.104 s]
[INFO] Fabric8 :: Kubernetes Model Schema2Pojo Customizer . SUCCESS [  2.551 s]
[INFO] Fabric8 :: Kubernetes Model :: FlowControl ......... SUCCESS [  5.342 s]
[INFO] Fabric8 :: Kubernetes Model :: Node ................ SUCCESS [  5.045 s]
[INFO] Fabric8 :: OpenShift Operator Model ................ SUCCESS [ 17.813 s]
[INFO] Fabric8 :: OpenShift OperatorHub Model ............. SUCCESS [ 10.815 s]
[INFO] Fabric8 :: OpenShift Monitoring Model .............. SUCCESS [  8.553 s]
[INFO] Fabric8 :: OpenShift Console Model ................. SUCCESS [  5.955 s]
[INFO] Fabric8 :: Kubernetes :: Java Client ............... SUCCESS [ 17.407 s]
[INFO] Fabric8 :: Kubernetes :: Server Mock ............... SUCCESS [  1.852 s]
[INFO] Fabric8 :: Openshift :: Java Client ................ SUCCESS [  7.331 s]
[INFO] Fabric8 :: Kubernetes :: Extensions ................ SUCCESS [  0.698 s]
[INFO] Fabric8 :: Knative :: Extension .................... SUCCESS [  0.751 s]
[INFO] Fabric8 :: Knative :: Model ........................ SUCCESS [ 18.825 s]
[INFO] Fabric8 :: Knative :: Client ....................... SUCCESS [  2.890 s]
[INFO] Fabric8 :: Knative :: Server Mock .................. SUCCESS [  2.223 s]
[INFO] Fabric8 :: Knative :: Examples ..................... SUCCESS [  2.032 s]
[INFO] Fabric8 :: Knative :: Tests ........................ SUCCESS [  1.361 s]
[INFO] Fabric8 :: Tekton :: Extension ..................... SUCCESS [  0.749 s]
[INFO] Fabric8 :: Tekton :: Model v1alpha1 ................ SUCCESS [  8.293 s]
[INFO] Fabric8 :: Tekton :: Model v1beta1 ................. SUCCESS [  9.219 s]
[INFO] Fabric8 :: Tekton :: Model triggers ................ SUCCESS [  5.743 s]
[INFO] Fabric8 :: Tekton :: Client ........................ SUCCESS [  2.584 s]
[INFO] Fabric8 :: Tekton :: Server Mock ................... SUCCESS [  2.016 s]
[INFO] Fabric8 :: Tekton :: Examples ...................... SUCCESS [  1.887 s]
[INFO] Fabric8 :: Tekton :: Tests ......................... SUCCESS [  1.438 s]
[INFO] Fabric8 :: Service Catalog :: Extension ............ SUCCESS [  0.803 s]
[INFO] Fabric8 :: Service Catalog :: Model ................ SUCCESS [  8.172 s]
[INFO] Fabric8 :: Service Catalog :: Client ............... SUCCESS [  2.286 s]
[INFO] Fabric8 :: Service Catalog :: Server Mock .......... SUCCESS [  1.820 s]
[INFO] Fabric8 :: Service Catalog :: Examples ............. SUCCESS [  1.952 s]
[INFO] Fabric8 :: Service Catalog :: Tests ................ SUCCESS [  1.393 s]
[INFO] Fabric8 :: Volume Snapshot :: Extension ............ SUCCESS [  0.806 s]
[INFO] Fabric8 :: Volume Snapshot :: Model ................ SUCCESS [  4.326 s]
[INFO] Fabric8 :: Volume Snapshot :: Client ............... SUCCESS [  2.092 s]
[INFO] Fabric8 :: Volume Snapshot :: Server Mock .......... SUCCESS [  2.245 s]
[INFO] Fabric8 :: Volume Snapshot :: Examples ............. SUCCESS [  1.909 s]
[INFO] Fabric8 :: Volume Snapshot :: Tests ................ SUCCESS [  1.281 s]
[INFO] Fabric8 :: Chaos Mesh :: Extension ................. SUCCESS [  0.823 s]
[INFO] Fabric8 :: Chaos Mesh :: Model ..................... SUCCESS [  9.716 s]
[INFO] Fabric8 :: Chaos Mesh :: Client .................... SUCCESS [  2.099 s]
[INFO] Fabric8 :: Chaos Mesh :: Server Mock ............... SUCCESS [  1.785 s]
[INFO] Fabric8 :: Chaos Mesh :: Examples .................. SUCCESS [  1.937 s]
[INFO] Fabric8 :: Chaos Mesh :: Tests ..................... SUCCESS [  1.375 s]
[INFO] Fabric8 :: Camel-K :: Extension .................... SUCCESS [  0.684 s]
[INFO] Fabric8 :: Camel K :: Model v1 ..................... SUCCESS [  6.183 s]
[INFO] Fabric8 :: Camel-K :: Model v1alpha1 ............... SUCCESS [  4.332 s]
[INFO] Fabric8 :: Camel-K :: Client ....................... SUCCESS [  2.334 s]
[INFO] Fabric8 :: Camel-K :: Server Mock .................. SUCCESS [  1.832 s]
[INFO] Fabric8 :: Camel-K :: Tests ........................ SUCCESS [  1.328 s]
[INFO] Fabric8 :: Openshift :: Server Mock ................ SUCCESS [  1.882 s]
[INFO] Fabric8 :: Kubernetes :: Client Examples ........... SUCCESS [  3.169 s]
[INFO] Fabric8 :: Kubernetes :: Platforms ................. SUCCESS [  0.681 s]
[INFO] Fabric8 :: Kubernetes :: Platforms :: Karaf ........ SUCCESS [  0.755 s]
[INFO] Fabric8 :: Kubernetes :: Platforms :: Karaf :: Features SUCCESS [  2.490 s]
[INFO] Fabric8 :: Kubernetes :: Platforms :: Karaf :: Tests SUCCESS [ 47.436 s]
[INFO] Fabric8 :: CRD generator :: Parent ................. SUCCESS [  0.701 s]
[INFO] Fabric8 :: CRD generator :: API .................... SUCCESS [  1.900 s]
[INFO] Fabric8 :: CRD generator :: Annotation Processor ... SUCCESS [  3.453 s]
[INFO] Fabric8 :: Kubernetes :: Tests ..................... SUCCESS [  3.142 s]
[INFO] Fabric8 :: Kubernetes and Openshift :: UberJar ..... SUCCESS [ 14.005 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  10:31 min
[INFO] Finished at: 2023-11-13T20:45:06Z
[INFO] ------------------------------------------------------------------------
/home/wtwhite/code/reproducible-central/bin/includes/rebuildToolMvn.sh: line 71: dos2unix: command not found

[FAIL] /========================================================================
[FAIL] |  failed to convert buildinfo newlines 
[FAIL] \========================================================================
[FAIL]  ---------->>> PROCESS WAS ABORTED <<<---------- 


real	12m44.046s
user	0m19.170s
sys	0m8.533s

Run 2 gives different results -- 189 files OK instead of 186:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ time ./rebuild.sh content/io/fabric8/kubernetes-client/kubernetes-client-5.4.1.buildspec 
--snip--
[ERROR] Reproducible Build output summary: 189 files ok, 63 different
[ERROR] see diff target/reference/kubernetes-openshift-uberjar-5.4.1.buildinfo uberjar/target/kubernetes-openshift-uberjar-5.4.1.buildinfo
[ERROR] see also https://maven.apache.org/guides/mini/guide-reproducible-builds.html
[INFO] Reproducible Build output comparison saved to /var/maven/app/uberjar/target/kubernetes-openshift-uberjar-5.4.1.buildcompare
[INFO] Aggregate buildcompare copied to /var/maven/app/target/kubernetes-client-project-5.4.1.buildcompare
--snip--
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:56 min
[INFO] Finished at: 2023-11-13T21:35:09Z
[INFO] ------------------------------------------------------------------------
/home/wtwhite/code/reproducible-central/bin/includes/rebuildToolMvn.sh: line 71: dos2unix: command not found

[FAIL] /========================================================================
[FAIL] |  failed to convert buildinfo newlines 
[FAIL] \========================================================================
[FAIL]  ---------->>> PROCESS WAS ABORTED <<<---------- 


real	3m58.837s
user	0m0.403s
sys	0m0.907s

Genuine build-to-build variability, or just an artifact of caching? Run 3 suggests the latter:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ time ./rebuild.sh content/io/fabric8/kubernetes-client/kubernetes-client-5.4.1.buildspec 
--snip--
[ERROR] Reproducible Build output summary: 189 files ok, 63 different
--snip--
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:02 min
[INFO] Finished at: 2023-11-13T21:42:47Z
[INFO] ------------------------------------------------------------------------
/home/wtwhite/code/reproducible-central/bin/includes/rebuildToolMvn.sh: line 71: dos2unix: command not found

[FAIL] /========================================================================
[FAIL] |  failed to convert buildinfo newlines 
[FAIL] \========================================================================
[FAIL]  ---------->>> PROCESS WAS ABORTED <<<---------- 


real	4m4.509s
user	0m0.361s
sys	0m0.880s

[wtwhite]

Building in a freshly checked out source tree, in first see 186 OK files (the same as in the initial run):

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central-FRESH$ time ./rebuild.sh content/io/fabric8/kubernetes-client/kubernetes-client-5.4.1.buildspec
--snip--
[ERROR] Reproducible Build output summary: 186 files ok, 66 different
[ERROR] see diff target/reference/kubernetes-openshift-uberjar-5.4.1.buildinfo uberjar/target/kubernetes-openshift-uberjar-5.4.1.buildinfo
[ERROR] see also https://maven.apache.org/guides/mini/guide-reproducible-builds.html
[INFO] Reproducible Build output comparison saved to /var/maven/app/uberjar/target/kubernetes-openshift-uberjar-5.4.1.buildcompare
[INFO] Aggregate buildcompare copied to /var/maven/app/target/kubernetes-client-project-5.4.1.buildcompare
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Fabric8 :: Kubernetes :: Project 5.4.1:
[INFO] 
[INFO] Fabric8 :: Kubernetes :: Project ................... SUCCESS [ 15.301 s]
[INFO] Fabric8 :: Kubernetes Model Parent ................. SUCCESS [  1.467 s]
[INFO] Fabric8 :: Kubernetes Model :: Common .............. SUCCESS [  2.631 s]
[INFO] Fabric8 :: Kubernetes :: Model Annotator ........... SUCCESS [  0.932 s]
[INFO] Fabric8 :: Kubernetes Model :: Core ................ SUCCESS [ 21.042 s]
[INFO] Fabric8 :: Kubernetes Model :: RBAC ................ SUCCESS [  3.727 s]
[INFO] Fabric8 :: Kubernetes Model :: Admission Registration, Authentication and Authorization SUCCESS [  9.772 s]
[INFO] Fabric8 :: Kubernetes Model :: Apps ................ SUCCESS [  8.244 s]
[INFO] Fabric8 :: Kubernetes Model :: Autoscaling ......... SUCCESS [  7.860 s]
[INFO] Fabric8 :: Kubernetes Model :: API Extensions ...... SUCCESS [  8.414 s]
[INFO] Fabric8 :: Kubernetes Model :: Batch ............... SUCCESS [  4.687 s]
[INFO] Fabric8 :: Kubernetes Model :: Certificates ........ SUCCESS [  4.751 s]
[INFO] Fabric8 :: Kubernetes Model :: Coordination ........ SUCCESS [  3.322 s]
[INFO] Fabric8 :: Kubernetes Model :: Discovery ........... SUCCESS [  4.707 s]
[INFO] Fabric8 :: Kubernetes Model :: Events .............. SUCCESS [  3.922 s]
[INFO] Fabric8 :: Kubernetes Model :: Extensions .......... SUCCESS [  5.801 s]
[INFO] Fabric8 :: Kubernetes Model :: Networking .......... SUCCESS [  5.638 s]
[INFO] Fabric8 :: Kubernetes Model :: Metrics ............. SUCCESS [  3.727 s]
[INFO] Fabric8 :: Kubernetes Model :: Policy .............. SUCCESS [  5.224 s]
[INFO] Fabric8 :: Kubernetes Model :: Scheduling .......... SUCCESS [  3.939 s]
[INFO] Fabric8 :: Kubernetes Model :: Storage Class ....... SUCCESS [  5.571 s]
[INFO] Fabric8 :: OpenShift Model ......................... SUCCESS [ 21.481 s]
[INFO] Fabric8 :: Kubernetes Model ........................ SUCCESS [  1.185 s]
[INFO] Fabric8 :: Kubernetes Model Schema2Pojo Customizer . SUCCESS [  1.935 s]
[INFO] Fabric8 :: Kubernetes Model :: FlowControl ......... SUCCESS [  3.947 s]
[INFO] Fabric8 :: Kubernetes Model :: Node ................ SUCCESS [  3.747 s]
[INFO] Fabric8 :: OpenShift Operator Model ................ SUCCESS [ 11.418 s]
[INFO] Fabric8 :: OpenShift OperatorHub Model ............. SUCCESS [  6.391 s]
[INFO] Fabric8 :: OpenShift Monitoring Model .............. SUCCESS [  6.005 s]
[INFO] Fabric8 :: OpenShift Console Model ................. SUCCESS [  3.317 s]
[INFO] Fabric8 :: Kubernetes :: Java Client ............... SUCCESS [ 11.617 s]
[INFO] Fabric8 :: Kubernetes :: Server Mock ............... SUCCESS [  0.599 s]
[INFO] Fabric8 :: Openshift :: Java Client ................ SUCCESS [  4.526 s]
[INFO] Fabric8 :: Kubernetes :: Extensions ................ SUCCESS [  0.274 s]
[INFO] Fabric8 :: Knative :: Extension .................... SUCCESS [  0.324 s]
[INFO] Fabric8 :: Knative :: Model ........................ SUCCESS [ 14.760 s]
[INFO] Fabric8 :: Knative :: Client ....................... SUCCESS [  1.969 s]
[INFO] Fabric8 :: Knative :: Server Mock .................. SUCCESS [  0.427 s]
[INFO] Fabric8 :: Knative :: Examples ..................... SUCCESS [  0.370 s]
[INFO] Fabric8 :: Knative :: Tests ........................ SUCCESS [  0.514 s]
[INFO] Fabric8 :: Tekton :: Extension ..................... SUCCESS [  0.324 s]
[INFO] Fabric8 :: Tekton :: Model v1alpha1 ................ SUCCESS [  6.523 s]
[INFO] Fabric8 :: Tekton :: Model v1beta1 ................. SUCCESS [  7.456 s]
[INFO] Fabric8 :: Tekton :: Model triggers ................ SUCCESS [  5.438 s]
[INFO] Fabric8 :: Tekton :: Client ........................ SUCCESS [  1.828 s]
[INFO] Fabric8 :: Tekton :: Server Mock ................... SUCCESS [  0.465 s]
[INFO] Fabric8 :: Tekton :: Examples ...................... SUCCESS [  0.524 s]
[INFO] Fabric8 :: Tekton :: Tests ......................... SUCCESS [  0.440 s]
[INFO] Fabric8 :: Service Catalog :: Extension ............ SUCCESS [  0.308 s]
[INFO] Fabric8 :: Service Catalog :: Model ................ SUCCESS [  7.119 s]
[INFO] Fabric8 :: Service Catalog :: Client ............... SUCCESS [  1.508 s]
[INFO] Fabric8 :: Service Catalog :: Server Mock .......... SUCCESS [  0.428 s]
[INFO] Fabric8 :: Service Catalog :: Examples ............. SUCCESS [  0.415 s]
[INFO] Fabric8 :: Service Catalog :: Tests ................ SUCCESS [  1.276 s]
[INFO] Fabric8 :: Volume Snapshot :: Extension ............ SUCCESS [  0.683 s]
[INFO] Fabric8 :: Volume Snapshot :: Model ................ SUCCESS [  4.262 s]
[INFO] Fabric8 :: Volume Snapshot :: Client ............... SUCCESS [  1.786 s]
[INFO] Fabric8 :: Volume Snapshot :: Server Mock .......... SUCCESS [  1.667 s]
[INFO] Fabric8 :: Volume Snapshot :: Examples ............. SUCCESS [  1.607 s]
[INFO] Fabric8 :: Volume Snapshot :: Tests ................ SUCCESS [  1.396 s]
[INFO] Fabric8 :: Chaos Mesh :: Extension ................. SUCCESS [  0.708 s]
[INFO] Fabric8 :: Chaos Mesh :: Model ..................... SUCCESS [  8.953 s]
[INFO] Fabric8 :: Chaos Mesh :: Client .................... SUCCESS [  2.042 s]
[INFO] Fabric8 :: Chaos Mesh :: Server Mock ............... SUCCESS [  1.629 s]
[INFO] Fabric8 :: Chaos Mesh :: Examples .................. SUCCESS [  1.571 s]
[INFO] Fabric8 :: Chaos Mesh :: Tests ..................... SUCCESS [  1.248 s]
[INFO] Fabric8 :: Camel-K :: Extension .................... SUCCESS [  0.723 s]
[INFO] Fabric8 :: Camel K :: Model v1 ..................... SUCCESS [  5.915 s]
[INFO] Fabric8 :: Camel-K :: Model v1alpha1 ............... SUCCESS [  3.870 s]
[INFO] Fabric8 :: Camel-K :: Client ....................... SUCCESS [  2.205 s]
[INFO] Fabric8 :: Camel-K :: Server Mock .................. SUCCESS [  1.614 s]
[INFO] Fabric8 :: Camel-K :: Tests ........................ SUCCESS [  1.233 s]
[INFO] Fabric8 :: Openshift :: Server Mock ................ SUCCESS [  1.653 s]
[INFO] Fabric8 :: Kubernetes :: Client Examples ........... SUCCESS [  2.485 s]
[INFO] Fabric8 :: Kubernetes :: Platforms ................. SUCCESS [  0.686 s]
[INFO] Fabric8 :: Kubernetes :: Platforms :: Karaf ........ SUCCESS [  0.692 s]
[INFO] Fabric8 :: Kubernetes :: Platforms :: Karaf :: Features SUCCESS [  2.049 s]
[INFO] Fabric8 :: Kubernetes :: Platforms :: Karaf :: Tests SUCCESS [ 27.773 s]
[INFO] Fabric8 :: CRD generator :: Parent ................. SUCCESS [  0.655 s]
[INFO] Fabric8 :: CRD generator :: API .................... SUCCESS [  1.771 s]
[INFO] Fabric8 :: CRD generator :: Annotation Processor ... SUCCESS [  2.604 s]
[INFO] Fabric8 :: Kubernetes :: Tests ..................... SUCCESS [  1.983 s]
[INFO] Fabric8 :: Kubernetes and Openshift :: UberJar ..... SUCCESS [  8.576 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  05:57 min
[INFO] Finished at: 2023-11-13T21:59:26Z
[INFO] ------------------------------------------------------------------------
/home/wtwhite/code/reproducible-central-FRESH/bin/includes/rebuildToolMvn.sh: line 71: dos2unix: command not found

[FAIL] /========================================================================
[FAIL] |  failed to convert buildinfo newlines 
[FAIL] \========================================================================
[FAIL]  ---------->>> PROCESS WAS ABORTED <<<---------- 


real	6m47.552s
user	0m18.313s
sys	0m7.554s

Run 2 gives 189, same as run 2 from the original source tree:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central-FRESH$ time ./rebuild.sh content/io/fabric8/kubernetes-client/kubernetes-client-5.4.1.buildspec
--snip--
[ERROR] Reproducible Build output summary: 189 files ok, 63 different
--snip--
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:00 min
[INFO] Finished at: 2023-11-13T22:05:21Z
[INFO] ------------------------------------------------------------------------
/home/wtwhite/code/reproducible-central-FRESH/bin/includes/rebuildToolMvn.sh: line 71: dos2unix: command not found

[FAIL] /========================================================================
[FAIL] |  failed to convert buildinfo newlines 
[FAIL] \========================================================================
[FAIL]  ---------->>> PROCESS WAS ABORTED <<<---------- 


real	4m2.728s
user	0m0.500s
sys	0m0.963s

Run 3 also gives 186goes back to 186, unlike run 3 from the original source tree:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central-FRESH$ time ./rebuild.sh content/io/fabric8/kubernetes-client/kubernetes-client-5.4.1.buildspec
--snip--
[ERROR] Reproducible Build output summary: 186 files ok, 66 different
--snip--
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:55 min
[INFO] Finished at: 2023-11-13T22:10:50Z
[INFO] ------------------------------------------------------------------------
/home/wtwhite/code/reproducible-central-FRESH/bin/includes/rebuildToolMvn.sh: line 71: dos2unix: command not found

[FAIL] /========================================================================
[FAIL] |  failed to convert buildinfo newlines 
[FAIL] \========================================================================
[FAIL]  ---------->>> PROCESS WAS ABORTED <<<---------- 


real	3m57.401s
user	0m0.340s
sys	0m0.838s

Conclusion: Some kind of internal caching done by rebuild.sh causes the number of reproducible jars to change from 186 to 189 in a seemingly nondeterministic way -- in one source tree, we got the sequence 186, 189, 189 while in a second we saw 186, 189, 186. This is a bug in rebuild.sh and unfortunately means that for a strong test, we need to either

• fix it, or
• run each build from a fresh working copy and do a lot of unnecessary re-downloading.

Since the full build seems to only take 2-3x as long as the second build, I'll do the latter.

[wtwhite]

To test build-to-build variability, I ran the entire build 10 times, each in a separate worktree. Each logfile reported 186 files OK (matching Maven Central repo), just like the manual builds in the 2 worktrees described so far:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ for ((i = 1; i <= 10; ++i)); do echo "FULL BUILD REP $i"; ( git worktree add ~/code/reproducible-central-kubernetes-client-5.4.1-TAKE2-REP$i && cd ~/code/reproducible-central-kubernetes-client-5.4.1-TAKE2-REP$i && time ./rebuild.sh content/io/fabric8/kubernetes-client/kubernetes-client-5.4.1.buildspec ) > kubernetes-client-5.4.1-TAKE2-REP$i.out 2>&1; done
FULL BUILD REP 1
FULL BUILD REP 2
FULL BUILD REP 3
FULL BUILD REP 4
FULL BUILD REP 5
FULL BUILD REP 6
FULL BUILD REP 7
FULL BUILD REP 8
FULL BUILD REP 9
FULL BUILD REP 10
wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ grep 'Reproducible Build output summary:.*[^0-9][0-9][0-9][0-9] files ok' kubernetes-client-5.4.1-TAKE2-REP*.out
kubernetes-client-5.4.1-TAKE2-REP10.out:[ERROR] Reproducible Build output summary: 186 files ok, 66 different
kubernetes-client-5.4.1-TAKE2-REP1.out:[ERROR] Reproducible Build output summary: 186 files ok, 66 different
kubernetes-client-5.4.1-TAKE2-REP2.out:[ERROR] Reproducible Build output summary: 186 files ok, 66 different
kubernetes-client-5.4.1-TAKE2-REP3.out:[ERROR] Reproducible Build output summary: 186 files ok, 66 different
kubernetes-client-5.4.1-TAKE2-REP4.out:[ERROR] Reproducible Build output summary: 186 files ok, 66 different
kubernetes-client-5.4.1-TAKE2-REP5.out:[ERROR] Reproducible Build output summary: 186 files ok, 66 different
kubernetes-client-5.4.1-TAKE2-REP6.out:[ERROR] Reproducible Build output summary: 186 files ok, 66 different
kubernetes-client-5.4.1-TAKE2-REP7.out:[ERROR] Reproducible Build output summary: 186 files ok, 66 different
kubernetes-client-5.4.1-TAKE2-REP8.out:[ERROR] Reproducible Build output summary: 186 files ok, 66 different
kubernetes-client-5.4.1-TAKE2-REP9.out:[ERROR] Reproducible Build output summary: 186 files ok, 66 different

It's not clear how to find the set of 186+66=252 objects referred to, since there are 83 projects. Overapproximating by looking at all .jar files produced under the buildcache folder gives 461 jars, with all 10 builds producing differences somewhere:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ for ((i = 1; i <= 10; ++i)); do echo $i; ( cd ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP$i && time md5sum `find content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/ -name '*.jar'` > all_jars.md5 ); done
--snip--
wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ wc ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP1/all_jars.md5 
  461   922 89152 ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP1/all_jars.md5
wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ md5sum ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP*/all_jars.md5
8c992373b307401a5e3df0606e32c812  ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP10/all_jars.md5
bcd035c9e7e0ef2dd822f80e5429a48b  ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP1/all_jars.md5
641056da352189804d58acc3b536f04c  ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP2/all_jars.md5
9e2617ba778656a9f37e606fb30e8f40  ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP3/all_jars.md5
48f8d546266db5ef3e5dddce42223c5b  ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP4/all_jars.md5
cb54d4637a154d2ee51f16cfef78ce25  ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP5/all_jars.md5
48d4e72ecdfe1e49645b3317869c9aee  ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP6/all_jars.md5
81b02849859a62ecb38a2a6ed076289f  ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP7/all_jars.md5
7f080ad8c3b499e84e5760efdf4fac3a  ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP8/all_jars.md5
bd3385cf524d862fb2472c5c6a4d240d  ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP9/all_jars.md5

Comparing builds 1 and 2 shows 24 different jars:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ diff ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP{1,2}/all_jars.md5|wc -l
24
wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ diff ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP{1,2}/all_jars.md5
4c4
< 3bffae5e86965e0d7a4d7122af654dda  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/kubernetes-client/target/kubernetes-client-5.4.1.jar
---
> a712cf98027342a93197bef195779727  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/kubernetes-client/target/kubernetes-client-5.4.1.jar
14c14
< 9d8494d4a5b819f2e8a9889ffc9c7aa3  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/kubernetes-server-mock/target/kubernetes-server-mock-5.4.1.jar
---
> 407147022c3da42dfc50117f9796b28e  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/kubernetes-server-mock/target/kubernetes-server-mock-5.4.1.jar
134c134
< 20609e544d3aa6e0f040e1dc7bab7eb1  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/openshift-client/target/openshift-client-5.4.1.jar
---
> d0378d1d1d9960015b3ccb67f0af16ff  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/openshift-client/target/openshift-client-5.4.1.jar
147c147
< b22c966d16f54cf30dbad32af2fdc37f  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/kubernetes-examples/target/kubernetes-examples-5.4.1.jar
---
> 5433db04315bcd8a7030c78738ec2c2c  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/kubernetes-examples/target/kubernetes-examples-5.4.1.jar
155c155
< 504b02edb00cfb582269591e0edb25c4  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/crd-generator/apt/target/crd-generator-apt-5.4.1.jar
---
> ec7c85980ae989a70681882ae65bae38  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/crd-generator/apt/target/crd-generator-apt-5.4.1.jar
207c207
< 83a99b0c194359441b4e3bcb2bfad13d  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/platforms/karaf/itests/target/kubernetes-karaf-itests-5.4.1.jar
---
> 72bc573380d65a69cc3a76c1d9316da7  content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/platforms/karaf/itests/target/kubernetes-karaf-itests-5.4.1.jar

Taking the first differing jar, kubernetes-client-5.4.1.jar, as an example, the only difference is META-INF/jandex.idx:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ diff <(cd ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP1 && unzip -l -v content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/kubernetes-client/target/kubernetes-client-5.4.1.jar) <(cd ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP2 && unzip -l -v content/io/fabric8/kubernetes-client/buildcache/kubernetes-client-project/kubernetes-client/target/kubernetes-client-5.4.1.jar)
138c138
<   176933  Defl:N    85210  52% 2020-11-14 12:24 e291f61e  META-INF/jandex.idx
---
>   176934  Defl:N    85186  52% 2020-11-14 12:24 1ffcbf63  META-INF/jandex.idx
796c796
<  3050612           964691  68%                            791 files
---
>  3050613           964667  68%                            791 files

jandex.idx is a binary file, possibly produce by this tool. It's mentioned several times in https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/content/io/fabric8/kubernetes-client/kubernetes-client-project-5.4.1.diffoscope so it seems to be a known source of nondeterminism.

[wtwhite]

All jars appear in the same order in each all_jars.md5 file:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ for f in ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP*/all_jars.md5; do echo `cut -d' ' -f3 < $f|md5sum`" $f"; done
7505fa52aa1a85c520a31682f5fa91cd - ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP10/all_jars.md5
7505fa52aa1a85c520a31682f5fa91cd - ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP1/all_jars.md5
7505fa52aa1a85c520a31682f5fa91cd - ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP2/all_jars.md5
7505fa52aa1a85c520a31682f5fa91cd - ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP3/all_jars.md5
7505fa52aa1a85c520a31682f5fa91cd - ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP4/all_jars.md5
7505fa52aa1a85c520a31682f5fa91cd - ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP5/all_jars.md5
7505fa52aa1a85c520a31682f5fa91cd - ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP6/all_jars.md5
7505fa52aa1a85c520a31682f5fa91cd - ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP7/all_jars.md5
7505fa52aa1a85c520a31682f5fa91cd - ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP8/all_jars.md5
7505fa52aa1a85c520a31682f5fa91cd - ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP9/all_jars.md5

Since unzip -l -v outputs CRC-32 checksums, which are unlikely to collide for differing files, we can test whether all .class files are bitwise identical by running md5sum on the concatenated listings of only .class files from each jar. Doing this reveals that all .class files in all jars are identical across all 10 builds:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central$ for f in ../reproducible-central-kubernetes-client-5.4.1-TAKE2-REP*/all_jars.md5; do ( cd `dirname $f` && ( for z in `cut -d' ' -f3 < $f`; do unzip -l -v $z | grep '\.class$'; done ) |md5sum; ); done
295d44a48a921d7edfb0bb1260d0673b  -
295d44a48a921d7edfb0bb1260d0673b  -
295d44a48a921d7edfb0bb1260d0673b  -
295d44a48a921d7edfb0bb1260d0673b  -
295d44a48a921d7edfb0bb1260d0673b  -
295d44a48a921d7edfb0bb1260d0673b  -
295d44a48a921d7edfb0bb1260d0673b  -
295d44a48a921d7edfb0bb1260d0673b  -
295d44a48a921d7edfb0bb1260d0673b  -
295d44a48a921d7edfb0bb1260d0673b  -

[wtwhite]

Unlike running in a freshly checked-out worktree, which appears to always produce results where 186 files are identical to those on Maven Central repo, rerunning the build in the same worktree nondeterministically results in either 186 or 189 files being identical:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central-FRESH$ for ((i = 11; i <= 20; ++i)); do echo $i; ( time ./rebuild.sh content/io/fabric8/kubernetes-client/kubernetes-client-5.4.1.buildspec ) > RUN_IN_PLACE-$i.log 2>&1; grep 'ok=' RUN_IN_PLACE-$i.log; done
11
           ok=186
12
           ok=189
13
           ok=186
14
           ok=189
15
           ok=186
16
           ok=189
17
           ok=189
18
           ok=186
19
           ok=189
20
           ok=186
wtwhite@wtwhite-vuw-vm:~/code/reproducible-central-FRESH$ 

clean is the first mvn target that ./rebuild.sh runs, so this is a bug in some part of the build process.

The 3 files that differ between runs 11 and 12:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central-FRESH$ diff RUN_IN_PLACE-1{1,2}.okfiles
20a21
> chaosmesh-server-mock-5.4.1.jar
125a127
> model-annotator-5.4.1.jar
182a185
> volumesnapshot-server-mock-5.4.1.jar

It's always the same set of 3 files that differ:

wtwhite@wtwhite-vuw-vm:~/code/reproducible-central-FRESH$ for f in RUN_IN_PLACE-*.log; do perl -lne 'print join "\n", split / /, $1 if /okFiles="(.*)"/' < $f | sort > ${f%%.log}.okfiles; done
wtwhite@wtwhite-vuw-vm:~/code/reproducible-central-FRESH$ md5sum *.okfiles|cut -d' ' -f1|sort|uniq -c
      6 43e9edc108d6ecb41d7b456ee63649bd
      9 dad24fdfb4da7e9a78b0b26cdfdb1b8e

This is not the "ideal kind" of nondeterminism for us (the ideal kind would be nondeterminism within a clean build environment). But this may be worth investigating nevertheless.