EvoSuite under JDK 8 generates some tests that fail to compile

[wtwhite]

On #15, running EvoSuite under JDK 8 instead of 11 enabled it to actually generate some tests for com.google.common.collect.testing.Helpers where before it failed. However, in the course of generating those tests, it tried to generate several more that it couldn't include because it was unable to compile them (it seems that compiling its own generated tests is done as a sanity check before including them in the final output).

This is likely an internal issue with either EvoSuite or JDK 8's javac -- I won't look into this further.

Things look pretty good initially:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre$ time java -jar /home/wtwhite/code/regression-test-generation/evosuite/evosuite-1.2.0.jar -class 'com.google.common.collect.testing.Helpers' -projectCP /home/wtwhite/code/craw-redhat-oss/wget/crawl/repo1.maven.org/maven2/com/google/guava/guava-testlib/32.1.2-jre/guava-testlib-32.1.2-jre.jar:$(echo t/dependency/*|perl -lpe 's/ /:/g') 2>&1 | tee 'evosuite.com.google.common.collect.testing.Helpers.JDK8.log'
* EvoSuite 1.2.0
* Going to generate test cases for class: com.google.common.collect.testing.Helpers
* Starting Client-0
* Connecting to master process on port 6880
* Analyzing classpath: 
  - /home/wtwhite/code/craw-redhat-oss/wget/crawl/repo1.maven.org/maven2/com/google/guava/guava-testlib/32.1.2-jre/guava-testlib-32.1.2-jre.jar
  - t/dependency/asm-9.1.jar
  - t/dependency/auto-value-annotations-1.8.1.jar
  - t/dependency/checker-qual-3.33.0.jar
  - t/dependency/error_prone_annotations-2.18.0.jar
  - t/dependency/failureaccess-1.0.1.jar
  - t/dependency/guava-32.1.2-jre.jar
  - t/dependency/hamcrest-core-1.3.jar
  - t/dependency/j2objc-annotations-2.8.jar
  - t/dependency/jsr305-3.0.2.jar
  - t/dependency/junit-4.13.2.jar
  - t/dependency/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
  - t/dependency/truth-1.1.3.jar
* Finished analyzing classpath
* Generating tests for class com.google.common.collect.testing.Helpers
* Test criteria:
  - Line Coverage
  - Branch Coverage
  - Exception
  - Mutation testing (weak)
  - Method-Output Coverage
  - Top-Level Method Coverage
  - No-Exception Top-Level Method Coverage
  - Context Branch Coverage
[Progress:>                             0%] [Cov:>                                  0%]* Total number of test goals for DYNAMOSA: 942
* Using seed 1729659303012
* Starting evolution
* Initial Number of Goals in DynaMOSA = 350 / 942
[MASTER] 17:55:05.339 [logback-1] WARN  TestCluster - Failed to check cache for java.util.LinkedList<E> : Type points to itself
[MASTER] 17:55:05.341 [logback-1] WARN  TestCluster - Failed to check cache for java.lang.Comparable<T> : Type points to itself
[MASTER] 17:55:05.341 [logback-1] WARN  TestCluster - Failed to check cache for java.util.AbstractMap<K, V>.SimpleImmutableEntry<K, V> : Type points to itself
[Progress:==>                           8%] [Cov:======================>            65%][MASTER] 17:55:11.033 [logback-1] WARN  TestCluster - Failed to check cache for java.util.Iterator<E> : Type points to itself
[Progress:=======>                      25%] [Cov:=========================>         74%][MASTER] 17:55:20.452 [logback-1] WARN  TestCluster - Failed to check cache for java.util.HashMap<K, V> : Type points to itself
[Progress:=========>                    33%] [Cov:==========================>        76%][MASTER] 17:55:25.429 [logback-1] WARN  CastClassManager - Nothing is assignable after adding class java.util.Locale$FilteringMode for type Variable T with bounds [java.lang.Comparable<? super T>] of declaration Comparator#naturalOrder. The owner var map is {}
[MASTER] 17:55:25.716 [logback-1] WARN  CastClassManager - Nothing is assignable after adding java.lang.Comparable<T> for type Variable V with bounds [java.lang.Comparable<? super V>] of declaration Entry#comparingByValue. The owner var map is {K=java.lang.Comparable<java.lang.Comparable<T>>}
[Progress:============>                 41%] [Cov:==========================>        76%][MASTER] 17:55:30.435 [logback-1] WARN  TestCluster - Failed to check cache for java.util.function.Function<T, R> : Type points to itself
[Progress:===============>              50%] [Cov:==========================>        75%][MASTER] 17:55:35.206 [logback-1] WARN  CastClassManager - Nothing is assignable after adding class java.lang.Integer for type Variable K with bounds [java.lang.Comparable<? super K>] of declaration Entry#comparingByKey. The owner var map is {}
[Progress:================>             55%] [Cov:==========================>        75%][MASTER] 17:55:38.397 [logback-1] WARN  TestCluster - Failed to check cache for java.util.LinkedHashSet<E> : Type points to itself
[Progress:===================>          65%] [Cov:==========================>        75%][MASTER] 17:55:45.054 [logback-1] WARN  TestCluster - Failed to check cache for java.util.function.Predicate<java.lang.Object> : Type points to itself
[Progress:======================>       75%] [Cov:==========================>        76%][MASTER] 17:55:50.493 [logback-1] WARN  TestCluster - Failed to check cache for com.google.common.collect.DiscreteDomain<C> : Type points to itself
[Progress:========================>     83%] [Cov:==========================>        76%][MASTER] 17:55:55.620 [logback-1] WARN  CastClassManager - Nothing is assignable after adding java.lang.Comparable<T> for type Variable V with bounds [java.lang.Comparable<? super V>] of declaration Entry#comparingByValue. The owner var map is {K=java.lang.Comparable<com.google.common.collect.testing.Helpers$NullsBeforeB>}
[Progress:==========================>   88%] [Cov:==========================>        76%][MASTER] 17:55:59.432 [logback-1] WARN  TestCluster - Failed to check cache for java.util.HashSet<E> : Type points to itself
[Progress:=============================>98%] [Cov:==========================>        76%][MASTER] 17:56:04.162 [logback-1] WARN  CastClassManager - Nothing is assignable after adding class java.lang.Integer for type Variable K with bounds [java.lang.Comparable<? super K>] of declaration Entry#comparingByKey. The owner var map is {}
[Progress:==============================100%] [Cov:==========================>        76%]
* Search finished after 62s and 134 generations, 59387 statements, best individual has fitness: 216.0
* Minimizing test suite
* Going to analyze the coverage criteria
* Coverage analysis for criterion LINE
* Coverage of criterion LINE: 81%
* Total number of goals: 177
* Number of covered goals: 144
* Coverage analysis for criterion BRANCH
* Coverage of criterion BRANCH: 81%
* Total number of goals: 149
* Number of covered goals: 121
* Coverage analysis for criterion EXCEPTION
* Coverage of criterion EXCEPTION: 100%
* Total number of goals: 41
* Number of covered goals: 41
* Coverage analysis for criterion WEAKMUTATION
* Coverage of criterion WEAKMUTATION: 84%
* Total number of goals: 341
* Number of covered goals: 287
* Coverage analysis for criterion OUTPUT
* Coverage of criterion OUTPUT: 53%
* Total number of goals: 62
* Number of covered goals: 33
* Coverage analysis for criterion METHOD
* Coverage of criterion METHOD: 97%
* Total number of goals: 32
* Number of covered goals: 31
* Coverage analysis for criterion METHODNOEXCEPTION
* Coverage of criterion METHODNOEXCEPTION: 91%
* Total number of goals: 32
* Number of covered goals: 29
* Coverage analysis for criterion CBRANCH
* Coverage of criterion CBRANCH: 81%
* Total number of goals: 149
* Number of covered goals: 120
* Generated 83 tests with total length 246
* Resulting test suite's coverage: 76% (average coverage for all fitness functions)
* Generating assertions
* Resulting test suite's mutation score: 35%
* Compiling and checking tests
--snip--

But then it seems that EvoSuite tries to compile the tests itself, and finds many that fail compilation:

[MASTER] 17:56:12.130 [logback-2] ERROR JUnitAnalyzer - Compilation failed on compilation units: [RegularFileObject[/tmp/EvoSuite_43_1729659372112/com/google/common/collect/testing/Helpers_43_tmp__ESTest.java], RegularFileObject[/tmp/EvoSuite_43_1729659372112/com/google/common/collect/testing/Helpers_43_tmp__ESTest_scaffolding.java]]
[MASTER] 17:56:12.130 [logback-2] ERROR JUnitAnalyzer - Classpath: /home/wtwhite/code/craw-redhat-oss/wget/crawl/repo1.maven.org/maven2/com/google/guava/guava-testlib/32.1.2-jre/guava-testlib-32.1.2-jre.jar:t/dependency/asm-9.1.jar:t/dependency/auto-value-annotations-1.8.1.jar:t/dependency/checker-qual-3.33.0.jar:t/dependency/error_prone_annotations-2.18.0.jar:t/dependency/failureaccess-1.0.1.jar:t/dependency/guava-32.1.2-jre.jar:t/dependency/hamcrest-core-1.3.jar:t/dependency/j2objc-annotations-2.8.jar:t/dependency/jsr305-3.0.2.jar:t/dependency/junit-4.13.2.jar:t/dependency/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:t/dependency/truth-1.1.3.jar:/home/wtwhite/code/regression-test-generation/evosuite/evosuite-1.2.0.jar:/home/wtwhite/code/craw-redhat-oss/wget/crawl/repo1.maven.org/maven2/com/google/guava/guava-testlib/32.1.2-jre/guava-testlib-32.1.2-jre.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/asm-9.1.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/auto-value-annotations-1.8.1.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/checker-qual-3.33.0.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/error_prone_annotations-2.18.0.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/failureaccess-1.0.1.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/guava-32.1.2-jre.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/hamcrest-core-1.3.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/j2objc-annotations-2.8.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/jsr305-3.0.2.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/junit-4.13.2.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/truth-1.1.3.jar
[MASTER] 17:56:12.130 [logback-2] ERROR JUnitAnalyzer - evosuiteCP: /home/wtwhite/code/regression-test-generation/evosuite/evosuite-1.2.0.jar:/home/wtwhite/code/craw-redhat-oss/wget/crawl/repo1.maven.org/maven2/com/google/guava/guava-testlib/32.1.2-jre/guava-testlib-32.1.2-jre.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/asm-9.1.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/auto-value-annotations-1.8.1.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/checker-qual-3.33.0.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/error_prone_annotations-2.18.0.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/failureaccess-1.0.1.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/guava-32.1.2-jre.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/hamcrest-core-1.3.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/j2objc-annotations-2.8.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/jsr305-3.0.2.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/junit-4.13.2.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/wtwhite/code/regression-test-generation/find-diff-classes/try_manually/testgen/mvnc/com/google/guava/guava-testlib/32.1.2-jre/t/dependency/truth-1.1.3.jar
[MASTER] 17:56:12.130 [logback-2] ERROR JUnitAnalyzer - Diagnostic: incompatible types: java.util.Comparator<java.util.Map.Entry<java.util.Locale.FilteringMode,java.lang.Object>> cannot be converted to java.util.Comparator<java.util.Map.Entry<java.util.Locale.FilteringMode,java.util.LinkedList<java.lang.Comparable>>>: 29
[MASTER] 17:56:12.131 [logback-2] ERROR JUnitAnalyzer - Diagnostic: /tmp/EvoSuite_43_1729659372112/com/google/common/collect/testing/Helpers_43_tmp__ESTest.java uses unchecked or unsafe operations.: -1
[MASTER] 17:56:12.131 [logback-2] ERROR JUnitAnalyzer - Diagnostic: Recompile with -Xlint:unchecked for details.: -1
[MASTER] 17:56:12.131 [logback-2] ERROR JUnitAnalyzer - RegularFileObject[/tmp/EvoSuite_43_1729659372112/com/google/common/collect/testing/Helpers_43_tmp__ESTest.java]
1: /*
2:  * This file was automatically generated by EvoSuite
3:  * Wed Oct 23 04:56:12 GMT 2024
4:  */
5: 
6: package com.google.common.collect.testing;
7: 
8: import org.junit.Test;
9: import static org.junit.Assert.*;
10: import static org.evosuite.shaded.org.mockito.Mockito.*;
11: import com.google.common.collect.testing.Helpers;
12: import java.util.Comparator;
13: import java.util.LinkedList;
14: import java.util.Locale;
15: import java.util.Map;
16: import java.util.function.ToLongFunction;
17: import org.evosuite.runtime.EvoRunner;
18: import org.evosuite.runtime.EvoRunnerParameters;
19: import org.evosuite.runtime.ViolatedAssumptionAnswer;
20: import org.junit.runner.RunWith;
21: 
22: @RunWith(EvoRunner.class) @EvoRunnerParameters(mockJVMNonDeterminism = true, useVFS = true, useVNET = true, resetStaticState = true) 
23: public class Helpers_43_tmp__ESTest extends Helpers_43_tmp__ESTest_scaffolding {
24: 
25:   @Test(timeout = 4000)
26:   public void test0()  throws Throwable  {
27:       ToLongFunction<Locale.FilteringMode> toLongFunction0 = (ToLongFunction<Locale.FilteringMode>) mock(ToLongFunction.class, new ViolatedAssumptionAnswer());
28:       Comparator<Locale.FilteringMode> comparator0 = Comparator.comparingLong((ToLongFunction<? super Locale.FilteringMode>) toLongFunction0);
29:       Comparator<Map.Entry<Locale.FilteringMode, LinkedList<Comparable>>> comparator1 = (Comparator<Map.Entry<Locale.FilteringMode, LinkedList<Comparable>>>)Helpers.entryComparator((Comparator<? super Locale.FilteringMode>) comparator0);
30:       assertNotNull(comparator1);
31:   }
32: }
RegularFileObject[/tmp/EvoSuite_43_1729659372112/com/google/common/collect/testing/Helpers_43_tmp__ESTest.java]
--snip--

[MASTER] 17:56:12.131 [logback-2] ERROR JUnitAnalyzer - Failed to compile test case:
ToLongFunction<Locale.FilteringMode> toLongFunction0 = (ToLongFunction<Locale.FilteringMode>) mock(ToLongFunction.class, new ViolatedAssumptionAnswer());
Comparator<Locale.FilteringMode> comparator0 = Comparator.comparingLong((ToLongFunction<? super Locale.FilteringMode>) toLongFunction0);
Comparator<Map.Entry<Locale.FilteringMode, LinkedList<Comparable>>> comparator1 = (Comparator<Map.Entry<Locale.FilteringMode, LinkedList<Comparable>>>)Helpers.entryComparator((Comparator<? super Locale.FilteringMode>) comparator0);
assertNotNull(comparator1);

I dug into the test0() generated method, and I think the issue is that Helpers.entryComparator() has 2 generic parameters, but one of them (V) cannot be inferred from any argument types in the call, so the JDK 8 compiler is assigning it Object. Rather than trying to cast the result of the Helpers.entryComparator() method call, EvoSuite should be explicitly specifying it as part of the method call, i.e., change line 29 to:

Comparator<Map.Entry<Locale.FilteringMode, LinkedList<Comparable>>> comparator1 = Helpers.<Locale.FilteringMode, LinkedList<Comparable>>entryComparator((Comparator<? super Locale.FilteringMode>) comparator0);

Nevertheless 78 tests do still get produced for the Helpers class, so I think we can consider switching to JDK 8 a success.