2 differing test outcomes between mvnc and gaoss in `io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest`

[wtwhite]

The mvnc version of io.netty:netty-codec-http:4.1.102.Final has 1 failure (test40()), while gaoss has that plus 2 additional failures (test50(), test51()).

This was present in this table, but I missed it at the time.

Need to investigate this quickly.

[wtwhite]

The 3 errors are 10x reproducible with gaoss when running against the JaCoCo-instrumented version of the class under test:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ for i in 1 2 3 4 5 6 7 8 9; do echo $i; RUNDIR=run.repro$i ./run_gaoss_AbstractDiskHttpData_ESTest_using_jacoco.sh ; done
--snip--
wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ tail -n 2 run.repro*/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out
==> run.repro1/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro2/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro3/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro4/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro5/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro6/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro7/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro8/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro9/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3

While mvnc produced only 1 error on each of the 10 runs:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ for i in 1 2 3 4 5 6 7 8 9; do echo $i; RUNDIR=run.repro$i ./run_mvnc_AbstractDiskHttpData_ESTest_using_jacoco.sh ; done
--snip--
wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ tail -n 2 run.repro*/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out
==> run.repro1/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro2/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro3/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro4/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro5/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro6/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro7/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro8/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro9/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1

Trying to repro without JaCoCo still makes no difference.

3 failures on gaoss:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ for i in 1 2 3 4 5 6 7 8 9; do echo $i; RUNDIR=run.repro-no-jacoco$i ./run_gaoss_AbstractDiskHttpData_ESTest.sh ; done
--snip--
wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ tail -n 2 run.repro-no-jacoco*/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out
==> run.repro-no-jacoco1/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro-no-jacoco2/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro-no-jacoco3/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro-no-jacoco4/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro-no-jacoco5/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro-no-jacoco6/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro-no-jacoco7/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro-no-jacoco8/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3


==> run.repro-no-jacoco9/gaoss/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 3

1 failure on mvnc:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ for i in 1 2 3 4 5 6 7 8 9; do echo $i; RUNDIR=run.repro-no-jacoco$i ./run_mvnc_AbstractDiskHttpData_ESTest.sh ; done
--snip--
wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ tail -n 2 run.repro-no-jacoco*/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out
==> run.repro-no-jacoco1/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro-no-jacoco2/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro-no-jacoco3/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro-no-jacoco4/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro-no-jacoco5/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro-no-jacoco6/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro-no-jacoco7/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro-no-jacoco8/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1


==> run.repro-no-jacoco9/mvnc/io/netty/netty-codec-http/4.1.102.Final/io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.out <==
Tests run: 68,  Failures: 1

So the difference doesn't appear to be flaky. Next step: Debugging.

[wtwhite]

TL;DR: This is basically the reverse of #36 (this time, gaoss was compiled with an old compiler, mvnc with a new one), but the differences only surface in the first place due to a test-interference bug (i.e., running each failing test individually would show identical results for mvnc and gaoss, even though a behavioural difference does exist).

Investigation

I opened each of the 2 (un-JaCoCofied) Maven projects in Intellij, set the JDK to 8, and disabled the timeout for test50(). I can repro the difference (3 failures for gaoss, 1 for mvnc) in Intellij's debugger when debugging the entire test class. However, debugging individual test methods behaves differently.

Recall that test50() and test51() succeed (meaning that they "correctly" throw NoSuchMethodError) only on mvnc.

In the mvnc artifact:

• 10 manual debuggings of just test50() (not the whole class) all failed (I expected success)
• I set a breakpoint inside test50() on the line that should trigger the NoSuchMethodError, and debugged just this test method. The debugger paused at the breakpoint; single-stepping with F8 caused the line to execute without throwing an exception. Trying to continue the program hung the program.
• I moved the breakpoint to the next line, which should not be reached under mvnc.
  • Debugging just this method resulted in the debugger stopping at the breakpoint, contrary to expectations.
  • Debugging the entire test class did not hit the breakpoint.

Finally, moving the breakpoint to the verifyException() call makes the debugger hit the breakpoint, and the NoSuchMethodError's e.detailMessage reveals it's our old friend java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;, called from here.

Intellij refuses to step past this point, complaining that the source differs from the bytecode, but setting conditional breakpoints on length == 1042 still works (test50() is the only direct caller of getChunk() that supplies this argument value).

After setting conditional breakpoints at that line and in various places in getChunk():

• Debugging just test50() stops at getChunk() as expected.
  • read is 0, so flip() is never called, so the NoSuchMethodError is never triggered. Falls through to fail("Expecting exception: NoSuchMethodError") which is not expected.
• Debugging the entire test class stops at getChunk() as expected.
  • read is 2, so flip() is called, triggering NoSuchMethodError and causing the test to pass.
  • The first 2 bytes in byteBuffer.hb are 102, 72 ("fH") which doesn't correspond to anything in this test method -- but it is a string provided as an argument in several method calls in other test methods, namely test09()andtest48()`:

  public void test09()  throws Throwable  {
    Future<?> future = executor.submit(new Runnable(){ 
            @Override public void run() { 
        try {
          Charset charset0 = Charset.defaultCharset();
          MockFile mockFile0 = new MockFile("fH");
          DiskAttribute diskAttribute0 = new DiskAttribute("fH", charset0);
          diskAttribute0.setContent((File) mockFile0);
          diskAttribute0.setValue("fH");
--snip--
  }
--snip--
  public void test48()  throws Throwable  {
      Charset charset0 = Charset.defaultCharset();
      MockFile mockFile0 = new MockFile("fH");
      DiskAttribute diskAttribute0 = new DiskAttribute("fH", charset0);
      diskAttribute0.setContent((File) mockFile0);
--snip--
  }

mvnc was compiled at JDK 19:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ unzip -p /home/wtwhite/code/craw-redhat-oss/wget/crawl/repo1.maven.org/maven2/io/netty/netty-codec-http/4.1.102.Final/netty-codec-http-4.1.102.Final.jar META-INF/MANIFEST.MF
Manifest-Version: 1.0
Created-By: Apache Maven Bundle Plugin 5.1.8
Built-By: chris
Build-Jdk: 19.0.2
Implementation-Title: Netty/Codec/HTTP
Implementation-Version: 4.1.102.Final
Implementation-Vendor-Id: io.netty
Implementation-Vendor: The Netty Project
Implementation-URL: https://netty.io/netty-codec-http/
Automatic-Module-Name: io.netty.codec.http
Bnd-LastModified: 1702419701527
Build-Jdk-Spec: 19
Bundle-Description: Netty is an asynchronous event-driven network applic
 ation framework for    rapid development of maintainable high performan
 ce protocol servers and    clients.
Bundle-DocURL: https://netty.io/
Bundle-License: https://www.apache.org/licenses/LICENSE-2.0
Bundle-ManifestVersion: 2
Bundle-Name: Netty/Codec/HTTP
Bundle-SymbolicName: io.netty.codec-http
Bundle-Vendor: The Netty Project
Bundle-Version: 4.1.102.Final
Export-Package: io.netty.handler.codec.http;uses:="io.netty.buffer,io.ne
 tty.channel,io.netty.channel.embedded,io.netty.handler.codec,io.netty.h
 andler.codec.compression,io.netty.handler.codec.http.cookie,io.netty.ha
 ndler.stream,io.netty.util";version="4.1.102",io.netty.handler.codec.ht
 tp.cookie;version="4.1.102",io.netty.handler.codec.http.cors;uses:="io.
 netty.channel,io.netty.handler.codec.http";version="4.1.102",io.netty.h
 andler.codec.http.multipart;uses:="io.netty.buffer,io.netty.channel,io.
 netty.handler.codec,io.netty.handler.codec.http,io.netty.handler.stream
 ,io.netty.util";version="4.1.102",io.netty.handler.codec.http.websocket
 x;uses:="io.netty.buffer,io.netty.channel,io.netty.handler.codec,io.net
 ty.handler.codec.http,io.netty.handler.stream,io.netty.util,io.netty.ut
 il.internal.logging";version="4.1.102",io.netty.handler.codec.http.webs
 ocketx.extensions;uses:="io.netty.channel,io.netty.handler.codec,io.net
 ty.handler.codec.http,io.netty.handler.codec.http.websocketx";version="
 4.1.102",io.netty.handler.codec.http.websocketx.extensions.compression;
 uses:="io.netty.channel,io.netty.handler.codec.http.websocketx.extensio
 ns";version="4.1.102",io.netty.handler.codec.rtsp;uses:="io.netty.buffe
 r,io.netty.channel,io.netty.handler.codec.http,io.netty.util";version="
 4.1.102",io.netty.handler.codec.spdy;uses:="io.netty.buffer,io.netty.ch
 annel,io.netty.handler.codec,io.netty.handler.codec.http,io.netty.util"
 ;version="4.1.102"
Import-Package: com.aayushatharva.brotli4j.encoder;resolution:=optional,
 com.jcraft.jzlib;resolution:=optional,io.netty.buffer;version="[4.1,5)"
 ,io.netty.channel;version="[4.1,5)",io.netty.channel.embedded;version="
 [4.1,5)",io.netty.handler.codec;version="[4.1,5)",io.netty.handler.code
 c.base64;version="[4.1,5)",io.netty.handler.codec.compression;version="
 [4.1,5)",io.netty.handler.codec.http,io.netty.handler.codec.http.cookie
 ,io.netty.handler.codec.http.websocketx,io.netty.handler.codec.http.web
 socketx.extensions,io.netty.handler.ssl;version="[4.1,5)",io.netty.hand
 ler.stream;version="[4.1,5)",io.netty.util;version="[4.1,5)",io.netty.u
 til.concurrent;version="[4.1,5)",io.netty.util.internal;version="[4.1,5
 )",io.netty.util.internal.logging;version="[4.1,5)",sun.nio.ch;resoluti
 on:=optional,org.eclipse.jetty.npn;version="[1,2)";resolution:=optional
 ,org.eclipse.jetty.alpn;version="[1,2)";resolution:=optional
Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.8))"
Tool: Bnd-6.3.1.202206071316

So although it targets JDK 8, it tries to invoke java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ javap -c -p -v -cp /home/wtwhite/code/craw-redhat-oss/wget/crawl/repo1.maven.org/maven2/io/netty/netty-codec-http/4.1.102.Final/netty-codec-http-4.1.102.Final.jar io.netty.handler.codec.http.multipart.AbstractDiskHttpData
--snip--
  major version: 52
--snip--
  public io.netty.buffer.ByteBuf getChunk(int) throws java.io.IOException;
    descriptor: (I)Lio/netty/buffer/ByteBuf;
    flags: (0x0001) ACC_PUBLIC
    Code:
      stack=4, locals=5, args_size=2
         0: aload_0
--snip--
       125: invokevirtual #209                // Method java/nio/ByteBuffer.flip:()Ljava/nio/ByteBuffer;
--snip--
       151: areturn
--snip--

OTOH, gaoss was built at JDK 8:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ unzip -p /home/wtwhite/code/craw-redhat-oss/wget/crawl/google-aoss/io/netty/netty-codec-http/4.1.102.Final/netty-codec-http-4.1.102.Final.jar META-INF/MANIFEST.MF
Manifest-Version: 1.0
Implementation-Title: Netty/Codec/HTTP
Bundle-Description: Netty is an asynchronous event-driven network appl
 ication framework for    rapid development of maintainable high perfo
 rmance protocol servers and    clients.
Automatic-Module-Name: io.netty.codec.http
Bundle-License: https://www.apache.org/licenses/LICENSE-2.0
Bundle-SymbolicName: io.netty.codec-http
Implementation-Version: 4.1.102.Final
Built-By: root
Bnd-LastModified: 1703219446220
Bundle-ManifestVersion: 2
Implementation-Vendor-Id: io.netty
Bundle-DocURL: https://netty.io/
Bundle-Vendor: The Netty Project
Import-Package: com.aayushatharva.brotli4j.encoder;resolution:=optiona
 l,com.jcraft.jzlib;resolution:=optional,io.netty.buffer;version="[4.1
 ,5)",io.netty.channel;version="[4.1,5)",io.netty.channel.embedded;ver
 sion="[4.1,5)",io.netty.handler.codec;version="[4.1,5)",io.netty.hand
 ler.codec.base64;version="[4.1,5)",io.netty.handler.codec.compression
 ;version="[4.1,5)",io.netty.handler.codec.http,io.netty.handler.codec
 .http.cookie,io.netty.handler.codec.http.websocketx,io.netty.handler.
 codec.http.websocketx.extensions,io.netty.handler.ssl;version="[4.1,5
 )",io.netty.handler.stream;version="[4.1,5)",io.netty.util;version="[
 4.1,5)",io.netty.util.concurrent;version="[4.1,5)",io.netty.util.inte
 rnal;version="[4.1,5)",io.netty.util.internal.logging;version="[4.1,5
 )",sun.nio.ch;resolution:=optional,org.eclipse.jetty.npn;version="[1,
 2)";resolution:=optional,org.eclipse.jetty.alpn;version="[1,2)";resol
 ution:=optional
Require-Capability: osgi.ee;filter:="(&(osgi.ee=JavaSE)(version=1.6))"
Tool: Bnd-6.3.1.202206071316
Implementation-Vendor: The Netty Project
Export-Package: io.netty.handler.codec.http;uses:="io.netty.buffer,io.
 netty.channel,io.netty.channel.embedded,io.netty.handler.codec,io.net
 ty.handler.codec.compression,io.netty.handler.codec.http.cookie,io.ne
 tty.handler.stream,io.netty.util";version="4.1.102",io.netty.handler.
 codec.http.cookie;version="4.1.102",io.netty.handler.codec.http.cors;
 uses:="io.netty.channel,io.netty.handler.codec.http";version="4.1.102
 ",io.netty.handler.codec.http.multipart;uses:="io.netty.buffer,io.net
 ty.channel,io.netty.handler.codec,io.netty.handler.codec.http,io.nett
 y.handler.stream,io.netty.util";version="4.1.102",io.netty.handler.co
 dec.http.websocketx;uses:="io.netty.buffer,io.netty.channel,io.netty.
 handler.codec,io.netty.handler.codec.http,io.netty.handler.stream,io.
 netty.util,io.netty.util.internal.logging";version="4.1.102",io.netty
 .handler.codec.http.websocketx.extensions;uses:="io.netty.channel,io.
 netty.handler.codec,io.netty.handler.codec.http,io.netty.handler.code
 c.http.websocketx";version="4.1.102",io.netty.handler.codec.http.webs
 ocketx.extensions.compression;uses:="io.netty.channel,io.netty.handle
 r.codec.http.websocketx.extensions";version="4.1.102",io.netty.handle
 r.codec.rtsp;uses:="io.netty.buffer,io.netty.channel,io.netty.handler
 .codec.http,io.netty.util";version="4.1.102",io.netty.handler.codec.s
 pdy;uses:="io.netty.buffer,io.netty.channel,io.netty.handler.codec,io
 .netty.handler.codec.http,io.netty.util";version="4.1.102"
Bundle-Name: Netty/Codec/HTTP
Bundle-Version: 4.1.102.Final
Build-Jdk-Spec: 1.8
Created-By: Apache Maven Bundle Plugin 5.1.8
Build-Jdk: 1.8.0_352
Implementation-URL: https://netty.io/netty-codec-http/

And targets JDK 1.6, so it calls java/nio/ByteBuffer.flip:()Ljava/nio/Buffer;:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ javap -c -p -v -cp /home/wtwhite/code/craw-redhat-oss/wget/crawl/google-aoss/io/netty/netty-codec-http/4.1.102.Final/netty-codec-http-4.1.102.Final.jar io.netty.handler.codec.http.multipart.AbstractDiskHttpData
--snip--
  major version: 50
--snip--
  public io.netty.buffer.ByteBuf getChunk(int) throws java.io.IOException;
    descriptor: (I)Lio/netty/buffer/ByteBuf;
    flags: (0x0001) ACC_PUBLIC
    Code:
      stack=4, locals=5, args_size=2
         0: aload_0
--snip--
       124: invokevirtual #78                 // Method java/nio/ByteBuffer.flip:()Ljava/nio/Buffer;
--snip--
       150: areturn
--snip--

Conclusions

Test methods are interfering with each other when multiple tests are run at once, and ironically this bug is necessary to show the difference in behaviour between the two providers.

Summarising:

• gaoss was compiled under JDK 8, thus invokes java.nio.ByteBuffer.flip()Ljava/nio/Buffer;, which does exist at runtime at JDK 8, so no NoSuchMethodError is thrown, so the test fails
• mvnc was compiled under JDK 19, thus invokes java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;, which does not exist at runtime at JDK 8, so a NoSuchMethodError is thrown, and the test passes
  • But only if flip() was actually called, which only happens when multiple tests are run at the same time and trigger a test-interference bug (in either EvoSuite or JUnit)

[wtwhite]

Rerunning the tests under JDK 11

We expect that running these tests under JDK 11 should cause all 3 (test40(), test50() and test51()) to fail with Expecting exception: NoSuchMethodError not only for gaoss but also for mvnc, since now java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer; does exist at runtime.

gaoss

Running the tests on gaoss under JDK 11 produces the 3 expected failures, as well as 13 other failures (that likely stem from EvoSuite unhappiness with JDK 11):

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes/compile/gaoss/io/netty/netty-codec-http/4.1.102.Final$ CLASSPATH="/home/wtwhite/code/craw-redhat-oss/wget/crawl/google-aoss/io/netty/netty-codec-http/4.1.102.Final/netty-codec-http-4.1.102.Final.jar:$(echo t/dependency/*|perl -lpe 's/ /:/g'):/home/wtwhite/code/regression-test-generation/evosuite/evosuite-standalone-runtime-1.2.0.jar:.:/home/wtwhite/.m2/repository/junit/junit/4.13.2/junit-4.13.2.jar:/home/wtwhite/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar" time /usr/lib/jvm/java-11-openjdk-amd64/bin/java org.junit.runner.JUnitCore "io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest" > gaoss_jdk11.out 2> gaoss_jdk11.err
wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes/compile/gaoss/io/netty/netty-codec-http/4.1.102.Final$ cat gaoss_jdk11.out 
--snip--
Time: 5.715
There were 16 failures:
1) test60(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Expecting exception: NoSuchMethodError
        at org.junit.Assert.fail(Assert.java:89)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test60(AbstractDiskHttpData_ESTest.java:1043)
2) test20(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: expected:<17> but was:<0>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test20(AbstractDiskHttpData_ESTest.java:379)
3) test23(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.readFrom(AbstractDiskHttpData.java:455)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.get(AbstractDiskHttpData.java:301)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test23(AbstractDiskHttpData_ESTest.java:407)
4) test29(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Exception was not thrown in org.evosuite.runtime.vfs.VirtualFileSystem but in org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.jav
a:104): java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.EvoAssertions.assertThrownBy(EvoAssertions.java:111)
        at org.evosuite.runtime.EvoAssertions.verifyException(EvoAssertions.java:48)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test29(AbstractDiskHttpData_ESTest.java:512)
5) test33(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Exception was not thrown in org.evosuite.runtime.vfs.VirtualFileSystem but in org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.jav
a:104): java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.EvoAssertions.assertThrownBy(EvoAssertions.java:111)
        at org.evosuite.runtime.EvoAssertions.verifyException(EvoAssertions.java:48)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test33(AbstractDiskHttpData_ESTest.java:589)
6) test37(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Expecting exception: FileNotFoundException
        at org.junit.Assert.fail(Assert.java:89)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test37(AbstractDiskHttpData_ESTest.java:669)
7) test40(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Expecting exception: NoSuchMethodError
        at org.junit.Assert.fail(Assert.java:89)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test40(AbstractDiskHttpData_ESTest.java:715)
8) test43(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.renameTo(AbstractDiskHttpData.java:386)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test43(AbstractDiskHttpData_ESTest.java:755)
9) test42(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.readFrom(AbstractDiskHttpData.java:455)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.getString(AbstractDiskHttpData.java:363)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.getString(AbstractDiskHttpData.java:351)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test42(AbstractDiskHttpData_ESTest.java:742)
10) test44(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.renameTo(AbstractDiskHttpData.java:386)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test44(AbstractDiskHttpData_ESTest.java:767)
11) test46(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Exception was not thrown in org.evosuite.runtime.vfs.VirtualFileSystem but in org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104): java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.EvoAssertions.assertThrownBy(EvoAssertions.java:111)
        at org.evosuite.runtime.EvoAssertions.verifyException(EvoAssertions.java:48)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test46(AbstractDiskHttpData_ESTest.java:808)
12) test07(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.renameTo(AbstractDiskHttpData.java:386)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test07(AbstractDiskHttpData_ESTest.java:217)
13) test08(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Exception was not thrown in org.evosuite.runtime.vfs.VirtualFileSystem but in org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104): java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.EvoAssertions.assertThrownBy(EvoAssertions.java:111)
        at org.evosuite.runtime.EvoAssertions.verifyException(EvoAssertions.java:48)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test08(AbstractDiskHttpData_ESTest.java:239)
14) test50(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Expecting exception: NoSuchMethodError
        at org.junit.Assert.fail(Assert.java:89)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test50(AbstractDiskHttpData_ESTest.java:865)
15) test51(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Expecting exception: NoSuchMethodError
        at org.junit.Assert.fail(Assert.java:89)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test51(AbstractDiskHttpData_ESTest.java:885)
16) test14(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: expected:<2> but was:<0>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test14(AbstractDiskHttpData_ESTest.java:322)

FAILURES!!!
Tests run: 68,  Failures: 16

mvnc

On mvnc at JDK 11, we get the same:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes/compile/mvnc/io/netty/netty-codec-http/4.1.102.Final$ CLASSPATH=/home/wtwhite/code/craw-redhat-oss/wget/crawl/repo1.maven.org/maven2/io/netty/netty-codec-http/4.1.102.Final/netty-codec-http-4.1.102.Final.jar:$(echo t/dependency/*|perl -lpe 's/ /:/g'):/home/wtwhite/code/regression-test-generation/evosuite/evosuite-standalone-runtime-1.2.0.jar:.:/home/wtwhite/.m2/repository/junit/junit/4.13.2/junit-4.13.2.jar:/home/wtwhite/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar time /usr/lib/jvm/java-11-openjdk-amd64/bin/java org.junit.runner.JUnitCore io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest > mvnc_jdk11.out 2> mvnc_jdk11.err
wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes/compile/mvnc/io/netty/netty-codec-http/4.1.102.Final$ cat mvnc_jdk11.out 
--snip--
Time: 5.533
There were 16 failures:
1) test60(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Expecting exception: NoSuchMethodError
        at org.junit.Assert.fail(Assert.java:89)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test60(AbstractDiskHttpData_ESTest.java:1043)
2) test20(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: expected:<17> but was:<0>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test20(AbstractDiskHttpData_ESTest.java:379)
3) test23(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.readFrom(AbstractDiskHttpData.java:455)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.get(AbstractDiskHttpData.java:301)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test23(AbstractDiskHttpData_ESTest.java:407)
4) test29(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Exception was not thrown in org.evosuite.runtime.vfs.VirtualFileSystem but in org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.jav
a:104): java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.EvoAssertions.assertThrownBy(EvoAssertions.java:111)
        at org.evosuite.runtime.EvoAssertions.verifyException(EvoAssertions.java:48)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test29(AbstractDiskHttpData_ESTest.java:512)
5) test33(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Exception was not thrown in org.evosuite.runtime.vfs.VirtualFileSystem but in org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.jav
a:104): java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.EvoAssertions.assertThrownBy(EvoAssertions.java:111)
        at org.evosuite.runtime.EvoAssertions.verifyException(EvoAssertions.java:48)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test33(AbstractDiskHttpData_ESTest.java:589)
6) test37(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Expecting exception: FileNotFoundException
        at org.junit.Assert.fail(Assert.java:89)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test37(AbstractDiskHttpData_ESTest.java:669)
7) test40(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Expecting exception: NoSuchMethodError
        at org.junit.Assert.fail(Assert.java:89)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test40(AbstractDiskHttpData_ESTest.java:715)
8) test43(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.renameTo(AbstractDiskHttpData.java:386)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test43(AbstractDiskHttpData_ESTest.java:755)
9) test42(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.readFrom(AbstractDiskHttpData.java:455)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.getString(AbstractDiskHttpData.java:363)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.getString(AbstractDiskHttpData.java:351)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test42(AbstractDiskHttpData_ESTest.java:742)
10) test44(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.renameTo(AbstractDiskHttpData.java:386)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test44(AbstractDiskHttpData_ESTest.java:767)
11) test46(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Exception was not thrown in org.evosuite.runtime.vfs.VirtualFileSystem but in org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104): java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.EvoAssertions.assertThrownBy(EvoAssertions.java:111)
        at org.evosuite.runtime.EvoAssertions.verifyException(EvoAssertions.java:48)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test46(AbstractDiskHttpData_ESTest.java:808)
12) test07(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData.renameTo(AbstractDiskHttpData.java:386)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test07(AbstractDiskHttpData_ESTest.java:217)
13) test08(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Exception was not thrown in org.evosuite.runtime.vfs.VirtualFileSystem but in org.evosuite.runtime.mock.java.io.MockRandomAccessFile.<init>(MockRandomAccessFile.java:104): java.io.FileNotFoundException: File does not exist, and RandomAccessFile is not open in write mode
        at org.evosuite.runtime.EvoAssertions.assertThrownBy(EvoAssertions.java:111)
        at org.evosuite.runtime.EvoAssertions.verifyException(EvoAssertions.java:48)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test08(AbstractDiskHttpData_ESTest.java:239)
14) test50(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Expecting exception: NoSuchMethodError
        at org.junit.Assert.fail(Assert.java:89)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test50(AbstractDiskHttpData_ESTest.java:865)
15) test51(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: Expecting exception: NoSuchMethodError
        at org.junit.Assert.fail(Assert.java:89)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test51(AbstractDiskHttpData_ESTest.java:885)
16) test14(io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest)
java.lang.AssertionError: expected:<2> but was:<0>
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at io.netty.handler.codec.http.multipart.AbstractDiskHttpData_ESTest.test14(AbstractDiskHttpData_ESTest.java:322)

FAILURES!!!
Tests run: 68,  Failures: 16

Conclusion: As expected, all 3 tests fail under mvnc when the tests are run at JDK 11.