66 coverage differences occur despite identical test outcomes

[wtwhite]

After #43, we can get all 66 "interesting" coverage differences easily with a DB view. All but 3 are in Guava:

sqlite> select * from same_test_results_diff_coverage order by 1, 2, 3, 4, 5, 6, 7, 8, 9, 10;
┌───────────┬───────────┬────────────────────┬────────────────────────┬───────────────┬─────────────────────────────────────────────────────────────────┬────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┬────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ provider1 │ provider2 │      groupId       │       artifactId       │    version    │                            testClass                            │                package                 │                                               class                                               │               method               │                                                                               descriptor                                                                                │
├───────────┼───────────┼────────────────────┼────────────────────────┼───────────────┼─────────────────────────────────────────────────────────────────┼────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.collect.Sets_ESTest                           │ com/google/common/collect              │ com/google/common/collect/Sets                                                                    │ newHashSet                         │ (Ljava/lang/Iterable;)Ljava/util/HashSet;                                                                                                                               │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.reflect.ClassPath_ESTest                      │ com/google/common/collect              │ com/google/common/collect/ImmutableSet$Builder                                                    │ addAll                             │ (Ljava/util/Iterator;)Lcom/google/common/collect/ImmutableSet$Builder;                                                                                                  │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractCatchingFuture_ESTest │ com/google/common/base                 │ com/google/common/base/Preconditions                                                              │ checkState                         │ (ZLjava/lang/String;Ljava/lang/Object;)V                                                                                                                                │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractCatchingFuture_ESTest │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractCatchingFuture                                          │ afterDone                          │ ()V                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractCatchingFuture_ESTest │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ cancellationExceptionWithCause     │ (Ljava/lang/String;Ljava/lang/Throwable;)Ljava/util/concurrent/CancellationException;                                                                                   │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractCatchingFuture_ESTest │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ get                                │ ()Ljava/lang/Object;                                                                                                                                                    │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractCatchingFuture_ESTest │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ getDoneValue                       │ (Ljava/lang/Object;)Ljava/lang/Object;                                                                                                                                  │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractCatchingFuture_ESTest │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ maybePropagateCancellationTo       │ (Ljava/util/concurrent/Future;)V                                                                                                                                        │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractCatchingFuture_ESTest │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ setException                       │ (Ljava/lang/Throwable;)Z                                                                                                                                                │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractCatchingFuture_ESTest │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ tryInternalFastPathGetFailure      │ ()Ljava/lang/Throwable;                                                                                                                                                 │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractCatchingFuture_ESTest │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/Futures                                                         │ getDone                            │ (Ljava/util/concurrent/Future;)Ljava/lang/Object;                                                                                                                       │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractCatchingFuture_ESTest │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/Platform                                                        │ isInstanceOfThrowableClass         │ (Ljava/lang/Throwable;Ljava/lang/Class;)Z                                                                                                                               │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractCatchingFuture_ESTest │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/Uninterruptibles                                                │ getUninterruptibly                 │ (Ljava/util/concurrent/Future;)Ljava/lang/Object;                                                                                                                       │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractFuture_ESTest         │ com/google/common/base                 │ com/google/common/base/Preconditions                                                              │ checkState                         │ (ZLjava/lang/String;Ljava/lang/Object;)V                                                                                                                                │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractFuture_ESTest         │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/Futures                                                         │ getDone                            │ (Ljava/util/concurrent/Future;)Ljava/lang/Object;                                                                                                                       │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractFuture_ESTest         │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/InterruptibleTask                                               │ waitForInterrupt                   │ (Ljava/lang/Thread;)V                                                                                                                                                   │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractFuture_ESTest         │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/InterruptibleTask$Blocker                                       │ <init>                             │ (Lcom/google/common/util/concurrent/InterruptibleTask;)V                                                                                                                │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractFuture_ESTest         │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/InterruptibleTask$Blocker                                       │ setOwner                           │ (Ljava/lang/Thread;)V                                                                                                                                                   │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AbstractFuture_ESTest         │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/Uninterruptibles                                                │ getUninterruptibly                 │ (Ljava/util/concurrent/Future;)Ljava/lang/Object;                                                                                                                       │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/base                 │ com/google/common/base/Preconditions                                                              │ checkState                         │ (ZLjava/lang/String;Ljava/lang/Object;)V                                                                                                                                │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ get                                │ ()Ljava/lang/Object;                                                                                                                                                    │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ getDoneValue                       │ (Ljava/lang/Object;)Ljava/lang/Object;                                                                                                                                  │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AggregateFuture                                                 │ collectValueFromNonCancelledFuture │ (ILjava/util/concurrent/Future;)V                                                                                                                                       │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/CombinedFuture$CombinedFutureInterruptibleTask                  │ afterRanInterruptiblyFailure       │ (Ljava/lang/Throwable;)V                                                                                                                                                │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/FluentFuture$TrustedFuture                                      │ get                                │ ()Ljava/lang/Object;                                                                                                                                                    │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/Futures                                                         │ getDone                            │ (Ljava/util/concurrent/Future;)Ljava/lang/Object;                                                                                                                       │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/NullnessCasts                                                   │ uncheckedNull                      │ ()Ljava/lang/Object;                                                                                                                                                    │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/TrustedListenableFutureTask$TrustedFutureInterruptibleTask      │ afterRanInterruptiblySuccess       │ (Ljava/lang/Object;)V                                                                                                                                                   │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/Uninterruptibles                                                │ getUninterruptibly                 │ (Ljava/util/concurrent/Future;)Ljava/lang/Object;                                                                                                                       │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ afterDone                          │ ()V                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ clearListeners                     │ (Lcom/google/common/util/concurrent/AbstractFuture$Listener;)Lcom/google/common/util/concurrent/AbstractFuture$Listener;                                                │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ complete                           │ (Lcom/google/common/util/concurrent/AbstractFuture;)V                                                                                                                   │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ executeListener                    │ (Ljava/lang/Runnable;Ljava/util/concurrent/Executor;)V                                                                                                                  │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ getFutureValue                     │ (Lcom/google/common/util/concurrent/ListenableFuture;)Ljava/lang/Object;                                                                                                │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ getUninterruptibly                 │ (Ljava/util/concurrent/Future;)Ljava/lang/Object;                                                                                                                       │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ releaseWaiters                     │ ()V                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ setFuture                          │ (Lcom/google/common/util/concurrent/ListenableFuture;)Z                                                                                                                 │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture                                                  │ wasInterrupted                     │ ()Z                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture$UnsafeAtomicHelper                               │ casValue                           │ (Lcom/google/common/util/concurrent/AbstractFuture;Ljava/lang/Object;Ljava/lang/Object;)Z                                                                               │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractFuture$UnsafeAtomicHelper                               │ casWaiters                         │ (Lcom/google/common/util/concurrent/AbstractFuture;Lcom/google/common/util/concurrent/AbstractFuture$Waiter;Lcom/google/common/util/concurrent/AbstractFuture$Waiter;)Z │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/DirectExecutor                                                  │ execute                            │ (Ljava/lang/Runnable;)V                                                                                                                                                 │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/ExecutionSequencer$3                                            │ run                                │ ()V                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/Futures$NonCancellationPropagatingFuture                        │ afterDone                          │ ()V                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/Futures$NonCancellationPropagatingFuture                        │ run                                │ ()V                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/ImmediateFuture                                                 │ get                                │ ()Ljava/lang/Object;                                                                                                                                                    │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/ImmediateFuture                                                 │ isCancelled                        │ ()Z                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/ImmediateFuture                                                 │ isDone                             │ ()Z                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/SettableFuture                                                  │ setFuture                          │ (Lcom/google/common/util/concurrent/ListenableFuture;)Z                                                                                                                 │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/TrustedListenableFutureTask                                     │ afterDone                          │ ()V                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.ExecutionSequencer_ESTest     │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/TrustedListenableFutureTask$TrustedFutureInterruptibleAsyncTask │ afterRanInterruptiblyFailure       │ (Ljava/lang/Throwable;)V                                                                                                                                                │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.Futures_ESTest                │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/ExecutionList                                                   │ execute                            │ ()V                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.Futures_ESTest                │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/ExecutionList                                                   │ executeListener                    │ (Ljava/lang/Runnable;Ljava/util/concurrent/Executor;)V                                                                                                                  │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.Futures_ESTest                │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/ListenableFutureTask                                            │ done                               │ ()V                                                                                                                                                                     │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.Futures_ESTest                │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/TrustedListenableFutureTask$TrustedFutureInterruptibleTask      │ afterRanInterruptiblySuccess       │ (Ljava/lang/Object;)V                                                                                                                                                   │
│ gaoss     │ mvnc      │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.Futures_ESTest                │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/TrustedListenableFutureTask$TrustedFutureInterruptibleTask      │ runInterruptibly                   │ ()Ljava/lang/Object;                                                                                                                                                    │
│ gaoss     │ mvnc      │ io/netty           │ netty-buffer           │ 4.1.102.Final │ io.netty.buffer.AbstractByteBufAllocator_ESTest                 │ io/netty/buffer                        │ io/netty/buffer/PoolArena$DirectArena                                                             │ destroyChunk                       │ (Lio/netty/buffer/PoolChunk;)V                                                                                                                                          │
│ mvnc      │ gaoss     │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/base                 │ com/google/common/base/Preconditions                                                              │ checkNotNull                       │ (Ljava/lang/Object;Ljava/lang/String;Ljava/lang/Object;)Ljava/lang/Object;                                                                                              │
│ mvnc      │ gaoss     │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/base                 │ com/google/common/base/Strings                                                                    │ lenientFormat                      │ (Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/String;                                                                                                               │
│ mvnc      │ gaoss     │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/base                 │ com/google/common/base/Strings                                                                    │ lenientToString                    │ (Ljava/lang/Object;)Ljava/lang/String;                                                                                                                                  │
│ mvnc      │ gaoss     │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/CombinedFuture$AsyncCallableInterruptibleTask                   │ runInterruptibly                   │ ()Lcom/google/common/util/concurrent/ListenableFuture;                                                                                                                  │
│ mvnc      │ gaoss     │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/InterruptibleTask$Blocker                                       │ <init>                             │ (Lcom/google/common/util/concurrent/InterruptibleTask;)V                                                                                                                │
│ mvnc      │ gaoss     │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.AggregateFuture_ESTest        │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/InterruptibleTask$Blocker                                       │ setOwner                           │ (Ljava/lang/Thread;)V                                                                                                                                                   │
│ mvnc      │ gaoss     │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.Futures_ESTest                │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/AbstractCatchingFuture                                          │ afterDone                          │ ()V                                                                                                                                                                     │
│ mvnc      │ gaoss     │ com/google/guava   │ guava                  │ 31.0.1-jre    │ com.google.common.util.concurrent.Futures_ESTest                │ com/google/common/util/concurrent      │ com/google/common/util/concurrent/TrustedListenableFutureTask$TrustedFutureInterruptibleTask      │ afterRanInterruptiblyFailure       │ (Ljava/lang/Throwable;)V                                                                                                                                                │
│ mvnc      │ gaoss     │ io/undertow        │ undertow-servlet       │ 2.2.28.Final  │ io.undertow.servlet.spec.UpgradeServletOutputStream_ESTest      │ io/undertow/servlet/spec               │ io/undertow/servlet/spec/UpgradeServletOutputStream$3                                             │ run                                │ ()V                                                                                                                                                                     │
│ mvnc      │ gaoss     │ org/apache/commons │ commons-configuration2 │ 2.7           │ org.apache.commons.configuration2.DataConfiguration_ESTest      │ org/apache/commons/configuration2/tree │ org/apache/commons/configuration2/tree/NodeSelector                                               │ equals                             │ (Ljava/lang/Object;)Z                                                                                                                                                   │
└───────────┴───────────┴────────────────────┴────────────────────────┴───────────────┴─────────────────────────────────────────────────────────────────┴────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Investigate.

[wtwhite]

Explanation of DB views

The above results are from the view same_test_results_diff_coverage, defined here. It uses 2 other views (that are useful in their own right):

• differing_coverage_methods has a row for each (provider1, provider2, <generated-test-class>, <method-in-class-under-test>) tuple in which some provider1 test in <generated-test-class> hits <method-in-class-under-test> but none of provider2's tests in <generated-test-class> do; <generated-test-class> here is actually represented as several other DB columns that together identify an EvoSuite-generated test class, and <method-in-class-under-test> here is actually represented as several other DB columns that together identify a (JaCoCo-instrumented) method in the class under test.
• differing_failed_test_methods has a row for each (provider1, provider2, <generated-test-class>, testMethodSuffix) tuple in which the testNN() method with numeric suffix testMethodSuffix in <generated-test-class> fails when run against provider1 but not when run against provider2.

same_test_results_diff_coverage takes differing_coverage_methods and filters out any rows with <generated-test-class>es that had one or more test methods with different results between the two given providers (these differing test results could appear in differing_failed_test_methods with providers in either "orientation").

Note: The above reflects the fact that coverage is currently "merged" across all test methods in a generated test class. (This is normally what you want -- to see whether all your tests together achieve good coverage.) We could increase "resolution" by running each test method separately and measuring its coverage. @jensdietrich WDYT?

[jensdietrich]

I have setup guava-31.0.1 for debugging in coverage-debug.

I have then tried to verify the first two rows in above table in the debugger.

com.google.common.collect.Sets_ESTest invokes com/google/common/collect/Sets::newHashSet(Ljava/lang/Iterable;)Ljava/util/HashSet;

for both projects Scenario1GAOSS and Scenario1MVNC:

1. open project in IntelliJ, set JDK to 1.8
2. place breakpoint in com/google/common/collect/Sets::newHashSet(Ljava/lang/Iterable;)Ljava/util/HashSet;
3. run com.google.common.collect.Sets_ESTest in debug mode

Result: for both projects, the debugger stops at the breakpoint, i.e. in both cases the invocation is observed. The context is the same for both and can be easily verified: a direct invocation from test05.

com.google.common.reflect.ClassPath_ESTest invokes com/google/common/collect/ImmutableSet$Builder::addAll(Ljava/util/Iterator;)Lcom/google/common/collect/ImmutableSet$Builder;

same process as above. Result: the execution does not stop at the breakpoint for either scenario.

So in summary I cannot confirm that the coverage is different in those cases.

@wtwhite - please have a look and perhaps debug through some records for a different GAV.

[wtwhite]

Thanks @jensdietrich, I'll try to repro this Wednesday morning. Tim

…

________________________________ From: Jens Dietrich ***@***.***> Sent: Monday, 18 November 2024 4:11 pm To: binaryeq/regression-test-generation ***@***.***> Cc: wtwhite ***@***.***>; Mention ***@***.***> Subject: Re: [binaryeq/regression-test-generation] 66 coverage differences occur despite identical test outcomes (Issue #44) I have setup guava-31.0.1 for debugging in coverage-debug<https://github.com/binaryeq/regression-test-generation/tree/main/coverage-debug>. I have then tried to verify the first two rows in above table in the debugger. com.google.common.collect.Sets_ESTest invokes com/google/common/collect/Sets::newHashSet(Ljava/lang/Iterable;)Ljava/util/HashSet; for both projects Scenario1GAOSS<https://github.com/binaryeq/regression-test-generation/tree/main/coverage-debug/Scenario1GAOSS> and Scenario1MVNC<https://github.com/binaryeq/regression-test-generation/tree/main/coverage-debug/Scenario1MVNC>: 1. open project in IntelliJ, set JDK to 1.8 2. place breakpoint in com/google/common/collect/Sets::newHashSet(Ljava/lang/Iterable;)Ljava/util/HashSet; 3. run com.google.common.collect.Sets_ESTest in debug mode Result: for both projects, the debugger stops at the breakpoint, i.e. in both cases the invocation is observed. The context is the same for both and can be easily verified: a direct invocation from test05. com.google.common.reflect.ClassPath_ESTest invokes com/google/common/collect/ImmutableSet$Builder::addAll(Ljava/util/Iterator;)Lcom/google/common/collect/ImmutableSet$Builder; same process as above. Result: the execution does not stop at the breakpoint for either scenario. So in summary I cannot confirm that the coverage is different in those cases. @wtwhite<https://github.com/wtwhite> - please have a look and perhaps debug through some records for a different GAV. — Reply to this email directly, view it on GitHub<#44 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACXVNL3DXJJ7FYPY7Z67O432BFLHFAVCNFSM6AAAAABR2JBBA2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBRHA2TINJXGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[wtwhite]

com.google.common.collect.Sets_ESTest

Per the table at the top, during the execution of tests in this class, gaoss invokes com/google/common/collect/Sets::newHashSet(Ljava/lang/Iterable;)Ljava/util/HashSet; while mvnc does not.

Intellij debugger

I could repro @jensdietrich's results in the Intellij debugger: Both mvnc and gaoss call this method. This is not surprising since as he noted, test05() in the Sets_ESTest test class explicitly calls this version of newHashSet() at top level:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ cat testgen/mvnc/com/google/guava/guava/31.0.1-jre/evosuite-tests/com/google/common/collect/Sets_ESTest.java
--snip--
  @Test(timeout = 4000)
  public void test05()  throws Throwable  {
      // Undeclared exception!
      try { 
        Sets.newHashSet((Iterable<? extends Integer>) null);
        fail("Expecting exception: NullPointerException");
      
      } catch(NullPointerException e) {
         //
         // no message in exception (getMessage() returned null)
         //
         verifyException("com.google.common.collect.Sets", e);
      }
  }
--snip--

JaCoCo XML reports

For gaoss, JaCoCo reports that the newHashSet() method that takes an Iterable was called (as expected):

--snip--
    <class name="com/google/common/collect/Sets" sourcefilename="Sets.java">
--snip--
      <method name="newHashSet" desc="()Ljava/util/HashSet;" line="181">
        <counter type="INSTRUCTION" missed="4" covered="0"/>
        <counter type="LINE" missed="1" covered="0"/>
        <counter type="COMPLEXITY" missed="1" covered="0"/>
        <counter type="METHOD" missed="1" covered="0"/>
      </method>
      <method name="newHashSet" desc="([Ljava/lang/Object;)Ljava/util/HashSet;" line="198">
        <counter type="INSTRUCTION" missed="10" covered="0"/>
        <counter type="LINE" missed="3" covered="0"/>
        <counter type="COMPLEXITY" missed="1" covered="0"/>
        <counter type="METHOD" missed="1" covered="0"/>
      </method>
      <method name="newHashSet" desc="(Ljava/lang/Iterable;)Ljava/util/HashSet;" line="222">
        <counter type="INSTRUCTION" missed="9" covered="4"/>
        <counter type="BRANCH" missed="1" covered="1"/>
        <counter type="LINE" missed="1" covered="1"/>
        <counter type="COMPLEXITY" missed="1" covered="1"/>
        <counter type="METHOD" missed="0" covered="1"/>
      </method>
      <method name="newHashSet" desc="(Ljava/util/Iterator;)Ljava/util/HashSet;" line="240">
        <counter type="INSTRUCTION" missed="8" covered="0"/>
        <counter type="LINE" missed="3" covered="0"/>
        <counter type="COMPLEXITY" missed="1" covered="0"/>
        <counter type="METHOD" missed="1" covered="0"/>
      </method>
--snip--

However, for mvnc, JaCoCo reports that no Sets.newHashSet() method calls were made:

--snip--
    <class name="com/google/common/collect/Sets" sourcefilename="Sets.java">
--snip--
      <method name="newHashSet" desc="()Ljava/util/HashSet;" line="181">
        <counter type="INSTRUCTION" missed="4" covered="0"/>
        <counter type="LINE" missed="1" covered="0"/>
        <counter type="COMPLEXITY" missed="1" covered="0"/>
        <counter type="METHOD" missed="1" covered="0"/>
      </method>
      <method name="newHashSet" desc="([Ljava/lang/Object;)Ljava/util/HashSet;" line="198">
        <counter type="INSTRUCTION" missed="10" covered="0"/>
        <counter type="LINE" missed="3" covered="0"/>
        <counter type="COMPLEXITY" missed="1" covered="0"/>
        <counter type="METHOD" missed="1" covered="0"/>
      </method>
      <method name="newHashSet" desc="(Ljava/lang/Iterable;)Ljava/util/HashSet;" line="222">
        <counter type="INSTRUCTION" missed="13" covered="0"/>
        <counter type="BRANCH" missed="2" covered="0"/>
        <counter type="LINE" missed="3" covered="0"/>
        <counter type="COMPLEXITY" missed="2" covered="0"/>
        <counter type="METHOD" missed="1" covered="0"/>
      </method>
      <method name="newHashSet" desc="(Ljava/util/Iterator;)Ljava/util/HashSet;" line="240">
        <counter type="INSTRUCTION" missed="8" covered="0"/>
        <counter type="LINE" missed="3" covered="0"/>
        <counter type="COMPLEXITY" missed="1" covered="0"/>
        <counter type="METHOD" missed="1" covered="0"/>
      </method>
--snip--

So the problem seems to be in JaCoCo's instrumentation, rather than in loading the raw JaCoCo results into the DB, or the query to extract them (as I'd thought).

Since test05() passes on both mvnc and gaoss, the expected NPE is indeed triggered in both. Per the JaCoCo FAQ, coverage can be missed when exceptions are thrown. The most likely explanation is therefore that JaCoCo's probe was placed slightly later in the instrumented mvnc Sets.class, thus not triggering before the NPE occurred.

[wtwhite]

TL;DR: JaCoCo indeed changes the location of its coverage-recording probe between mvnc and gaoss (despite identical original bytecode).

The disassembly of the JaCoCo-fied version of the mvnc newHashSet() method:

  public static <E extends java.lang.Object> java.util.HashSet<E> newHashSet(java.lang.Iterable<? extends E>);
    descriptor: (Ljava/lang/Iterable;)Ljava/util/HashSet;
    flags: (0x0009) ACC_PUBLIC, ACC_STATIC
    Code:
      stack=6, locals=3, args_size=1
         0: invokestatic  #603                // Method $jacocoInit:()[Z
         3: astore_2
         4: aload_0
         5: instanceof    #5                  // class java/util/Collection
         8: ifeq          35
        11: aload_2
        12: bipush        19
        14: iconst_1
        15: bastore
        16: new           #18                 // class java/util/HashSet
        19: dup
        20: aload_0
        21: checkcast     #5                  // class java/util/Collection
        24: invokespecial #22                 // Method java/util/HashSet."<init>":(Ljava/util/Collection;)V
        27: aload_2
        28: bipush        20
        30: iconst_1
        31: bastore
        32: goto          49
        35: aload_0
        36: invokeinterface #9,  1            // InterfaceMethod java/lang/Iterable.iterator:()Ljava/util/Iterator;
        41: invokestatic  #23                 // Method newHashSet:(Ljava/util/Iterator;)Ljava/util/HashSet;
        44: aload_2
        45: bipush        21
        47: iconst_1
        48: bastore
        49: aload_2
        50: bipush        22
        52: iconst_1
        53: bastore
        54: areturn

The null argument causes the instanceof at offset 5 to push 0, causing the following ifeq to branch to offset 35. The invokeinterface at offset 36 triggers NPE, so the bastores at offsets 48 and 53 are never called. The latter store occurs on all non-exceptional paths through the function, so must be JaCoCo's method-level coverage flag.

The corresponding disassembly for gaoss:

  public static <E extends java.lang.Object> java.util.HashSet<E> newHashSet(java.lang.Iterable<? extends E>);
    descriptor: (Ljava/lang/Iterable;)Ljava/util/HashSet;
    flags: (0x0009) ACC_PUBLIC, ACC_STATIC
    Code:
      stack=6, locals=3, args_size=1
         0: invokestatic  #612                // Method $jacocoInit:()[Z
         3: astore_2
         4: aload_0
         5: instanceof    #179                // class java/util/Collection
         8: ifeq          30
        11: new           #205                // class java/util/HashSet
        14: dup
        15: aload_0
        16: checkcast     #179                // class java/util/Collection
        19: invokespecial #22                 // Method java/util/HashSet."<init>":(Ljava/util/Collection;)V
        22: aload_2
        23: bipush        19
        25: iconst_1
        26: bastore
        27: goto          49
        30: aload_0
        31: aload_2
        32: bipush        20
        34: iconst_1
        35: bastore
        36: invokeinterface #9,  1            // InterfaceMethod java/lang/Iterable.iterator:()Ljava/util/Iterator;
        41: invokestatic  #23                 // Method newHashSet:(Ljava/util/Iterator;)Ljava/util/HashSet;
        44: aload_2
        45: bipush        21
        47: iconst_1
        48: bastore
        49: aload_2
        50: bipush        22
        52: iconst_1
        53: bastore
        54: areturn

This time, the null argument causes a branch from offset 8 to offset 30, and the bastore at offset 35 occurs before the NPE-triggering invokeinterface that follows, so the method coverage is recorded.

It's slightly surprising that JaCoCo reversed the placement of its probe, since the original (pre-JaCoCo) bytecode instructions are in fact identical between mvnc and gaoss (up to constant pool reordering and mvnc having 1 additional line number in its LineNumberTable, not shown):

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/coverage-debug/Scenario1GAOSS$ cat newHashSet.original.javap 
  public static <E extends java.lang.Object> java.util.HashSet<E> newHashSet(java.lang.Iterable<? extends E>);
    descriptor: (Ljava/lang/Iterable;)Ljava/util/HashSet;
    flags: (0x0009) ACC_PUBLIC, ACC_STATIC
    Code:
      stack=3, locals=1, args_size=1
         0: aload_0
         1: instanceof    #5                  // class java/util/Collection
         4: ifeq          21
         7: new           #18                 // class java/util/HashSet
        10: dup
        11: aload_0
        12: checkcast     #5                  // class java/util/Collection
        15: invokespecial #22                 // Method java/util/HashSet."<init>":(Ljava/util/Collection;)V
        18: goto          30
        21: aload_0
        22: invokeinterface #9,  1            // InterfaceMethod java/lang/Iterable.iterator:()Ljava/util/Iterator;
        27: invokestatic  #23                 // Method newHashSet:(Ljava/util/Iterator;)Ljava/util/HashSet;
        30: areturn
--snip--

[wtwhite]

io.netty.buffer.AbstractByteBufAllocator_ESTest

Row 56 in the table at the top indicates that gaoss invokes io.netty.buffer.PoolArena$DirectArena.destroyChunk() during this test while mvnc does not. Looking to repro this in the Intellij debugger by setting a breakpoint on the first line of this method.

Maven projects

• mvnc
• gaoss

I opened each in Intellij, and changed the JDK to 1.8 in "Project Structure" to debug it.

In mvnc

Side quest: Intellij enables assertions, causing spurious test failures

test06() fails reliably with:

java.lang.AssertionError
	at io.netty.buffer.CompositeByteBuf.shiftComps(CompositeByteBuf.java:2341)
	at io.netty.buffer.CompositeByteBuf.addComponents0(CompositeByteBuf.java:392)
	at io.netty.buffer.CompositeByteBuf.<init>(CompositeByteBuf.java:89)
	at io.netty.buffer.AbstractByteBufAllocator_ESTest.test06(AbstractByteBufAllocator_ESTest.java:83)
--snip--

This must be caused by assert i >= 0 && i <= size && count > 0; (in Intellij's decompilation of shiftComps()) as that's the only assert in the method.

This seems to be a consequence of Intellij debugging turning on -ea, while EvoSuite did not have it turned on during test generation, and nor do the usual JUnit4 runners during batch processing.

This assertion failure is not surprising, given that the test calls a package-private constructor with an offset (last parameter) that exceeds the size of the ByteBuf passed in:

      ByteBuf[] byteBufArray0 = new ByteBuf[0];
      CompositeByteBuf compositeByteBuf0 = new CompositeByteBuf(pooledByteBufAllocator0, true, 41, byteBufArray0, 41);

Removing -ea from the java command line in the Intellij run configuration causes test06() to reliably pass.

Continuing with assertions off

PoolArena$DirectArena.destroyChunk() is called from a finalizer thread:

destroyChunk:715, PoolArena$DirectArena (io.netty.buffer)
destroy:257, PoolChunkList (io.netty.buffer)
destroyPoolChunkLists:619, PoolArena (io.netty.buffer)
finalize:607, PoolArena (io.netty.buffer)
invokeFinalize:1285, System$2 (java.lang)
runFinalizer:102, Finalizer (java.lang.ref)
access$100:34, Finalizer (java.lang.ref)
run:189, Finalizer$FinalizerThread (java.lang.ref)

Another thread stack shows that test00() triggered it:

<init>:145, SizeClasses (io.netty.buffer)
<init>:318, PooledByteBufAllocator (io.netty.buffer)
<init>:225, PooledByteBufAllocator (io.netty.buffer)
<init>:214, PooledByteBufAllocator (io.netty.buffer)
<init>:200, PooledByteBufAllocator (io.netty.buffer)
<init>:195, PooledByteBufAllocator (io.netty.buffer)
test00:29, AbstractByteBufAllocator_ESTest (io.netty.buffer)
--snip--

However:

• Rerunning hit the breakpoint again in a finalizer thread, but this time there was no thread corresponding to a test.
• Rerunning the individual test00() method 10 more times did not hit the breakpoint
• Rerunning the entire test class 10 times hit the breakpoint 6 times, each time on a finalizer thread, with the console pane showing test00 "paused", but test00() not in any thread's stack frame

In gaoss

The situation is already clear, but for completeness, I also tested gaoss in Intellij with assertions off:

• Running the individual test00() test method 10 times did not hit the breakpoint
• Running the entire test class 10 times hit the breakpoint 4 times, each time on a finalizer thread, with the console pane showing either test22 (once) or test00 (3 times) "paused", but that method was not in any thread's stack frame

Conclusion: Whether or not mvnc calls destroyChunk() during io.netty.buffer.AbstractByteBufAllocator_ESTest is flaky, depending on how far a finalizer thread gets before JVM termination. Accordingly row 56 is spurious.

[wtwhite]

org.apache.commons.configuration2.DataConfiguration_ESTest

Row 66 in the table at the top indicates that mvnc invokes org.apache.commons.configuration2.tree.NodeSelector.equals() during this test, while gaoss does not. Looking to repro this in the Intellij debugger by setting a breakpoint on the first line of this method.

Maven projects

• mvnc
• gaoss

I opened each in Intellij, changed the JDK to 1.8 in "Project Structure" to debug it, and disabled assertions.

In mvnc

• Ran whole test class 10 times, breakpoint hit 1 time (run 4) in finalizer thread: Flaky

In gaoss

• Ran whole test class 10 times, breakpoint hit 3 times in finalizer thread: Flaky

Conclusion: Another instance of flaky coverage due to the finalizer thread racing with the other threads. Disregard.

[wtwhite]

io.undertow.servlet.spec.UpgradeServletOutputStream_ESTest

Row 65 in the table at the top indicates that mvnc invokes io.undertow.servlet.spec.UpgradeServletOutputStream$3.run() during this test, while gaoss does not. Looking to repro this in the Intellij debugger by setting a breakpoint on the first line of this method.

Maven projects

• mvnc
• gaoss

I opened each in Intellij, changed the JDK to 1.8 in "Project Structure" to debug it, and disabled assertions.

In mvnc

Placing the breakpoint is complicated by the fact that UpgradeServletOutputStream$3 is one of 3 functionally identical lambda functions, all of which simply call StreamSinkChannel.resumeWrites(), which can't be distinguished in the decompiled code. Looking at the disassembly shows the only creation of a UpgradeServletOutputStream$3 object is in the top-level class's setWriteListener() method, so I placed the breakpoint there (i.e., here).

Debugging hit the breakpoint inside test21(), which directly calls setWriteListener():

  @Test(timeout = 4000)
  public void test21()  throws Throwable  {
      NullStreamSinkChannel nullStreamSinkChannel0 = new NullStreamSinkChannel((XnioIoThread) null);
      ThreadPoolExecutor.DiscardPolicy threadPoolExecutor_DiscardPolicy0 = new ThreadPoolExecutor.DiscardPolicy();
      ScheduledThreadPoolExecutor scheduledThreadPoolExecutor0 = new ScheduledThreadPoolExecutor(0, threadPoolExecutor_DiscardPolicy0);
      UpgradeServletOutputStream upgradeServletOutputStream0 = new UpgradeServletOutputStream(nullStreamSinkChannel0, scheduledThreadPoolExecutor0);
      WriteListener writeListener0 = mock(WriteListener.class, new ViolatedAssumptionAnswer());
      upgradeServletOutputStream0.setWriteListener(writeListener0);
      upgradeServletOutputStream0.flush();
  }

In fact, multiple tests (those numbered 10, 11, 16, 17, 19, 20, 21, 24) directly call this method. I tried debugging each test method separately:

• Tests 16, 19, 20, 21 and 24 hit the breakpoint
• Other tests do not, despite direct top-level calls in some cases -- e.g., test10() does not, because an NPE is reliably triggered beforehand (which the test expects)

Hypothesis: mvnc and gaoss compiled lambdas in a different order.

In gaoss

My hypothesis was wrong: setWriteListener still instantiates and calls run() on UpgradeServletOutputStream$3 in gaoss, just as in mvnc:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ javap -c -p -v -cp /home/wtwhite/code/craw-redhat-oss/wget/crawl/google-aoss/io/undertow/undertow-servlet/2.2.28.Final/undertow-servlet-2.2.28.Final.jar io.undertow.servlet.spec.UpgradeServletOutputStream
--snip--
  public void setWriteListener(javax.servlet.WriteListener);
    descriptor: (Ljavax/servlet/WriteListener;)V
    flags: (0x0001) ACC_PUBLIC
    Code:
      stack=5, locals=2, args_size=2
         0: aload_1
--snip--
        73: new           #44                 // class io/undertow/servlet/spec/UpgradeServletOutputStream$3
        76: dup
        77: aload_0
        78: invokespecial #45                 // Method io/undertow/servlet/spec/UpgradeServletOutputStream$3."<init>":(Lio/undertow/servlet/spec/UpgradeServletOutputStream;)V
        81: invokeinterface #27,  2           // InterfaceMethod java/util/concurrent/Executor.execute:(Ljava/lang/Runnable;)V
        86: return
--snip--

Debugging hit the breakpoint inside test21(), just as it did for mvnc. So how come JaCoCo reported that the method was not hit in gaoss? test21() does not trigger any exceptions, so the rationale from Sets_ESTest (JaCoCo probes being skipped on exceptional control flows) does not apply here.

I checked the JaCoCo XML report for gaoss, and indeed it shows that, although the constructor was called, run() was not:

--snip--
    <class name="io/undertow/servlet/spec/UpgradeServletOutputStream$3" sourcefilename="UpgradeServletOutputStream.java">
      <method name="&lt;init&gt;" desc="(Lio/undertow/servlet/spec/UpgradeServletOutputStream;)V" line="188">
        <counter type="INSTRUCTION" missed="0" covered="6"/>
        <counter type="LINE" missed="0" covered="1"/>
        <counter type="COMPLEXITY" missed="0" covered="1"/>
        <counter type="METHOD" missed="0" covered="1"/>
      </method>
      <method name="run" desc="()V" line="191">
        <counter type="INSTRUCTION" missed="5" covered="0"/>
        <counter type="LINE" missed="2" covered="0"/>
        <counter type="COMPLEXITY" missed="1" covered="0"/>
        <counter type="METHOD" missed="1" covered="0"/>
      </method>
--snip--

However, puzzlingly, the same "missing coverage" is true for mvnc:

--snip--
    <class name="io/undertow/servlet/spec/UpgradeServletOutputStream$3" sourcefilename="UpgradeServletOutputStream.java">
      <method name="&lt;init&gt;" desc="(Lio/undertow/servlet/spec/UpgradeServletOutputStream;)V" line="188">
        <counter type="INSTRUCTION" missed="0" covered="6"/>
        <counter type="LINE" missed="0" covered="1"/>
        <counter type="COMPLEXITY" missed="0" covered="1"/>
        <counter type="METHOD" missed="0" covered="1"/>
      </method>
      <method name="run" desc="()V" line="191">
        <counter type="INSTRUCTION" missed="5" covered="0"/>
        <counter type="LINE" missed="2" covered="0"/>
        <counter type="COMPLEXITY" missed="1" covered="0"/>
        <counter type="METHOD" missed="1" covered="0"/>
      </method>
--snip--

This is wrong in 2 ways: (1) The run() method definitely is hit in mvnc; (2) If JaCoCo reports that it was not hit there, then there should therefore be no entry in the table at the top.

[wtwhite]

Mystery (2) solved -- the XML snippets in the previous comment came from the wrong version (2.2.25.Final, instead of 2.2.28.Final). Looking at the right versions shows that mvnc did hit the run() method while gaoss did not:

mvnc:

--snip--
    <class name="io/undertow/servlet/spec/UpgradeServletOutputStream$3" sourcefilename="UpgradeServletOutputStream.java">
      <method name="&lt;init&gt;" desc="(Lio/undertow/servlet/spec/UpgradeServletOutputStream;)V" line="188">
        <counter type="INSTRUCTION" missed="0" covered="6"/>
        <counter type="LINE" missed="0" covered="1"/>
        <counter type="COMPLEXITY" missed="0" covered="1"/>
        <counter type="METHOD" missed="0" covered="1"/>
      </method>
      <method name="run" desc="()V" line="191">
        <counter type="INSTRUCTION" missed="0" covered="5"/>
        <counter type="LINE" missed="0" covered="2"/>
        <counter type="COMPLEXITY" missed="0" covered="1"/>
        <counter type="METHOD" missed="0" covered="1"/>
      </method>
--snip--

gaoss:

--snip--
    <class name="io/undertow/servlet/spec/UpgradeServletOutputStream$3" sourcefilename="UpgradeServletOutputStream.java">
      <method name="&lt;init&gt;" desc="(Lio/undertow/servlet/spec/UpgradeServletOutputStream;)V" line="188">
        <counter type="INSTRUCTION" missed="0" covered="6"/>
        <counter type="LINE" missed="0" covered="1"/>
        <counter type="COMPLEXITY" missed="0" covered="1"/>
        <counter type="METHOD" missed="0" covered="1"/>
      </method>
      <method name="run" desc="()V" line="191">
        <counter type="INSTRUCTION" missed="5" covered="0"/>
        <counter type="LINE" missed="2" covered="0"/>
        <counter type="COMPLEXITY" missed="1" covered="0"/>
        <counter type="METHOD" missed="1" covered="0"/>
      </method>
--snip--

However, the mystery of why run() is not hit in gaoss remains.

[wtwhite]

I looked at the disassembly of the actual Jacoco-instrumented class-under-test, but that is just as clear -- setWriteListener() creates a UpgradeServletOutputStream$3:

wtwhite@wtwhite-vuw-vm:~/code/regression-test-generation/find-diff-classes$ javap -c -p -v -cp /home/wtwhite/code/craw-redhat-oss/wget/crawl/google-aoss/io/undertow/undertow-servlet/2.2.28.Final/jacocofied/undertow-servlet-2.2.28.Final.jar io.undertow.servlet.spec.UpgradeServletOutputStream
--snip--
  public void setWriteListener(javax.servlet.WriteListener);
    descriptor: (Ljavax/servlet/WriteListener;)V
    flags: (0x0001) ACC_PUBLIC
    Code:
      stack=8, locals=4, args_size=2
         0: invokestatic  #222                // Method $jacocoInit:()[Z
--snip--
       107: new           #44                 // class io/undertow/servlet/spec/UpgradeServletOutputStream$3
       110: dup
       111: aload_0
       112: invokespecial #45                 // Method io/undertow/servlet/spec/UpgradeServletOutputStream$3."<init>":(Lio/undertow/servlet/spec/UpgradeServletOutputStream;)V
       115: invokeinterface #27,  2           // InterfaceMethod java/util/concurrent/Executor.execute:(Ljava/lang/Runnable;)V
--snip--

But it doesn't directly call run() on it -- it only passes it to Executor.execute(), which will call run() at some point in the future, possibly on a different thread.

Conclusion: The call to run() races with the main thread, leading to flaky coverage which we observed here.