Shrinker performance benchmark

[jmid]

This PR adds a shrinker benchmark.
It iterates the (factored) shrinker tests in QCheck_tests.ml and QCheck2_tests.ml in pairs.
For each such pair of tests we measure

• time in seconds, denoted Q1/s and Q2/s
• the number of succesful shrinks, denoted #succ
• the number of shrink attempts, denoted #att

This is iterated across several seeds to make sure observations hold across more than one random run.
Finally we add up the timings to spot general trends.

Overall, the benchmark gives a reasonable idea of the state of affairs (see output below):

• the QCheck list shrinker tries way too many candidates, often making it unbearably inefficient on large lists, e.g, spending 9.8 secs over 5 milion shrink attempts of which only 1.696 are successful(!)
• the QCheck string shrinker does not perform as well as the QCheck2 shrinker, e.g., spending 1.342 seconds on 4465 shrink-steps when QCheck2 gets away with only 24 in 0.002 secs
• the QCheck2 list shrinker could also use a hand, e.g., spending 17.010 secs (twice of QCheck) on lists shorter than 4332
• the function shrinkers in both QCheck and QCheck2 suffer from poor list-shrinking and the involved domain+range shrinkers, e.g., QCheck2 spending 15.278 secs on fold_left fold_right uncurried in one iteration, whereas QCheck spends 28.631 secs on only 125 successful shrink attempts out of 27750 on fold_left test, fun first in another iteration.

The benchmark starts to drill the framework:

• On one seed I had the segfault reported in Segfault on fac test with OCaml 4.12.0 #175
• I've also had to cherry-pick random seeds that don't make the benchmark take hours on my laptop.

I've therefore drafted (QCheck) performance improvements that will make such hacks unnecessary. I'll commit them separately.

                                                         iteration seed 1234                   iteration seed 8743                   iteration seed 6789               total
Shrink test name                                  Q1/s  #succ/#att   Q2/s  #succ/#att   Q1/s  #succ/#att   Q2/s  #succ/#att   Q1/s  #succ/#att   Q2/s  #succ/#att    Q1/s   Q2/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
big bound issue59                                - skipped as generator is stateful, making it non-repeatable
long_shrink                                       0.051  149/351     1.700 3039/3099    0.010  148/349     1.064 3068/3127    0.013  146/345     1.172 3063/3124    0.074  3.937
ints arent 0 mod 3                                0.000   84/216     0.000    2/2       0.000   71/204     0.000    1/1       0.000   88/215     0.000   88/305     0.000  0.000
ints are 0                                        0.000   62/63      0.000   61/123     0.000   61/62      0.000   61/122     0.000   62/63      0.000   61/123     0.000  0.000
ints < 209609                                    - skipped as generator is stateful, making it non-repeatable
nat < 5001                                        0.000    6/56      0.000    7/77      0.000    6/47      0.000    7/69      0.000    3/24      0.000    8/85      0.000  0.000
char is never produces 'abcdef'                   0.000    0/0       0.000    1/1       0.000    0/0       0.000    0/0       0.000    0/0       0.000    0/0       0.000  0.000
strings are empty                                 0.000  249/250     0.000    8/16      0.050 4466/4467    0.002   13/26      0.000    0/1       0.000    1/2       0.051  0.002
string never has a \000 char                      0.000   25/40      0.000   22/167     0.159 4466/4519    0.003   56/254     0.000   15/18      0.000   15/48      0.159  0.003
string never has a \255 char                      0.001  249/316     0.001   59/318     0.163 4466/4520    0.005   97/529     0.645 9260/9365    0.003   41/194     0.808  0.009
strings have unique chars                         0.003  248/269     0.000   18/30      1.342 4465/4536    0.002   24/52      0.000   14/34      0.000   15/20      1.345  0.002
pairs have different components                   0.000    0/4       0.000    0/6       0.000    0/4       0.000    0/6       0.000    0/6       0.000    0/10      0.000  0.000
pairs have same components                        0.000  125/126     0.000   63/125     0.000  124/125     0.000   62/123     0.000  119/120     0.000   63/125     0.000  0.000
pairs have a zero component                       0.000  124/188     0.000  122/306     0.000  123/186     0.000  122/306     0.000  118/182     0.000  123/308     0.000  0.000
pairs are (0,0)                                   0.000  125/126     0.000   63/125     0.000  124/125     0.000   62/123     0.000  119/120     0.000   63/125     0.000  0.000
pairs are ordered                                 0.000  827/17626   0.000   94/1217    0.000  690/12946   0.000   85/865     0.000  687/13963   0.000   94/1326    0.001  0.001
pairs are ordered reversely                       0.000  124/125     0.000   62/124     0.000  123/124     0.000   62/124     0.000  122/123     0.000   62/124     0.000  0.000
pairs sum to less than 128                        0.000  116/129     0.000   56/126     0.000  120/146     0.000   59/138     0.000  119/141     0.000   57/130     0.000  0.000
pairs lists rev concat                            0.032  140/332     0.016   83/168     0.017  137/335     0.005   75/152     0.000  130/318     0.000   67/136     0.049  0.021
pairs lists no overlap                            0.001   22/47      0.006   27/60      0.000   17/41      0.008   18/41      0.000    6/20      0.000   11/28      0.001  0.014
triples have pair-wise different components       0.000    7/31      0.000    3/15      0.000    6/6       0.000    3/3       0.000    2/6       0.000    3/3       0.000  0.000
triples have same components                      0.000  188/252     0.000   64/127     0.000  177/240     0.000   64/128     0.000  182/246     0.000   62/122     0.000  0.000
triples are ordered                               0.000  188/252     0.000    3/4       0.000  177/178     0.000    3/4       0.000  187/250     0.003   91/1021    0.000  0.003
triples are ordered reversely                     0.000  188/189     0.000   64/126     0.000  177/240     0.000  124/247     0.000  182/183     0.000   65/127     0.000  0.000
quadruples have pair-wise different components    0.000   23/41      0.000    4/4       0.000   11/11      0.000    4/4       0.000   14/38      0.000    4/11      0.000  0.000
quadruples have same components                   0.000  250/377     0.000  126/313     0.000  237/424     0.000  115/292     0.000  242/425     0.000  123/307     0.000  0.001
quadruples are ordered                            0.000  251/315     0.000    5/6       0.000  239/240     0.000    4/5       0.000  244/308     0.000    5/6       0.000  0.000
quadruples are ordered reversely                  0.000  251/252     0.000   66/128     0.000  239/302     0.000  126/250     0.000  244/245     0.000   66/128     0.000  0.000
bind ordered pairs                                0.000  125/125     0.000    1/1       0.000  124/124     0.000    1/1       0.000  120/120     0.000    1/1       0.000  0.000
bind list_size constant                           0.000   50/358     0.000   12/26      0.000   48/338     0.000   12/25      0.000   48/342     0.000   11/21      0.000  0.001
lists are empty                                   0.000   11/16      0.000    8/16      0.000   19/27      0.004   13/26      0.000    4/9       0.000    1/2       0.000  0.004
lists shorter than 10                             0.000   50/1198    0.000   16/30      0.001   71/1637    0.004   21/42      0.000   36/868     0.000   15/29      0.001  0.004
lists shorter than 432                            9.842 1696/5118102  1.213  412/457     9.516 1612/4863421  1.183  405/450     9.506 1667/5037661  0.188  419/447    28.863  2.585
lists shorter than 4332                           3.785   13/190735  6.262 4022/4087    2.496   11/126052  6.124 4020/4067    2.378    7/126607  4.623 4013/4055    8.659 17.010
lists equal to duplication                        0.215   20/23      0.582    4/7       0.000    7/13      0.000    3/6       0.020   20/25      0.140   17/35      0.235  0.722
lists have unique elems                           0.000    7/17      0.000   11/22      0.003   12/44      0.007   17/30      0.000    6/16      0.000   10/20      0.003  0.007
tree contains only 42                             0.000   10/10      0.000    2/2       0.000   10/10      0.000    2/2       0.000   12/13      0.000    2/2       0.000  0.000
fail_pred_map_commute                             0.002  127/628     0.000   16/59      0.000  112/567     0.000   14/65      0.000  114/665     0.002  117/373     0.002  0.002
fail_pred_strings                                 0.000    1/2       0.000    1/4       0.000    1/2       0.000    1/4       0.000    1/3       0.000    2/5       0.000  0.000
fold_left fold_right                              0.000   25/74      0.000   22/73      0.003   43/161     0.011   58/139     0.001   17/45      0.000   39/95      0.004  0.012
fold_left fold_right uncurried                    4.361  111/99424   0.054  325/984     0.141   36/304    15.278 4811/8969    0.000    5/19      0.000    2/13      4.502 15.332
fold_left fold_right uncurried fun last           0.000   26/89      0.000   25/86      0.004   42/136     0.003   54/176     0.000   16/52      0.000   40/93      0.004  0.004
fold_left test, fun first                         0.002   40/57      0.001   15/28     28.631  125/27750   5.639   45/11947  20.790  282/87674   1.635  168/27748  49.423  7.276
                                                                                                                                                                   94.186 46.951

[jmid]

I have rebased this benchmark PR on top of the merged test PRs.

We still have a few tests printing to stdout, so to avoid garbling the two outputs I changed the program to write the results to a file - for now just hard-coded to shrink_bench.log.

For your viewing pleasure, below follows the output from a fresh run. The situation is (unsurprisingly) the same as I described above in the fall - but PRs #235 and #240 start to change that:

                                                         iteration seed 1234                   iteration seed 8743                   iteration seed 6789               total
Shrink test name                                  Q1/s  #succ/#att   Q2/s  #succ/#att   Q1/s  #succ/#att   Q2/s  #succ/#att   Q1/s  #succ/#att   Q2/s  #succ/#att    Q1/s   Q2/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
big bound issue59                                - skipped as generator is stateful, making it non-repeatable
long_shrink                                       0.022  149/351     0.885 3039/3099    0.007  148/349     0.567 3068/3127    0.006  146/345     0.620 3063/3124    0.035  2.072
ints arent 0 mod 3                                0.000   84/216     0.000    2/2       0.000   71/204     0.000    1/1       0.000   88/215     0.000   88/305     0.000  0.000
ints are 0                                        0.000   62/63      0.000   61/123     0.000   61/62      0.000   61/122     0.000   62/63      0.000   61/123     0.000  0.000
ints < 209609                                    - skipped as generator is stateful, making it non-repeatable
nat < 5001                                        0.000    6/56      0.000    7/77      0.000    6/47      0.000    7/69      0.000    3/24      0.000    8/85      0.000  0.000
char never produces 'abcdef'                      0.000    0/0       0.000    1/1       0.000    0/0       0.000    0/0       0.000    0/0       0.000    0/0       0.000  0.000
strings are empty                                 0.000  249/250     0.000    8/16      0.031 4466/4467    0.000   13/26      0.000    0/1       0.000    1/2       0.031  0.000
string never has a \000 char                      0.000   25/40      0.000   22/167     0.093 4466/4519    0.002   56/254     0.000   15/18      0.000   15/48      0.093  0.002
string never has a \255 char                      0.000  249/316     0.001   59/318     0.097 4466/4520    0.002   97/529     0.403 9260/9365    0.002   41/194     0.500  0.005
strings have unique chars                         0.003  248/269     0.000   18/30      0.912 4465/4536    0.001   24/52      0.000   14/34      0.000   15/20      0.915  0.002
pairs have different components                   0.000    0/4       0.000    0/6       0.000    0/4       0.000    0/6       0.000    0/6       0.000    0/10      0.000  0.000
pairs have same components                        0.000  125/126     0.000   63/125     0.000  124/125     0.000   62/123     0.000  119/120     0.000   63/125     0.000  0.000
pairs have a zero component                       0.000  124/188     0.000  122/306     0.000  123/186     0.000  122/306     0.000  118/182     0.000  123/308     0.000  0.000
pairs are (0,0)                                   0.000  125/126     0.000   63/125     0.000  124/125     0.000   62/123     0.000  119/120     0.000   63/125     0.000  0.000
pairs are ordered                                 0.000  827/17626   0.000   94/1217    0.001  690/12946   0.000   85/865     0.000  687/13963   0.000   94/1326    0.001  0.000
pairs are ordered reversely                       0.000  124/125     0.000   62/124     0.000  123/124     0.000   62/124     0.000  122/123     0.000   62/124     0.000  0.000
pairs sum to less than 128                        0.000  116/129     0.000   56/126     0.000  120/146     0.000   59/138     0.000  119/141     0.000   57/130     0.000  0.000
pairs lists rev concat                            0.014  140/332     0.008   83/168     0.008  137/335     0.002   75/152     0.000  130/318     0.000   67/136     0.021  0.011
pairs lists no overlap                            0.000   22/47      0.004   27/60      0.000   17/41      0.002   18/41      0.000    6/20      0.000   11/28      0.001  0.006
triples have pair-wise different components       0.000    7/31      0.000    3/15      0.000    6/6       0.000    3/3       0.000    2/6       0.000    3/3       0.000  0.000
triples have same components                      0.000  188/252     0.000   64/127     0.000  177/240     0.000   64/128     0.000  182/246     0.000   62/122     0.000  0.000
triples are ordered                               0.000  188/252     0.000    3/4       0.000  177/178     0.000    3/4       0.000  187/250     0.000   91/1021    0.000  0.000
triples are ordered reversely                     0.000  188/189     0.000   64/126     0.000  177/240     0.000  124/247     0.000  182/183     0.000   65/127     0.000  0.000
quadruples have pair-wise different components    0.000   23/41      0.000    4/4       0.000   11/11      0.000    4/4       0.000   14/38      0.000    4/11      0.000  0.000
quadruples have same components                   0.000  250/377     0.000  126/313     0.000  237/424     0.001  115/292     0.000  242/425     0.000  123/307     0.000  0.001
quadruples are ordered                            0.000  251/315     0.000    5/6       0.000  239/240     0.000    4/5       0.000  244/308     0.000    5/6       0.000  0.000
quadruples are ordered reversely                  0.000  251/252     0.000   66/128     0.000  239/302     0.000  126/250     0.000  244/245     0.000   66/128     0.000  0.000
forall (a, b) in nat: a < b                       0.000   13/23      0.000    6/16      0.000   10/15      0.000    6/15      0.000    5/6       0.000    4/7       0.000  0.000
forall (a, b, c) in nat: a < b < c                0.000   15/22      0.000    3/7       0.000   26/53      0.000    7/28      0.000    9/9       0.000    3/3       0.000  0.000
forall (a, b, c, d) in nat: a < b < c < d         0.000   23/29      0.000    4/4       0.000   30/56      0.000    4/4       0.000   13/13      0.000    4/4       0.000  0.000
forall (a, b, c, d, e) in nat: a < b < c < d < e  0.000   28/28      0.000    5/5       0.000   33/33      0.000    5/5       0.000   14/14      0.000    5/5       0.000  0.000
forall (a, b, c, d, e, f) in nat: a < b < c < d   0.000   30/30      0.000    6/6       0.000   38/38      0.000    6/6       0.000   16/16      0.000    6/6       0.000  0.000
forall (a, b, c, d, e, f, g) in nat: a < b < c <  0.000   31/31      0.000    7/7       0.000   41/41      0.000    7/7       0.000   22/22      0.000    7/7       0.000  0.000
forall (a, b, c, d, e, f, g, h) in nat: a < b <   0.000   35/35      0.000    8/8       0.000   48/48      0.000    8/8       0.000   22/22      0.000    7/7       0.000  0.000
forall (a, b, c, d, e, f, g, h, i) in nat: a < b  0.000   42/42      0.000    9/9       0.000   55/55      0.000    9/9       0.000   26/26      0.000    8/8       0.000  0.000
bind ordered pairs                                0.000  125/125     0.000    1/1       0.000  124/124     0.000    1/1       0.000  120/120     0.000    1/1       0.000  0.000
bind list_size constant                           0.000   50/358     0.000   12/26      0.000   48/338     0.001   12/25      0.000   48/342     0.000   11/21      0.000  0.001
lists are empty                                   0.000   11/16      0.000    8/16      0.002   19/27      0.003   13/26      0.000    4/9       0.000    1/2       0.002  0.004
lists shorter than 10                             0.000   50/1198    0.000   16/30      0.000   71/1637    0.002   21/42      0.000   36/868     0.000   15/29      0.001  0.002
lists shorter than 432                            6.851 1696/5118102  1.131  412/457     6.873 1612/4863421  1.058  405/450     6.909 1667/5037661  0.123  419/447    20.633  2.312
lists shorter than 4332                           2.713   13/190735  3.794 4022/4087    1.609   11/126052  4.977 4020/4067    1.790    7/126607  3.707 4013/4055    6.112 12.478
lists equal to duplication                        0.195   20/23      0.667    4/7       0.000    7/13      0.000    3/6       0.018   20/25      0.114   17/35      0.213  0.781
lists have unique elems                           0.000    7/17      0.000   11/22      0.003   12/44      0.006   17/30      0.000    6/16      0.000   10/20      0.003  0.007
tree contains only 42                             0.000   10/10      0.000    2/2       0.000   10/10      0.000    2/2       0.000   12/13      0.000    2/2       0.000  0.000
fail_pred_map_commute                             0.000  127/628     0.000   16/59      0.000  112/567     0.000   14/65      0.000  114/665     0.001  117/373     0.001  0.001
fail_pred_strings                                 0.000    1/2       0.000    1/4       0.000    1/2       0.000    1/4       0.000    1/3       0.000    2/5       0.000  0.000
fold_left fold_right                              0.000   25/74      0.000   22/73      0.002   43/161     0.007   58/139     0.000   17/45      0.000   39/95      0.002  0.007
fold_left fold_right uncurried                    3.261  111/99424   0.047  325/984     0.105   36/304     9.131 4811/8969    0.000    5/19      0.000    2/13      3.366  9.177
fold_left fold_right uncurried fun last           0.000   26/89      0.000   25/86      0.001   42/136     0.003   54/176     0.000   16/52      0.000   40/93      0.001  0.003
fold_left test, fun first                         0.001   40/57      0.001   15/28     20.486  125/27750   4.533   45/11947  14.653  282/87674   1.182  168/27748  35.140  5.716
                                                                                                                                                                   67.071 32.588