Benchmarking the ethdebug implementation

[clonker]

Abstract

Stringing along debugging data

• externally, e.g., serializing it into a comment-style format or into a separate file to reparse it when the language is being lowered, or
• internally, in terms of data structures and on/off switches in the code that decide whether debug data should be recorded,

cannot have significant impact on the time it takes to invoke the compiler if debugging info is switched off. Any regression must be justified.

To ensure this, benchmarks should be run regularly.

Specification

Benchmarking real-world contracts can be slow for big projects which are also most representative. For faster feedback loops, performance proxy contracts / source files should be devised by, e.g., programmatically generating code. These are expected to be smaller and faster to compile and yet give reasonable indication of the expected outcome on benchmarks run on real-world data. Said proxy contracts should explore the characteristics of real-world contracts in terms of structural patterns - e.g., deeply nested vs wide - as well as feature usage - e.g., heavy ABI, storage, control flow, etc.

For bigger PRs and changes, run the real-world ones, too. For these, a single standard-json file should be extracted out of the external benchmark projects so that costs and variances inherent to the frameworks are not considered.

All the benchmarks themselves run on a dedicated machine with minimal external load and as-stable-as-possible CPU frequency to get reproducible results, triggered manually and/or via labels. Each benchmark should run a minimum of three times (ideally more often, depending on invocation speed) to get more reliable data. Report at least median, mean, and variance.

The scripts to run the benchmarks should be structured so that they can also be run locally with ease to inform about performance implication while, e.g., developing code. To this end there should be a single entry point to run performance tests with the ability to select individual benchmarks (real-world as well as generated) as well as the full suite or a subset thereof.

[cameel]

All the benchmarks themselves run on a dedicated machine

So this is not a self-contained task I guess? It depends on the infra work that @r0qs is doing in https://github.com/argotorg/tofu/issues/28, right?

Also, it sounds like there will be two parts to it, the first one involving putting all the necessary scripts in place for general benchmarking (done by us) and the second one actually implementing ethdebug specific benchmarks (done by Walnut).

[clonker]

So this is not a self-contained task I guess? It depends on the infra work that @r0qs is doing in argotorg/tofu#28, right?

Yes.

Also, it sounds like there will be two parts to it, the first one involving putting all the necessary scripts in place for general benchmarking (done by us) and the second one actually implementing ethdebug specific benchmarks (done by Walnut).

At the very least it is two parts, yes. Who takes care of them is another question.