Is your feature request related to a problem? Please describe.
A top InterUSS priority currently is ensuring sufficient performance of DSS deployments to meet users needs. To do this effectively, we need to be able to easily and repeatably measure the metrics of interest that map directly to meeting user needs. The current tool of choice to attempt this is currently locust in the loadtest folder of this repo. However, the means of applying load (how many users, what do they do, etc) and the metrics of interest (total QPS, QPS per user, latency, etc) are often different and it's not clear which approach will map most directly to meeting user needs.
Describe the solution you'd like
I would like to create a general-purpose benchmarker tool that easily and flexibly allows users and contributors to measure the relevant performance metrics of a system in a repeatable fashion. The importance of such a tool was emphasized in the recent user focus group July 8, and I expect we will want to use this tool to characterize all future DSS releases.
One concern I believe this tool will address is whether our current metrics actually map to meeting user needs. I think modeling system performance with the Universal Scalability Law (or shorter description) will provide good conceptual backing for understanding how we load the system and what the results mean for overall scalability. I think it will be valuable to think about this qualitative graph shape:
Briefly: the blue dotted line shows ideal linear scaling (if going from 0 load to dL load produces dT throughput, then going from L load to L+dL load should produce dT additional throughput -- each incremental load is served with the same efficiency as very low load). This is not practically achievable, so instead each additional load is served a little less efficiently and the curve bends downward away from linear. Additionally, each additional load requires some fixed amount of resources to service, so at some point adding load both doesn't substantially increase throughput due to diminishing returns and actually lowers throughput because resources previously producing throughput have to be rededicated to handling the fixed cost of the additional load. With this effect, the curve peaks at some point and drops below that max throughput as additional load is added.
This max throughput point is an extremely important point. The system should never be operated to in the operational regime to the right of the max throughput point. Any performance metrics captured in the regime to the right of this max throughput point are invalid as they could be improved almost for free with load shedding. Unfortunately, I think we may have been measuring performance in this regime in some of our past benchmarking operations.
Even the max throughput point is not one we want to reach because requests at that point will be served with very high latency compared to requests served at lower load/throughput. Instead, this max throughput point should be the absolute maximum load the system should see when working through a burst of load.
If we step up load in many small steps, we can draw out this curve, and then characterize it with a best fit to the USL equation and therefore express behavior with just three parameters:
- Performance coefficient (slope of line near zero load)
- Contention/serialization coefficient (how much less throughput each additional unit of load results in)
- Crosstalk/coherency-cost coefficient (how much throughput is consumed by each additional unit of load)
Describe alternatives you've considered
This tool is effectively #1519, but designed to scale to a much more broad and powerful utility, especially with capability to test non-local deployments and to be configured by file rather than by flags/environment variables.
I considered whether this tool would fit into the uss_qualifier framework, but I think the tasks and user journeys are sufficiently different to warrant a different tool (though certainly we should reuse capabilities between the tools whenever possible).
Is your feature request related to a problem? Please describe.
A top InterUSS priority currently is ensuring sufficient performance of DSS deployments to meet users needs. To do this effectively, we need to be able to easily and repeatably measure the metrics of interest that map directly to meeting user needs. The current tool of choice to attempt this is currently locust in the loadtest folder of this repo. However, the means of applying load (how many users, what do they do, etc) and the metrics of interest (total QPS, QPS per user, latency, etc) are often different and it's not clear which approach will map most directly to meeting user needs.
Describe the solution you'd like
I would like to create a general-purpose benchmarker tool that easily and flexibly allows users and contributors to measure the relevant performance metrics of a system in a repeatable fashion. The importance of such a tool was emphasized in the recent user focus group July 8, and I expect we will want to use this tool to characterize all future DSS releases.
One concern I believe this tool will address is whether our current metrics actually map to meeting user needs. I think modeling system performance with the Universal Scalability Law (or shorter description) will provide good conceptual backing for understanding how we load the system and what the results mean for overall scalability. I think it will be valuable to think about this qualitative graph shape:
Briefly: the blue dotted line shows ideal linear scaling (if going from 0 load to dL load produces dT throughput, then going from L load to L+dL load should produce dT additional throughput -- each incremental load is served with the same efficiency as very low load). This is not practically achievable, so instead each additional load is served a little less efficiently and the curve bends downward away from linear. Additionally, each additional load requires some fixed amount of resources to service, so at some point adding load both doesn't substantially increase throughput due to diminishing returns and actually lowers throughput because resources previously producing throughput have to be rededicated to handling the fixed cost of the additional load. With this effect, the curve peaks at some point and drops below that max throughput as additional load is added.
This max throughput point is an extremely important point. The system should never be operated to in the operational regime to the right of the max throughput point. Any performance metrics captured in the regime to the right of this max throughput point are invalid as they could be improved almost for free with load shedding. Unfortunately, I think we may have been measuring performance in this regime in some of our past benchmarking operations.
Even the max throughput point is not one we want to reach because requests at that point will be served with very high latency compared to requests served at lower load/throughput. Instead, this max throughput point should be the absolute maximum load the system should see when working through a burst of load.
If we step up load in many small steps, we can draw out this curve, and then characterize it with a best fit to the USL equation and therefore express behavior with just three parameters:
Describe alternatives you've considered
This tool is effectively #1519, but designed to scale to a much more broad and powerful utility, especially with capability to test non-local deployments and to be configured by file rather than by flags/environment variables.
I considered whether this tool would fit into the uss_qualifier framework, but I think the tasks and user journeys are sufficiently different to warrant a different tool (though certainly we should reuse capabilities between the tools whenever possible).