test: regression tests show higher memory, throughput and loss values for some settings and models

[kmehant]

Regression is done as part of addition of kernels for GraniteMoeHybridForCausalLM see: #143.

Final reg plots that includes past models

memory increased	loss	throughput increased

Outliers

3 classes of outliers can be identified

1. increased throughput
2. increased memory consumption
3. increased loss

Loss regression

Models ibm-granite/granite-3.0-3b-a800m-instruct and ibm-research/moe-7b-1b-active-shared-experts all padding-free runs regressed from previous bench loss showing larger losses than previous bench loss. However, its not clear if it has to do with padding-free since other models in the benchmark set didn't regress with padding free on.

All outliers

Additional failed runs compared to previous benchmark

Reason: OOM