the question about LLM inference performance

[davidray222]

Thank you for providing such outstanding research!

I tested the llama7b model, and after pruning, both the memory usage and inference speed are not significantly different from the original model. May I ask if you mentioned any methods to accelerate inference for pruned models?

GPU:NVIDIA A6000
torch 2.2.0
transformers 4.31.0
accelerate 0.21.0

[vlejd]

There are a few options.

1. you can use 2:4 sparsity pattern and GPU that supports this type of acceleration,
2. Prune to high sparsity -> 90% to get real speedups and use sparse multiplication implementations like Sputnik, or DASP
3. Use new method that is still under research development: https://github.com/vlejd/macko_spmv. You should get 1.5x speedup and 1.5x memory reduction for 50% sparsity.