You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+8-1Lines changed: 8 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,11 @@
1
-
# Modified HeCBench for Roofline Analysis
1
+
# *gpuFLOPBench*: Counting Without Running: Evaluating LLMs’ Reasoning About Code Complexity
2
+
3
+
This repo is based off of the [HeCBench Suite](https://github.com/zjin-lcf/HeCBench), where we build, profile, and categorize all the CUDA codes to create the **gpuFLOPBench** dataset.
4
+
This dataset is designed to test the FLOP prediction capability of state-of-the-art LLMs, where we only supply them with soure code, compiler args, and command-line input arguments, expecting the LLMs to perform constant propagation and predict the number of FLOPs a target CUDA kernel would perform.
5
+
The querying is done with simple zero-shot prompting techniques and tool calls, without any agentic or MCP features.
6
+
This work gives us a baseline understanding of where current SoTA models are at w.r.t GPU performance prediction from the perspective of FLOP counts.
7
+
8
+
## Modified HeCBench for GPU FLOP Performance Prediction using LLMs
2
9
3
10
We took this version of HeCBench and modified it to build the CUDA and OMP codes to gather their roofline performance data.
4
11
So far we have a large portion of the CUDA and OMP codes building without issue. We use CMake because the `autohecbench.py` was giving us trouble with easily switching out compilers and build options. There were also many issues with individual makefiles, so we decided to put all the build commands into one big `CMakeLists.txt` file for simplicity. We also wanted to create distinct phases of building and gathering data which wasn't too easy with `autohecbench.py`.
0 commit comments