From efa95715a77f8ac15dbd93893f8b198a730d17a5 Mon Sep 17 00:00:00 2001 From: aarushjain29 Date: Fri, 29 May 2026 16:41:55 +0000 Subject: [PATCH] tools readme --- tools/README.md | 120 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 tools/README.md diff --git a/tools/README.md b/tools/README.md new file mode 100644 index 00000000000..68ff3bb4e8e --- /dev/null +++ b/tools/README.md @@ -0,0 +1,120 @@ +# MIGraphX `tools/` + +This directory contains the helper scripts and supporting files used to develop, test, +profile, and maintain MIGraphX. They are not part of the runtime library; they are the +utilities that contributors and continuous integration run alongside it. + +This page serves as an index for locating the appropriate tool. Tools that provide their +own detailed instructions link to a dedicated `README.md` within their subfolder. + +> Most Python tools accept `-h`/`--help`; run `python3 tools/.py --help` for usage +> details. Unless otherwise noted, scripts should be run from the repository root. + +## Task reference + +| Task | Tool | +|---|---| +| Install build prerequisites or set up an environment | [`install_prereqs.sh`](#environment-and-build-setup), [`docker/`](#environment-and-build-setup) | +| Format source before pushing | [`format.py`](#code-quality-and-ci-gates) | +| Add or verify license-header year stamps | [`license_stamper.py`](#code-quality-and-ci-gates), [`check_stamped.py`](#code-quality-and-ci-gates) | +| Run the custom static-analysis (cppcheck) checks | [`cppcheck/`](#code-quality-and-ci-gates) | +| Regenerate the public C/Python API after editing headers | [`generate.py`](#api-and-code-generation) | +| Verify MIGraphX results against a reference | [`accuracy/`](#model-testing-and-verification), [`ort/`](#model-testing-and-verification) | +| Run a folder of ONNX test cases | [`test_runner.py`](#model-testing-and-verification), [`model_zoo/`](#model-testing-and-verification) | +| Convert an ONNX model's opset, or `.mxr` to ONNX | [`convert_onnx_version.py`](#model-testing-and-verification), [`converters/`](#model-testing-and-verification) | +| Determine the fastest environment-variable configuration for a model | [`autotune_perf/`](#performance-and-profiling) | +| Profile per-operator runtime or compile time | [`roctx.py`](#performance-and-profiling), [`compile_analysis.py`](#performance-and-profiling) | +| Build and test the ONNX Runtime and MIGraphX integration | [`build_and_test_onnxrt.sh`](#onnx-runtime-integration) | +| Review pull request status across the repository | [`pr_review_dashboard/`](#repository-and-workflow) | +| Enable syntax highlighting for MIGraphX IR dumps | [`syntax/`](#editor-support) | + +--- + +## Environment and build setup + +| Path | Description | +|---|---| +| `install_prereqs.sh` | Installs MIGraphX build prerequisites (CMake, the ROCm/MIOpen/rocBLAS development packages, `rbuild`, and Python dependencies) for Ubuntu and SLES. Used by the Docker images and for provisioning a development machine. Optional arguments: `install_prereqs.sh [PREFIX] [DEPS_DIR]`. | +| `requirements-py.txt` | Pinned Python packages (onnx, numpy, protobuf, pytest, and others) required by the Python-based tools and the ONNX Runtime unit tests. Install with `pip3 install -r tools/requirements-py.txt`. | +| `docker/` | Dockerfiles for reproducible build and test environments: `ubuntu_2204.dockerfile`, `ubuntu_2404.dockerfile`, `sles.docker`, `ort.dockerfile` (ONNX Runtime), `migraphx_with_onnxruntime_pytorch.docker`, and `therock_deb.docker`. | + +## Code quality and CI gates + +These scripts implement the `format`, `licensing`, and `cppcheck` gates that block pull +request merges. + +| Path | Description | +|---|---| +| `format.py` | Runs `clang-format` on changed C/C++/HIP files and `yapf` on changed Python files. Run `python3 tools/format.py origin/develop` to display diffs, or add `-i` to apply changes in place. This should be run before every push. | +| `check_stamped.py` | Verifies that every file changed relative to a base branch carries the current-year MIT license stamp (the continuous integration `licensing` gate). Usage: `python3 tools/check_stamped.py origin/develop`. | +| `license_stamper.py` | Adds or updates license-header year stamps. `python3 tools/license_stamper.py origin/develop` stamps changed files; `--all` rewrites every tracked file using its last-commit year. | +| `stamp_status.py` | Shared helper for the two scripts above, providing the stamp-detection and year-update logic. Not intended to be run directly. | +| `git_tools.py` | Shared helper used by the format and stamp scripts to query git for the merge base, changed files, and commit year. Not intended to be run directly. | +| `cppcheck/` | MIGraphX's custom static analysis: a cppcheck addon (`migraphx.py`), XML pattern rules (`rules.xml`), and `test.sh` to run them. See [`cppcheck/test/README.md`](./cppcheck/test/README.md) for the test suite. | + +## API and code generation + +MIGraphX's stable C/C++/Python API and its type-erased interfaces are generated from +templates and header definitions. Edit the inputs described below, then regenerate. + +| Path | Description | +|---|---| +| `generate.py` | Entry point for the CMake `generate` target. Regenerates `src/api/api.cpp`, `src/api/include/migraphx/migraphx.h`, and the type-erased headers under `src/include/migraphx/`, then applies clang-format. Run `make generate` from the build directory, or `python3 tools/generate.py -f `. | +| `api.py` | The API-generation engine consumed by `generate.py`. It parses the API definitions and emits the C header and `api.cpp`. | +| `te.py` | The type-erasure generator, which converts the interface headers in `include/` into their concrete type-erased classes. | +| `api/` | Input templates for the public API: `migraphx.h` and `api.cpp`, processed by `api.py`. | +| `include/` | Interface header definitions (`operation.hpp`, `pass.hpp`, `target.hpp`, `context.hpp`, and others) consumed by `te.py` to generate the type-erased versions. | +| `CMakeLists.txt` | Defines the `generate` custom target (requires Python 3 and clang-format). | + +## Model testing and verification + +| Path | Description | +|---|---| +| `accuracy/` | Compares MIGraphX output against ONNX Runtime for a given ONNX model and reports `PASSED` or `FAILED`. See [`accuracy/README.md`](./accuracy/README.md). | +| `ort/` | A lightweight ONNX Runtime driver for quickly running and validating an ONNX file through ONNX Runtime. See [`ort/README.md`](./ort/README.md). | +| `test_runner.py` | Runs a folder of ONNX test cases (a model with reference inputs and outputs) through MIGraphX on a chosen target (`ref` or `gpu`) with configurable tolerances. Usage: `python3 tools/test_runner.py --target gpu`. | +| `model_zoo/` | Helpers to fetch and run larger model collections: the ONNX Model Zoo and a dataset-driven test generator. See [`model_zoo/README.md`](./model_zoo/README.md), [`model_zoo/onnx_zoo/README.md`](./model_zoo/onnx_zoo/README.md), and [`model_zoo/test_generator/README.md`](./model_zoo/test_generator/README.md). | +| `convert_onnx_version.py` | Converts an ONNX model to a target opset, optionally inferring shapes. Usage: `python3 tools/convert_onnx_version.py --model in.onnx --output out.onnx --opset N`. | +| `converters/` | `mxr_to_onnx.py` reconstructs an ONNX graph from a serialized MIGraphX (`.mxr`) program, which is useful for inspecting or replaying compiled models. | + +## Performance and profiling + +| Path | Description | +|---|---| +| `autotune_perf/` | Sweeps MIGraphX environment-variable settings on top of `migraphx-driver perf`, reports the fastest configuration, and writes a sourceable `.tune` file. See [`autotune_perf/README.md`](./autotune_perf/README.md). | +| `roctx.py` | Parses ROCTX marker traces (JSON) produced by a profiled MIGraphX run into per-operator CSV summaries for analysis. Usage: `python3 tools/roctx.py --json-path --out `. | +| `compile_analysis.py` | Parses a MIGraphX compile log for per-stage timings and renders an interactive HTML breakdown of compile time. Usage: `python3 tools/compile_analysis.py --file_path `. | + +## ONNX Runtime integration + +Scripts used to build and continuously test the MIGraphX execution provider within ONNX +Runtime. + +| Path | Description | +|---|---| +| `build_and_test_onnxrt.sh` | Builds ONNX Runtime with the MIGraphX execution provider and runs its test suites (used by the ONNX Runtime continuous integration image). | +| `pai_test_launcher.sh` | Runs `onnxruntime_test_all`, excluding the tests listed in `pai-excluded-tests.txt`. | +| `pai_provider_test_launcher.sh` | Runs `onnxruntime_provider_test`, excluding the tests listed in `pai-excluded-tests.txt`. | + +## Repository and workflow + +| Path | Description | +|---|---| +| `pr_review_dashboard/` | `pr-review-status.py` categorizes open pull requests by reviewer and approval state (terminal output or `--json`), accompanied by `pr-review-config.json` and an `index.html` dashboard view. Requires a `GITHUB_TOKEN`. | + +## Editor support + +| Path | Description | +|---|---| +| `syntax/` | `migx.vim` provides Vim syntax highlighting for MIGraphX IR (program and module) text dumps. | + +--- + +### Conventions for tools + +- New or modified `.py`, `.sh`, `.cpp`, `.hpp`, and similar files require a current-year MIT + license header. Run `python3 tools/license_stamper.py origin/develop` to apply it. +- Format Python with `yapf` and C/C++ with `clang-format` via + `python3 tools/format.py origin/develop -i`. +- Refer to the repository-root `README.md` and the `CONTRIBUTING`/`AGENTS.md` files for the + complete build and pull request workflow.