Skip to content

Commit 8424b6e

Browse files
uncleDecartuncleDecart
committed
Update README
Signed-off-by: Pavel Abramov <uncle.decart@gmail.com>
1 parent 4d124c2 commit 8424b6e

1 file changed

Lines changed: 191 additions & 18 deletions

File tree

‎README.md‎

Lines changed: 191 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
11
# rtos_bench: benchmarking suite to analyse real-time (RT) performance of an operating system
22

3-
A lightweight, configurable Python framework for running system and application benchmarks using Hydra for flexible experiment management.
4-
This repository provides a single entry-point script to run various performance tests with reproducible configurations defined in conf/config.yaml.
3+
A comprehensive Python framework for benchmarking, analyzing and validating real-time (RT) performance of operating systems. Combines Docker-containerized benchmarks with statistical analysis tools based on Extreme Value Theory (EVT) to determine if a system meets real-time requirements.
4+
5+
## Key Features
6+
7+
- **Containerized Benchmarks**: Run reproducible RT benchmarks (Caterpillar, Cyclictest, iperf3, CODESYS) in Docker or on your host system
8+
- **Intel RDT Integration**: Full support for Cache Allocation Technology (CAT) and Memory Bandwidth Allocation (MBA)
9+
- **Statistical RT Validation**: EVT-based analysis with Region of Acceptance (RoA) for probabilistic WCET estimation
10+
- **BIOS Collection via Redfish**: Automatically capture BIOS settings from BMC/iDRAC before benchmarks
11+
- **Jupyter Analysis Notebooks**: Interactive reports for analyzing benchmark results and RT readiness
512

613
## Prerequisites:
714

@@ -19,12 +26,32 @@ curl -LsSf https://astral.sh/uv/install.sh | sh
1926
uv sync
2027
```
2128

22-
## How to run benchmark
29+
4. Additional system requirements:
30+
- Docker (for containerized execution)
31+
- `intel-cmt-cat` package (for pqos/Intel RDT support)
32+
- Root access (required for pqos, IRQ affinity, and some metrics)
33+
34+
35+
36+
## Quick Start
2337

2438
```bash
25-
uv run main.py
39+
40+
# Install dependencies and virtual environment (venv)
41+
uv sync
42+
43+
# Build all Docker images first
44+
sudo .venv/bin/python3 main.py run.command=build
45+
46+
# Run a benchmark (e.g. caterpillar)
47+
sudo .venv/bin/python3 main.py run.command=caterpillar
48+
49+
# Analyze results in Jupyter
50+
uv run jupyter-lab
2651
```
2752

53+
> **Note**: Benchmarks require root access for pqos, IRQ affinity configuration, and hardware monitoring. Use `sudo .venv/bin/python3/main.py` instead of `uv run main.py`
54+
2855
## How to run jupyter notebook (analysis software)
2956

3057
```
@@ -56,7 +83,6 @@ After that you can open any report and run it, just double-click on it like here
5683
├── iperf3/
5784
├── mega-benchmark/
5885
├── codesys-jitter-benchmark/
59-
├── data/ # Store experiments here
6086
├── outputs/ # Where we run experiment bundles
6187
├── notebooks/ # Jupyter notebooks to analyse data
6288
├── src/ # libraries
@@ -73,24 +99,171 @@ All experiment parameters are controlled via Hydra’s configuration file at:
7399
conf/config.yaml
74100
```
75101

76-
## Example configuration
102+
You can override any configuration parameter from the command line:
77103

104+
```bash
105+
sudo .venv/bin/python3 main.py run.command=cyclictest run.t_core="3,5"
78106
```
107+
108+
## Run Configuration
109+
110+
```yaml
79111
run:
80-
command: "caterpillar"
81-
llc_cache_mask: "0x000f"
82-
t_core: "3"
83-
stressor: true
84-
tests_path: "tests"
112+
command: "caterpillar" # Benchmark to run
113+
t_core: "9,11" # Target CPU cores
114+
numa_node: "1" # NUMA node for cpuset-mems (should be same as NUMA node for t_core)
115+
stressor: true # Enable stress workload
116+
metrics: true # Enable metrics monitoring
117+
docker: true # Run inside Docker container
118+
cat_clos_pinning:
119+
enable: true # Pin test PID to CLOS
120+
clos: 1 # CLOS ID to use
121+
```
122+
123+
| Parameter | Type | Description |
124+
| ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------- |
125+
| `run.command` | str | Benchmark to run: `caterpillar`, `cyclictest`, `iperf3`, `mega-benchmark`, `codesys-jitter-benchmark`, `codesys-opcua-pubsub`, or `build`. |
126+
| `run.t_core` | str | Target CPU cores for running the benchmark (e.g., `"3,5,7,9"` or `"9,11"`) |
127+
| `run.numa_node` | str | NUMA node for cpuset-mems (should be same as NUMA node for t_core) |
128+
| `run.stressor` | bool | Enables additional stress workload during the benchmark |
129+
| `run.metrics` | bool | Enable real-time metrics monitoring (CPU temp, IRQs, memory, etc.) |
130+
| `run.docker` | bool | Run benchmark inside Docker container (if `false`, runs on host) |
131+
| `run.cat_clos_pinning.enable`| bool | Enable pinning test PID to specified CLOS (caterpillar/cyclictest only) |
132+
| `run.cat_clos_pinning.clos` | int | CLOS ID to pin the test process to |
133+
134+
## Intel RDT/CAT Configuration (pqos)
135+
136+
Configure Intel Resource Director Technology (Cache Allocation Technology, Memory Bandwidth Allocation):
137+
138+
```yaml
139+
pqos:
140+
interface: "os" # 'os' for resctrl (recommended), 'msr' for direct access
141+
reset_before_apply: true # Reset all allocations before applying new ones
142+
143+
classes:
144+
- id: 1
145+
description: "real-time workload"
146+
l3_mask: "0x00ff" # L3 cache mask (8 cache ways)
147+
l2_mask: "0x00ff" # L2 cache mask
148+
mba: 100 # Memory Bandwidth Allocation (%)
149+
pids: [] # PIDs to assign to this class
150+
cores: [] # CPU cores to assign to this class
151+
- id: 0
152+
description: "background worker"
153+
l3_mask: "0x7f00" # Different cache ways for isolation
154+
l2_mask: "0xff00"
155+
mba: 10
156+
cores: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]
157+
pids: [115, 118]
85158
```
86159

87-
| Parameter | Type | Description |
88-
| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------- |
89-
| `command` | str | Benchmark to run. One of: `caterpillar`, `cyclictest`, `iperf3`, `mega-benchmark`, `codesys-jitter-benchmark`. |
90-
| `llc_cache_mask` | str | Hexadecimal mask for Last-Level Cache (LLC) configuration. |
91-
| `t_core` | str/int | Target CPU core for running the benchmark (i.e. '3,5,7,9') |
92-
| `stressor` | bool | Enables additional stress workload during the benchmark. |
93-
| `tests_path` | str | Path to the directory containing benchmark implementations. |
160+
| Parameter | Type | Description |
161+
| -------------------------- | ------ | ------------------------------------------------------------------ |
162+
| `pqos.interface` | str | Interface mode: `os` (resctrl, required for PIDs) or `msr` (direct)|
163+
| `pqos.reset_before_apply` | bool | Reset all allocations before applying new configuration |
164+
| `pqos.classes[].id` | int | Class of Service (CLOS) ID |
165+
| `pqos.classes[].l3_mask` | str | Hexadecimal L3 cache way mask |
166+
| `pqos.classes[].l2_mask` | str | Hexadecimal L2 cache way mask |
167+
| `pqos.classes[].mba` | int | Memory Bandwidth Allocation percentage (10-100) |
168+
| `pqos.classes[].cores` | list | CPU cores to assign to this CLOS leave empty if not used |
169+
| `pqos.classes[].pids` | list | Process IDs to assign to this CLOS leave empty if not used |
170+
171+
172+
## IRQ Affinity Configuration
173+
174+
Configure IRQ and RCU task affinity to isolate real-time cores:
175+
176+
```yaml
177+
irq_affinity:
178+
enabled: true
179+
housekeeping_cores: "0-1" # Cores for handling IRQs and RCU
180+
```
181+
182+
## BIOS Settings Collection via Redfish
183+
184+
Automatically collect BIOS settings from servers with Redfish-enabled BMC (e.g., Dell iDRAC) before running benchmarks:
185+
186+
```yaml
187+
bios:
188+
enable: true
189+
redfish:
190+
host: "192.168.1.100" # BMC/iDRAC IP address
191+
username: "root"
192+
password: "YOUR_PASSWORD"
193+
verify_ssl: false # Set to true for valid SSL certificates
194+
timeout: 15
195+
196+
output:
197+
format: "json" # Output format: json, yaml, or text
198+
file: "${hydra:run.dir}/bios.json"
199+
pretty: true
200+
```
201+
202+
| Parameter | Type | Description |
203+
| ------------------------- | ------ | ------------------------------------------------------------------ |
204+
| `bios.enable` | bool | Enable/disable BIOS settings collection |
205+
| `bios.redfish.host` | str | BMC/iDRAC hostname or IP address |
206+
| `bios.redfish.username` | str | Username for Redfish API authentication |
207+
| `bios.redfish.password` | str | Password for Redfish API authentication |
208+
| `bios.redfish.verify_ssl` | bool | Verify SSL certificates (set `false` for self-signed certs) |
209+
| `bios.redfish.timeout` | int | Connection timeout in seconds |
210+
| `bios.output.format` | str | Output format: `json`, `yaml`, or `text` |
211+
| `bios.output.file` | str | Path to save BIOS settings (supports Hydra interpolation) |
212+
| `bios.output.pretty` | bool | Enable pretty-printing for JSON output |
213+
214+
## Test-Specific Configuration
215+
216+
### Caterpillar
217+
```yaml
218+
caterpillar:
219+
n_cycles: 7200 # Number of measurement cycles
220+
```
221+
222+
### Cyclictest
223+
```yaml
224+
cyclictest:
225+
loops: 100000 # Number of test loops
226+
```
227+
228+
## Metrics Monitoring
229+
230+
When `run.metrics: true`, the following monitors collect data during benchmark execution:
231+
232+
| Monitor | Output File | Description |
233+
| ---------------- | ------------------------------ | ---------------------------------------- |
234+
| CPU Monitor | `cpu_monitor.csv` | Per-core CPU temperatures |
235+
| IRQ Monitor | `irq_monitor.csv` | Interrupt counts per CPU |
236+
| MemInfo Monitor | `meminfo_monitor.csv` | Memory statistics from `/proc/meminfo` |
237+
| SoftIRQ Monitor | `softirq_monitor.csv` | Software interrupt statistics |
238+
| CPUStat Monitor | `cpustat_monitor.csv` | CPU usage statistics |
239+
| PQOS Monitor | `pqos_monitor.csv` | Intel RDT monitoring data |
240+
241+
Configure monitoring intervals in the config:
242+
243+
```yaml
244+
cpu_monitor:
245+
path: "${hydra:run.dir}/cpu_monitor.csv"
246+
interval: 1.0
247+
```
248+
249+
## Output Files
250+
251+
Each benchmark run creates a timestamped directory in `outputs/` containing:
252+
253+
- `output.csv` - Benchmark results
254+
- `sysinfo.json` - System information snapshot (includes Hydra configuration)
255+
- `bios.json` - BIOS settings (if enabled)
256+
- `*_monitor.csv` - Various metrics logs (if enabled)
257+
- `.hydra/` - Hydra configuration logs
258+
259+
## Security Note
94260

261+
⚠️ **Important**: The Redfish password is stored in the configuration file. Consider:
262+
- Using environment variables for sensitive credentials
263+
- Restricting file permissions on `config.yaml`
264+
- Not committing passwords to version control
95265

266+
## References
96267

268+
- [Dealing with Uncertainty in pWCET Estimations](https://dl.acm.org/doi/abs/10.1145/3396234) - Region of Acceptance methodology
269+
- [Probabilistic-WCET Reliability](https://dl.acm.org/doi/10.1145/3126495) - EVT hypothesis validation

0 commit comments

Comments
 (0)