Skip to content

Commit 77830e4

Browse files
smuppandsmuppand
authored
Merge pull request qualcomm-linux#336 from smuppand/kernel
Add generic module reload validation and harden fastrpc unload hang coverage
2 parents 91d881b + 3b5dd95 commit 77830e4

6 files changed

Lines changed: 1613 additions & 0 deletions

File tree

‎Runner/suites/Kernel/Baseport/Module_Reload_Validation/Module_Reload_Validation.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
metadata:
2+
name: Module_Reload_Validation
3+
format: "Lava-Test Test Definition 1.0"
4+
description: "Generic profiled kernel module unload/reload regression validation"
5+
maintainer:
6+
- Srikanth kumar
7+
os:
8+
- linux
9+
scope:
10+
- functional
11+
12+
params:
13+
PROFILE: ""
14+
ITERATIONS: "3"
15+
MODE: ""
16+
TIMEOUT_UNLOAD: "30"
17+
TIMEOUT_LOAD: "30"
18+
TIMEOUT_SETTLE: "20"
19+
ENABLE_SYSRQ_HANG_DUMP: "1"
20+
21+
run:
22+
steps:
23+
- REPO_PATH=$PWD
24+
- cd Runner/suites/Kernel/Baseport/Module_Reload_Validation
25+
- SYSRQ_ARG=""
26+
- if [ "${ENABLE_SYSRQ_HANG_DUMP}" = "0" ]; then SYSRQ_ARG="--disable-sysrq-hang-dump"; fi
27+
- ./run.sh --module "${PROFILE}" --iterations "${ITERATIONS}" --mode "${MODE}" --timeout-unload "${TIMEOUT_UNLOAD}" --timeout-load "${TIMEOUT_LOAD}" --timeout-settle "${TIMEOUT_SETTLE}" ${SYSRQ_ARG} || true
28+
- $REPO_PATH/Runner/utils/send-to-lava.sh Module_Reload_Validation.res

‎Runner/suites/Kernel/Baseport/Module_Reload_Validation/Module_Reload_Validation_README.md‎

Lines changed: 227 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,227 @@
1+
# Module_Reload_Validation
2+
3+
## Overview
4+
`Module_Reload_Validation` is a generic, profile-driven kernel module unload/reload regression suite.
5+
6+
It is intended to catch issues such as:
7+
- module unload hangs,
8+
- failed reloads,
9+
- service/device rebind regressions after reload,
10+
- issues that reproduce on the 1st, 2nd, or later reload iteration.
11+
12+
The suite uses:
13+
- a **generic engine** in `run.sh`,
14+
- shared helper logic in `Runner/utils/lib_module_reload.sh`,
15+
- **module-specific profiles** under `profiles/`.
16+
17+
## Folder layout
18+
19+
```text
20+
Runner/suites/Kernel/Baseport/Module_Reload_Validation/
21+
├── run.sh
22+
├── Module_Reload_Validation.yaml
23+
├── profiles/
24+
│ ├── enabled.list
25+
│ └── fastrpc.profile
26+
27+
Runner/utils/
28+
└── lib_module_reload.sh
29+
```
30+
31+
## Main components
32+
33+
### `run.sh`
34+
Thin orchestration layer that:
35+
- parses CLI arguments,
36+
- resolves the selected profile(s),
37+
- invokes the generic library engine,
38+
- writes `Module_Reload_Validation.res`.
39+
40+
### `lib_module_reload.sh`
41+
Shared module reload engine that handles:
42+
- module state checks,
43+
- timeout-controlled unload/load execution,
44+
- per-iteration evidence collection,
45+
- timeout-path hang evidence,
46+
- profile hook dispatch,
47+
- result handling.
48+
49+
### `profiles/*.profile`
50+
Each profile provides module-specific metadata and optional hook logic.
51+
52+
Example profile fields:
53+
- `PROFILE_NAME`
54+
- `PROFILE_DESCRIPTION`
55+
- `MODULE_NAME`
56+
- `PROFILE_MODE_DEFAULT`
57+
- `PROFILE_REQUIRED_CMDS`
58+
- `PROFILE_SERVICES`
59+
- `PROFILE_DEVICE_PATTERNS`
60+
- `PROFILE_SYSFS_PATTERNS`
61+
62+
Optional hooks:
63+
- `profile_prepare`
64+
- `profile_warmup`
65+
- `profile_quiesce`
66+
- `profile_post_unload`
67+
- `profile_post_load`
68+
- `profile_smoke`
69+
- `profile_finalize`
70+
71+
## Current starter profile
72+
73+
### `fastrpc.profile`
74+
Current profile covers FastRPC unload/reload validation and supports service lifecycle-based testing.
75+
76+
Default mode:
77+
- `daemon_lifecycle`
78+
79+
Relevant services:
80+
- `adsprpcd.service`
81+
- `cdsprpcd.service`
82+
83+
## Execution flow
84+
For each selected profile:
85+
86+
1. Validate the profile.
87+
2. Ensure the module is loaded before starting iteration work.
88+
3. Run warmup hook.
89+
4. Capture pre-state logs.
90+
5. Run quiesce hook.
91+
6. Attempt module unload with timeout.
92+
7. Validate module absence.
93+
8. Run post-unload hook.
94+
9. Attempt module reload with timeout.
95+
10. Validate module presence.
96+
11. Run post-load hook.
97+
12. Run smoke hook.
98+
13. Capture post-load state.
99+
14. Repeat for all iterations.
100+
15. Run finalize hook.
101+
102+
## Hang handling policy
103+
Sysrq dump is **not** triggered on normal passing iterations.
104+
105+
It is triggered only when an unload/load action actually times out.
106+
107+
Current behavior:
108+
- normal pass -> no sysrq dump
109+
- quick non-timeout failure -> normal failure evidence only
110+
- timeout / hang -> hang evidence bundle + optional sysrq dump
111+
112+
Default behavior in current suite:
113+
- sysrq hang dump enabled
114+
- but only used on timeout paths
115+
116+
## Evidence collected
117+
Per profile / iteration, the suite can capture:
118+
- command logs,
119+
- `lsmod`,
120+
- `modinfo`,
121+
- `ps`,
122+
- `dmesg`,
123+
- service status and recent journal,
124+
- profiled device path presence,
125+
- profiled sysfs path presence,
126+
- `/sys/module/<module>/holders`,
127+
- timeout PID `/proc` snapshots,
128+
- optional sysrq task/block dumps.
129+
130+
Results are stored under:
131+
132+
```text
133+
results/Module_Reload_Validation/<profile>/iter_XX/
134+
```
135+
136+
## CLI usage
137+
138+
### Run one profile
139+
```sh
140+
./run.sh --module fastrpc
141+
```
142+
143+
### Run one profile with more iterations
144+
```sh
145+
./run.sh --module fastrpc --iterations 5
146+
```
147+
148+
### Override mode
149+
```sh
150+
./run.sh --module fastrpc --mode daemon_lifecycle
151+
./run.sh --module fastrpc --mode basic
152+
```
153+
154+
### Override timeouts
155+
```sh
156+
./run.sh --module fastrpc --timeout-unload 60 --timeout-load 60 --timeout-settle 30
157+
```
158+
159+
### Disable sysrq timeout-path dumps
160+
```sh
161+
./run.sh --module fastrpc --disable-sysrq-hang-dump
162+
```
163+
164+
### Run all enabled profiles
165+
```sh
166+
./run.sh
167+
```
168+
169+
If `--module` is empty or not given, the suite runs all profiles listed in `profiles/enabled.list`.
170+
171+
If `--mode` is empty or not given, the profile default mode is used.
172+
173+
## YAML usage in LAVA
174+
Current YAML is generic and takes profile input from the test plan.
175+
176+
Important params:
177+
- `PROFILE`
178+
- `ITERATIONS`
179+
- `MODE`
180+
- `TIMEOUT_UNLOAD`
181+
- `TIMEOUT_LOAD`
182+
- `TIMEOUT_SETTLE`
183+
- `ENABLE_SYSRQ_HANG_DUMP`
184+
185+
Behavior:
186+
- `PROFILE="fastrpc"` -> runs only `fastrpc.profile`
187+
- `PROFILE=""` -> runs all enabled profiles
188+
- `MODE=""` -> uses the profile default mode
189+
190+
## How to add a new profile
191+
1. Create a new file under `profiles/`, for example:
192+
- `profiles/ath11k_pci.profile`
193+
2. Define the required metadata:
194+
- `PROFILE_NAME`
195+
- `MODULE_NAME`
196+
3. Add hooks only if module-specific lifecycle handling is needed.
197+
4. Add the profile basename to `profiles/enabled.list`.
198+
5. Run locally with:
199+
200+
```sh
201+
./run.sh --module ath11k_pci
202+
```
203+
204+
No YAML duplication is needed for new profiles.
205+
206+
## Result policy
207+
208+
### PASS
209+
- all requested iterations for a profile pass successfully.
210+
211+
### FAIL
212+
- unload/load timeout,
213+
- unload/load command failure,
214+
- module state validation failure,
215+
- profile hook failure,
216+
- smoke validation failure.
217+
218+
### SKIP
219+
- module not present,
220+
- module built into the kernel,
221+
- required commands not available,
222+
- profile explicitly not reloadable.
223+
224+
## Notes
225+
- This suite is intended for **profiled, supported modules**, not blind reload of every loaded kernel module.
226+
- The current structure avoids hidden run.sh-to-library globals as much as possible by passing explicit arguments and hook context.
227+
- Profile hooks receive context arguments from the engine, so module-specific logic can store logs in the correct iteration directory without depending on hidden globals.

‎Runner/suites/Kernel/Baseport/Module_Reload_Validation/profiles/enabled.list‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
fastrpc

‎Runner/suites/Kernel/Baseport/Module_Reload_Validation/profiles/fastrpc.profile‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/sh
2+
# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries.
3+
# SPDX-License-Identifier: BSD-3-Clause-Clear
4+
5+
PROFILE_NAME="fastrpc"
6+
PROFILE_DESCRIPTION="FastRPC module reload validation"
7+
MODULE_NAME="fastrpc"
8+
9+
MODULE_RELOAD_SUPPORTED="yes"
10+
PROFILE_MODE_DEFAULT="daemon_lifecycle"
11+
12+
PROFILE_REQUIRED_CMDS="modprobe rmmod systemctl ps sed"
13+
PROFILE_SERVICES="adsprpcd.service cdsprpcd.service"
14+
PROFILE_PROC_PATTERNS="/usr/bin/adsprpcd /usr/bin/cdsprpcd"
15+
PROFILE_DEVICE_PATTERNS="/dev/fastrpc-* /dev/*dsp*"
16+
PROFILE_SYSFS_PATTERNS="/sys/module/fastrpc /sys/class/remoteproc/*"

0 commit comments

Comments
 (0)