qat_init.sh: check_driver() times out on 420xx hardware when both qat_4xxx and qat_420xx are loaded

[aleksandrov-denis]

Problem

qat.service fails to start on 420xx hardware with:

QAT driver is still not present after 20s. Aborting qat_init

even when the hardware and drivers are fully functional.

Root cause

get_module_state() builds a grep pattern for every name in SUPPORTED_DRIVER_NAMES (qat_4xxx qat_420xx) and returns the fifth field (module state) for every match in /proc/modules:

get_module_state() {
    CMD=""
    for SUPPORTED_DRIVER_NAME in $SUPPORTED_DRIVER_NAMES;
    do
        CMD="$CMD -e ^$SUPPORTED_DRIVER_NAME"
    done
    echo "$(cat /proc/modules | grep $CMD | cut -d' ' -f5)"
}

On 420xx hardware the kernel loads both modules — qat_420xx for the physical devices and qat_4xxx as a dependency — so get_module_state() returns two lines:

Live
Live

check_driver() then compares this multi-line string against the literal "Live":

while [ "$CURRENT_STATE" != "Live" ]

"Live\nLive" != "Live" is always true, so the loop runs until the 20-second timeout and the service exits with an error, despite both drivers being fully ready.

Reproduction

On any 420xx host (PCI device ID 0x4946):

# Both modules are present even though qat_4xxx has no devices bound
lsmod | grep qat
# qat_420xx  ...
# qat_4xxx   ...   ← loaded, no bound devices
# intel_qat  ... 2 qat_4xxx,qat_420xx

# Simulate what get_module_state() returns
grep -e '^qat_4xxx' -e '^qat_420xx' /proc/modules | cut -d' ' -f5
# Live
# Live    ← two lines; never matches the string "Live"

systemctl start qat
# → "QAT driver is still not present after 20s. Aborting qat_init"

The workaround is to manually unload the idle module first (rmmod qat_4xxx), after which only one line is returned and the comparison succeeds.

Proposed Fix

Replace the string-equality check with a grep that succeeds as soon as any driver in the output reaches the Live state:

# before
while [ "$CURRENT_STATE" != "Live" ]

# after
while ! echo "$CURRENT_STATE" | grep -q "^Live$"

This is the correct semantic: the relevant hardware driver (qat_420xx) being Live is sufficient signal to proceed. The state of co-loaded but idle drivers (qat_4xxx) is irrelevant.

Testing

Verified on an Intel 420xx SR630v3 system running RHEL 9 (kernel 5.14.0-687.el9.x86_64, qatlib 25.08.0) with both qat_4xxx and qat_420xx loaded. Prior to the fix systemctl start qat timed out consistently; after the fix the service
starts cleanly and all 8 PFs are configured correctly.

[aleksandrov-denis]

I have a PR ready with the proposed fix, please let me know if I can push it. Thanks :)

[gcabiddu]

Thanks @aleksandrov-denis, would be great if you could push your fix.

I'm curious to know why is the 4xxx driver is getting loaded on a platform with a QAT 420xx device? I checked the drivers and there is no code-level dependency between qat_420xx and qat_4xxx.

IOW, both qat_420xx and qat_4xxx get loaded independently to handle their respective PCI endpoints and there shouldn't be a platform with both devices of such type.

[aleksandrov-denis]

Hi @gcabiddu, the original issue was caused by an error in a test script, sorry for the confusion. The script was reloading all qat drivers manually and unnecessarily, instead of a general systemctl restart qat.service which would have handled everything automatically. Due to this, one of the qat drivers was useless, but caused the above described error when systemctl start qat.service was called.

Perhaps it is still a valuable fix in case a user loads an unnecessary driver, what do you think?

Thanks!

[fionatrahe]

Thanks for your fix @aleksandrov-denis. If we merge the fix you suggested it may have an unwanted side effect. The Live check was introduced to make sure the kernel driver was fully finished its initialisation before qatlib started to use the device. Your change may result in that check exiting too soon, as soon as the qat_4xxx module is loaded even though qat_420xx module is not finished initialising.

We'll investigate and come up with a fix in the next release. Co-existence of multiple device types is not actually possible with the current devices, so this can only happen with manual manipulation.

[aleksandrov-denis]

Hi @fionatrahe, thanks for the update! That makes sense, I'll also take a look to see if I can find an alternative solution.

[Ahsan-Atta]

Hi @aleksandrov-denis : We are resolving this issue with an alternate method here #151.
Can you please review it and close this thread if you are satisfied with the solution.

Thanks,
Ahsan

[aleksandrov-denis]

@Ahsan-Atta thanks for finding an alternative, lgtm!