CI: ptest workflow fails on release PRs - testdata.json never created

[rpcme]

Problem

The ptest testing workflow consistently fails on release PRs (e.g., #15252) with the error:

ERROR: File .../core-image-minimal-*.rootfs.testdata.json not found
Have you built the image with IMAGE_CLASSES += "testimage" in the conf/local.conf?

This blocks all release PRs from merging and requires admin override.

Root Cause

The workflow has two phases:

1. Build phase: Sets IMAGE_CLASSES += "testimage" in local.conf, builds individual recipes (NOT the image)
2. Test phase: Adds ptest packages to IMAGE_INSTALL, attempts to build image and run testimage

The testdata.json file is required by do_testimage but is never created because:

• Build phase doesn't build the image, only individual recipes
• Test phase finds cached image from previous runs (without IMAGE_CLASSES set)
• Even forcing image rebuild with -c image_complete -f doesn't create testdata.json
• The testimage class doesn't properly initialize when IMAGE_CLASSES is added after initial image build

Failed Fix Attempts

All documented in PR #15252:

1. ❌ bitbake -c testdata_json -f - task doesn't exist
2. ❌ bitbake core-image-minimal - uses sstate cache, doesn't rebuild
3. ❌ bitbake -c rootfs -f - testdata.json created in image_complete, not rootfs
4. ❌ bitbake -c image_complete -f - testdata.json STILL not created despite IMAGE_CLASSES being set

Impact

• All release PRs fail and require admin override to merge
• No ptest coverage on release PRs (defeats purpose of having the test)
• Wastes CI resources running tests that always fail
• Blocks releases and creates manual work

Affected PRs

• Release: Merge master-next to master #15252 (current release PR - master)
• Likely affects all future release PRs with multiple recipes

Reproduction

1. Create PR from master-next → master with 15+ recipe changes
2. CI runs build-test-recipe workflow
3. Build phase succeeds
4. Test phase fails with testdata.json not found

Proposed Solutions

Option A: Skip ptest for release PRs (SHORT TERM)

# In test phase
if [ $(echo $RECIPES | wc -w) -gt 10 ]; then
  echo "Skipping ptest for release PR with many recipes"
  exit 0
fi

Pros: Unblocks releases immediately, individual recipe PRs still tested
Cons: No ptest coverage on releases

Option B: Build image in build phase (MEDIUM TERM)

Add bitbake core-image-minimal to build phase after building recipes
Pros: testdata.json created from the start
Cons: Longer build time, test phase still needs to rebuild with ptest packages

Option C: Clean image before test (SLOW)

Run bitbake core-image-minimal -c cleanall before test phase
Pros: Forces complete rebuild
Cons: Very slow, loses sstate benefits

Option D: Investigate testimage.bbclass (LONG TERM)

Deep dive into Yocto's testimage.bbclass to understand exact conditions for testdata.json creation
Pros: Proper fix
Cons: Time-consuming, may require Yocto expertise

Recommendation

Implement Option A immediately to unblock releases, then investigate Option D for proper long-term fix.

Related Issues

• CI: Improve BitBake version mapping for Yocto releases #15260 - BitBake version mapping (separate CI issue)
• run ptest with credentials to connect to aws #13481 - ptest with AWS credentials (enhancement, not blocker)

Workflow File

.github/workflows/build-test-recipe.yml