uni01alpha: wait tempest pod completion

Modernize uni01alpha test automation to match current patterns:
- Add documentation fields to all stages explaining their purpose
- Add wait_pod_completion for all 4 tempest workflow steps
- Set 3-hour timeouts for multi/single-thread tests (steps 2-3)
- Add "Wait for expected compute services" stage (OSPRH-10942)
- Reorder tempest-tests.yml workflow: ironic tests first

Fix incorrect documentation in other scenarios.

Assisted-By: Claude (claude-4.5-sonnet)
Signed-off-by: Harald Jensås <hjensas@redhat.com>

Author: hjensas

scenarios/multi-nodeset/test-operator/automation-vars.yml

@@ -92,8 +92,7 @@ stages:
     documentation: >-
       Executes comprehensive OpenStack validation tests using the Tempest framework.
       These tests validate API functionality, resource management, and integration
-      between OpenStack services in the baremetal environment. Includes both the
-      test execution phase (workflowStep=0) and log collection phase (workflowStep=1)
+      between OpenStack services. Includes multi-threaded testing (workflowStep=0)
       to ensure complete validation results are captured for analysis.
     manifest: tempest-tests.yml
     wait_conditions:

scenarios/sno-2-bm-ipv6/test-operator/automation-vars.yml

@@ -374,8 +374,8 @@ stages:
     documentation: >-
       Executes comprehensive OpenStack validation tests using the Tempest framework.
       These tests validate API functionality, resource management, and integration
-      between OpenStack services in the baremetal environment. Includes both the
-      test execution phase (workflowStep=0) and log collection phase (workflowStep=1)
+      between OpenStack services in the baremetal environment. Includes ironic
+      scenario testing (workflowStep=0) and ironic API testing (workflowStep=1)
       to ensure complete validation results are captured for analysis.
     manifest: tempest-tests.yml
     wait_conditions:

scenarios/sno-2-bm/test-operator/automation-vars.yml

@@ -358,8 +358,8 @@ stages:
     documentation: >-
       Executes comprehensive OpenStack validation tests using the Tempest framework.
       These tests validate API functionality, resource management, and integration
-      between OpenStack services in the baremetal environment. Includes both the
-      test execution phase (workflowStep=0) and log collection phase (workflowStep=1)
+      between OpenStack services in the baremetal environment. Includes ironic
+      scenario testing (workflowStep=0) and ironic API testing (workflowStep=1)
       to ensure complete validation results are captured for analysis.
     manifest: tempest-tests.yml
     wait_conditions:

scenarios/uni01alpha/test-operator/automation-vars.yml

@@ -1,13 +1,21 @@
 ---
 stages:
   - name: Apply ironic network-attachement-definition
+    documentation: >-
+      Creates a Multus NetworkAttachmentDefinition that allows pods to attach to the ironic
+      provisioning network. This is required for sushy-emulator to communicate with baremetal
+      nodes during provisioning operations via the dedicated provisioning network.
     manifest: manifests/nad.yaml
     wait_conditions:
       - >-
         oc wait -n sushy-emulator network-attachment-definitions.k8s.cni.cncf.io ironic
         --for jsonpath='{.metadata.annotations}' --timeout=30s
 
   - name: Patch RedFish Sushy Emulator Deployment - add network attachment
+    documentation: >-
+      Modifies the sushy-emulator deployment to attach to the ironic provisioning network
+      via CNI network annotation. This enables the BMC simulator to receive Redfish API
+      calls on the correct network segment where baremetal nodes expect their BMC interfaces.
     shell: |
       set -xe -o pipefail
 
@@ -29,6 +37,10 @@ stages:
       - "oc -n sushy-emulator wait deployments.apps sushy-emulator --for condition=Available --timeout=300s"
 
   - name: Set a multiattach volume type and create it if needed
+    documentation: >-
+      Creates a Cinder volume type with multiattach capability enabled, allowing volumes
+      to be simultaneously attached to multiple instances. This is essential for shared
+      storage scenarios like clustered databases or file systems in the test environment.
     shell: |
       set -xe -o pipefail
       oc project openstack
@@ -39,6 +51,11 @@ stages:
       oc rsh openstackclient openstack volume type set --property multiattach="<is> True" multiattach
 
   - name: Create public network if needed
+    documentation: >-
+      Establishes the external network that provides floating IP connectivity to instances.
+      This network is mapped to the physical 'datacentre' network and serves as the gateway
+      for external traffic routing. Essential for instances to reach external services and
+      for external clients to access instance services via floating IPs.
     shell: |
       set -xe -o pipefail
       oc project openstack
@@ -52,6 +69,11 @@ stages:
           --provider-physical-network datacentre
 
   - name: Create subnet on public network if needed
+    documentation: >-
+      Defines the IP address pool and routing configuration for the public network.
+      The 192.168.122.0/24 range with allocation pool 171-250 provides floating IPs
+      while preserving lower addresses for infrastructure. Gateway 192.168.122.1
+      routes traffic to external networks.
     shell: |
       set -xe -o pipefail
       oc project openstack
@@ -65,6 +87,11 @@ stages:
           --dhcp
 
   - name: Create private network if needed
+    documentation: >-
+      Creates the default tenant network for instance-to-instance communication.
+      This shared network allows multiple tenants to deploy instances that can
+      communicate privately while being isolated from external networks until
+      floating IPs are assigned.
     shell: |
       set -xe -o pipefail
       oc project openstack
@@ -73,6 +100,11 @@ stages:
         oc rsh openstackclient openstack network create private --share
 
   - name: Create subnet on private network if needed
+    documentation: >-
+      Configures the private tenant network with RFC1918 addressing (10.2.0.0/24).
+      This subnet provides DHCP-assigned IP addresses for instances deployed on
+      the private network, enabling inter-instance communication before floating
+      IP assignment for external access.
     shell: |
       set -xe -o pipefail
       oc project openstack
@@ -86,6 +118,11 @@ stages:
           --dhcp
 
   - name: Create network for ironic provisioning if needed
+    documentation: >-
+      Establishes the dedicated baremetal provisioning network mapped to the physical
+      'ironic' network interface. This isolated network carries PXE boot traffic, DHCP
+      for baremetal nodes, and communication between Ironic services and nodes during
+      deployment operations, keeping provisioning traffic separate from tenant networks.
     shell: |
       set -xe -o pipefail
       oc project openstack
@@ -99,6 +136,11 @@ stages:
             --provider-network-type flat
 
   - name: Create subnet for ironic provisioning if needed
+    documentation: >-
+      Configures IP addressing for the baremetal provisioning network (172.20.1.0/24).
+      DNS server 192.168.122.80 provides name resolution during node deployment.
+      The allocation pool 100-200 reserves addresses for DHCP assignment to baremetal
+      nodes during their provisioning lifecycle.
     shell: |
       set -xe -o pipefail
       oc project openstack
@@ -113,6 +155,12 @@ stages:
             --allocation-pool start=172.20.1.100,end=172.20.1.200
 
   - name: Create baremetal flavor if needed
+    documentation: >-
+      Defines Nova flavor for baremetal instances with custom resource requirements.
+      Uses CUSTOM_BAREMETAL=1 to match against baremetal node resource classes,
+      while setting standard resources (VCPU, MEMORY_MB, DISK_GB) to 0 since
+      physical resources are managed by Ironic rather than Nova's scheduler.
+      UEFI boot mode ensures compatibility with modern baremetal hardware.
     shell: |
       set -xe -o pipefail
       oc project openstack
@@ -131,18 +179,33 @@ stages:
             --property capabilities:boot_mode=uefi
 
   - name: Copy ironic_nodes.yaml to the openstackclient pod
+    documentation: >-
+      Transfers the baremetal node definition file containing BMC credentials,
+      hardware specifications, and network configurations from the local filesystem
+      to the OpenStack client pod. This file defines the physical infrastructure
+      that Ironic will manage for baremetal provisioning.
     shell: |
       set -xe -o pipefail
       oc project openstack
       oc cp ~/data/ironic_nodes.yaml openstackclient:ironic_nodes.yaml
 
   - name: Enroll nodes in ironic
+    documentation: >-
+      Registers physical baremetal nodes with the Ironic service using the node
+      definitions from ironic_nodes.yaml. This creates Ironic node records with
+      BMC access credentials, hardware profiles, and port configurations, marking
+      the beginning of the node lifecycle management in OpenStack.
     shell: |
       set -xe -o pipefail
       oc project openstack
       oc rsh openstackclient openstack baremetal create ironic_nodes.yaml
 
   - name: Wait for ironic nodes to get to state - enroll
+    documentation: >-
+      Monitors node state transition to 'enroll' status, indicating that Ironic
+      has successfully registered the nodes and validated basic BMC connectivity.
+      This is the first state in the baremetal provisioning lifecycle, confirming
+      that nodes are recognized by the system before management operations begin.
     shell: |
       oc project openstack
 
@@ -160,6 +223,11 @@ stages:
       done
 
   - name: Manage ironic nodes
+    documentation: >-
+      Initiates the transition from 'enroll' to 'manageable' state by instructing
+      Ironic to perform hardware introspection and validation. During this process,
+      Ironic will power on nodes, inspect hardware capabilities, and prepare them
+      for provisioning operations while validating BMC access and power management.
     shell: |
       set -xe -o pipefail
       oc project openstack
@@ -168,6 +236,11 @@ stages:
       oc rsh openstackclient openstack baremetal node manage ironic1
 
   - name: Wait for ironic nodes to get to state - manageable
+    documentation: >-
+      Waits for nodes to complete hardware introspection and reach 'manageable' state.
+      In this state, Ironic has successfully inventoried hardware resources, validated
+      BMC functionality, and confirmed the nodes are ready for cleaning and provisioning
+      operations. This is a prerequisite for making nodes available to tenants.
     shell: |
       oc project openstack
 
@@ -185,22 +258,24 @@ stages:
       done
 
   - name: Power off the ironic nodes
+    documentation: >-
+      Ensures all baremetal nodes are powered down before configuration changes.
+      This prevents potential issues during capability updates and ensures a clean
+      state before transitioning nodes to 'available'. Power management validation
+      also confirms BMC functionality is working correctly.
     shell: |
       set -xe -o pipefail
       oc project openstack
 
       oc rsh openstackclient openstack baremetal node power off ironic0
       oc rsh openstackclient openstack baremetal node power off ironic1
 
-  - name: Set capabilities boot_mode:uefi for ironic nodes
-    shell: |
-      set -xe -o pipefail
-      oc project openstack
-
-      oc rsh openstackclient openstack baremetal node set --property capabilities='boot_mode:uefi' ironic0
-      oc rsh openstackclient openstack baremetal node set --property capabilities='boot_mode:uefi' ironic1
-
   - name: Ensure ironic nodes are powered off
+    documentation: >-
+      Verifies that power management commands have taken effect and all nodes
+      report 'power off' status. This confirmation is critical before transitioning
+      to 'available' state, as Nova expects baremetal nodes to be powered off
+      when not actively hosting instances.
     shell: |
       oc project openstack
 
@@ -218,6 +293,11 @@ stages:
       done
 
   - name: Provide ironic nodes
+    documentation: >-
+      Transitions nodes from 'manageable' to 'available' state, making them eligible
+      for tenant provisioning. This triggers automated cleaning processes to ensure
+      nodes are in a pristine state, removing any residual data from previous
+      deployments and preparing them for new instance launches.
     shell: |
       set -xe -o pipefail
       oc project openstack
@@ -226,6 +306,11 @@ stages:
       oc rsh openstackclient openstack baremetal node provide ironic1
 
   - name: Wait for ironic nodes to get to state - available
+    documentation: >-
+      Confirms nodes have completed the cleaning process and reached 'available' state.
+      Available nodes appear in Nova's resource inventory and can be allocated to
+      instance requests that match the baremetal flavor. This state indicates the
+      baremetal infrastructure is fully operational and ready for workload deployment.
     shell: |
       oc project openstack
 
@@ -258,5 +343,61 @@ stages:
         sleep 10
       done
 
+  - name: Wait for expected compute services (OSPRH-10942)
+    documentation: >-
+      Waits for Nova compute services to register and become available in the
+      Nova service registry. The hotstack-nova-discover-hosts utility ensures
+      that all compute services (2 EDPM computes + 1 Ironic conductor) are
+      properly registered as compute nodes, enabling Nova scheduler to place
+      both VM and baremetal instances. References bug OSPRH-10942 related to
+      compute service discovery timing issues.
+    wait_conditions:
+      - >-
+        timeout --foreground 5m hotstack-nova-discover-hosts
+        --namespace openstack --num-computes 3
+
   - name: Run tempest
+    documentation: >-
+      Executes comprehensive OpenStack validation tests using the Tempest framework.
+      These tests validate API functionality, resource management, and integration
+      between OpenStack services in the baremetal environment.
+
+      Workflow steps:
+      - Step 0: ironic-scenario-testing
+      - Step 1: ironic-api-testing
+      - Step 2: multi-thread-testing
+      - Step 3: single-thread-testing
     manifest: tempest-tests.yml
+    wait_conditions:
+      - >-
+        oc wait -n openstack tempests.test.openstack.org tempest-tests
+        --for condition=ServiceConfigReady --timeout=120s
+    wait_pod_completion:
+      - namespace: openstack
+        labels:
+          operator: test-operator
+          service: tempest
+          workflowStep: "0"
+        timeout: 900
+        poll_interval: 15
+      - namespace: openstack
+        labels:
+          operator: test-operator
+          service: tempest
+          workflowStep: "1"
+        timeout: 900
+        poll_interval: 15
+      - namespace: openstack
+        labels:
+          operator: test-operator
+          service: tempest
+          workflowStep: "2"
+        timeout: 10800
+        poll_interval: 30
+      - namespace: openstack
+        labels:
+          operator: test-operator
+          service: tempest
+          workflowStep: "3"
+        timeout: 10800
+        poll_interval: 30