Add stuck pod describe - #1208
Open
RaunakJalan wants to merge 15 commits into
Open
Conversation
When FIO pods are stuck in PodInitializing or time out during wait_fio, save kubectl describe output to stuck_pod_describes/ directory for post-mortem debugging of volume mount or CSI failures.
TestSingleNodeOutage: The checksum utility pod was deleted with --wait=false, causing a name collision when re-created 24s later. The stale pod returned empty checksums, failing the assertion. Fix: delete_pod() now accepts wait=True; _generate_checksums_dual uses it to block until the pod is fully removed. TestSingleNodeFailure: Stale VolumeSnapshots (snapshot-1, snapshot-2) from TestSingleNodeOutage persisted because cleanup used --wait=false and cleanup_k8s_leftovers only matched snap-* prefix. The next test's kubectl apply hit "persistentVolumeClaimName is immutable", silently reusing the stale snapshot that pointed to a deleted backend object. Fixes: - delete_pod/delete_volume_snapshot: add wait parameter - _generate_checksums_dual: wait for pod deletion - Teardown: wait for snapshot deletion + catch-all for untracked snapshots - cleanup_k8s_leftovers: match snapshot-* in addition to snap-* - create_volume_snapshot: detect and remove stale snapshot before apply
- Create audit pool in Phase 0 before read-only audits so pool.get, pool.iostats, volume.crud, and snapshot.crud are no longer skipped with "no pools available" - Change all not_tested and interface_error severity from WARNING/INFO to ERROR — if an interface fails, it should fail the test - Capture full API call details in every finding: the CLI command or HTTP method+path, HTTP status code, and response preview - _run_cli now returns a dict with data, stdout, stderr, and command so failures include the actual stderr output for debugging - Count mismatches now include sample IDs from each interface so it's clear which items are present vs missing - Report HTML updated with new columns: API Call, HTTP Status, Response preview, and sample IDs for count mismatches - pool.crud uses a separate pool name (parity_crud_pool) so it doesn't conflict with the audit pool lifecycle
Full end-to-end runbook covering Phase 1 (R25.x legacy Helm deployment), Phase 2 (pre-upgrade data setup with FIO/MD5/snapshots/clones), Phase 3 (10-step maintenance window migration), and Phase 4 (post-upgrade validation including old data verify, new provisioning, and outage tests).
- TC-BCK-018: wait for backup completion before PVC deletion (K8s operator re-resolves PVC during reconciliation, causing BackupSourceResolutionError when PVC is deleted mid-backup) - TC-BCK-172: pass restore_size="10G" for resized lvol restore (PVC size must match backup size) - TC-BCK-175: remove -d debug flag that caused false positive error assertion on stderr - Move topology backup tests (TestBackupAfterNodeAdd, TestBackupWithFioOnNewNode, TestBackupAfterNodeMigration, TestBackupDuringMigration) into separate get_backup_topology_tests() so they don't run in the regular backup pipeline without required NEW_NODE_IPS / migrate_to_worker params - Add "backup-topology" keyword to e2e.py test runner Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only TestSequentialNodeAdd needs 2 new nodes; all other add-node tests work with 1. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- csi_repository: add default simplyblock/spdkcsi (was empty) - csi_tag: add default latest (was empty) - ifc_names: br-ex:enp2s0f0 (was ens18:enp1s0) - cluster_environment: openshift-baremetal (was local) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add cluster_security input (none/backup) to add-node and migration workflows (both workflow_call and workflow_dispatch) - Deploy MinIO + backup-credentials secret when backup is enabled - Add BACKUP_SPEC to StorageCluster CR for backup-enabled runs - Auto-detect backup tests by testname containing "Backup" - Pass cluster_security through topology suite parent workflows Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove cluster_security from workflow_dispatch inputs (exceeds GitHub's 25-input limit). Keep it in workflow_call for programmatic use. Backup is auto-enabled when testname contains "Backup" so no functionality lost. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change `inputs.tls_enabled == 'true'` to `inputs.tls_enabled` in if conditions. The truthy check works for both boolean true (from workflow_call) and string "true" (from workflow_dispatch), avoiding type coercion issues when parent workflow passes boolean to child. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The DaemonSet schedules storage-node pods on any node with the simplyblock.io/role=mgmt-plane label. Pre-labeling new_worker_nodes caused pods to start on nodes not yet in the StorageNodeSet, resulting in Init:CrashLoopBackOff. The test itself handles labeling when it adds the node via StorageNodeSet CR update. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Labels persist across pipeline runs. During cleanup, remove simplyblock.io/role from new_worker_nodes and reset their hugepages so the DaemonSet doesn't schedule storage-node pods on them before the test adds them via StorageNodeSet CR. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…elete stuck pods Three fixes for K8s add-node and migration pipeline failures: 1. Pipeline cleanup: remove /etc/simplyblock from all worker nodes (both initial and new) during cleanup phase, preventing stale device config from causing init container CrashLoopBackOff on subsequent runs. 2. Add-node test: add new workers one at a time instead of all at once. Each node's StorageNode CR is created, stale pods are deleted, and the node is waited on to come online before proceeding to the next. 3. Both tests: after creating StorageNode/StorageNodeOps CRs, delete any existing simplyblock-storage-node-ds pods on the target worker so the DaemonSet recreates them with correct StorageNodeSet configuration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When creating StorageNode CRs for add-node expansion, read driveSizeRange and pcieModel from the parent StorageNodeSet and include them in the overrides block. This ensures the init container can discover the correct SSD devices on the new worker node. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: after creating a StorageNode CR, the test immediately deleted the stale DaemonSet pod. The operator hadn't yet updated the per-node-config ConfigMap with the new worker's MAX_LVOL value, so the recreated pod started with MAX_LVOL=0 and crashed in s-node-api-config-generator init container. Fix: poll the per-node-config ConfigMap until it has an entry for the worker node before deleting any stale pods. This ensures the DaemonSet recreates the pod with the correct configuration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.