fix(tests): prevent Chrome OOM in e2e incident lifecycle test

Split the alert polling into two phases to avoid Chrome OOM (exit code 137):

Phase 1: Poll the Thanos Querier API via cy.exec/oc to check if the alert
is actually firing. This is lightweight — a single shell command per
iteration with zero Chrome DOM interaction or Cypress command log growth.

Phase 2: Once the alert is confirmed firing, use the UI traversal to find
the incident, but with doubled interval (2 min) and fewer max iterations.
This reduces the number of heavy DOM traversals that accumulate Cypress
command snapshots in Chrome memory.

Also fix poll-ci-status.py to use --repo openshift/monitoring-plugin flag
so it queries the upstream repo instead of the local fork.

CI run: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_monitoring-plugin/860/pull-ci-openshift-monitoring-plugin-main-e2e-incidents/2036818563108442112
Classifications: INFRA_OOM (2 consecutive runs)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: DavidRajnoha

.claude/commands/cypress/scripts/poll-ci-status.py

@@ -27,7 +27,7 @@
 def poll(pr, job_substring="e2e-incidents", max_attempts=30, interval=300):
     for attempt in range(max_attempts):
         result = subprocess.run(
-            ["gh", "pr", "checks", pr, "--json", "name,state,link"],
+            ["gh", "pr", "checks", pr, "--repo", "openshift/monitoring-plugin", "--json", "name,state,link"],
             capture_output=True,
             text=True,
         )

web/cypress/e2e/incidents/00.coo_incidents_e2e.cy.ts

@@ -41,16 +41,37 @@ describe('BVT: Incidents - e2e', { tags: ['@smoke', '@slow', '@incidents', '@e2e
     cy.log('1.1 Navigate to Incidents page and clear filters');
     incidentsPage.goTo();
     incidentsPage.clearAllFilters();
-    
+
     const intervalMs = 60_000;
-    const maxMinutes = 30; 
+    const maxMinutes = 30;
+
+    cy.log('1.2 Wait for alert to start firing on cluster');
+    // Phase 1: Poll the Thanos Querier API to check if the alert is actually
+    // firing. This is lightweight — a single cy.exec per iteration with no
+    // Chrome DOM interaction, preventing the OOM (exit code 137) caused by
+    // repeated heavy UI traversals accumulating Cypress command log snapshots.
+    const kubeconfigPath = Cypress.env('KUBECONFIG_PATH');
+    cy.waitUntil(
+      () => cy.exec(
+        `oc get --raw '/api/v1/namespaces/openshift-monitoring/services/thanos-querier:web/proxy/api/v1/rules?type=alert' --kubeconfig ${kubeconfigPath}`,
+        { failOnNonZeroExit: false, timeout: 20000 },
+      ).then((result) => result.code === 0 && result.stdout.includes(currentAlertName)),
+      {
+        interval: 30_000,
+        timeout: 15 * 60_000,
+        errorMsg: `Alert ${currentAlertName} not firing on cluster within 15 minutes`,
+      }
+    );
 
-    cy.log('1.2 Wait for incident with custom alert to appear');
+    cy.log('1.2.1 Wait for incident detection to pick up the firing alert');
+    // Phase 2: Alert is confirmed firing. Wait for incident detection to group
+    // it into an incident. Uses the UI traversal but with fewer iterations
+    // since incident detection typically takes 5-10 minutes after alert fires.
     cy.waitUntil(
       () => incidentsPage.findIncidentWithAlert(currentAlertName),
-      { 
-        interval: intervalMs, 
-        timeout: maxMinutes * intervalMs,
+      {
+        interval: 2 * intervalMs,
+        timeout: 20 * intervalMs,
       }
     );