Refactor Performance Tests and expand Monitoring Stack

[rendre-greyling]

Refactor Performance Tests and expand Monitoring Stack

This PR is a continuation of #561 which has been closed due to being out of date.

NOTE: This PR is ~30K lines - a large majority of this is generic test kpt packages (~19k lines) and large Grafana json dashboards

Description

New Performance Tests and Suite

The previous performance tests were split between 2 performance packages, one under test/performance folder, the other under test/e2e/performance which was the Iterative test. These tests were outdated and lacked certain metrics. Features of both have now been consolidated under one test/performance folder.

The new Performance Tests work with both the v1alpha1 and v1alpha2 api's of porch and can be expanded in future.

There are now 2 new tests which replace the previous tests and functionality:

• Load Test: Creates a preset amount of repositories, packages, and revisions to simulate load on the Porch system.
• Maximum Package Revisions: Tests the maximum number of package revisions that can be handled by Porch in a single repository.

See the test/performance/README.md for all info on the data that is collected by the tests and customization arguments. The data is stored in csv files for later analysis. The data is also exposed to OTel/Prometheus if the -enable-prometheus is set.

The Package Revisions which are deployed can be selected from 3 generic Kpt packages:

• small-package: Minimal footprint and 2 basic kpt mutators for 1 file in the package.
• complex-package: Multiple mutators and validators across a dozen files in the package to simulate loads on function pods.
• large-package: Multiple mutators and validators across a couple of files in the package and one very large (~17k lines) deployment.yaml file to simulate the impact of large files on the system.

New Performance Monitoring Stack

The OTel interface has been expanded to expose various new API metrics as well as Memory and CPU data. A new deploy-monitoring.sh script along with make targets has been added which will deploy Grafana, Prometheus, Postgres Exporter (for the porch DB cache metrics), Pyroscope (using alloy to get the data from pprof) and Jaeger pods in the kind cluster for visual analysis of the data.

The make targets are:

• make deploy-monitoring: Deploys the base Grafana, Prometheus and Postgres Exporter pods, services and sets up port forwarding.
• make deploy-monitoring-jaeger: Deploys the Jaeger pod, service and sets up port forwarding.
• make deploy-monitoring-pyroscope: Deploys the Pyroscope and Alloy pods, services and port forwarding.
• make cleanup-monitoring: Deletes all monitoring pods, services and port forwarding.
• make restart-monitoring: Restarts all monitoring pods, services and port forwarding.

Grafana has 4 dashboards:

• Porch API: PR, PRR and external Git requests time taken per API call, as well as req/s per user account. (these metrics are tracked for both v1alpha1 and v1alpha2)
• Porch Performance Test Dashboard: All recorded metrics from the performance tests
• Porch Resource Usage: Memory and CPU usage of the porch-server, function runner, controllers, DB Cache and function pods.
• Porch Postgres Dashboard: Displays metrics of the Postgres DB such as QPS, deadlocks and cache hits and misses.
• Porch Package Size Dashboard: remains unchanged from: Add OpenTelemetry metrics for package resources size #1018 except for smaller vertical height of the panel.
• Pyroscope – Porch profiling: Advanced CPU, Memory and Go profiling data with the use of flame graphs.

Prometheus UI can be accessed by http://localhost:9092/

Pyroscope UI can be accessed by http://localhost:4040/ after running the make deploy-monitoring-pyroscope` for more granularity.

Jaeger UI can be accessed by http://localhost:16686/ after running the make deploy-monitoring-jaeger` for advanced tracing.

---

Type of Change

• Bug fix
• New feature
• Enhancement
• Refactor
• Documentation
• Tests
• Other: ________

---

Checklist

• Code follows project style guidelines
• Self-reviewed changes
• Tests added/updated
• Documentation added/updated
• All tests and gating checks pass

---

Testing Instructions (Optional)

1. Setup kind cluster
2. Follow instructions in test/performance/README.md

---

AI Disclosure

[x] I have used AI in the creation of this PR.

• Cursor composer 2.5 and Sonnet 4.6 used for the creation of the generic kpt packages and Grafana dashboards, squaring away the scripts and deployments as well as the csv creation. Also assisted in splitting the original implementation of the performance tests to be able to use both API types.

[netlify]

✅ Deploy Preview for kpt-porch ready!

Name	Link
🔨 Latest commit	489e08f
🔍 Latest deploy log	https://app.netlify.com/projects/kpt-porch/deploys/6a5ddaa0e7fa1e00087bba46
😎 Deploy Preview	https://deploy-preview-1122--kpt-porch.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull request overview

This PR modernizes Porch’s performance testing by consolidating and expanding the test/performance suite (supporting both v1alpha1 and v1alpha2) and introduces a richer local monitoring/profiling stack (Prometheus/Grafana + optional Jaeger, Pyroscope, and Postgres exporter) to observe API, controller, resource usage, and profiling metrics.

Changes:

• Refactors performance tests into a new test/performance suite with explicit API-version drivers, richer operation metrics, and CSV/log outputs.
• Expands telemetry to record API/controller durations, request counts by user, external repo operations, and performance-test metrics via OTel/Prometheus.
• Adds/extends local monitoring stack deployment (Prometheus/Grafana/Postgres exporter + optional Jaeger/Pyroscope/Alloy) and updates Porch manifests to expose pprof and support profiling.

Reviewed changes

Copilot reviewed 78 out of 81 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
test/performance/types.go	Adds API version typing, operation key constants, and richer metrics structures.
test/performance/README.md	Rewrites documentation for the new perf test suite and monitoring workflow.
test/performance/promql_queries.txt	Removes legacy PromQL query cheat sheet (superseded by new dashboards/docs).
test/performance/prometheus_metrics.go	Removes legacy Prometheus client metrics implementation (migrated to OTel/telemetry).
test/performance/packages/small-package/Kptfile	Adds minimal kpt package used as perf-test payload.
test/performance/packages/small-package/deployment.yaml	Adds small test Deployment manifest with setters.
test/performance/packages/large-package/Kptfile	Adds large kpt package pipeline stressing kpt functions and payload size.
test/performance/packages/large-package/limitrange.yaml	Adds LimitRange resource used by the large package.
test/performance/packages/large-package/starlark-run.yaml	Adds StarlarkRun to mutate/validate large-package workloads.
test/performance/packages/large-package/apply-replacements.yaml	Adds ApplyReplacements resource for cross-resource field propagation.
test/performance/packages/large-package/gatekeeper-constraints.yaml	Adds Gatekeeper constraint template + constraint (local-config) for load realism.
test/performance/packages/complex-package/Kptfile	Adds complex package pipeline covering many catalog functions.
test/performance/packages/complex-package/configmap.yaml	Adds complex-package ConfigMap with setter-driven values.
test/performance/packages/complex-package/deployment.yaml	Adds complex-package Deployment with probes/resources/env.
test/performance/packages/complex-package/service.yaml	Adds Service for complex-package.
test/performance/packages/complex-package/serviceaccount.yaml	Adds ServiceAccount for complex-package.
test/performance/packages/complex-package/role.yaml	Adds Role for complex-package (RBAC realism).
test/performance/packages/complex-package/rolebinding.yaml	Adds RoleBinding for complex-package.
test/performance/packages/complex-package/networkpolicy.yaml	Adds NetworkPolicy for complex-package.
test/performance/packages/complex-package/poddisruptionbudget.yaml	Adds PDB for complex-package.
test/performance/packages/complex-package/limitrange.yaml	Adds LimitRange for complex-package.
test/performance/packages/complex-package/ingress.yaml	Adds Ingress for complex-package.
test/performance/packages/complex-package/hpa.yaml	Adds HPA for complex-package.
test/performance/packages/complex-package/starlark-run.yaml	Adds StarlarkRun for complex-package workload mutation.
test/performance/packages/complex-package/apply-replacements.yaml	Adds ApplyReplacements rules for complex-package.
test/performance/packages/complex-package/gatekeeper-constraints.yaml	Adds Gatekeeper constraints (local-config) for complex-package.
test/performance/operations.go	Centralizes operation lists/headings per API version and lifecycle filtering helpers.
test/performance/driver.go	Introduces shared lifecycle driver base (create/update/propose/approve/delete + telemetry hooks).
test/performance/driver_v1alpha1.go	Implements v1alpha1 lifecycle driver, including approval update via Porch clientset.
test/performance/driver_v1alpha2.go	Implements v1alpha2 lifecycle driver with controller reconciliation waits and deletion semantics.
test/performance/csv_generation.go	Adds CSV generation for lifecycle, per-operation, and deletion metrics.
test/performance/logger.go	Refactors logging into file-backed loggers for consolidated output and results.
test/performance/metrics.go	Removes older perf test implementation (replaced by new suite/driver architecture).
test/e2e/performance/utils.go	Removes legacy iterative perf test utilities.
test/e2e/performance/rules.yml	Removes legacy Prometheus rule set for iterative perf test.
test/e2e/performance/README.md	Removes legacy iterative perf test docs.
test/e2e/performance/prometheus.go	Removes legacy Prometheus client metrics code for iterative perf test.
test/e2e/performance/prometheus_in_docker.sh	Removes legacy dockerized Prometheus helper.
test/e2e/performance/performance_suite.go	Removes legacy iterative perf test suite scaffolding.
test/e2e/performance/iterative_types.go	Removes legacy iterative metrics structures.
test/e2e/performance/iterative_test.go	Removes legacy iterative perf test itself.
test/e2e/performance/config.yml	Removes legacy Prometheus scrape config for iterative test.
scripts/deploy-monitoring.sh	Expands monitoring deployment script (base + jaeger + pyroscope), configmaps, port-forward management, and trace env toggles.
make/deploy.mk	Adds Make targets wrapping monitoring deployment actions.
deployments/metrics-resources/prometheus-config.yaml	Adds scrape jobs for postgres exporter, cadvisor, and perf-test host metrics.
deployments/metrics/grafana-deployment.yaml	Adds Pyroscope datasource configuration for Grafana.
deployments/metrics/jaeger-deployment.yaml	Adds Jaeger all-in-one deployment and services.
deployments/metrics/pyroscope-deployment.yaml	Adds Pyroscope + Alloy deployment and RBAC for profiling discovery/scrape.
deployments/metrics/postgres-exporter-deployment.yaml	Adds Postgres exporter deployment/service/secret for DB telemetry.
deployments/metrics-resources/grafana-pyroscope-dashboard.json	Adds Pyroscope flamegraph dashboard for profiling.
deployments/metrics-resources/grafana-database-dashboard.json	Adds PostgreSQL dashboard for exporter metrics.
deployments/metrics-resources/grafana-package-sizes-dashboard.json	Tweaks existing package size dashboard panel sizing.
deployments/porch/3-porch-postgres-bundle.yaml	Grants pg_monitor to porch user for postgres-exporter compatibility.
deployments/porch/3-porch-server.yaml	Enables pprof port/annotations and adjusts memory limit for heavier perf workloads.
deployments/porch/2-function-runner.yaml	Enables pprof port/annotations for profiling function-runner.
deployments/porch/9-controllers.yaml	Enables pprof port/annotations for profiling controllers.
internal/telemetry/metrics.go	Adds new OTel metrics instruments and recording helpers (API calls, request counts, perf-test metrics, external repo ops).
internal/telemetry/metrics_test.go	Adds coverage for new API duration/request metrics recording path.
internal/telemetry/otel.go	Starts/stops profiling server as part of OTel lifecycle.
internal/telemetry/pprof.go	Adds optional pprof HTTP server controlled by env var.
internal/telemetry/pyroscope.go	Adds optional in-process Pyroscope profiling integration (env-driven).
pkg/registry/porch/packagerevision.go	Records API request counts and call durations for PackageRevision endpoints.
pkg/registry/porch/packagerevisionresources.go	Records API request counts and call durations for PackageRevisionResources endpoints.
pkg/registry/porch/packagerevision_approval.go	Records API request counts and call durations for approval endpoints.
pkg/externalrepo/git/git.go	Records external repo FETCH/PUSH operation durations and request counts via telemetry.
controllers/packagerevisions/pkg/controllers/packagerevision/render.go	Records controller operation metrics around render/update flow.
controllers/packagerevisions/pkg/controllers/packagerevision/packagerevision_controller.go	Records controller operation metrics for lifecycle/source reconciliation.
controllers/packagerevisions/pkg/controllers/packagerevision/ownership.go	Records controller operation metrics for deletion cleanup path.
pkg/apiserver/apiserver.go	Disables controller-runtime metrics server in a manager instance (BindAddress: "0").
func/server/server.go	Disables controller-runtime metrics server in function config manager instance (BindAddress: "0").
go.mod	Adds Pyroscope Go dependency and adjusts indirect requirements.
go.sum	Adds checksums for new Pyroscope-related dependencies.
.gitignore	Ignores CSV outputs and perf-test result files; adds Cursor IDE folder.
.env.template	Adds Grafana admin password example env var.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
91.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud