fix(grafana): Resolve issue with add-grafana make command & add new K8s dashboard (#1018)

## Summary

- Moved Grafana resources (`grafana.yaml`, dashboards, RBAC) into the
`with-grafana` overlay
- Fixed Grafana Prometheus auth: init container injects SA token into
datasource config at pod startup
- Fixed PVC mount path (`/var/lib/grafana` instead of
`/var/lib/grafana/data`) so `grafana.db` persists across restarts
- Removed `GF_SECURITY_ADMIN_PASSWORD` env var that was resetting
password on every restart
- Added K8s infrastructure dashboards (cluster, nodes, namespace, pods)
- PVC is managed separately from kustomize so `make clean-observability`
preserves Grafana data

## Dashboards

| Dashboard | Screenshot | Description |
|-----------|------------|-------------|
| K8s Cluster Monitoring |
![](https://github.com/user-attachments/assets/077fac80-5995-4235-b634-c6f8d77e0e75)
| Cluster-level CPU, memory, network |
| K8s Nodes |
![](https://github.com/user-attachments/assets/342e8ccd-c9bc-4266-b2a4-29b8c5923b9e)
| Node-level resource usage |
| K8s Namespace |
![](https://github.com/user-attachments/assets/726da1b3-6390-46a6-8a3d-6fe23facd0fc)|
Namespace-level resource usage |
| K8s Pods |
![](https://github.com/user-attachments/assets/7f5ef084-d8d6-40db-9f1d-0252b0cbc281)
| Pod-level resource usage |

## Test plan

- [x] `make add-grafana` deploys without errors
- [x] `make clean-observability` removes stack but preserves PVC
- [x] Grafana password persists across pod restarts
- [x] All dashboards load and show data from Prometheus

> Note: These charts are taken from community dashboards like
https://github.com/dotdc/grafana-dashboards-kubernetes, so some data
might be missing depending on the cluster setup. I'll be addressing them
in another change request.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Introduced pre-provisioned Grafana dashboards for Kubernetes cluster
and pod monitoring, automatically deployed without manual configuration.
* Added simplified Grafana installation workflow using the `make
add-grafana` command.

* **Documentation**
  * Updated installation and cleanup instructions for Grafana.
  * Added comprehensive guide for managing and customizing dashboards.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Rahul Shetty <rashetty@redhat.com>
Co-authored-by: Jeremy Eder <jeder@redhat.com>

Author: 2 people

Makefile

@@ -288,15 +288,17 @@ deploy-observability: ## Deploy observability (OTel + OpenShift Prometheus)
 
 add-grafana: ## Add Grafana on top of observability stack
 	@echo "$(COLOR_BLUE)▶$(COLOR_RESET) Adding Grafana..."
+	@kubectl apply -f components/manifests/observability/overlays/with-grafana/grafana-pvc.yaml
 	@kubectl apply -k components/manifests/observability/overlays/with-grafana/
 	@echo "$(COLOR_GREEN)✓$(COLOR_RESET) Grafana deployed"
 	@echo "  Create route: oc create route edge grafana --service=grafana -n $(NAMESPACE)"
 
-clean-observability: ## Remove observability components
+clean-observability: ## Remove observability components (preserves Grafana PVC)
 	@echo "$(COLOR_BLUE)▶$(COLOR_RESET) Removing observability..."
 	@kubectl delete -k components/manifests/observability/overlays/with-grafana/ 2>/dev/null || true
 	@kubectl delete -k components/manifests/observability/ 2>/dev/null || true
 	@echo "$(COLOR_GREEN)✓$(COLOR_RESET) Observability removed"
+	@echo "  To also delete Grafana data: kubectl delete pvc grafana-storage -n $(NAMESPACE)"
 
 grafana-dashboard: ## Open Grafana (create route first)
 	@echo "$(COLOR_BLUE)▶$(COLOR_RESET) Opening Grafana..."

components/manifests/observability/README.md

@@ -40,11 +40,14 @@ Open **OpenShift Console → Observe → Metrics** and query:
 If you want custom dashboards:
 
 ```bash
-# Add Grafana overlay
+make add-grafana
+
+# Or manually
+kubectl apply -f components/manifests/observability/overlays/with-grafana/grafana-pvc.yaml
 kubectl apply -k components/manifests/observability/overlays/with-grafana/
 ```
 
-**Adds**: Grafana (additional 128MB) - still uses OpenShift Prometheus
+**Adds**: Grafana (additional 128MB) with pre-provisioned dashboards - still uses OpenShift Prometheus
 
 **Access Grafana**:
 ```bash
@@ -53,10 +56,10 @@ oc create route edge grafana --service=grafana -n ambient-code
 
 # Get URL
 oc get route grafana -n ambient-code -o jsonpath='{.spec.host}'
-# Login: admin/admin
+# Login: admin/admin (change on first login)
 ```
 
-**Import dashboard**: Upload `dashboards/ambient-operator-dashboard.json` in Grafana UI
+**Dashboards** are provisioned automatically from `overlays/with-grafana/dashboards/`. See [dashboards/README.md](./overlays/with-grafana/dashboards/README.md) for how to add new ones.
 
 ---
 
@@ -185,6 +188,6 @@ EOF
 ## Cleanup
 
 ```bash
-kubectl delete -k components/manifests/observability/overlays/with-grafana/  # If Grafana deployed
-kubectl delete -k components/manifests/observability/
+make clean-observability                    # Removes stack but preserves Grafana PVC
+kubectl delete pvc grafana-storage -n ambient-code  # Also delete Grafana data
 ```

components/manifests/observability/base/kustomization.yaml

@@ -0,0 +1,8 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ambient-code
+
+resources:
+  - otel-collector.yaml
+  - servicemonitor.yaml