Commit 8eee3af
authored
fix: remove project label from projectstorage metrics to reduce cardinality (#558)
## Summary
- Remove the `project` label from all three `projectstorage_*` metrics
## Problem
The `project` label creates cardinality explosion in VictoriaMetrics
production storage (datum-cloud/infra#2113). PVCs are at 93% capacity.
| Metric | Before (per pod) | After (per pod) |
|--------|-----------------|-----------------|
| `projectstorage_first_ready_seconds` | 414 × 82 × 12 = 407K series |
82 × 12 = 984 series |
| `projectstorage_child_creations_total` | 414 × 82 = 34K series | 82
series |
| `projectstorage_reinitializing_errors_total` | 414 × 82 × 7 = 237K
series | 82 × 7 = 574 series |
| **Total** | **~678K / pod, ~6.1M across 9 pods** | **~1.6K / pod, ~14K
across 9 pods** |
**~430x reduction in cardinality.**
## What changed
- Removed `project` from label dimensions on all three metrics
- Removed `project` field from `instrumentedStorage` struct
- Updated `recordFirstReady`, `incrReinit`, and
`childCreations.WithLabelValues` call sites
The distribution by `resource_group` and `resource_kind` is the useful
signal for understanding storage init performance. Per-project
granularity is not actionable and is the source of the cardinality
problem.
## Test plan
- [ ] `go build ./internal/apiserver/storage/project/` passes
- [ ] Deploy to staging, verify metrics still emit with reduced labels
- [ ] Confirm VictoriaMetrics series count drops after old series expire1 file changed
Lines changed: 17 additions & 19 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | | - | |
| 38 | + | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
| 41 | + | |
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
51 | | - | |
| 51 | + | |
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | | - | |
| 60 | + | |
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | | - | |
73 | | - | |
| 72 | + | |
| 73 | + | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
| 76 | + | |
77 | 77 | | |
78 | | - | |
| 78 | + | |
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
| |||
107 | 107 | | |
108 | 108 | | |
109 | 109 | | |
110 | | - | |
111 | | - | |
112 | | - | |
| 110 | + | |
| 111 | + | |
113 | 112 | | |
114 | 113 | | |
115 | 114 | | |
116 | 115 | | |
117 | 116 | | |
118 | 117 | | |
119 | 118 | | |
120 | | - | |
| 119 | + | |
121 | 120 | | |
122 | 121 | | |
123 | 122 | | |
124 | | - | |
| 123 | + | |
125 | 124 | | |
126 | 125 | | |
127 | 126 | | |
| |||
239 | 238 | | |
240 | 239 | | |
241 | 240 | | |
242 | | - | |
243 | | - | |
244 | | - | |
245 | | - | |
246 | | - | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
247 | 245 | | |
248 | 246 | | |
249 | 247 | | |
250 | 248 | | |
251 | | - | |
| 249 | + | |
252 | 250 | | |
253 | 251 | | |
254 | 252 | | |
| |||
0 commit comments