diff --git a/FOCUS-1.4-PLAN.md b/FOCUS-1.4-PLAN.md new file mode 100644 index 000000000..198a27260 --- /dev/null +++ b/FOCUS-1.4-PLAN.md @@ -0,0 +1,195 @@ +# FOCUS 1.4 implementation plan + +> **Temporary working document** for the `flanakin/focus14` integration branch. This file tracks the design and work breakdown for adding FOCUS 1.4 support across the FinOps toolkit. It will be **removed before the `flanakin/focus14` → `dev` PR is marked ready for review**. Progress is mirrored in the umbrella PR checklist. + +## 1. Goal + +Add full FOCUS 1.4 support across the FinOps toolkit: a new `v1_4` hub schema (tables, functions, transforms), Power BI reports, documentation, open data metadata, AI/agent knowledge files, and tests — following the proven `v1_0` → `v1_2` upgrade pattern, staged as small PRs into `flanakin/focus14`, then one PR into `dev`. + +Microsoft Cost Management does **not** yet export FOCUS 1.4. This plan ships the `v1_4` hub schema converting all currently supported export versions (`1.0-preview(v1)`, `1.0`, `1.0r2`, `1.2-preview`, `1.2`) up to FOCUS 1.4, and pre-wires detection for native 1.4 exports. Export-side registration (schema mapping file, default export version) is staged as a follow-up gated on the Cost Management release. + +## 2. Grounding: what changed in FOCUS 1.2 → 1.4 + +Sources: FOCUS spec repo tags `v1.2`/`v1.3`/`v1.4` (tags are ground truth; `working_draft` already contains 1.5 drift), `CHANGELOG.md`, `ERRATA.md`, and `supporting_content/appendix/version_migration_guidance.md` (1.3→1.4 guide added 2026-07-09, post-1.4-tag). + +Both 1.2→1.3 and 1.3→1.4 are **backward-compatible releases** (no incompatible changes per spec changelog); the only removals are `ProviderName`/`PublisherName` (deprecated 1.3, removed 1.4). + +### CostAndUsage (Costs) — 65 columns in 1.4 (vs 57 in 1.2) + +New columns to add: + +| Column | Introduced | Type | Feature level | Nulls | Hubs source (today) | +|---|---|---|---|---|---| +| AllocatedMethodDetails | 1.3 | JSON | Recommended | ✔ | null (no CM source) | +| AllocatedMethodId | 1.3 | String | Conditional | ✔ | null | +| AllocatedResourceId | 1.3 | String | Conditional | ✔ | null | +| AllocatedResourceName | 1.3 | String | Conditional | ✔ | null | +| AllocatedTags | 1.3 | JSON | Conditional | ✔ | null | +| ContractApplied | 1.3 (format revised 1.4) | JSON | Conditional | ✔ | null | +| HostProviderName | 1.3 | String | **Mandatory** | ✔ | derived (see D3) | +| ServiceProviderName | 1.3 | String | **Mandatory** | ✘ | derived (see D3) | +| CommitmentProgramEligibilityDetails | 1.4 | JSON | Conditional | ✔ | null | +| InvoiceDetailId | 1.4 | String | Conditional | ✔ | null | + +Removed columns: **ProviderName**, **PublisherName** (successor mapping is semantic, not a rename — see D3). + +Changed: `InvoiceId` feature level Recommended → Conditional (1.4); `PricingCurrency` and `PricingCurrencyEffectiveCost` non-nullable (1.4 errata fixes); `BilledCost`/`EffectiveCost` normative requirements substantially tightened (1.4); several price/quantity columns gained null-consistency rules tied to `SkuPriceId` (1.3). **No allowed-value changes to any pre-existing enum column across 1.2→1.4** (verified by value-set diff) — no value remapping needed for existing data. + +### New FOCUS datasets (no Cost Management source yet) + +- **ContractCommitment** (1.3: 13 columns → 1.4: 30 columns) +- **BillingPeriod** (1.4: 6 columns) +- **InvoiceDetail** (1.4: 22 columns) + +### Attributes / metadata (docs + conformance impact only) + +1.4 removes `ColumnHandling` → `FocusColumnHandling` + `CustomColumnHandling`, removes `DiscountHandling` (→ appendix) and `InvoiceHandling` (→ `DeliveryHandling` + `DatasetCompleteness`); revises `CurrencyFormat` (ISO 4217 only — virtual currency allowance removed), `NumericFormat`, `UnitFormat`, `StringHandling`, `NullHandling`. 1.3 added DatasetInstance + Recency metadata groups. + +## 3. Design decisions + +### D1. Single `v1_4` hub schema; skip `v1_3` + +Hubs converts to the *latest* supported FOCUS version. 1.3 is a strict subset of 1.4 additions, Cost Management will never export 1.3, and both releases are backward-compatible. A `v1_3` schema layer would double the KQL surface for zero consumer value. **Decision: one new schema version, `v1_4`.** + +### D2. Replicate the v1_2 architecture exactly + +Per the v12 precedent (PR #1688 et al.): + +- `{ds}_raw` update policies switch to `{ds}_transform_v1_4()` → `{ds}_final_v1_4` (new). The v1_2 update policies are flipped to `"IsEnabled": false` in `IngestionSetup_v1_2.kql`; `_final_v1_2` tables are kept (historical data stays put; no data migration). +- `{ds}_transform_v1_2()` docstrings marked deprecated (same as v1_0 functions were). +- Hub DB: new `{ds}_v1_4()` functions union `_final_v1_4 ∪ _final_v1_2 ∪ _final_v1_0` (up-converting older rows); existing `{ds}_v1_0()`/`{ds}_v1_2()` functions extended to also union `_final_v1_4` **down-converted** (back-compat guarantee). +- `HubSetup_Latest.kql` (and the unversioned tail of the newest HubSetup file) repoints `{ds}()` → `{ds}_v1_4()`. +- New FOCUS 1.4 superset columns added to `{ds}_raw` in `IngestionSetup_RawTables.kql` with `// FOCUS 1.3+` / `// FOCUS 1.4+` annotations (existing convention). +- `decimal`→`real` and JSON→`dynamic` type conventions carry forward from v1_2. + +### D3. ProviderName / PublisherName → ServiceProviderName / HostProviderName + +The spec migration guide (lines 197–230) defines a context-dependent mapping, not a rename. For Microsoft-billed data: + +- **HostProviderName** = `'Microsoft'` (Microsoft operates the hosting infrastructure for all charges in CM exports). Nullable per spec, but always populated for our data. +- **ServiceProviderName** = `PublisherName` for Marketplace charges; `'Microsoft'` otherwise. (For native services PublisherName is already Microsoft, so `coalesce`-style logic collapses cleanly.) + +Down-conversion (v1_2/v1_0 views over `_final_v1_4` rows): `ProviderName = HostProviderName`, `PublisherName = ServiceProviderName`. This round-trips correctly for both native and Marketplace charges. Original values preserved via the existing `x_SourceValues` mechanism (#1767 precedent) — for removed columns, follow the removed-column convention (`checkString(..., old_X, '')`) so originals are always captured. + +> ⚠️ Note: a plain `ServiceProviderName = PublisherName` fallback is insufficient — the existing PublisherName fix-up can yield `''` (e.g., `x_PublisherCategory == 'Cloud Provider'` branch), violating the non-null mandate. Use the Marketplace-conditional mapping with a `'Microsoft'` fallback. Both #2126 and #2128 currently have this mapping **swapped** (Host↔Service inverted); fix must be coordinated across ingestion + hub + plugin docs. + +### D4. New FOCUS datasets in scope (revised after PR review) + +Original recommendation was to defer ContractCommitment, BillingPeriod, and InvoiceDetail (no Cost Management source). **Superseded by explicit owner direction** in PR #2126 review comments 3277231717/3277243626, which requested the tables — and the work already exists across the PR stack (raw tables, transforms, `_final_v1_4` tables, `{ds}_v1_4()` functions, unversioned aliases). **Decision: keep them in scope.** They remain empty until a data source ships; all docs/agent surfaces describing them must carry an explicit "no data until Cost Management ships an export" caveat. + +### D5. Export-side registration staged behind Cost Management + +The seams (all verified in code): + +1. `Exports/schemas/focuscost_1.4*.json` — column mapping for the actual CM export; **cannot be authored correctly until CM publishes the 1.4 export column list.** +2. `Exports/app.bicep` `schemaFiles` registration. +3. `ManagedExports/app.bicep` `exportDataVersions.focuscost` (currently `1.2-preview`). +4. `New-FinOpsCostExport.ps1` `$DatasetVersion` default + Pester tests + PS docs. +5. `open-data/dataset-metadata/FocusCost_1.4*.json` + example CSV. + +These ship as a **follow-up PR when CM announces FOCUS 1.4 exports** (tracked in the checklist as blocked). Meanwhile, `Costs_transform_v1_4()` pre-wires `x_SourceVersion` detection for 1.4 exports by column presence (e.g., `ServiceProviderName`), extending the existing heuristic (`IngestionSetup_v1_2.kql` ~L385), so day-one ingestion works once a schema file exists. The ETL pipeline already fails gracefully with an "upgrade" error for unknown schema versions. + +### D6. Testing: add a static KQL/regression Pester harness (new) + +There is **no existing KQL setup-script test coverage** (`HubsIngestionQueries.Tests.ps1` covers Recommendations query JSON only; the v1_2 rollout was manually verified — and shipped 4 regressions). Add `src/powershell/Tests/Unit/HubsSchemaSetup.Tests.ps1` (static analysis, no ADX needed), asserting: + +1. **Registration completeness**: every `IngestionSetup_v1_*.kql` / `HubSetup_v1_*.kql` file is listed in `.build.config` (Fabric bundle) *and* loaded in `Analytics/app.bicep` — prevents regression #1777 (v1_2 script missing from Fabric build). +2. **Version-string consistency**: within `*_v1_X.kql`, function/table/docstring references use the file's own version except in an explicit allowlist (cross-version unions, deprecation pointers) — prevents the recurring copy-paste defects (#e036c2cd; paired-file drift). +3. **Structural pairing**: each managed dataset has `{ds}_transform_v1_4()`, `{ds}_final_v1_4` table, and an update policy wiring them; exactly one schema version has enabled update policies. +4. **Column contract**: `_final_v1_4` Costs columns match the FOCUS 1.4 column inventory (golden list checked into the test) plus documented `x_` extensions; `_raw` superset ⊇ every transform input. + +Plus existing gates: `bicep build` via `Build-Toolkit`, `Test-PowerShell -Lint`, full Pester run, and a manual `Deploy-Toolkit -Build -Test` ADX validation pass before the umbrella PR is marked ready (ingest sample exports for **every** supported dataset version — prevents regression #1906, pre-1.2 data breaking in the new transform; and check hub-function join cardinality for duplicates — regression #1907). + +### D7. Docs shipped in-band, not lagging + +The v1_2 release left Power BI docs stale for months. Every code PR updates its own docs surface; the docs PR covers the schema/version matrices: `data-model.md` (schema version table + column tables), `upgrade.md`, `compatibility.md`, `data-processing.md`, `data-dictionary.md`, power-bi docs, changelog (Unreleased). `docs-mslearn` edits require manually setting `ms.date` to today. + +### D8. FOCUS conformance docs regenerated for 1.4 + +`Build-FocusConformance.ps1` defaults `$Branch = "v1.2"` → bump to `v1.4` and regenerate `docs-mslearn/focus/*` (metadata, convert, conformance-summary, conformance-full-report). The 1.4 attribute restructuring (removed/renamed attributes) means the conformance report layout must be re-validated, not just re-run. + +## 4. Work breakdown + +Small PRs targeting `flanakin/focus14`; stacked where a hard dependency exists. Existing draft PRs (2123, 2126, 2128, 2135, 2136, 2194) are mapped to these work items after a full review pass (§6) — reworked, restructured, or superseded as needed. + +| # | Work item | Depends on | Existing PR | Scope (files) | +|---|---|---|---|---| +| **W1** | Plan + umbrella PR | — | — | This doc; draft PR `flanakin/focus14` → `dev` | +| **W2** | Ingestion v1_4 | — | #2126 | `IngestionSetup_RawTables.kql` (superset columns), new `IngestionSetup_v1_4.kql` (transforms incl. D3 mapping + 1.4 source detection, `_final_v1_4` tables, update policies), `IngestionSetup_v1_2.kql` (disable policies, deprecate docstrings), `Analytics/app.bicep`, `.build.config`, changelog | +| **W3** | Hub functions v1_4 | W2 | #2128 | New `HubSetup_v1_4.kql`, `HubSetup_v1_0.kql`/`HubSetup_v1_2.kql` (union `_final_v1_4`, down-convert per D3), `HubSetup_Latest.kql`, `dashboard.json`, changelog | +| **W4** | KQL test harness | W2, W3 | #2136 (part) | New `HubsSchemaSetup.Tests.ps1` per D6; runs green against W2+W3 output | +| **W5** | Power BI | W3 | — | `kql/Shared.Dataset` TMDL repoint `Costs_v1_2`→`Costs_v1_4` (+ ReservationRecommendations), `storage/Shared.Dataset` `Has14` normalization + `expressions.tmdl` version inference, report.json "FOCUS 1.2" strings, `ftk_NormalizeSchema.pq` if needed, power-bi docs | +| **W6** | Core docs | W2, W3 | #2123 (upgrade.md) | `data-model.md`, `upgrade.md`, `compatibility.md`, `data-processing.md`, `data-dictionary.md`, hub README support matrix, changelog consolidation | +| **W7** | FOCUS conformance + open data | — | — | `Build-FocusConformance.ps1` branch bump, regenerated `docs-mslearn/focus/*`; open-data updates that are **not** CM-gated | +| **W8** | AI/agent surfaces | W3 | #2135 | copilot-studio knowledge files (`schema-reference.md` etc.), `agent-skills` references, claude-plugin output style, `Build-QueryCatalog`/`src/queries` `_v1_2` refs | +| **W9** | Recommendations enrichment | — | #2194 | `x_RecommendationCategory`/`x_RecommendationSubcategory` — independent feature; evaluated on its own merits | +| **W10** | *Blocked on CM 1.4 export release* | W2 | — | `focuscost_1.4*.json` schema, `Exports/app.bicep` registration, `ManagedExports` default, `New-FinOpsCostExport` default + tests + PS docs, open-data `FocusCost_1.4*.json` metadata + example CSV | + +Execution order: W2 → W3 → W4 (stacked); W5–W8 parallel after W3; W9 anytime; W10 blocked. + +### D9. KQL file-size limit (added after PR review) + +Bicep `loadTextContent()` caps embedded files at **131,072 characters**. `IngestionSetup_v1_4.kql` on the PR stack is ~128 K before enrichments; PR #2194's +55 lines already broke CI (`Error BCP184`). **Decision: split `IngestionSetup_v1_4.kql` into two scripts** (e.g., core managed datasets vs. supplemental FOCUS datasets), following the existing `OpenDataFunctions_*.kql` split precedent, registered in both `app.bicep` and `.build.config`. The W4 registration-completeness test must cover both files. + +## 5. Risks and lessons applied (from the v1_2 rollout) + +| Regression (v1_2) | Mitigation here | +|---|---| +| #1777 — `IngestionSetup_v1_2.kql` missing from `.build.config` (Fabric build shipped broken) | `.build.config` updated in W2 itself + W4 test asserts registration completeness | +| #1906 — `Costs_transform_v1_2()` broke on pre-1.2 exports | W2 transform handles every `x_SourceVersion`; manual ADX pass ingests all 5 export versions | +| #1907 — duplicate rows from Services open-data join in `Costs_v1_2()` | W3 review checks join cardinality; ADX pass includes row-count reconciliation raw vs final vs hub function | +| #e036c2cd — stale cross-version references in paired KQL files | W4 version-string consistency test | +| Power BI docs lagged release by months | D7: docs in-band per PR | + +## 6. Existing PR dispositions (reviewed 2026-07-15) + +Stack order (nothing merged yet; phases 3–5 were closed/superseded): **#2126 (phase1) → #2128 (phase2) → #2135 (phase6) → #2136 (phase7) → #2194 (phase9)**. #2123 (phase0) is independent. Later PRs' diffs currently include all upstream content; retarget each PR's base to its parent branch to shrink review surface. + +| PR | Disposition | Blocking fixes | +|---|---|---| +| #2123 → W6a | **Keep + fix** | 5 unaddressed 2026-06-29 comments (`sinc` typo, "into"→"in" ×2, "it"→"them", PR body GA/steps mismatch); decide forward-dated `v1_4` Learn content gating; **rescope as "upgrade procedure process docs"** — column-change tables for `upgrade.md`/`compatibility.md` etc. move to a new W6b docs PR | +| #2126 → W2 | **Keep + fix** | C1 Host/ServiceProviderName fallbacks swapped (L765/783); C2 C360 transforms (`ActualCosts`/`AmortizedCosts`) don't project the 22 new `Costs_raw` columns → transactional update policy breaks C360 ingestion (retrofit v1_0/v1_2 too, per repo convention); C3 v1_2 update policies not disabled + no deprecation docstrings; H1 no native-1.4 `x_SourceVersion` detection; M1 removed columns not always preserved in `x_SourceValues`; 12 unaddressed round-2 comments; D9 file split | +| #2128 → W3 | **Keep + fix** | F1 `Prices/Recommendations/Transactions/CommitmentDiscountUsage_v1_4()` missing `_final_v1_2` union arm (near-total data loss for existing hubs); F2 `HubSetup_v1_0/v1_2.kql` down-convert arms never added; F3 same provider-name swap in both up-convert arms; F4 remove duplicated unversioned tails (v1_4 L644-686 **and** stale v1_2 tail L552-588 — `HubSetup_Latest.kql` is single owner); F5 `dashboard.json` still `Costs_v1_2`; F7 opportunistic `take_any` on CDU Services lookup; 11 unaddressed comments; retarget base to phase1 | +| #2135 → W8 | **Rework** | C1 unresolved merge-conflict markers committed in `HubSetup_v1_4.kql` (~L283-296, from merge `676da5e8` — fix the merge resolution); C2 Service/HostProviderName descriptions swapped in `ftk-database-query.md` (other two files correct); M1 `// FOCUS 1.3+` comments on columns landing in v1_4 schema; M2 "no data yet" caveats for new dataset functions; copilot-studio knowledge files NOT covered — track separately in W8; retarget base to phase2 | +| #2136 → W4 | **Rework** | Missing D6 checks 2 (version-string consistency + allowlist — would have caught the conflict markers) and 3 (exactly-one-enabled-policy-version — would have caught C3); hardcoded v1_4 filenames → glob discovery; tighten substring-match tests (`_v1_0()` alias check, occurrence counts); D3 round-trip mapping untested; red-green validate new checks against current broken state | +| #2194 → W9 | **Keep + fix** | Bicep 131 K size break (CI-verified, → D9 split); 22 of ~23 hub-native recommendation queries silently fall to subcategory `Other` (map them); category/subcategory coherence rule + test; 6 open review comments; PR body test claims contradict CI (2006 passed / 1 failed) | + +## 7. Open questions (non-blocking; recommendations applied unless overridden) + +1. ~~Q1 (D4)~~ Resolved: new datasets in scope per owner review comments on #2126 (see D4). +2. ~~Q2 (D3)~~ Resolved: derivation confirmed by owner review comments 3309129381/3309175754 on #2126; both PRs currently implement it swapped — fix in W2/W3. +3. **Q3**: Target release for `flanakin/focus14` → `dev` (v14 shipped 2026-07; assume next release). Changelog entries land under "Unreleased" (stack currently uses "FinOps hubs v15.0.0" inside the commented Unreleased block — consistent). +4. ~~Q4~~ Resolved (owner: "just do it", 2026-07-15): `ContractCommitments_raw`/`_final_v1_4` reconciled to the 30 FOCUS 1.4 spec column IDs (12 renames + `ContractCommitmentDescription` + `ServiceProviderName` added), verified against the spec `v1.4` tag; tests, changelog, plugin/copilot docs, and schema docs updated across the stack. +5. **Q5 (back-port the 1.4 columns older exports don't carry — owner-directed, NOT deferred to an issue)**: Three FOCUS 1.4 `Costs` columns are projected with `column_ifexists()` and land null when ingesting pre-1.4 exports (`Costs_transform_v1_4()` in `IngestionSetup_v1_4.kql`). Per owner direction (2026-07-26, #2126 round-3 closeout) these must be owned by a concrete PR in this stack, not a research issue. Feasibility differs: + - **`CommitmentProgramEligibilityDetails`** (#2126 thread 3309107501) — *feasible now*. The `CommitmentDiscountEligibility.csv` open dataset (MeterId → spend/usage eligibility, ~121K rows) is the source; needs a materialized lookup + the FOCUS 1.4 JSON shape (verify against the spec `v1.4` tag). Owned by **W11a** — new PR stacked on the stack tail (phase9 head); owner: focus14-lead, after #2126 review closes. + - **`InvoiceDetailId`** (#2126 thread 3309133481) — *needs a new source*. Not in current Cost Management exports; requires a new invoice-details API (not available for all account types) plus a custom lookup. Owned by **W11b** (separate PR; blocked on the API — stays null until then). + - **`ContractCommitment*` per-row columns** (#2126 thread 3309131626) — *needs multiple sources*. Some data is in cost data, but not all; pulling from multiple APIs needs design. Owned by **W11c** (separate PR; stays null until then). + Consolidation: standalone issues #2217 and #2159 (native 1.4 export column research) both closed into this plan + umbrella #2207; W11a owner: focus14-lead (agreed 2026-07-26). + +## 8. Progress + +- [x] Research: FOCUS 1.2→1.4 spec delta (tags, changelog, errata, migration guides) +- [x] Research: FTK FOCUS-version surface inventory +- [x] Research: v1_0→v1_2 precedent + post-release regressions +- [x] W1: Plan committed; umbrella draft PR opened ([#2207](https://github.com/microsoft/finops-toolkit/pull/2207)) +- [x] Review 6 existing PRs; dispositions recorded in §6 +- [x] W2: Ingestion v1_4 — #2126 fixed (`30f926d3`: provider mapping, C360 retrofit, v1_2 policies disabled, 1.4 detection, D9 file split; build green) — **awaiting review** +- [x] W3: Hub functions v1_4 — #2128 fixed (`cd2873b7`+`7181c9aa`: v1_2 union arms, v1_0/v1_2 down-convert arms, mapping consistency, tails removed, dashboard repoint; build green; base retargeted to phase1) — **awaiting review** +- [x] W4: KQL test harness — #2136 reworked (`778080f2`+`9eb79cad`: 207 static tests, D6 checks 1-4, all red-green validated; full suite 2047/0) — **awaiting review** +- [x] W5: Power BI — new [#2209](https://github.com/microsoft/finops-toolkit/pull/2209) (kql repoint + 22 model columns, storage `Has14` normalization, ProviderName/PublisherName kept as derived columns to protect visuals/DAX; 2 `.pbix` binaries flagged for release-time regen) — **awaiting review** +- [x] W6: Core docs — W6a #2123 fixed (`463c2735`); W6b new [#2211](https://github.com/microsoft/finops-toolkit/pull/2211) (upgrade/compatibility/data-processing/data-dictionary; 111/0 column cross-check) — **awaiting review** +- [x] W7: Conformance — new [#2208](https://github.com/microsoft/finops-toolkit/pull/2208) (`Build-FocusConformance.ps1` fixed for 1.3+ spec layout + curation preservation; 1,021 new 1.3/1.4 requirements marked Not Evaluated, no invented claims) — **awaiting review** +- [x] W8: AI/agent surfaces — #2135 reworked (`968207f4`+`0824600c`: conflict markers removed, provider semantics fixed, no-data caveats); W8b new [#2210](https://github.com/microsoft/finops-toolkit/pull/2210) (copilot-studio knowledge 175/175 column check, query catalog + removed-column migration) — **awaiting review** +- [x] W9: Recommendations enrichment — #2194 fixed (`503d3a63`+`a6c91087`+`47b406ed`: 22/23 queries mapped to real subcategories, taxonomy coherence datatable + tests, symmetric defaults; suite 2093/0) — **awaiting review** +- [x] Dev sync: `origin/dev` merged through the entire stack (merge commits only) — brings PR-environment workflows (#2033) and dev's KQL/PBI fixes; dev's v1_2-era fixes (#2151 strcat OOM, #2189 price-lookup max()) ported into the v1_4 transforms with a version-normalized parity diff +- [ ] W11: Back-port 1.4 columns from older exports (Q5) — **W11a** `CommitmentProgramEligibilityDetails` via `CommitmentDiscountEligibility.csv` (feasible now; new PR on stack tail, owner focus14-lead); **W11b** `InvoiceDetailId` (blocked on new invoice API); **W11c** `ContractCommitment*` (needs multi-API design). Null until landed, per Q5. +- [ ] ADX validation via **PR environments** (replaces the manual pass): "Hubs + ADX (managed)" checked on #2194 (full stack) and #2207 (umbrella); deploys queue for `ftk-pr` environment approval. Validate: ingest samples for all 5 export versions, row-count/cardinality reconciliation raw → final → hub functions +- [ ] Remove this plan file; mark umbrella PR ready +- [ ] *Blocked*: W10 export-side registration (awaiting Cost Management FOCUS 1.4 export) + +### Recommended review/merge order + +1. [#2123](https://github.com/microsoft/finops-toolkit/pull/2123) (independent process docs) +2. Stack, in order: [#2126](https://github.com/microsoft/finops-toolkit/pull/2126) → [#2128](https://github.com/microsoft/finops-toolkit/pull/2128) → [#2135](https://github.com/microsoft/finops-toolkit/pull/2135) → [#2136](https://github.com/microsoft/finops-toolkit/pull/2136) → [#2194](https://github.com/microsoft/finops-toolkit/pull/2194) (bases auto-retarget as parents merge) +3. Stacked on phase7: [#2209](https://github.com/microsoft/finops-toolkit/pull/2209) (Power BI), [#2210](https://github.com/microsoft/finops-toolkit/pull/2210) (copilot-studio/queries), [#2211](https://github.com/microsoft/finops-toolkit/pull/2211) (schema docs). Note: #2194, #2209 all append to `changelog.md` from the same base — merge conflicts there are additive; keep both sides per repo convention. +4. [#2208](https://github.com/microsoft/finops-toolkit/pull/2208) (conformance, independent) diff --git a/docs-mslearn/toolkit/hubs/data-model.md b/docs-mslearn/toolkit/hubs/data-model.md index e06a19519..d9b5cf6c8 100644 --- a/docs-mslearn/toolkit/hubs/data-model.md +++ b/docs-mslearn/toolkit/hubs/data-model.md @@ -3,7 +3,7 @@ title: FinOps hubs data model description: Learn about the tables and functions available in FinOps hubs to build your own queries, reports, and dashboards. author: flanakin ms.author: micflan -ms.date: 04/01/2026 +ms.date: 07/26/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit @@ -52,15 +52,22 @@ Note the use of "Data Explorer" covers both Azure Data Explorer and Microsoft Fa ## Schema version -One of the goals of the FinOps hubs data model is to guarantee backwards compatibility. To support this, each FinOps hub release uses a specific schema version which aligns to a specific FOCUS version. The **schema version** defines the columns, data types, and allowed values in the tables and functions for each [managed dataset](#managed-datasets-in-finops-hubs). +One of the goals of the FinOps hubs data model is to guarantee backwards compatibility. To support this, each FinOps hub release uses a specific schema version which aligns to a specific FOCUS version. For instance, the `v1_4` schema version aligns with FOCUS 1.4 and the `v1_2` schema with FOCUS 1.2. The **schema version** defines the columns, data types, and allowed values in the tables and functions for each [managed dataset](#managed-datasets-in-finops-hubs). + +During ingestion, data is transformed to the schema version of your current FinOps hub release, regardless of the FOCUS version of the source data. For instance, if your hub uses the `v1_4` schema, incoming FOCUS 1.0 data is converted to FOCUS 1.4 as it's ingested. Data ingested by an earlier release stays in the schema version it was originally ingested into. For instance, data ingested when your hub used the `v1_2` schema remains in the `v1_2` format even after you upgrade to a newer schema version. + +The functions in the `Hub` database pull from all tables and transform them into a specific version. For instance, `Costs_v1_0` will convert `v1_4` and `v1_2` data back to the `v1_0` format, ensuring backwards compatibility despite upgrading to newer versions. When you're ready to upgrade, you can selectively upgrade each report or integration to use `Costs_v1_2` or `Costs_v1_4` without breaking others. However, if you want a quick, convenience function that always returns the latest version, you can use the unversioned functions, like `Costs` and `Prices`. Unversioned functions are convenient for ad hoc use while versioned functions are recommended for integration points that require a consistent format after upgrades. The following table indicates the schema version for each FinOps hub release and which FOCUS version they align to. | Release | Schema | FOCUS version | | ------- | ------ | ------------- | -| 12+ | `v1_2` | 1.2 | +| 15+ | `v1_4` | 1.4 | +| 12-14 | `v1_2` | 1.2 | | 0.7-11 | `v1_0` | 1.0 | +Cost Management export availability for a given FOCUS version is separate from FinOps hub support for that version. FinOps hubs transforms export data to the schema version of your current release during ingestion, whether the export uses an older or newer FOCUS version than that schema. +
## Managed datasets in FinOps hubs @@ -81,7 +88,7 @@ Managed datasets also include the following assets for FinOps hubs with Data Exp - An unversioned function in the **Hub** database (for example, **Costs()**). - A table in Power BI KQL reports that wraps the corresponding versioned function. -During data ingestion, FinOps hubs transform data to the latest supported [schema version](#schema-version). Unversioned functions, like **Costs()**, use the latest schema version. Unversioned functions, like **Costs_v1_0()**, are backwards compatible, should remain consistent, and do not need to change when upgrading your FinOps hub instance. To learn more about the data ingestion process, see [How data is processed in FinOps hubs](data-processing.md). +During data ingestion, FinOps hubs transform data to the latest supported [schema version](#schema-version). Unversioned functions, like **Costs()**, use the latest schema version. Versioned functions, like **Costs_v1_0()**, are backwards compatible, should remain consistent, and do not need to change when upgrading your FinOps hub instance. To learn more about the data ingestion process, see [How data is processed in FinOps hubs](data-processing.md). When querying data in FinOps hubs, always use the **Hub** database and avoid working with the tables and functions in the **Ingestion** database. Use unversioned functions for ad-hoc analysis or reports that do not require long-term backwards compatibility. Use the versioned functions for reports or systems that do require backwards compatibility and you do not want to be impacted by FinOps hub updates, which may change column names, data types, and values. diff --git a/src/templates/finops-hub/docs/README.md b/src/templates/finops-hub/docs/README.md index 2cff88bf9..a639316eb 100644 --- a/src/templates/finops-hub/docs/README.md +++ b/src/templates/finops-hub/docs/README.md @@ -2,10 +2,10 @@ ## Data ingestion workflow -- All data is ingested into tables named "*_raw". +- All data is ingested into tables named "\*\_raw". - Raw tables are declared in [IngestionSetup_raw.kql](../modules/scripts/IngestionSetup_raw.kql). - These tables have a union schema to support multiple sources and versions. -- All data is transformed to the latest FOCUS schema using an update policy into a "*_final_vX_Y" table named after the version (for example, "1.0" = "_v1_0"). Final tables, Transform functions, and update policies are in the versioned setup files: +- All data is transformed to the latest FOCUS schema using an update policy into a "\*\_final_vX_Y" table named after the version (for example, "1.0" = "\_v1_0"). Final tables, Transform functions, and update policies are in the versioned setup files: - [IngestionSetup_v1_0.kql](../modules/scripts/IngestionSetup_v1_0.kql) - Data ingestion from previous version of hubs will remain in the versioned tables. - Data is read from versioned functions in the Hub database. See HubSetup_vX_Y.kql in the [modules/scripts folder](../modules/scripts) for details. @@ -17,27 +17,43 @@ - Each versioned function unions data from versioned tables in the Ingestion database and transforms it to that FOCUS version for back compat. - Consumers should use the unversioned function for the latest and the versioned functions for back compat. -To add a new FOCUS versions: +To add a new FOCUS version: +0. Confirm dependencies + 1. Ensure the FOCUS specification has been ratified for the target version. Support for working draft FOCUS versions should not be shipped before official ratification to prevent customer churn since FOCUS working drafts may change without notice. + 2. Check whether Microsoft Cost Management has shipped a matching FOCUS export dataset version. The hub depends on a `focuscost_X.Y.json` schema mapping file in [Microsoft.CostManagement/Exports/schemas](../modules/Microsoft.CostManagement/Exports/schemas/) when ingesting from Cost Management. If the export is not yet available, the hub schema can still ship as GA. Note the gap in the changelog so adopters know what additional setup will be required. 1. Add schema mapping file 1. Create new schema mapping file for the Cost Management export dataset version in the schemas folder 2. Add file to file upload list in [storage.bicep](../modules/storage.bicep) 2. Update ingestion database scripts - 1. Add new columns to the *_raw tables per dataset in [IngestionSetup_RawTables.kql](../modules/scripts/IngestionSetup_RawTables.kql) + 1. Add new columns to the \*\_raw tables per dataset in [IngestionSetup_RawTables.kql](../modules/scripts/IngestionSetup_RawTables.kql) 2. Save a copy of the latest version of the IngestionSetup_vX_Y.kql using the latest FOCUS version - If updating the same version, increment the release number (e.g., `r2`) 3. Rename all functions, tables, and policies in the new file to the new version (leave the old as-is) - 4. Update the *_final_vX_Y tables to account for any new columns - 5. Update the *_transform_vX_Y functions to account for any new columns - 6. Update the script to delete the old update policy for the *_raw tables + 4. Update the \*\_final_vX_Y tables to account for any new columns + 5. Update the \*\_transform_vX_Y functions to account for any new columns + 6. Update the script to delete the old update policy for the \*\_raw tables 7. Confirm the new file has the update policy set to use the latest version of the transform function and final table 3. Update hub database scripts 1. Add new FOCUS version section after the latest version section and before existing version sections - 2. Create new *_vX_Y functions per dataset that transforms older data to the new FOCUS version - 3. Update the unversioned functions to use the new *_vX_Y functions - 4. Update older versioned functions to also pull from the new *_vX_Y functions and transform to the old schema + 2. Create new \*\_vX_Y functions per dataset that transforms older data to the new FOCUS version + 3. Update the unversioned functions to use the new \*\_vX_Y functions + 4. Update older versioned functions to also pull from the new \*\_vX_Y functions and transform to the old schema 4. Update reports and dashboards 1. Update the storage reports to use the new columns 2. Update the KQL reports to use the new versioned functions 3. Update the ADX dashboard to use the new versioned functions 4. Update the FOCUS queries in the best practices library to use the new versioned functions +5. Update open-data metadata + 1. Create a `FocusCost_.json` file in [src/open-data/dataset-metadata](../../../open-data/dataset-metadata/) for each Cost Management cost export version. `` must match the precise Cost Management export version, including `-preview`, if applicable (for example, `FocusCost_1.2-preview.json`). + 2. Create a `FinOpsHubs__.json` file in [src/open-data/dataset-metadata](../../../open-data/dataset-metadata/) for each FinOps hub managed dataset schema version. + 3. Mirror the schema details (columns, types, descriptions) from the matching Cost Management export or FinOps hubs schema so downstream consumers see consistent metadata. +6. Update plugin skill files + 1. Refresh the FOCUS schema and function references in the following files so plugin guidance does not go stale: + - [src/templates/agent-skills/finops-toolkit/references/finops-hubs.md](../../agent-skills/finops-toolkit/references/finops-hubs.md) + - [src/templates/agent-skills/finops-toolkit/references/finops-hubs-deployment.md](../../agent-skills/finops-toolkit/references/finops-hubs-deployment.md) + - [src/templates/agent-skills/azure-cost-management/references/azure-cost-exports.md](../../agent-skills/azure-cost-management/references/azure-cost-exports.md) + - [src/templates/claude-plugin/agents/ftk-database-query.md](../../claude-plugin/agents/ftk-database-query.md) + - [src/templates/claude-plugin/output-styles/ftk-output-style.md](../../claude-plugin/output-styles/ftk-output-style.md) +7. Update changelog + 1. Add an entry under the next version in [docs-mslearn/toolkit/changelog.md](../../../../docs-mslearn/toolkit/changelog.md) describing the new FOCUS version support and any preview status.