From f6036480e4e22cfbd7b2b867442229e645e6c16c Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Wed, 15 Jul 2026 12:11:43 -0700 Subject: [PATCH 1/8] docs: add FOCUS 1.4 implementation plan (temporary working doc) Grounded in FOCUS spec tags v1.2-v1.4, the v1_0->v1_2 rollout precedent, and a full inventory of FOCUS-version seams across the toolkit. Tracks design decisions, work breakdown, testing strategy, and progress for the flanakin/focus14 integration branch. Removed before merge to dev. Co-Authored-By: Claude Fable 5 --- FOCUS-1.4-PLAN.md | 174 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 FOCUS-1.4-PLAN.md diff --git a/FOCUS-1.4-PLAN.md b/FOCUS-1.4-PLAN.md new file mode 100644 index 000000000..12508e582 --- /dev/null +++ b/FOCUS-1.4-PLAN.md @@ -0,0 +1,174 @@ +# FOCUS 1.4 implementation plan + +> **Temporary working document** for the `flanakin/focus14` integration branch. This file tracks the design and work breakdown for adding FOCUS 1.4 support across the FinOps toolkit. It will be **removed before the `flanakin/focus14` → `dev` PR is marked ready for review**. Progress is mirrored in the umbrella PR checklist. + +## 1. Goal + +Add full FOCUS 1.4 support across the FinOps toolkit: a new `v1_4` hub schema (tables, functions, transforms), Power BI reports, documentation, open data metadata, AI/agent knowledge files, and tests — following the proven `v1_0` → `v1_2` upgrade pattern, staged as small PRs into `flanakin/focus14`, then one PR into `dev`. + +Microsoft Cost Management does **not** yet export FOCUS 1.4. This plan ships the `v1_4` hub schema converting all currently supported export versions (`1.0-preview(v1)`, `1.0`, `1.0r2`, `1.2-preview`, `1.2`) up to FOCUS 1.4, and pre-wires detection for native 1.4 exports. Export-side registration (schema mapping file, default export version) is staged as a follow-up gated on the Cost Management release. + +## 2. Grounding: what changed in FOCUS 1.2 → 1.4 + +Sources: FOCUS spec repo tags `v1.2`/`v1.3`/`v1.4` (tags are ground truth; `working_draft` already contains 1.5 drift), `CHANGELOG.md`, `ERRATA.md`, and `supporting_content/appendix/version_migration_guidance.md` (1.3→1.4 guide added 2026-07-09, post-1.4-tag). + +Both 1.2→1.3 and 1.3→1.4 are **backward-compatible releases** (no incompatible changes per spec changelog); the only removals are `ProviderName`/`PublisherName` (deprecated 1.3, removed 1.4). + +### CostAndUsage (Costs) — 65 columns in 1.4 (vs 57 in 1.2) + +New columns to add: + +| Column | Introduced | Type | Feature level | Nulls | Hubs source (today) | +|---|---|---|---|---|---| +| AllocatedMethodDetails | 1.3 | JSON | Recommended | ✔ | null (no CM source) | +| AllocatedMethodId | 1.3 | String | Conditional | ✔ | null | +| AllocatedResourceId | 1.3 | String | Conditional | ✔ | null | +| AllocatedResourceName | 1.3 | String | Conditional | ✔ | null | +| AllocatedTags | 1.3 | JSON | Conditional | ✔ | null | +| ContractApplied | 1.3 (format revised 1.4) | JSON | Conditional | ✔ | null | +| HostProviderName | 1.3 | String | **Mandatory** | ✔ | derived (see D3) | +| ServiceProviderName | 1.3 | String | **Mandatory** | ✘ | derived (see D3) | +| CommitmentProgramEligibilityDetails | 1.4 | JSON | Conditional | ✔ | null | +| InvoiceDetailId | 1.4 | String | Conditional | ✔ | null | + +Removed columns: **ProviderName**, **PublisherName** (successor mapping is semantic, not a rename — see D3). + +Changed: `InvoiceId` feature level Recommended → Conditional (1.4); `PricingCurrency` and `PricingCurrencyEffectiveCost` non-nullable (1.4 errata fixes); `BilledCost`/`EffectiveCost` normative requirements substantially tightened (1.4); several price/quantity columns gained null-consistency rules tied to `SkuPriceId` (1.3). **No allowed-value changes to any pre-existing enum column across 1.2→1.4** (verified by value-set diff) — no value remapping needed for existing data. + +### New FOCUS datasets (no Cost Management source yet) + +- **ContractCommitment** (1.3: 13 columns → 1.4: 30 columns) +- **BillingPeriod** (1.4: 6 columns) +- **InvoiceDetail** (1.4: 22 columns) + +### Attributes / metadata (docs + conformance impact only) + +1.4 removes `ColumnHandling` → `FocusColumnHandling` + `CustomColumnHandling`, removes `DiscountHandling` (→ appendix) and `InvoiceHandling` (→ `DeliveryHandling` + `DatasetCompleteness`); revises `CurrencyFormat` (ISO 4217 only — virtual currency allowance removed), `NumericFormat`, `UnitFormat`, `StringHandling`, `NullHandling`. 1.3 added DatasetInstance + Recency metadata groups. + +## 3. Design decisions + +### D1. Single `v1_4` hub schema; skip `v1_3` + +Hubs converts to the *latest* supported FOCUS version. 1.3 is a strict subset of 1.4 additions, Cost Management will never export 1.3, and both releases are backward-compatible. A `v1_3` schema layer would double the KQL surface for zero consumer value. **Decision: one new schema version, `v1_4`.** + +### D2. Replicate the v1_2 architecture exactly + +Per the v12 precedent (PR #1688 et al.): + +- `{ds}_raw` update policies switch to `{ds}_transform_v1_4()` → `{ds}_final_v1_4` (new). The v1_2 update policies are flipped to `"IsEnabled": false` in `IngestionSetup_v1_2.kql`; `_final_v1_2` tables are kept (historical data stays put; no data migration). +- `{ds}_transform_v1_2()` docstrings marked deprecated (same as v1_0 functions were). +- Hub DB: new `{ds}_v1_4()` functions union `_final_v1_4 ∪ _final_v1_2 ∪ _final_v1_0` (up-converting older rows); existing `{ds}_v1_0()`/`{ds}_v1_2()` functions extended to also union `_final_v1_4` **down-converted** (back-compat guarantee). +- `HubSetup_Latest.kql` (and the unversioned tail of the newest HubSetup file) repoints `{ds}()` → `{ds}_v1_4()`. +- New FOCUS 1.4 superset columns added to `{ds}_raw` in `IngestionSetup_RawTables.kql` with `// FOCUS 1.3+` / `// FOCUS 1.4+` annotations (existing convention). +- `decimal`→`real` and JSON→`dynamic` type conventions carry forward from v1_2. + +### D3. ProviderName / PublisherName → ServiceProviderName / HostProviderName + +The spec migration guide (lines 197–230) defines a context-dependent mapping, not a rename. For Microsoft-billed data: + +- **HostProviderName** = `'Microsoft'` (Microsoft operates the hosting infrastructure for all charges in CM exports). Nullable per spec, but always populated for our data. +- **ServiceProviderName** = `PublisherName` for Marketplace charges; `'Microsoft'` otherwise. (For native services PublisherName is already Microsoft, so `coalesce`-style logic collapses cleanly.) + +Down-conversion (v1_2/v1_0 views over `_final_v1_4` rows): `ProviderName = HostProviderName`, `PublisherName = ServiceProviderName`. This round-trips correctly for both native and Marketplace charges. Original values preserved via the existing `x_SourceValues` mechanism (#1767 precedent) when changed. + +### D4. New FOCUS datasets deferred (recommended; open question Q1) + +ContractCommitment, BillingPeriod, and InvoiceDetail have no Cost Management export and no hub data source. Creating empty tables/functions adds deployment surface and doc burden with no consumer value, and the eventual CM shape may differ. **Decision: do not create hub tables for them in this effort.** Document the position in `data-model.md`. Revisit when a data source exists. + +### D5. Export-side registration staged behind Cost Management + +The seams (all verified in code): + +1. `Exports/schemas/focuscost_1.4*.json` — column mapping for the actual CM export; **cannot be authored correctly until CM publishes the 1.4 export column list.** +2. `Exports/app.bicep` `schemaFiles` registration. +3. `ManagedExports/app.bicep` `exportDataVersions.focuscost` (currently `1.2-preview`). +4. `New-FinOpsCostExport.ps1` `$DatasetVersion` default + Pester tests + PS docs. +5. `open-data/dataset-metadata/FocusCost_1.4*.json` + example CSV. + +These ship as a **follow-up PR when CM announces FOCUS 1.4 exports** (tracked in the checklist as blocked). Meanwhile, `Costs_transform_v1_4()` pre-wires `x_SourceVersion` detection for 1.4 exports by column presence (e.g., `ServiceProviderName`), extending the existing heuristic (`IngestionSetup_v1_2.kql` ~L385), so day-one ingestion works once a schema file exists. The ETL pipeline already fails gracefully with an "upgrade" error for unknown schema versions. + +### D6. Testing: add a static KQL/regression Pester harness (new) + +There is **no existing KQL setup-script test coverage** (`HubsIngestionQueries.Tests.ps1` covers Recommendations query JSON only; the v1_2 rollout was manually verified — and shipped 4 regressions). Add `src/powershell/Tests/Unit/HubsSchemaSetup.Tests.ps1` (static analysis, no ADX needed), asserting: + +1. **Registration completeness**: every `IngestionSetup_v1_*.kql` / `HubSetup_v1_*.kql` file is listed in `.build.config` (Fabric bundle) *and* loaded in `Analytics/app.bicep` — prevents regression #1777 (v1_2 script missing from Fabric build). +2. **Version-string consistency**: within `*_v1_X.kql`, function/table/docstring references use the file's own version except in an explicit allowlist (cross-version unions, deprecation pointers) — prevents the recurring copy-paste defects (#e036c2cd; paired-file drift). +3. **Structural pairing**: each managed dataset has `{ds}_transform_v1_4()`, `{ds}_final_v1_4` table, and an update policy wiring them; exactly one schema version has enabled update policies. +4. **Column contract**: `_final_v1_4` Costs columns match the FOCUS 1.4 column inventory (golden list checked into the test) plus documented `x_` extensions; `_raw` superset ⊇ every transform input. + +Plus existing gates: `bicep build` via `Build-Toolkit`, `Test-PowerShell -Lint`, full Pester run, and a manual `Deploy-Toolkit -Build -Test` ADX validation pass before the umbrella PR is marked ready (ingest sample exports for **every** supported dataset version — prevents regression #1906, pre-1.2 data breaking in the new transform; and check hub-function join cardinality for duplicates — regression #1907). + +### D7. Docs shipped in-band, not lagging + +The v1_2 release left Power BI docs stale for months. Every code PR updates its own docs surface; the docs PR covers the schema/version matrices: `data-model.md` (schema version table + column tables), `upgrade.md`, `compatibility.md`, `data-processing.md`, `data-dictionary.md`, power-bi docs, changelog (Unreleased). `docs-mslearn` edits require manually setting `ms.date` to today. + +### D8. FOCUS conformance docs regenerated for 1.4 + +`Build-FocusConformance.ps1` defaults `$Branch = "v1.2"` → bump to `v1.4` and regenerate `docs-mslearn/focus/*` (metadata, convert, conformance-summary, conformance-full-report). The 1.4 attribute restructuring (removed/renamed attributes) means the conformance report layout must be re-validated, not just re-run. + +## 4. Work breakdown + +Small PRs targeting `flanakin/focus14`; stacked where a hard dependency exists. Existing draft PRs (2123, 2126, 2128, 2135, 2136, 2194) are mapped to these work items after a full review pass (§6) — reworked, restructured, or superseded as needed. + +| # | Work item | Depends on | Existing PR | Scope (files) | +|---|---|---|---|---| +| **W1** | Plan + umbrella PR | — | — | This doc; draft PR `flanakin/focus14` → `dev` | +| **W2** | Ingestion v1_4 | — | #2126 | `IngestionSetup_RawTables.kql` (superset columns), new `IngestionSetup_v1_4.kql` (transforms incl. D3 mapping + 1.4 source detection, `_final_v1_4` tables, update policies), `IngestionSetup_v1_2.kql` (disable policies, deprecate docstrings), `Analytics/app.bicep`, `.build.config`, changelog | +| **W3** | Hub functions v1_4 | W2 | #2128 | New `HubSetup_v1_4.kql`, `HubSetup_v1_0.kql`/`HubSetup_v1_2.kql` (union `_final_v1_4`, down-convert per D3), `HubSetup_Latest.kql`, `dashboard.json`, changelog | +| **W4** | KQL test harness | W2, W3 | #2136 (part) | New `HubsSchemaSetup.Tests.ps1` per D6; runs green against W2+W3 output | +| **W5** | Power BI | W3 | — | `kql/Shared.Dataset` TMDL repoint `Costs_v1_2`→`Costs_v1_4` (+ ReservationRecommendations), `storage/Shared.Dataset` `Has14` normalization + `expressions.tmdl` version inference, report.json "FOCUS 1.2" strings, `ftk_NormalizeSchema.pq` if needed, power-bi docs | +| **W6** | Core docs | W2, W3 | #2123 (upgrade.md) | `data-model.md`, `upgrade.md`, `compatibility.md`, `data-processing.md`, `data-dictionary.md`, hub README support matrix, changelog consolidation | +| **W7** | FOCUS conformance + open data | — | — | `Build-FocusConformance.ps1` branch bump, regenerated `docs-mslearn/focus/*`; open-data updates that are **not** CM-gated | +| **W8** | AI/agent surfaces | W3 | #2135 | copilot-studio knowledge files (`schema-reference.md` etc.), `agent-skills` references, claude-plugin output style, `Build-QueryCatalog`/`src/queries` `_v1_2` refs | +| **W9** | Recommendations enrichment | — | #2194 | `x_RecommendationCategory`/`x_RecommendationSubcategory` — independent feature; evaluated on its own merits | +| **W10** | *Blocked on CM 1.4 export release* | W2 | — | `focuscost_1.4*.json` schema, `Exports/app.bicep` registration, `ManagedExports` default, `New-FinOpsCostExport` default + tests + PS docs, open-data `FocusCost_1.4*.json` metadata + example CSV | + +Execution order: W2 → W3 → W4 (stacked); W5–W8 parallel after W3; W9 anytime; W10 blocked. + +## 5. Risks and lessons applied (from the v1_2 rollout) + +| Regression (v1_2) | Mitigation here | +|---|---| +| #1777 — `IngestionSetup_v1_2.kql` missing from `.build.config` (Fabric build shipped broken) | `.build.config` updated in W2 itself + W4 test asserts registration completeness | +| #1906 — `Costs_transform_v1_2()` broke on pre-1.2 exports | W2 transform handles every `x_SourceVersion`; manual ADX pass ingests all 5 export versions | +| #1907 — duplicate rows from Services open-data join in `Costs_v1_2()` | W3 review checks join cardinality; ADX pass includes row-count reconciliation raw vs final vs hub function | +| #e036c2cd — stale cross-version references in paired KQL files | W4 version-string consistency test | +| Power BI docs lagged release by months | D7: docs in-band per PR | + +## 6. Existing PR alignment (pending review) + +The 6 draft PRs were built earlier (some initially targeting FOCUS 1.3) and have unaddressed review feedback. Each gets a full review against this plan and the spec delta before rework: + +- [ ] #2123 (phase0-docs — upgrade procedure) → W6 +- [ ] #2126 (phase1 — IngestionSetup_v1_4.kql) → W2 +- [ ] #2128 (phase2 — HubSetup_v1_4.kql) → W3 +- [ ] #2135 (phase6 — claude plugin) → W8 +- [ ] #2136 (phase7 — tests + changelog) → W4 +- [ ] #2194 (phase9 — recommendation categories) → W9 + +Disposition per PR (keep/rework/supersede) recorded here after review. + +## 7. Open questions (non-blocking; recommendations applied unless overridden) + +1. **Q1 (D4)**: Create empty `v1_4` tables/functions for ContractCommitment / BillingPeriod / InvoiceDetail now, or defer until a data source exists? **Recommendation: defer** (applied). +2. **Q2 (D3)**: Confirm `ServiceProviderName`/`HostProviderName` derivation for Microsoft data (Marketplace → publisher; else Microsoft). **Recommendation as stated** (applied). +3. **Q3**: Target release for `flanakin/focus14` → `dev` (v14 shipped 2026-07; assume next release). Changelog entries land under "Unreleased". + +## 8. Progress + +- [x] Research: FOCUS 1.2→1.4 spec delta (tags, changelog, errata, migration guides) +- [x] Research: FTK FOCUS-version surface inventory +- [x] Research: v1_0→v1_2 precedent + post-release regressions +- [x] W1: Plan committed; umbrella draft PR opened +- [ ] Review 6 existing PRs; record dispositions in §6 +- [ ] W2: Ingestion v1_4 +- [ ] W3: Hub functions v1_4 +- [ ] W4: KQL test harness +- [ ] W5: Power BI +- [ ] W6: Core docs +- [ ] W7: Conformance + open data +- [ ] W8: AI/agent surfaces +- [ ] W9: Recommendations enrichment (#2194) +- [ ] Manual ADX validation pass (all export versions, cardinality reconciliation) +- [ ] Remove this plan file; mark umbrella PR ready +- [ ] *Blocked*: W10 export-side registration (awaiting Cost Management FOCUS 1.4 export) From d94bd5365f34670c8c7b055ee9b28889e8b0a70e Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Wed, 15 Jul 2026 12:25:41 -0700 Subject: [PATCH 2/8] docs: record PR dispositions and revised decisions in FOCUS 1.4 plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - D4 revised: new FOCUS datasets in scope per owner review direction on #2126 - D3 refined: provider-name mapping non-null guarantee; swap defect noted - D9 added: split IngestionSetup_v1_4.kql for the Bicep 131K loadTextContent limit - §6: dispositions + blocking fix lists for all 6 reviewed PRs Co-Authored-By: Claude Fable 5 --- FOCUS-1.4-PLAN.md | 42 ++++++++++++++++++++++++------------------ 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/FOCUS-1.4-PLAN.md b/FOCUS-1.4-PLAN.md index 12508e582..c70a51d6e 100644 --- a/FOCUS-1.4-PLAN.md +++ b/FOCUS-1.4-PLAN.md @@ -69,11 +69,13 @@ The spec migration guide (lines 197–230) defines a context-dependent mapping, - **HostProviderName** = `'Microsoft'` (Microsoft operates the hosting infrastructure for all charges in CM exports). Nullable per spec, but always populated for our data. - **ServiceProviderName** = `PublisherName` for Marketplace charges; `'Microsoft'` otherwise. (For native services PublisherName is already Microsoft, so `coalesce`-style logic collapses cleanly.) -Down-conversion (v1_2/v1_0 views over `_final_v1_4` rows): `ProviderName = HostProviderName`, `PublisherName = ServiceProviderName`. This round-trips correctly for both native and Marketplace charges. Original values preserved via the existing `x_SourceValues` mechanism (#1767 precedent) when changed. +Down-conversion (v1_2/v1_0 views over `_final_v1_4` rows): `ProviderName = HostProviderName`, `PublisherName = ServiceProviderName`. This round-trips correctly for both native and Marketplace charges. Original values preserved via the existing `x_SourceValues` mechanism (#1767 precedent) — for removed columns, follow the removed-column convention (`checkString(..., old_X, '')`) so originals are always captured. -### D4. New FOCUS datasets deferred (recommended; open question Q1) +> ⚠️ Note: a plain `ServiceProviderName = PublisherName` fallback is insufficient — the existing PublisherName fix-up can yield `''` (e.g., `x_PublisherCategory == 'Cloud Provider'` branch), violating the non-null mandate. Use the Marketplace-conditional mapping with a `'Microsoft'` fallback. Both #2126 and #2128 currently have this mapping **swapped** (Host↔Service inverted); fix must be coordinated across ingestion + hub + plugin docs. -ContractCommitment, BillingPeriod, and InvoiceDetail have no Cost Management export and no hub data source. Creating empty tables/functions adds deployment surface and doc burden with no consumer value, and the eventual CM shape may differ. **Decision: do not create hub tables for them in this effort.** Document the position in `data-model.md`. Revisit when a data source exists. +### D4. New FOCUS datasets in scope (revised after PR review) + +Original recommendation was to defer ContractCommitment, BillingPeriod, and InvoiceDetail (no Cost Management source). **Superseded by explicit owner direction** in PR #2126 review comments 3277231717/3277243626, which requested the tables — and the work already exists across the PR stack (raw tables, transforms, `_final_v1_4` tables, `{ds}_v1_4()` functions, unversioned aliases). **Decision: keep them in scope.** They remain empty until a data source ships; all docs/agent surfaces describing them must carry an explicit "no data until Cost Management ships an export" caveat. ### D5. Export-side registration staged behind Cost Management @@ -125,6 +127,10 @@ Small PRs targeting `flanakin/focus14`; stacked where a hard dependency exists. Execution order: W2 → W3 → W4 (stacked); W5–W8 parallel after W3; W9 anytime; W10 blocked. +### D9. KQL file-size limit (added after PR review) + +Bicep `loadTextContent()` caps embedded files at **131,072 characters**. `IngestionSetup_v1_4.kql` on the PR stack is ~128 K before enrichments; PR #2194's +55 lines already broke CI (`Error BCP184`). **Decision: split `IngestionSetup_v1_4.kql` into two scripts** (e.g., core managed datasets vs. supplemental FOCUS datasets), following the existing `OpenDataFunctions_*.kql` split precedent, registered in both `app.bicep` and `.build.config`. The W4 registration-completeness test must cover both files. + ## 5. Risks and lessons applied (from the v1_2 rollout) | Regression (v1_2) | Mitigation here | @@ -135,32 +141,32 @@ Execution order: W2 → W3 → W4 (stacked); W5–W8 parallel after W3; W9 anyti | #e036c2cd — stale cross-version references in paired KQL files | W4 version-string consistency test | | Power BI docs lagged release by months | D7: docs in-band per PR | -## 6. Existing PR alignment (pending review) - -The 6 draft PRs were built earlier (some initially targeting FOCUS 1.3) and have unaddressed review feedback. Each gets a full review against this plan and the spec delta before rework: +## 6. Existing PR dispositions (reviewed 2026-07-15) -- [ ] #2123 (phase0-docs — upgrade procedure) → W6 -- [ ] #2126 (phase1 — IngestionSetup_v1_4.kql) → W2 -- [ ] #2128 (phase2 — HubSetup_v1_4.kql) → W3 -- [ ] #2135 (phase6 — claude plugin) → W8 -- [ ] #2136 (phase7 — tests + changelog) → W4 -- [ ] #2194 (phase9 — recommendation categories) → W9 +Stack order (nothing merged yet; phases 3–5 were closed/superseded): **#2126 (phase1) → #2128 (phase2) → #2135 (phase6) → #2136 (phase7) → #2194 (phase9)**. #2123 (phase0) is independent. Later PRs' diffs currently include all upstream content; retarget each PR's base to its parent branch to shrink review surface. -Disposition per PR (keep/rework/supersede) recorded here after review. +| PR | Disposition | Blocking fixes | +|---|---|---| +| #2123 → W6a | **Keep + fix** | 5 unaddressed 2026-06-29 comments (`sinc` typo, "into"→"in" ×2, "it"→"them", PR body GA/steps mismatch); decide forward-dated `v1_4` Learn content gating; **rescope as "upgrade procedure process docs"** — column-change tables for `upgrade.md`/`compatibility.md` etc. move to a new W6b docs PR | +| #2126 → W2 | **Keep + fix** | C1 Host/ServiceProviderName fallbacks swapped (L765/783); C2 C360 transforms (`ActualCosts`/`AmortizedCosts`) don't project the 22 new `Costs_raw` columns → transactional update policy breaks C360 ingestion (retrofit v1_0/v1_2 too, per repo convention); C3 v1_2 update policies not disabled + no deprecation docstrings; H1 no native-1.4 `x_SourceVersion` detection; M1 removed columns not always preserved in `x_SourceValues`; 12 unaddressed round-2 comments; D9 file split | +| #2128 → W3 | **Keep + fix** | F1 `Prices/Recommendations/Transactions/CommitmentDiscountUsage_v1_4()` missing `_final_v1_2` union arm (near-total data loss for existing hubs); F2 `HubSetup_v1_0/v1_2.kql` down-convert arms never added; F3 same provider-name swap in both up-convert arms; F4 remove duplicated unversioned tails (v1_4 L644-686 **and** stale v1_2 tail L552-588 — `HubSetup_Latest.kql` is single owner); F5 `dashboard.json` still `Costs_v1_2`; F7 opportunistic `take_any` on CDU Services lookup; 11 unaddressed comments; retarget base to phase1 | +| #2135 → W8 | **Rework** | C1 unresolved merge-conflict markers committed in `HubSetup_v1_4.kql` (~L283-296, from merge `676da5e8` — fix the merge resolution); C2 Service/HostProviderName descriptions swapped in `ftk-database-query.md` (other two files correct); M1 `// FOCUS 1.3+` comments on columns landing in v1_4 schema; M2 "no data yet" caveats for new dataset functions; copilot-studio knowledge files NOT covered — track separately in W8; retarget base to phase2 | +| #2136 → W4 | **Rework** | Missing D6 checks 2 (version-string consistency + allowlist — would have caught the conflict markers) and 3 (exactly-one-enabled-policy-version — would have caught C3); hardcoded v1_4 filenames → glob discovery; tighten substring-match tests (`_v1_0()` alias check, occurrence counts); D3 round-trip mapping untested; red-green validate new checks against current broken state | +| #2194 → W9 | **Keep + fix** | Bicep 131 K size break (CI-verified, → D9 split); 22 of ~23 hub-native recommendation queries silently fall to subcategory `Other` (map them); category/subcategory coherence rule + test; 6 open review comments; PR body test claims contradict CI (2006 passed / 1 failed) | ## 7. Open questions (non-blocking; recommendations applied unless overridden) -1. **Q1 (D4)**: Create empty `v1_4` tables/functions for ContractCommitment / BillingPeriod / InvoiceDetail now, or defer until a data source exists? **Recommendation: defer** (applied). -2. **Q2 (D3)**: Confirm `ServiceProviderName`/`HostProviderName` derivation for Microsoft data (Marketplace → publisher; else Microsoft). **Recommendation as stated** (applied). -3. **Q3**: Target release for `flanakin/focus14` → `dev` (v14 shipped 2026-07; assume next release). Changelog entries land under "Unreleased". +1. ~~Q1 (D4)~~ Resolved: new datasets in scope per owner review comments on #2126 (see D4). +2. ~~Q2 (D3)~~ Resolved: derivation confirmed by owner review comments 3309129381/3309175754 on #2126; both PRs currently implement it swapped — fix in W2/W3. +3. **Q3**: Target release for `flanakin/focus14` → `dev` (v14 shipped 2026-07; assume next release). Changelog entries land under "Unreleased" (stack currently uses "FinOps hubs v15.0.0" inside the commented Unreleased block — consistent). ## 8. Progress - [x] Research: FOCUS 1.2→1.4 spec delta (tags, changelog, errata, migration guides) - [x] Research: FTK FOCUS-version surface inventory - [x] Research: v1_0→v1_2 precedent + post-release regressions -- [x] W1: Plan committed; umbrella draft PR opened -- [ ] Review 6 existing PRs; record dispositions in §6 +- [x] W1: Plan committed; umbrella draft PR opened ([#2207](https://github.com/microsoft/finops-toolkit/pull/2207)) +- [x] Review 6 existing PRs; dispositions recorded in §6 - [ ] W2: Ingestion v1_4 - [ ] W3: Hub functions v1_4 - [ ] W4: KQL test harness From 4e98402d787acfb67b347543075bb3ec4d76cb13 Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Wed, 15 Jul 2026 13:18:14 -0700 Subject: [PATCH 3/8] docs: mark W2, W3, W6a complete in FOCUS 1.4 plan (awaiting review) Co-Authored-By: Claude Fable 5 --- FOCUS-1.4-PLAN.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/FOCUS-1.4-PLAN.md b/FOCUS-1.4-PLAN.md index c70a51d6e..6916fbaae 100644 --- a/FOCUS-1.4-PLAN.md +++ b/FOCUS-1.4-PLAN.md @@ -167,11 +167,11 @@ Stack order (nothing merged yet; phases 3–5 were closed/superseded): **#2126 ( - [x] Research: v1_0→v1_2 precedent + post-release regressions - [x] W1: Plan committed; umbrella draft PR opened ([#2207](https://github.com/microsoft/finops-toolkit/pull/2207)) - [x] Review 6 existing PRs; dispositions recorded in §6 -- [ ] W2: Ingestion v1_4 -- [ ] W3: Hub functions v1_4 -- [ ] W4: KQL test harness +- [x] W2: Ingestion v1_4 — #2126 fixed (`30f926d3`: provider mapping, C360 retrofit, v1_2 policies disabled, 1.4 detection, D9 file split; build green) — **awaiting review** +- [x] W3: Hub functions v1_4 — #2128 fixed (`cd2873b7`+`7181c9aa`: v1_2 union arms, v1_0/v1_2 down-convert arms, mapping consistency, tails removed, dashboard repoint; build green; base retargeted to phase1) — **awaiting review** +- [ ] W4: KQL test harness (#2136 rework) - [ ] W5: Power BI -- [ ] W6: Core docs +- [ ] W6: Core docs — W6a #2123 fixed (`463c2735`, awaiting review); W6b column tables pending - [ ] W7: Conformance + open data - [ ] W8: AI/agent surfaces - [ ] W9: Recommendations enrichment (#2194) From 95511f9ad563c8a1f9b1553df47a18d862d33ac6 Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Wed, 15 Jul 2026 15:02:51 -0700 Subject: [PATCH 4/8] docs: record W4-W9 completion, review order, and Q4 in FOCUS 1.4 plan Co-Authored-By: Claude Fable 5 --- FOCUS-1.4-PLAN.md | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/FOCUS-1.4-PLAN.md b/FOCUS-1.4-PLAN.md index 6916fbaae..1d0fc11aa 100644 --- a/FOCUS-1.4-PLAN.md +++ b/FOCUS-1.4-PLAN.md @@ -159,6 +159,7 @@ Stack order (nothing merged yet; phases 3–5 were closed/superseded): **#2126 ( 1. ~~Q1 (D4)~~ Resolved: new datasets in scope per owner review comments on #2126 (see D4). 2. ~~Q2 (D3)~~ Resolved: derivation confirmed by owner review comments 3309129381/3309175754 on #2126; both PRs currently implement it swapped — fix in W2/W3. 3. **Q3**: Target release for `flanakin/focus14` → `dev` (v14 shipped 2026-07; assume next release). Changelog entries land under "Unreleased" (stack currently uses "FinOps hubs v15.0.0" inside the commented Unreleased block — consistent). +4. **Q4 (NEW — needs owner decision)**: `ContractCommitments_final_v1_4` uses unprefixed column names (`BenefitCategory`, `Created`, `Model`, …) that don't match the FOCUS 1.4 spec column IDs (`ContractCommitmentBenefitCategory`, …) and omits spec columns `ContractCommitmentDescription` and `ServiceProviderName` (hubs: 28; spec: 30). Verified against the spec `v1.4` tag by two independent checks. If strict FOCUS conformance is the goal, the table/function/tests/docs should be reconciled — cheap now while the table is empty. Deliberately NOT changed pending your call, since the current shape follows the existing branch content. ## 8. Progress @@ -169,12 +170,19 @@ Stack order (nothing merged yet; phases 3–5 were closed/superseded): **#2126 ( - [x] Review 6 existing PRs; dispositions recorded in §6 - [x] W2: Ingestion v1_4 — #2126 fixed (`30f926d3`: provider mapping, C360 retrofit, v1_2 policies disabled, 1.4 detection, D9 file split; build green) — **awaiting review** - [x] W3: Hub functions v1_4 — #2128 fixed (`cd2873b7`+`7181c9aa`: v1_2 union arms, v1_0/v1_2 down-convert arms, mapping consistency, tails removed, dashboard repoint; build green; base retargeted to phase1) — **awaiting review** -- [ ] W4: KQL test harness (#2136 rework) -- [ ] W5: Power BI -- [ ] W6: Core docs — W6a #2123 fixed (`463c2735`, awaiting review); W6b column tables pending -- [ ] W7: Conformance + open data -- [ ] W8: AI/agent surfaces -- [ ] W9: Recommendations enrichment (#2194) -- [ ] Manual ADX validation pass (all export versions, cardinality reconciliation) +- [x] W4: KQL test harness — #2136 reworked (`778080f2`+`9eb79cad`: 207 static tests, D6 checks 1-4, all red-green validated; full suite 2047/0) — **awaiting review** +- [x] W5: Power BI — new [#2209](https://github.com/microsoft/finops-toolkit/pull/2209) (kql repoint + 22 model columns, storage `Has14` normalization, ProviderName/PublisherName kept as derived columns to protect visuals/DAX; 2 `.pbix` binaries flagged for release-time regen) — **awaiting review** +- [x] W6: Core docs — W6a #2123 fixed (`463c2735`); W6b new [#2211](https://github.com/microsoft/finops-toolkit/pull/2211) (upgrade/compatibility/data-processing/data-dictionary; 111/0 column cross-check) — **awaiting review** +- [x] W7: Conformance — new [#2208](https://github.com/microsoft/finops-toolkit/pull/2208) (`Build-FocusConformance.ps1` fixed for 1.3+ spec layout + curation preservation; 1,021 new 1.3/1.4 requirements marked Not Evaluated, no invented claims) — **awaiting review** +- [x] W8: AI/agent surfaces — #2135 reworked (`968207f4`+`0824600c`: conflict markers removed, provider semantics fixed, no-data caveats); W8b new [#2210](https://github.com/microsoft/finops-toolkit/pull/2210) (copilot-studio knowledge 175/175 column check, query catalog + removed-column migration) — **awaiting review** +- [x] W9: Recommendations enrichment — #2194 fixed (`503d3a63`+`a6c91087`+`47b406ed`: 22/23 queries mapped to real subcategories, taxonomy coherence datatable + tests, symmetric defaults; suite 2093/0) — **awaiting review** +- [ ] Manual ADX validation pass (all export versions, cardinality reconciliation) — requires an Azure deployment (`Deploy-Toolkit -Build -Test`); needs owner to run or authorize - [ ] Remove this plan file; mark umbrella PR ready - [ ] *Blocked*: W10 export-side registration (awaiting Cost Management FOCUS 1.4 export) + +### Recommended review/merge order + +1. [#2123](https://github.com/microsoft/finops-toolkit/pull/2123) (independent process docs) +2. Stack, in order: [#2126](https://github.com/microsoft/finops-toolkit/pull/2126) → [#2128](https://github.com/microsoft/finops-toolkit/pull/2128) → [#2135](https://github.com/microsoft/finops-toolkit/pull/2135) → [#2136](https://github.com/microsoft/finops-toolkit/pull/2136) → [#2194](https://github.com/microsoft/finops-toolkit/pull/2194) (bases auto-retarget as parents merge) +3. Stacked on phase7: [#2209](https://github.com/microsoft/finops-toolkit/pull/2209) (Power BI), [#2210](https://github.com/microsoft/finops-toolkit/pull/2210) (copilot-studio/queries), [#2211](https://github.com/microsoft/finops-toolkit/pull/2211) (schema docs). Note: #2194, #2209 all append to `changelog.md` from the same base — merge conflicts there are additive; keep both sides per repo convention. +4. [#2208](https://github.com/microsoft/finops-toolkit/pull/2208) (conformance, independent) From 9ec85db26b0cdede45483a16e1119520648492cf Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Wed, 15 Jul 2026 17:29:35 -0700 Subject: [PATCH 5/8] docs: record Q4 resolution, dev sync, and PR-environment validation in FOCUS 1.4 plan Co-Authored-By: Claude Fable 5 --- FOCUS-1.4-PLAN.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/FOCUS-1.4-PLAN.md b/FOCUS-1.4-PLAN.md index 1d0fc11aa..3e0d6bfae 100644 --- a/FOCUS-1.4-PLAN.md +++ b/FOCUS-1.4-PLAN.md @@ -159,7 +159,7 @@ Stack order (nothing merged yet; phases 3–5 were closed/superseded): **#2126 ( 1. ~~Q1 (D4)~~ Resolved: new datasets in scope per owner review comments on #2126 (see D4). 2. ~~Q2 (D3)~~ Resolved: derivation confirmed by owner review comments 3309129381/3309175754 on #2126; both PRs currently implement it swapped — fix in W2/W3. 3. **Q3**: Target release for `flanakin/focus14` → `dev` (v14 shipped 2026-07; assume next release). Changelog entries land under "Unreleased" (stack currently uses "FinOps hubs v15.0.0" inside the commented Unreleased block — consistent). -4. **Q4 (NEW — needs owner decision)**: `ContractCommitments_final_v1_4` uses unprefixed column names (`BenefitCategory`, `Created`, `Model`, …) that don't match the FOCUS 1.4 spec column IDs (`ContractCommitmentBenefitCategory`, …) and omits spec columns `ContractCommitmentDescription` and `ServiceProviderName` (hubs: 28; spec: 30). Verified against the spec `v1.4` tag by two independent checks. If strict FOCUS conformance is the goal, the table/function/tests/docs should be reconciled — cheap now while the table is empty. Deliberately NOT changed pending your call, since the current shape follows the existing branch content. +4. ~~Q4~~ Resolved (owner: "just do it", 2026-07-15): `ContractCommitments_raw`/`_final_v1_4` reconciled to the 30 FOCUS 1.4 spec column IDs (12 renames + `ContractCommitmentDescription` + `ServiceProviderName` added), verified against the spec `v1.4` tag; tests, changelog, plugin/copilot docs, and schema docs updated across the stack. ## 8. Progress @@ -176,7 +176,8 @@ Stack order (nothing merged yet; phases 3–5 were closed/superseded): **#2126 ( - [x] W7: Conformance — new [#2208](https://github.com/microsoft/finops-toolkit/pull/2208) (`Build-FocusConformance.ps1` fixed for 1.3+ spec layout + curation preservation; 1,021 new 1.3/1.4 requirements marked Not Evaluated, no invented claims) — **awaiting review** - [x] W8: AI/agent surfaces — #2135 reworked (`968207f4`+`0824600c`: conflict markers removed, provider semantics fixed, no-data caveats); W8b new [#2210](https://github.com/microsoft/finops-toolkit/pull/2210) (copilot-studio knowledge 175/175 column check, query catalog + removed-column migration) — **awaiting review** - [x] W9: Recommendations enrichment — #2194 fixed (`503d3a63`+`a6c91087`+`47b406ed`: 22/23 queries mapped to real subcategories, taxonomy coherence datatable + tests, symmetric defaults; suite 2093/0) — **awaiting review** -- [ ] Manual ADX validation pass (all export versions, cardinality reconciliation) — requires an Azure deployment (`Deploy-Toolkit -Build -Test`); needs owner to run or authorize +- [x] Dev sync: `origin/dev` merged through the entire stack (merge commits only) — brings PR-environment workflows (#2033) and dev's KQL/PBI fixes; dev's v1_2-era fixes (#2151 strcat OOM, #2189 price-lookup max()) ported into the v1_4 transforms with a version-normalized parity diff +- [ ] ADX validation via **PR environments** (replaces the manual pass): "Hubs + ADX (managed)" checked on #2194 (full stack) and #2207 (umbrella); deploys queue for `ftk-pr` environment approval. Validate: ingest samples for all 5 export versions, row-count/cardinality reconciliation raw → final → hub functions - [ ] Remove this plan file; mark umbrella PR ready - [ ] *Blocked*: W10 export-side registration (awaiting Cost Management FOCUS 1.4 export) From 4f69e568e03eff6403055ba5f4f497268f6efdb9 Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Sun, 26 Jul 2026 09:56:36 -0700 Subject: [PATCH 6/8] Refresh FinOps hub upgrade procedure for multi-version FOCUS adoption (#2123) Co-authored-by: Claude Opus 4.7 (1M context) Co-authored-by: copilot-pull-request-reviewer Co-authored-by: github-actions[bot] Co-authored-by: RolandKrummenacher --- docs-mslearn/toolkit/hubs/data-model.md | 15 ++++++++--- src/templates/finops-hub/docs/README.md | 36 ++++++++++++++++++------- 2 files changed, 37 insertions(+), 14 deletions(-) diff --git a/docs-mslearn/toolkit/hubs/data-model.md b/docs-mslearn/toolkit/hubs/data-model.md index e06a19519..d9b5cf6c8 100644 --- a/docs-mslearn/toolkit/hubs/data-model.md +++ b/docs-mslearn/toolkit/hubs/data-model.md @@ -3,7 +3,7 @@ title: FinOps hubs data model description: Learn about the tables and functions available in FinOps hubs to build your own queries, reports, and dashboards. author: flanakin ms.author: micflan -ms.date: 04/01/2026 +ms.date: 07/26/2026 ms.topic: reference ms.service: finops ms.subservice: finops-toolkit @@ -52,15 +52,22 @@ Note the use of "Data Explorer" covers both Azure Data Explorer and Microsoft Fa ## Schema version -One of the goals of the FinOps hubs data model is to guarantee backwards compatibility. To support this, each FinOps hub release uses a specific schema version which aligns to a specific FOCUS version. The **schema version** defines the columns, data types, and allowed values in the tables and functions for each [managed dataset](#managed-datasets-in-finops-hubs). +One of the goals of the FinOps hubs data model is to guarantee backwards compatibility. To support this, each FinOps hub release uses a specific schema version which aligns to a specific FOCUS version. For instance, the `v1_4` schema version aligns with FOCUS 1.4 and the `v1_2` schema with FOCUS 1.2. The **schema version** defines the columns, data types, and allowed values in the tables and functions for each [managed dataset](#managed-datasets-in-finops-hubs). + +During ingestion, data is transformed to the schema version of your current FinOps hub release, regardless of the FOCUS version of the source data. For instance, if your hub uses the `v1_4` schema, incoming FOCUS 1.0 data is converted to FOCUS 1.4 as it's ingested. Data ingested by an earlier release stays in the schema version it was originally ingested into. For instance, data ingested when your hub used the `v1_2` schema remains in the `v1_2` format even after you upgrade to a newer schema version. + +The functions in the `Hub` database pull from all tables and transform them into a specific version. For instance, `Costs_v1_0` will convert `v1_4` and `v1_2` data back to the `v1_0` format, ensuring backwards compatibility despite upgrading to newer versions. When you're ready to upgrade, you can selectively upgrade each report or integration to use `Costs_v1_2` or `Costs_v1_4` without breaking others. However, if you want a quick, convenience function that always returns the latest version, you can use the unversioned functions, like `Costs` and `Prices`. Unversioned functions are convenient for ad hoc use while versioned functions are recommended for integration points that require a consistent format after upgrades. The following table indicates the schema version for each FinOps hub release and which FOCUS version they align to. | Release | Schema | FOCUS version | | ------- | ------ | ------------- | -| 12+ | `v1_2` | 1.2 | +| 15+ | `v1_4` | 1.4 | +| 12-14 | `v1_2` | 1.2 | | 0.7-11 | `v1_0` | 1.0 | +Cost Management export availability for a given FOCUS version is separate from FinOps hub support for that version. FinOps hubs transforms export data to the schema version of your current release during ingestion, whether the export uses an older or newer FOCUS version than that schema. +
## Managed datasets in FinOps hubs @@ -81,7 +88,7 @@ Managed datasets also include the following assets for FinOps hubs with Data Exp - An unversioned function in the **Hub** database (for example, **Costs()**). - A table in Power BI KQL reports that wraps the corresponding versioned function. -During data ingestion, FinOps hubs transform data to the latest supported [schema version](#schema-version). Unversioned functions, like **Costs()**, use the latest schema version. Unversioned functions, like **Costs_v1_0()**, are backwards compatible, should remain consistent, and do not need to change when upgrading your FinOps hub instance. To learn more about the data ingestion process, see [How data is processed in FinOps hubs](data-processing.md). +During data ingestion, FinOps hubs transform data to the latest supported [schema version](#schema-version). Unversioned functions, like **Costs()**, use the latest schema version. Versioned functions, like **Costs_v1_0()**, are backwards compatible, should remain consistent, and do not need to change when upgrading your FinOps hub instance. To learn more about the data ingestion process, see [How data is processed in FinOps hubs](data-processing.md). When querying data in FinOps hubs, always use the **Hub** database and avoid working with the tables and functions in the **Ingestion** database. Use unversioned functions for ad-hoc analysis or reports that do not require long-term backwards compatibility. Use the versioned functions for reports or systems that do require backwards compatibility and you do not want to be impacted by FinOps hub updates, which may change column names, data types, and values. diff --git a/src/templates/finops-hub/docs/README.md b/src/templates/finops-hub/docs/README.md index 2cff88bf9..a639316eb 100644 --- a/src/templates/finops-hub/docs/README.md +++ b/src/templates/finops-hub/docs/README.md @@ -2,10 +2,10 @@ ## Data ingestion workflow -- All data is ingested into tables named "*_raw". +- All data is ingested into tables named "\*\_raw". - Raw tables are declared in [IngestionSetup_raw.kql](../modules/scripts/IngestionSetup_raw.kql). - These tables have a union schema to support multiple sources and versions. -- All data is transformed to the latest FOCUS schema using an update policy into a "*_final_vX_Y" table named after the version (for example, "1.0" = "_v1_0"). Final tables, Transform functions, and update policies are in the versioned setup files: +- All data is transformed to the latest FOCUS schema using an update policy into a "\*\_final_vX_Y" table named after the version (for example, "1.0" = "\_v1_0"). Final tables, Transform functions, and update policies are in the versioned setup files: - [IngestionSetup_v1_0.kql](../modules/scripts/IngestionSetup_v1_0.kql) - Data ingestion from previous version of hubs will remain in the versioned tables. - Data is read from versioned functions in the Hub database. See HubSetup_vX_Y.kql in the [modules/scripts folder](../modules/scripts) for details. @@ -17,27 +17,43 @@ - Each versioned function unions data from versioned tables in the Ingestion database and transforms it to that FOCUS version for back compat. - Consumers should use the unversioned function for the latest and the versioned functions for back compat. -To add a new FOCUS versions: +To add a new FOCUS version: +0. Confirm dependencies + 1. Ensure the FOCUS specification has been ratified for the target version. Support for working draft FOCUS versions should not be shipped before official ratification to prevent customer churn since FOCUS working drafts may change without notice. + 2. Check whether Microsoft Cost Management has shipped a matching FOCUS export dataset version. The hub depends on a `focuscost_X.Y.json` schema mapping file in [Microsoft.CostManagement/Exports/schemas](../modules/Microsoft.CostManagement/Exports/schemas/) when ingesting from Cost Management. If the export is not yet available, the hub schema can still ship as GA. Note the gap in the changelog so adopters know what additional setup will be required. 1. Add schema mapping file 1. Create new schema mapping file for the Cost Management export dataset version in the schemas folder 2. Add file to file upload list in [storage.bicep](../modules/storage.bicep) 2. Update ingestion database scripts - 1. Add new columns to the *_raw tables per dataset in [IngestionSetup_RawTables.kql](../modules/scripts/IngestionSetup_RawTables.kql) + 1. Add new columns to the \*\_raw tables per dataset in [IngestionSetup_RawTables.kql](../modules/scripts/IngestionSetup_RawTables.kql) 2. Save a copy of the latest version of the IngestionSetup_vX_Y.kql using the latest FOCUS version - If updating the same version, increment the release number (e.g., `r2`) 3. Rename all functions, tables, and policies in the new file to the new version (leave the old as-is) - 4. Update the *_final_vX_Y tables to account for any new columns - 5. Update the *_transform_vX_Y functions to account for any new columns - 6. Update the script to delete the old update policy for the *_raw tables + 4. Update the \*\_final_vX_Y tables to account for any new columns + 5. Update the \*\_transform_vX_Y functions to account for any new columns + 6. Update the script to delete the old update policy for the \*\_raw tables 7. Confirm the new file has the update policy set to use the latest version of the transform function and final table 3. Update hub database scripts 1. Add new FOCUS version section after the latest version section and before existing version sections - 2. Create new *_vX_Y functions per dataset that transforms older data to the new FOCUS version - 3. Update the unversioned functions to use the new *_vX_Y functions - 4. Update older versioned functions to also pull from the new *_vX_Y functions and transform to the old schema + 2. Create new \*\_vX_Y functions per dataset that transforms older data to the new FOCUS version + 3. Update the unversioned functions to use the new \*\_vX_Y functions + 4. Update older versioned functions to also pull from the new \*\_vX_Y functions and transform to the old schema 4. Update reports and dashboards 1. Update the storage reports to use the new columns 2. Update the KQL reports to use the new versioned functions 3. Update the ADX dashboard to use the new versioned functions 4. Update the FOCUS queries in the best practices library to use the new versioned functions +5. Update open-data metadata + 1. Create a `FocusCost_.json` file in [src/open-data/dataset-metadata](../../../open-data/dataset-metadata/) for each Cost Management cost export version. `` must match the precise Cost Management export version, including `-preview`, if applicable (for example, `FocusCost_1.2-preview.json`). + 2. Create a `FinOpsHubs__.json` file in [src/open-data/dataset-metadata](../../../open-data/dataset-metadata/) for each FinOps hub managed dataset schema version. + 3. Mirror the schema details (columns, types, descriptions) from the matching Cost Management export or FinOps hubs schema so downstream consumers see consistent metadata. +6. Update plugin skill files + 1. Refresh the FOCUS schema and function references in the following files so plugin guidance does not go stale: + - [src/templates/agent-skills/finops-toolkit/references/finops-hubs.md](../../agent-skills/finops-toolkit/references/finops-hubs.md) + - [src/templates/agent-skills/finops-toolkit/references/finops-hubs-deployment.md](../../agent-skills/finops-toolkit/references/finops-hubs-deployment.md) + - [src/templates/agent-skills/azure-cost-management/references/azure-cost-exports.md](../../agent-skills/azure-cost-management/references/azure-cost-exports.md) + - [src/templates/claude-plugin/agents/ftk-database-query.md](../../claude-plugin/agents/ftk-database-query.md) + - [src/templates/claude-plugin/output-styles/ftk-output-style.md](../../claude-plugin/output-styles/ftk-output-style.md) +7. Update changelog + 1. Add an entry under the next version in [docs-mslearn/toolkit/changelog.md](../../../../docs-mslearn/toolkit/changelog.md) describing the new FOCUS version support and any preview status. From 99106f00d8bcef033ddfe7dc1ffc7e828de6954b Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Sun, 26 Jul 2026 16:37:40 -0700 Subject: [PATCH 7/8] docs: add Q5/W11 back-port work items to FOCUS 1.4 plan Ports the #2126 round-3 closeout tracking from pr-update-agent to the canonical plan: CommitmentProgramEligibilityDetails (W11a, feasible via CommitmentDiscountEligibility.csv), InvoiceDetailId (W11b, blocked on invoice API), ContractCommitment* per-row (W11c, needs design). No standalone tracking issues per owner direction; #2217 closed into plan. Co-Authored-By: Claude Fable 5 --- FOCUS-1.4-PLAN.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/FOCUS-1.4-PLAN.md b/FOCUS-1.4-PLAN.md index 3e0d6bfae..ca162370c 100644 --- a/FOCUS-1.4-PLAN.md +++ b/FOCUS-1.4-PLAN.md @@ -160,6 +160,11 @@ Stack order (nothing merged yet; phases 3–5 were closed/superseded): **#2126 ( 2. ~~Q2 (D3)~~ Resolved: derivation confirmed by owner review comments 3309129381/3309175754 on #2126; both PRs currently implement it swapped — fix in W2/W3. 3. **Q3**: Target release for `flanakin/focus14` → `dev` (v14 shipped 2026-07; assume next release). Changelog entries land under "Unreleased" (stack currently uses "FinOps hubs v15.0.0" inside the commented Unreleased block — consistent). 4. ~~Q4~~ Resolved (owner: "just do it", 2026-07-15): `ContractCommitments_raw`/`_final_v1_4` reconciled to the 30 FOCUS 1.4 spec column IDs (12 renames + `ContractCommitmentDescription` + `ServiceProviderName` added), verified against the spec `v1.4` tag; tests, changelog, plugin/copilot docs, and schema docs updated across the stack. +5. **Q5 (back-port the 1.4 columns older exports don't carry — owner-directed, NOT deferred to an issue)**: Three FOCUS 1.4 `Costs` columns are projected with `column_ifexists()` and land null when ingesting pre-1.4 exports (`Costs_transform_v1_4()` in `IngestionSetup_v1_4.kql`). Per owner direction (2026-07-26, #2126 round-3 closeout) these must be owned by a concrete PR in this stack, not a research issue. Feasibility differs: + - **`CommitmentProgramEligibilityDetails`** (#2126 thread 3309107501) — *feasible now*. The `CommitmentDiscountEligibility.csv` open dataset (MeterId → spend/usage eligibility, ~121K rows) is the source; needs a materialized lookup + the FOCUS 1.4 JSON shape (verify against the spec `v1.4` tag). Owned by **W11a** — new PR stacked on the stack tail (phase9 head); owner: focus14-lead, after #2126 review closes. + - **`InvoiceDetailId`** (#2126 thread 3309133481) — *needs a new source*. Not in current Cost Management exports; requires a new invoice-details API (not available for all account types) plus a custom lookup. Owned by **W11b** (separate PR; blocked on the API — stays null until then). + - **`ContractCommitment*` per-row columns** (#2126 thread 3309131626) — *needs multiple sources*. Some data is in cost data, but not all; pulling from multiple APIs needs design. Owned by **W11c** (separate PR; stays null until then). + Consolidation: standalone issue #2217 closed into this plan + umbrella #2207; owner to decide whether #2159 (native 1.4 export column research) closes into W11 the same way. ## 8. Progress @@ -177,6 +182,7 @@ Stack order (nothing merged yet; phases 3–5 were closed/superseded): **#2126 ( - [x] W8: AI/agent surfaces — #2135 reworked (`968207f4`+`0824600c`: conflict markers removed, provider semantics fixed, no-data caveats); W8b new [#2210](https://github.com/microsoft/finops-toolkit/pull/2210) (copilot-studio knowledge 175/175 column check, query catalog + removed-column migration) — **awaiting review** - [x] W9: Recommendations enrichment — #2194 fixed (`503d3a63`+`a6c91087`+`47b406ed`: 22/23 queries mapped to real subcategories, taxonomy coherence datatable + tests, symmetric defaults; suite 2093/0) — **awaiting review** - [x] Dev sync: `origin/dev` merged through the entire stack (merge commits only) — brings PR-environment workflows (#2033) and dev's KQL/PBI fixes; dev's v1_2-era fixes (#2151 strcat OOM, #2189 price-lookup max()) ported into the v1_4 transforms with a version-normalized parity diff +- [ ] W11: Back-port 1.4 columns from older exports (Q5) — **W11a** `CommitmentProgramEligibilityDetails` via `CommitmentDiscountEligibility.csv` (feasible now; new PR on stack tail, owner focus14-lead); **W11b** `InvoiceDetailId` (blocked on new invoice API); **W11c** `ContractCommitment*` (needs multi-API design). Null until landed, per Q5. - [ ] ADX validation via **PR environments** (replaces the manual pass): "Hubs + ADX (managed)" checked on #2194 (full stack) and #2207 (umbrella); deploys queue for `ftk-pr` environment approval. Validate: ingest samples for all 5 export versions, row-count/cardinality reconciliation raw → final → hub functions - [ ] Remove this plan file; mark umbrella PR ready - [ ] *Blocked*: W10 export-side registration (awaiting Cost Management FOCUS 1.4 export) From 8f30cdbe2373ae9c7e1fbcc79a4a4f09bc45ae74 Mon Sep 17 00:00:00 2001 From: Michael Flanakin Date: Sun, 26 Jul 2026 16:43:37 -0700 Subject: [PATCH 8/8] docs: record #2159 closure into W11 and W11a ownership Co-Authored-By: Claude Fable 5 --- FOCUS-1.4-PLAN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/FOCUS-1.4-PLAN.md b/FOCUS-1.4-PLAN.md index ca162370c..198a27260 100644 --- a/FOCUS-1.4-PLAN.md +++ b/FOCUS-1.4-PLAN.md @@ -164,7 +164,7 @@ Stack order (nothing merged yet; phases 3–5 were closed/superseded): **#2126 ( - **`CommitmentProgramEligibilityDetails`** (#2126 thread 3309107501) — *feasible now*. The `CommitmentDiscountEligibility.csv` open dataset (MeterId → spend/usage eligibility, ~121K rows) is the source; needs a materialized lookup + the FOCUS 1.4 JSON shape (verify against the spec `v1.4` tag). Owned by **W11a** — new PR stacked on the stack tail (phase9 head); owner: focus14-lead, after #2126 review closes. - **`InvoiceDetailId`** (#2126 thread 3309133481) — *needs a new source*. Not in current Cost Management exports; requires a new invoice-details API (not available for all account types) plus a custom lookup. Owned by **W11b** (separate PR; blocked on the API — stays null until then). - **`ContractCommitment*` per-row columns** (#2126 thread 3309131626) — *needs multiple sources*. Some data is in cost data, but not all; pulling from multiple APIs needs design. Owned by **W11c** (separate PR; stays null until then). - Consolidation: standalone issue #2217 closed into this plan + umbrella #2207; owner to decide whether #2159 (native 1.4 export column research) closes into W11 the same way. + Consolidation: standalone issues #2217 and #2159 (native 1.4 export column research) both closed into this plan + umbrella #2207; W11a owner: focus14-lead (agreed 2026-07-26). ## 8. Progress