Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql) by flanakin · Pull Request #2126 · microsoft/finops-toolkit

flanakin · 2026-05-06T15:07:27Z

🛠️ Description

Adds the FOCUS 1.4 ingestion schema (v1_4) for FinOps hubs, converting all currently supported export versions up to FOCUS 1.4 and pre-wiring detection for native 1.3/1.4 exports.

Adds IngestionSetup_v1_4.kql (Prices, Costs, ActualCosts/AmortizedCosts C360 transforms) and IngestionSetup_v1_4_Datasets.kql (BillingPeriods, CommitmentDiscountUsage, ContractCommitments, InvoiceDetails, Recommendations, Transactions) — split into two self-contained files to stay under the Bicep loadTextContent() 131,072-character limit; each dataset's transform, final table, and update policy live in the same file.
Adds FOCUS 1.3/1.4 superset columns to Costs_raw and ContractCommitments_raw in IngestionSetup_RawTables.kql.
Maps removed ProviderName/PublisherName to HostProviderName/ServiceProviderName (falls back to the deprecated columns; guarantees non-null ServiceProviderName per spec); original values are always preserved in x_SourceValues.
Detects native FOCUS 1.3/1.4 exports in x_SourceVersion (by ServiceProviderName/HostProviderName/InvoiceDetailId presence) and treats 1.4 as the current version when labeling x_SourceChanges.
Projects all 22 new Costs_raw columns from the C360 (ActualCosts/AmortizedCosts) transforms in v1_0, v1_2, and v1_4 so the transactional update policies validate against the raw table schema.
Disables all v1_2 update policies and marks the v1_2 transform docstrings deprecated, pointing to the _v1_4() transforms (same pattern as the v1_0 → v1_2 upgrade).
Registers both v1_4 scripts in Analytics/app.bicep and the .build.config Fabric KQL bundle.

New FOCUS 1.4 datasets (BillingPeriods, ContractCommitments, InvoiceDetails) define schema in advance; they remain empty until Cost Management ships an export for them.

Verification

Build-Toolkit finops-hub succeeds (bicep build + loadTextContent; both v1_4 files under the 131,072-character limit: 91,437 and 37,175 bytes).
Both C360 transforms in v1_0, v1_2, and v1_4 project the exact Costs_raw column set (166 columns; verified programmatically).
IngestionSetup_v1_2.kql has zero enabled update policies; v1_4 files have 4 + 6 enabled policies.
Fabric KQL bundle (finops-hub-fabric-setup-Ingestion.kql) includes both v1_4 files.
HubsIngestionQueries.Tests.ps1: 192 passed, 0 failed.
Manual ADX deployment validation (all supported export versions) is tracked for the umbrella PR.

📋 Checklist

🔬 How did you test this change?

🤏 Lint tests

🤞 PS -WhatIf / az validate

👍 Manually deployed + verified

💪 Unit tests

🙌 Integration tests

📦 Deploy to test?

Hubs + ADX (managed)

Hubs + Fabric (manual) — URI:

Hubs (manual)

Hubs (no data)

Workbooks

Alerts

🙋‍♀️ Do any of the following that apply?

🚨 This is a breaking change.

🤏 The change is less than 20 lines of code.

🤖 Generated with Claude Code

Adds IngestionSetup_v1_3.kql with Costs/Prices/CommitmentDiscountUsage/ Recommendations/Transactions transforms and final tables renamed for FOCUS 1.3. Cost and Usage gains 8 new FOCUS 1.3 columns: AllocatedMethodId/Details/ResourceId/ResourceName/Tags (data-generator split cost allocation), ContractApplied (per-row contract commitment application), ServiceProviderName + HostProviderName (replacing the deprecated ProviderName/PublisherName, with empty-fallback for back compat). Costs_raw now carries the new columns so downstream v1_2 transforms keep working and v1_3 transforms can read them directly. The v1_3 file is wired into the Bicep deployment alongside v1_0 and v1_2. The unversioned Costs() function still aliases to v1_2 — phase 2 adds Costs_v1_3() and retargets Latest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Consolidates FOCUS 1.3 and 1.4 column additions into a single v1_4 ingestion setup file. Removes the intermediate v1_3 file, adds ContractCommitment_raw table definition, and updates app.bicep and .build.config to reference v1_4. Co-Authored-By: Claude <noreply@anthropic.com>

flanakin

Changes here need to also be applied to the other 3 FOCUS 1.4 PRs

flanakin · 2026-05-20T21:26:00Z

@@ -359,6 +455,11 @@

 // Costs_raw table -- Redefine all columns
 .alter table Costs_raw (


FOCUS 1.4 has a requirement to include ALL native columns. This will significantly bloat the table size for cross-cloud data. Given we don't have a full set of columns for this yet, we need to have a way to ingest any additional columns either by auto-accepting new columns on ingestion or dumping them into a single x_ column. We need to research options and weigh pros/cons. This can happen in a separate PR, but don't resolve this without a link to that PR.

🤖 [AI][Claude Code] 🤔 Needs discussion

Filed a tracking issue to scope the research: #2159 — Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat. The issue captures the three candidate approaches (auto-accept new columns, single x_NativeColumns JSON, hybrid) and explicitly scopes implementation to a separate PR.

This PR ships the FOCUS 1.4 schema with the columns we know today; native-columns handling will follow #2159's resolution. Leaving this thread unresolved per your request until the follow-up PR is linked.

- Pluralize ContractCommitments_raw / _transform_v1_4 / _final_v1_4 / mapping / update policy - Add BillingPeriods_raw + IngestionSetup_v1_4 transforms/finals/policies (FOCUS 1.4 supplemental dataset) - Add InvoiceDetails_raw + IngestionSetup_v1_4 transforms/finals/policies (FOCUS 1.4 supplemental dataset) - Add 12 ContractCommitment* per-row columns + CommitmentProgramEligibilityDetails + InvoiceDetailId to Costs_raw, Costs_final_v1_4, and Costs_transform_v1_4 - Fix x_Source* annotations on ContractCommitments_raw to Hubs v1_4+ - Update Costs_raw ProviderName/PublisherName annotations to FOCUS 0.5-1.3 (removed in 1.4; kept for back compat) - Update Costs_raw Region annotation: deprecated -> removed - File #2159 for the FOCUS 1.4 native-columns architecture research 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

flanakin · 2026-05-26T07:22:37Z

🤖 [AI][Claude Code] PR Update Summary

Addressed: 7 thread(s)

✅ Implemented: 6
🤔 Needs discussion: 1 (Thread 7 — filed as issue Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat #2159)

Key changes:

Pluralized ContractCommitment_* -> ContractCommitments_* across tables, functions, mapping, and update policies (Thread 1)
Updated x_Source* annotations Hubs v1_3+ -> v1_4+ on ContractCommitments_raw (Thread 2)
Updated ProviderName/PublisherName/Region annotations to mark them as removed (Thread 3)
Added BillingPeriods_raw and InvoiceDetails_raw datasets with transforms/finals/policies (Thread 4)
Added 12 ContractCommitment* per-row columns + CommitmentProgramEligibilityDetails to Costs_raw (Thread 5)
Added InvoiceDetailId to Costs_raw (Thread 6)
Filed Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat #2159 for native-columns research (Thread 7)

Cascading next: Same pluralization + new BillingPeriods/InvoiceDetails datasets will be added to PRs #2128 (hub functions), #2135 (plugin docs), and #2136 (tests + changelog).

- Rename ContractCommitment_v1_4() -> ContractCommitments_v1_4() (and update folder label) - Add BillingPeriods_v1_4() and InvoiceDetails_v1_4() hub functions - Add BillingPeriods(), ContractCommitments() (renamed), and InvoiceDetails() unversioned aliases in HubSetup_Latest.kql - Add CommitmentProgramEligibilityDetails + 12 ContractCommitment* + InvoiceDetailId column defaults to Costs_final_v1_0 and Costs_final_v1_2 union arms (back-compat shims) and add them to the project list 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

- Rename ContractCommitment() -> ContractCommitments() in skill references and agent docs - Add BillingPeriods() and InvoiceDetails() to the function list and dedicated agent doc sections - Mention the new FOCUS 1.4 per-row columns (CommitmentProgramEligibilityDetails, 12 ContractCommitment* columns, InvoiceDetailId) - Update output-style guidance to reference all three supplemental datasets 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

Tests: - Pluralize ContractCommitments_raw / _final_v1_4 / _v1_4 assertions - Assert new BillingPeriods_raw + InvoiceDetails_raw datasets exist with their FOCUS 1.4 columns - Assert new BillingPeriods_v1_4(), InvoiceDetails_v1_4(), ContractCommitments_v1_4() hub functions - Assert HubSetup_Latest aliases for BillingPeriods(), ContractCommitments(), InvoiceDetails() - Assert Costs_raw and Costs_final_v1_4 include the 14 new FOCUS 1.4 columns - Assert NO singular ContractCommitment_raw / ContractCommitment_v1_4 / ContractCommitment() exists - Assert Costs_raw keeps removed ProviderName/PublisherName/Region for back compat Changelog: - Expand v15.0.0 entry with the 14 new FOCUS 1.4 columns and the three new supplemental datasets (ContractCommitments, BillingPeriods, InvoiceDetails) - Update plugin entry to list ContractCommitments() / BillingPeriods() / InvoiceDetails() 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

flanakin · 2026-05-27T06:41:39Z

+    ProviderName:               string,    // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat)
+    PublisherName:              string,    // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat)
+    Region:                     string,    // FOCUS 0.5-1.0-preview (removed; kept for back compat)


Suggested change

ProviderName: string, // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat)

PublisherName: string, // FOCUS 0.5-1.3 (removed in 1.4; kept for back compat)

Region: string, // FOCUS 0.5-1.0-preview (removed; kept for back compat)

ProviderName: string, // FOCUS 0.5-1.3 (removed; keep for back-compat)

PublisherName: string, // FOCUS 0.5-1.3 (removed; keep for back-compat)

Region: string, // FOCUS 0.5-1.0-preview (removed; keep for back-compat)

🤖 [AI][Claude Code] ✅ Implemented

Applied the suggested wording. Costs_raw now annotates ProviderName/PublisherName as // FOCUS 0.5-1.3 (removed; keep for back-compat) and Region as // FOCUS 0.5-1.0-preview (removed; keep for back-compat).

flanakin · 2026-05-27T07:00:18Z

+        // FOCUS 1.3
+        isnotempty(CommitmentDiscountQuantity), CommitmentDiscountQuantity,
+        // FOCUS 1.0-preview, 1.0
+        isempty(CommitmentDiscountStatus), real(null),
+        CommitmentDiscountCategory == 'Spend', EffectiveCost / coalesce(x_BillingExchangeRate, real(1)),
+        CommitmentDiscountCategory == 'Usage' and isnotempty(x_CommitmentDiscountNormalizedRatio), PricingQuantity / coalesce(x_PricingBlockSize, real(1)) * x_CommitmentDiscountNormalizedRatio,
+        real(null)
+    )
+    | extend old_CommitmentDiscountUnit = CommitmentDiscountUnit, CommitmentDiscountUnit = case(
+        // FOCUS 1.3


Suggested change

// FOCUS 1.3

isnotempty(CommitmentDiscountQuantity), CommitmentDiscountQuantity,

// FOCUS 1.0-preview, 1.0

isempty(CommitmentDiscountStatus), real(null),

CommitmentDiscountCategory == 'Spend', EffectiveCost / coalesce(x_BillingExchangeRate, real(1)),

CommitmentDiscountCategory == 'Usage' and isnotempty(x_CommitmentDiscountNormalizedRatio), PricingQuantity / coalesce(x_PricingBlockSize, real(1)) * x_CommitmentDiscountNormalizedRatio,

real(null)

)

| extend old_CommitmentDiscountUnit = CommitmentDiscountUnit, CommitmentDiscountUnit = case(

// FOCUS 1.3

// FOCUS 1.2+

isnotempty(CommitmentDiscountQuantity), CommitmentDiscountQuantity,

// FOCUS 1.0-preview, 1.0

isempty(CommitmentDiscountStatus), real(null),

CommitmentDiscountCategory == 'Spend', EffectiveCost / coalesce(x_BillingExchangeRate, real(1)),

CommitmentDiscountCategory == 'Usage' and isnotempty(x_CommitmentDiscountNormalizedRatio), PricingQuantity / coalesce(x_PricingBlockSize, real(1)) * x_CommitmentDiscountNormalizedRatio,

real(null)

)

| extend old_CommitmentDiscountUnit = CommitmentDiscountUnit, CommitmentDiscountUnit = case(

// FOCUS 1.2+

🤖 [AI][Claude Code] ✅ Implemented

Applied. The CommitmentDiscountQuantity and CommitmentDiscountUnit fallbacks are now labeled // FOCUS 1.2+.

flanakin · 2026-05-27T07:00:53Z

+    //
+    // Pricing
+    | extend old_x_AmortizationClass = x_AmortizationClass, x_AmortizationClass = case(
+        // FOCUS 1.3


Suggested change

// FOCUS 1.3

// FOCUS 1.2+

🤖 [AI][Claude Code] ✅ Implemented

Applied. Comment updated to // FOCUS 1.2+.

flanakin · 2026-05-27T07:05:05Z

+    )
+    | extend x_ConsumedCoreHours = iff(ConsumedUnit == 'Hours' and isnotempty(x_SkuCoreCount), x_SkuCoreCount * ConsumedQuantity, real(null))
+    | extend SkuPriceDetails = case(
+        // FOCUS 1.3


Suggested change

// FOCUS 1.3

// FOCUS 1.2+

🤖 [AI][Claude Code] ✅ Implemented

Applied. Comment updated to // FOCUS 1.2+.

flanakin · 2026-05-27T07:10:12Z

+```
+
+
+//===| BillingPeriods |=================================================================================================


BillingPeriods should be first in this file (alphabetized list)

🤖 [AI][Claude Code] ✅ Implemented

Datasets are now alphabetized with BillingPeriods first. They moved into a new IngestionSetup_v1_4_Datasets.kql (BillingPeriods → CommitmentDiscountUsage → ContractCommitments → InvoiceDetails → Recommendations → Transactions) because IngestionSetup_v1_4.kql was near the Bicep loadTextContent() 131,072-char limit. Prices + Costs stay in IngestionSetup_v1_4.kql since Costs_transform_v1_4() references Prices_final_v1_4 and each file must be self-contained (scripts deploy unordered). Both registered in app.bicep and .build.config.

flanakin · 2026-05-27T07:19:49Z

+        ContractApplied  = parse_json(ContractApplied),  // FOCUS 1.3+
+        ContractCommitmentBenefitCategory          = column_ifexists('ContractCommitmentBenefitCategory', ''),                  // FOCUS 1.4+
+        ContractCommitmentCreated                  = column_ifexists('ContractCommitmentCreated', datetime(null)),              // FOCUS 1.4+
+        ContractCommitmentDiscountPercentage       = column_ifexists('ContractCommitmentDiscountPercentage', real(null)),       // FOCUS 1.4+
+        ContractCommitmentDurationType             = column_ifexists('ContractCommitmentDurationType', ''),                     // FOCUS 1.4+
+        ContractCommitmentFulfillmentInterval      = column_ifexists('ContractCommitmentFulfillmentInterval', ''),              // FOCUS 1.4+
+        ContractCommitmentLastUpdated              = column_ifexists('ContractCommitmentLastUpdated', datetime(null)),          // FOCUS 1.4+
+        ContractCommitmentLifecycleStatus          = column_ifexists('ContractCommitmentLifecycleStatus', ''),                  // FOCUS 1.4+
+        ContractCommitmentModel                    = column_ifexists('ContractCommitmentModel', ''),                            // FOCUS 1.4+
+        ContractCommitmentOfferCategory            = column_ifexists('ContractCommitmentOfferCategory', ''),                    // FOCUS 1.4+
+        ContractCommitmentPaymentInterval          = column_ifexists('ContractCommitmentPaymentInterval', ''),                  // FOCUS 1.4+
+        ContractCommitmentPaymentModel             = column_ifexists('ContractCommitmentPaymentModel', ''),                     // FOCUS 1.4+
+        ContractCommitmentPaymentUpfrontPercentage = column_ifexists('ContractCommitmentPaymentUpfrontPercentage', real(null)), // FOCUS 1.4+


Port from older FOCUS exports

🤖 [AI][Claude Code] 🤔 Needs discussion

Tracked in #2217 alongside the other back-port gaps. No source-data mapping exists yet for the ContractCommitment* per-row columns, so back-filling them here would be speculative — they're projected with column_ifexists() (null for older exports) until the mapping is designed.

🤖 [AI][Claude Code] 🤔 Correction — not deferred to an issue

Superseding my earlier reply (#2217 closed; tracking consolidated into umbrella #2207 / FOCUS-1.4-PLAN.md).

Agreed this needs thought: some ContractCommitment* data is in cost data but not all — pulling from multiple APIs needs a design pass, so it's a separate PR. Tracked as W11c. Projected with column_ifexists() (null for older exports) until then.

flanakin · 2026-05-27T07:20:09Z

+        ContractedUnitPrice = coalesce(ContractedUnitPrice, x_OnDemandUnitPrice),
+        EffectiveCost,
+        HostProviderName = iff(isempty(HostProviderName), PublisherName, HostProviderName),  // FOCUS 1.3+; falls back to deprecated PublisherName for back compat
+        InvoiceDetailId = column_ifexists('InvoiceDetailId', ''),  // FOCUS 1.4+


Port from older FOCUS exports

🤖 [AI][Claude Code] 🤔 Needs discussion

Tracked in #2217. InvoiceDetailId is projected with column_ifexists() and left empty for pre-1.4 exports until we have a source mapping to derive it from.

🤖 [AI][Claude Code] 🤔 Correction — not deferred to an issue

Superseding my earlier reply (#2217 closed; tracking consolidated into umbrella #2207 / FOCUS-1.4-PLAN.md).

InvoiceDetailId needs a new invoice-details API (not available for all account types) plus a custom lookup to map back into the Costs dataset — separate PR. Tracked as W11b. column_ifexists() leaves it empty for pre-1.4 exports until then.

flanakin · 2026-05-27T07:26:44Z

+        PricingCurrency = coalesce(PricingCurrency, x_PricingCurrency),
+        PricingQuantity,
+        PricingUnit,
+        // ProviderName and PublisherName removed in FOCUS 1.4. Use ServiceProviderName / HostProviderName.


Suggested change

// ProviderName and PublisherName removed in FOCUS 1.4. Use ServiceProviderName / HostProviderName.

🤖 [AI][Claude Code] ✅ Implemented

Applied — the placement comment was deleted.

flanakin · 2026-05-27T07:27:26Z

+        ResourceType,
+        ServiceCategory,
+        ServiceName,
+        ServiceProviderName = iff(isempty(ServiceProviderName), ProviderName, ServiceProviderName),  // FOCUS 1.3+; falls back to deprecated ProviderName for back compat


Suggested change

ServiceProviderName = iff(isempty(ServiceProviderName), ProviderName, ServiceProviderName), // FOCUS 1.3+; falls back to deprecated ProviderName for back compat

ServiceProviderName = iff(isempty(ServiceProviderName), PublisherName, ServiceProviderName), // FOCUS 1.3+; falls back to deprecated PublisherName for back compat

🤖 [AI][Claude Code] ✅ Implemented

Applied, with one addition beyond the suggestion: ServiceProviderName falls back to PublisherName, then ProviderName, then 'Microsoft'. The extra arms are needed because the earlier PublisherName fix-up can yield '' (when x_PublisherCategory isn't Cloud Provider), and FOCUS mandates a non-null ServiceProviderName.

flanakin · 2026-05-27T07:32:23Z

+    PricingCurrency:            string,
+    PricingQuantity:            real,
+    PricingUnit:                string,
+    // ProviderName and PublisherName removed in FOCUS 1.4


Suggested change

// ProviderName and PublisherName removed in FOCUS 1.4

🤖 [AI][Claude Code] ✅ Implemented

Applied — the placement comment was deleted.

- D4 revised: new FOCUS datasets in scope per owner review direction on #2126 - D3 refined: provider-name mapping non-null guarantee; swap defect noted - D9 added: split IngestionSetup_v1_4.kql for the Bicep 131K loadTextContent limit - §6: dispositions + blocking fix lists for all 6 reviewed PRs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- Fix swapped HostProviderName/ServiceProviderName fallbacks and guarantee non-null ServiceProviderName (C1) - Project all 22 new Costs_raw columns from the ActualCosts/AmortizedCosts C360 transforms in v1_0, v1_2, and v1_4 (C2) - Disable v1_2 update policies and add deprecation docstrings pointing to the _v1_4() transforms (C3) - Detect native FOCUS 1.3/1.4 exports in x_SourceVersion and treat 1.4 as the current version for x_SourceChanges (H1) - Always preserve removed ProviderName/PublisherName values in x_SourceValues (M1) - Apply review suggestions: raw table annotation wording, FOCUS 1.2+ comment fixes, remove stale placement comments - Split IngestionSetup_v1_4.kql into IngestionSetup_v1_4.kql (Prices + Costs) and IngestionSetup_v1_4_Datasets.kql (remaining datasets, alphabetized) to stay under the Bicep loadTextContent() 131,072-character limit; register the new file in app.bicep and .build.config Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

flanakin · 2026-07-15T19:59:28Z

Pushed 30f926d addressing the round-2 review feedback and the remaining W2 fix list. Mapping of changes to review comments:

Applied

3308931802 — Costs_raw annotation wording updated to (removed; keep for back-compat) for ProviderName/PublisherName/Region
3309022571 / 3309026016 / 3309054996 — // FOCUS 1.3 → // FOCUS 1.2+ on the CommitmentDiscountQuantity, CommitmentDiscountUnit, x_AmortizationClass, and SkuPriceDetails fallbacks
3309129381 — HostProviderName now falls back to deprecated ProviderName (suggestion applied as-is)
3309175754 — ServiceProviderName now falls back to deprecated PublisherName, then ProviderName, then 'Microsoft'. Went one step beyond the suggestion because the earlier PublisherName fix-up can yield '' (when x_PublisherCategory isn't Cloud Provider), and FOCUS mandates non-null ServiceProviderName — the extra arms guarantee that
3309171288 / 3309205246 — both "ProviderName and PublisherName removed…" placement comments deleted
3309082327 — the new dataset sections are now alphabetized with BillingPeriods first. They landed in a new IngestionSetup_v1_4_Datasets.kql (BillingPeriods, CommitmentDiscountUsage, ContractCommitments, InvoiceDetails, Recommendations, Transactions) because IngestionSetup_v1_4.kql was ~2K characters from the Bicep loadTextContent() 131,072-character limit (PR Add x_RecommendationCategory and x_RecommendationSubcategory enrichments to v1_4 Recommendations #2194 already tripped it). Prices stays with Costs in IngestionSetup_v1_4.kql because Costs_transform_v1_4() joins Prices_final_v1_4 and the Costs_raw update policy validates that reference at deployment time — scripts in the hub-database.bicep map deploy as an unordered loop, so each file must be self-contained. Both files registered in app.bicep and .build.config.

Also in this push (W2 fix list)

C2: all 22 new Costs_raw columns projected from the C360 ActualCosts/AmortizedCosts transforms (v1_4, plus v1_0/v1_2 retrofits per the InvoiceId = '' convention) — verified programmatically against the raw table (166 columns, exact match ×6 transforms)
C3: all 7 v1_2 update policies disabled + deprecation docstrings pointing to _v1_4()
H1: x_SourceVersion infers '1.4' from ServiceProviderName/HostProviderName/InvoiceDetailId presence; x_SourceChanges now treats '1.4' as current (everything else labeled Legacy/UnknownFocusVersion)
M1: ProviderName/PublisherName always captured in x_SourceValues via the removed-column convention (checkString(..., old_X, ''))

Deliberately not done (tracked follow-ups — native-column research, #2159)

3309107501 — back-porting CommitmentProgramEligibilityDetails from older exports (needs the CommitmentDiscountEligibility open dataset); left null for now
3309131626 / 3309133481 — porting ContractCommitment* and InvoiceDetailId from older exports; no source data mapping exists yet, so inventing one here would be speculative

Verification: Build-Toolkit finops-hub green; both v1_4 files ≤ 110K (91,437 / 37,175 bytes); zero conflict markers; zero v1_3 identifiers; HubsIngestionQueries.Tests.ps1 192/192 passed.

Resolved IngestionSetup_v1_2.kql docstring conflict by combining dev's AmortizedCost copy-paste fix with the DEPRECATED annotation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Ports #2151 (strcat-free emptiness checks), #2189 (max()-based price lookup for duplicate on-demand rows), and the AmortizedCost docstring copy-paste fix into IngestionSetup_v1_4.kql, which was copied from v1_2 before those fixes landed on dev. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Renames 12 unprefixed columns (BenefitCategory, Created, Model, ...) to their spec IDs (ContractCommitmentBenefitCategory, ...) and adds the two missing columns (ContractCommitmentDescription, ServiceProviderName), bringing the dataset to the full 30 FOCUS columns across the raw table, parquet mapping, transform, and final table. Hub x_ add-on columns are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

flanakin · 2026-07-16T00:04:11Z

Synced with dev and applied the Q4 schema fix:

Merge: merged flanakin/focus14 (now in sync with origin/dev). One conflict in IngestionSetup_v1_2.kql — dev's AmortizedCost docstring copy-paste fix vs the DEPRECATED annotation; resolved by combining both.
Ported dev fixes to v1.4: IngestionSetup_v1_4.kql was copied from v1_2 before [Hubs] Fix ingestion OOM errors by eliminating strcat() in emptiness checks #2151 and fix(hubs): use max() when duplicate on-demand price rows collapse under one lookup key #2189 landed. Ported the strcat-free emptiness checks, the max()-based on-demand price lookup, and the AmortizedCost docstring fix. Version-normalized diff vs v1_2 confirms the transform fix regions are at parity.
Q4 — ContractCommitments spec alignment: renamed 12 unprefixed columns to their FOCUS 1.4 spec IDs (e.g., BenefitCategory → ContractCommitmentBenefitCategory) and added the 2 missing columns (ContractCommitmentDescription, ServiceProviderName), verified against the FOCUS 1.4 spec dataset definition — now the full 30 FOCUS columns across the raw table, parquet mapping, transform, and final table. Hub x_ add-on columns unchanged. The flattened ContractCommitment* per-row columns in the Costs tables are untouched (separate design).

Verified: Build-Toolkit finops-hub exit 0; v1_4 KQL files ≤ 110 KB (91.6 KB / 38.0 KB); no conflict markers.

Ports the #2126 round-3 closeout tracking from pr-update-agent to the canonical plan: CommitmentProgramEligibilityDetails (W11a, feasible via CommitmentDiscountEligibility.csv), InvoiceDetailId (W11b, blocked on invoice API), ContractCommitment* per-row (W11c, needs design). No standalone tracking issues per owner direction; #2217 closed into plan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings May 6, 2026 15:07

flanakin requested review from MSBrett and RolandKrummenacher as code owners May 6, 2026 15:07

microsoft-github-policy-service Bot requested a review from helderpinto May 6, 2026 15:08

microsoft-github-policy-service Bot added the Needs: Review 👀 PR that is ready to be reviewed label May 6, 2026

microsoft-github-policy-service Bot assigned MSBrett May 6, 2026

microsoft-github-policy-service Bot added the Tool: FinOps hubs Data pipeline solution label May 6, 2026

microsoft-github-policy-service Bot assigned helderpinto and RolandKrummenacher May 6, 2026

Copilot started reviewing on behalf of flanakin May 6, 2026 15:08 View session

This was referenced May 6, 2026

Add FOCUS 1.4 hub functions (HubSetup_v1_4.kql) #2128

Draft

FOCUS 1.3 and 1.4 support across the toolkit #2120

Open

flanakin marked this pull request as draft May 6, 2026 23:16

flanakin added this to the v15 milestone May 15, 2026

flanakin changed the title ~~Add FOCUS 1.3 ingestion (IngestionSetup_v1_3.kql)~~ Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql) May 17, 2026

This was referenced May 17, 2026

Add ContractCommitment dataset #2130

Closed

Add FOCUS 1.4-preview ingestion (IngestionSetup_v1_4.kql) #2132

Closed

flanakin commented May 26, 2026

View reviewed changes

flanakin mentioned this pull request May 26, 2026

Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat #2159

Closed

flanakin mentioned this pull request May 26, 2026

Refresh Claude Code plugin skill files for FOCUS 1.4 #2135

Draft

flanakin mentioned this pull request May 27, 2026

Add FOCUS 1.4 schema tests and changelog entries #2136

Draft

flanakin commented May 28, 2026

View reviewed changes

flanakin removed the Needs: Review 👀 PR that is ready to be reviewed label May 28, 2026

flanakin added the Needs: Attention 👋 Issue or PR needs to be reviewed by the author or it will be closed due to no activity label May 28, 2026

RolandKrummenacher mentioned this pull request Jun 3, 2026

[Hubs] Fix ingestion OOM errors by eliminating strcat() in emptiness checks #2151

Merged

flanakin mentioned this pull request Jul 15, 2026

Add FOCUS 1.4 support #2207

Draft

30 tasks

flanakin mentioned this pull request Jul 15, 2026

Refresh FinOps hub upgrade procedure for multi-version FOCUS adoption #2123

Merged

microsoft-github-policy-service Bot added Needs: Review 👀 PR that is ready to be reviewed and removed Needs: Attention 👋 Issue or PR needs to be reviewed by the author or it will be closed due to no activity labels Jul 15, 2026

flanakin and others added 3 commits July 15, 2026 16:55

merge: sync flanakin/focus14 (origin/dev) into phase1

14bf92c

Resolved IngestionSetup_v1_2.kql docstring conflict by combining dev's AmortizedCost copy-paste fix with the DEPRECATED annotation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

flanakin mentioned this pull request Jul 26, 2026

Back-port FOCUS 1.4 columns from older export versions during ingestion #2217

Closed

		@@ -359,6 +455,11 @@

		// Costs_raw table -- Redefine all columns
		.alter table Costs_raw (

		```


		//===\| BillingPeriods \|=================================================================================================

	ServiceProviderName = iff(isempty(ServiceProviderName), ProviderName, ServiceProviderName), // FOCUS 1.3+; falls back to deprecated ProviderName for back compat
	ServiceProviderName = iff(isempty(ServiceProviderName), PublisherName, ServiceProviderName), // FOCUS 1.3+; falls back to deprecated PublisherName for back compat

Uh oh!

Conversation

flanakin commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛠️ Description

Verification

📋 Checklist

🔬 How did you test this change?

📦 Deploy to test?

🙋‍♀️ Do any of the following that apply?

Uh oh!

flanakin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flanakin commented May 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flanakin commented Jul 15, 2026

Uh oh!

flanakin commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

flanakin commented May 6, 2026 •

edited

Loading