diff --git a/.agents/sow/current/SOW-0102-20260603-quality-complexity-duplication-coverage.md b/.agents/sow/current/SOW-0102-20260603-quality-complexity-duplication-coverage.md index 782850e..d87afc3 100644 --- a/.agents/sow/current/SOW-0102-20260603-quality-complexity-duplication-coverage.md +++ b/.agents/sow/current/SOW-0102-20260603-quality-complexity-duplication-coverage.md @@ -4,7 +4,7 @@ Status: in-progress -Sub-state: twenty-fifth implementation slice validated; ASN database processing PR pending +Sub-state: twenty-sixth implementation slice validated; entity detail aggregation duplication PR pending ## Requirements @@ -2023,6 +2023,52 @@ Open decisions: - No new user design decision is required because the slice is behavior-preserving quality work under the previously approved quality plan. +## Slice 26 Results + +Changes made: + +- Added `pkg/engine/home_entity_facets.go` with a shared detail facet accumulator for category and maintainer totals. +- Reused the shared accumulator from both country and ASN detail builders. +- Removed duplicated category and maintainer aggregation from `countryDetailBuilder.addFeed` and `asnDetailBuilder.addFeed`. +- Removed duplicated top-category and top-maintainer summary construction from country and ASN detail sidecar builds. +- Preserved country-specific ASN rollups, ASN-specific country rollups, ASN country distribution, feed ordering, summary sorting, maintainer slug/URL fill-in behavior, and public payload schemas. + +Measured result: + +- Baseline source-only `jscpd`: `11` clones, `511` duplicated lines (`0.52%`), `4595` duplicated tokens (`0.53%`). +- After refactor source-only `jscpd`: `9` clones, `433` duplicated lines (`0.44%`), `3761` duplicated tokens (`0.44%`). +- Go clones moved from `6` to `4`; Go duplicated lines moved from `169` (`0.29%`) to `91` (`0.16%`). +- The `home_entity_precompute.go` duplicate blocks are no longer reported by `jscpd`. +- `pkg/engine` coverage remains `71.9%`. +- Shared helper coverage includes `newDetailFacetAccumulator` at `100.0%`, `(*detailFacetAccumulator).add` at `94.7%`, `topCategories` at `80.0%`, and `topMaintainers` at `90.0%`. +- `countryDetailBuilder.addFeed` and `asnDetailBuilder.addFeed` are each `100.0%` covered after the split. +- Root coverage by `go tool cover -func=coverage.out` remains `72.9%`. +- `tools/archposture` after this slice: source files `633`, source lines `127377`, large files `49`, large functions `25`, and production large functions `0`. + +Tests or equivalent validation: + +- `go test ./pkg/engine`: passed. +- `go test -coverprofile=/tmp/update-ipsets-engine-slice26.cover -covermode=atomic ./pkg/engine`: passed, `71.9%`. +- `go tool cover -func=/tmp/update-ipsets-engine-slice26.cover`: passed; shared facet helper covered through entity detail behavior tests. +- `go run ./tools/archposture -root . > /tmp/update-ipsets-archposture-slice26.json`: passed. +- Source-only `jscpd` with project hygiene exclusions: passed; clone count reduced from `11` to `9`. +- `make lint`: passed. +- `make staticcheck`: passed. +- `make golangci-lint`: passed with `0 issues`. +- `CI=true make coverage`: passed, root total `72.9%`. +- `make test-strict`: passed. +- `git diff --check`: passed. +- Durable-artifact forbidden-name scan over added diff lines found no newly added personal name, authorship, tool, or vendor-attribution text. + +Artifact maintenance gate: + +- AGENTS.md: no update needed. +- Runtime project skills: no update needed; no new durable process rule was found. +- Specs: no update needed; entity detail payload semantics are unchanged. +- End-user/operator docs: no update needed. +- End-user/operator skills: no update needed. +- SOW lifecycle: remains in `.agents/sow/current/`; Slice 26 is validated and pending PR merge. + ## Slice 25 Results Changes made: @@ -2072,7 +2118,89 @@ Artifact maintenance gate: - Specs: no update needed; ASN provider lifecycle semantics are unchanged. - End-user/operator docs: no update needed. - End-user/operator skills: no update needed. -- SOW lifecycle: remains in `.agents/sow/current/`; Slice 25 is validated and pending PR merge. +- SOW lifecycle: remains in `.agents/sow/current/`; Slice 25 merged through PR #29 as merge commit `05218cbe2d0ac4158037bb18f66e743a83def376`. + +## Pre-Implementation Gate - Slice 26 + +Status: ready. + +Problem / root-cause model: + +- Facts: after the Slice 25 merge, local `tools/archposture` reports zero production large functions. +- Facts: source-only `jscpd` reports 11 clones and `511` duplicated source lines (`0.52%`) with tests, generated assets, SOW artifacts, dependencies, and archposture testdata excluded. +- Facts: the largest backend duplication candidates include `pkg/engine/home_entity_precompute.go` country and ASN detail builders, where category and maintainer aggregation are repeated across `countryDetailBuilder` and `asnDetailBuilder`. +- Facts: `go tool cover -func=coverage.out` reports `countryDetailBuilder.addFeed` at `95.0%`, `asnDetailBuilder.addFeed` at `95.5%`, `countryDetailBuilder.build` at `73.5%`, and `asnDetailBuilder.build` at `84.1%`. +- Working theory: the repeated category/maintainer aggregation is a good next slice because it is real backend source duplication, not intentional optimized IPv4/IPv6 duplication, and it can be extracted without changing country/ASN-specific payload construction. + +Evidence reviewed: + +- `/tmp/update-ipsets-archposture-after-pr29.json` +- `/tmp/update-ipsets-jscpd-slice26/jscpd-report.json` +- `pkg/engine/home_entity_precompute.go` +- `pkg/engine/home_detail_test.go` +- `pkg/engine/entity_surgical_test.go` +- `pkg/engine/home_entity_detail_live_test.go` +- `coverage.out` +- Project coding, testing, hygiene, Go best-practices, Go behavioral-testing, and content-surface skills. + +Affected contracts and surfaces: + +- Country detail sidecar category and maintainer rollups. +- ASN detail sidecar category and maintainer rollups. +- Sort order for top categories and top maintainers. +- Feed rows, country/ASN-specific top lists, country distribution, provider metadata, totals, and public payload schemas must not change. +- SOW only; no docs or specs are expected to change because the slice is behavior-preserving. + +Existing patterns to reuse: + +- Existing `detailCategoryAggregate` and `detailMaintainerAggregate` types. +- Existing `DetailCategorySummary` and `DetailMaintainerSummary` payload rows. +- Existing `maintainerSlugify` and maintainer URL fill-in behavior. +- Existing entity detail tests that validate public country and ASN detail payload materialization. +- Existing local `jscpd` source-only scan command from project hygiene. + +Risk and blast radius: + +- Public country and ASN detail payloads depend on these rollups. +- Category and maintainer tie-break ordering must stay byte-for-byte compatible for existing tests and stable public artifacts. +- The shared helper must not erase country-specific ASN rollups or ASN-specific country distribution logic. +- No downloader, scheduler queueing, public serving fallback, install behavior, UI behavior, or ASN/geolocation attribution algorithm should change. + +Sensitive data handling plan: + +- This slice uses only local source code, synthetic test fixtures already in the repository, temporary paths, and local posture/duplication/coverage metrics. +- No secrets, tokens, cookies, private endpoints, customer data, or personal data are needed. +- Durable artifacts will record only file paths, metrics, validation outcomes, and sanitized command evidence. + +Implementation plan: + +1. Introduce a small shared detail facet accumulator for category and maintainer totals. +2. Move category rollup addition and maintainer rollup addition into the shared accumulator. +3. Move top-category and top-maintainer summary construction into shared helper methods. +4. Update country and ASN detail builders to embed or own the shared accumulator while keeping country/ASN-specific fields separate. +5. Re-run engine tests, coverage, archposture, and `jscpd` to prove behavior and duplication posture. + +Validation plan: + +- Run `go test ./pkg/engine`. +- Run `go test -coverprofile=/tmp/update-ipsets-engine-slice26.cover -covermode=atomic ./pkg/engine` and inspect `go tool cover -func`. +- Run `go run ./tools/archposture -root . > /tmp/update-ipsets-archposture-slice26.json`. +- Run source-only `jscpd` with the project hygiene exclusions and compare clone counts against Slice 26 baseline. +- Run `make lint`, `make staticcheck`, `make golangci-lint`, `CI=true make coverage`, and `make test-strict`. +- Run whitespace and durable-artifact forbidden-name scans over the changed files before commit. + +Artifact impact plan: + +- AGENTS.md: no update expected. +- Runtime project skills: update only if a repeatable duplication triage lesson is found. +- Specs: no update expected because entity detail payload semantics are intended to stay unchanged. +- End-user/operator docs: no update expected. +- End-user/operator skills: no update expected. +- SOW lifecycle: this SOW remains in `.agents/sow/current/`; Slice 26 results will be recorded after validation. + +Open decisions: + +- No new user design decision is required because the slice is behavior-preserving quality work under the previously approved quality plan. ## Slice 16 Results @@ -3264,7 +3392,7 @@ Open decisions: ## Outcome -First through twenty-fourth implementation slices are complete, validated locally, and merged. The twenty-fifth implementation slice is complete and validated locally. The SOW remains open for the next focused coverage, complexity, or duplication slice. +First through twenty-fifth implementation slices are complete, validated locally, and merged. The twenty-sixth implementation slice is complete and validated locally. The SOW remains open for the next focused coverage, complexity, or duplication slice. ## Lessons Extracted diff --git a/pkg/engine/home_entity_facets.go b/pkg/engine/home_entity_facets.go new file mode 100644 index 0000000..b4e0ddb --- /dev/null +++ b/pkg/engine/home_entity_facets.go @@ -0,0 +1,110 @@ +package engine + +import ( + "sort" + "strings" +) + +type detailFacetAccumulator struct { + categoryTotals map[string]*detailCategoryAggregate + maintainerTotals map[string]*detailMaintainerAggregate +} + +func newDetailFacetAccumulator() detailFacetAccumulator { + return detailFacetAccumulator{ + categoryTotals: make(map[string]*detailCategoryAggregate), + maintainerTotals: make(map[string]*detailMaintainerAggregate), + } +} + +func (a *detailFacetAccumulator) add(category, maintainer, maintainerURL string, attributedIPs uint64) { + a.ensureMaps() + categoryAgg := a.categoryTotals[category] + if categoryAgg == nil { + categoryAgg = &detailCategoryAggregate{} + a.categoryTotals[category] = categoryAgg + } + categoryAgg.feedCount++ + categoryAgg.attributedIPs += attributedIPs + + maintainerName := strings.TrimSpace(maintainer) + if maintainerName == "" { + return + } + slug := maintainerSlugify(maintainerName) + maintainerAgg := a.maintainerTotals[slug] + if maintainerAgg == nil { + maintainerAgg = &detailMaintainerAggregate{ + slug: slug, + name: maintainerName, + url: maintainerURL, + } + a.maintainerTotals[slug] = maintainerAgg + } + if maintainerAgg.url == "" && maintainerURL != "" { + maintainerAgg.url = maintainerURL + } + maintainerAgg.feedCount++ + maintainerAgg.attributedIPs += attributedIPs +} + +func (a *detailFacetAccumulator) ensureMaps() { + if a.categoryTotals == nil { + a.categoryTotals = make(map[string]*detailCategoryAggregate) + } + if a.maintainerTotals == nil { + a.maintainerTotals = make(map[string]*detailMaintainerAggregate) + } +} + +func (a detailFacetAccumulator) categoryCount() int { + return len(a.categoryTotals) +} + +func (a detailFacetAccumulator) maintainerCount() int { + return len(a.maintainerTotals) +} + +func (a detailFacetAccumulator) topCategories() []DetailCategorySummary { + topCategories := make([]DetailCategorySummary, 0, len(a.categoryTotals)) + for category, agg := range a.categoryTotals { + topCategories = append(topCategories, DetailCategorySummary{ + Category: category, + FeedCount: agg.feedCount, + AttributedIPs: agg.attributedIPs, + }) + } + sort.Slice(topCategories, func(i, j int) bool { + if topCategories[i].AttributedIPs != topCategories[j].AttributedIPs { + return topCategories[i].AttributedIPs > topCategories[j].AttributedIPs + } + if topCategories[i].FeedCount != topCategories[j].FeedCount { + return topCategories[i].FeedCount > topCategories[j].FeedCount + } + return topCategories[i].Category < topCategories[j].Category + }) + return topCategories +} + +func (a detailFacetAccumulator) topMaintainers() []DetailMaintainerSummary { + topMaintainers := make([]DetailMaintainerSummary, 0, len(a.maintainerTotals)) + for _, agg := range a.maintainerTotals { + topMaintainers = append(topMaintainers, DetailMaintainerSummary{ + Slug: agg.slug, + Name: agg.name, + URL: agg.url, + FeedCount: agg.feedCount, + AttributedIPs: agg.attributedIPs, + }) + } + sort.Slice(topMaintainers, func(i, j int) bool { + if topMaintainers[i].AttributedIPs != topMaintainers[j].AttributedIPs { + return topMaintainers[i].AttributedIPs > topMaintainers[j].AttributedIPs + } + if topMaintainers[i].FeedCount != topMaintainers[j].FeedCount { + return topMaintainers[i].FeedCount > topMaintainers[j].FeedCount + } + return topMaintainers[i].Name < topMaintainers[j].Name + }) + return topMaintainers +} diff --git a/pkg/engine/home_entity_precompute.go b/pkg/engine/home_entity_precompute.go index 19ef381..52b928d 100644 --- a/pkg/engine/home_entity_precompute.go +++ b/pkg/engine/home_entity_precompute.go @@ -15,54 +15,25 @@ type countryDetailASNAggregate struct { } type countryDetailBuilder struct { - code string - feeds []countryDetailFeedBase - categoryTotals map[string]*detailCategoryAggregate - maintainerTotals map[string]*detailMaintainerAggregate - asnTotals map[uint32]*countryDetailASNAggregate - totalAttributed uint64 + code string + feeds []countryDetailFeedBase + facets detailFacetAccumulator + asnTotals map[uint32]*countryDetailASNAggregate + totalAttributed uint64 } func newCountryDetailBuilder(code string) *countryDetailBuilder { return &countryDetailBuilder{ - code: strings.ToUpper(strings.TrimSpace(code)), - categoryTotals: make(map[string]*detailCategoryAggregate), - maintainerTotals: make(map[string]*detailMaintainerAggregate), - asnTotals: make(map[uint32]*countryDetailASNAggregate), + code: strings.ToUpper(strings.TrimSpace(code)), + facets: newDetailFacetAccumulator(), + asnTotals: make(map[uint32]*countryDetailASNAggregate), } } func (b *countryDetailBuilder) addFeed(row countryDetailFeedBase, maintainerURL string) { b.feeds = append(b.feeds, row) b.totalAttributed += row.AttributedIPs - - categoryAgg := b.categoryTotals[row.Category] - if categoryAgg == nil { - categoryAgg = &detailCategoryAggregate{} - b.categoryTotals[row.Category] = categoryAgg - } - categoryAgg.feedCount++ - categoryAgg.attributedIPs += row.AttributedIPs - - maintainerName := strings.TrimSpace(row.Maintainer) - if maintainerName == "" { - return - } - slug := maintainerSlugify(maintainerName) - maintainerAgg := b.maintainerTotals[slug] - if maintainerAgg == nil { - maintainerAgg = &detailMaintainerAggregate{ - slug: slug, - name: maintainerName, - url: maintainerURL, - } - b.maintainerTotals[slug] = maintainerAgg - } - if maintainerAgg.url == "" && maintainerURL != "" { - maintainerAgg.url = maintainerURL - } - maintainerAgg.feedCount++ - maintainerAgg.attributedIPs += row.AttributedIPs + b.facets.add(row.Category, row.Maintainer, maintainerURL, row.AttributedIPs) } func (b *countryDetailBuilder) addASN(asn uint32, name string, count uint64) { @@ -92,43 +63,8 @@ func (b *countryDetailBuilder) build(geoProvider, asnProvider HomeSummaryProvide return b.feeds[i].Name < b.feeds[j].Name }) - topCategories := make([]DetailCategorySummary, 0, len(b.categoryTotals)) - for category, agg := range b.categoryTotals { - topCategories = append(topCategories, DetailCategorySummary{ - Category: category, - FeedCount: agg.feedCount, - AttributedIPs: agg.attributedIPs, - }) - } - sort.Slice(topCategories, func(i, j int) bool { - if topCategories[i].AttributedIPs != topCategories[j].AttributedIPs { - return topCategories[i].AttributedIPs > topCategories[j].AttributedIPs - } - if topCategories[i].FeedCount != topCategories[j].FeedCount { - return topCategories[i].FeedCount > topCategories[j].FeedCount - } - return topCategories[i].Category < topCategories[j].Category - }) - - topMaintainers := make([]DetailMaintainerSummary, 0, len(b.maintainerTotals)) - for _, agg := range b.maintainerTotals { - topMaintainers = append(topMaintainers, DetailMaintainerSummary{ - Slug: agg.slug, - Name: agg.name, - URL: agg.url, - FeedCount: agg.feedCount, - AttributedIPs: agg.attributedIPs, - }) - } - sort.Slice(topMaintainers, func(i, j int) bool { - if topMaintainers[i].AttributedIPs != topMaintainers[j].AttributedIPs { - return topMaintainers[i].AttributedIPs > topMaintainers[j].AttributedIPs - } - if topMaintainers[i].FeedCount != topMaintainers[j].FeedCount { - return topMaintainers[i].FeedCount > topMaintainers[j].FeedCount - } - return topMaintainers[i].Name < topMaintainers[j].Name - }) + topCategories := b.facets.topCategories() + topMaintainers := b.facets.topMaintainers() topASNs := make([]CountryDetailASN, 0, len(b.asnTotals)) for asn, agg := range b.asnTotals { @@ -156,8 +92,8 @@ func (b *countryDetailBuilder) build(geoProvider, asnProvider HomeSummaryProvide Totals: CountryDetailTotals{ FeedsMatching: len(b.feeds), AttributedIPsInFeed: b.totalAttributed, - Categories: len(b.categoryTotals), - Maintainers: len(b.maintainerTotals), + Categories: b.facets.categoryCount(), + Maintainers: b.facets.maintainerCount(), ASNs: len(b.asnTotals), }, Feeds: b.feeds, @@ -177,8 +113,7 @@ type asnDetailBuilder struct { name string description string feeds []asnDetailFeedBase - categoryTotals map[string]*detailCategoryAggregate - maintainerTotals map[string]*detailMaintainerAggregate + facets detailFacetAccumulator countryTotals map[string]*asnDetailCountryAggregate distributionCounts map[string]uint64 totalAttributed uint64 @@ -188,8 +123,7 @@ type asnDetailBuilder struct { func newASNDetailBuilder(asn uint32) *asnDetailBuilder { return &asnDetailBuilder{ asn: asn, - categoryTotals: make(map[string]*detailCategoryAggregate), - maintainerTotals: make(map[string]*detailMaintainerAggregate), + facets: newDetailFacetAccumulator(), countryTotals: make(map[string]*asnDetailCountryAggregate), distributionCounts: make(map[string]uint64), } @@ -201,34 +135,7 @@ func (b *asnDetailBuilder) addFeed(row asnDetailFeedBase, maintainerURL, observe if b.name == "" && observedName != "" { b.name = observedName } - - categoryAgg := b.categoryTotals[row.Category] - if categoryAgg == nil { - categoryAgg = &detailCategoryAggregate{} - b.categoryTotals[row.Category] = categoryAgg - } - categoryAgg.feedCount++ - categoryAgg.attributedIPs += row.AttributedIPs - - maintainerName := strings.TrimSpace(row.Maintainer) - if maintainerName == "" { - return - } - slug := maintainerSlugify(maintainerName) - maintainerAgg := b.maintainerTotals[slug] - if maintainerAgg == nil { - maintainerAgg = &detailMaintainerAggregate{ - slug: slug, - name: maintainerName, - url: maintainerURL, - } - b.maintainerTotals[slug] = maintainerAgg - } - if maintainerAgg.url == "" && maintainerURL != "" { - maintainerAgg.url = maintainerURL - } - maintainerAgg.feedCount++ - maintainerAgg.attributedIPs += row.AttributedIPs + b.facets.add(row.Category, row.Maintainer, maintainerURL, row.AttributedIPs) } func (b *asnDetailBuilder) addCountry(code string, count uint64) { @@ -258,43 +165,8 @@ func (b *asnDetailBuilder) build(asnProvider, geoProvider HomeSummaryProvider) * return b.feeds[i].Name < b.feeds[j].Name }) - topCategories := make([]DetailCategorySummary, 0, len(b.categoryTotals)) - for category, agg := range b.categoryTotals { - topCategories = append(topCategories, DetailCategorySummary{ - Category: category, - FeedCount: agg.feedCount, - AttributedIPs: agg.attributedIPs, - }) - } - sort.Slice(topCategories, func(i, j int) bool { - if topCategories[i].AttributedIPs != topCategories[j].AttributedIPs { - return topCategories[i].AttributedIPs > topCategories[j].AttributedIPs - } - if topCategories[i].FeedCount != topCategories[j].FeedCount { - return topCategories[i].FeedCount > topCategories[j].FeedCount - } - return topCategories[i].Category < topCategories[j].Category - }) - - topMaintainers := make([]DetailMaintainerSummary, 0, len(b.maintainerTotals)) - for _, agg := range b.maintainerTotals { - topMaintainers = append(topMaintainers, DetailMaintainerSummary{ - Slug: agg.slug, - Name: agg.name, - URL: agg.url, - FeedCount: agg.feedCount, - AttributedIPs: agg.attributedIPs, - }) - } - sort.Slice(topMaintainers, func(i, j int) bool { - if topMaintainers[i].AttributedIPs != topMaintainers[j].AttributedIPs { - return topMaintainers[i].AttributedIPs > topMaintainers[j].AttributedIPs - } - if topMaintainers[i].FeedCount != topMaintainers[j].FeedCount { - return topMaintainers[i].FeedCount > topMaintainers[j].FeedCount - } - return topMaintainers[i].Name < topMaintainers[j].Name - }) + topCategories := b.facets.topCategories() + topMaintainers := b.facets.topMaintainers() topCountries := make([]ASNDetailCountry, 0, len(b.countryTotals)) for code, agg := range b.countryTotals { @@ -334,8 +206,8 @@ func (b *asnDetailBuilder) build(asnProvider, geoProvider HomeSummaryProvider) * Totals: ASNDetailTotals{ FeedsMatching: len(b.feeds), AttributedIPs: b.totalAttributed, - Categories: len(b.categoryTotals), - Maintainers: len(b.maintainerTotals), + Categories: b.facets.categoryCount(), + Maintainers: b.facets.maintainerCount(), Countries: len(b.countryTotals), }, Feeds: b.feeds,