Release v0.6.32

Release v0.6.32: FamilySearch improvements, Ancestry automation, and tree visualizations

Author: atomantic

.changelog/v0.6.x.md

@@ -0,0 +1,216 @@
+# Release v0.6.x - FamilySearch Normalization & Architecture Refinement
+
+Released: YYYY-MM-DD
+
+## Overview
+
+This release normalizes FamilySearch as a downstream provider (equal to Ancestry, WikiTree, etc.) and establishes SparseTree SQLite as the canonical source of truth. Major architectural cleanup removes all legacy fallback code in favor of migration scripts.
+
+## New Features
+
+### Ancestry Update Automation Page (v0.6.x)
+- New database-scoped page at `/db/:dbId/ancestry-update` for bulk Ancestry synchronization
+- BFS traversal of direct-ancestor sparse tree from selected root person
+- Configurable generation depth (1-4 or full tree)
+- Per-person pipeline: check Ancestry link, process free hints, download/scrape provider data
+- Real-time SSE progress with emoji-prefixed execution log
+- Test mode (dry run) for previewing what would happen
+- Adds "Ancestry Update" link to database sidebar
+- API endpoints: GET `/api/ancestry-update/status`, GET `/:dbId/events` (SSE), POST `/:dbId/cancel`, GET `/:dbId/validate/:personId`
+
+### Ancestry Free Hints Automation (v0.6.x)
+- New "Hints" button on Ancestry row in Provider Data table to process free record hints
+- Playwright automation navigates to hints page, reviews each hint, accepts and saves to tree
+- Supports checkbox selection for adding related people from hints
+- Includes consecutive failure safeguard (stops after 3 failures) and remaining hints re-check
+- SSE-based progress events for real-time UI updates
+- API endpoints: POST/GET `/api/ancestry-hints/:dbId/:personId`, cancel, status
+
+### Ancestry Vital Info Upload (v0.6.x)
+- Extended Ancestry upload to support birth date, birth place, death date, death place, and name (not just photos)
+- Upload dialog now shows field-by-field comparison between local SparseTree data and cached Ancestry data
+- Browser automation uses Ancestry's Quick Edit sidebar to update person facts
+- Automatically handles Living/Deceased status toggle when setting death information
+- Pre-selects all differing fields for upload convenience
+
+### FamilySearch Normalization (v0.6.5)
+- FamilySearch now treated as equal downstream provider (no special priority)
+- Photos use `-familysearch` suffix like other providers: `{personId}-familysearch.jpg`
+- Added `linkFamilySearch()` and `getFamilySearchPhotoPath()` to augmentation service
+- ID resolution uses alphabetical order (no FamilySearch priority)
+
+### Architecture: Source of Truth Principle (v0.6.x)
+- SparseTree SQLite is now the canonical source of truth
+- Provider cache (`data/provider-cache/`) is for comparison only, not a fallback data source
+- All data reads come from SQLite; provider cache shows what providers have vs what SparseTree has
+
+## Bug Fixes
+
+### SQLite Vital Event Row ID Preservation (v0.6.x)
+- Changed vital_event INSERT from `INSERT OR REPLACE` to `ON CONFLICT DO UPDATE`
+- Preserves row IDs which are critical for local_override references
+- INSERT OR REPLACE was deleting and re-inserting rows, orphaning override references
+
+### Improved Vital Event Date Parsing (v0.6.x)
+- Now uses actual `birth.date` and `death.date` objects instead of parsing lifespan string
+- Captures full dates (e.g., "1979-07-31") rather than just years from lifespan ("1979-2020")
+- Added `parseYearFromDate()` function to handle various date formats including BC notation
+
+### Ancestry Update Log Memory Fix (v0.6.x)
+- Limited execution log entries to 500 to prevent browser memory issues on large trees
+- Shows notice when older entries are truncated
+
+### FamilySearch Upload Improvements (v0.6.x)
+- Auto-refresh FamilySearch cache after successful upload so UI shows updated data
+- Improved photo comparison to detect already-uploaded photos via symlink matching
+- Fixed Google login button selector for updated FamilySearch login page
+- Improved vitals page editing: handles conclusion dialogs and edit buttons
+- Added autocomplete handling for date/place fields (clicks first suggestion)
+
+### Scraper Auto-Login (v0.6.x)
+- Scraper now attempts auto-login via Google OAuth when landing on FamilySearch login page
+- Added `handleLoginIfNeeded()` function to detect login pages and complete OAuth flow
+- Navigates directly to target URL after successful login
+
+### Provider Comparison Not Using Local Overrides (v0.6.x)
+- Fixed comparison service using original database values instead of user-applied overrides
+- Added `applyLocalOverrides()` to fetch and apply vital_event overrides before comparison
+- FamilySearch/Ancestry now correctly show "match" when user has applied their values
+
+### Date Format Differences Showing as Different (v0.6.x)
+- Added date normalization to expand month abbreviations (e.g., "AUG" → "august")
+- "29 AUG 1933" now correctly matches "29 August 1933"
+- Reduces false positives in provider comparison diff counts
+
+### Unicode Escape in View Mode Selector (v0.6.x)
+- Fixed checkmark showing as literal `\u2713` text instead of ✓ in tree view mode dropdown
+- JSX text content requires actual Unicode characters, not escape sequences
+
+### Fan Chart Ancestor Count (v0.6.x)
+- Fixed footer showing incorrect count like "109 of 62 ancestors shown"
+- Now correctly counts only ancestors within the visible generation range
+
+### Stale Browser Connection Detection (v0.6.x)
+- Added `verifyAndReconnect()` to browser service that checks both Playwright state and actual browser process
+- Augmentation service now detects and recovers from stale connections before scraping
+- Fixes silent failures when browser was closed but connection state was still "connected"
+
+### Auto-Login on Sync (v0.6.x)
+- FamilySearch sync now attempts automatic login when auth token is missing
+- Uses `providerService.ensureAuthenticated()` to try Google SSO or stored credentials
+- Eliminates need for manual browser login before sync operations
+- Applies to both `refreshPerson()` and `fetchPersonDisplayName()` methods
+
+### FamilySearch Scraper Not Extracting Data (v0.6.x)
+- Updated scraper selectors to match current FamilySearch DOM structure
+- Name now extracted using `[data-testid="fullName"]`
+- Birth/death dates now extracted from `[data-testid="conclusionDisplay:BIRTH"]` and `[data-testid="conclusionDisplay:DEATH"]` containers
+- Added handling for collapsed Vitals section (auto-expands before extracting)
+- Photo extraction uses avatar container near `[data-testid="update-portrait-button"]`
+- Fixed TypeScript `__name` decorator issue causing `page.evaluate` to fail in browser context
+- Added detailed debug logging for photo extraction troubleshooting
+- Improved UI feedback: loading toast during scrape, success toast shows extracted fields
+
+### "Use" Button Not Updating SparseTree Row (v0.6.x)
+- Fixed clicking "Use" on provider field values not updating the SparseTree row display
+- Root causes fixed:
+  - Client: `onFieldChanged` callback now refetches person data after applying override
+  - Server: `getPerson()` now applies local overrides to returned person data
+- Added `applyLocalOverridesToPerson()` to database service for consistent override application
+- Birth/death date/place values from providers now properly display after clicking "Use"
+
+### "Set as Primary" Photo Not Updating (v0.6.x)
+- Fixed photo not updating after clicking "Set as Primary" on provider photos
+- Root causes fixed:
+  - `getPhotoPath()` now checks for primary photo (`{personId}.jpg`) before provider-suffixed photos
+  - Photo priority in PersonDetail header and SparseTree row now shows primary photo first
+  - Added `Cache-Control: no-cache` headers to all photo "exists" endpoints to prevent stale responses
+- Added FamilySearch-specific photo endpoint: `GET /augment/:personId/familysearch-photo`
+- Added client API methods: `getFsPhotoUrl()`, `hasFsPhotoProvider()`
+- Added `photoVersion` state for cache-busting photo URLs after changes
+
+### HMR Context Resilience (v0.6.x)
+- Fixed "useSidebar must be used within a SidebarProvider" error during HMR transitions
+- `useSidebar()` and `useTheme()` hooks now return safe defaults during dev hot reloads
+- Prevents crashes when React context identity changes during development
+
+## Improvements
+
+### FamilySearch Upload Performance (v0.6.x)
+- Eliminated redundant post-login delays in FamilySearch upload automation
+- After Google OAuth redirect, now checks if already on target page before re-navigating
+- Replaced fixed `waitForTimeout` delays with element-based `waitForSelector` for responsiveness
+- Added PM2 timestamps to server logs for better debugging (`time: true` in ecosystem config)
+
+### Page-Level Padding Control (v0.6.x)
+- Moved `p-6` padding from Layout to individual pages for flexibility
+- Full-height pages (tree views, PersonDetail, SparseTreePage) control their own layout
+- Standard content pages receive consistent padding via their root element
+- Fixed missing padding on SparseTreePage and PersonDetail
+- Removed double padding from AIProviders page (external component handles its own padding)
+
+### Tree Visualization Parity with Ancestry.com (v0.6.x)
+- Added 5 tree view modes matching Ancestry.com functionality
+- **Fan Chart** (default): Radial pedigree with lineage-colored wedges (paternal=blue/teal, maternal=red/coral)
+- **Horizontal Pedigree**: Root left, ancestors right with expandable nodes
+- **Vertical Family**: Ancestors top, root middle with generation labels
+- **Columns** and **Focus**: Existing views retained
+- URL-based routing: `/tree/:dbId/:personId/:viewMode`
+- New shared components: TreeCanvas (D3 zoom/pan), TreeControls, AncestorNode
+- New utilities: lineageColors, treeLayout, arcGenerator
+- Fixed classic view bugs: dynamic centering, ResizeObserver instead of setTimeout
+
+### Vertical Family View Improvements (v0.6.x)
+- Refactored layout to use multi-pass collision resolution algorithm
+- Parents properly avoid overlapping when multiple branches are expanded
+- Added horizontal coupling bars between parent pairs (matching Ancestry.com style)
+- Fixed expand button centering on cards
+- Expansion now maintains current zoom level and pans to center on expanded node
+- No longer zooms out and shifts to bottom-right when expanding ancestors
+- Fixed expand/collapse button positioning to overlap card tops consistently
+- Moved spouse connection line from middle of card to name/date area
+
+### Avatar Placeholders (v0.6.x)
+- Replaced emoji-based avatar placeholders with proper SVG components throughout tree views
+- New `AvatarPlaceholder` component renders gender-aware silhouettes (male/female/unknown profiles)
+- Updated PersonCard, FocusNavigatorView, GenerationalColumnsView, PedigreeChartView, and SparseTreePage
+- Softened gender color palette for better visual harmony (less saturated blue/pink)
+
+### Legacy Code Removal (v0.6.x)
+- Removed all legacy fallback code from services (no more dual-path lookups)
+- Services updated: `familysearch-refresh`, `multi-platform-comparison`, `augmentation`, `scraper`, `person.routes`
+- Architecture policy: use migration scripts instead of maintaining legacy paths
+
+## Migration Scripts
+
+### Photo Migration
+- `scripts/migrate-fs-photos.ts` - Renames FamilySearch photos from `{personId}.jpg` to `{personId}-familysearch.jpg`
+
+### Cache Migration
+- `scripts/migrate-legacy-cache.ts` - Moves JSON files from `data/person/` to `data/provider-cache/familysearch/`
+
+## Technical Details
+
+### Source of Truth Architecture
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                     Layer 3: Local Overrides                    │
+│  User edits that take precedence and survive provider re-sync   │
+├─────────────────────────────────────────────────────────────────┤
+│                    Layer 2: Normalized Data (SQLite)            │
+│  CANONICAL SOURCE OF TRUTH - all reads come from here           │
+├─────────────────────────────────────────────────────────────────┤
+│                     Layer 1: Raw Provider Cache                 │
+│  For comparison only - shows what providers have                │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+### Key Principles
+1. **Initial Seeding**: When indexing, data writes to both SQLite (as record) AND provider-cache (for comparison)
+2. **No Fallback Code**: Migration scripts upgrade data in place; no dual-path lookups
+3. **Explicit Apply**: Downloaded provider data is cached but never auto-applied; users click "Use" buttons
+4. **Provider Equality**: All providers treated equally as downstream data sources
+
+## Full Changelog
+
+**Full Diff**: https://github.com/atomantic/SparseTree/compare/v0.5.4...v0.6.x