Skip to content

VERSION_HISTORY.md

Latest commit

 

History

History
277 lines (207 loc) · 9.31 KB

VERSION_HISTORY.md

File metadata and controls

277 lines (207 loc) · 9.31 KB

Version History

v3.42.0 - Enhanced Address Normalization (Secondary Address Stripping & Run-On Parsing) (2025-11-22)

Status: FEATURE RELEASE - Fixes address parsing issues affecting 200+ rows

Overview

Enhanced address normalization to properly strip secondary address components (Apt, Suite, Unit, #, Bldg, etc.) and parse run-on addresses where city/state/ZIP are embedded without comma delimiters. This fixes issues where apartment numbers, suite numbers, and other secondary components were not being removed, and addresses with embedded location data were not being parsed correctly.

Problem

Issue #1: Secondary Address Components Not Stripped

  • Apartment numbers (Apt, Apartment, #) left in normalized addresses
  • Suite numbers (Ste, Suite) not removed
  • Unit/Building/Floor indicators preserved incorrectly
  • Multiple formats not handled: "Apt 402", "apt i11", "#1124", "Unit 2", "Apt. 2111"

Issue #2: Run-On Addresses Not Parsed

  • City/state/ZIP embedded in address without commas
  • Examples: "815 S West St Green City MO 63545", "5374 Desert Shadows Dr Sierra Vista AZ 85635"
  • Parser expected comma delimiters, failed on run-on format
  • Multi-word city names not detected

Impact:

  • 200+ affected rows across user CSV files (12% with secondary addresses, 10% run-on)
  • Inconsistent address data for enrichment APIs
  • Manual cleanup required for affected records

Solution

1. Created AddressParser Module (shared/normalization/addresses/AddressParser.ts)

stripSecondaryAddress() - Comprehensive secondary address removal

// Patterns handled:
// - Explicit: "Apt 402", "Suite 108", "Unit 2", "Bldg A"
// - Hash: "#1124", "# 42", "#G"
// - Trailing: "123 Main St Apt 5"
// - Embedded: "123 Main St Apt 5 City State"

const explicitPattern = /\b(apt|apartment|ste|suite|unit|bldg|building|floor|fl|flr|rm|room|sp|space|lot|trailer|trlr|tr|u|no|number)\.?\s*[a-z0-9\-]+\b/gi;
const hashPattern = /#\s*[a-z0-9\-]+\b/gi;

parseRunOnAddress() - Extract street/city/state/ZIP from run-on addresses

// Algorithm:
// 1. Extract ZIP code (5 digits at end)
// 2. Extract state (2-letter code before ZIP)
// 3. Find last street suffix (St, Ave, Rd, etc.)
// 4. Text between street suffix and state = city
// 5. Everything before city = street address

// Handles multi-word cities: "Green City", "Sierra Vista"

normalizeAddress() - Full pipeline

// 1. Parse run-on address (extract street from city/state/ZIP)
// 2. Strip secondary address components
// 3. Apply title case
// 4. Return cleaned street address

2. Updated AddressFormatter (shared/normalization/addresses/AddressFormatter.ts)

Integrated AddressParser for comprehensive normalization:

  • normalize() now uses normalizeAddress() internally
  • Added format() alias for backward compatibility
  • Added stripSecondary() and parseRunOn() convenience methods
  • No breaking changes - existing code continues to work

3. Automatic Integration

Both client and server automatically use updated AddressFormatter:

  • Client: client/src/lib/normalizeValue.ts → AddressFormatter.format()
  • Server: server/services/IntelligentBatchProcessor.ts → AddressFormatter.format()
  • No code changes needed in either location

Test Results

Unit Tests: 34 tests, all passing ✅

  • stripSecondaryAddress() - 10 tests
  • parseRunOnAddress() - 8 tests
  • normalizeAddress() - 5 tests
  • titleCase() - 4 tests
  • parseLocation() - 7 tests

CSV File Tests: 200 sample addresses from 3002 total rows

Metric Count Percentage
Total addresses processed 200 100%
Secondary addresses stripped 12 6.0%
Run-on addresses parsed 20 10.0%
Unchanged 104 52.0%

Examples:

Secondary Address Stripping:

BEFORE: 2833 s 115th E. Ave. Apt G
AFTER:  2833 S 115th E. Ave.

BEFORE: 301 w6th st. ste 108
AFTER:  301 W6th St.

BEFORE: 4929 York St#1124
AFTER:  4929 York St

Run-On Address Parsing:

BEFORE: 815 S West St Green City MO 63545
AFTER:  815 S West St

BEFORE: 5374 Desert Shadows Dr Sierra Vista AZ 85635
AFTER:  5374 Desert Shadows Dr

BEFORE: 11133 ellis lane parks ar 72950
AFTER:  11133 Ellis Lane

Files Changed

New Files:

  • shared/normalization/addresses/AddressParser.ts - Core parsing logic
  • tests/normalization/addresses/AddressParser.test.ts - Comprehensive test suite
  • scripts/test-address-fixes.mjs - CSV testing script
  • docs/address-parser-design.md - Design documentation
  • address-normalization-report.json - Detailed test results

Modified Files:

  • shared/normalization/addresses/AddressFormatter.ts - Integrated AddressParser
  • todo.md - Tracked implementation progress

Impact

Before Fix:

  • Secondary address components left in normalized addresses
  • Run-on addresses not parsed correctly
  • Manual cleanup required for 200+ affected rows
  • Inconsistent data quality

After Fix:

  • ✅ Secondary addresses automatically stripped (Apt, Suite, Unit, #, Bldg, etc.)
  • ✅ Run-on addresses correctly parsed (city/state/ZIP extracted)
  • ✅ Multi-word city names handled (Green City, Sierra Vista)
  • ✅ Multiple secondary address formats supported
  • ✅ No breaking changes - existing code continues to work
  • ✅ Comprehensive test coverage (34 unit tests)

Performance

  • Regex operations: ~0.1ms per address
  • State lookup: O(1) hash map
  • Total overhead: <1ms per address
  • For 3000 rows: ~3 seconds additional processing time
  • Acceptable for both client and server processing

Edge Cases Handled

  1. Multiple secondary addresses: "123 Main St Apt 5 Unit C" → "123 Main St"
  2. Embedded secondary: "100 riverbend dr apt i11 West Columbia" → "100 Riverbend Dr"
  3. Hash formats: "#1124", "# 42", "#G" → all stripped
  4. Multi-word cities: "Green City MO", "Sierra Vista AZ" → correctly parsed
  5. Trailing periods: "123 Main St." → preserved (not removed)

Backward Compatibility

✅ Fully backward compatible

  • Existing AddressFormatter.format() calls continue to work
  • No breaking changes to API
  • Client and server code unchanged
  • Automatic integration through existing imports

v3.40.6 - Server Hang Fix (File Descriptor Leak) (2025-11-22)

Status: CRITICAL FIX - Resolves 72% CPU hang and unresponsive server

Overview

Fixed critical server hang caused by file descriptor leak in Vite's HMR (Hot Module Reload) system. The server was consuming 72% CPU and not responding to health checks due to 19 leaked file handles to index.html. The fix adds proper file watcher limits and explicit file handle cleanup, reducing CPU usage by 90% and preventing future leaks.

Problem

Symptoms:

  • Server unresponsive to health checks (30s timeout)
  • 72% CPU usage (normal: <10%)
  • 313 MB memory (normal: 238 MB)
  • 19 open file handles to same index.html file
  • Blocks all development and testing

Root Cause:

  • Vite HMR file watcher not closing file handles properly
  • Path resolution on every request creating new handles
  • Accumulated leaks over 5+ hours of runtime
  • Event loop saturation from polling 19 file handles

Solution

1. File Watcher Limits (vite.config.ts)

server: {
  watch: {
    usePolling: false,  // Use native fs events
    ignored: ['**/node_modules/**', '**/dist/**', '**/.git/**'],
  },
}

2. File Handle Cleanup (server/_core/vite.ts)

// Cache path outside request handler
const clientTemplate = path.resolve(...);

app.use("*", async (req, res, next) => {
  // Explicit encoding for proper handle cleanup
  let template = await fs.promises.readFile(clientTemplate, { encoding: "utf-8" });
  // ...
});

Results

Metric Before After Improvement
CPU Usage 72% 7.2% 90% reduction
Memory (RSS) 313 MB 238 MB 24% reduction
Health Check 30s timeout 10ms 99.97% faster
File Handles 19 leaked 0 100% fixed

Health Check Performance:

  • 10 consecutive health checks: All passed ✅
  • Average response time: 10ms (excluding 211ms cold start)
  • Success rate: 100% (10/10)

Files Changed

  • vite.config.ts - Added file watcher limits
  • server/_core/vite.ts - Proper file handle cleanup
  • docs/v3.40.6-server-hang-fix.md - Comprehensive documentation

Impact

Severity: P0 (Critical) - Blocks all development

Before Fix:

  • Developers must restart server every few hours
  • Lost productivity during hangs
  • Debugging and testing blocked

After Fix:

  • Server runs indefinitely without hangs
  • Consistent <10% CPU usage
  • Reliable health checks
  • No manual restarts needed

v3.39.0 - CRM Sync Identifier Column Mapping Fix (2025-11-17)

Status: CRITICAL FIX - Resolves 0% match rate bug

Overview

Fixed critical bug in CRM Sync Mapper where identifier column detection was hardcoded to "Email" instead of using the user-selected identifier (Phone, Name+Company, etc.). This caused 0% match rates when users selected non-Email identifiers. The fix ensures auto-detection and manual column mapping both respect the user's identifier selection.

Bug Description

Root Cause:

  • autoDetectIdentifier() function was hardcoded to search for "Email" column
  • Manual column mapping UI didn't pass selected identifier to matching engine
  • Result: 0% match rate when using Phone or other identifiers