.cursor-updates

# Cursor Updates

## 2024-12-19

- Added URL cleaning function to detect and remove extra closing brackets that cause 404 errors in OpenGraph data fetching
- Implemented fallback mechanism in LinkPreviewCard to retry with cleaned URL if first attempt fails
- Added logging for debugging when URLs are cleaned
- Fixed child_datasets data not being passed through to StudyDetail component in search page
- Fixed styling inconsistency in child datasets section header to match other collapsible sections
- Added cleanup API function to services/api.ts for AI-powered dataset summarization using JSON schema data
- Updated system prompts to focus on academic research dataset summarization with metadata and variables
- Integrated cleanup API into StudyDetail component AI Summary section that sends dataset_schema as JSON
- AI Summary uses dataset schema from original SearchResult for comprehensive dataset analysis
- Enhanced AI Summary display to properly format array response with question_text objects as paragraphs
- Added support for basic markdown formatting (bold text) in AI Summary display
- Optimized AI Summary API calls by filtering out large variableMeasured arrays and replacing with variableCount
- Updated system prompts to handle variableCount field instead of full variable lists
- Added knowledge-based AI summarization option that uses AI's existing knowledge of studies
- Implemented automatic fallback from knowledge-based to schema-based summarization when AI doesn't know the study
- Removed toggle UI and made the fallback process seamless for users
- Added conditional display to only show AI Summary section for studies (resourceType === "study")
- Improved knowledge-based prompt to return empty responses for generic study names
- Implemented desktop study detail pane expansion feature with smooth transitions
- Added responsive layout that expands detail pane to 80% width when clicked (20% for results)
- Created collapsed SearchResults view showing study images/logos in card format maintaining height
- Added sticky chevron button to collapse back to 50/50 split instead of clicking results area
- Improved UX by not collapsing when selecting different results - stays expanded until manually collapsed
- Moved collapse button to fixed position at top-right of results panel, outside scroll area for always-visible access
- Simplified collapsed results view to show normal cards with content hidden below image line for cleaner sliding effect
- Refactored to use absolute positioning overlay approach - results stay at 50% width, detail pane overlays with flexbox
- Added subtle drop shadow to detail pane when expanded for better visual hierarchy
- Moved collapse button to detail pane (top-left) with click event prevention to avoid triggering detail click
- Added blue borders on both left and right sides of selected result cards for visibility when detail pane is expanded
- Enhanced filter panel with hybrid chips/autocomplete component for categorical filters with frequency data
- Implemented get-keyword-phrases API endpoint integration for intelligent alpha calculation based on study names and abbreviations
- Modified search logic to use alpha=0 for queries matching keyword phrases instead of heuristic word-count based calculation
- Updated all fetchResultByUuid calls to include alpha parameter for consistent search behavior across lookup operations
- Simplified related datasets table to show only dataset name, number of variables, and time coverage
- Updated dataset links to open internal pages at /items/{slug} instead of external URLs
- Enhanced /items/{slug} route to support both slugs and UUIDs for child datasets from lookup API
- Added UUID detection logic to automatically handle datasets without slugs
- Fixed static generation issue by adding return_variables_within_parent=false parameter to dataset search API calls
- Updated generateStaticParams to create pages for both slug and UUID versions of each dataset
- Fixed UUID detection to recognize hash-based IDs (32 hex characters) in addition to proper UUIDs with dashes
- Optimized static generation to only pre-compile child datasets from studies instead of all datasets
- Simplified dynamic page rendering to use direct API lookups instead of searching through all studies
- Enhanced fetchResultByUuid to support both UUID and slug lookups with automatic fallback
- Simplified static generation to build studies by both slug and UUID (with UUID redirects to slug)
- Updated child dataset pages to use StudyDetail component with proper data transformation
- Switched XLSX library from npm package to local xlsx.min.js (v0.20.3) in public/js to prevent security vulnerabilities
- Removed vercel package from dependencies (not needed - Vercel builds automatically detect Next.js projects)
- Removed all/matching toggle from variable table and simplified to always use QuickFilter with initial query value
- Reorganized variable table toolbar to have search box and harmony/save buttons on a single line
- Set initial QuickFilter value to mainSearchQuery when page loads with a query for immediate matching results
- Added debouncing to variable table search (500ms delay) to reduce server calls while typing, ensuring last call always executes
- Modified API calls to pass min_original_vector_score (1 - max_vector_distance) instead of max_vector_distance
- Added three-way toggle to Max Distance slider to switch between sending max_distance, min_score, or both parameters to API calls (search, lookup, get_variables)- Changed default resultsPerPage from 100 to 500 to work better with Weaviate fusion
- Only update totalHits (num_results) from the first page call (page 1, no offset, no exclusions) to keep reliable estimate
- Added retry logic to get_variables: if no results with search term, retry without search term; hide variables table if still no results
- Fixed issue where results list was not cleared when search field is deleted and there are no filters
- Added localStorage persistence for advanced search settings (useSearch2, hybridWeight, maxDistance, maxDistanceMode, directMatchWeight, paginationStrategy) - settings are saved when changed and restored on page load if not in URL params
- Updated variable count logic to prioritize num_descendant_variables from lookup object as primary source in CompactResultCard and StudyDetail components
- Fixed issue where results list was not cleared when search field is deleted and there are no filters
- Added localStorage persistence for advanced search settings (useSearch2, hybridWeight, maxDistance, maxDistanceMode, directMatchWeight, paginationStrategy) - settings are saved when changed and restored on page load if not in URL params
- Enhanced trust estimate mode to handle JSON parse errors gracefully: if batch lookup fails, tries individual UUID lookups, continues processing all batches, logs unparseable UUIDs to console, and displays failed count in results summary
- Made Response Options column in variables table dynamically hidden when API response contains no response options data - column only appears if at least one variable has response_options or options data
- Added narrow URL column to variables table with external link icon - column only appears if variables have URLs array, displays clickable icon linking to first URL in array
- Connected MUI DataGrid sorting to API sort_order parameter - when user clicks column header to sort, sortModel is extracted from params and converted to "field:asc/desc" format for API call
- Changed filterMode and sortingMode to "client" (pagination remains "server") - filtering and sorting now happen client-side on fetched data, rowCount from API response (num_hits) is passed to DataGrid for proper pagination display
- Updated rowCount logic: on first page with filter, use results.length if < pageSize (exact) or pageSize if full page (minimum); on first page without filter, use num_hits (unfiltered total). When no query, only send ancestor_uuid, offset, and num_results to API (table handles offset/limit). Green bar variable count captures num_hits from first DataGrid request (num_hits is always unfiltered total regardless of query). Simplified refs - removed complex tracking, just reset request key when study changes to prevent duplicate calls. QuickFilter only initialized with mainSearchQuery if variablesWhichMatched exists and has items, otherwise set to empty
- Removed duplicate request prevention that was blocking legitimate DataGrid requests
- Fixed 'No results' message flashing during search - now only shows after search actually completes (loading goes from true to false), preventing false "0 results" display while search is in progress
- Fixed variables table hiding logic - now correctly hides when get_variables returns no results on first page, regardless of whether there's a search query
- Replaced /items/[slug] with catch-all route /items/[[...slug]] that pre-generates and pre-renders all item routes (by slug and UUID, 21k+ routes) with data at build time, just like studies pages - fetches from cache/API during build and passes to component, fetchResultByUuid handles both slugs and UUIDs
- Removed all optional parameters (query, alpha, maxVectorDistance, maxDistanceMode) from fetchResultByUuid and fetchResultsByUuids lookup endpoints - now only passes uuid or slug to improve API performance, batch lookup still supported
- Added Google Analytics 4 with anonymized IP tracking (no cookie consent required) - configured with anonymize_ip, allow_google_signals: false, and allow_ad_personalization_signals: false for GDPR compliance
- Added Hotjar tracking gated behind harmonyCookieConsent cookie (shared with harmonydata.github.io) - created HotjarTracker client component that checks for consent cookie and polls for changes to enable cross-site consent sharing
- Fixed TypeScript errors in admin page by adding explicit type annotations to valueFormatter functions and using match_reported field instead of non-existent question_index
- Added parseFilters helper function in admin.ts to handle both old (JSON string) and new (object/map) filter formats for backward compatibility
- Fixed variables table search input losing focus when get_variables returns - extracted QuickFilter toolbar to stable component (VariablesGridToolbar) to prevent re-mount on parent re-render