Backup Integrity Verification, Storage Explorer Caching, and Multiple Improvements

.github/copilot-instructions.md

@@ -95,26 +95,31 @@ return { success: boolean, message?: string, data?: any, error?: string }
 ## Developer Workflows
 
 ```bash
-pnpm dev                  # Start dev server (http://localhost:3000)
+pnpm dev                  # Start dev server - auto-applies pending migrations on startup
 pnpm test                 # Unit tests (vitest)
 pnpm test:integration     # Integration tests against real DB containers
 pnpm test:ui              # Spin up 16 test DBs + seed local DB for manual testing
 pnpm run build            # Production build (validate before commit)
-npx prisma migrate dev    # Create DB migration
+npx prisma migrate dev    # Create a new DB migration (stop dev server first)
+pnpm run database:reset   # Reset dev DB from scratch (drops + recreates via all migrations)
 ```
 
 **Test Infrastructure**: See [docker-compose.test.yml](docker-compose.test.yml) for MySQL/PG/Mongo containers.
 
 ### Prisma Migrations - IMPORTANT
 
+`pnpm dev` automatically runs `prisma migrate deploy` on startup, so the local DB is always up to date with all pending migrations. No manual step needed after pulling changes that include new migrations.
+
 **Never run `prisma migrate dev` while `pnpm dev` is running.** The dev server holds an open SQLite connection. `migrate dev` can trigger an interactive DB reset (on drift), which conflicts with the file lock and crashes the Node process - and often VS Code with it.
 
+**Never use `prisma db push`.** It applies schema changes without creating a migration file, causing the local `_prisma_migrations` table to diverge from the actual schema. This breaks `database:deploy` in production and for other developers. Always create a proper migration.
+
 **Safe workflow for schema changes:**
 1. Stop the dev server first (Ctrl+C in the node terminal)
 2. Run `npx prisma migrate dev --name <migration-name>`
-3. Restart `pnpm dev`
+3. Restart `pnpm dev` - migrations apply automatically on startup
 
-**Alternative for local dev only:** `npx prisma db push` - applies schema changes without migration history, safe to run alongside the dev server, and never prompts for a reset.
+**Reset dev DB from scratch:** `pnpm run database:reset` (runs `prisma migrate reset` - drops and recreates from all migrations).
 
 ## Queue System (`src/lib/execution/queue-manager.ts`)
 

api-docs/openapi.yaml

@@ -1,7 +1,7 @@
 openapi: 3.1.0
 info:
   title: DBackup API
-  version: 2.6.0
+  version: 2.7.0
   description: |
     REST API for DBackup - a self-hosted database backup automation platform with encryption, compression, and smart retention.
 

docs/.vitepress/config.mts

@@ -171,6 +171,7 @@ export default defineConfig({
           collapsed: false,
           items: [
             { text: 'Storage Explorer', link: '/user-guide/features/storage-explorer' },
+            { text: 'Backup Verification', link: '/user-guide/features/backup-verification' },
             { text: 'Restore', link: '/user-guide/features/restore' },
             { text: 'Notifications', link: '/user-guide/features/notifications' },
             { text: 'System Backup', link: '/user-guide/features/system-backup' },
@@ -212,7 +213,8 @@ export default defineConfig({
             { text: 'Icon System', link: '/developer-guide/core/icons' },
             { text: 'Download Tokens', link: '/developer-guide/core/download-tokens' },
             { text: 'Rate Limiting', link: '/developer-guide/core/rate-limiting' },
-            { text: 'Update Service', link: '/developer-guide/core/updates' }
+            { text: 'Update Service', link: '/developer-guide/core/updates' },
+            { text: 'Storage List Cache', link: '/developer-guide/core/storage-cache' }
           ]
         },
         {

docs/changelog.md

@@ -2,6 +2,48 @@
 
 All notable changes to DBackup are documented here.
 
+## v2.7.0 - Backup Integrity Verification, Storage Explorer Caching, and Multiple Improvements
+*Released: June 14, 2026*
+
+### ✨ Features
+
+- **integrity**: Added backup file integrity verification. The Storage Explorer shows SHA-256/MD5 checksums and last verification result per file, with a Verify Now button that runs as a tracked async execution with live progress and cancel support.
+- **integrity**: Native checksum verification for S3/R2/Hetzner (SHA-256 via object metadata), Google Drive (MD5 via API), OneDrive (SHA-256 via Graph API), and local filesystem - no download required. Both checksums are stored in `.meta.json` at upload time.
+- **integrity**: Post-upload verification is now opt-in for all storage destinations via system settings (local filesystem always verifies).
+- **integrity**: Scheduled integrity checks support two scan modes - **Jobs** (default, only verifies files linked to backup jobs) and **All Files** (full storage scan) - plus configurable filters (skip already-passed, max age, max file size). Jobs and storage destinations can individually opt out via a "Skip Verification" toggle.
+- **storage**: Storage Explorer file listings are now cached in SQLite for instant repeat visits. Cache is invalidated on backup create, delete, verify, and lock changes.
+- **history**: Execution logs can now be copied to the clipboard or downloaded as a `.log` file directly from the log dialog. Sensitive data (IPs, credentials, connection strings) is automatically redacted before export.
+
+### 🐛 Bug Fixes
+
+- **storage**: Google Drive, OneDrive, and Dropbox now live-validate their stored OAuth token when the Connection tab is opened, showing an expiry warning with a Re-authorize button instead of a misleading green "authorized" status.
+
+### 🎨 Improvements
+
+- **dev**: `pnpm dev` now automatically applies pending Prisma migrations and regenerates the Prisma client on startup.
+- **storage**: Storage Explorer cache uses surgical updates - create, delete, lock, and verify each patch only the affected cache entry instead of invalidating the full cache.
+- **storage**: A new "Pre-warm Storage Cache" system task (enabled by default, hourly) reconciles caches against remote storage and pre-populates the cache for adapters not yet visited.
+- **storage**: Stale cache entries trigger a background reconciliation via `adapter.list()` to detect files deleted outside DBackup without re-reading sidecars.
+- **storage**: Long backup names in the Storage Explorer are truncated with a tooltip showing the full name on hover.
+- **integrity**: Manual system task runs auto-redirect to the live execution history when "Auto-redirect on job start" is enabled.
+- **ui**: All data tables (Storage, Jobs, Sources, Destinations, Notifications) now support horizontal scrolling when columns overflow.
+- **explorer**: Database version history now detects and visually distinguishes downgrades - the change log shows an orange downward arrow for downgrades vs. green upward arrow for upgrades, and downgrade notifications are sent with a distinct "Downgrade" label and warning color.
+- **explorer**: Database Explorer table list now supports searching by name and sorting by Name, Type, Rows, or Size via clickable column headers.
+
+### 🧪 Tests
+
+- **tests**: Updated unit tests for integrity service, upload step, storage service, and system task service to match the refactored verification interfaces.
+- **tests**: Fixed integrity service tests to use destinations scan mode explicitly and added `skipVerification` field to job service create test.
+- **tests**: Added unit tests for `VerificationService` (all verification code paths), `SystemTaskRunner` (execution lifecycle), `logs/sanitize` (credential and IP redaction), and `logs/format` (log text export). Extended `IntegrityService` tests to cover the jobs-mode scan path.
+
+### 🐳 Docker
+
+- **Image**: `skyfay/dbackup:v2.7.0`
+- **Also tagged as**: `latest`, `v2`
+- **CI Image**: `skyfay/dbackup:ci`
+- **Platforms**: linux/amd64, linux/arm64
+
+
 ## v2.6.0 - Security Update, Vault Credential Profiles, OAuth Improvements, and Multiple Bug Fixes
 *Released: June 6, 2026*
 

docs/developer-guide/core/storage-cache.md

@@ -0,0 +1,122 @@
+# Storage List Cache
+
+The Storage Explorer calls `adapter.list("")` (recursive folder traversal) plus one `adapter.read()` per `.meta.json` sidecar on every load. For remote adapters like Google Drive this means dozens of API calls per page view. The storage list cache stores the full enriched listing in SQLite so repeat visits are instant.
+
+## Database Model
+
+```prisma
+model StorageListCache {
+  adapterConfigId String        @id
+  filesJson       String        // JSON array of RichFileInfo — full list, no typeFilter
+  cachedAt        DateTime      @default(now())
+  adapterConfig   AdapterConfig @relation(..., onDelete: Cascade)
+}
+```
+
+One row per storage adapter. `cachedAt` drives the staleness check.
+
+## Read Path
+
+`StorageService.listFilesWithMetadata(adapterConfigId, typeFilter?, bypassCache?)`:
+
+1. If `bypassCache = false` (default): query `StorageListCache` by `adapterConfigId`.
+2. **Cache hit**: check age. If `cachedAt` is older than `CACHE_STALENESS_HOURS` (2 h), fire-and-forget `reconcileStorageListCache()` in the background, then return cached data immediately (stale-while-revalidate).
+3. **Cache miss**: run full fetch — `adapter.list("")` + parallel `.meta.json` reads + DB fallbacks — write result to `StorageListCache`, then return.
+
+TypeFilter (`BACKUP` / `SYSTEM`) is applied **after** cache retrieval, so the cache always stores the full unfiltered list.
+
+## Write Path
+
+After a full fetch (cache miss), the result is persisted with a non-blocking `upsert`:
+
+```typescript
+prisma.storageListCache.upsert({
+    where:  { adapterConfigId },
+    create: { adapterConfigId, filesJson },
+    update: { filesJson, cachedAt: new Date() },
+}).catch(() => {});
+```
+
+## Surgical Update Methods
+
+Instead of dropping the entire cache row on every change, these methods patch only the affected entry:
+
+| Method | When to use |
+|--------|-------------|
+| `appendStorageListCacheEntry(id, entry)` | After a successful backup upload |
+| `removeStorageListCacheEntry(id, filePath)` | After a file is deleted (manual or retention) |
+| `updateStorageListCacheEntry(id, filePath, updates)` | After lock toggle or verification result written |
+
+All three follow the same read-modify-write pattern against the JSON array. If no cache row exists they no-op — the next `listFilesWithMetadata` call does a fresh fetch and populates the cache.
+
+**Adding a new surgical update point:**
+
+```typescript
+import("@/services/storage/storage-service").then(({ storageService }) => {
+    storageService.removeStorageListCacheEntry(configId, filePath).catch(() => {});
+});
+```
+
+Use a dynamic import with fire-and-forget to avoid circular dependencies and to keep the calling code non-blocking.
+
+## Reconciliation (Stale-While-Revalidate)
+
+Files deleted directly on the remote storage (outside DBackup) are invisible to the surgical update methods. The reconciliation job detects these:
+
+1. Call `adapter.list("")` — returns only file names and paths, no `.meta.json` reads.
+2. Diff remote paths against cached paths.
+3. **Removed files**: filter them out of the cache.
+4. **New files** (added outside DBackup or missed during a previous run): fetch their `.meta.json` sidecars and enrich only those files using `enrichSingleFile()`.
+5. Write the updated array back and reset `cachedAt`.
+
+Reconciliation runs in the background (non-blocking) whenever a cached listing is served and its `cachedAt` is older than `CACHE_STALENESS_HOURS`. The threshold is defined at the top of `storage-service.ts`:
+
+```typescript
+const CACHE_STALENESS_HOURS = 2;
+```
+
+## Pre-warm / Reconcile System Task
+
+The `system.warmup_storage_cache` task keeps the cache consistent for all storage adapters.
+
+- **Startup delay**: 10 seconds (standard for all startup tasks, controlled by the scheduler).
+- **Recurring schedule**: Every hour.
+- **Enabled by default**: yes.
+- **Concurrency**: adapters are processed sequentially to avoid simultaneous rate-limit hits.
+
+**Per-adapter logic:**
+- **Cache exists**: calls `reconcileStorageListCache()` — runs `adapter.list()`, diffs against the cached list, removes entries for files deleted externally, enriches and appends new files. Detects changes made outside DBackup within the hour.
+- **No cache row**: calls `listFilesWithMetadata()` — full fetch to populate the cache from scratch.
+
+## Force Refresh
+
+Pass `?refresh=true` on the files API route to bypass the cache and force a full re-fetch:
+
+```
+GET /api/storage/:id/files?refresh=true
+```
+
+This is wired to the Refresh button in the Storage Explorer UI. After the live fetch completes, the new result is written back to the cache.
+
+## Cache Invalidation Summary
+
+| Trigger | Method | Location |
+|---------|--------|----------|
+| Backup uploaded | `appendStorageListCacheEntry` | `src/lib/runner/steps/03-upload.ts` |
+| Retention deleted a file | `removeStorageListCacheEntry` | `src/lib/runner/steps/05-retention.ts` |
+| Manual file delete | `removeStorageListCacheEntry` | `StorageService.deleteFile()` |
+| File lock toggled | `updateStorageListCacheEntry` | `StorageService.toggleLock()` |
+| Verification result written | `updateStorageListCacheEntry` | `VerificationService.writeVerificationResult()` |
+| Cache older than 2 h | `reconcileStorageListCache()` background | `StorageService.listFilesWithMetadata()` |
+| User clicks Refresh | `invalidateStorageListCache()` + full fetch | `GET /api/storage/:id/files?refresh=true` |
+
+## Key Files
+
+| File | Role |
+|------|------|
+| `src/services/storage/storage-service.ts` | All cache methods, reconciliation, enrichment |
+| `src/services/storage/verification-service.ts` | Surgical update after verification |
+| `src/lib/runner/steps/03-upload.ts` | Append on upload |
+| `src/lib/runner/steps/05-retention.ts` | Remove per deleted file |
+| `src/services/system/system-task-service.ts` | Pre-warm task definition and runner |
+| `prisma/schema.prisma` | `StorageListCache` model |

docs/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dbackup-docs",
-  "version": "2.6.0",
+  "version": "2.7.0",
   "private": true,
   "scripts": {
     "dev": "vitepress dev",