diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 73ff02db..5427e37f 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -95,26 +95,31 @@ return { success: boolean, message?: string, data?: any, error?: string } ## Developer Workflows ```bash -pnpm dev # Start dev server (http://localhost:3000) +pnpm dev # Start dev server - auto-applies pending migrations on startup pnpm test # Unit tests (vitest) pnpm test:integration # Integration tests against real DB containers pnpm test:ui # Spin up 16 test DBs + seed local DB for manual testing pnpm run build # Production build (validate before commit) -npx prisma migrate dev # Create DB migration +npx prisma migrate dev # Create a new DB migration (stop dev server first) +pnpm run database:reset # Reset dev DB from scratch (drops + recreates via all migrations) ``` **Test Infrastructure**: See [docker-compose.test.yml](docker-compose.test.yml) for MySQL/PG/Mongo containers. ### Prisma Migrations - IMPORTANT +`pnpm dev` automatically runs `prisma migrate deploy` on startup, so the local DB is always up to date with all pending migrations. No manual step needed after pulling changes that include new migrations. + **Never run `prisma migrate dev` while `pnpm dev` is running.** The dev server holds an open SQLite connection. `migrate dev` can trigger an interactive DB reset (on drift), which conflicts with the file lock and crashes the Node process - and often VS Code with it. +**Never use `prisma db push`.** It applies schema changes without creating a migration file, causing the local `_prisma_migrations` table to diverge from the actual schema. This breaks `database:deploy` in production and for other developers. Always create a proper migration. + **Safe workflow for schema changes:** 1. Stop the dev server first (Ctrl+C in the node terminal) 2. Run `npx prisma migrate dev --name ` -3. Restart `pnpm dev` +3. Restart `pnpm dev` - migrations apply automatically on startup -**Alternative for local dev only:** `npx prisma db push` - applies schema changes without migration history, safe to run alongside the dev server, and never prompts for a reset. +**Reset dev DB from scratch:** `pnpm run database:reset` (runs `prisma migrate reset` - drops and recreates from all migrations). ## Queue System (`src/lib/execution/queue-manager.ts`) diff --git a/api-docs/openapi.yaml b/api-docs/openapi.yaml index 3ae76fd9..df88ef14 100644 --- a/api-docs/openapi.yaml +++ b/api-docs/openapi.yaml @@ -1,7 +1,7 @@ openapi: 3.1.0 info: title: DBackup API - version: 2.6.0 + version: 2.7.0 description: | REST API for DBackup - a self-hosted database backup automation platform with encryption, compression, and smart retention. diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index 0e444571..19ac9f28 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -171,6 +171,7 @@ export default defineConfig({ collapsed: false, items: [ { text: 'Storage Explorer', link: '/user-guide/features/storage-explorer' }, + { text: 'Backup Verification', link: '/user-guide/features/backup-verification' }, { text: 'Restore', link: '/user-guide/features/restore' }, { text: 'Notifications', link: '/user-guide/features/notifications' }, { text: 'System Backup', link: '/user-guide/features/system-backup' }, @@ -212,7 +213,8 @@ export default defineConfig({ { text: 'Icon System', link: '/developer-guide/core/icons' }, { text: 'Download Tokens', link: '/developer-guide/core/download-tokens' }, { text: 'Rate Limiting', link: '/developer-guide/core/rate-limiting' }, - { text: 'Update Service', link: '/developer-guide/core/updates' } + { text: 'Update Service', link: '/developer-guide/core/updates' }, + { text: 'Storage List Cache', link: '/developer-guide/core/storage-cache' } ] }, { diff --git a/docs/changelog.md b/docs/changelog.md index 3cdf097d..c7fb9ca3 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -2,6 +2,48 @@ All notable changes to DBackup are documented here. +## v2.7.0 - Backup Integrity Verification, Storage Explorer Caching, and Multiple Improvements +*Released: June 14, 2026* + +### โœจ Features + +- **integrity**: Added backup file integrity verification. The Storage Explorer shows SHA-256/MD5 checksums and last verification result per file, with a Verify Now button that runs as a tracked async execution with live progress and cancel support. +- **integrity**: Native checksum verification for S3/R2/Hetzner (SHA-256 via object metadata), Google Drive (MD5 via API), OneDrive (SHA-256 via Graph API), and local filesystem - no download required. Both checksums are stored in `.meta.json` at upload time. +- **integrity**: Post-upload verification is now opt-in for all storage destinations via system settings (local filesystem always verifies). +- **integrity**: Scheduled integrity checks support two scan modes - **Jobs** (default, only verifies files linked to backup jobs) and **All Files** (full storage scan) - plus configurable filters (skip already-passed, max age, max file size). Jobs and storage destinations can individually opt out via a "Skip Verification" toggle. +- **storage**: Storage Explorer file listings are now cached in SQLite for instant repeat visits. Cache is invalidated on backup create, delete, verify, and lock changes. +- **history**: Execution logs can now be copied to the clipboard or downloaded as a `.log` file directly from the log dialog. Sensitive data (IPs, credentials, connection strings) is automatically redacted before export. + +### ๐Ÿ› Bug Fixes + +- **storage**: Google Drive, OneDrive, and Dropbox now live-validate their stored OAuth token when the Connection tab is opened, showing an expiry warning with a Re-authorize button instead of a misleading green "authorized" status. + +### ๐ŸŽจ Improvements + +- **dev**: `pnpm dev` now automatically applies pending Prisma migrations and regenerates the Prisma client on startup. +- **storage**: Storage Explorer cache uses surgical updates - create, delete, lock, and verify each patch only the affected cache entry instead of invalidating the full cache. +- **storage**: A new "Pre-warm Storage Cache" system task (enabled by default, hourly) reconciles caches against remote storage and pre-populates the cache for adapters not yet visited. +- **storage**: Stale cache entries trigger a background reconciliation via `adapter.list()` to detect files deleted outside DBackup without re-reading sidecars. +- **storage**: Long backup names in the Storage Explorer are truncated with a tooltip showing the full name on hover. +- **integrity**: Manual system task runs auto-redirect to the live execution history when "Auto-redirect on job start" is enabled. +- **ui**: All data tables (Storage, Jobs, Sources, Destinations, Notifications) now support horizontal scrolling when columns overflow. +- **explorer**: Database version history now detects and visually distinguishes downgrades - the change log shows an orange downward arrow for downgrades vs. green upward arrow for upgrades, and downgrade notifications are sent with a distinct "Downgrade" label and warning color. +- **explorer**: Database Explorer table list now supports searching by name and sorting by Name, Type, Rows, or Size via clickable column headers. + +### ๐Ÿงช Tests + +- **tests**: Updated unit tests for integrity service, upload step, storage service, and system task service to match the refactored verification interfaces. +- **tests**: Fixed integrity service tests to use destinations scan mode explicitly and added `skipVerification` field to job service create test. +- **tests**: Added unit tests for `VerificationService` (all verification code paths), `SystemTaskRunner` (execution lifecycle), `logs/sanitize` (credential and IP redaction), and `logs/format` (log text export). Extended `IntegrityService` tests to cover the jobs-mode scan path. + +### ๐Ÿณ Docker + +- **Image**: `skyfay/dbackup:v2.7.0` +- **Also tagged as**: `latest`, `v2` +- **CI Image**: `skyfay/dbackup:ci` +- **Platforms**: linux/amd64, linux/arm64 + + ## v2.6.0 - Security Update, Vault Credential Profiles, OAuth Improvements, and Multiple Bug Fixes *Released: June 6, 2026* diff --git a/docs/developer-guide/core/storage-cache.md b/docs/developer-guide/core/storage-cache.md new file mode 100644 index 00000000..eedfb011 --- /dev/null +++ b/docs/developer-guide/core/storage-cache.md @@ -0,0 +1,122 @@ +# Storage List Cache + +The Storage Explorer calls `adapter.list("")` (recursive folder traversal) plus one `adapter.read()` per `.meta.json` sidecar on every load. For remote adapters like Google Drive this means dozens of API calls per page view. The storage list cache stores the full enriched listing in SQLite so repeat visits are instant. + +## Database Model + +```prisma +model StorageListCache { + adapterConfigId String @id + filesJson String // JSON array of RichFileInfo โ€” full list, no typeFilter + cachedAt DateTime @default(now()) + adapterConfig AdapterConfig @relation(..., onDelete: Cascade) +} +``` + +One row per storage adapter. `cachedAt` drives the staleness check. + +## Read Path + +`StorageService.listFilesWithMetadata(adapterConfigId, typeFilter?, bypassCache?)`: + +1. If `bypassCache = false` (default): query `StorageListCache` by `adapterConfigId`. +2. **Cache hit**: check age. If `cachedAt` is older than `CACHE_STALENESS_HOURS` (2 h), fire-and-forget `reconcileStorageListCache()` in the background, then return cached data immediately (stale-while-revalidate). +3. **Cache miss**: run full fetch โ€” `adapter.list("")` + parallel `.meta.json` reads + DB fallbacks โ€” write result to `StorageListCache`, then return. + +TypeFilter (`BACKUP` / `SYSTEM`) is applied **after** cache retrieval, so the cache always stores the full unfiltered list. + +## Write Path + +After a full fetch (cache miss), the result is persisted with a non-blocking `upsert`: + +```typescript +prisma.storageListCache.upsert({ + where: { adapterConfigId }, + create: { adapterConfigId, filesJson }, + update: { filesJson, cachedAt: new Date() }, +}).catch(() => {}); +``` + +## Surgical Update Methods + +Instead of dropping the entire cache row on every change, these methods patch only the affected entry: + +| Method | When to use | +|--------|-------------| +| `appendStorageListCacheEntry(id, entry)` | After a successful backup upload | +| `removeStorageListCacheEntry(id, filePath)` | After a file is deleted (manual or retention) | +| `updateStorageListCacheEntry(id, filePath, updates)` | After lock toggle or verification result written | + +All three follow the same read-modify-write pattern against the JSON array. If no cache row exists they no-op โ€” the next `listFilesWithMetadata` call does a fresh fetch and populates the cache. + +**Adding a new surgical update point:** + +```typescript +import("@/services/storage/storage-service").then(({ storageService }) => { + storageService.removeStorageListCacheEntry(configId, filePath).catch(() => {}); +}); +``` + +Use a dynamic import with fire-and-forget to avoid circular dependencies and to keep the calling code non-blocking. + +## Reconciliation (Stale-While-Revalidate) + +Files deleted directly on the remote storage (outside DBackup) are invisible to the surgical update methods. The reconciliation job detects these: + +1. Call `adapter.list("")` โ€” returns only file names and paths, no `.meta.json` reads. +2. Diff remote paths against cached paths. +3. **Removed files**: filter them out of the cache. +4. **New files** (added outside DBackup or missed during a previous run): fetch their `.meta.json` sidecars and enrich only those files using `enrichSingleFile()`. +5. Write the updated array back and reset `cachedAt`. + +Reconciliation runs in the background (non-blocking) whenever a cached listing is served and its `cachedAt` is older than `CACHE_STALENESS_HOURS`. The threshold is defined at the top of `storage-service.ts`: + +```typescript +const CACHE_STALENESS_HOURS = 2; +``` + +## Pre-warm / Reconcile System Task + +The `system.warmup_storage_cache` task keeps the cache consistent for all storage adapters. + +- **Startup delay**: 10 seconds (standard for all startup tasks, controlled by the scheduler). +- **Recurring schedule**: Every hour. +- **Enabled by default**: yes. +- **Concurrency**: adapters are processed sequentially to avoid simultaneous rate-limit hits. + +**Per-adapter logic:** +- **Cache exists**: calls `reconcileStorageListCache()` โ€” runs `adapter.list()`, diffs against the cached list, removes entries for files deleted externally, enriches and appends new files. Detects changes made outside DBackup within the hour. +- **No cache row**: calls `listFilesWithMetadata()` โ€” full fetch to populate the cache from scratch. + +## Force Refresh + +Pass `?refresh=true` on the files API route to bypass the cache and force a full re-fetch: + +``` +GET /api/storage/:id/files?refresh=true +``` + +This is wired to the Refresh button in the Storage Explorer UI. After the live fetch completes, the new result is written back to the cache. + +## Cache Invalidation Summary + +| Trigger | Method | Location | +|---------|--------|----------| +| Backup uploaded | `appendStorageListCacheEntry` | `src/lib/runner/steps/03-upload.ts` | +| Retention deleted a file | `removeStorageListCacheEntry` | `src/lib/runner/steps/05-retention.ts` | +| Manual file delete | `removeStorageListCacheEntry` | `StorageService.deleteFile()` | +| File lock toggled | `updateStorageListCacheEntry` | `StorageService.toggleLock()` | +| Verification result written | `updateStorageListCacheEntry` | `VerificationService.writeVerificationResult()` | +| Cache older than 2 h | `reconcileStorageListCache()` background | `StorageService.listFilesWithMetadata()` | +| User clicks Refresh | `invalidateStorageListCache()` + full fetch | `GET /api/storage/:id/files?refresh=true` | + +## Key Files + +| File | Role | +|------|------| +| `src/services/storage/storage-service.ts` | All cache methods, reconciliation, enrichment | +| `src/services/storage/verification-service.ts` | Surgical update after verification | +| `src/lib/runner/steps/03-upload.ts` | Append on upload | +| `src/lib/runner/steps/05-retention.ts` | Remove per deleted file | +| `src/services/system/system-task-service.ts` | Pre-warm task definition and runner | +| `prisma/schema.prisma` | `StorageListCache` model | diff --git a/docs/package.json b/docs/package.json index d2f3906c..a5d4179e 100644 --- a/docs/package.json +++ b/docs/package.json @@ -1,6 +1,6 @@ { "name": "dbackup-docs", - "version": "2.6.0", + "version": "2.7.0", "private": true, "scripts": { "dev": "vitepress dev", diff --git a/docs/user-guide/features/backup-verification.md b/docs/user-guide/features/backup-verification.md new file mode 100644 index 00000000..ea033c9b --- /dev/null +++ b/docs/user-guide/features/backup-verification.md @@ -0,0 +1,120 @@ +# Backup Verification + +Detect corrupted or tampered backup files before you need to restore them. + +## Why Verification Matters + +A backup that silently corrupts mid-upload is worse than no backup at all - you only discover the problem at the worst possible moment, during a restore. Verification catches this early by comparing a cryptographic checksum of the backup file against the original value recorded immediately after the dump. + +## How It Works + +### Checksum Generation (at upload time) + +Every backup file gets a SHA-256 checksum computed from the final processed file - after compression and encryption if enabled. For adapters that natively support MD5 (Google Drive), an MD5 checksum is computed in the same pass at no extra cost. + +Both values are stored in the `.meta.json` sidecar file alongside the backup: + +```json +{ + "checksum": "a3f1b2c4...", + "checksumMd5": "d8e9f0a1...", + "verification": { + "verifiedAt": "2026-06-10T14:32:00Z", + "passed": true, + "trigger": "post-upload" + } +} +``` + +### Verification Logic + +When a verification is triggered, DBackup uses the best available method for each storage adapter: + +| Adapter | Method | No Download Needed | +| :--- | :--- | :--- | +| **Local Filesystem** | Direct file hash via stream | Yes | +| **S3 / R2 / Hetzner** | `HeadObject` - reads SHA-256 from object metadata | Yes | +| **Google Drive** | `files.get` API - reads native MD5 field | Yes | +| **OneDrive** | Graph API - reads native SHA-256 hash | Yes | +| SFTP / FTP | Download + compute SHA-256 | No | +| SMB | Download + compute SHA-256 | No | +| WebDAV | Download + compute SHA-256 | No | +| Dropbox | Download + compute SHA-256 | No | +| Rsync | Download + compute SHA-256 | No | + +For adapters without native checksum APIs, DBackup downloads the full file, recomputes the hash, and compares it against the stored value. + +The result is written back into the `.meta.json` sidecar so it persists across sessions and appears immediately in the Storage Explorer without re-verifying. + +## Triggering a Verification + +### Manual Verification (on-demand) + +In the **Storage Explorer**, each backup row has a shield icon in the Actions column: + +- **Gray shield**: Never verified +- **Green shield**: Last check passed +- **Red shield**: Last check failed + +Click the icon to trigger a verification. A loading toast appears while the check runs, then the result is shown and the badge in the table updates automatically. + +::: tip Re-verify anytime +You can re-verify a backup at any time - clicking the green shield on an already-verified backup runs a fresh check. +::: + +The **Integrity** column shows the same status at a glance without opening the actions menu. + +### Post-Upload Verification (automatic) + +DBackup can verify each backup immediately after it finishes uploading. This is controlled by the `backup.postUploadVerify` system setting. + +**Local filesystem destinations always verify**, regardless of this setting - the check is a direct file read with near-zero overhead. + +For remote destinations (S3, Google Drive, SFTP, etc.), post-upload verification is **opt-in** and off by default. Enable it in **Settings - System** if you want automatic verification for all destinations. + +::: warning Bandwidth for download-based adapters +For SFTP, FTP, SMB, WebDAV, Dropbox, and Rsync, automatic post-upload verification downloads the full backup file a second time to recompute the hash. For large backups on slow or metered connections this adds significant time and transfer costs. For S3, Google Drive, OneDrive, and local storage, verification has near-zero overhead and is safe to enable freely. +::: + +### Scheduled Integrity Check + +DBackup includes a **Scheduled Integrity Check** job that periodically verifies all backups across all storage destinations. It runs through each destination, reads the `.meta.json` sidecar for each backup file, and runs the same native-first verification logic as the manual check. + +Results are written back to the sidecars as they complete, so the Storage Explorer badges stay up to date without any manual action. + +The scheduler can be configured under **Settings - Scheduler** (cron expression). A weekly or monthly check on your full archive is a reasonable default for most setups. + +::: info What counts as "scheduled" +The `trigger` field in `.meta.json` records how each check was triggered: `manual`, `post-upload`, or `scheduled`. This lets you distinguish between a fresh post-upload check and an older scheduled check in the metadata. +::: + +## Interpreting Results + +| Status | Meaning | +| :--- | :--- | +| **Verified (green)** | File matches its checksum - no corruption or tampering detected | +| **Failed (red)** | Hash mismatch - file may be corrupted or was modified after upload | +| **No checksum** | Backup predates checksum support - no baseline to compare against | +| **No metadata** | File has no `.meta.json` sidecar (manually placed file) | +| **-** (dash) | Never been verified | + +A **Failed** result does not necessarily mean the backup is unrestorable - it means something changed since the original upload. Possible causes: + +- Storage provider-level corruption (rare but documented on HDD-backed storage) +- Partial re-upload or file replacement +- Ransomware or accidental modification +- Network error during original upload that was not caught + +If you see a failed check on a backup you intend to use for a restore, treat it as suspect and try an older backup if available. + +## Limitations + +- **Encrypted backups**: The checksum covers the encrypted file, not the plaintext. A passing integrity check confirms the encrypted bytes are intact, but does not guarantee the decryption key is still available. +- **Legacy backups**: Files uploaded before checksum support was added have no baseline. Verification will report "No checksum" and no comparison is possible. +- **Native SHA-256 on S3**: The SHA-256 is stored as S3 custom metadata (`dbackup-sha256`) during upload. If the object was copied or re-uploaded via external tools without preserving this metadata, native verification will fall back gracefully and report "unsupported" - which triggers a download-based check instead. + +## Next Steps + +- [Storage Explorer](/user-guide/features/storage-explorer) - Browse and manage backup files +- [Restore](/user-guide/features/restore) - Restore a backup to a database +- [Encryption](/user-guide/security/encryption) - Encrypt backup files at rest diff --git a/package.json b/package.json index 5d1b5c49..cff016dc 100644 --- a/package.json +++ b/package.json @@ -1,9 +1,9 @@ { "name": "dbackup", - "version": "2.6.0", + "version": "2.7.0", "private": true, "scripts": { - "dev": "next dev", + "dev": "prisma migrate deploy && prisma generate && next dev", "build": "next build", "start": "next start", "lint": "eslint", @@ -24,6 +24,7 @@ "version:bump": "./scripts/sync-version.sh", "changelog:next": "./scripts/sync-version.sh --changelog-init", "database:deploy": "prisma migrate deploy && prisma generate", + "database:reset": "prisma migrate reset", "docs:install": "cd docs && pnpm install", "docs:dev": "cd docs && pnpm dev", "docs:build": "cd docs && pnpm build", diff --git a/prisma/migrations/20260611000001_add_storage_list_cache/migration.sql b/prisma/migrations/20260611000001_add_storage_list_cache/migration.sql new file mode 100644 index 00000000..5ba5d810 --- /dev/null +++ b/prisma/migrations/20260611000001_add_storage_list_cache/migration.sql @@ -0,0 +1,7 @@ +-- CreateTable +CREATE TABLE "StorageListCache" ( + "adapterConfigId" TEXT NOT NULL PRIMARY KEY, + "filesJson" TEXT NOT NULL, + "cachedAt" DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT "StorageListCache_adapterConfigId_fkey" FOREIGN KEY ("adapterConfigId") REFERENCES "AdapterConfig" ("id") ON DELETE CASCADE ON UPDATE CASCADE +); diff --git a/prisma/migrations/20260612072631_add_job_skip_verification/migration.sql b/prisma/migrations/20260612072631_add_job_skip_verification/migration.sql new file mode 100644 index 00000000..22cefb06 --- /dev/null +++ b/prisma/migrations/20260612072631_add_job_skip_verification/migration.sql @@ -0,0 +1,2 @@ +-- AlterTable +ALTER TABLE "Job" ADD COLUMN "skipVerification" BOOLEAN NOT NULL DEFAULT false; diff --git a/prisma/schema.prisma b/prisma/schema.prisma index a3ae6a24..365fcf1c 100644 --- a/prisma/schema.prisma +++ b/prisma/schema.prisma @@ -38,6 +38,7 @@ model AdapterConfig { healthLogs HealthCheckLog[] versionHistory DbVersionHistory[] + storageListCache StorageListCache? } model CredentialProfile { @@ -120,6 +121,7 @@ model Job { source AdapterConfig @relation("Source", fields: [sourceId], references: [id]) notifications AdapterConfig[] @relation("Notifications") notificationEvents String @default("ALWAYS") // "ALWAYS", "FAILURE_ONLY", "SUCCESS_ONLY" + skipVerification Boolean @default(false) } model JobDestination { @@ -397,3 +399,10 @@ model NotificationLog { @@index([sentAt]) @@index([executionId]) } + +model StorageListCache { + adapterConfigId String @id + filesJson String + cachedAt DateTime @default(now()) + adapterConfig AdapterConfig @relation(fields: [adapterConfigId], references: [id], onDelete: Cascade) +} diff --git a/public/openapi.yaml b/public/openapi.yaml index ced1abba..36cfc1d1 100644 --- a/public/openapi.yaml +++ b/public/openapi.yaml @@ -1,7 +1,7 @@ openapi: 3.1.0 info: title: DBackup API - version: 2.6.0 + version: 2.7.0 description: | REST API for DBackup - a self-hosted database backup automation platform with encryption, compression, and smart retention. diff --git a/src/app/actions/settings/integrity-settings.ts b/src/app/actions/settings/integrity-settings.ts new file mode 100644 index 00000000..32c361bc --- /dev/null +++ b/src/app/actions/settings/integrity-settings.ts @@ -0,0 +1,61 @@ +"use server" + +import prisma from "@/lib/prisma"; +import { revalidatePath } from "next/cache"; +import { z } from "zod"; +import { checkPermission } from "@/lib/auth/access-control"; +import { PERMISSIONS } from "@/lib/auth/permissions"; +import { logger } from "@/lib/logging/logger"; +import { wrapError } from "@/lib/logging/errors"; + +const log = logger.child({ action: "integrity-settings" }); + +const schema = z.object({ + skipPassed: z.boolean(), + maxAgeDays: z.coerce.number().min(0).max(3650), + maxFileSizeMb: z.coerce.number().min(0).max(1_000_000), + scanMode: z.enum(["jobs", "destinations"]).default("jobs"), +}); + +export async function saveIntegritySettings(data: z.infer) { + await checkPermission(PERMISSIONS.SETTINGS.WRITE); + + const result = schema.safeParse(data); + if (!result.success) { + return { success: false, error: result.error.issues[0].message }; + } + + try { + const { skipPassed, maxAgeDays, maxFileSizeMb, scanMode } = result.data; + + await prisma.systemSetting.upsert({ + where: { key: "integrity.skipPassed" }, + update: { value: String(skipPassed) }, + create: { key: "integrity.skipPassed", value: String(skipPassed) }, + }); + + await prisma.systemSetting.upsert({ + where: { key: "integrity.maxAgeDays" }, + update: { value: String(maxAgeDays) }, + create: { key: "integrity.maxAgeDays", value: String(maxAgeDays) }, + }); + + await prisma.systemSetting.upsert({ + where: { key: "integrity.maxFileSizeMb" }, + update: { value: String(maxFileSizeMb) }, + create: { key: "integrity.maxFileSizeMb", value: String(maxFileSizeMb) }, + }); + + await prisma.systemSetting.upsert({ + where: { key: "integrity.scanMode" }, + update: { value: scanMode }, + create: { key: "integrity.scanMode", value: scanMode }, + }); + + revalidatePath("/dashboard/settings"); + return { success: true }; + } catch (e: unknown) { + log.error("Failed to save integrity settings", {}, wrapError(e)); + return { success: false, error: "Failed to save settings" }; + } +} diff --git a/src/app/api/adapters/dropbox/validate-token/route.ts b/src/app/api/adapters/dropbox/validate-token/route.ts new file mode 100644 index 00000000..d1553060 --- /dev/null +++ b/src/app/api/adapters/dropbox/validate-token/route.ts @@ -0,0 +1,58 @@ +import { NextRequest, NextResponse } from "next/server"; +import { headers } from "next/headers"; +import { getAuthContext } from "@/lib/auth/access-control"; +import { getDecryptedCredentialData } from "@/services/auth/credential-service"; +import type { OAuthData } from "@/lib/core/credentials"; +import { Dropbox } from "dropbox"; +import { logger } from "@/lib/logging/logger"; + +const log = logger.child({ route: "adapters/dropbox/validate-token" }); + +/** + * POST /api/adapters/dropbox/validate-token + * Validates whether the stored refresh token for an OAuth credential profile is still accepted by Dropbox. + * Body: { credentialId: string } + * Response: { valid: boolean, message?: string } + */ +export async function POST(req: NextRequest) { + const ctx = await getAuthContext(await headers()); + if (!ctx) { + return NextResponse.json({ error: "Unauthorized" }, { status: 401 }); + } + + try { + const { credentialId } = await req.json(); + if (!credentialId) { + return NextResponse.json({ valid: false, message: "Missing credentialId" }, { status: 400 }); + } + + const profile = (await getDecryptedCredentialData(credentialId, "OAUTH")) as OAuthData; + + if (!profile.clientId || !profile.clientSecret || !profile.refreshToken) { + return NextResponse.json({ valid: false, message: "Incomplete credentials" }); + } + + const dbx = new Dropbox({ + clientId: profile.clientId, + clientSecret: profile.clientSecret, + refreshToken: profile.refreshToken, + }); + + await dbx.usersGetCurrentAccount(); + + return NextResponse.json({ valid: true }); + } catch (error: unknown) { + const message = error instanceof Error ? error.message : String(error); + const isExpired = + message.includes("invalid_access_token") || + message.includes("expired_access_token") || + message.includes("invalid_grant"); + log.warn("Dropbox token validation failed", { message }); + return NextResponse.json({ + valid: false, + message: isExpired + ? "Authorization expired. Please re-authorize with Dropbox." + : message, + }); + } +} diff --git a/src/app/api/adapters/google-drive/validate-token/route.ts b/src/app/api/adapters/google-drive/validate-token/route.ts new file mode 100644 index 00000000..33b4767d --- /dev/null +++ b/src/app/api/adapters/google-drive/validate-token/route.ts @@ -0,0 +1,57 @@ +import { NextRequest, NextResponse } from "next/server"; +import { google } from "googleapis"; +import { headers } from "next/headers"; +import { getAuthContext } from "@/lib/auth/access-control"; +import { getDecryptedCredentialData } from "@/services/auth/credential-service"; +import type { OAuthData } from "@/lib/core/credentials"; +import { logger } from "@/lib/logging/logger"; + +const log = logger.child({ route: "adapters/google-drive/validate-token" }); + +/** + * POST /api/adapters/google-drive/validate-token + * Validates whether the stored refresh token for an OAuth credential profile is still accepted by Google. + * Body: { credentialId: string } + * Response: { valid: boolean, message?: string } + */ +export async function POST(req: NextRequest) { + const ctx = await getAuthContext(await headers()); + if (!ctx) { + return NextResponse.json({ error: "Unauthorized" }, { status: 401 }); + } + + try { + const { credentialId } = await req.json(); + if (!credentialId) { + return NextResponse.json({ valid: false, message: "Missing credentialId" }, { status: 400 }); + } + + const profile = (await getDecryptedCredentialData(credentialId, "OAUTH")) as OAuthData; + + if (!profile.clientId || !profile.clientSecret || !profile.refreshToken) { + return NextResponse.json({ valid: false, message: "Incomplete credentials" }); + } + + const origin = process.env.BETTER_AUTH_URL || req.nextUrl.origin; + const redirectUri = `${origin}/api/adapters/google-drive/callback`; + + const oauth2Client = new google.auth.OAuth2( + profile.clientId, + profile.clientSecret, + redirectUri + ); + oauth2Client.setCredentials({ refresh_token: profile.refreshToken }); + + await oauth2Client.getAccessToken(); + + return NextResponse.json({ valid: true }); + } catch (error: unknown) { + const message = error instanceof Error ? error.message : String(error); + const isExpired = message.includes("invalid_grant") || message.includes("Token has been expired"); + log.warn("Google Drive token validation failed", { message }); + return NextResponse.json({ + valid: false, + message: isExpired ? "Authorization expired. Please re-authorize with Google." : message, + }); + } +} diff --git a/src/app/api/adapters/onedrive/validate-token/route.ts b/src/app/api/adapters/onedrive/validate-token/route.ts new file mode 100644 index 00000000..843eaaf3 --- /dev/null +++ b/src/app/api/adapters/onedrive/validate-token/route.ts @@ -0,0 +1,67 @@ +import { NextRequest, NextResponse } from "next/server"; +import { headers } from "next/headers"; +import { getAuthContext } from "@/lib/auth/access-control"; +import { getDecryptedCredentialData } from "@/services/auth/credential-service"; +import type { OAuthData } from "@/lib/core/credentials"; +import { logger } from "@/lib/logging/logger"; + +const log = logger.child({ route: "adapters/onedrive/validate-token" }); +const TOKEN_URL = "https://login.microsoftonline.com/common/oauth2/v2.0/token"; + +/** + * POST /api/adapters/onedrive/validate-token + * Validates whether the stored refresh token for an OAuth credential profile is still accepted by Microsoft. + * Body: { credentialId: string } + * Response: { valid: boolean, message?: string } + */ +export async function POST(req: NextRequest) { + const ctx = await getAuthContext(await headers()); + if (!ctx) { + return NextResponse.json({ error: "Unauthorized" }, { status: 401 }); + } + + try { + const { credentialId } = await req.json(); + if (!credentialId) { + return NextResponse.json({ valid: false, message: "Missing credentialId" }, { status: 400 }); + } + + const profile = (await getDecryptedCredentialData(credentialId, "OAUTH")) as OAuthData; + + if (!profile.clientId || !profile.clientSecret || !profile.refreshToken) { + return NextResponse.json({ valid: false, message: "Incomplete credentials" }); + } + + const body = new URLSearchParams({ + client_id: profile.clientId, + client_secret: profile.clientSecret, + refresh_token: profile.refreshToken, + grant_type: "refresh_token", + scope: "Files.ReadWrite.All offline_access", + }); + + const res = await fetch(TOKEN_URL, { + method: "POST", + headers: { "Content-Type": "application/x-www-form-urlencoded" }, + body, + }); + + if (!res.ok) { + const data = await res.json().catch(() => ({})); + const isExpired = data.error === "invalid_grant" || data.error === "interaction_required"; + log.warn("OneDrive token validation failed", { error: data.error }); + return NextResponse.json({ + valid: false, + message: isExpired + ? "Authorization expired. Please re-authorize with Microsoft." + : `Token refresh failed: ${data.error_description ?? res.statusText}`, + }); + } + + return NextResponse.json({ valid: true }); + } catch (error: unknown) { + const message = error instanceof Error ? error.message : String(error); + log.warn("OneDrive token validation error", { message }); + return NextResponse.json({ valid: false, message }); + } +} diff --git a/src/app/api/jobs/[id]/route.ts b/src/app/api/jobs/[id]/route.ts index 321bf1ef..61653ad5 100644 --- a/src/app/api/jobs/[id]/route.ts +++ b/src/app/api/jobs/[id]/route.ts @@ -38,7 +38,7 @@ export async function PUT( const params = await props.params; try { const body = await req.json(); - const { name, schedule, sourceId, databases, destinations, notificationIds, enabled, encryptionProfileId, compression, pgCompression, notificationEvents, namingTemplateId, schedulePresetId } = body; + const { name, schedule, sourceId, databases, destinations, notificationIds, enabled, encryptionProfileId, compression, pgCompression, notificationEvents, namingTemplateId, schedulePresetId, skipVerification } = body; const updatedJob = await jobService.updateJob(params.id, { name, @@ -59,6 +59,7 @@ export async function PUT( notificationEvents, namingTemplateId: namingTemplateId !== undefined ? (namingTemplateId ?? null) : undefined, schedulePresetId: schedulePresetId !== undefined ? (schedulePresetId ?? null) : undefined, + skipVerification: skipVerification !== undefined ? skipVerification : undefined, }); return NextResponse.json(updatedJob); diff --git a/src/app/api/jobs/route.ts b/src/app/api/jobs/route.ts index 924c88fd..752cfd3b 100644 --- a/src/app/api/jobs/route.ts +++ b/src/app/api/jobs/route.ts @@ -59,7 +59,7 @@ export async function POST(req: NextRequest) { checkPermissionWithContext(ctx, PERMISSIONS.JOBS.WRITE); const body = await req.json(); - const { name, schedule, sourceId, databases, destinations, notificationIds, enabled, encryptionProfileId, compression, pgCompression, notificationEvents, namingTemplateId, schedulePresetId } = body; + const { name, schedule, sourceId, databases, destinations, notificationIds, enabled, encryptionProfileId, compression, pgCompression, notificationEvents, namingTemplateId, schedulePresetId, skipVerification } = body; if (!name || !schedule || !sourceId || !destinations || !Array.isArray(destinations) || destinations.length === 0) { return NextResponse.json({ error: "Missing required fields (name, schedule, sourceId, destinations)" }, { status: 400 }); @@ -84,6 +84,7 @@ export async function POST(req: NextRequest) { notificationEvents, namingTemplateId: namingTemplateId ?? null, schedulePresetId: schedulePresetId ?? null, + skipVerification: skipVerification ?? false, }); return NextResponse.json(newJob, { status: 201 }); diff --git a/src/app/api/settings/system-tasks/route.ts b/src/app/api/settings/system-tasks/route.ts index a683d592..19c356fc 100644 --- a/src/app/api/settings/system-tasks/route.ts +++ b/src/app/api/settings/system-tasks/route.ts @@ -107,8 +107,9 @@ export async function PUT(req: NextRequest) { if (!taskId) return NextResponse.json({ error: "Missing taskId" }, { status: 400 }); - // Run async - systemTaskService.runTask(taskId); + const user = await prisma.user.findUnique({ where: { id: ctx.userId }, select: { name: true } }); + + const executionId = await systemTaskService.runTask(taskId, "Manual", user?.name ?? "Manual"); await auditService.log( ctx.userId, @@ -118,5 +119,5 @@ export async function PUT(req: NextRequest) { taskId ); - return NextResponse.json({ success: true, message: "Task started" }); + return NextResponse.json({ success: true, ...(executionId ? { executionId } : {}) }); } diff --git a/src/app/api/storage/[id]/files/route.ts b/src/app/api/storage/[id]/files/route.ts index 4472e0dd..99b3ec06 100644 --- a/src/app/api/storage/[id]/files/route.ts +++ b/src/app/api/storage/[id]/files/route.ts @@ -25,9 +25,10 @@ export async function GET(req: NextRequest, props: { params: Promise<{ id: strin const params = await props.params; const url = new URL(req.url); const typeFilter = url.searchParams.get("typeFilter") || undefined; + const bypassCache = url.searchParams.get("refresh") === "true"; // Delegate logic to Service - const enrichedFiles = await storageService.listFilesWithMetadata(params.id, typeFilter); + const enrichedFiles = await storageService.listFilesWithMetadata(params.id, typeFilter, bypassCache); return NextResponse.json(enrichedFiles); diff --git a/src/app/api/storage/[id]/verify-async/route.ts b/src/app/api/storage/[id]/verify-async/route.ts new file mode 100644 index 00000000..58b77495 --- /dev/null +++ b/src/app/api/storage/[id]/verify-async/route.ts @@ -0,0 +1,93 @@ +import { NextRequest, NextResponse } from "next/server"; +import { verificationService } from "@/services/storage/verification-service"; +import { SystemTaskRunner } from "@/lib/runner/system-task-runner"; +import prisma from "@/lib/prisma"; +import { headers } from "next/headers"; +import { getAuthContext, checkPermissionWithContext } from "@/lib/auth/access-control"; +import { PERMISSIONS } from "@/lib/auth/permissions"; +import { logger } from "@/lib/logging/logger"; +import { wrapError, getErrorMessage } from "@/lib/logging/errors"; + +const log = logger.child({ route: "storage/verify-async" }); + +const VERIFICATION_STAGE_PROGRESS_MAP: Record = { + "Initializing": [0, 5], + "Verifying": [5, 95], + "Completed": [100, 100], + "Failed": [100, 100], +}; + +export async function POST(req: NextRequest, props: { params: Promise<{ id: string }> }) { + const ctx = await getAuthContext(await headers()); + if (!ctx) return NextResponse.json({ error: "Unauthorized" }, { status: 401 }); + + const params = await props.params; + + try { + checkPermissionWithContext(ctx, PERMISSIONS.STORAGE.READ); + + const body = await req.json(); + const { file } = body; + + if (!file || typeof file !== "string" || file.includes("..") || file.startsWith("/")) { + return NextResponse.json({ error: "Invalid file path" }, { status: 400 }); + } + + const user = await prisma.user.findUnique({ where: { id: ctx.userId }, select: { name: true } }); + + const runner = await SystemTaskRunner.create( + "Verification", + "Manual", + user?.name ?? "Manual", + VERIFICATION_STAGE_PROGRESS_MAP + ); + + // Run verification in the background โ€” return executionId immediately. + (async () => { + try { + await runner.start(); + runner.setStage("Initializing"); + runner.logEntry(`Verifying ${file}`, "info"); + + runner.setStage("Verifying"); + const result = await verificationService.verifyFile(params.id, file, "manual"); + + if (result.status === "passed") { + runner.setStage("Completed"); + runner.logEntry("Checksum verified successfully", "success"); + await runner.finish("Success"); + } else if (result.status === "failed") { + runner.setStage("Completed"); + runner.logEntry( + "Checksum mismatch", + "error", + "general", + `Expected: ${result.expectedChecksum ?? "unknown"}\nActual: ${result.actualChecksum ?? "unknown"}` + ); + await runner.finish("Failed"); + } else { + const skipReasons: Record = { + no_metadata: "No metadata file found", + no_checksum: "No checksum stored in metadata", + download_error: "Download failed", + skipped: "Already verified", + }; + const reason = skipReasons[result.status] ?? result.status; + runner.setStage("Completed"); + runner.logEntry(`Skipped: ${reason}`, "info"); + await runner.finish("Success"); + } + } catch (e: unknown) { + log.error("Async verification failed", { file, storageConfigId: params.id }, wrapError(e)); + runner.logEntry(getErrorMessage(e), "error"); + runner.setStage("Failed"); + await runner.finish("Failed"); + } + })(); + + return NextResponse.json({ success: true, executionId: runner.id }); + } catch (error: unknown) { + log.error("Verify-async route error", { id: params.id }, wrapError(error)); + return NextResponse.json({ error: getErrorMessage(error) }, { status: 500 }); + } +} diff --git a/src/app/api/storage/[id]/verify/route.ts b/src/app/api/storage/[id]/verify/route.ts new file mode 100644 index 00000000..6c4bb3a7 --- /dev/null +++ b/src/app/api/storage/[id]/verify/route.ts @@ -0,0 +1,37 @@ +import { NextRequest, NextResponse } from "next/server"; +import { verificationService } from "@/services/storage/verification-service"; +import { headers } from "next/headers"; +import { getAuthContext, checkPermissionWithContext } from "@/lib/auth/access-control"; +import { PERMISSIONS } from "@/lib/auth/permissions"; +import { logger } from "@/lib/logging/logger"; +import { wrapError, getErrorMessage } from "@/lib/logging/errors"; + +const log = logger.child({ route: "storage/verify" }); + +export async function POST(req: NextRequest, props: { params: Promise<{ id: string }> }) { + const ctx = await getAuthContext(await headers()); + + if (!ctx) { + return NextResponse.json({ error: "Unauthorized" }, { status: 401 }); + } + + const params = await props.params; + + try { + checkPermissionWithContext(ctx, PERMISSIONS.STORAGE.READ); + + const body = await req.json(); + const { file } = body; + + if (!file || typeof file !== 'string' || file.includes('..') || file.startsWith('/')) { + return NextResponse.json({ error: "Invalid file path" }, { status: 400 }); + } + + const result = await verificationService.verifyFile(params.id, file, 'manual'); + + return NextResponse.json({ success: true, data: result }); + } catch (error: unknown) { + log.error("Verify route error", { id: params.id }, wrapError(error)); + return NextResponse.json({ error: getErrorMessage(error) }, { status: 500 }); + } +} diff --git a/src/app/dashboard/history/columns.tsx b/src/app/dashboard/history/columns.tsx index 86f94718..318e25a0 100644 --- a/src/app/dashboard/history/columns.tsx +++ b/src/app/dashboard/history/columns.tsx @@ -24,6 +24,10 @@ export interface Execution { triggerLabel?: string | null; } +const SYSTEM_TASK_TYPE_LABELS: Record = { + IntegrityCheck: "Backup Integrity Check", +}; + export const createColumns = (onViewLogs: (execution: Execution) => void): ColumnDef[] => [ { id: "jobName", @@ -163,3 +167,89 @@ export const createColumns = (onViewLogs: (execution: Execution) => void): Colum } } ]; + +export const createSystemTaskColumns = (onViewLogs: (execution: Execution) => void): ColumnDef[] => [ + { + id: "taskName", + accessorFn: (row) => SYSTEM_TASK_TYPE_LABELS[row.type ?? ""] ?? row.type ?? "System Task", + header: "Task", + cell: ({ row }) => ( + {row.getValue("taskName")} + ), + }, + { + id: "trigger", + accessorFn: (row) => row.triggerType ?? "", + header: "Trigger", + filterFn: (row, _id, value) => { + return value.includes(row.original.triggerType ?? ""); + }, + cell: ({ row }) => { + const triggerType = row.original.triggerType; + const triggerLabel = row.original.triggerLabel; + + if (!triggerType) { + return -; + } + + const iconClass = "h-3.5 w-3.5 shrink-0"; + let icon: React.ReactNode; + let badgeClass: string; + + if (triggerType === "Scheduler") { + icon = ; + badgeClass = "bg-violet-100 text-violet-700 border-violet-200 dark:bg-violet-950 dark:text-violet-300 dark:border-violet-800"; + } else { + icon = ; + badgeClass = "bg-sky-100 text-sky-700 border-sky-200 dark:bg-sky-950 dark:text-sky-300 dark:border-sky-800"; + } + + return ( + + {icon} + {triggerLabel || triggerType} + + ); + }, + }, + { + accessorKey: "status", + header: "Status", + cell: ({ row }) => { + const status = row.getValue("status") as string; + if (status === "Success") { + return Success; + } else if (status === "Failed") { + return Failed; + } else if (status === "Running") { + return Running; + } + return {status}; + }, + filterFn: (row, id, value) => value.includes(row.getValue(id)), + }, + { + accessorKey: "startedAt", + header: "Started At", + cell: ({ row }) => , + }, + { + accessorKey: "endedAt", + header: "Duration", + cell: ({ row }) => { + const start = new Date(row.original.startedAt); + const end = row.original.endedAt ? new Date(row.original.endedAt) : null; + if (!end) return Running...; + return {formatDuration(end.getTime() - start.getTime())}; + }, + }, + { + id: "actions", + cell: ({ row }) => ( + + ), + }, +]; diff --git a/src/app/dashboard/history/page.tsx b/src/app/dashboard/history/page.tsx index 1cc1d575..d66af9f8 100644 --- a/src/app/dashboard/history/page.tsx +++ b/src/app/dashboard/history/page.tsx @@ -9,16 +9,18 @@ import { DialogTitle, } from "@/components/ui/dialog" import { DataTable } from "@/components/ui/data-table"; -import { createColumns, Execution } from "./columns"; +import { createColumns, createSystemTaskColumns, Execution } from "./columns"; import { createNotificationLogColumns, NotificationLogRow } from "./notification-log-columns"; import { NotificationPreview } from "./notification-preview"; import { useSearchParams, useRouter } from "next/navigation"; -import { Loader2, Square } from "lucide-react"; +import { Loader2, Square, Copy, Download } from "lucide-react"; import { Progress } from "@/components/ui/progress"; import { DateDisplay } from "@/components/utils/date-display"; import { Card, CardContent, CardHeader, CardTitle, CardDescription } from "@/components/ui/card"; import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"; import { LogViewer } from "@/components/execution/log-viewer"; +import { sanitizeLogs } from "@/lib/logs/sanitize"; +import { formatLogsAsText, generateLogFilename } from "@/lib/logs/format"; import { ScrollArea } from "@/components/ui/scroll-area"; import { Badge } from "@/components/ui/badge"; import { Button } from "@/components/ui/button"; @@ -32,6 +34,7 @@ export default function HistoryPage() { function HistoryContent() { const [executions, setExecutions] = useState([]); + const [systemTasks, setSystemTasks] = useState([]); const [systemTimezone, setSystemTimezone] = useState("UTC"); const [selectedLog, setSelectedLog] = useState(null); const [activeTab, setActiveTab] = useState("activity"); @@ -50,26 +53,23 @@ function HistoryContent() { // Sync selectedLog with latest executions data to enable live updates in modal useEffect(() => { if (selectedLog) { - const updatedLog = executions.find(e => e.id === selectedLog.id); - // Only update if the content has actually changed to prevent loops + const allExecs = [...executions, ...systemTasks]; + const updatedLog = allExecs.find(e => e.id === selectedLog.id); if (updatedLog && JSON.stringify(updatedLog) !== JSON.stringify(selectedLog)) { setSelectedLog(updatedLog); } } - }, [executions, selectedLog]); + }, [executions, systemTasks, selectedLog]); useEffect(() => { - if (executionId && executions.length > 0) { - // Check if we are already viewing it or explicitly closed it (not easily tracked here without ref, but let's assume if query param exists we want to open) - // To prevent re-opening, we remove the query param immediately after finding the log - const found = executions.find(e => e.id === executionId); + if (executionId && (executions.length > 0 || systemTasks.length > 0)) { + const found = [...executions, ...systemTasks].find(e => e.id === executionId); if (found && !selectedLog) { setSelectedLog(found); - // Clear the query param so it doesn't re-trigger on close router.replace("/dashboard/history", { scroll: false }); } } - }, [executions, executionId, selectedLog, router]); + }, [executions, systemTasks, executionId, selectedLog, router]); const fetchInFlight = useRef(false); @@ -80,7 +80,9 @@ function HistoryContent() { const res = await fetch("/api/history"); if (res.ok) { const data = await res.json(); - setExecutions(data.executions); + const systemTypes = ["IntegrityCheck", "Verification"]; + setSystemTasks(data.executions.filter((e: Execution) => systemTypes.includes(e.type ?? ""))); + setExecutions(data.executions.filter((e: Execution) => !systemTypes.includes(e.type ?? ""))); setSystemTimezone(data.systemTimezone); } } catch (_e) { @@ -102,10 +104,10 @@ function HistoryContent() { } }, []); - // Poll history: 5s default, 2s when a job is running for live feel + // Poll history: 5s default, 2s when any job or system task is running for live feel const hasRunningJob = useMemo( - () => executions.some(e => e.status === "Running" || e.status === "Pending"), - [executions] + () => [...executions, ...systemTasks].some(e => e.status === "Running" || e.status === "Pending"), + [executions, systemTasks] ); useEffect(() => { @@ -151,7 +153,50 @@ function HistoryContent() { } }, [fetchHistory]); + const handleCopyLogs = useCallback(() => { + if (!selectedLog) return; + const logs = sanitizeLogs(parseLogs(selectedLog.logs)); + const text = formatLogsAsText(logs, { + jobName: selectedLog.job?.name ?? selectedLog.type ?? "Unknown", + type: selectedLog.type ?? "Backup", + status: selectedLog.status, + startedAt: selectedLog.startedAt, + endedAt: selectedLog.endedAt, + triggerType: selectedLog.triggerType, + triggerLabel: selectedLog.triggerLabel, + }); + navigator.clipboard.writeText(text) + .then(() => toast.success("Logs copied to clipboard")) + .catch(() => toast.error("Failed to copy logs")); + }, [selectedLog]); + + const handleDownloadLog = useCallback(() => { + if (!selectedLog) return; + const logs = sanitizeLogs(parseLogs(selectedLog.logs)); + const text = formatLogsAsText(logs, { + jobName: selectedLog.job?.name ?? selectedLog.type ?? "Unknown", + type: selectedLog.type ?? "Backup", + status: selectedLog.status, + startedAt: selectedLog.startedAt, + endedAt: selectedLog.endedAt, + triggerType: selectedLog.triggerType, + triggerLabel: selectedLog.triggerLabel, + }); + const filename = generateLogFilename( + selectedLog.job?.name ?? selectedLog.type ?? "log", + selectedLog.startedAt, + ); + const blob = new Blob([text], { type: "text/plain" }); + const url = URL.createObjectURL(blob); + const a = document.createElement("a"); + a.href = url; + a.download = filename; + a.click(); + URL.revokeObjectURL(url); + }, [selectedLog]); + const columns = useMemo(() => createColumns(setSelectedLog), []); + const systemTaskColumns = useMemo(() => createSystemTaskColumns(setSelectedLog), []); const notificationColumns = useMemo( () => createNotificationLogColumns(setSelectedNotification), [] @@ -187,6 +232,26 @@ function HistoryContent() { }, ], []); + const systemTaskFilterableColumns = useMemo(() => [ + { + id: "status", + title: "Status", + options: [ + { label: "Success", value: "Success" }, + { label: "Failed", value: "Failed" }, + { label: "Running", value: "Running" }, + ], + }, + { + id: "trigger", + title: "Trigger", + options: [ + { label: "Manual", value: "Manual" }, + { label: "Scheduler", value: "Scheduler" }, + ], + }, + ], []); + const notificationFilterableColumns = useMemo(() => [ { id: "adapterId", @@ -239,9 +304,8 @@ function HistoryContent() { Activity Logs - - Notification Logs - + System Tasks + Notification Logs @@ -263,6 +327,25 @@ function HistoryContent() { + + + + System Tasks + History of automated system operations such as integrity checks. + + + + + + + @@ -289,18 +372,34 @@ function HistoryContent() { { if(!open) setSelectedLog(null); }}> - - {selectedLog?.status === "Running" && } - {selectedLog?.job?.name || selectedLog?.type || "Manual Job"} - {selectedLog?.status && ( - - {selectedLog.status} - - )} - - - {selectedLog?.startedAt && } - +
+
+ + {selectedLog?.status === "Running" && } + {selectedLog?.job?.name || (selectedLog?.type === "IntegrityCheck" ? "Backup Integrity Check" : selectedLog?.type) || "Manual Job"} + {selectedLog?.status && ( + + {selectedLog.status} + + )} + + + {selectedLog?.startedAt && } + +
+ {selectedLog?.status !== "Running" && selectedLog?.status !== "Pending" && ( +
+ + +
+ )} +
{(selectedLog?.status === "Running" || selectedLog?.status === "Pending") && ( @@ -311,20 +410,22 @@ function HistoryContent() { {detail && - {detail}} {selectedLog?.status === "Running" && progress > 0 && !detail && {progress}%} - + {( + + )} {selectedLog?.status === "Running" && ( progress > 0 ? ( diff --git a/src/app/dashboard/settings/page.tsx b/src/app/dashboard/settings/page.tsx index ada25e36..f29c8429 100644 --- a/src/app/dashboard/settings/page.tsx +++ b/src/app/dashboard/settings/page.tsx @@ -92,6 +92,17 @@ export default async function SettingsPage() { retention: configRetention ? parseInt(configRetention.value) : 10, }; + const integritySkipPassed = await prisma.systemSetting.findUnique({ where: { key: 'integrity.skipPassed' } }); + const integrityMaxAgeDays = await prisma.systemSetting.findUnique({ where: { key: 'integrity.maxAgeDays' } }); + const integrityMaxFileSizeMb = await prisma.systemSetting.findUnique({ where: { key: 'integrity.maxFileSizeMb' } }); + const integrityScanMode = await prisma.systemSetting.findUnique({ where: { key: 'integrity.scanMode' } }); + const integritySettings = { + skipPassed: integritySkipPassed?.value === 'true', + maxAgeDays: integrityMaxAgeDays ? parseInt(integrityMaxAgeDays.value) || 0 : 0, + maxFileSizeMb: integrityMaxFileSizeMb ? parseInt(integrityMaxFileSizeMb.value) || 0 : 0, + scanMode: (integrityScanMode?.value === 'destinations' ? 'destinations' : 'jobs') as 'jobs' | 'destinations', + }; + // Load Rate Limit Settings const rateLimitConfig = await getRateLimitConfig(); @@ -138,7 +149,7 @@ export default async function SettingsPage() {
- + void; onToggleLock: (file: FileInfo) => void; onGenerateLink: (file: FileInfo) => void; + onVerify: (file: FileInfo) => void; canDownload: boolean; canRestore: boolean; canDelete: boolean; } -export const getColumns = ({ onRestore, onDownload, onDelete, onToggleLock, onGenerateLink, canDownload, canRestore, canDelete }: ColumnsProps): ColumnDef[] => [ +export const getColumns = ({ onRestore, onDownload, onDelete, onToggleLock, onGenerateLink, onVerify, canDownload, canRestore, canDelete }: ColumnsProps): ColumnDef[] => [ { accessorKey: "name", header: ({ column }) => { @@ -224,6 +232,7 @@ export const getColumns = ({ onRestore, onDownload, onDelete, onToggleLock, onGe onDelete={onDelete} onToggleLock={onToggleLock} onGenerateLink={onGenerateLink} + onVerify={onVerify} canDownload={canDownload} canRestore={canRestore} canDelete={canDelete} diff --git a/src/app/dashboard/storage/storage-client.tsx b/src/app/dashboard/storage/storage-client.tsx index 7f719e07..b5642768 100644 --- a/src/app/dashboard/storage/storage-client.tsx +++ b/src/app/dashboard/storage/storage-client.tsx @@ -37,6 +37,7 @@ import { Label } from "@/components/ui/label"; import { getColumns, FileInfo } from "./columns"; import { lockBackup } from "@/app/actions/storage/lock"; import { DownloadLinkModal } from "@/components/dashboard/storage/download-link-modal"; +import { IntegrityModal } from "@/components/dashboard/storage/integrity-modal"; import { StorageHistoryTab } from "@/components/dashboard/storage/storage-history-tab"; import { StorageSettingsTab } from "@/components/dashboard/storage/storage-settings-tab"; import { Skeleton } from "@/components/ui/skeleton"; @@ -82,6 +83,9 @@ export function StorageClient({ canDownload, canRestore, canDelete }: StorageCli // Download Link Modal State const [downloadLinkFile, setDownloadLinkFile] = useState(null); + // Integrity Modal State + const [verifyModalFile, setVerifyModalFile] = useState(null); + // Encryption Key Resolution Dialog State (decrypted download fallback) const [decryptKeyDialogOpen, setDecryptKeyDialogOpen] = useState(false); const [pendingDecryptFile, setPendingDecryptFile] = useState(null); @@ -118,11 +122,12 @@ export function StorageClient({ canDownload, canRestore, canDelete }: StorageCli } }, [selectedDestination, showSystemConfigs]); - const fetchFiles = async (destId: string, showSystem: boolean) => { + const fetchFiles = async (destId: string, showSystem: boolean, bypassCache = false) => { setLoading(true); try { const typeFilter = showSystem ? "SYSTEM" : "BACKUP"; - const res = await fetch(`/api/storage/${destId}/files?typeFilter=${typeFilter}`); + const qs = bypassCache ? `typeFilter=${typeFilter}&refresh=true` : `typeFilter=${typeFilter}`; + const res = await fetch(`/api/storage/${destId}/files?${qs}`); if (res.ok) { const fetchedFiles: FileInfo[] = await res.json(); setFiles(fetchedFiles); @@ -259,6 +264,10 @@ export function StorageClient({ canDownload, canRestore, canDelete }: StorageCli setDownloadLinkFile(file); }, [canDownload]); + const handleVerify = useCallback((file: FileInfo) => { + setVerifyModalFile(file); + }, []); + const confirmDelete = async () => { if (!fileToDelete) return; setDeleting(true); @@ -290,10 +299,11 @@ export function StorageClient({ canDownload, canRestore, canDelete }: StorageCli onDelete: handleDeleteClick, onToggleLock: handleToggleLock, onGenerateLink: handleGenerateLink, + onVerify: handleVerify, canDownload, canRestore, canDelete - }), [handleRestoreClick, handleDownload, handleDeleteClick, handleToggleLock, handleGenerateLink, canDownload, canRestore, canDelete]); + }), [handleRestoreClick, handleDownload, handleDeleteClick, handleToggleLock, handleGenerateLink, handleVerify, canDownload, canRestore, canDelete]); const filterableColumns = useMemo(() => { const jobs = Array.from(new Set(files.map(f => f.jobName).filter(Boolean).filter(n => n !== "Unknown"))) as string[]; @@ -321,7 +331,7 @@ export function StorageClient({ canDownload, canRestore, canDelete }: StorageCli const handleRefresh = useCallback(() => { switch (activeTab) { case "explorer": - fetchFiles(selectedDestination, showSystemConfigs); + fetchFiles(selectedDestination, showSystemConfigs, true); break; case "history": historyRef.current?.refresh(); @@ -541,6 +551,17 @@ export function StorageClient({ canDownload, canRestore, canDelete }: StorageCli /> )} + {/* Integrity Modal */} + {verifyModalFile && ( + { if (!o) setVerifyModalFile(null); }} + file={verifyModalFile} + storageConfigId={selectedDestination} + onVerifyComplete={() => fetchFiles(selectedDestination, showSystemConfigs)} + /> + )} + {/* Encryption Key Resolution Dialog (decrypted download fallback) */} (initialMeta.healthNotificationsDisabled === true); + // Disable verification (storage only) + const [skipVerification, setSkipVerification] = useState(initialMeta.skipVerification === true); + // Exclude from restore (database only) const [isRestoreExcluded, setIsRestoreExcluded] = useState(initialMeta.isRestoreExcluded === true); @@ -208,7 +211,7 @@ export function AdapterForm({ type, adapters, onSuccess, initialData, onBack }: // Build metadata with health notification preference for database/storage adapters const existingMeta = initialData?.metadata ? JSON.parse(initialData.metadata) : {}; const metadata = (type === 'database' || type === 'storage') - ? { ...existingMeta, healthNotificationsDisabled, ...(type === 'database' ? { isRestoreExcluded } : {}) } + ? { ...existingMeta, healthNotificationsDisabled, ...(type === 'database' ? { isRestoreExcluded } : {}), ...(type === 'storage' ? { skipVerification } : {}) } : existingMeta; const payload = { @@ -420,6 +423,8 @@ export function AdapterForm({ type, adapters, onSuccess, initialData, onBack }: initialData={initialData} healthNotificationsDisabled={healthNotificationsDisabled} onHealthNotificationsDisabledChange={setHealthNotificationsDisabled} + skipVerification={skipVerification} + onSkipVerificationChange={setSkipVerification} primaryCredentialId={primaryCredentialId} sshCredentialId={sshCredentialId} onPrimaryChange={setPrimaryCredentialId} diff --git a/src/components/adapter/dropbox-oauth-button.tsx b/src/components/adapter/dropbox-oauth-button.tsx index 811aafd7..40d3aa45 100644 --- a/src/components/adapter/dropbox-oauth-button.tsx +++ b/src/components/adapter/dropbox-oauth-button.tsx @@ -1,8 +1,8 @@ "use client"; -import { useState } from "react"; +import { useState, useEffect } from "react"; import { Button } from "@/components/ui/button"; -import { ExternalLink, CheckCircle2, AlertCircle, Loader2 } from "lucide-react"; +import { ExternalLink, CheckCircle2, AlertCircle, Loader2, AlertTriangle } from "lucide-react"; import { toast } from "sonner"; import { Alert, AlertDescription } from "@/components/ui/alert"; @@ -21,6 +21,28 @@ interface DropboxOAuthButtonProps { */ export function DropboxOAuthButton({ credentialId, authorized, onAuthorized }: DropboxOAuthButtonProps) { const [isLoading, setIsLoading] = useState(false); + const [tokenCheck, setTokenCheck] = useState<{ id: string; auth: boolean; result: "valid" | "expired" } | null>(null); + + const tokenState: "checking" | "valid" | "expired" | null = + !authorized || !credentialId ? null + : tokenCheck?.id === credentialId && tokenCheck?.auth === authorized ? tokenCheck.result + : "checking"; + + useEffect(() => { + if (!authorized || !credentialId) { + return; + } + let active = true; + fetch("/api/adapters/dropbox/validate-token", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ credentialId }), + }) + .then((r) => r.json()) + .then((data) => { if (active) setTokenCheck({ id: credentialId, auth: !!authorized, result: data.valid ? "valid" : "expired" }); }) + .catch(() => { if (active) setTokenCheck({ id: credentialId, auth: !!authorized, result: "valid" }); }); + return () => { active = false; }; + }, [authorized, credentialId]); if (!credentialId) { return ( @@ -33,7 +55,37 @@ export function DropboxOAuthButton({ credentialId, authorized, onAuthorized }: D ); } - if (authorized) { + if (authorized && tokenState === "checking") { + return ( + + + Verifying Dropbox authorization... + + ); + } + + if (authorized && tokenState === "expired") { + return ( + + + + Authorization expired. Please re-authorize with Dropbox. + + + + ); + } + + if (authorized && (tokenState === "valid" || tokenState === null)) { return ( @@ -73,7 +125,6 @@ export function DropboxOAuthButton({ credentialId, authorized, onAuthorized }: D ); if (!popup) { - // Popup blocked - fall back to full navigation. window.location.href = data.data.authUrl; return; } @@ -94,7 +145,6 @@ export function DropboxOAuthButton({ credentialId, authorized, onAuthorized }: D window.addEventListener("message", handleMessage); - // Fallback: detect when the popup is closed without a message. const pollClosed = setInterval(() => { if (popup.closed) { clearInterval(pollClosed); diff --git a/src/components/adapter/form-sections.tsx b/src/components/adapter/form-sections.tsx index 4e5bf759..19119961 100644 --- a/src/components/adapter/form-sections.tsx +++ b/src/components/adapter/form-sections.tsx @@ -134,6 +134,30 @@ function HealthCheckNotificationSwitch({ ); } +function DisableVerificationSwitch({ + disabled, + onChange, +}: { + disabled: boolean; + onChange: (disabled: boolean) => void; +}) { + return ( +
+
+ +

+ Exclude this destination from scheduled integrity checks. +

+
+ +
+ ); +} + function RestoreExcludedSwitch({ excluded, onChange, @@ -666,18 +690,20 @@ export function StorageFormContent({ initialData: _initialData, healthNotificationsDisabled, onHealthNotificationsDisabledChange, + skipVerification, + onSkipVerificationChange, primaryCredentialId, sshCredentialId: _sshCredentialId, onPrimaryChange, onSshChange: _onSshChange, -}: { adapter: AdapterDefinition; initialData?: AdapterConfig; healthNotificationsDisabled?: boolean; onHealthNotificationsDisabledChange?: (disabled: boolean) => void } & CredentialPickerHostProps) { +}: { adapter: AdapterDefinition; initialData?: AdapterConfig; healthNotificationsDisabled?: boolean; onHealthNotificationsDisabledChange?: (disabled: boolean) => void; skipVerification?: boolean; onSkipVerificationChange?: (disabled: boolean) => void } & CredentialPickerHostProps) { const { watch } = useFormContext(); const authType = watch("config.authType"); const storageClass = watch("config.storageClass"); const isArchivedStorageClass = storageClass === "GLACIER" || storageClass === "DEEP_ARCHIVE"; const hasRealConfigKeys = hasFields(adapter, STORAGE_CONFIG_KEYS); - // Always show Configuration tab for storage adapters (health check switch lives there) - const hasConfigKeys = hasRealConfigKeys || !!onHealthNotificationsDisabledChange; + // Always show Configuration tab for storage adapters (health check and verification switches live there) + const hasConfigKeys = hasRealConfigKeys || !!onHealthNotificationsDisabledChange || !!onSkipVerificationChange; const isGoogleDrive = adapter.id === 'google-drive'; const isDropbox = adapter.id === 'dropbox'; const isOneDrive = adapter.id === 'onedrive'; @@ -799,6 +825,12 @@ export function StorageFormContent({ onChange={onHealthNotificationsDisabledChange} /> )} + {onSkipVerificationChange && ( + + )}
)}
diff --git a/src/components/adapter/google-drive-oauth-button.tsx b/src/components/adapter/google-drive-oauth-button.tsx index 1419d828..b172b2c7 100644 --- a/src/components/adapter/google-drive-oauth-button.tsx +++ b/src/components/adapter/google-drive-oauth-button.tsx @@ -1,8 +1,8 @@ "use client"; -import { useState } from "react"; +import { useState, useEffect } from "react"; import { Button } from "@/components/ui/button"; -import { ExternalLink, CheckCircle2, AlertCircle, Loader2 } from "lucide-react"; +import { ExternalLink, CheckCircle2, AlertCircle, Loader2, AlertTriangle } from "lucide-react"; import { toast } from "sonner"; import { Alert, AlertDescription } from "@/components/ui/alert"; @@ -21,6 +21,28 @@ interface GoogleDriveOAuthButtonProps { */ export function GoogleDriveOAuthButton({ credentialId, authorized, onAuthorized }: GoogleDriveOAuthButtonProps) { const [isLoading, setIsLoading] = useState(false); + const [tokenCheck, setTokenCheck] = useState<{ id: string; auth: boolean; result: "valid" | "expired" } | null>(null); + + const tokenState: "checking" | "valid" | "expired" | null = + !authorized || !credentialId ? null + : tokenCheck?.id === credentialId && tokenCheck?.auth === authorized ? tokenCheck.result + : "checking"; + + useEffect(() => { + if (!authorized || !credentialId) { + return; + } + let active = true; + fetch("/api/adapters/google-drive/validate-token", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ credentialId }), + }) + .then((r) => r.json()) + .then((data) => { if (active) setTokenCheck({ id: credentialId, auth: !!authorized, result: data.valid ? "valid" : "expired" }); }) + .catch(() => { if (active) setTokenCheck({ id: credentialId, auth: !!authorized, result: "valid" }); }); + return () => { active = false; }; + }, [authorized, credentialId]); if (!credentialId) { return ( @@ -33,7 +55,37 @@ export function GoogleDriveOAuthButton({ credentialId, authorized, onAuthorized ); } - if (authorized) { + if (authorized && tokenState === "checking") { + return ( + + + Verifying Google Drive authorization... + + ); + } + + if (authorized && tokenState === "expired") { + return ( + + + + Authorization expired. Please re-authorize with Google. + + + + ); + } + + if (authorized && (tokenState === "valid" || tokenState === null)) { return ( diff --git a/src/components/adapter/onedrive-oauth-button.tsx b/src/components/adapter/onedrive-oauth-button.tsx index da6b39ca..c9144c90 100644 --- a/src/components/adapter/onedrive-oauth-button.tsx +++ b/src/components/adapter/onedrive-oauth-button.tsx @@ -1,8 +1,8 @@ "use client"; -import { useState } from "react"; +import { useState, useEffect } from "react"; import { Button } from "@/components/ui/button"; -import { ExternalLink, CheckCircle2, AlertCircle, Loader2 } from "lucide-react"; +import { ExternalLink, CheckCircle2, AlertCircle, Loader2, AlertTriangle } from "lucide-react"; import { toast } from "sonner"; import { Alert, AlertDescription } from "@/components/ui/alert"; @@ -21,6 +21,28 @@ interface OneDriveOAuthButtonProps { */ export function OneDriveOAuthButton({ credentialId, authorized, onAuthorized }: OneDriveOAuthButtonProps) { const [isLoading, setIsLoading] = useState(false); + const [tokenCheck, setTokenCheck] = useState<{ id: string; auth: boolean; result: "valid" | "expired" } | null>(null); + + const tokenState: "checking" | "valid" | "expired" | null = + !authorized || !credentialId ? null + : tokenCheck?.id === credentialId && tokenCheck?.auth === authorized ? tokenCheck.result + : "checking"; + + useEffect(() => { + if (!authorized || !credentialId) { + return; + } + let active = true; + fetch("/api/adapters/onedrive/validate-token", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ credentialId }), + }) + .then((r) => r.json()) + .then((data) => { if (active) setTokenCheck({ id: credentialId, auth: !!authorized, result: data.valid ? "valid" : "expired" }); }) + .catch(() => { if (active) setTokenCheck({ id: credentialId, auth: !!authorized, result: "valid" }); }); + return () => { active = false; }; + }, [authorized, credentialId]); if (!credentialId) { return ( @@ -33,7 +55,37 @@ export function OneDriveOAuthButton({ credentialId, authorized, onAuthorized }: ); } - if (authorized) { + if (authorized && tokenState === "checking") { + return ( + + + Verifying OneDrive authorization... + + ); + } + + if (authorized && tokenState === "expired") { + return ( + + + + Authorization expired. Please re-authorize with Microsoft. + + + + ); + } + + if (authorized && (tokenState === "valid" || tokenState === null)) { return ( @@ -73,7 +125,6 @@ export function OneDriveOAuthButton({ credentialId, authorized, onAuthorized }: ); if (!popup) { - // Popup blocked - fall back to full navigation. window.location.href = data.data.authUrl; return; } @@ -94,7 +145,6 @@ export function OneDriveOAuthButton({ credentialId, authorized, onAuthorized }: window.addEventListener("message", handleMessage); - // Fallback: detect when the popup is closed without a message. const pollClosed = setInterval(() => { if (popup.closed) { clearInterval(pollClosed); diff --git a/src/components/dashboard/explorer/database-table-list.tsx b/src/components/dashboard/explorer/database-table-list.tsx index 3d2a586a..423e0540 100644 --- a/src/components/dashboard/explorer/database-table-list.tsx +++ b/src/components/dashboard/explorer/database-table-list.tsx @@ -1,11 +1,24 @@ "use client"; -import { useState, useCallback, useEffect } from "react"; +import { useState, useCallback, useEffect, useMemo } from "react"; import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"; import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from "@/components/ui/table"; +import { Input } from "@/components/ui/input"; +import { Button } from "@/components/ui/button"; import { Skeleton } from "@/components/ui/skeleton"; import { Badge } from "@/components/ui/badge"; -import { AlertTriangle, TableIcon, Database, Rows3 } from "lucide-react"; +import { + AlertTriangle, + TableIcon, + Database, + Rows3, + Search, + ArrowUpDown, + ArrowUp, + ArrowDown, + X, + RefreshCw, +} from "lucide-react"; import { toast } from "sonner"; import { formatBytes } from "@/lib/utils"; @@ -22,6 +35,8 @@ interface DatabaseTableListProps { onTableClick: (tableName: string) => void; } +type SortCol = "name" | "type" | "rows" | "size"; + const TYPE_BADGE: Record = { table: { label: "Table", variant: "secondary" }, view: { label: "View", variant: "outline" }, @@ -33,6 +48,9 @@ export function DatabaseTableList({ sourceId, database, onTableClick }: Database const [tables, setTables] = useState([]); const [isLoading, setIsLoading] = useState(false); const [error, setError] = useState(null); + const [search, setSearch] = useState(""); + const [sortCol, setSortCol] = useState(null); + const [sortDir, setSortDir] = useState<"asc" | "desc">("asc"); const fetchTables = useCallback(async () => { setIsLoading(true); @@ -68,6 +86,44 @@ export function DatabaseTableList({ sourceId, database, onTableClick }: Database const hasSize = tables.some(t => t.sizeInBytes != null); const totalRows = tables.reduce((s, t) => s + (t.rowCount ?? 0), 0); + const displayed = useMemo(() => { + let result = tables.filter(t => + t.name.toLowerCase().includes(search.toLowerCase()) + ); + if (sortCol) { + result = [...result].sort((a, b) => { + let cmp = 0; + if (sortCol === "name") { + cmp = a.name.localeCompare(b.name); + } else if (sortCol === "type") { + cmp = (a.type ?? "").localeCompare(b.type ?? ""); + } else if (sortCol === "rows") { + cmp = (a.rowCount ?? -1) - (b.rowCount ?? -1); + } else if (sortCol === "size") { + cmp = (a.sizeInBytes ?? -1) - (b.sizeInBytes ?? -1); + } + return sortDir === "asc" ? cmp : -cmp; + }); + } + return result; + }, [tables, search, sortCol, sortDir]); + + function toggleSort(col: SortCol) { + if (sortCol === col) { + setSortDir(d => (d === "asc" ? "desc" : "asc")); + } else { + setSortCol(col); + setSortDir("asc"); + } + } + + function SortIcon({ col }: { col: SortCol }) { + if (sortCol !== col) return ; + return sortDir === "asc" + ? + : ; + } + return ( @@ -83,6 +139,15 @@ export function DatabaseTableList({ sourceId, database, onTableClick }: Database : `${tables.length} object${tables.length !== 1 ? "s" : ""}${totalRows > 0 ? ` ยท ~${totalRows.toLocaleString()} rows` : ""}`} + @@ -110,53 +175,118 @@ export function DatabaseTableList({ sourceId, database, onTableClick }: Database

No tables found in this database.

) : ( -
- - - - Name - Type - Rows - {hasSize && Size} - - - - {tables.map(table => { - const typeMeta = TYPE_BADGE[table.type ?? "table"] ?? TYPE_BADGE.table; - return ( - onTableClick(table.name)} + <> +
+
+ + setSearch(e.target.value)} + className="pl-8 h-8 text-sm" + /> +
+ {search && ( + + )} +
+
+
+ + + toggleSort("name")} > - - - - {table.name} - - - - - {typeMeta.label} - - - - - - {table.rowCount != null ? table.rowCount.toLocaleString() : "-"} + + Name + + + toggleSort("type")} + > + + Type + + + toggleSort("rows")} + > + + Rows + + + {hasSize && ( + toggleSort("size")} + > + + Size + + )} + + + + {displayed.length === 0 ? ( + + + No tables match your search. - {hasSize && ( - - {table.sizeInBytes != null ? formatBytes(table.sizeInBytes) : "-"} - - )} - ); - })} - -
-
+ ) : ( + displayed.map(table => { + const typeMeta = TYPE_BADGE[table.type ?? "table"] ?? TYPE_BADGE.table; + return ( + onTableClick(table.name)} + > + + + + {table.name} + + + + + {typeMeta.label} + + + + + + {table.rowCount != null ? table.rowCount.toLocaleString() : "-"} + + + {hasSize && ( + + {table.sizeInBytes != null ? formatBytes(table.sizeInBytes) : "-"} + + )} + + ); + }) + )} + + + + )}
diff --git a/src/components/dashboard/explorer/source-version-history.tsx b/src/components/dashboard/explorer/source-version-history.tsx index 82693783..c383ebf6 100644 --- a/src/components/dashboard/explorer/source-version-history.tsx +++ b/src/components/dashboard/explorer/source-version-history.tsx @@ -8,8 +8,9 @@ import { Badge } from "@/components/ui/badge"; import { ChartContainer, ChartTooltip, ChartTooltipContent } from "@/components/ui/chart"; import type { ChartConfig } from "@/components/ui/chart"; import { CartesianGrid, Line, LineChart, XAxis, YAxis } from "recharts"; -import { ArrowRight, History, AlertTriangle, Server } from "lucide-react"; +import { ArrowUp, ArrowDown, History, AlertTriangle, Server } from "lucide-react"; import { useDateFormatter } from "@/hooks/use-date-formatter"; +import { compareVersions } from "@/lib/utils"; interface VersionHistoryEntry { id: string; @@ -72,12 +73,9 @@ export function SourceVersionHistory({ sourceId, currentVersion }: SourceVersion // Each unique `newVersion` is mapped to a numeric Y position (0..n-1) // because recharts cannot plot string values on a numeric axis. const ordered = [...history].reverse(); - const versionOrder: string[] = []; - for (const entry of ordered) { - if (!versionOrder.includes(entry.newVersion)) { - versionOrder.push(entry.newVersion); - } - } + const versionSet = new Set(); + for (const entry of ordered) versionSet.add(entry.newVersion); + const versionOrder = [...versionSet].sort((a, b) => compareVersions(a, b)); const chartData = ordered.map((entry) => ({ detectedAt: entry.detectedAt, versionIndex: versionOrder.indexOf(entry.newVersion), @@ -212,13 +210,22 @@ export function SourceVersionHistory({ sourceId, currentVersion }: SourceVersion
- {entry.previousVersion ? ( - <> - {entry.previousVersion} - - {entry.newVersion} - - ) : ( + {entry.previousVersion ? (() => { + const isDowngrade = compareVersions(entry.previousVersion, entry.newVersion) > 0; + return isDowngrade ? ( + <> + {entry.previousVersion} + + {entry.newVersion} + + ) : ( + <> + {entry.previousVersion} + + {entry.newVersion} + + ); + })() : ( <> Initial: {entry.newVersion} diff --git a/src/components/dashboard/jobs/job-form.tsx b/src/components/dashboard/jobs/job-form.tsx index 897401ca..d1db9442 100644 --- a/src/components/dashboard/jobs/job-form.tsx +++ b/src/components/dashboard/jobs/job-form.tsx @@ -76,6 +76,7 @@ export interface JobData { namingTemplateId?: string | null; schedulePresetId?: string | null; schedulePreset?: { id: string; name: string; schedule: string } | null; + skipVerification?: boolean; notifications: { id: string, name: string }[]; destinations: { configId: string; @@ -142,6 +143,7 @@ const jobSchema = z.object({ notificationIds: z.array(z.string()).optional(), notificationEvents: z.enum(["ALWAYS", "FAILURE_ONLY", "SUCCESS_ONLY"]).default("ALWAYS"), enabled: z.boolean().default(true), + skipVerification: z.boolean().default(false), }); const defaultRetentionValue = { mode: "NONE" as const, simple: { keepCount: 10 }, smart: { daily: 7, weekly: 4, monthly: 12, yearly: 2 } }; @@ -165,6 +167,7 @@ interface JobFormProps { namingTemplateId?: string | null; schedulePresetId?: string | null; schedulePreset?: { id: string; name: string; schedule: string } | null; + skipVerification?: boolean; notifications: { id: string; name: string }[]; destinations: { configId: string; priority: number; retention: string; retentionPolicyId?: string | null }[]; } | null; @@ -244,6 +247,7 @@ export function JobForm({ sources, destinations, notifications, encryptionProfil notificationIds: initialData?.notifications?.map((n) => n.id) || [], notificationEvents: (initialData?.notificationEvents as "ALWAYS" | "FAILURE_ONLY" | "SUCCESS_ONLY") || "ALWAYS", enabled: initialData?.enabled ?? true, + skipVerification: initialData?.skipVerification ?? false, } }); @@ -364,6 +368,7 @@ export function JobForm({ sources, destinations, notifications, encryptionProfil const { pgCompressionAlgo: _algo, pgCompressionLevel: _level, ...rest } = data; const payload = { ...rest, + skipVerification: data.skipVerification, pgCompression, encryptionProfileId: data.encryptionProfileId === "no-encryption" ? "" : data.encryptionProfileId, namingTemplateId: data.namingTemplateId || null, @@ -878,6 +883,22 @@ export function JobForm({ sources, destinations, notifications, encryptionProfil
)} + ( + +
+
+ Skip Verification + + Exclude this job from scheduled integrity checks. + +
+ + + +
+ +
+ )} /> {/* TAB 4: NOTIFICATIONS */} diff --git a/src/components/dashboard/storage/cells/actions-cell.tsx b/src/components/dashboard/storage/cells/actions-cell.tsx index 329990ed..3daebeab 100644 --- a/src/components/dashboard/storage/cells/actions-cell.tsx +++ b/src/components/dashboard/storage/cells/actions-cell.tsx @@ -1,7 +1,7 @@ import { Button } from "@/components/ui/button"; import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from "@/components/ui/tooltip"; import { DropdownMenu, DropdownMenuContent, DropdownMenuItem, DropdownMenuSeparator, DropdownMenuTrigger } from "@/components/ui/dropdown-menu"; -import { Download, RotateCcw, Trash2, Lock, FileLock2, FileCheck, Terminal } from "lucide-react"; +import { Download, RotateCcw, Trash2, Lock, FileLock2, FileCheck, Terminal, ShieldCheck, ShieldX, Shield } from "lucide-react"; import { FileInfo } from "@/app/dashboard/storage/columns"; const ARCHIVED_STORAGE_CLASSES = ["GLACIER", "DEEP_ARCHIVE"]; @@ -13,6 +13,7 @@ interface ActionsCellProps { onDelete: (file: FileInfo) => void; onToggleLock?: (file: FileInfo) => void; onGenerateLink?: (file: FileInfo) => void; + onVerify?: (file: FileInfo) => void; canDownload: boolean; canRestore: boolean; canDelete: boolean; @@ -25,6 +26,7 @@ export function ActionsCell({ onDelete, onToggleLock, onGenerateLink, + onVerify, canDownload, canRestore, canDelete @@ -32,8 +34,45 @@ export function ActionsCell({ const isArchived = ARCHIVED_STORAGE_CLASSES.includes(file.storageClass ?? ""); const archivedTooltip = "This backup is archived in S3 Glacier or Deep Archive. Restore it via the AWS Console first (S3 - select object - Actions - Initiate restore)."; + const verifyIcon = file.verification + ? file.verification.passed + ? + : + : ; + + const verifyClass = file.verification + ? file.verification.passed + ? "text-green-600 hover:text-green-700" + : "text-red-600 hover:text-red-700" + : "text-muted-foreground hover:text-foreground"; + + const verifyTooltip = file.verification + ? file.verification.passed + ? "Integrity verified - click to re-verify" + : "Integrity check failed - click to re-verify" + : "Verify integrity"; + return (
+ {onVerify && ( + + + + + + +

{verifyTooltip}

+
+
+
+ )} {onToggleLock && canDelete && ( diff --git a/src/components/dashboard/storage/cells/name-cell.tsx b/src/components/dashboard/storage/cells/name-cell.tsx index 3552981c..491deec0 100644 --- a/src/components/dashboard/storage/cells/name-cell.tsx +++ b/src/components/dashboard/storage/cells/name-cell.tsx @@ -10,18 +10,25 @@ interface NameCellProps { export function NameCell({ name, path, isEncrypted }: NameCellProps) { return (
-
+
{isEncrypted && ( - + Encrypted Backup )} - {name} + + + + {name} + + {name} + +
{path} diff --git a/src/components/dashboard/storage/integrity-modal.tsx b/src/components/dashboard/storage/integrity-modal.tsx new file mode 100644 index 00000000..82e553c4 --- /dev/null +++ b/src/components/dashboard/storage/integrity-modal.tsx @@ -0,0 +1,214 @@ +"use client"; + +import { useState } from "react"; +import { useRouter } from "next/navigation"; +import { ShieldCheck, ShieldX, Shield, Copy, Check, Loader2 } from "lucide-react"; +import { Dialog, DialogContent, DialogHeader, DialogTitle, DialogDescription } from "@/components/ui/dialog"; +import { Button } from "@/components/ui/button"; +import { DateDisplay } from "@/components/utils/date-display"; +import { toast } from "sonner"; +import { useUserPreferences } from "@/hooks/use-user-preferences"; +import type { FileInfo } from "@/app/dashboard/storage/columns"; + +interface IntegrityModalProps { + open: boolean; + onOpenChange: (open: boolean) => void; + file: FileInfo; + storageConfigId: string; + onVerifyComplete: () => void; +} + +type LiveResult = { + status: 'passed' | 'failed' | 'no_checksum' | 'no_metadata' | 'download_error'; + verifiedAt: string; + actualChecksum?: string; +}; + +function CopyButton({ value }: { value: string }) { + const [copied, setCopied] = useState(false); + + const handleCopy = async () => { + try { + await navigator.clipboard.writeText(value); + setCopied(true); + setTimeout(() => setCopied(false), 1500); + } catch {} + }; + + return ( + + ); +} + +function ChecksumRow({ label, value }: { label: string; value?: string }) { + if (!value) return null; + return ( +
+ {label} + {value} + +
+ ); +} + +function VerificationStatus({ verification, liveResult }: { + verification: FileInfo['verification']; + liveResult: LiveResult | null; +}) { + const status = liveResult ?? (verification ? { status: verification.passed ? 'passed' : 'failed', verifiedAt: verification.verifiedAt } : null); + + if (!status) { + return ( +
+ + Not yet verified +
+ ); + } + + if (status.status === 'passed') { + return ( +
+
+ + Passed +
+

+ Verified + {liveResult ? null : verification?.trigger ? ` via ${verification.trigger}` : null} +

+
+ ); + } + + if (status.status === 'failed') { + return ( +
+
+ + Failed - backup may be corrupted +
+

+ Checked +

+ {(status as any).actualChecksum && ( +
+

Actual hash on storage:

+

{(status as any).actualChecksum}

+
+ )} +
+ ); + } + + if (status.status === 'no_checksum') { + return ( +
+ + No checksum stored - legacy backup cannot be verified +
+ ); + } + + if (status.status === 'no_metadata') { + return ( +
+ + No metadata file found for this backup +
+ ); + } + + return ( +
+ + Verification error: {status.status} +
+ ); +} + +export function IntegrityModal({ open, onOpenChange, file, storageConfigId, onVerifyComplete }: IntegrityModalProps) { + const [verifying, setVerifying] = useState(false); + const [liveResult, setLiveResult] = useState(null); + const router = useRouter(); + const { autoRedirectOnJobStart } = useUserPreferences(); + + const hasChecksums = !!(file.checksum || file.checksumMd5); + + const handleVerify = async () => { + setVerifying(true); + try { + const res = await fetch(`/api/storage/${storageConfigId}/verify-async`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ file: file.path }), + }); + const data = await res.json(); + if (!res.ok) { + setLiveResult({ status: 'download_error', verifiedAt: new Date().toISOString() }); + return; + } + if (autoRedirectOnJobStart && data.executionId) { + onOpenChange(false); + router.push(`/dashboard/history?executionId=${data.executionId}`); + } else { + toast.info("Verification started. Check History for results."); + onVerifyComplete(); + onOpenChange(false); + } + } catch { + setLiveResult({ status: 'download_error', verifiedAt: new Date().toISOString() }); + } finally { + setVerifying(false); + } + }; + + return ( + { if (!o && !verifying) { setLiveResult(null); onOpenChange(false); } }}> + + + Integrity Check + {file.name} + + +
+ {hasChecksums ? ( +
+

Stored Checksums

+
+ + +
+
+ ) : ( +
+

No checksums available. This is a legacy backup that was uploaded before checksum support was added.

+
+ )} + +
+

Last Verification

+ +
+
+ +
+ +
+
+
+ ); +} diff --git a/src/components/email/system-notification-template.tsx b/src/components/email/system-notification-template.tsx index b3f7f5e2..693f69fe 100644 --- a/src/components/email/system-notification-template.tsx +++ b/src/components/email/system-notification-template.tsx @@ -74,14 +74,14 @@ function getStatusStyle(color: string | undefined, success: boolean, badge?: str // Color-based variations for success-type events switch (color) { case "#3b82f6": // blue โ€“ informational (login, etc.) - return { label: "Info", icon: "โ„น", accent: tokens.blue, bg: "#eff6ff", border: "#bfdbfe" }; + return { label: badge ?? "Info", icon: "โ„น", accent: tokens.blue, bg: "#eff6ff", border: "#bfdbfe" }; case "#8b5cf6": // purple โ€“ config backup - return { label: "Completed", icon: "โœ“", accent: tokens.purple, bg: "#f5f3ff", border: "#ddd6fe" }; + return { label: badge ?? "Completed", icon: "โœ“", accent: tokens.purple, bg: "#f5f3ff", border: "#ddd6fe" }; case "#f59e0b": // amber โ€“ warning - return { label: "Warning", icon: "โš ", accent: tokens.amber, bg: "#fffbeb", border: "#fde68a" }; + return { label: badge ?? "Warning", icon: "โš ", accent: tokens.amber, bg: "#fffbeb", border: "#fde68a" }; default: // green โ€“ success return { - label: "Success", + label: badge ?? "Success", icon: "โœ“", accent: tokens.success, bg: tokens.successBg, diff --git a/src/components/execution/log-viewer.tsx b/src/components/execution/log-viewer.tsx index d491c13b..38d55332 100644 --- a/src/components/execution/log-viewer.tsx +++ b/src/components/execution/log-viewer.tsx @@ -1,6 +1,6 @@ "use client"; -import React, { useRef, useEffect, useState, useMemo } from "react"; +import React, { useRef, useEffect, useState, useMemo, startTransition } from "react"; import { CheckCircle2, AlertCircle, @@ -13,7 +13,7 @@ import { Clock } from "lucide-react"; import { cn, formatDuration } from "@/lib/utils"; -import { LogEntry, BACKUP_STAGE_ORDER, RESTORE_STAGE_ORDER } from "@/lib/core/logs"; +import { LogEntry, BACKUP_STAGE_ORDER, RESTORE_STAGE_ORDER, INTEGRITY_CHECK_STAGE_ORDER, VERIFICATION_STAGE_ORDER } from "@/lib/core/logs"; import { Accordion, AccordionContent, @@ -68,10 +68,13 @@ export function LogViewer({ logs, className, autoScroll = true, status, executio // Grouping Logic - group by stage, sort by stage order, fill pending stages const groupedLogs = useMemo(() => { - // Detect execution type: explicit prop, or infer from stage names const stageOrder = executionType === "Restore" ? RESTORE_STAGE_ORDER - : BACKUP_STAGE_ORDER; + : executionType === "IntegrityCheck" + ? INTEGRITY_CHECK_STAGE_ORDER + : executionType === "Verification" + ? VERIFICATION_STAGE_ORDER + : BACKUP_STAGE_ORDER; // Build a map of stage โ†’ logs const stageMap = new Map(); @@ -135,23 +138,24 @@ export function LogViewer({ logs, className, autoScroll = true, status, executio return groups; }, [parsedLogs, status, executionType]); - // Auto-expand latest running stage only if user hasn't manually collapsed/expanded things + // Track the currently running stage so the effect fires when it transitions. + const runningStage = useMemo( + () => groupedLogs.find(g => g.status === "running")?.stage ?? null, + [groupedLogs] + ); + + // Auto-expand the active stage; collapse the previous one. useEffect(() => { - if (userInteracted) return; - - const lastGroup = groupedLogs[groupedLogs.length - 1]; - if (lastGroup) { - setActiveStages(prev => { - // If the last group is not in the active list, switch to it ONLY. - // This effectively collapses previous success stages. - if (!prev.includes(lastGroup.stage)) { - return [lastGroup.stage]; - } - return prev; - }); - } - // eslint-disable-next-line react-hooks/exhaustive-deps - }, [groupedLogs.length, userInteracted]); + if (userInteracted) return; + if (!runningStage) return; + const stage = runningStage; + startTransition(() => { + setActiveStages(prev => { + if (prev.includes(stage)) return prev; + return [stage]; + }); + }); + }, [runningStage, userInteracted]); // Scroll to bottom on new logs if sticky useEffect(() => { diff --git a/src/components/settings/integrity-check-settings-modal.tsx b/src/components/settings/integrity-check-settings-modal.tsx new file mode 100644 index 00000000..1f899e5b --- /dev/null +++ b/src/components/settings/integrity-check-settings-modal.tsx @@ -0,0 +1,174 @@ +"use client"; + +import { useState } from "react"; +import { Dialog, DialogContent, DialogHeader, DialogTitle, DialogDescription, DialogFooter } from "@/components/ui/dialog"; +import { Button } from "@/components/ui/button"; +import { Input } from "@/components/ui/input"; +import { Label } from "@/components/ui/label"; +import { Switch } from "@/components/ui/switch"; +import { toast } from "sonner"; +import { saveIntegritySettings } from "@/app/actions/settings/integrity-settings"; + +export interface IntegritySettings { + skipPassed: boolean; + maxAgeDays: number; + maxFileSizeMb: number; + scanMode: "jobs" | "destinations"; +} + +interface IntegrityCheckSettingsModalProps { + open: boolean; + onOpenChange: (open: boolean) => void; + initialSettings: IntegritySettings; + onSaved: (settings: IntegritySettings) => void; +} + +export function IntegrityCheckSettingsModal({ open, onOpenChange, initialSettings, onSaved }: IntegrityCheckSettingsModalProps) { + const [skipPassed, setSkipPassed] = useState(initialSettings.skipPassed); + const [maxAgeDays, setMaxAgeDays] = useState(String(initialSettings.maxAgeDays)); + const [maxFileSizeMb, setMaxFileSizeMb] = useState(String(initialSettings.maxFileSizeMb)); + const [scanMode, setScanMode] = useState<"jobs" | "destinations">(initialSettings.scanMode); + const [saving, setSaving] = useState(false); + + const handleSave = async () => { + setSaving(true); + try { + const result = await saveIntegritySettings({ + skipPassed, + maxAgeDays: parseInt(maxAgeDays) || 0, + maxFileSizeMb: parseInt(maxFileSizeMb) || 0, + scanMode, + }); + if (result.success) { + toast.success("Integrity check settings saved"); + onSaved({ + skipPassed, + maxAgeDays: parseInt(maxAgeDays) || 0, + maxFileSizeMb: parseInt(maxFileSizeMb) || 0, + scanMode, + }); + onOpenChange(false); + } else { + toast.error(result.error || "Failed to save settings"); + } + } catch { + toast.error("Failed to save settings"); + } finally { + setSaving(false); + } + }; + + return ( + { if (!saving) onOpenChange(o); }}> + + + Integrity Check Settings + + Configure which backups are included in scheduled integrity checks. + + + +
+ {/* Scan Mode */} +
+ +
+ + +
+
+ +
+
+ +

+ Only check backups that have never been verified or previously failed. + Backups with a passing result are skipped. +

+
+ +
+ +
+ +
+ setMaxAgeDays(e.target.value)} + /> + 0 = no limit +
+

+ Only verify backups newer than this many days. Older backups are counted as skipped. +

+
+ +
+ +
+ setMaxFileSizeMb(e.target.value)} + /> + 0 = no limit +
+

+ Skip files larger than this threshold. Mainly relevant for remote adapters + (SFTP, SMB, FTP, WebDAV, Dropbox) that must download the full file to verify. +

+
+
+ + + + + +
+
+ ); +} diff --git a/src/components/settings/preferences-form.tsx b/src/components/settings/preferences-form.tsx index 52d60902..95636e7a 100644 --- a/src/components/settings/preferences-form.tsx +++ b/src/components/settings/preferences-form.tsx @@ -54,7 +54,7 @@ export function PreferencesForm({ userId, autoRedirectOnJobStart: initialValue }

Automatically navigate to the History page and open the live execution view - when a backup or restore job starts. + when a backup, restore, or system task starts.

diff --git a/src/components/settings/system-tasks-settings.tsx b/src/components/settings/system-tasks-settings.tsx index 067facab..407d1300 100644 --- a/src/components/settings/system-tasks-settings.tsx +++ b/src/components/settings/system-tasks-settings.tsx @@ -1,14 +1,18 @@ "use client"; import { useEffect, useState } from "react"; +import { useRouter } from "next/navigation"; import { Button } from "@/components/ui/button"; import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"; import { Input } from "@/components/ui/input"; import { toast } from "sonner"; -import { Play, Loader2, Clock, CalendarClock } from "lucide-react"; +import { Play, Loader2, Clock, CalendarClock, Settings2 } from "lucide-react"; import { Switch } from "@/components/ui/switch"; import { Label } from "@/components/ui/label"; import { DateDisplay } from "@/components/utils/date-display"; +import { IntegrityCheckSettingsModal, type IntegritySettings } from "@/components/settings/integrity-check-settings-modal"; +import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from "@/components/ui/tooltip"; +import { useUserPreferences } from "@/hooks/use-user-preferences"; interface SystemTask { id: string; @@ -22,12 +26,22 @@ interface SystemTask { timezone: string; } -export function SystemTasksSettings() { +interface SystemTasksSettingsProps { + initialIntegritySettings?: IntegritySettings; +} + +export function SystemTasksSettings({ initialIntegritySettings }: SystemTasksSettingsProps) { const [tasks, setTasks] = useState([]); const [loading, setLoading] = useState(false); const [editing, setEditing] = useState>({}); const [saving, setSaving] = useState>({}); const [running, setRunning] = useState>({}); + const [integritySettingsOpen, setIntegritySettingsOpen] = useState(false); + const [integritySettings, setIntegritySettings] = useState( + initialIntegritySettings ?? { skipPassed: false, maxAgeDays: 0, maxFileSizeMb: 0, scanMode: "jobs" as const } + ); + const router = useRouter(); + const { autoRedirectOnJobStart } = useUserPreferences(); useEffect(() => { fetchTasks(); @@ -133,7 +147,12 @@ export function SystemTasksSettings() { body: JSON.stringify({ taskId }), }); if (res.ok) { - toast.success("Task started in background"); + const data = await res.json(); + if (data.executionId && autoRedirectOnJobStart) { + router.push(`/dashboard/history?executionId=${data.executionId}`); + } else { + toast.success("Task started in background"); + } } else { toast.error("Failed to start task"); } @@ -145,6 +164,7 @@ export function SystemTasksSettings() { }; return ( + <> System Tasks @@ -205,7 +225,22 @@ export function SystemTasksSettings() { )} -
+ {task.id === 'system.integrity_check' && ( +
+ + + + + + Configure check filters + + +
+ )} + +
+ + + ); } diff --git a/src/components/ui/data-table.tsx b/src/components/ui/data-table.tsx index eed5e3a9..24ab8090 100644 --- a/src/components/ui/data-table.tsx +++ b/src/components/ui/data-table.tsx @@ -245,7 +245,7 @@ export function DataTable({ )}
-
+
{table.getHeaderGroups().map((headerGroup) => ( diff --git a/src/lib/adapters/storage/google-drive.ts b/src/lib/adapters/storage/google-drive.ts index 3513d4ae..8150dbae 100644 --- a/src/lib/adapters/storage/google-drive.ts +++ b/src/lib/adapters/storage/google-drive.ts @@ -313,7 +313,7 @@ export const GoogleDriveAdapter: StorageAdapter = { // Navigate to subdir if specified if (dir && dir !== ".") { - rootFolderId = await resolveOrCreatePath(drive, config.folderId, dir + "/dummy"); + rootFolderId = await resolveOrCreatePath(drive, rootFolderId, dir + "/dummy"); } return await listFilesRecursive(drive, rootFolderId, dir || ""); @@ -344,6 +344,26 @@ export const GoogleDriveAdapter: StorageAdapter = { } }, + async verifyChecksum(config: GoogleDriveConfig, remotePath: string, checksums: { sha256?: string; md5?: string }): Promise<'passed' | 'failed' | 'unsupported'> { + if (!checksums.md5) return 'unsupported'; + try { + const drive = createDriveClient(config); + const fileName = path.basename(remotePath); + const dirPath = path.posix.dirname(remotePath); + const folderId = dirPath === '.' + ? (config.folderId || 'root') + : await resolveOrCreatePath(drive, config.folderId, dirPath + '/dummy'); + const file = await findFile(drive, folderId, fileName); + if (!file?.id) return 'unsupported'; + const details = await drive.files.get({ fileId: file.id, fields: 'md5Checksum' }); + const md5 = details.data.md5Checksum; + if (!md5) return 'unsupported'; + return md5.toLowerCase() === checksums.md5.toLowerCase() ? 'passed' : 'failed'; + } catch { + return 'unsupported'; + } + }, + async test(config: GoogleDriveConfig): Promise<{ success: boolean; message: string }> { try { const drive = createDriveClient(config); diff --git a/src/lib/adapters/storage/local.ts b/src/lib/adapters/storage/local.ts index 8f121287..59f4c59d 100644 --- a/src/lib/adapters/storage/local.ts +++ b/src/lib/adapters/storage/local.ts @@ -1,4 +1,5 @@ import { StorageAdapter, FileInfo } from "@/lib/core/interfaces"; +import { calculateFileChecksum } from "@/lib/crypto/checksum"; import { LogLevel, LogType } from "@/lib/core/logs"; import { LocalStorageSchema } from "@/lib/adapters/definitions"; import fs from "fs/promises"; @@ -188,6 +189,18 @@ export const LocalFileSystemAdapter: StorageAdapter = { } }, + async verifyChecksum(config: { basePath: string }, remotePath: string, checksums: { sha256?: string; md5?: string }): Promise<'passed' | 'failed' | 'unsupported'> { + if (!checksums.sha256) return 'unsupported'; + try { + const filePath = resolveSafePath(config.basePath, remotePath); + await fs.access(filePath); + const actual = await calculateFileChecksum(filePath); + return actual === checksums.sha256 ? 'passed' : 'failed'; + } catch { + return 'unsupported'; + } + }, + async test(config: { basePath: string }): Promise<{ success: boolean; message: string }> { const testFile = path.join(config.basePath, `.connection-test-${Date.now()}`); let written = false; diff --git a/src/lib/adapters/storage/onedrive.ts b/src/lib/adapters/storage/onedrive.ts index 1e681dc5..15348e03 100644 --- a/src/lib/adapters/storage/onedrive.ts +++ b/src/lib/adapters/storage/onedrive.ts @@ -423,6 +423,22 @@ export const OneDriveAdapter: StorageAdapter = { } }, + async verifyChecksum(config: OneDriveConfig, remotePath: string, checksums: { sha256?: string; md5?: string }): Promise<'passed' | 'failed' | 'unsupported'> { + if (!checksums.sha256) return 'unsupported'; + try { + const accessToken = await getAccessToken(config); + const client = createGraphClient(accessToken); + const drivePath = buildDrivePath(config.folderPath, remotePath); + const item = await client.api(driveItemPath(drivePath)).select('id,file').get(); + const sha256Base64: string | undefined = item?.file?.hashes?.sha256Hash; + if (!sha256Base64) return 'unsupported'; + const sha256Hex = Buffer.from(sha256Base64, 'base64').toString('hex').toLowerCase(); + return sha256Hex === checksums.sha256.toLowerCase() ? 'passed' : 'failed'; + } catch { + return 'unsupported'; + } + }, + async test(config: OneDriveConfig): Promise<{ success: boolean; message: string }> { try { const accessToken = await getAccessToken(config); diff --git a/src/lib/adapters/storage/s3.ts b/src/lib/adapters/storage/s3.ts index e4f0b88b..ffa7d2fe 100644 --- a/src/lib/adapters/storage/s3.ts +++ b/src/lib/adapters/storage/s3.ts @@ -1,6 +1,6 @@ -import { StorageAdapter, FileInfo } from "@/lib/core/interfaces"; +import { StorageAdapter, FileInfo, UploadOptions } from "@/lib/core/interfaces"; import { S3GenericSchema, S3AWSSchema, S3R2Schema, S3HetznerSchema } from "@/lib/adapters/definitions"; -import { S3Client, ListObjectsV2Command, GetObjectCommand, DeleteObjectCommand, PutObjectCommand, StorageClass } from "@aws-sdk/client-s3"; +import { S3Client, ListObjectsV2Command, GetObjectCommand, DeleteObjectCommand, PutObjectCommand, HeadObjectCommand, StorageClass } from "@aws-sdk/client-s3"; import { Upload } from "@aws-sdk/lib-storage"; import { createReadStream, createWriteStream } from "fs"; import { pipeline } from "stream/promises"; @@ -40,7 +40,7 @@ class S3ClientFactory { // --- Shared Implementation --- -async function s3Upload(internalConfig: S3InternalConfig, localPath: string, remotePath: string, onProgress?: (percent: number) => void, onLog?: (msg: string, level?: LogLevel, type?: LogType, details?: string) => void): Promise { +async function s3Upload(internalConfig: S3InternalConfig, localPath: string, remotePath: string, onProgress?: (percent: number) => void, onLog?: (msg: string, level?: LogLevel, type?: LogType, details?: string) => void, options?: UploadOptions): Promise { const client = S3ClientFactory.create(internalConfig); const targetKey = S3ClientFactory.getTargetKey(internalConfig, remotePath); @@ -55,6 +55,7 @@ async function s3Upload(internalConfig: S3InternalConfig, localPath: string, rem Key: targetKey, Body: fileStream, StorageClass: (internalConfig.storageClass as StorageClass) || undefined, + Metadata: options?.checksumSha256 ? { 'dbackup-sha256': options.checksumSha256 } : undefined, }, }); @@ -201,6 +202,24 @@ async function s3Delete( } } +async function s3VerifyChecksum( + internalConfig: S3InternalConfig, + remotePath: string, + checksums: { sha256?: string; md5?: string } +): Promise<'passed' | 'failed' | 'unsupported'> { + if (!checksums.sha256) return 'unsupported'; + const client = S3ClientFactory.create(internalConfig); + const targetKey = S3ClientFactory.getTargetKey(internalConfig, remotePath); + try { + const response = await client.send(new HeadObjectCommand({ Bucket: internalConfig.bucket, Key: targetKey })); + const stored = response.Metadata?.['dbackup-sha256']; + if (!stored) return 'unsupported'; + return stored === checksums.sha256 ? 'passed' : 'failed'; + } catch { + return 'unsupported'; + } +} + async function s3Test(internalConfig: S3InternalConfig): Promise<{ success: boolean; message: string }> { const client = S3ClientFactory.create(internalConfig); const testFile = `.backup-manager-test-${Date.now()}`; @@ -289,7 +308,15 @@ export const S3GenericAdapter: StorageAdapter = { credentials: { accessKeyId: config.accessKeyId, secretAccessKey: config.secretAccessKey }, forcePathStyle: config.forcePathStyle, pathPrefix: config.pathPrefix - }, ...args) + }, ...args), + verifyChecksum: (config, remotePath, checksums) => s3VerifyChecksum({ + endpoint: config.endpoint, + region: config.region, + bucket: config.bucket, + credentials: { accessKeyId: config.accessKeyId, secretAccessKey: config.secretAccessKey }, + forcePathStyle: config.forcePathStyle, + pathPrefix: config.pathPrefix + }, remotePath, checksums), }; // 2. AWS S3 @@ -332,7 +359,13 @@ export const S3AWSAdapter: StorageAdapter = { bucket: config.bucket, credentials: { accessKeyId: config.accessKeyId, secretAccessKey: config.secretAccessKey }, pathPrefix: config.pathPrefix - }, ...args) + }, ...args), + verifyChecksum: (config, remotePath, checksums) => s3VerifyChecksum({ + region: config.region, + bucket: config.bucket, + credentials: { accessKeyId: config.accessKeyId, secretAccessKey: config.secretAccessKey }, + pathPrefix: config.pathPrefix + }, remotePath, checksums), }; function r2Endpoint(accountId: string, jurisdiction?: string): string { @@ -386,7 +419,14 @@ export const S3R2Adapter: StorageAdapter = { bucket: config.bucket, credentials: { accessKeyId: config.accessKeyId, secretAccessKey: config.secretAccessKey }, pathPrefix: config.pathPrefix - }, ...args) + }, ...args), + verifyChecksum: (config, remotePath, checksums) => s3VerifyChecksum({ + endpoint: r2Endpoint(config.accountId, config.jurisdiction), + region: "auto", + bucket: config.bucket, + credentials: { accessKeyId: config.accessKeyId, secretAccessKey: config.secretAccessKey }, + pathPrefix: config.pathPrefix + }, remotePath, checksums), }; // 4. Hetzner Object Storage @@ -434,5 +474,12 @@ export const S3HetznerAdapter: StorageAdapter = { bucket: config.bucket, credentials: { accessKeyId: config.accessKeyId, secretAccessKey: config.secretAccessKey }, pathPrefix: config.pathPrefix - }, ...args) + }, ...args), + verifyChecksum: (config, remotePath, checksums) => s3VerifyChecksum({ + endpoint: `https://${config.region}.your-objectstorage.com`, + region: config.region, + bucket: config.bucket, + credentials: { accessKeyId: config.accessKeyId, secretAccessKey: config.secretAccessKey }, + pathPrefix: config.pathPrefix + }, remotePath, checksums), }; diff --git a/src/lib/core/interfaces.ts b/src/lib/core/interfaces.ts index 49fc9508..eb2f84ec 100644 --- a/src/lib/core/interfaces.ts +++ b/src/lib/core/interfaces.ts @@ -46,6 +46,15 @@ export interface BackupMetadata { }; /** SHA-256 checksum of the final backup file (after compression/encryption) */ checksum?: string; + /** MD5 checksum of the final backup file - enables native verification for Google Drive and OneDrive */ + checksumMd5?: string; + /** Result of the most recent integrity verification */ + verification?: { + verifiedAt: string; + passed: boolean; + trigger: 'manual' | 'post-upload' | 'scheduled'; + actualChecksum?: string; + }; /** Trigger information - what initiated the backup */ trigger?: { type: "Manual" | "Scheduler" | "Api"; @@ -220,6 +229,12 @@ export type FileInfo = { storageClass?: string; }; +/** Optional options passed to upload() for adapters that support native checksum storage. */ +export interface UploadOptions { + checksumSha256?: string; + checksumMd5?: string; +} + /** * A persistent upload session that reuses a single connection for multiple uploads. * Returned by `StorageAdapter.openSession()` when an adapter supports connection reuse. @@ -233,7 +248,8 @@ export interface StorageSession { localPath: string, remotePath: string, onProgress?: (percent: number) => void, - onLog?: (msg: string, level?: LogLevel, type?: LogType, details?: string) => void + onLog?: (msg: string, level?: LogLevel, type?: LogType, details?: string) => void, + options?: UploadOptions ): Promise; close(): Promise; } @@ -241,9 +257,24 @@ export interface StorageSession { export interface StorageAdapter extends BaseAdapter { type: 'storage'; /** - * Uploads a local file to the storage destination + * Uploads a local file to the storage destination. + * Pass `options.checksumSha256` / `options.checksumMd5` so adapters that support + * native checksum storage (S3, etc.) can attach the hash during the upload request. */ - upload(config: AdapterConfig, localPath: string, remotePath: string, onProgress?: (percent: number) => void, onLog?: (msg: string, level?: LogLevel, type?: LogType, details?: string) => void): Promise; + upload(config: AdapterConfig, localPath: string, remotePath: string, onProgress?: (percent: number) => void, onLog?: (msg: string, level?: LogLevel, type?: LogType, details?: string) => void, options?: UploadOptions): Promise; + + /** + * Optional: Verifies the integrity of a remote file without downloading it. + * Adapters implement this when the storage API provides native checksum access + * (S3 custom metadata, Google Drive md5Checksum, OneDrive file.hashes.sha256Hash). + * Returns 'unsupported' when the adapter cannot verify natively (e.g. SFTP, FTP). + * The VerificationService calls this first and falls back to download+hash if unsupported. + */ + verifyChecksum?( + config: AdapterConfig, + remotePath: string, + checksums: { sha256?: string; md5?: string } + ): Promise<'passed' | 'failed' | 'unsupported'>; /** * Optional: Opens a persistent session for multiple uploads on a single connection. diff --git a/src/lib/core/logs.ts b/src/lib/core/logs.ts index 2c8bdff6..98512a95 100644 --- a/src/lib/core/logs.ts +++ b/src/lib/core/logs.ts @@ -66,6 +66,38 @@ export const RESTORE_STAGE_ORDER: string[] = [ RESTORE_STAGES.COMPLETED, ]; +/** Integrity Check stages */ +export const INTEGRITY_CHECK_STAGES = { + INITIALIZING: "Initializing", + SCANNING: "Scanning Storage", + VERIFYING_CHECKSUMS: "Verifying Checksums", + COMPLETED: "Completed", + FAILED: "Failed", +} as const; + +/** Ordered list of integrity check stages for frontend rendering */ +export const INTEGRITY_CHECK_STAGE_ORDER: string[] = [ + "Initializing", + "Scanning Storage", + "Verifying Checksums", + "Completed", +]; + +export const VERIFICATION_STAGE_ORDER: string[] = [ + "Initializing", + "Verifying", + "Completed", +]; + +/** Progress ranges [min, max] for integrity check stages */ +export const INTEGRITY_CHECK_STAGE_PROGRESS_MAP: Record = { + "Initializing": [0, 5], + "Scanning Storage": [5, 20], + "Verifying Checksums": [20, 95], + "Completed": [100, 100], + "Failed": [100, 100], +}; + /** @deprecated Use BACKUP_STAGE_ORDER instead */ export const STAGE_ORDER: PipelineStage[] = BACKUP_STAGE_ORDER as PipelineStage[]; diff --git a/src/lib/crypto/checksum.ts b/src/lib/crypto/checksum.ts index 9dd4ad3c..3b70e0eb 100644 --- a/src/lib/crypto/checksum.ts +++ b/src/lib/crypto/checksum.ts @@ -32,6 +32,28 @@ export function calculateChecksum(data: Buffer | string): string { return crypto.createHash(ALGORITHM).update(data).digest("hex"); } +/** + * Calculates both SHA-256 and MD5 checksums for a file in a single streaming pass. + * More efficient than calling calculateFileChecksum() twice for large backup files. + * + * @param filePath - Absolute path to the file + * @returns Object with hex-encoded sha256 and md5 hash strings + */ +export function calculateFileChecksums(filePath: string): Promise<{ sha256: string; md5: string }> { + return new Promise((resolve, reject) => { + const sha256 = crypto.createHash("sha256"); + const md5 = crypto.createHash("md5"); + const stream = fs.createReadStream(filePath, { highWaterMark: 1024 * 1024 }); + + stream.on("data", (chunk) => { + sha256.update(chunk); + md5.update(chunk); + }); + stream.on("end", () => resolve({ sha256: sha256.digest("hex"), md5: md5.digest("hex") })); + stream.on("error", (err) => reject(err)); + }); +} + /** * Verifies a file against an expected checksum. * diff --git a/src/lib/logs/format.ts b/src/lib/logs/format.ts new file mode 100644 index 00000000..7dfd8bd3 --- /dev/null +++ b/src/lib/logs/format.ts @@ -0,0 +1,108 @@ +import { LogEntry } from "@/lib/core/logs"; + +export interface ExportMeta { + jobName: string; + type: string; + status: string; + startedAt: string; + endedAt?: string | null; + triggerType?: string | null; + triggerLabel?: string | null; +} + +function formatUtcTimestamp(iso: string): string { + const d = new Date(iso); + if (isNaN(d.getTime())) return iso; + const hh = String(d.getUTCHours()).padStart(2, "0"); + const mm = String(d.getUTCMinutes()).padStart(2, "0"); + const ss = String(d.getUTCSeconds()).padStart(2, "0"); + return `${hh}:${mm}:${ss}`; +} + +function formatUtcDatetime(iso: string): string { + const d = new Date(iso); + if (isNaN(d.getTime())) return iso; + return d.toISOString().replace("T", " ").replace(/\.\d+Z$/, " UTC"); +} + +function formatDurationMs(ms: number): string { + if (ms < 1000) return `${ms}ms`; + const s = (ms / 1000).toFixed(1); + return `${s}s`; +} + +function levelPad(level: string): string { + return level.toUpperCase().padEnd(7); +} + +export function formatLogsAsText(logs: LogEntry[], meta: ExportMeta): string { + const lines: string[] = []; + + lines.push("DBackup Execution Log"); + lines.push("=".repeat(48)); + lines.push(`Job: ${meta.jobName}`); + lines.push(`Type: ${meta.type}`); + lines.push(`Status: ${meta.status}`); + lines.push(`Started: ${formatUtcDatetime(meta.startedAt)}`); + if (meta.endedAt) { + lines.push(`Ended: ${formatUtcDatetime(meta.endedAt)}`); + } + if (meta.triggerType) { + lines.push(`Trigger: ${meta.triggerType}`); + } + lines.push("=".repeat(48)); + + // Group by stage + const stageMap = new Map(); + for (const entry of logs) { + const stage = entry.stage ?? "General"; + if (!stageMap.has(stage)) stageMap.set(stage, []); + stageMap.get(stage)!.push(entry); + } + + for (const [stage, entries] of stageMap) { + lines.push(""); + + let stageHeader = `[STAGE: ${stage}]`; + if (entries.length >= 2) { + const first = new Date(entries[0].timestamp).getTime(); + const last = new Date(entries[entries.length - 1].timestamp).getTime(); + const durationMs = last - first; + if (!isNaN(durationMs) && durationMs >= 0) { + stageHeader += ` (${formatDurationMs(durationMs)})`; + } + } + lines.push(stageHeader); + + for (const entry of entries) { + const ts = formatUtcTimestamp(entry.timestamp); + const lvl = levelPad(entry.level); + const durationSuffix = entry.durationMs != null ? ` [${formatDurationMs(entry.durationMs)}]` : ""; + lines.push(`${ts} ${lvl} ${entry.message}${durationSuffix}`); + if (entry.details) { + lines.push(" [DETAILS]"); + for (const detailLine of entry.details.split("\n")) { + lines.push(` ${detailLine}`); + } + } + } + } + + lines.push(""); + return lines.join("\n"); +} + +export function generateLogFilename(jobName: string, startedAt: string): string { + const slug = jobName + .toLowerCase() + .replace(/[^a-z0-9]+/g, "-") + .replace(/^-+|-+$/g, "") + .slice(0, 40); + + const d = new Date(startedAt); + if (isNaN(d.getTime())) return `dbackup-${slug}.log`; + + const date = d.toISOString().slice(0, 10); + const hhmm = `${String(d.getUTCHours()).padStart(2, "0")}-${String(d.getUTCMinutes()).padStart(2, "0")}`; + return `dbackup-${slug}-${date}-${hhmm}.log`; +} diff --git a/src/lib/logs/sanitize.ts b/src/lib/logs/sanitize.ts new file mode 100644 index 00000000..a72f947c --- /dev/null +++ b/src/lib/logs/sanitize.ts @@ -0,0 +1,38 @@ +import { LogEntry } from "@/lib/core/logs"; +import { SENSITIVE_KEYS } from "@/lib/crypto"; + +const IPV4_PATTERN = /\b(?:\d{1,3}\.){3}\d{1,3}\b/g; +const IPV6_PATTERN = /\b(?:[0-9a-fA-F]{0,4}:){2,7}[0-9a-fA-F]{0,4}\b/g; +const CONNECTION_STRING_PATTERN = /((?:mongodb(?:\+srv)?|mysql|postgres(?:ql)?|redis|ftp|sftp):\/\/)[^@\s]+(@)/gi; + +function redactString(value: string): string { + return value + .replace(CONNECTION_STRING_PATTERN, "$1[CREDENTIALS REDACTED]$2") + .replace(IPV4_PATTERN, "[IP REDACTED]") + .replace(IPV6_PATTERN, "[IP REDACTED]"); +} + +function redactContext(ctx: Record): Record { + const result: Record = {}; + for (const [key, value] of Object.entries(ctx)) { + if (SENSITIVE_KEYS.includes(key)) { + result[key] = "[REDACTED]"; + } else if (value && typeof value === "object" && !Array.isArray(value)) { + result[key] = redactContext(value as Record); + } else if (typeof value === "string") { + result[key] = redactString(value); + } else { + result[key] = value; + } + } + return result; +} + +export function sanitizeLogs(logs: LogEntry[]): LogEntry[] { + return logs.map((entry) => ({ + ...entry, + message: redactString(entry.message), + details: entry.details ? redactString(entry.details) : entry.details, + context: entry.context ? redactContext(entry.context) : entry.context, + })); +} diff --git a/src/lib/notifications/templates.ts b/src/lib/notifications/templates.ts index 3bd3ab13..047bc426 100644 --- a/src/lib/notifications/templates.ts +++ b/src/lib/notifications/templates.ts @@ -384,9 +384,28 @@ function dbVersionChangedTemplate( data: DbVersionChangedData ): NotificationPayload { const from = data.previousVersion ?? "unknown"; + if (data.isDowngrade) { + return { + title: `Database Version Downgraded: ${data.sourceName}`, + message: `Source '${data.sourceName}' was downgraded (${from} โ†’ ${data.newVersion}).`, + fields: [ + { name: "Source", value: data.sourceName, inline: true }, + { name: "Adapter", value: data.adapterId, inline: true }, + { name: "Previous Version", value: from, inline: true }, + { name: "New Version", value: data.newVersion, inline: true }, + ...(data.edition + ? [{ name: "Edition", value: data.edition, inline: true }] + : []), + { name: "Time", value: data.timestamp, inline: true }, + ], + color: "#f59e0b", // amber - warning + success: false, + badge: "Downgrade", + }; + } return { - title: `Database Version Changed: ${data.sourceName}`, - message: `Source '${data.sourceName}' reported a new engine version (${from} โ†’ ${data.newVersion}).`, + title: `Database Version Upgraded: ${data.sourceName}`, + message: `Source '${data.sourceName}' was upgraded (${from} โ†’ ${data.newVersion}).`, fields: [ { name: "Source", value: data.sourceName, inline: true }, { name: "Adapter", value: data.adapterId, inline: true }, @@ -399,7 +418,7 @@ function dbVersionChangedTemplate( ], color: "#3b82f6", // blue success: true, - badge: "Version", + badge: "Upgrade", }; } diff --git a/src/lib/notifications/types.ts b/src/lib/notifications/types.ts index f3699665..c56af1f9 100644 --- a/src/lib/notifications/types.ts +++ b/src/lib/notifications/types.ts @@ -181,6 +181,7 @@ export interface DbVersionChangedData { newVersion: string; edition?: string; timestamp: string; + isDowngrade: boolean; } /** Union of all event data types for type-safe template dispatch */ diff --git a/src/lib/runner/steps/03-upload.ts b/src/lib/runner/steps/03-upload.ts index 661345f4..20a16ad9 100644 --- a/src/lib/runner/steps/03-upload.ts +++ b/src/lib/runner/steps/03-upload.ts @@ -10,10 +10,10 @@ import { createEncryptionStream } from "@/lib/crypto/stream"; import { getCompressionStream, getCompressionExtension, CompressionType } from "@/lib/crypto/compression"; import { ProgressMonitorStream } from "@/lib/streams/progress-monitor"; import { formatBytes } from "@/lib/utils"; -import { calculateFileChecksum, verifyFileChecksum } from "@/lib/crypto/checksum"; -import { getTempDir } from "@/lib/temp-dir"; +import { calculateFileChecksums } from "@/lib/crypto/checksum"; import { PIPELINE_STAGES } from "@/lib/core/logs"; import { withStorageSession } from "./upload-helpers"; +import { verificationService } from "@/services/storage/verification-service"; export async function stepUpload(ctx: RunnerContext) { if (!ctx.job || ctx.destinations.length === 0 || !ctx.tempFile) throw new Error("Context not ready for upload"); @@ -117,10 +117,11 @@ export async function stepUpload(ctx: RunnerContext) { } } - // --- CHECKSUM CALCULATION --- - ctx.log("Calculating SHA-256 checksum..."); - const checksum = await calculateFileChecksum(ctx.tempFile); - ctx.log(`Checksum: ${checksum}`); + // --- CHECKSUM CALCULATION (SHA-256 + MD5 in single stream pass) --- + ctx.log("Calculating checksums..."); + const { sha256: checksum, md5: checksumMd5 } = await calculateFileChecksums(ctx.tempFile); + ctx.log(`SHA-256: ${checksum}`); + ctx.log(`MD5: ${checksumMd5}`); // --- PRIVACY SETTING: include actor in metadata? --- const privacySetting = await prisma.systemSetting.findUnique({ where: { key: "privacy.includeActorInMetadata" } }); @@ -153,6 +154,7 @@ export async function stepUpload(ctx: RunnerContext) { compression: compressionMeta, encryption: encryptionMeta, checksum, + checksumMd5, multiDb: ctx.metadata?.multiDb, trigger, locked: ctx.lock === true, @@ -167,9 +169,6 @@ export async function stepUpload(ctx: RunnerContext) { ctx.setStage(PIPELINE_STAGES.UPLOADING); - // Collect destinations that need post-upload integrity verification - const verifyQueue: { dest: typeof ctx.destinations[number]; destLabel: string }[] = []; - for (let i = 0; i < totalDests; i++) { const dest = ctx.destinations[i]; const destLabel = `[${dest.configName}]`; @@ -206,12 +205,13 @@ export async function stepUpload(ctx: RunnerContext) { sessionLog ); - // Upload main backup file + // Upload main backup file (pass checksums so adapters can store them natively) const uploadSuccess = await session.upload( ctx.tempFile!, remotePath, destProgress, - sessionLog + sessionLog, + { checksumSha256: checksum, checksumMd5 } ); if (!uploadSuccess) { @@ -221,11 +221,31 @@ export async function stepUpload(ctx: RunnerContext) { dest.uploadResult = { success: true, path: remotePath }; ctx.log(`${destLabel} Upload complete: ${remotePath}`); - - // Queue integrity verification for local storage - if (dest.adapterId === "local-filesystem") { - verifyQueue.push({ dest, destLabel }); - } + const dbCount = typeof metadata.databases === 'object' && 'count' in metadata.databases + ? (metadata.databases as { count: number }).count + : (Array.isArray(metadata.databases) ? metadata.databases.length : 0); + const richEntry = { + name: path.basename(remotePath), + path: remotePath, + size: ctx.dumpSize ?? 0, + lastModified: new Date(), + jobName: metadata.jobName, + sourceName: metadata.sourceName, + sourceType: metadata.sourceType, + engineVersion: metadata.engineVersion, + engineEdition: metadata.engineEdition, + dbInfo: { count: dbCount, label: dbCount <= 1 ? "Single DB" : `${dbCount} DBs` }, + isEncrypted: metadata.encryption?.enabled, + encryptionProfileId: metadata.encryption?.profileId, + compression: metadata.compression, + locked: metadata.locked ?? false, + trigger: metadata.trigger as { type: string; actor?: string } | undefined, + checksum: metadata.checksum, + checksumMd5: metadata.checksumMd5, + }; + import("@/services/storage/storage-service").then(({ storageService }) => { + storageService.appendStorageListCacheEntry(dest.configId, richEntry).catch(() => {}); + }); } catch (e: unknown) { const message = e instanceof Error ? e.message : String(e); @@ -238,30 +258,34 @@ export async function stepUpload(ctx: RunnerContext) { await fs.unlink(metaPath).catch(() => {}); // --- POST-UPLOAD VERIFICATION --- - ctx.setStage(PIPELINE_STAGES.VERIFYING); + const postVerifySetting = await prisma.systemSetting.findUnique({ where: { key: "backup.postUploadVerify" } }); + const postVerifyEnabled = postVerifySetting?.value === 'true'; + + // Always verify local destinations (native, no bandwidth cost); verify remote only if setting enabled + const destinationsToVerify = ctx.destinations.filter(d => + d.uploadResult?.success && (d.adapterId === "local-filesystem" || postVerifyEnabled) + ); - if (verifyQueue.length > 0) { - for (const { dest, destLabel } of verifyQueue) { + if (destinationsToVerify.length > 0) { + ctx.setStage(PIPELINE_STAGES.VERIFYING); + + for (const dest of destinationsToVerify) { + const destLabel = `[${dest.configName}]`; try { ctx.log(`${destLabel} Verifying upload integrity...`); - const verifyPath = path.join(getTempDir(), `verify_${Date.now()}_${path.basename(ctx.tempFile)}`); - const downloadOk = await dest.adapter.download(dest.config, remotePath, verifyPath); - if (downloadOk) { - const result = await verifyFileChecksum(verifyPath, checksum); - if (result.valid) { - ctx.log(`${destLabel} Integrity check passed โœ“`, 'success'); - } else { - ctx.log(`${destLabel} WARNING: Integrity check FAILED! Expected: ${result.expected}, Got: ${result.actual}`, 'warning'); - } - await fs.unlink(verifyPath).catch(() => {}); + const verifyResult = await verificationService.verifyFile(dest.configId, remotePath, 'post-upload'); + if (verifyResult.status === 'passed') { + ctx.log(`${destLabel} Integrity check passed`, 'success'); + } else if (verifyResult.status === 'failed') { + ctx.log(`${destLabel} WARNING: Integrity check FAILED. Expected: ${verifyResult.expectedChecksum}, Got: ${verifyResult.actualChecksum}`, 'warning'); + } else { + ctx.log(`${destLabel} Integrity verification skipped: ${verifyResult.status}`, 'info'); } } catch (e: unknown) { const message = e instanceof Error ? e.message : String(e); - ctx.log(`${destLabel} Integrity verification skipped: ${message}`, 'warning'); + ctx.log(`${destLabel} Integrity verification error: ${message}`, 'warning'); } } - } else { - ctx.log("No local destinations - skipping integrity verification"); } // --- EVALUATE RESULTS --- diff --git a/src/lib/runner/steps/05-retention.ts b/src/lib/runner/steps/05-retention.ts index 0c95496a..4ff5fbe9 100644 --- a/src/lib/runner/steps/05-retention.ts +++ b/src/lib/runner/steps/05-retention.ts @@ -40,6 +40,7 @@ export async function stepRetention(ctx: RunnerContext) { } } + async function applyRetentionForDestination(ctx: RunnerContext, dest: DestinationContext, timezone: string): Promise { const destLabel = `[${dest.configName}]`; const policy = dest.retention; @@ -115,6 +116,9 @@ async function applyRetentionForDestination(ctx: RunnerContext, dest: Destinatio const metaPath = file.path + ".meta.json"; await dest.adapter.delete(dest.config, metaPath).catch(() => {}); deletedCount++; + import("@/services/storage/storage-service").then(({ storageService }) => { + storageService.removeStorageListCacheEntry(dest.configId, file.path).catch(() => {}); + }); } } catch (delError: unknown) { const message = delError instanceof Error ? delError.message : String(delError); diff --git a/src/lib/runner/steps/upload-helpers.ts b/src/lib/runner/steps/upload-helpers.ts index 30eaadc4..9618d493 100644 --- a/src/lib/runner/steps/upload-helpers.ts +++ b/src/lib/runner/steps/upload-helpers.ts @@ -32,8 +32,8 @@ export async function withStorageSession( function createStatelessSessionShim(adapter: StorageAdapter, config: AdapterConfig): StorageSession { return { - upload: (localPath, remotePath, onProgress, onLog) => - adapter.upload(config, localPath, remotePath, onProgress, onLog), + upload: (localPath, remotePath, onProgress, onLog, options) => + adapter.upload(config, localPath, remotePath, onProgress, onLog, options), close: async () => { }, }; } diff --git a/src/lib/runner/system-task-runner.ts b/src/lib/runner/system-task-runner.ts new file mode 100644 index 00000000..dcad940b --- /dev/null +++ b/src/lib/runner/system-task-runner.ts @@ -0,0 +1,181 @@ +import prisma from "@/lib/prisma"; +import { LogEntry, LogLevel, LogType, INTEGRITY_CHECK_STAGE_PROGRESS_MAP } from "@/lib/core/logs"; +import { logger } from "@/lib/logging/logger"; +import { wrapError } from "@/lib/logging/errors"; +import { formatDuration } from "@/lib/utils"; + +const log = logger.child({ module: "SystemTaskRunner" }); + +export class SystemTaskRunner { + private executionId: string; + private logs: LogEntry[] = []; + private currentStage = "Initializing"; + private currentProgress = 0; + private currentDetail = ""; + private stageStartTimes = new Map(); + private lastLogFlush = 0; + private isFlushing = false; + private hasPendingFlush = false; + private stageProgressMap: Record; + + private constructor( + executionId: string, + stageProgressMap: Record + ) { + this.executionId = executionId; + this.stageProgressMap = stageProgressMap; + } + + static async create( + taskType: string, + triggerType?: "Manual" | "Scheduler", + triggerLabel?: string, + stageProgressMap: Record = INTEGRITY_CHECK_STAGE_PROGRESS_MAP + ): Promise { + const initialLog: LogEntry = { + timestamp: new Date().toISOString(), + level: "info", + type: "general", + message: "Task queued", + stage: "Initializing", + }; + + const execution = await prisma.execution.create({ + data: { + jobId: null, + type: taskType, + status: "Pending", + logs: JSON.stringify([initialLog]), + metadata: JSON.stringify({ progress: 0, stage: "Initializing" }), + triggerType: triggerType ?? null, + triggerLabel: triggerLabel ?? null, + }, + }); + + return new SystemTaskRunner(execution.id, stageProgressMap); + } + + get id(): string { + return this.executionId; + } + + async start(): Promise { + const claimed = await prisma.execution.updateMany({ + where: { id: this.executionId, status: "Pending" }, + data: { status: "Running", startedAt: new Date() }, + }); + + if (claimed.count === 0) { + throw new Error("Execution already claimed by a concurrent call"); + } + } + + async finish(status: "Success" | "Failed"): Promise { + await this.flushLogs(true); + await prisma.execution.update({ + where: { id: this.executionId }, + data: { + status, + endedAt: new Date(), + logs: JSON.stringify(this.logs), + metadata: JSON.stringify({ + progress: this.currentProgress, + stage: this.currentStage, + }), + }, + }); + } + + logEntry(message: string, level: LogLevel = "info", type: LogType = "general", details?: string): void { + const entry: LogEntry = { + timestamp: new Date().toISOString(), + level, + type, + message, + stage: this.currentStage, + details, + }; + this.logs.push(entry); + void this.flushLogs(); + } + + setStage(stage: string): void { + const prevStart = this.stageStartTimes.get(this.currentStage); + if (prevStart && this.currentStage !== stage) { + const durationMs = Date.now() - prevStart; + this.logs.push({ + timestamp: new Date().toISOString(), + level: "success", + type: "general", + message: `${this.currentStage} completed (${formatDuration(durationMs)})`, + stage: this.currentStage, + durationMs, + }); + } + + this.currentStage = stage; + this.currentDetail = ""; + this.stageStartTimes.set(stage, Date.now()); + + const range = this.stageProgressMap[stage]; + this.currentProgress = range ? range[0] : this.currentProgress; + + void this.flushLogs(true); + } + + updateStageProgress(internalPercent: number): void { + const range = this.stageProgressMap[this.currentStage]; + if (range) { + const [min, max] = range; + const clamped = Math.max(0, Math.min(100, internalPercent)); + this.currentProgress = Math.round(min + (max - min) * (clamped / 100)); + } + void this.flushLogs(); + } + + updateDetail(detail: string): void { + this.currentDetail = detail; + void this.flushLogs(); + } + + async flushLogs(force = false): Promise { + const now = Date.now(); + const shouldRun = force || now - this.lastLogFlush > 1000; + if (!shouldRun) return; + + if (this.isFlushing) { + this.hasPendingFlush = true; + return; + } + + this.isFlushing = true; + const performUpdate = async () => { + try { + this.lastLogFlush = Date.now(); + await prisma.execution.update({ + where: { id: this.executionId }, + data: { + logs: JSON.stringify(this.logs), + metadata: JSON.stringify({ + progress: this.currentProgress, + stage: this.currentStage, + detail: this.currentDetail, + }), + }, + }); + } catch (error) { + log.error("Failed to flush logs", { executionId: this.executionId }, wrapError(error)); + } + }; + + try { + await performUpdate(); + if (this.hasPendingFlush) { + this.hasPendingFlush = false; + await performUpdate(); + } + } finally { + this.isFlushing = false; + } + } +} diff --git a/src/services/backup/integrity-service.ts b/src/services/backup/integrity-service.ts index a1f252e7..25235f50 100644 --- a/src/services/backup/integrity-service.ts +++ b/src/services/backup/integrity-service.ts @@ -1,19 +1,15 @@ import prisma from "@/lib/prisma"; import { registry } from "@/lib/core/registry"; import { registerAdapters } from "@/lib/adapters"; -import { StorageAdapter, BackupMetadata } from "@/lib/core/interfaces"; +import { StorageAdapter } from "@/lib/core/interfaces"; import { resolveAdapterConfig } from "@/lib/adapters/config-resolver"; -import { getTempDir } from "@/lib/temp-dir"; -import { verifyFileChecksum } from "@/lib/crypto/checksum"; +import { verificationService } from "@/services/storage/verification-service"; import { logger } from "@/lib/logging/logger"; import { wrapError } from "@/lib/logging/errors"; -import path from "path"; -import fs from "fs"; -import crypto from "crypto"; +import { INTEGRITY_CHECK_STAGES } from "@/lib/core/logs"; const log = logger.child({ service: "IntegrityService" }); -// Ensure adapters are loaded registerAdapters(); export interface IntegrityCheckResult { @@ -30,17 +26,27 @@ export interface IntegrityCheckResult { }>; } -/** - * Service for verifying the integrity of backup files on storage. - * Iterates over all storage destinations, downloads each backup file, - * computes its SHA-256 checksum, and compares it against the stored metadata. - */ +export interface IntegrityProgressCallbacks { + onLog: (message: string, level?: "info" | "success" | "warning" | "error", details?: string) => void; + onStage: (stage: string) => void; + onFileProgress: (done: number, total: number, currentFile?: string) => void; +} + +interface IntegrityFilters { + skipPassed: boolean; + maxAgeDays: number; + maxFileSizeBytes: number; +} + +interface WorkItem { + storageConfigId: string; + destinationName: string; + remotePath: string; + fileName: string; +} + export class IntegrityService { - /** - * Runs an integrity check on all backup files across all storage destinations. - * Only checks files that have a checksum in their metadata sidecar. - */ - async runFullIntegrityCheck(): Promise { + async runFullIntegrityCheck(callbacks?: IntegrityProgressCallbacks): Promise { log.info("Starting full backup integrity check"); const result: IntegrityCheckResult = { @@ -52,21 +58,133 @@ export class IntegrityService { errors: [], }; - // Get all storage adapters - const storageConfigs = await prisma.adapterConfig.findMany({ - where: { type: "storage" }, + const [skipPassedSetting, maxAgeSetting, maxSizeSetting, scanModeSetting] = await Promise.all([ + prisma.systemSetting.findUnique({ where: { key: 'integrity.skipPassed' } }), + prisma.systemSetting.findUnique({ where: { key: 'integrity.maxAgeDays' } }), + prisma.systemSetting.findUnique({ where: { key: 'integrity.maxFileSizeMb' } }), + prisma.systemSetting.findUnique({ where: { key: 'integrity.scanMode' } }), + ]); + + const filters: IntegrityFilters = { + skipPassed: skipPassedSetting?.value === 'true', + maxAgeDays: parseInt(maxAgeSetting?.value ?? '0') || 0, + maxFileSizeBytes: (parseInt(maxSizeSetting?.value ?? '0') || 0) * 1024 * 1024, + }; + + const isJobsMode = !scanModeSetting || scanModeSetting.value !== 'destinations'; + + log.info("Integrity check config", { + mode: isJobsMode ? 'jobs' : 'destinations', + skipPassed: filters.skipPassed, + maxAgeDays: filters.maxAgeDays, + maxFileSizeMb: filters.maxFileSizeBytes / 1024 / 1024, }); - for (const storageConfig of storageConfigs) { + callbacks?.onStage(INTEGRITY_CHECK_STAGES.INITIALIZING); + + const filterParts: string[] = []; + if (filters.maxAgeDays > 0) filterParts.push(`max age ${filters.maxAgeDays} days`); + if (filters.maxFileSizeBytes > 0) filterParts.push(`max size ${filters.maxFileSizeBytes / 1024 / 1024} MB`); + if (filters.skipPassed) filterParts.push("skip already-passed files"); + callbacks?.onLog( + filterParts.length > 0 ? `Filters: ${filterParts.join(", ")}` : "No filters configured - checking all files" + ); + + callbacks?.onLog(`Scan mode: ${isJobsMode ? 'Jobs (job-linked files only)' : 'All destinations (full storage scan)'}`); + + // Pass 1: Collect work items according to scan mode + callbacks?.onStage(INTEGRITY_CHECK_STAGES.SCANNING); + + const allWorkItems: WorkItem[] = []; + + if (isJobsMode) { try { - await this.checkDestination(storageConfig, result); + const items = await this.gatherFilesFromJobs(filters, callbacks); + allWorkItems.push(...items); } catch (e: unknown) { - log.error( - "Failed to check storage destination", - { destination: storageConfig.name }, - wrapError(e) + log.error("Failed to gather files from jobs", {}, wrapError(e)); + callbacks?.onLog("Failed to scan job-linked files", "error"); + } + } else { + const storageConfigs = await prisma.adapterConfig.findMany({ + where: { type: "storage" }, + }); + + callbacks?.onLog(`Found ${storageConfigs.length} storage destination${storageConfigs.length !== 1 ? "s" : ""}`); + + for (const storageConfig of storageConfigs) { + try { + const items = await this.gatherFilesFromDestination(storageConfig, filters, callbacks); + allWorkItems.push(...items); + } catch (e: unknown) { + log.error("Failed to scan storage destination", { destination: storageConfig.name }, wrapError(e)); + callbacks?.onLog(`Failed to scan ${storageConfig.name}`, "error"); + } + } + } + + result.totalFiles = allWorkItems.length; + callbacks?.onLog( + `Total: ${allWorkItems.length} file${allWorkItems.length !== 1 ? "s" : ""} to verify`, + "info" + ); + + if (allWorkItems.length === 0) { + callbacks?.onLog("No files to verify - check filters or storage contents", "info"); + return result; + } + + // Pass 2: Verify all collected files + callbacks?.onStage(INTEGRITY_CHECK_STAGES.VERIFYING_CHECKSUMS); + + for (let i = 0; i < allWorkItems.length; i++) { + const item = allWorkItems[i]; + callbacks?.onFileProgress(i, allWorkItems.length, item.fileName); + + try { + const verifyResult = await verificationService.verifyFile( + item.storageConfigId, + item.remotePath, + "scheduled", + { skipIfPassed: filters.skipPassed } ); + + if (verifyResult.status === "passed") { + result.verified++; + result.passed++; + callbacks?.onLog(`${item.fileName}`, "success"); + } else if (verifyResult.status === "failed") { + result.verified++; + result.failed++; + result.errors.push({ + file: item.fileName, + destination: item.destinationName, + expected: verifyResult.expectedChecksum ?? "", + actual: verifyResult.actualChecksum ?? "", + }); + callbacks?.onLog( + `${item.fileName} - checksum mismatch`, + "error", + `Expected: ${verifyResult.expectedChecksum ?? "unknown"}\nActual: ${verifyResult.actualChecksum ?? "unknown"}` + ); + } else { + result.skipped++; + const skipReasons: Record = { + skipped: "already verified", + no_metadata: "no metadata file", + no_checksum: "no checksum stored", + download_error: "download failed", + }; + const reason = skipReasons[verifyResult.status] ?? verifyResult.status; + callbacks?.onLog(`${item.fileName} skipped (${reason})`, "info"); + } + } catch (e: unknown) { + log.error("Failed to verify file", { file: item.fileName, destination: item.destinationName }, wrapError(e)); + callbacks?.onLog(`Failed to verify ${item.fileName}`, "error"); + result.skipped++; } + + callbacks?.onFileProgress(i + 1, allWorkItems.length, item.fileName); } log.info("Integrity check completed", { @@ -80,186 +198,182 @@ export class IntegrityService { return result; } - private async checkDestination( + private async gatherFilesFromJobs( + filters: IntegrityFilters, + callbacks?: IntegrityProgressCallbacks + ): Promise { + const jobs = await prisma.job.findMany({ + where: { enabled: true }, + include: { + destinations: { + include: { config: true }, + }, + }, + }); + + callbacks?.onLog(`Found ${jobs.length} job${jobs.length !== 1 ? "s" : ""} to scan`); + + const eligibleJobs = jobs.filter((j) => !j.skipVerification); + const skippedJobs = jobs.filter((j) => j.skipVerification); + + for (const job of skippedJobs) { + callbacks?.onLog(`${job.name}: verification disabled - skipping`, "info"); + } + + const seen = new Set(); + const workItems: WorkItem[] = []; + + // Track per-job file counts for logging + const jobCounts = new Map(); + for (const job of eligibleJobs) jobCounts.set(job.name, 0); + + // Collect all unique destinations across eligible jobs, then list each once + const destJobMap = new Map(); + for (const job of eligibleJobs) { + for (const dest of job.destinations) { + const destMeta = dest.config.metadata ? JSON.parse(dest.config.metadata) : {}; + if (destMeta.skipVerification === true) continue; + const existing = destJobMap.get(dest.configId); + if (existing) { + existing.jobNames.push(job.name); + } else { + destJobMap.set(dest.configId, { dest, jobNames: [job.name] }); + } + } + } + + for (const { dest, jobNames } of destJobMap.values()) { + const adapter = registry.get(dest.config.adapterId) as StorageAdapter; + if (!adapter) { + callbacks?.onLog(`${dest.config.name}: adapter '${dest.config.adapterId}' not found`, "error"); + continue; + } + + let config: Awaited>; + try { + config = await resolveAdapterConfig(dest.config); + } catch (e: unknown) { + const msg = e instanceof Error ? e.message : String(e); + callbacks?.onLog(`${dest.config.name}: config resolution failed - ${msg}`, "error"); + continue; + } + + let allFiles: Awaited> = []; + try { + allFiles = await adapter.list(config, ""); + } catch (e: unknown) { + const msg = e instanceof Error ? e.message : String(e); + log.warn("Could not list destination", { destination: dest.config.name }, wrapError(e)); + callbacks?.onLog(`${dest.config.name}: listing failed - ${msg}`, "error"); + continue; + } + + // Keep only files that belong to one of the eligible jobs (path starts with jobName/) + const backupFiles = allFiles.filter( + (f) => !f.name.endsWith(".meta.json") && jobNames.some((n) => f.path.startsWith(n + "/")) + ); + + for (const file of backupFiles) { + if (filters.maxAgeDays > 0 && file.lastModified) { + const ageDays = (Date.now() - new Date(file.lastModified).getTime()) / 86_400_000; + if (ageDays > filters.maxAgeDays) continue; + } + + if (filters.maxFileSizeBytes > 0 && file.size > filters.maxFileSizeBytes) continue; + + const key = `${dest.configId}:${file.path}`; + if (seen.has(key)) continue; + seen.add(key); + + workItems.push({ + storageConfigId: dest.configId, + destinationName: dest.config.name, + remotePath: file.path, + fileName: file.name, + }); + + const matchingJob = jobNames.find((n) => file.path.startsWith(n + "/")); + if (matchingJob) jobCounts.set(matchingJob, (jobCounts.get(matchingJob) ?? 0) + 1); + } + } + + for (const [jobName, count] of jobCounts.entries()) { + callbacks?.onLog(`${jobName}: found ${count} file${count !== 1 ? "s" : ""} to verify`); + } + + return workItems; + } + + private async gatherFilesFromDestination( storageConfig: any, - result: IntegrityCheckResult - ) { + filters: IntegrityFilters, + callbacks?: IntegrityProgressCallbacks + ): Promise { + // Check if this destination has verification disabled + const meta = storageConfig.metadata ? JSON.parse(storageConfig.metadata) : {}; + if (meta.skipVerification === true) { + callbacks?.onLog(`${storageConfig.name}: verification disabled - skipping`, "info"); + return []; + } + + const workItems: WorkItem[] = []; const adapter = registry.get(storageConfig.adapterId) as StorageAdapter; if (!adapter) { - log.warn("Storage adapter not found", { - adapterId: storageConfig.adapterId, - }); - return; + log.warn("Storage adapter not found", { adapterId: storageConfig.adapterId }); + return []; } const config = await resolveAdapterConfig(storageConfig); - // List all top-level folders in storage (not just active jobs). - // This ensures backups from deleted jobs are also verified. - let folders: string[] = []; + // Collect all files from storage. Adapters return flat recursive listings where + // file.name is the basename and file.path is the full relative path โ€” use file.path + // as remotePath so downstream download/read calls resolve the correct location. + let allFiles: Awaited> = []; try { - const topLevel = await adapter.list(config, ""); - folders = topLevel - .filter((f) => f.name && !f.name.endsWith(".meta.json")) - .map((f) => f.name); + allFiles = await adapter.list(config, ""); } catch (e: unknown) { log.warn( "Could not list storage root, falling back to active jobs", { destination: storageConfig.name }, wrapError(e) ); - // Fallback: use known job names const jobs = await prisma.job.findMany({ where: { destinations: { some: { configId: storageConfig.id } } }, select: { name: true }, }); - folders = jobs.map((j) => j.name); - } - - for (const folder of folders) { - try { - // List files in folder - const files = await adapter.list(config, folder); - - // Filter to only backup files (exclude .meta.json) - const backupFiles = files.filter( - (f) => !f.name.endsWith(".meta.json") - ); - - for (const file of backupFiles) { - result.totalFiles++; - - try { - await this.verifyFile( - adapter, - config, - `${folder}/${file.name}`, - storageConfig.name, - result - ); - } catch (e: unknown) { - log.error( - "Failed to verify file", - { file: file.name, destination: storageConfig.name }, - wrapError(e) - ); - result.skipped++; - } + for (const job of jobs) { + try { + const files = await adapter.list(config, job.name); + allFiles.push(...files); + } catch { + // skip unreachable job folders } - } catch (e: unknown) { - log.warn( - "Failed to list files for folder", - { folder, destination: storageConfig.name }, - wrapError(e) - ); } } - } - private async verifyFile( - adapter: StorageAdapter, - config: any, - remotePath: string, - destinationName: string, - result: IntegrityCheckResult - ) { - const fileName = path.basename(remotePath); + const backupFiles = allFiles.filter((f) => !f.name.endsWith(".meta.json")); - // 1. Try to read metadata sidecar - let metadata: BackupMetadata | null = null; - - if (adapter.read) { - try { - const metaContent = await adapter.read( - config, - remotePath + ".meta.json" - ); - if (metaContent) { - metadata = JSON.parse(metaContent); - } - } catch { - // No metadata file, skip + for (const file of backupFiles) { + if (filters.maxAgeDays > 0 && file.lastModified) { + const ageDays = (Date.now() - new Date(file.lastModified).getTime()) / 86_400_000; + if (ageDays > filters.maxAgeDays) continue; } - } - if (!metadata) { - // Try download-based approach - const tempMetaPath = path.join( - getTempDir(), - `integrity_meta_${crypto.randomUUID()}.json` - ); - try { - const ok = await adapter.download( - config, - remotePath + ".meta.json", - tempMetaPath - ); - if (ok) { - const content = await fs.promises.readFile(tempMetaPath, "utf-8"); - metadata = JSON.parse(content); - } - } catch { - // No metadata - } finally { - await fs.promises.unlink(tempMetaPath).catch(() => {}); - } - } + if (filters.maxFileSizeBytes > 0 && file.size > filters.maxFileSizeBytes) continue; - if (!metadata?.checksum) { - log.debug("No checksum in metadata, skipping", { file: fileName }); - result.skipped++; - return; + workItems.push({ + storageConfigId: storageConfig.id, + destinationName: storageConfig.name, + remotePath: file.path, + fileName: file.name, + }); } - // 2. Download the backup file to temp - const tempFilePath = path.join( - getTempDir(), - `integrity_${crypto.randomUUID()}_${fileName}` + callbacks?.onLog( + `${storageConfig.name}: found ${workItems.length} file${workItems.length !== 1 ? "s" : ""} to verify` ); - try { - const downloadOk = await adapter.download( - config, - remotePath, - tempFilePath - ); - if (!downloadOk) { - log.warn("Could not download file for verification", { - file: fileName, - }); - result.skipped++; - return; - } - - // 3. Verify checksum - const verification = await verifyFileChecksum( - tempFilePath, - metadata.checksum - ); - result.verified++; - - if (verification.valid) { - log.debug("File integrity OK", { file: fileName }); - result.passed++; - } else { - log.error("INTEGRITY FAILURE", { - file: fileName, - destination: destinationName, - expected: verification.expected, - actual: verification.actual, - }); - result.failed++; - result.errors.push({ - file: fileName, - destination: destinationName, - expected: verification.expected, - actual: verification.actual, - }); - } - } finally { - // Always cleanup temp file - await fs.promises.unlink(tempFilePath).catch(() => {}); - } + return workItems; } } diff --git a/src/services/jobs/job-service.ts b/src/services/jobs/job-service.ts index 21f4d1af..8badd1eb 100644 --- a/src/services/jobs/job-service.ts +++ b/src/services/jobs/job-service.ts @@ -26,6 +26,7 @@ export interface CreateJobInput { notificationEvents?: string; namingTemplateId?: string | null; schedulePresetId?: string | null; + skipVerification?: boolean; } export interface UpdateJobInput { @@ -42,6 +43,7 @@ export interface UpdateJobInput { notificationEvents?: string; namingTemplateId?: string | null; schedulePresetId?: string | null; + skipVerification?: boolean; } const jobInclude = { @@ -84,7 +86,7 @@ export class JobService { } async createJob(input: CreateJobInput) { - const { name, schedule, sourceId, databases, destinations, notificationIds, enabled, encryptionProfileId, compression, pgCompression, notificationEvents } = input; + const { name, schedule, sourceId, databases, destinations, notificationIds, enabled, encryptionProfileId, compression, pgCompression, notificationEvents, skipVerification } = input; // Check name uniqueness const existingByName = await prisma.job.findFirst({ where: { name } }); @@ -105,6 +107,7 @@ export class JobService { compression: compression || "NONE", pgCompression: pgCompression ?? "", notificationEvents: notificationEvents || "ALWAYS", + skipVerification: skipVerification ?? false, notifications: { connect: notificationIds?.map((id) => ({ id })) || [] }, @@ -126,7 +129,7 @@ export class JobService { } async updateJob(id: string, input: UpdateJobInput) { - const { name, schedule, sourceId, databases, destinations, notificationIds, enabled, encryptionProfileId, compression, pgCompression, notificationEvents, namingTemplateId } = input; + const { name, schedule, sourceId, databases, destinations, notificationIds, enabled, encryptionProfileId, compression, pgCompression, notificationEvents, namingTemplateId, skipVerification } = input; // Check name uniqueness (excluding current job) if (name) { @@ -167,6 +170,7 @@ export class JobService { namingTemplateId: namingTemplateId !== undefined ? (namingTemplateId ?? null) : undefined, schedulePresetId: input.schedulePresetId !== undefined ? (input.schedulePresetId ?? null) : undefined, encryptionProfileId: encryptionProfileId === "" ? null : encryptionProfileId, + skipVerification: skipVerification !== undefined ? skipVerification : undefined, notifications: { set: [], connect: notificationIds?.map((id) => ({ id })) || [] diff --git a/src/services/storage/storage-service.ts b/src/services/storage/storage-service.ts index 904a9495..11d9ee03 100644 --- a/src/services/storage/storage-service.ts +++ b/src/services/storage/storage-service.ts @@ -31,8 +31,18 @@ export type RichFileInfo = FileInfo & { compression?: string; locked?: boolean; trigger?: { type: string; actor?: string }; + checksum?: string; + checksumMd5?: string; + verification?: { + verifiedAt: string; + passed: boolean; + trigger: 'manual' | 'post-upload' | 'scheduled'; + }; }; +// After this many hours a cached listing is considered stale and triggers background reconciliation. +const CACHE_STALENESS_HOURS = 2; + export class StorageService { async toggleLock(adapterConfigId: string, filePath: string) { const adapterConfig = await prisma.adapterConfig.findUnique({ @@ -44,12 +54,10 @@ export class StorageService { const adapter = registry.get(adapterConfig.adapterId) as StorageAdapter; const config = await resolveAdapterConfig(adapterConfig); - // Define paths const metaPath = filePath + ".meta.json"; let metadata: BackupMetadata; - // 1. Read existing metadata try { if (!adapter.read) throw new Error("Adapter does not support reading metadata"); const content = await adapter.read(config, metaPath); @@ -61,13 +69,8 @@ export class StorageService { throw new Error(`Could not read metadata for this backup: ${message}`); } - // 2. Toggle Lock metadata.locked = !metadata.locked; - // 3. Write back - // StorageAdapter interface usually only has 'upload' (from local file). - // We need 'write' (string content) or we create a temp file. - // Let's create a temp file. const tempPath = path.join(getTempDir(), `meta-${Date.now()}.json`); await fs.writeFile(tempPath, JSON.stringify(metadata, null, 2)); @@ -77,15 +80,14 @@ export class StorageService { await fs.unlink(tempPath).catch(() => {}); } + await this.updateStorageListCacheEntry(adapterConfigId, filePath, { locked: metadata.locked }); + return metadata.locked; } /** * Lists files from a specific storage adapter configuration. - * @param adapterConfigId The ID of the AdapterConfig in the database. - * @param subPath Optional subpath to list. - * @param typeFilter Optional filter for source type (e.g. "SYSTEM") */ async listFiles(adapterConfigId: string, subPath: string = "", _typeFilter?: string): Promise { const adapterConfig = await prisma.adapterConfig.findUnique({ @@ -105,7 +107,6 @@ export class StorageService { throw new Error(`Storage adapter implementation '${adapterConfig.adapterId}' not found in registry.`); } - // Resolve config (merges credential profile if present) let config: any; try { config = await resolveAdapterConfig(adapterConfig); @@ -116,10 +117,268 @@ export class StorageService { return await adapter.list(config, subPath); } + async invalidateStorageListCache(adapterConfigId: string): Promise { + await prisma.storageListCache.deleteMany({ where: { adapterConfigId } }); + } + + async appendStorageListCacheEntry(adapterConfigId: string, entry: RichFileInfo): Promise { + const cached = await prisma.storageListCache.findUnique({ where: { adapterConfigId } }); + if (!cached) return; + const files = JSON.parse(cached.filesJson) as RichFileInfo[]; + if (files.some(f => f.path === entry.path)) return; + files.push(entry); + await prisma.storageListCache.update({ + where: { adapterConfigId }, + data: { filesJson: JSON.stringify(files), cachedAt: new Date() }, + }); + } + + async removeStorageListCacheEntry(adapterConfigId: string, filePath: string): Promise { + const cached = await prisma.storageListCache.findUnique({ where: { adapterConfigId } }); + if (!cached) return; + const files = JSON.parse(cached.filesJson) as RichFileInfo[]; + const filtered = files.filter(f => f.path !== filePath); + if (filtered.length === files.length) return; + await prisma.storageListCache.update({ + where: { adapterConfigId }, + data: { filesJson: JSON.stringify(filtered), cachedAt: new Date() }, + }); + } + + async updateStorageListCacheEntry(adapterConfigId: string, filePath: string, updates: Partial): Promise { + const cached = await prisma.storageListCache.findUnique({ where: { adapterConfigId } }); + if (!cached) return; + const files = JSON.parse(cached.filesJson) as RichFileInfo[]; + const idx = files.findIndex(f => f.path === filePath); + if (idx === -1) return; + files[idx] = { ...files[idx], ...updates }; + await prisma.storageListCache.update({ + where: { adapterConfigId }, + data: { filesJson: JSON.stringify(files), cachedAt: new Date() }, + }); + } + + private applyTypeFilter(files: RichFileInfo[], typeFilter?: string): RichFileInfo[] { + if (typeFilter === "SYSTEM") return files.filter(f => f.sourceType === "SYSTEM"); + if (typeFilter === "BACKUP") return files.filter(f => f.sourceType !== "SYSTEM"); + return files; + } + + private enrichSingleFile( + file: FileInfo, + metadataMap: Map, + jobMap: Map, + executionMap: Map + ): RichFileInfo { + const sidecar = metadataMap.get(file.name); + let isEncrypted = file.name.endsWith('.enc'); + let encryptionProfileId: string | undefined = undefined; + let compression: string | undefined = undefined; + + if (sidecar) { + let count = 0; + let label = "Unknown"; + const isConfigBackup = sidecar.sourceType === "SYSTEM" || file.name.startsWith("config_backup_"); + + if (isConfigBackup) { + count = 1; + label = "System Config"; + } else { + count = typeof sidecar.databases === 'object' ? (sidecar.databases as any).count : (typeof sidecar.databases === 'number' ? sidecar.databases : 0); + label = count === 0 ? "Unknown" : (count === 1 ? "Single DB" : `${count} DBs`); + } + + if (sidecar.encryption?.enabled) isEncrypted = true; + encryptionProfileId = sidecar.encryption?.profileId; + compression = sidecar.compression; + + return { + ...file, + jobName: sidecar.jobName || (isConfigBackup ? "Config Backup" : undefined), + sourceName: sidecar.sourceName || (isConfigBackup ? "System" : undefined), + sourceType: sidecar.sourceType || (isConfigBackup ? "SYSTEM" : undefined), + engineVersion: sidecar.engineVersion, + engineEdition: sidecar.engineEdition, + dbInfo: { count, label }, + isEncrypted, + encryptionProfileId, + compression, + locked: sidecar.locked, + trigger: sidecar.trigger as { type: string; actor?: string } | undefined, + checksum: sidecar.checksum, + checksumMd5: sidecar.checksumMd5, + verification: sidecar.verification ? { + verifiedAt: sidecar.verification.verifiedAt, + passed: sidecar.verification.passed, + trigger: sidecar.verification.trigger, + } : undefined, + }; + } + + if (file.name.endsWith('.gz')) compression = 'GZIP'; + else if (file.name.endsWith('.br')) compression = 'BROTLI'; + + let potentialJobName = null; + const parts = file.path.split('/'); + if (parts.length > 2 && parts[0] === 'backups') { + potentialJobName = parts[1]; + } else if (parts.length > 1 && parts[0] !== 'backups') { + potentialJobName = parts[0]; + } else { + const match = file.name.match(/^(.+?)_\d{4}-\d{2}-\d{2}/); + if (match) potentialJobName = match[1]; + } + + const job = potentialJobName ? jobMap.get(potentialJobName) : null; + let dbInfo: { count: string | number; label: string } = { count: 'Unknown', label: '' }; + + const metaStr = executionMap.get(file.path); + if (metaStr) { + try { + const meta = JSON.parse(metaStr); + if (meta.label) { + dbInfo = { count: meta.count || '?', label: meta.label }; + } + if (meta.jobName) { + const realType = meta.adapterId || meta.sourceType; + return { + ...file, + jobName: meta.jobName, + sourceName: meta.sourceName, + sourceType: realType, + dbInfo, + isEncrypted, + encryptionProfileId, + compression + }; + } + } catch {} + } + + if (job) { + return { + ...file, + jobName: job.name, + sourceName: job.source.name, + sourceType: job.source.type, + dbInfo, + isEncrypted, + encryptionProfileId, + compression + }; + } + + const isConfigBackup = potentialJobName === "config-backups" || potentialJobName === "config_backup" || file.name.startsWith("config_backup_"); + + return { + ...file, + jobName: isConfigBackup ? "Config Backup" : (potentialJobName || 'Unknown'), + sourceName: isConfigBackup ? "System" : 'Unknown', + sourceType: isConfigBackup ? "SYSTEM" : 'unknown', + dbInfo: isConfigBackup ? { count: 1, label: "System Config" } : dbInfo, + isEncrypted, + encryptionProfileId, + compression + }; + } + + async reconcileStorageListCache(adapterConfigId: string): Promise { + const adapterConfig = await prisma.adapterConfig.findUnique({ where: { id: adapterConfigId } }); + if (!adapterConfig || adapterConfig.type !== "storage") return; + + const adapter = registry.get(adapterConfig.adapterId) as StorageAdapter; + if (!adapter) return; + + let config: any; + try { + config = await resolveAdapterConfig(adapterConfig); + } catch { return; } + + const allRemoteFiles = await adapter.list(config, ""); + const remoteBackups = allRemoteFiles.filter(f => !f.name.endsWith('.meta.json')); + const remoteMetaFiles = allRemoteFiles.filter(f => f.name.endsWith('.meta.json')); + const remotePathSet = new Set(remoteBackups.map(f => f.path)); + + const cached = await prisma.storageListCache.findUnique({ where: { adapterConfigId } }); + if (!cached) return; + + const cachedFiles = JSON.parse(cached.filesJson) as RichFileInfo[]; + const cachedPathSet = new Set(cachedFiles.map(f => f.path)); + + const removedPaths = new Set([...cachedPathSet].filter(p => !remotePathSet.has(p))); + const newFiles = remoteBackups.filter(f => !cachedPathSet.has(f.path)); + + if (removedPaths.size === 0 && newFiles.length === 0) { + await prisma.storageListCache.update({ + where: { adapterConfigId }, + data: { cachedAt: new Date() }, + }); + return; + } + + let updatedFiles = cachedFiles.filter(f => !removedPaths.has(f.path)); + + if (newFiles.length > 0) { + const metadataMap = new Map(); + if (adapter.read) { + const newFileNames = new Set(newFiles.map(f => f.name)); + const relevantMetaFiles = remoteMetaFiles.filter(mf => newFileNames.has(mf.name.slice(0, -10))); + await Promise.all(relevantMetaFiles.map(async (metaFile) => { + try { + const content = await adapter.read!(config, metaFile.path); + if (content) metadataMap.set(metaFile.name.slice(0, -10), JSON.parse(content) as BackupMetadata); + } catch { /* ignore */ } + })); + } + + const allJobs = await prisma.job.findMany({ include: { source: true } }); + const jobMap = new Map(); + allJobs.forEach(j => { + jobMap.set(j.name.replace(/[^a-z0-9]/gi, '_'), j); + jobMap.set(j.name, j); + }); + + const executions = await prisma.execution.findMany({ + where: { status: 'Success', path: { not: null } }, + select: { path: true, metadata: true } + }); + const executionMap = new Map(); + executions.forEach(ex => { + if (ex.path) { + executionMap.set(ex.path, ex.metadata); + if (ex.path.startsWith('/')) executionMap.set(ex.path.substring(1), ex.metadata); + else executionMap.set('/' + ex.path, ex.metadata); + } + }); + + const enrichedNew = newFiles.map(f => this.enrichSingleFile(f, metadataMap, jobMap, executionMap)); + updatedFiles = [...updatedFiles, ...enrichedNew]; + } + + await prisma.storageListCache.update({ + where: { adapterConfigId }, + data: { filesJson: JSON.stringify(updatedFiles), cachedAt: new Date() }, + }); + log.debug("Reconciled storage cache", { adapterConfigId, removed: removedPaths.size, added: newFiles.length }); + } + /** * Lists files and enriches them with metadata from sidecars and database history. + * Results are cached in SQLite; pass bypassCache=true to force a live re-fetch. + * Stale caches (> CACHE_STALENESS_HOURS) trigger a background reconciliation. */ - async listFilesWithMetadata(adapterConfigId: string, typeFilter?: string): Promise { + async listFilesWithMetadata(adapterConfigId: string, typeFilter?: string, bypassCache = false): Promise { + if (!bypassCache) { + const cached = await prisma.storageListCache.findUnique({ where: { adapterConfigId } }); + if (cached) { + const ageHours = (Date.now() - cached.cachedAt.getTime()) / 3_600_000; + if (ageHours > CACHE_STALENESS_HOURS) { + this.reconcileStorageListCache(adapterConfigId).catch(() => {}); + } + return this.applyTypeFilter(JSON.parse(cached.filesJson) as RichFileInfo[], typeFilter); + } + } + const adapterConfig = await prisma.adapterConfig.findUnique({ where: { id: adapterConfigId } }); @@ -144,18 +403,13 @@ export class StorageService { throw new Error(`Failed to decrypt configuration for ${adapterConfigId}: ${(e as Error).message}`); } - // TODO: Pass typeFilter to adapter.list if adapters supported optimized filtering. - // For now, we fetch all and filter in memory. const allFiles = await adapter.list(config, ""); - // Filter Backups vs Metadata const backups = allFiles.filter(f => !f.name.endsWith('.meta.json')); const metadataFiles = allFiles.filter(f => f.name.endsWith('.meta.json')); - // Load Sidecar Metadata const metadataMap = new Map(); if (adapter.read) { - // Parallel read const metaReads = metadataFiles.map(async (metaFile) => { try { const content = await adapter.read!(config, metaFile.path); @@ -171,7 +425,6 @@ export class StorageService { await Promise.all(metaReads); } - // Fetch jobs for fallback logic const allJobs = await prisma.job.findMany({ include: { source: true } }); @@ -183,7 +436,6 @@ export class StorageService { jobMap.set(j.name, j); }); - // Fetch executions for metadata const executions = await prisma.execution.findMany({ where: { status: 'Success', @@ -208,133 +460,17 @@ export class StorageService { } }); - const results = backups.map(file => { - // 1. Check Sidecar Metadata (Primary Source of Truth) - const sidecar = metadataMap.get(file.name); - let isEncrypted = file.name.endsWith('.enc'); - let encryptionProfileId: string | undefined = undefined; - let compression: string | undefined = undefined; - - if (sidecar) { - // Database Count from Metadata - let count = 0; - let label = "Unknown"; - - // Handle Config Backups - const isConfigBackup = sidecar.sourceType === "SYSTEM" || file.name.startsWith("config_backup_"); - - if (isConfigBackup) { - count = 1; // It's one config - label = "System Config"; - } else { - count = typeof sidecar.databases === 'object' ? (sidecar.databases as any).count : (typeof sidecar.databases === 'number' ? sidecar.databases : 0); - label = count === 0 ? "Unknown" : (count === 1 ? "Single DB" : `${count} DBs`); - } - - - if (sidecar.encryption?.enabled) isEncrypted = true; - encryptionProfileId = sidecar.encryption?.profileId; - compression = sidecar.compression; - - return { - ...file, - jobName: sidecar.jobName || (isConfigBackup ? "Config Backup" : undefined), - sourceName: sidecar.sourceName || (isConfigBackup ? "System" : undefined), - sourceType: sidecar.sourceType || (isConfigBackup ? "SYSTEM" : undefined), - engineVersion: sidecar.engineVersion, - engineEdition: sidecar.engineEdition, - dbInfo: { count, label }, - isEncrypted, - encryptionProfileId, - compression, - locked: sidecar.locked, - trigger: sidecar.trigger as { type: string; actor?: string } | undefined, - }; - } - - // Check for compression by extension if not found in sidecar - if (file.name.endsWith('.gz')) compression = 'GZIP'; - else if (file.name.endsWith('.br')) compression = 'BROTLI'; - - // 2. Fallback to Execution History / Regex Logic - let potentialJobName = null; - const parts = file.path.split('/'); - if (parts.length > 2 && parts[0] === 'backups') { - potentialJobName = parts[1]; - } else if (parts.length > 1 && parts[0] !== 'backups') { - potentialJobName = parts[0]; - } else { - const match = file.name.match(/^(.+?)_\d{4}-\d{2}-\d{2}/); - if (match) potentialJobName = match[1]; - } - - const job = potentialJobName ? jobMap.get(potentialJobName) : null; - - let dbInfo: { count: string | number; label: string } = { count: 'Unknown', label: '' }; - - // 1. Try to get metadata from Execution record - const metaStr = executionMap.get(file.path); - if (metaStr) { - try { - const meta = JSON.parse(metaStr); - if (meta.label) { - dbInfo = { count: meta.count || '?', label: meta.label }; - } - if (meta.jobName) { - const realType = meta.adapterId || meta.sourceType; - return { - ...file, - jobName: meta.jobName, - sourceName: meta.sourceName, - sourceType: realType, - dbInfo, - isEncrypted, - encryptionProfileId, - compression - } - } - } catch {} - } - - // 2. Existing Job Fallback - if (job) { - return { - ...file, - jobName: job.name, - sourceName: job.source.name, - sourceType: job.source.type, // e.g. 'database' - dbInfo, - isEncrypted, - encryptionProfileId, - compression - } - } - - // 3. Regex Fallback - const isConfigBackup = potentialJobName === "config-backups" || potentialJobName === "config_backup" || file.name.startsWith("config_backup_"); - - return { - ...file, - jobName: isConfigBackup ? "Config Backup" : (potentialJobName || 'Unknown'), - sourceName: isConfigBackup ? "System" : 'Unknown', - sourceType: isConfigBackup ? "SYSTEM" : 'unknown', - dbInfo: isConfigBackup ? { count: 1, label: "System Config" } : dbInfo, - isEncrypted, - encryptionProfileId, - compression - }; - }); + const results = backups.map(file => this.enrichSingleFile(file, metadataMap, jobMap, executionMap)); - if (typeFilter) { - if (typeFilter === "SYSTEM") { - return results.filter(f => (f as any).sourceType === "SYSTEM"); - } - if (typeFilter === "BACKUP") { - return results.filter(f => (f as any).sourceType !== "SYSTEM"); - } - } + // Persist to cache (full list without typeFilter applied) + const jsonStr = JSON.stringify(results); + prisma.storageListCache.upsert({ + where: { adapterConfigId }, + create: { adapterConfigId, filesJson: jsonStr }, + update: { filesJson: jsonStr, cachedAt: new Date() }, + }).catch(() => {}); - return results; + return this.applyTypeFilter(results, typeFilter); } /** @@ -365,11 +501,8 @@ export class StorageService { throw new Error(`Failed to decrypt configuration for ${adapterConfigId}: ${(e as Error).message}`); } - // Attempt to delete the main file const mainDelete = await adapter.delete(config, filePath); - // Also try to delete potential sidecar files (.meta.json) - // We don't fail the operation if this fails, but it's good practice to clean up. try { const metaPath = filePath + ".meta.json"; await adapter.delete(config, metaPath); @@ -377,6 +510,8 @@ export class StorageService { log.warn("Failed to delete associated metadata file", { filePath }, wrapError(e)); } + await this.removeStorageListCacheEntry(adapterConfigId, filePath); + return mainDelete; } @@ -408,9 +543,7 @@ export class StorageService { throw new Error(`Failed to decrypt configuration for ${adapterConfigId}: ${(e as Error).message}`); } - // 1. Decrypt if requested (Explicit Decryption) if (decrypt) { - // Download basic file first const success = await adapter.download(config, remotePath, localDestination); if (!success) return { success: false }; @@ -418,10 +551,8 @@ export class StorageService { const tempMetaPath = path.join(getTempDir(), "dlmeta_" + Date.now() + ".json"); try { - // Try to get metadata logic let meta: any = null; - // If adapter supports read, use it (faster) if (adapter.read) { try { const content = await adapter.read(config, metaRemotePath); @@ -429,7 +560,6 @@ export class StorageService { } catch {} } - // Fallback to download if read failed or not supported if (!meta) { const metaSuccess = await adapter.download(config, metaRemotePath, tempMetaPath).catch(() => false); if (metaSuccess) { @@ -439,18 +569,15 @@ export class StorageService { } } - // Determine Encryption Params (Support both Standard Object and Flat Config Backup formats) let encryptionParams: { profileId: string, iv: string, authTag: string } | null = null; if (meta && meta.encryption && typeof meta.encryption === 'object' && meta.encryption.enabled) { - // Standard Format encryptionParams = { profileId: meta.encryption.profileId, iv: meta.encryption.iv, authTag: meta.encryption.authTag }; } else if (meta && meta.encryptionProfileId && meta.iv && meta.authTag) { - // Legacy / Config Backup Flat Format encryptionParams = { profileId: meta.encryptionProfileId, iv: meta.iv, @@ -462,13 +589,10 @@ export class StorageService { let masterKey: Buffer; if (options?.rawKeyHex) { - // Caller-supplied raw key (from manual key resolution UI, passed via POST body) masterKey = Buffer.from(options.rawKeyHex, 'hex'); } else if (options?.profileIdOverride) { - // Caller explicitly chose a different vault profile masterKey = await getProfileMasterKey(options.profileIdOverride); } else { - // Auto-resolve: try the referenced profile first, then Smart Recovery const encryptionMeta = { enabled: true as const, profileId: encryptionParams.profileId, @@ -499,21 +623,18 @@ export class StorageService { const decryptStream = createDecryptionStream(masterKey, iv, authTag); const decryptedPath = localDestination + ".dec"; - // Decrypt to .dec file await pipeline( createReadStream(localDestination), decryptStream, createWriteStream(decryptedPath) ); - // Replace Original with Decrypted await fs.unlink(localDestination); await fs.rename(decryptedPath, localDestination); } return { success: true, isZip: false }; } catch (e: unknown) { - // Re-throw key-required errors unwrapped so the API route can detect them if (e instanceof Error && e.message.startsWith("ENCRYPTION_KEY_REQUIRED:")) { throw e; } @@ -522,7 +643,6 @@ export class StorageService { } } - // 2. Encrypted File Download (No Decryption) -> Bundle .meta.json if present if (remotePath.endsWith('.enc')) { const tempDir = path.dirname(localDestination); const baseName = path.basename(remotePath); @@ -531,11 +651,9 @@ export class StorageService { const metaRemotePath = remotePath + ".meta.json"; try { - // Download Main File to Temp const mainSuccess = await adapter.download(config, remotePath, tempMain); if (!mainSuccess) return { success: false }; - // Try Download Meta let metaFound = false; try { const metaSuccess = await adapter.download(config, metaRemotePath, tempMeta); @@ -543,7 +661,6 @@ export class StorageService { } catch {} if (metaFound) { - // Create ZIP try { const zip = new AdmZip(); zip.addLocalFile(tempMain, "", baseName); @@ -553,16 +670,13 @@ export class StorageService { return { success: true, isZip: true }; } catch (zipError) { log.error("Zip creation failed", { remotePath }, wrapError(zipError)); - // Fallback: Return original file only await fs.rename(tempMain, localDestination); return { success: true, isZip: false }; } finally { - // Cleanup temps try { await fs.unlink(tempMain); } catch {} try { await fs.unlink(tempMeta); } catch {} } } else { - // No metadata found, return original file await fs.rename(tempMain, localDestination); return { success: true, isZip: false }; } @@ -573,7 +687,6 @@ export class StorageService { } } - // 3. Normal File Download const success = await adapter.download(config, remotePath, localDestination); return { success, isZip: false }; } diff --git a/src/services/storage/verification-service.ts b/src/services/storage/verification-service.ts new file mode 100644 index 00000000..a7f49a1c --- /dev/null +++ b/src/services/storage/verification-service.ts @@ -0,0 +1,169 @@ +import prisma from "@/lib/prisma"; +import { registry } from "@/lib/core/registry"; +import { StorageAdapter, BackupMetadata } from "@/lib/core/interfaces"; +import { resolveAdapterConfig } from "@/lib/adapters/config-resolver"; +import { calculateFileChecksum } from "@/lib/crypto/checksum"; +import { getTempDir } from "@/lib/temp-dir"; +import { registerAdapters } from "@/lib/adapters"; +import { logger } from "@/lib/logging/logger"; +import { wrapError } from "@/lib/logging/errors"; +import path from "path"; +import fs from "fs/promises"; +import crypto from "crypto"; + +const log = logger.child({ service: "VerificationService" }); + +registerAdapters(); + +export type VerifyStatus = 'passed' | 'failed' | 'no_checksum' | 'no_metadata' | 'download_error' | 'skipped'; + +export interface FileVerificationResult { + status: VerifyStatus; + expectedChecksum?: string; + actualChecksum?: string; + verifiedAt: string; +} + +export class VerificationService { + async verifyFile( + adapterConfigId: string, + remotePath: string, + trigger: 'manual' | 'post-upload' | 'scheduled', + options?: { skipIfPassed?: boolean } + ): Promise { + const verifiedAt = new Date().toISOString(); + + const adapterConfig = await prisma.adapterConfig.findUnique({ where: { id: adapterConfigId } }); + if (!adapterConfig) throw new Error("Storage configuration not found"); + + const adapter = registry.get(adapterConfig.adapterId) as StorageAdapter; + if (!adapter) throw new Error(`Adapter '${adapterConfig.adapterId}' not found`); + + const config = await resolveAdapterConfig(adapterConfig); + + // 1. Read .meta.json sidecar + const metaPath = remotePath + ".meta.json"; + let metadata: BackupMetadata | null = null; + + try { + if (adapter.read) { + const content = await adapter.read(config, metaPath); + if (content) metadata = JSON.parse(content) as BackupMetadata; + } + + if (!metadata) { + const tempMeta = path.join(getTempDir(), `verify_meta_${crypto.randomUUID()}.json`); + try { + const ok = await adapter.download(config, metaPath, tempMeta); + if (ok) { + const content = await fs.readFile(tempMeta, 'utf-8'); + metadata = JSON.parse(content) as BackupMetadata; + } + } finally { + await fs.unlink(tempMeta).catch(() => {}); + } + } + } catch (e: unknown) { + log.warn("Could not read metadata for verification", { remotePath }, wrapError(e)); + } + + if (!metadata) { + return { status: 'no_metadata', verifiedAt }; + } + + if (!metadata.checksum && !metadata.checksumMd5) { + return { status: 'no_checksum', verifiedAt }; + } + + if (options?.skipIfPassed && metadata.verification?.passed === true) { + return { status: 'skipped', verifiedAt }; + } + + // 2. Try native adapter verification (no download needed for S3, local, GDrive, OneDrive) + if (adapter.verifyChecksum) { + try { + const nativeResult = await adapter.verifyChecksum(config, remotePath, { + sha256: metadata.checksum, + md5: metadata.checksumMd5, + }); + + if (nativeResult !== 'unsupported') { + const passed = nativeResult === 'passed'; + await this.writeVerificationResult(adapterConfigId, adapter, config, metaPath, metadata, { + verifiedAt, + passed, + trigger, + }); + return { status: nativeResult, expectedChecksum: metadata.checksum, verifiedAt }; + } + } catch (e: unknown) { + log.warn("Native checksum verification error, falling back to download", { remotePath }, wrapError(e)); + } + } + + // 3. Fallback: download + compute SHA-256 + if (!metadata.checksum) { + return { status: 'no_checksum', verifiedAt }; + } + + const tempFile = path.join(getTempDir(), `verify_${crypto.randomUUID()}_${path.basename(remotePath)}`); + try { + const downloadOk = await adapter.download(config, remotePath, tempFile); + if (!downloadOk) { + return { status: 'download_error', verifiedAt }; + } + + const actual = await calculateFileChecksum(tempFile); + const passed = actual === metadata.checksum; + + await this.writeVerificationResult(adapterConfigId, adapter, config, metaPath, metadata, { + verifiedAt, + passed, + trigger, + actualChecksum: passed ? undefined : actual, + }); + + return { + status: passed ? 'passed' : 'failed', + expectedChecksum: metadata.checksum, + actualChecksum: passed ? undefined : actual, + verifiedAt, + }; + } catch (e: unknown) { + log.error("Download-based verification failed", { remotePath }, wrapError(e)); + return { status: 'download_error', verifiedAt }; + } finally { + await fs.unlink(tempFile).catch(() => {}); + } + } + + private async writeVerificationResult( + adapterConfigId: string, + adapter: StorageAdapter, + config: any, + metaPath: string, + metadata: BackupMetadata, + verification: NonNullable + ) { + metadata.verification = verification; + const tempPath = path.join(getTempDir(), `meta-verify-${Date.now()}.json`); + await fs.writeFile(tempPath, JSON.stringify(metadata, null, 2)); + try { + await adapter.upload(config, tempPath, metaPath); + } finally { + await fs.unlink(tempPath).catch(() => {}); + } + const backupPath = metaPath.replace(/\.meta\.json$/, ""); + import("@/services/storage/storage-service").then(({ storageService }) => { + storageService.updateStorageListCacheEntry(adapterConfigId, backupPath, { + verification: { + verifiedAt: verification.verifiedAt, + passed: verification.passed, + trigger: verification.trigger, + }, + }).catch(() => {}); + }); + } +} + +export const verificationService = new VerificationService(); diff --git a/src/services/system/db-version-service.ts b/src/services/system/db-version-service.ts index 7085790b..0389ebb0 100644 --- a/src/services/system/db-version-service.ts +++ b/src/services/system/db-version-service.ts @@ -1,6 +1,7 @@ import prisma from "@/lib/prisma"; import { logger } from "@/lib/logging/logger"; import { wrapError } from "@/lib/logging/errors"; +import { compareVersions } from "@/lib/utils"; const log = logger.child({ service: "DbVersionService" }); @@ -11,6 +12,8 @@ export interface RecordVersionResult { previousVersion: string | null; /** The version stored as `newVersion` on the latest entry. */ newVersion: string; + /** True when the new version is numerically lower than the previous version. */ + isDowngrade: boolean; } /** @@ -38,7 +41,7 @@ export async function recordVersionIfChanged( ): Promise { const normalized = normalize(version); if (!normalized) { - return { changed: false, previousVersion: null, newVersion: version }; + return { changed: false, previousVersion: null, newVersion: version, isDowngrade: false }; } const latest = await prisma.dbVersionHistory.findFirst({ @@ -52,9 +55,11 @@ export async function recordVersionIfChanged( // No change when version (and edition, if present) match the last entry. if (latest && previousVersion === normalized && previousEdition === normalizedEdition) { - return { changed: false, previousVersion, newVersion: normalized }; + return { changed: false, previousVersion, newVersion: normalized, isDowngrade: false }; } + const isDowngrade = previousVersion !== null && compareVersions(previousVersion, normalized) > 0; + try { await prisma.dbVersionHistory.create({ data: { @@ -68,13 +73,14 @@ export async function recordVersionIfChanged( adapterConfigId, previousVersion, newVersion: normalized, + isDowngrade, }); } catch (e: unknown) { log.error("Failed to record version history", { adapterConfigId }, wrapError(e)); - return { changed: false, previousVersion, newVersion: normalized }; + return { changed: false, previousVersion, newVersion: normalized, isDowngrade: false }; } - return { changed: true, previousVersion, newVersion: normalized }; + return { changed: true, previousVersion, newVersion: normalized, isDowngrade }; } /** diff --git a/src/services/system/system-task-service.ts b/src/services/system/system-task-service.ts index fa8afa5b..f5c5936b 100644 --- a/src/services/system/system-task-service.ts +++ b/src/services/system/system-task-service.ts @@ -41,7 +41,8 @@ export const SYSTEM_TASKS = { SYNC_PERMISSIONS: "system.sync_permissions", CONFIG_BACKUP: "system.config_backup", INTEGRITY_CHECK: "system.integrity_check", - REFRESH_STORAGE_STATS: "system.refresh_storage_stats" + REFRESH_STORAGE_STATS: "system.refresh_storage_stats", + WARMUP_STORAGE_CACHE: "system.warmup_storage_cache" }; export const DEFAULT_TASK_CONFIG = { @@ -100,6 +101,13 @@ export const DEFAULT_TASK_CONFIG = { enabled: true, label: "Refresh Storage Statistics", description: "Queries all storage destinations to update file counts and total sizes displayed on the dashboard. Runs automatically after each backup." + }, + [SYSTEM_TASKS.WARMUP_STORAGE_CACHE]: { + interval: "0 * * * *", // Every hour + runOnStartup: true, + enabled: true, + label: "Pre-warm Storage Cache", + description: "Keeps the Storage Explorer cache fresh. Reconciles existing caches against remote storage to detect external changes, and pre-populates the cache for adapters not yet loaded." } }; @@ -214,7 +222,7 @@ export class SystemTaskService { }); } - async runTask(taskId: string) { + async runTask(taskId: string, triggerType?: "Manual" | "Scheduler", triggerLabel?: string): Promise { log.info("Running system task", { taskId }); await this.setTaskLastRunAt(taskId); @@ -242,20 +250,57 @@ export class SystemTaskService { } case SYSTEM_TASKS.INTEGRITY_CHECK: { const { integrityService } = await import("@/services/backup/integrity-service"); - const result = await integrityService.runFullIntegrityCheck(); - log.info("Integrity check results", { - total: result.totalFiles, - passed: result.passed, - failed: result.failed, - skipped: result.skipped - }); - break; + const { SystemTaskRunner } = await import("@/lib/runner/system-task-runner"); + const { INTEGRITY_CHECK_STAGES } = await import("@/lib/core/logs"); + const { getErrorMessage } = await import("@/lib/logging/errors"); + + const runner = await SystemTaskRunner.create( + "IntegrityCheck", + triggerType ?? "Scheduler", + triggerLabel ?? "Scheduler" + ); + + // Run async without blocking so callers receive the executionId immediately. + (async () => { + try { + await runner.start(); + const result = await integrityService.runFullIntegrityCheck({ + onLog: (msg, level, details) => runner.logEntry(msg, level ?? "info", "general", details), + onStage: (stage) => runner.setStage(stage), + onFileProgress: (done, total) => { + if (total > 0) runner.updateStageProgress((done / total) * 100); + }, + }); + runner.setStage(INTEGRITY_CHECK_STAGES.COMPLETED); + runner.logEntry( + `${result.passed} passed, ${result.failed} failed, ${result.skipped} skipped of ${result.totalFiles} total`, + result.failed > 0 ? "warning" : "success" + ); + await runner.finish("Success"); + log.info("Integrity check completed", { + total: result.totalFiles, + passed: result.passed, + failed: result.failed, + skipped: result.skipped, + }); + } catch (e: unknown) { + runner.logEntry(getErrorMessage(e), "error"); + runner.setStage(INTEGRITY_CHECK_STAGES.FAILED); + await runner.finish("Failed"); + log.error("Integrity check failed", {}, wrapError(e)); + } + })(); + + return runner.id; } case SYSTEM_TASKS.REFRESH_STORAGE_STATS: { const { refreshStorageStatsCache } = await import("@/services/dashboard-service"); await refreshStorageStatsCache(); break; } + case SYSTEM_TASKS.WARMUP_STORAGE_CACHE: + await this.runWarmupStorageCache(); + break; default: log.warn("Unknown system task", { taskId }); } @@ -490,6 +535,7 @@ export class SystemTaskService { newVersion: change.newVersion, edition, timestamp: new Date().toISOString(), + isDowngrade: change.isDowngrade, }, }); } @@ -516,6 +562,31 @@ export class SystemTaskService { } } + private async runWarmupStorageCache() { + const adapters = await prisma.adapterConfig.findMany({ + where: { type: "storage" }, + select: { id: true, name: true }, + }); + log.info("Pre-warming storage cache", { count: adapters.length }); + const { storageService } = await import("@/services/storage/storage-service"); + for (const adapter of adapters) { + try { + // If a cache row already exists: reconcile against remote to detect external changes. + // If no cache row: do a full fetch to populate it. + const cached = await prisma.storageListCache.findUnique({ where: { adapterConfigId: adapter.id } }); + if (cached) { + await storageService.reconcileStorageListCache(adapter.id); + log.debug("Reconciled storage cache", { adapterId: adapter.id, name: adapter.name }); + } else { + await storageService.listFilesWithMetadata(adapter.id); + log.debug("Warmed storage cache", { adapterId: adapter.id, name: adapter.name }); + } + } catch (e: unknown) { + log.warn("Failed to warm/reconcile cache for adapter", { adapterId: adapter.id, name: adapter.name }, wrapError(e)); + } + } + } + private async runSyncPermissions() { try { log.debug("Syncing permissions for SuperAdmin group"); diff --git a/tests/unit/lib/logs/format.test.ts b/tests/unit/lib/logs/format.test.ts new file mode 100644 index 00000000..423b7af9 --- /dev/null +++ b/tests/unit/lib/logs/format.test.ts @@ -0,0 +1,161 @@ +import { describe, it, expect } from 'vitest'; +import { formatLogsAsText, generateLogFilename, type ExportMeta } from '@/lib/logs/format'; +import type { LogEntry } from '@/lib/core/logs'; + +function makeEntry(msg: string, overrides: Partial = {}): LogEntry { + return { + timestamp: '2024-01-01T10:00:00.000Z', + level: 'info', + type: 'general', + message: msg, + ...overrides, + }; +} + +const baseMeta: ExportMeta = { + jobName: 'Test Job', + type: 'Backup', + status: 'Success', + startedAt: '2024-01-01T10:00:00.000Z', +}; + +describe('generateLogFilename', () => { + it('generates filename with slugified job name and UTC timestamp', () => { + const name = generateLogFilename('My Backup Job', '2024-03-15T14:30:00.000Z'); + expect(name).toBe('dbackup-my-backup-job-2024-03-15-14-30.log'); + }); + + it('replaces special characters with hyphens in slug', () => { + const name = generateLogFilename('Job/With:Special*Chars', '2024-01-01T00:00:00.000Z'); + expect(name).toMatch(/^dbackup-job-with-special-chars-/); + }); + + it('lowercases the job name', () => { + const name = generateLogFilename('UPPERCASE JOB', '2024-01-01T00:00:00.000Z'); + expect(name).toMatch(/^dbackup-uppercase-job-/); + }); + + it('truncates slug at 40 characters', () => { + const longName = 'This is a very long job name that exceeds the forty character limit by far'; + const name = generateLogFilename(longName, '2024-01-01T00:00:00.000Z'); + const slug = name + .replace(/^dbackup-/, '') + .replace(/-\d{4}-\d{2}-\d{2}-\d{2}-\d{2}\.log$/, ''); + expect(slug.length).toBeLessThanOrEqual(40); + }); + + it('returns fallback filename for invalid date', () => { + const name = generateLogFilename('My Job', 'not-a-date'); + expect(name).toBe('dbackup-my-job.log'); + }); + + it('uses UTC hours and minutes', () => { + const name = generateLogFilename('Job', '2024-06-01T23:45:00.000Z'); + expect(name).toContain('23-45'); + }); + + it('pads single-digit hours and minutes with leading zero', () => { + const name = generateLogFilename('Job', '2024-01-01T08:05:00.000Z'); + expect(name).toContain('08-05'); + }); +}); + +describe('formatLogsAsText', () => { + it('includes job name, type, and status in header', () => { + const output = formatLogsAsText([], baseMeta); + expect(output).toContain('Job: Test Job'); + expect(output).toContain('Type: Backup'); + expect(output).toContain('Status: Success'); + }); + + it('includes started timestamp in header', () => { + const output = formatLogsAsText([], baseMeta); + expect(output).toContain('Started:'); + expect(output).toContain('2024-01-01'); + }); + + it('includes ended timestamp when provided', () => { + const meta = { ...baseMeta, endedAt: '2024-01-01T10:05:00.000Z' }; + const output = formatLogsAsText([], meta); + expect(output).toContain('Ended:'); + }); + + it('omits ended line when endedAt is not provided', () => { + const output = formatLogsAsText([], baseMeta); + expect(output).not.toContain('Ended:'); + }); + + it('includes trigger type when provided', () => { + const meta = { ...baseMeta, triggerType: 'Scheduler' }; + const output = formatLogsAsText([], meta); + expect(output).toContain('Trigger: Scheduler'); + }); + + it('omits trigger line when triggerType is absent', () => { + const output = formatLogsAsText([], baseMeta); + expect(output).not.toContain('Trigger:'); + }); + + it('groups log entries by stage', () => { + const logs = [ + makeEntry('Starting up', { stage: 'Initializing' }), + makeEntry('Pushing file', { stage: 'Uploading' }), + ]; + const output = formatLogsAsText(logs, baseMeta); + expect(output).toContain('[STAGE: Initializing]'); + expect(output).toContain('[STAGE: Uploading]'); + }); + + it('includes log messages within their stage section', () => { + const logs = [makeEntry('Backup completed', { stage: 'Done' })]; + const output = formatLogsAsText(logs, baseMeta); + expect(output).toContain('Backup completed'); + }); + + it('appends duration suffix for entries with durationMs >= 1000', () => { + const logs = [makeEntry('Stage done', { stage: 'Done', durationMs: 2500 })]; + const output = formatLogsAsText(logs, baseMeta); + expect(output).toContain('[2.5s]'); + }); + + it('appends ms for durations under 1 second', () => { + const logs = [makeEntry('Fast op', { stage: 'Done', durationMs: 150 })]; + const output = formatLogsAsText(logs, baseMeta); + expect(output).toContain('[150ms]'); + }); + + it('renders details block when present', () => { + const logs = [makeEntry('Output', { stage: 'Done', details: 'line1\nline2' })]; + const output = formatLogsAsText(logs, baseMeta); + expect(output).toContain('[DETAILS]'); + expect(output).toContain('line1'); + expect(output).toContain('line2'); + }); + + it('renders UTC timestamps in HH:MM:SS format', () => { + const logs = [makeEntry('msg', { timestamp: '2024-01-15T14:30:45.000Z', stage: 'S' })]; + const output = formatLogsAsText(logs, baseMeta); + expect(output).toContain('14:30:45'); + }); + + it('assigns entries without stage to "General" group', () => { + const logs = [makeEntry('no stage', { stage: undefined })]; + const output = formatLogsAsText(logs, baseMeta); + expect(output).toContain('[STAGE: General]'); + }); + + it('includes stage duration in header when multiple entries share the same stage', () => { + const logs = [ + makeEntry('start', { timestamp: '2024-01-01T10:00:00.000Z', stage: 'Upload' }), + makeEntry('end', { timestamp: '2024-01-01T10:00:05.000Z', stage: 'Upload' }), + ]; + const output = formatLogsAsText(logs, baseMeta); + expect(output).toContain('[STAGE: Upload]'); + expect(output).toMatch(/\(5\.0s\)/); + }); + + it('includes the DBackup header line', () => { + const output = formatLogsAsText([], baseMeta); + expect(output).toContain('DBackup Execution Log'); + }); +}); diff --git a/tests/unit/lib/logs/sanitize.test.ts b/tests/unit/lib/logs/sanitize.test.ts new file mode 100644 index 00000000..7058b6ba --- /dev/null +++ b/tests/unit/lib/logs/sanitize.test.ts @@ -0,0 +1,115 @@ +import { describe, it, expect } from 'vitest'; +import { sanitizeLogs } from '@/lib/logs/sanitize'; +import type { LogEntry } from '@/lib/core/logs'; + +function makeEntry(overrides: Partial = {}): LogEntry { + return { + timestamp: '2024-01-01T00:00:00.000Z', + level: 'info', + type: 'general', + message: 'test message', + ...overrides, + }; +} + +describe('sanitizeLogs', () => { + it('redacts IPv4 addresses in message', () => { + const result = sanitizeLogs([makeEntry({ message: 'Connected to 192.168.1.100 successfully' })]); + expect(result[0].message).not.toContain('192.168.1.100'); + expect(result[0].message).toContain('[IP REDACTED]'); + }); + + it('redacts IPv6 addresses in message', () => { + const result = sanitizeLogs([makeEntry({ message: 'Server 2001:db8::1 timed out' })]); + expect(result[0].message).toContain('[IP REDACTED]'); + expect(result[0].message).not.toContain('2001:db8::1'); + }); + + it('redacts mongodb connection string credentials', () => { + const result = sanitizeLogs([makeEntry({ message: 'mongodb://admin:s3cr3t@host:27017/db' })]); + expect(result[0].message).not.toContain('s3cr3t'); + expect(result[0].message).toContain('[CREDENTIALS REDACTED]'); + }); + + it('redacts postgresql connection string credentials', () => { + const result = sanitizeLogs([makeEntry({ message: 'Connecting via postgresql://user:pass@host/mydb' })]); + expect(result[0].message).not.toContain('pass'); + expect(result[0].message).toContain('[CREDENTIALS REDACTED]'); + }); + + it('redacts mysql connection string credentials', () => { + const result = sanitizeLogs([makeEntry({ message: 'mysql://root:topsecret@db/schema' })]); + expect(result[0].message).not.toContain('topsecret'); + expect(result[0].message).toContain('[CREDENTIALS REDACTED]'); + }); + + it('redacts redis connection string credentials', () => { + const result = sanitizeLogs([makeEntry({ message: 'redis://user:pass@localhost:6379' })]); + expect(result[0].message).not.toContain('pass'); + expect(result[0].message).toContain('[CREDENTIALS REDACTED]'); + }); + + it('redacts credentials in details field', () => { + const result = sanitizeLogs([makeEntry({ details: 'sftp://user:secret@host/path' })]); + expect(result[0].details).not.toContain('secret'); + expect(result[0].details).toContain('[CREDENTIALS REDACTED]'); + }); + + it('redacts IPv4 addresses in details field', () => { + const result = sanitizeLogs([makeEntry({ details: 'Timeout on server 172.16.0.5' })]); + expect(result[0].details).not.toContain('172.16.0.5'); + expect(result[0].details).toContain('[IP REDACTED]'); + }); + + it('redacts SENSITIVE_KEYS in context', () => { + const result = sanitizeLogs([makeEntry({ context: { password: 'my-secret', jobId: 'job-1' } })]); + expect(result[0].context!.password).toBe('[REDACTED]'); + expect(result[0].context!.jobId).toBe('job-1'); + }); + + it('redacts apiKey in context', () => { + const result = sanitizeLogs([makeEntry({ context: { apiKey: 'sk-secret-key' } })]); + expect(result[0].context!.apiKey).toBe('[REDACTED]'); + }); + + it('redacts IP addresses inside non-sensitive context string values', () => { + const result = sanitizeLogs([makeEntry({ context: { info: 'server at 10.10.10.10 responded' } })]); + expect(result[0].context!.info).toContain('[IP REDACTED]'); + expect(result[0].context!.info).not.toContain('10.10.10.10'); + }); + + it('redacts nested sensitive keys in context objects', () => { + const result = sanitizeLogs([makeEntry({ context: { db: { password: 'nested-pass', port: 5432 } } })]); + expect(result[0].context!.db.password).toBe('[REDACTED]'); + expect(result[0].context!.db.port).toBe(5432); + }); + + it('preserves non-string non-object context values unchanged', () => { + const result = sanitizeLogs([makeEntry({ context: { count: 42, active: true } })]); + expect(result[0].context!.count).toBe(42); + expect(result[0].context!.active).toBe(true); + }); + + it('leaves entries without details or context unchanged', () => { + const result = sanitizeLogs([makeEntry({ details: undefined, context: undefined })]); + expect(result[0].details).toBeUndefined(); + expect(result[0].context).toBeUndefined(); + }); + + it('leaves clean messages without sensitive content unchanged', () => { + const result = sanitizeLogs([makeEntry({ message: 'Backup completed successfully' })]); + expect(result[0].message).toBe('Backup completed successfully'); + }); + + it('processes multiple entries independently', () => { + const logs = [ + makeEntry({ message: 'Server: 10.0.0.1 responded' }), + makeEntry({ message: 'No sensitive data here' }), + makeEntry({ message: 'mongodb://user:pass@host/db' }), + ]; + const result = sanitizeLogs(logs); + expect(result[0].message).toContain('[IP REDACTED]'); + expect(result[1].message).toBe('No sensitive data here'); + expect(result[2].message).toContain('[CREDENTIALS REDACTED]'); + }); +}); diff --git a/tests/unit/lib/notifications/templates.test.ts b/tests/unit/lib/notifications/templates.test.ts index 1817856f..38d172f2 100644 --- a/tests/unit/lib/notifications/templates.test.ts +++ b/tests/unit/lib/notifications/templates.test.ts @@ -655,16 +655,17 @@ describe("Notification Templates", () => { newVersion: "15.0.4360.2", edition: "Enterprise Edition", timestamp: "2026-05-31T10:00:00Z", + isDowngrade: false, }, }); - expect(payload.title).toBe("Database Version Changed: prod-mssql"); + expect(payload.title).toBe("Database Version Upgraded: prod-mssql"); expect(payload.message).toContain("prod-mssql"); expect(payload.message).toContain("15.0.4280.7"); expect(payload.message).toContain("15.0.4360.2"); expect(payload.success).toBe(true); expect(payload.color).toBe("#3b82f6"); - expect(payload.badge).toBe("Version"); + expect(payload.badge).toBe("Upgrade"); const fieldNames = payload.fields?.map((f) => f.name) ?? []; expect(fieldNames).toEqual( @@ -689,6 +690,7 @@ describe("Notification Templates", () => { previousVersion: null, newVersion: "8.0.31", timestamp: "2026-05-31T10:00:00Z", + isDowngrade: false, }, }); diff --git a/tests/unit/lib/runner/system-task-runner.test.ts b/tests/unit/lib/runner/system-task-runner.test.ts new file mode 100644 index 00000000..5809190e --- /dev/null +++ b/tests/unit/lib/runner/system-task-runner.test.ts @@ -0,0 +1,278 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { prismaMock } from '@/lib/testing/prisma-mock'; +import { SystemTaskRunner } from '@/lib/runner/system-task-runner'; + +vi.mock('@/lib/logging/logger', () => ({ + logger: { + child: vi.fn().mockReturnValue({ + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + debug: vi.fn(), + }), + }, +})); + +vi.mock('@/lib/logging/errors', () => ({ + wrapError: vi.fn((e) => e), +})); + +const STAGE_MAP: Record = { + Initializing: [0, 10], + Running: [10, 90], + Completed: [90, 100], +}; + +async function makeRunner(): Promise { + prismaMock.execution.create.mockResolvedValue({ id: 'exec-1' } as any); + return SystemTaskRunner.create('IntegrityCheck', 'Manual', 'Test trigger', STAGE_MAP); +} + +describe('SystemTaskRunner', () => { + beforeEach(() => { + vi.clearAllMocks(); + prismaMock.execution.update.mockResolvedValue({} as any); + prismaMock.execution.updateMany.mockResolvedValue({ count: 1 }); + }); + + describe('create()', () => { + it('creates an execution record with status Pending', async () => { + prismaMock.execution.create.mockResolvedValue({ id: 'exec-1' } as any); + + await SystemTaskRunner.create('IntegrityCheck'); + + expect(prismaMock.execution.create).toHaveBeenCalledWith( + expect.objectContaining({ + data: expect.objectContaining({ + status: 'Pending', + type: 'IntegrityCheck', + }), + }) + ); + }); + + it('stores the initial log entry in the execution record', async () => { + prismaMock.execution.create.mockResolvedValue({ id: 'exec-1' } as any); + + await SystemTaskRunner.create('IntegrityCheck'); + + const call = prismaMock.execution.create.mock.calls[0][0]; + const logs = JSON.parse(call.data.logs); + expect(logs).toHaveLength(1); + expect(logs[0].message).toBe('Task queued'); + }); + + it('passes triggerType and triggerLabel when provided', async () => { + prismaMock.execution.create.mockResolvedValue({ id: 'exec-1' } as any); + + await SystemTaskRunner.create('IntegrityCheck', 'Scheduler', 'daily-job'); + + expect(prismaMock.execution.create).toHaveBeenCalledWith( + expect.objectContaining({ + data: expect.objectContaining({ + triggerType: 'Scheduler', + triggerLabel: 'daily-job', + }), + }) + ); + }); + + it('exposes the execution id via the id getter', async () => { + prismaMock.execution.create.mockResolvedValue({ id: 'exec-abc' } as any); + const runner = await SystemTaskRunner.create('IntegrityCheck'); + expect(runner.id).toBe('exec-abc'); + }); + }); + + describe('start()', () => { + it('transitions execution status to Running', async () => { + const runner = await makeRunner(); + prismaMock.execution.updateMany.mockResolvedValue({ count: 1 }); + + await runner.start(); + + expect(prismaMock.execution.updateMany).toHaveBeenCalledWith( + expect.objectContaining({ + where: { id: 'exec-1', status: 'Pending' }, + data: expect.objectContaining({ status: 'Running' }), + }) + ); + }); + + it('throws when execution has already been claimed (concurrent call)', async () => { + const runner = await makeRunner(); + prismaMock.execution.updateMany.mockResolvedValue({ count: 0 }); + + await expect(runner.start()).rejects.toThrow('Execution already claimed by a concurrent call'); + }); + }); + + describe('finish()', () => { + it('updates execution status to Success', async () => { + const runner = await makeRunner(); + + await runner.finish('Success'); + + expect(prismaMock.execution.update).toHaveBeenCalledWith( + expect.objectContaining({ + where: { id: 'exec-1' }, + data: expect.objectContaining({ status: 'Success' }), + }) + ); + }); + + it('updates execution status to Failed', async () => { + const runner = await makeRunner(); + + await runner.finish('Failed'); + + expect(prismaMock.execution.update).toHaveBeenCalledWith( + expect.objectContaining({ + data: expect.objectContaining({ status: 'Failed' }), + }) + ); + }); + + it('sets endedAt on finish', async () => { + const runner = await makeRunner(); + + await runner.finish('Success'); + + const call = prismaMock.execution.update.mock.calls.at(-1)![0]; + expect(call.data.endedAt).toBeDefined(); + }); + }); + + describe('setStage()', () => { + it('updates currentStage and sets progress to the stage range minimum', async () => { + const runner = await makeRunner(); + + runner.setStage('Running'); + + // Force flush to check the update + await runner.flushLogs(true); + + const call = prismaMock.execution.update.mock.calls.at(-1)![0]; + const meta = JSON.parse(call.data.metadata as string); + expect(meta.stage).toBe('Running'); + expect(meta.progress).toBe(10); + }); + + it('emits a completion log for the previous stage when switching stages', async () => { + const runner = await makeRunner(); + runner.setStage('Running'); // start Running + runner.setStage('Completed'); // switch away from Running - adds completion log + + await runner.finish('Success'); + + const call = prismaMock.execution.update.mock.calls.at(-1)![0]; + const logs = JSON.parse(call.data.logs as string); + const completionEntry = logs.find( + (l: { message: string }) => l.message.includes('Running completed') + ); + expect(completionEntry).toBeDefined(); + }); + + it('does not emit completion log when stage stays the same', async () => { + const runner = await makeRunner(); + runner.setStage('Running'); + runner.setStage('Running'); // no change + + await runner.flushLogs(true); + + const call = prismaMock.execution.update.mock.calls.at(-1)![0]; + const logs = JSON.parse(call.data.logs as string); + const completionEntries = logs.filter( + (l: { message: string }) => l.message.includes('Running completed') + ); + expect(completionEntries).toHaveLength(0); + }); + }); + + describe('updateStageProgress()', () => { + it('interpolates internalPercent within the stage range', async () => { + const runner = await makeRunner(); + runner.setStage('Running'); // range [10, 90] + runner.updateStageProgress(50); // midpoint โ†’ 10 + (90-10)*0.5 = 50 + + // finish() drains all pending void flushes before its own final DB write + await runner.finish('Success'); + + const call = prismaMock.execution.update.mock.calls.at(-1)![0]; + const meta = JSON.parse(call.data.metadata as string); + expect(meta.progress).toBe(50); + }); + + it('clamps internalPercent to 0-100 range', async () => { + const runner = await makeRunner(); + runner.setStage('Running'); // range [10, 90] + runner.updateStageProgress(150); // clamped to 100 โ†’ max of range = 90 + + await runner.finish('Success'); + + const call = prismaMock.execution.update.mock.calls.at(-1)![0]; + const meta = JSON.parse(call.data.metadata as string); + expect(meta.progress).toBe(90); + }); + }); + + describe('logEntry()', () => { + it('adds a log entry with the current stage', async () => { + const runner = await makeRunner(); + runner.setStage('Running'); + runner.logEntry('Processing file', 'info'); + + // finish() ensures all pending void flushes complete before writing final state + await runner.finish('Success'); + + const call = prismaMock.execution.update.mock.calls.at(-1)![0]; + const logs = JSON.parse(call.data.logs as string); + const entry = logs.find((l: { message: string }) => l.message === 'Processing file'); + expect(entry).toBeDefined(); + expect(entry.stage).toBe('Running'); + expect(entry.level).toBe('info'); + }); + + it('includes optional details in the log entry', async () => { + const runner = await makeRunner(); + runner.logEntry('Command output', 'info', 'command', 'line1\nline2'); + + await runner.flushLogs(true); + + const call = prismaMock.execution.update.mock.calls.at(-1)![0]; + const logs = JSON.parse(call.data.logs as string); + const entry = logs.find((l: { message: string }) => l.message === 'Command output'); + expect(entry.details).toBe('line1\nline2'); + }); + }); + + describe('flushLogs()', () => { + it('persists current logs and metadata to the database', async () => { + const runner = await makeRunner(); + runner.logEntry('Test log'); + + await runner.flushLogs(true); + + expect(prismaMock.execution.update).toHaveBeenCalledWith( + expect.objectContaining({ + where: { id: 'exec-1' }, + data: expect.objectContaining({ + logs: expect.stringContaining('Test log'), + metadata: expect.any(String), + }), + }) + ); + }); + + it('skips flush when called within 1 second and force is false', async () => { + const runner = await makeRunner(); + + await runner.flushLogs(true); // forced flush, sets lastLogFlush to now + const callCount = prismaMock.execution.update.mock.calls.length; + + await runner.flushLogs(false); // within 1s, should skip + + expect(prismaMock.execution.update.mock.calls.length).toBe(callCount); + }); + }); +}); diff --git a/tests/unit/runner/multi-destination-upload.test.ts b/tests/unit/runner/multi-destination-upload.test.ts index 1725a7cc..b4d1a74d 100644 --- a/tests/unit/runner/multi-destination-upload.test.ts +++ b/tests/unit/runner/multi-destination-upload.test.ts @@ -20,8 +20,19 @@ vi.mock('@/services/backup/encryption-service', () => ({ })); vi.mock('@/lib/crypto/checksum', () => ({ calculateFileChecksum: vi.fn().mockResolvedValue('abc123'), + calculateFileChecksums: vi.fn().mockResolvedValue({ sha256: 'abc123', md5: 'def456' }), verifyFileChecksum: vi.fn().mockResolvedValue({ valid: true }), })); +vi.mock('@/services/storage/verification-service', () => ({ + verificationService: { + verifyFile: vi.fn().mockResolvedValue({ status: 'passed', verifiedAt: new Date().toISOString() }), + }, +})); +vi.mock('@/services/storage/storage-service', () => ({ + storageService: { + appendStorageListCacheEntry: vi.fn().mockResolvedValue(undefined), + }, +})); vi.mock('@/lib/temp-dir', () => ({ getTempDir: vi.fn().mockReturnValue('/tmp'), })); diff --git a/tests/unit/runner/steps/03-upload.test.ts b/tests/unit/runner/steps/03-upload.test.ts index b21836ab..d6098880 100644 --- a/tests/unit/runner/steps/03-upload.test.ts +++ b/tests/unit/runner/steps/03-upload.test.ts @@ -70,6 +70,7 @@ vi.mock('@/lib/utils', () => ({ vi.mock('@/lib/crypto/checksum', () => ({ calculateFileChecksum: vi.fn().mockResolvedValue('abc123checksum'), + calculateFileChecksums: vi.fn().mockResolvedValue({ sha256: 'abc123checksum', md5: 'def456checksum' }), verifyFileChecksum: vi.fn().mockResolvedValue({ valid: true, expected: 'abc123', actual: 'abc123' }), })); @@ -91,6 +92,19 @@ vi.mock('@/lib/prisma', () => ({ }, })); +vi.mock('@/services/storage/verification-service', () => ({ + verificationService: { + verifyFile: vi.fn().mockResolvedValue({ status: 'passed', verifiedAt: new Date().toISOString() }), + }, +})); + +vi.mock('@/services/storage/storage-service', () => ({ + storageService: { + appendStorageListCacheEntry: vi.fn().mockResolvedValue(undefined), + updateStorageListCacheEntry: vi.fn().mockResolvedValue(undefined), + }, +})); + // --- Helpers --- function makeDestination(overrides: Partial = {}): DestinationContext { @@ -239,13 +253,13 @@ describe('stepUpload', () => { }); it('calculates checksum and writes metadata sidecar', async () => { - const { calculateFileChecksum } = await import('@/lib/crypto/checksum'); + const { calculateFileChecksums } = await import('@/lib/crypto/checksum'); const fsPromises = await import('fs/promises'); const ctx = makeCtx(); await stepUpload(ctx); - expect(calculateFileChecksum).toHaveBeenCalledWith('/tmp/test_backup.sql'); + expect(calculateFileChecksums).toHaveBeenCalledWith('/tmp/test_backup.sql'); expect(fsPromises.default.writeFile).toHaveBeenCalledWith( '/tmp/test_backup.sql.meta.json', expect.stringContaining('abc123checksum'), @@ -378,23 +392,23 @@ describe('stepUpload', () => { }); it('performs post-upload integrity verification for local-filesystem destinations', async () => { - const { verifyFileChecksum } = await import('@/lib/crypto/checksum'); + const { verificationService } = await import('@/services/storage/verification-service'); const dest = makeDestination({ adapterId: 'local-filesystem' }); const ctx = makeCtx({ destinations: [dest] }); await stepUpload(ctx); - expect(dest.adapter.download).toHaveBeenCalled(); - expect(verifyFileChecksum).toHaveBeenCalled(); + expect(verificationService.verifyFile).toHaveBeenCalled(); expect(ctx.log).toHaveBeenCalledWith(expect.stringContaining('Integrity check passed'), 'success'); }); it('logs a warning when post-upload integrity check fails', async () => { - const { verifyFileChecksum } = await import('@/lib/crypto/checksum'); - (verifyFileChecksum as ReturnType).mockResolvedValueOnce({ - valid: false, - expected: 'abc123', - actual: 'badchecksum', + const { verificationService } = await import('@/services/storage/verification-service'); + vi.mocked(verificationService.verifyFile).mockResolvedValueOnce({ + status: 'failed', + expectedChecksum: 'abc123', + actualChecksum: 'badchecksum', + verifiedAt: new Date().toISOString(), }); const dest = makeDestination({ adapterId: 'local-filesystem' }); @@ -406,14 +420,13 @@ describe('stepUpload', () => { }); it('skips integrity verification for non-local-filesystem destinations', async () => { - const { verifyFileChecksum } = await import('@/lib/crypto/checksum'); + const { verificationService } = await import('@/services/storage/verification-service'); const dest = makeDestination({ adapterId: 's3' }); const ctx = makeCtx({ destinations: [dest] }); await stepUpload(ctx); - expect(verifyFileChecksum).not.toHaveBeenCalled(); - expect(ctx.log).toHaveBeenCalledWith(expect.stringContaining('No local destinations')); + expect(verificationService.verifyFile).not.toHaveBeenCalled(); }); it('uses percentage-based detail string when dumpSize is zero', async () => { diff --git a/tests/unit/runner/steps/05-retention.test.ts b/tests/unit/runner/steps/05-retention.test.ts index 26fd5321..cc45db50 100644 --- a/tests/unit/runner/steps/05-retention.test.ts +++ b/tests/unit/runner/steps/05-retention.test.ts @@ -22,6 +22,14 @@ vi.mock('@/services/dashboard-service', () => ({ refreshStorageStatsCache: vi.fn().mockResolvedValue(undefined), })); +vi.mock('@/services/storage/storage-service', () => ({ + storageService: { + appendStorageListCacheEntry: vi.fn().mockResolvedValue(undefined), + updateStorageListCacheEntry: vi.fn().mockResolvedValue(undefined), + removeStorageListCacheEntry: vi.fn().mockResolvedValue(undefined), + }, +})); + vi.mock('@/lib/prisma', () => ({ default: { systemSetting: { findUnique: vi.fn().mockResolvedValue(null) }, diff --git a/tests/unit/services/db-version-service.test.ts b/tests/unit/services/db-version-service.test.ts index f4ba291f..e8914628 100644 --- a/tests/unit/services/db-version-service.test.ts +++ b/tests/unit/services/db-version-service.test.ts @@ -43,7 +43,7 @@ describe("db-version-service", () => { const result = await recordVersionIfChanged(SOURCE_ID, "8.0.36"); - expect(result).toEqual({ changed: true, previousVersion: null, newVersion: "8.0.36" }); + expect(result).toEqual({ changed: true, previousVersion: null, newVersion: "8.0.36", isDowngrade: false }); expect(prismaMock.dbVersionHistory.create).toHaveBeenCalledWith({ data: { adapterConfigId: SOURCE_ID, @@ -59,7 +59,7 @@ describe("db-version-service", () => { const result = await recordVersionIfChanged(SOURCE_ID, "8.0.36"); - expect(result).toEqual({ changed: false, previousVersion: "8.0.36", newVersion: "8.0.36" }); + expect(result).toEqual({ changed: false, previousVersion: "8.0.36", newVersion: "8.0.36", isDowngrade: false }); expect(prismaMock.dbVersionHistory.create).not.toHaveBeenCalled(); }); @@ -69,7 +69,7 @@ describe("db-version-service", () => { const result = await recordVersionIfChanged(SOURCE_ID, "8.0.37"); - expect(result).toEqual({ changed: true, previousVersion: "8.0.36", newVersion: "8.0.37" }); + expect(result).toEqual({ changed: true, previousVersion: "8.0.36", newVersion: "8.0.37", isDowngrade: false }); expect(prismaMock.dbVersionHistory.create).toHaveBeenCalledWith({ data: { adapterConfigId: SOURCE_ID, diff --git a/tests/unit/services/integrity-service.test.ts b/tests/unit/services/integrity-service.test.ts index 9eeaad60..224477e5 100644 --- a/tests/unit/services/integrity-service.test.ts +++ b/tests/unit/services/integrity-service.test.ts @@ -2,11 +2,17 @@ import { describe, it, expect, vi, beforeEach } from 'vitest'; import { integrityService } from '@/services/backup/integrity-service'; import prisma from '@/lib/prisma'; import { registry } from '@/lib/core/registry'; +import { verificationService } from '@/services/storage/verification-service'; vi.mock('@/lib/prisma', () => ({ default: { adapterConfig: { findMany: vi.fn() }, job: { findMany: vi.fn() }, + systemSetting: { + findUnique: vi.fn().mockImplementation(({ where }: { where: { key: string } }) => + Promise.resolve(where.key === 'integrity.scanMode' ? { value: 'destinations' } : null) + ), + }, }, })); @@ -22,12 +28,10 @@ vi.mock('@/lib/adapters/config-resolver', () => ({ resolveAdapterConfig: vi.fn().mockResolvedValue({ bucket: 'test' }), })); -vi.mock('@/lib/temp-dir', () => ({ - getTempDir: vi.fn().mockReturnValue('/tmp'), -})); - -vi.mock('@/lib/crypto/checksum', () => ({ - verifyFileChecksum: vi.fn(), +vi.mock('@/services/storage/verification-service', () => ({ + verificationService: { + verifyFile: vi.fn(), + }, })); vi.mock('@/lib/logging/logger', () => ({ @@ -45,28 +49,14 @@ vi.mock('@/lib/logging/errors', () => ({ wrapError: vi.fn((e) => e), })); -// Shared mock functions hoisted so they're available when vi.mock factories run -const { mockFsReadFile, mockFsUnlink } = vi.hoisted(() => ({ - mockFsReadFile: vi.fn(), - mockFsUnlink: vi.fn().mockResolvedValue(undefined), -})); - -vi.mock('fs', () => ({ - default: { - promises: { - readFile: mockFsReadFile, - unlink: mockFsUnlink, - }, - }, - promises: { - readFile: mockFsReadFile, - unlink: mockFsUnlink, - }, -})); - describe('IntegrityService', () => { beforeEach(() => { vi.clearAllMocks(); + // Default: files have no metadata (results in skipped) + vi.mocked(verificationService.verifyFile).mockResolvedValue({ + status: 'no_metadata', + verifiedAt: new Date().toISOString(), + }); }); function makeStorageAdapter(overrides: Record = {}) { @@ -93,17 +83,17 @@ describe('IntegrityService', () => { it('skips files without checksum metadata', async () => { const adapter = makeStorageAdapter({ - list: vi.fn() - .mockResolvedValueOnce([{ name: 'My Job' }]) // top-level folder - .mockResolvedValueOnce([{ name: 'backup.sql' }]), // files in folder - read: vi.fn().mockResolvedValue(null), - download: vi.fn().mockResolvedValue(false), // no metadata download + list: vi.fn().mockResolvedValue([{ name: 'backup.sql', path: 'backup.sql' }]), }); (prisma.adapterConfig.findMany as ReturnType).mockResolvedValue([ { id: 's1', adapterId: 'local', name: 'Local', config: '{}', primaryCredentialId: null, sshCredentialId: null }, ]); (registry.get as ReturnType).mockReturnValue(adapter); + vi.mocked(verificationService.verifyFile).mockResolvedValue({ + status: 'no_checksum', + verifiedAt: new Date().toISOString(), + }); const result = await integrityService.runFullIntegrityCheck(); @@ -113,26 +103,17 @@ describe('IntegrityService', () => { }); it('passes file that matches checksum', async () => { - const { verifyFileChecksum } = await import('@/lib/crypto/checksum'); - - const metaJson = JSON.stringify({ checksum: 'sha256:abc123' }); - const adapter = makeStorageAdapter({ - list: vi.fn() - .mockResolvedValueOnce([{ name: 'My Job' }]) - .mockResolvedValueOnce([{ name: 'backup.sql' }]), - read: vi.fn().mockResolvedValue(metaJson), - download: vi.fn().mockResolvedValue(true), + list: vi.fn().mockResolvedValue([{ name: 'backup.sql', path: 'backup.sql' }]), }); (prisma.adapterConfig.findMany as ReturnType).mockResolvedValue([ { id: 's1', adapterId: 'local', name: 'Local', config: '{}', primaryCredentialId: null, sshCredentialId: null }, ]); (registry.get as ReturnType).mockReturnValue(adapter); - (verifyFileChecksum as ReturnType).mockResolvedValue({ - valid: true, - expected: 'sha256:abc123', - actual: 'sha256:abc123', + vi.mocked(verificationService.verifyFile).mockResolvedValue({ + status: 'passed', + verifiedAt: new Date().toISOString(), }); const result = await integrityService.runFullIntegrityCheck(); @@ -145,26 +126,19 @@ describe('IntegrityService', () => { }); it('records failed file when checksum mismatch', async () => { - const { verifyFileChecksum } = await import('@/lib/crypto/checksum'); - - const metaJson = JSON.stringify({ checksum: 'sha256:correct' }); - const adapter = makeStorageAdapter({ - list: vi.fn() - .mockResolvedValueOnce([{ name: 'Jobs' }]) - .mockResolvedValueOnce([{ name: 'backup.sql' }]), - read: vi.fn().mockResolvedValue(metaJson), - download: vi.fn().mockResolvedValue(true), + list: vi.fn().mockResolvedValue([{ name: 'backup.sql', path: 'backup.sql' }]), }); (prisma.adapterConfig.findMany as ReturnType).mockResolvedValue([ { id: 's1', adapterId: 'local', name: 'Storage1', config: '{}', primaryCredentialId: null, sshCredentialId: null }, ]); (registry.get as ReturnType).mockReturnValue(adapter); - (verifyFileChecksum as ReturnType).mockResolvedValue({ - valid: false, - expected: 'sha256:correct', - actual: 'sha256:tampered', + vi.mocked(verificationService.verifyFile).mockResolvedValue({ + status: 'failed', + expectedChecksum: 'sha256:correct', + actualChecksum: 'sha256:tampered', + verifiedAt: new Date().toISOString(), }); const result = await integrityService.runFullIntegrityCheck(); @@ -181,10 +155,8 @@ describe('IntegrityService', () => { it('falls back to job names when listing storage root fails', async () => { const adapter = makeStorageAdapter({ list: vi.fn() - .mockRejectedValueOnce(new Error('Permission denied')) // root listing fails - .mockResolvedValueOnce([{ name: 'backup.sql' }]), // folder listing succeeds - read: vi.fn().mockResolvedValue(null), - download: vi.fn().mockResolvedValue(false), + .mockRejectedValueOnce(new Error('Permission denied')) + .mockResolvedValueOnce([{ name: 'backup.sql', path: 'backup.sql' }]), }); (prisma.adapterConfig.findMany as ReturnType).mockResolvedValue([ @@ -197,7 +169,6 @@ describe('IntegrityService', () => { const result = await integrityService.runFullIntegrityCheck(); - // Fallback to job names was used - listing should have been attempted for 'Fallback Job' expect(prisma.job.findMany).toHaveBeenCalled(); expect(result.totalFiles).toBe(1); expect(result.skipped).toBe(1); @@ -219,9 +190,7 @@ describe('IntegrityService', () => { list: vi.fn().mockRejectedValue(new Error('Storage crash')), }); const passingAdapter = makeStorageAdapter({ - list: vi.fn() - .mockResolvedValueOnce([{ name: 'Jobs' }]) - .mockResolvedValueOnce([]), + list: vi.fn().mockResolvedValue([{ name: 'backup.sql', path: 'backup.sql' }]), }); (prisma.adapterConfig.findMany as ReturnType).mockResolvedValue([ @@ -235,28 +204,19 @@ describe('IntegrityService', () => { await expect(integrityService.runFullIntegrityCheck()).resolves.not.toThrow(); }); - it('uses download fallback for metadata when adapter has no read() method', async () => { - const { verifyFileChecksum } = await import('@/lib/crypto/checksum'); - - const metaJson = JSON.stringify({ checksum: 'sha256:xyz' }); - + it('delegates metadata download fallback to verificationService', async () => { const adapter = makeStorageAdapter({ - read: undefined, // No read() method - list: vi.fn() - .mockResolvedValueOnce([{ name: 'Folder' }]) - .mockResolvedValueOnce([{ name: 'backup.sql' }]), - download: vi.fn().mockResolvedValue(true), + read: undefined, + list: vi.fn().mockResolvedValue([{ name: 'backup.sql', path: 'backup.sql' }]), }); - mockFsReadFile.mockResolvedValue(metaJson); (prisma.adapterConfig.findMany as ReturnType).mockResolvedValue([ { id: 's1', adapterId: 'local', name: 'Local', config: '{}', primaryCredentialId: null, sshCredentialId: null }, ]); (registry.get as ReturnType).mockReturnValue(adapter); - (verifyFileChecksum as ReturnType).mockResolvedValue({ - valid: true, - expected: 'sha256:xyz', - actual: 'sha256:xyz', + vi.mocked(verificationService.verifyFile).mockResolvedValue({ + status: 'passed', + verifiedAt: new Date().toISOString(), }); const result = await integrityService.runFullIntegrityCheck(); @@ -265,17 +225,9 @@ describe('IntegrityService', () => { }); it('accumulates results across multiple destinations', async () => { - const { verifyFileChecksum } = await import('@/lib/crypto/checksum'); - - const metaJson = JSON.stringify({ checksum: 'sha256:ok' }); - function makePassingAdapter() { return makeStorageAdapter({ - list: vi.fn() - .mockResolvedValueOnce([{ name: 'Folder' }]) - .mockResolvedValueOnce([{ name: 'backup.sql' }]), - read: vi.fn().mockResolvedValue(metaJson), - download: vi.fn().mockResolvedValue(true), + list: vi.fn().mockResolvedValue([{ name: 'backup.sql', path: 'backup.sql' }]), }); } @@ -286,10 +238,9 @@ describe('IntegrityService', () => { (registry.get as ReturnType) .mockReturnValueOnce(makePassingAdapter()) .mockReturnValueOnce(makePassingAdapter()); - (verifyFileChecksum as ReturnType).mockResolvedValue({ - valid: true, - expected: 'sha256:ok', - actual: 'sha256:ok', + vi.mocked(verificationService.verifyFile).mockResolvedValue({ + status: 'passed', + verifiedAt: new Date().toISOString(), }); const result = await integrityService.runFullIntegrityCheck(); @@ -297,4 +248,202 @@ describe('IntegrityService', () => { expect(result.totalFiles).toBe(2); expect(result.passed).toBe(2); }); + + describe('Jobs scan mode (gatherFilesFromJobs)', () => { + function makeJobWithDest(jobName: string, destConfigId: string, overrides: Record = {}) { + return { + id: `job-${jobName}`, + name: jobName, + enabled: true, + skipVerification: false, + destinations: [ + { + configId: destConfigId, + config: { + id: destConfigId, + adapterId: 'local', + name: `Storage for ${jobName}`, + config: '{}', + primaryCredentialId: null, + sshCredentialId: null, + metadata: null, + }, + }, + ], + ...overrides, + }; + } + + beforeEach(() => { + // Switch to jobs mode: no scanMode setting + vi.mocked(prisma.systemSetting.findUnique).mockResolvedValue(null); + }); + + it('scans job-linked files when scanMode is not set', async () => { + const adapter = makeStorageAdapter({ + list: vi.fn().mockResolvedValue([ + { name: 'backup.sql', path: 'ProdJob/backup.sql' }, + ]), + }); + (prisma.job.findMany as ReturnType).mockResolvedValue([ + makeJobWithDest('ProdJob', 'dest-1'), + ]); + (registry.get as ReturnType).mockReturnValue(adapter); + vi.mocked(verificationService.verifyFile).mockResolvedValue({ + status: 'passed', + verifiedAt: new Date().toISOString(), + }); + + const result = await integrityService.runFullIntegrityCheck(); + + expect(result.totalFiles).toBe(1); + expect(result.passed).toBe(1); + }); + + it('skips jobs with skipVerification flag', async () => { + const adapter = makeStorageAdapter({ + list: vi.fn().mockResolvedValue([{ name: 'backup.sql', path: 'SkipJob/backup.sql' }]), + }); + (prisma.job.findMany as ReturnType).mockResolvedValue([ + makeJobWithDest('SkipJob', 'dest-1', { skipVerification: true }), + ]); + (registry.get as ReturnType).mockReturnValue(adapter); + + const result = await integrityService.runFullIntegrityCheck(); + + expect(result.totalFiles).toBe(0); + }); + + it('filters out files that do not start with the job name prefix', async () => { + const adapter = makeStorageAdapter({ + list: vi.fn().mockResolvedValue([ + { name: 'mine.sql', path: 'ProdJob/mine.sql' }, + { name: 'other.sql', path: 'OtherJob/other.sql' }, + ]), + }); + (prisma.job.findMany as ReturnType).mockResolvedValue([ + makeJobWithDest('ProdJob', 'dest-1'), + ]); + (registry.get as ReturnType).mockReturnValue(adapter); + vi.mocked(verificationService.verifyFile).mockResolvedValue({ + status: 'passed', + verifiedAt: new Date().toISOString(), + }); + + const result = await integrityService.runFullIntegrityCheck(); + + expect(result.totalFiles).toBe(1); + }); + + it('excludes .meta.json sidecar files from verification', async () => { + const adapter = makeStorageAdapter({ + list: vi.fn().mockResolvedValue([ + { name: 'backup.sql', path: 'Job/backup.sql' }, + { name: 'backup.sql.meta.json', path: 'Job/backup.sql.meta.json' }, + ]), + }); + (prisma.job.findMany as ReturnType).mockResolvedValue([ + makeJobWithDest('Job', 'dest-1'), + ]); + (registry.get as ReturnType).mockReturnValue(adapter); + vi.mocked(verificationService.verifyFile).mockResolvedValue({ + status: 'passed', + verifiedAt: new Date().toISOString(), + }); + + const result = await integrityService.runFullIntegrityCheck(); + + expect(result.totalFiles).toBe(1); + }); + + it('deduplicates files when two jobs share the same destination', async () => { + const adapter = makeStorageAdapter({ + list: vi.fn().mockResolvedValue([ + { name: 'job1.sql', path: 'Job1/job1.sql' }, + { name: 'job2.sql', path: 'Job2/job2.sql' }, + ]), + }); + const sharedDestId = 'shared-dest'; + (prisma.job.findMany as ReturnType).mockResolvedValue([ + makeJobWithDest('Job1', sharedDestId), + makeJobWithDest('Job2', sharedDestId), + ]); + (registry.get as ReturnType).mockReturnValue(adapter); + vi.mocked(verificationService.verifyFile).mockResolvedValue({ + status: 'passed', + verifiedAt: new Date().toISOString(), + }); + + const result = await integrityService.runFullIntegrityCheck(); + + // list() called once despite two jobs sharing the destination + expect(adapter.list).toHaveBeenCalledTimes(1); + expect(result.totalFiles).toBe(2); + }); + + it('skips destinations with skipVerification set in metadata', async () => { + const adapter = makeStorageAdapter({ + list: vi.fn().mockResolvedValue([{ name: 'backup.sql', path: 'Job/backup.sql' }]), + }); + const job = { + ...makeJobWithDest('Job', 'dest-1'), + destinations: [ + { + configId: 'dest-1', + config: { + id: 'dest-1', + adapterId: 'local', + name: 'Skipped Storage', + config: '{}', + primaryCredentialId: null, + sshCredentialId: null, + metadata: JSON.stringify({ skipVerification: true }), + }, + }, + ], + }; + (prisma.job.findMany as ReturnType).mockResolvedValue([job]); + (registry.get as ReturnType).mockReturnValue(adapter); + + const result = await integrityService.runFullIntegrityCheck(); + + expect(result.totalFiles).toBe(0); + }); + + it('applies maxAgeDays filter in jobs mode', async () => { + const oldDate = new Date(Date.now() - 10 * 86_400_000).toISOString(); + const recentDate = new Date(Date.now() - 1 * 86_400_000).toISOString(); + const adapter = makeStorageAdapter({ + list: vi.fn().mockResolvedValue([ + { name: 'old.sql', path: 'Job/old.sql', lastModified: oldDate, size: 100 }, + { name: 'new.sql', path: 'Job/new.sql', lastModified: recentDate, size: 100 }, + ]), + }); + (prisma.job.findMany as ReturnType).mockResolvedValue([ + makeJobWithDest('Job', 'dest-1'), + ]); + (registry.get as ReturnType).mockReturnValue(adapter); + // maxAgeDays = 5 + (prisma.systemSetting.findUnique as ReturnType).mockImplementation(({ where }: { where: { key?: string } }) => { + if (where.key === 'integrity.maxAgeDays') return Promise.resolve({ value: '5' }); + return Promise.resolve(null); + }); + vi.mocked(verificationService.verifyFile).mockResolvedValue({ + status: 'passed', + verifiedAt: new Date().toISOString(), + }); + + const result = await integrityService.runFullIntegrityCheck(); + + expect(result.totalFiles).toBe(1); + }); + + it('returns zero results when no jobs exist', async () => { + (prisma.job.findMany as ReturnType).mockResolvedValue([]); + + const result = await integrityService.runFullIntegrityCheck(); + + expect(result.totalFiles).toBe(0); + }); + }); }); diff --git a/tests/unit/services/job-service.test.ts b/tests/unit/services/job-service.test.ts index 363630cf..d7f8dcdd 100644 --- a/tests/unit/services/job-service.test.ts +++ b/tests/unit/services/job-service.test.ts @@ -59,6 +59,7 @@ describe('JobService', () => { compression: "NONE", pgCompression: "", notificationEvents: "ALWAYS", + skipVerification: false, notifications: { connect: [{ id: 'notif-1' }] }, diff --git a/tests/unit/services/storage-service.test.ts b/tests/unit/services/storage-service.test.ts index b111420b..3dce3ef7 100644 --- a/tests/unit/services/storage-service.test.ts +++ b/tests/unit/services/storage-service.test.ts @@ -151,6 +151,8 @@ describe('StorageService', () => { mockResolveAdapterConfig.mockImplementation((adapterConfig: any) => Promise.resolve(JSON.parse(adapterConfig.config)) ); + // storageListCache.upsert is a fire-and-forget cache write - must return a Promise + prismaMock.storageListCache.upsert.mockResolvedValue({} as any); // Reset fs mocks to success fsMocks.writeFile.mockResolvedValue(undefined); fsMocks.unlink.mockResolvedValue(undefined); diff --git a/tests/unit/services/system-task-service.test.ts b/tests/unit/services/system-task-service.test.ts index 8ea11542..67b5eae0 100644 --- a/tests/unit/services/system-task-service.test.ts +++ b/tests/unit/services/system-task-service.test.ts @@ -39,6 +39,19 @@ vi.mock('@/services/backup/integrity-service', () => ({ runFullIntegrityCheck: vi.fn().mockResolvedValue({ totalFiles: 5, passed: 5, failed: 0, skipped: 0 }), }, })); +vi.mock('@/lib/runner/system-task-runner', () => ({ + SystemTaskRunner: { + create: vi.fn().mockResolvedValue({ + id: 'runner-exec-1', + start: vi.fn().mockResolvedValue(undefined), + finish: vi.fn().mockResolvedValue(undefined), + logEntry: vi.fn(), + setStage: vi.fn(), + setProgress: vi.fn(), + }), + }, + INTEGRITY_CHECK_STAGE_PROGRESS_MAP: {}, +})); vi.mock('@/services/dashboard-service', () => ({ refreshStorageStatsCache: vi.fn().mockResolvedValue(undefined), cleanupOldSnapshots: vi.fn().mockResolvedValue(3), @@ -638,6 +651,7 @@ describe('SystemTaskService', () => { changed: true, previousVersion: '15.0.4280.7', newVersion: '15.0.4360.2', + isDowngrade: false, }); await service.runTask(SYSTEM_TASKS.UPDATE_DB_VERSIONS); @@ -675,6 +689,7 @@ describe('SystemTaskService', () => { changed: true, previousVersion: null, newVersion: '8.0.31', + isDowngrade: false, }); await service.runTask(SYSTEM_TASKS.UPDATE_DB_VERSIONS); @@ -702,6 +717,7 @@ describe('SystemTaskService', () => { changed: false, previousVersion: '16.2', newVersion: '16.2', + isDowngrade: false, }); await service.runTask(SYSTEM_TASKS.UPDATE_DB_VERSIONS); diff --git a/tests/unit/services/verification-service.test.ts b/tests/unit/services/verification-service.test.ts new file mode 100644 index 00000000..d1e90711 --- /dev/null +++ b/tests/unit/services/verification-service.test.ts @@ -0,0 +1,379 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import type { BackupMetadata } from '@/lib/core/interfaces'; + +vi.mock('@/lib/prisma', () => ({ + default: { + adapterConfig: { findUnique: vi.fn() }, + }, +})); + +vi.mock('@/lib/core/registry', () => ({ + registry: { get: vi.fn() }, +})); + +vi.mock('@/lib/adapters', () => ({ + registerAdapters: vi.fn(), +})); + +vi.mock('@/lib/adapters/config-resolver', () => ({ + resolveAdapterConfig: vi.fn().mockResolvedValue({ resolved: true }), +})); + +vi.mock('@/lib/crypto/checksum', () => ({ + calculateFileChecksum: vi.fn(), +})); + +vi.mock('@/lib/temp-dir', () => ({ + getTempDir: vi.fn().mockReturnValue('/tmp'), +})); + +vi.mock('fs/promises', () => { + const readFile = vi.fn().mockResolvedValue('{}'); + const unlink = vi.fn().mockResolvedValue(undefined); + const writeFile = vi.fn().mockResolvedValue(undefined); + return { + default: { readFile, unlink, writeFile }, + readFile, + unlink, + writeFile, + }; +}); + +vi.mock('@/lib/logging/logger', () => ({ + logger: { + child: vi.fn().mockReturnValue({ + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + debug: vi.fn(), + }), + }, +})); + +vi.mock('@/lib/logging/errors', () => ({ + wrapError: vi.fn((e) => e), +})); + +vi.mock('@/services/storage/storage-service', () => ({ + storageService: { + updateStorageListCacheEntry: vi.fn().mockResolvedValue(undefined), + }, +})); + +import prisma from '@/lib/prisma'; +import { registry } from '@/lib/core/registry'; +import { calculateFileChecksum } from '@/lib/crypto/checksum'; +import { verificationService } from '@/services/storage/verification-service'; +import fsPromises from 'fs/promises'; + +function makeAdapterConfig(overrides: Record = {}) { + return { + id: 'config-1', + adapterId: 'local', + name: 'Local Storage', + config: '{}', + primaryCredentialId: null, + sshCredentialId: null, + metadata: null, + ...overrides, + }; +} + +function makeAdapter(overrides: Record = {}) { + return { + download: vi.fn().mockResolvedValue(false), + upload: vi.fn().mockResolvedValue(undefined), + read: vi.fn().mockResolvedValue(null), + list: vi.fn().mockResolvedValue([]), + delete: vi.fn().mockResolvedValue(undefined), + verifyChecksum: undefined as ReturnType | undefined, + ...overrides, + }; +} + +function makeMetadata(overrides: Partial = {}): BackupMetadata { + return { + version: 1, + jobId: 'job-1', + jobName: 'Test Job', + sourceName: 'Test Source', + sourceType: 'mysql', + databases: ['testdb'], + timestamp: '2024-01-01T00:00:00.000Z', + originalFileName: 'backup.sql', + sourceId: 'src-1', + checksum: 'sha256:abc123', + ...overrides, + }; +} + +describe('VerificationService', () => { + beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(calculateFileChecksum).mockResolvedValue('sha256:abc123'); + }); + + it('throws when adapter config is not found', async () => { + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(null); + + await expect( + verificationService.verifyFile('missing-id', 'backup.sql', 'manual') + ).rejects.toThrow('Storage configuration not found'); + }); + + it('throws when adapter is not in registry', async () => { + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(null); + + await expect( + verificationService.verifyFile('config-1', 'backup.sql', 'manual') + ).rejects.toThrow("Adapter 'local' not found"); + }); + + it('returns no_metadata when adapter.read returns null and download returns false', async () => { + const adapter = makeAdapter({ read: vi.fn().mockResolvedValue(null) }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(result.status).toBe('no_metadata'); + }); + + it('returns no_metadata when adapter has no read method and download returns false', async () => { + const adapter = makeAdapter({ read: undefined }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(result.status).toBe('no_metadata'); + expect(adapter.download).toHaveBeenCalledWith( + expect.anything(), + 'backup.sql.meta.json', + expect.stringContaining('/tmp') + ); + }); + + it('returns no_checksum when metadata has no checksum fields', async () => { + const metadata = makeMetadata({ checksum: undefined, checksumMd5: undefined }); + const adapter = makeAdapter({ read: vi.fn().mockResolvedValue(JSON.stringify(metadata)) }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(result.status).toBe('no_checksum'); + }); + + it('returns skipped when skipIfPassed is true and metadata.verification.passed is true', async () => { + const metadata = makeMetadata({ + verification: { verifiedAt: '2024-01-01T00:00:00.000Z', passed: true, trigger: 'manual' }, + }); + const adapter = makeAdapter({ read: vi.fn().mockResolvedValue(JSON.stringify(metadata)) }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual', { skipIfPassed: true }); + + expect(result.status).toBe('skipped'); + }); + + it('does not skip when skipIfPassed is true but verification has not passed', async () => { + const metadata = makeMetadata({ + verification: { verifiedAt: '2024-01-01T00:00:00.000Z', passed: false, trigger: 'manual' }, + }); + const adapter = makeAdapter({ + read: vi.fn().mockResolvedValue(JSON.stringify(metadata)), + download: vi.fn().mockResolvedValue(true), + }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual', { skipIfPassed: true }); + + expect(result.status).not.toBe('skipped'); + }); + + it('returns passed via native verifyChecksum', async () => { + const metadata = makeMetadata(); + const adapter = makeAdapter({ + read: vi.fn().mockResolvedValue(JSON.stringify(metadata)), + verifyChecksum: vi.fn().mockResolvedValue('passed'), + }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(result.status).toBe('passed'); + expect(adapter.verifyChecksum).toHaveBeenCalledWith( + expect.anything(), + 'backup.sql', + expect.objectContaining({ sha256: 'sha256:abc123' }) + ); + }); + + it('returns failed via native verifyChecksum', async () => { + const metadata = makeMetadata(); + const adapter = makeAdapter({ + read: vi.fn().mockResolvedValue(JSON.stringify(metadata)), + verifyChecksum: vi.fn().mockResolvedValue('failed'), + }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(result.status).toBe('failed'); + }); + + it('falls back to download when native verifyChecksum returns unsupported', async () => { + const metadata = makeMetadata(); + const adapter = makeAdapter({ + read: vi.fn().mockResolvedValue(JSON.stringify(metadata)), + verifyChecksum: vi.fn().mockResolvedValue('unsupported'), + download: vi.fn().mockResolvedValue(true), + }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + vi.mocked(calculateFileChecksum).mockResolvedValue('sha256:abc123'); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(result.status).toBe('passed'); + expect(adapter.download).toHaveBeenCalledWith( + expect.anything(), + 'backup.sql', + expect.stringContaining('/tmp') + ); + }); + + it('returns passed when download checksum matches', async () => { + const metadata = makeMetadata({ checksum: 'sha256:expected' }); + const adapter = makeAdapter({ + read: vi.fn().mockResolvedValue(JSON.stringify(metadata)), + download: vi.fn().mockResolvedValue(true), + }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + vi.mocked(calculateFileChecksum).mockResolvedValue('sha256:expected'); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(result.status).toBe('passed'); + expect(result.expectedChecksum).toBe('sha256:expected'); + expect(result.actualChecksum).toBeUndefined(); + }); + + it('returns failed when download checksum does not match', async () => { + const metadata = makeMetadata({ checksum: 'sha256:expected' }); + const adapter = makeAdapter({ + read: vi.fn().mockResolvedValue(JSON.stringify(metadata)), + download: vi.fn().mockResolvedValue(true), + }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + vi.mocked(calculateFileChecksum).mockResolvedValue('sha256:tampered'); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(result.status).toBe('failed'); + expect(result.expectedChecksum).toBe('sha256:expected'); + expect(result.actualChecksum).toBe('sha256:tampered'); + }); + + it('returns download_error when adapter.download returns false for the backup file', async () => { + const metadata = makeMetadata(); + const adapter = makeAdapter({ + read: vi.fn().mockResolvedValue(JSON.stringify(metadata)), + download: vi.fn().mockResolvedValue(false), + }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(result.status).toBe('download_error'); + }); + + it('returns download_error when adapter.download throws', async () => { + const metadata = makeMetadata(); + const adapter = makeAdapter({ + read: vi.fn().mockResolvedValue(JSON.stringify(metadata)), + download: vi.fn().mockRejectedValue(new Error('Network timeout')), + }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(result.status).toBe('download_error'); + }); + + it('reads metadata via adapter.download when adapter has no read method', async () => { + const metadata = makeMetadata(); + const adapter = makeAdapter({ + read: undefined, + download: vi.fn().mockResolvedValue(true), + }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + vi.mocked(fsPromises.readFile as ReturnType).mockResolvedValueOnce(JSON.stringify(metadata)); + + await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(adapter.download).toHaveBeenCalledWith( + expect.anything(), + 'backup.sql.meta.json', + expect.stringContaining('/tmp') + ); + }); + + it('calls adapter.upload to persist verification result after native check', async () => { + const metadata = makeMetadata(); + const adapter = makeAdapter({ + read: vi.fn().mockResolvedValue(JSON.stringify(metadata)), + verifyChecksum: vi.fn().mockResolvedValue('passed'), + }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + + await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(adapter.upload).toHaveBeenCalledWith( + expect.anything(), + expect.stringContaining('/tmp'), + 'backup.sql.meta.json' + ); + }); + + it('calls adapter.upload to persist verification result after download-based check', async () => { + const metadata = makeMetadata(); + const adapter = makeAdapter({ + read: vi.fn().mockResolvedValue(JSON.stringify(metadata)), + download: vi.fn().mockResolvedValue(true), + }); + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(adapter); + vi.mocked(calculateFileChecksum).mockResolvedValue('sha256:abc123'); + + await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(adapter.upload).toHaveBeenCalledWith( + expect.anything(), + expect.stringContaining('/tmp'), + 'backup.sql.meta.json' + ); + }); + + it('includes verifiedAt in every result', async () => { + (prisma.adapterConfig.findUnique as ReturnType).mockResolvedValue(makeAdapterConfig()); + (registry.get as ReturnType).mockReturnValue(makeAdapter({ read: vi.fn().mockResolvedValue(null) })); + + const result = await verificationService.verifyFile('config-1', 'backup.sql', 'manual'); + + expect(result.verifiedAt).toBeDefined(); + expect(new Date(result.verifiedAt).getTime()).not.toBeNaN(); + }); +});