Skip to content

PLAN.md

Latest commit

 

History

History
405 lines (323 loc) · 22.7 KB

PLAN.md

File metadata and controls

405 lines (323 loc) · 22.7 KB

ShareX Hosting Site - undefinedlabs.tech

Context

Build a personal ShareX file hosting site that accepts image, file, and text uploads via ShareX custom uploader destinations. The site runs as a Blazor Server app in Docker, deployed on Dokploy, using a self-hosted Convex instance for database and file storage via the unofficial Convex.Client NuGet package. The Convex TypeScript backend (schema + functions) lives in the same repo but is deployed separately from the .NET app.

Architecture Overview

ShareX  -->  POST /i/, /f/, /t/  -->  Blazor Server  -->  Convex (self-hosted)
                                           |                    |
Browser -->  GET /i/{slug}        -->  Blazor Pages   -->  Convex Queries
Browser -->  GET /f/{slug}        -->  Blazor Pages   -->  Convex Queries
Browser -->  GET /t/{slug}        -->  Blazor Pages   -->  Convex Queries
Browser -->  GET /admin/dashboard -->  Admin (session) -->  Convex Overview Stats
Browser -->  GET /admin/images    -->  Admin (session) -->  Convex Image Stats + Image List
Browser -->  GET /admin/files     -->  Admin (session) -->  Convex File Stats + File List
Browser -->  GET /admin/texts     -->  Admin (session) -->  Convex Text Stats + Text List
Browser -->  GET /admin/tools     -->  Admin (session) -->  Hash Generator
Browser -->  GET /content/i/{slug}-->  Cached image content (ETag, range requests)
  • Upload: ShareX sends multipart form-data (images/files) or form-urlencoded (text) with auth header
  • Storage: All binary/text content stored in Convex file storage; metadata in Convex tables
  • Serving: Public GET routes query Convex by short slug, increment views, render styled pages
  • Admin: Session-based password auth (PBKDF2-SHA256), dashboard overview + per-type pages + tools page
  • Security: Rate limiting, security headers, upload size limits, proxy trust model, admin IP allowlist

Project Structure

undefinedlabs.tech/
├── convex/                              # Convex TypeScript backend
│   ├── convex.json                      # Functions directory config
│   ├── functions/
│   │   ├── schema.ts                    # 4 tables: images, files, texts, counters
│   │   ├── auth.ts                      # INTERNAL_API_SECRET validation
│   │   ├── crud.ts                      # Shared CRUD builder (buildCrud)
│   │   ├── counters.ts                  # Pre-aggregated counter tracking
│   │   ├── images.ts                    # Image create + re-exported CRUD
│   │   ├── files.ts                     # File create + re-exported CRUD
│   │   ├── texts.ts                     # Text create + re-exported CRUD
│   │   ├── stats.ts                     # Dashboard + type-specific stats queries
│   │   ├── storage.ts                   # generateUploadUrl + deleteStorageObject
│   │   └── helpers.ts                   # Slug generation, type/table mapping
│   ├── tsconfig.json
│   └── package.json
├── src/
│   └── UndefinedLabs/                   # Blazor Server .NET 10
│       ├── Components/
│       │   ├── Layout/
│       │   │   ├── MainLayout.razor
│       │   │   └── MainLayout.razor.css
│       │   ├── Pages/
│       │   │   ├── Home.razor               # Landing page (/)
│       │   │   ├── ImageView.razor          # /i/{slug}
│       │   │   ├── FileView.razor           # /f/{slug}
│       │   │   ├── TextView.razor           # /t/{slug}
│       │   │   ├── AdminLogin.razor         # /admin
│       │   │   ├── AdminDashboard.razor     # /admin/dashboard
│       │   │   ├── AdminImages.razor        # /admin/images
│       │   │   ├── AdminFiles.razor         # /admin/files
│       │   │   ├── AdminTexts.razor         # /admin/texts
│       │   │   ├── AdminTools.razor         # /admin/tools (hash generator)
│       │   │   ├── NotFound.razor           # 404 page
│       │   │   └── Error.razor              # Error page
│       │   ├── AdminRecordList.razor        # Shared record list component
│       │   ├── AdminNavigation.razor        # Admin nav component
│       │   ├── App.razor
│       │   ├── Routes.razor
│       │   └── _Imports.razor
│       ├── Features/
│       │   ├── Admin/
│       │   │   ├── AdminAuthEndpoints.cs    # Admin login/logout endpoints
│       │   │   └── AdminAccessMiddleware.cs # Admin access control
│       │   ├── Downloads/
│       │   │   └── DownloadEndpoints.cs     # Download + content route handlers
│       │   ├── Uploads/
│       │   │   └── UploadEndpoints.cs       # Upload route handlers
│       │   └── Public/
│       │       └── NotFoundRedirectMiddleware.cs
│       ├── Services/
│       │   ├── ConvexService.cs         # HTTP API wrapper for Convex
│       │   ├── AuthService.cs           # PBKDF2-SHA256 auth (ShareX key + admin password)
│       │   ├── UploadService.cs         # Orchestrates rollback-safe upload flow
│       │   └── ShareXUploaderService.cs # ShareX .sxcu config generation
│       ├── Models/
│       │   ├── UploadRecord.cs          # Shared DTO for image/file/text records
│       │   ├── DashboardStats.cs        # Stats DTO
│       │   ├── TypeStatsResult.cs       # Per-type stats DTO
│       │   ├── UploadResponse.cs        # POST response DTO
│       │   └── FlexibleLongJsonConverters.cs # Convex integer deserialization
│       ├── Configuration/
│       │   └── UploadLimitSettings.cs   # Max upload size config
│       ├── Infrastructure/Http/
│       │   ├── RequestRouteHelpers.cs   # Route parsing for rate limiting
│       │   ├── RateLimitSettings.cs     # Rate limit config
│       │   └── TrustedProxySettings.cs  # Proxy trust config
│       ├── Utilities/
│       │   └── DisplayFormatters.cs     # FormatBytes, FormatDate helpers
│       ├── wwwroot/
│       │   ├── css/app.css              # Dark theme styles
│       │   └── js/clipboard.js          # Copy-to-clipboard interop
│       ├── Program.cs                   # DI, middleware, security, minimal API endpoints
│       ├── appsettings.json
│       └── UndefinedLabs.csproj
├── tests/
│   └── UndefinedLabs.Tests/
│       ├── AuthRouteTests.cs
│       ├── AuthServiceTests.cs
│       ├── ConvexServiceTests.cs
│       ├── UploadServiceTests.cs
│       ├── RateLimitRouteTests.cs
│       ├── ShareXUploaderServiceTests.cs
│       ├── AdminStatsDeserializationTests.cs
│       └── UndefinedLabs.Tests.csproj
├── sharex/                              # ShareX custom uploader templates
│   ├── undefinedlabs-image.sxcu
│   ├── undefinedlabs-file.sxcu
│   └── undefinedlabs-text.sxcu
├── .github/
│   └── workflows/
│       └── deploy                       # Build, test, push image to GHCR
├── Dockerfile                           # Multi-stage .NET 10 build
├── .dockerignore
├── .gitignore
├── .env.example
└── UndefinedLabs.sln

Environment Variables

Required

Variable Purpose
CONVEX_URL Self-hosted Convex backend URL (e.g., https://convex.example.com)
CONVEX_ADMIN_KEY Convex admin key for server-side HTTP API calls (Authorization: Convex <key>)
INTERNAL_API_SECRET Shared secret passed to Convex mutations for function-level authorization
PUBLIC_BASE_URL Public site base URL used to build returned upload URLs (e.g., https://undefinedlabs.tech)
  • PUBLIC_BASE_URL should be normalized by trimming a trailing slash before URL construction

Auth (PBKDF2-SHA256 hashes)

Variable Purpose
ADMIN_PASSWORD_HASH PBKDF2-SHA256 hash of admin dashboard password. Random temporary password generated at startup if not set.
SHAREX_AUTH_KEY_HASH PBKDF2-SHA256 hash of ShareX authorization key. Random temporary key generated at startup if not set.

Optional Security

Variable Default Purpose
TRUSTED_PROXY_IPS (none) Comma-separated IPs/CIDRs for trusted reverse proxies
ADMIN_IP_ALLOWLIST (none) Comma-separated IPs/CIDRs allowed to access /admin*
RATE_LIMIT_UPLOAD_PER_MINUTE 20 Max uploads per IP per minute
RATE_LIMIT_VIEW_PER_MINUTE_PER_SLUG 60 Max views per IP+slug per minute
RATE_LIMIT_DOWNLOAD_PER_MINUTE_PER_SLUG 30 Max downloads per IP+slug per minute
RATE_LIMIT_ADMIN_LOGIN_PER_15MIN 5 Max admin login attempts per IP per 15 minutes
MAX_IMAGE_MB 20 Max image upload size in MB
MAX_FILE_MB 100 Max file upload size in MB
MAX_TEXT_KB 1024 Max text upload size in KB

Convex Backend

Schema (convex/functions/schema.ts)

Three upload tables share the same shape, plus a counters table for pre-aggregated stats:

const uploadRecordFields = {
  slug: v.string(),           // 8-char alphanumeric, indexed
  storageId: v.id("_storage"),
  fileName: v.string(),
  contentType: v.string(),
  fileSize: v.number(),
  views: v.number(),          // incremented on page view
  downloads: v.number(),      // incremented on download click
};

images: defineTable(uploadRecordFields).index("by_slug", ["slug"]),
files: defineTable(uploadRecordFields).index("by_slug", ["slug"]),
texts: defineTable(uploadRecordFields).index("by_slug", ["slug"]),

counters: defineTable({
  table: v.string(),          // "images" | "files" | "texts"
  count: v.number(),
  views: v.number(),
  downloads: v.number(),
  storageBytes: v.number(),
}).index("by_table", ["table"]),
  • _creationTime is automatic in Convex (no need for a created field)
  • Text is stored as a .txt file in Convex storage
  • Each upload table has a by_slug index for fast lookups
  • counters table provides O(1) aggregate stats (no full table scans for totals)

Authorization (convex/functions/auth.ts)

  • requireApiSecret(providedSecret) - validates the INTERNAL_API_SECRET env var against the provided secret
  • Called by all mutations and admin-facing queries (list, stats) to prevent unauthorized function calls
  • Public queries (getBySlug) do not require API secret

Shared CRUD (convex/functions/crud.ts)

buildCrud(table) returns shared implementations for each upload table:

  • getBySlug query - looks up by slug index, returns record + storage URL via ctx.storage.getUrl(storageId)
  • incrementViews mutation - patches views count, adjusts counter
  • incrementDownloads mutation - patches downloads count, adjusts counter
  • list query - cursor-paginated list ordered by creation time desc (limit default 50, max 50)
  • deleteById mutation - deletes record + storage object, adjusts counter

Per-table modules (images.ts, files.ts, texts.ts)

Each module exports:

  • create mutation - generates unique 8-char slug (retry on collision), inserts record, adjusts counter, returns { slug, id }
  • Re-exports from buildCrud(): getBySlug, incrementViews, incrementDownloads, list, deleteById

Counter tracking (convex/functions/counters.ts)

  • adjustCounter(ctx, table, delta) - atomically increments/decrements counter row for a table
  • getCounter(ctx, table) - reads current counter values (count, views, downloads, storageBytes)

Slug generation (convex/functions/helpers.ts)

  • 8-char string from [A-Za-z0-9] (62^8 = ~218 trillion combinations)
  • Generated inside mutations with collision check (while loop querying by_slug index)
  • Uses Web Crypto API (crypto.getRandomValues)
  • Server-side generation ensures atomicity (no race conditions)

Stats (convex/functions/stats.ts)

  • getOverviewStats query - returns dashboard metrics from counters (image/file/text counts, total views/downloads/storage bytes) plus cross-type top records sorted by views/downloads
  • getTypeStats query - returns metrics for one type (image/file/text): counter totals + cursor-paginated records
  • getImageStatsLive / getFileStatsLive / getTextStatsLive queries - queries accepting apiSecret for WebSocket subscriptions on admin per-type pages

Storage (convex/functions/storage.ts)

  • generateUploadUrl mutation - wraps ctx.storage.generateUploadUrl() (requires API secret)
  • deleteStorageObject mutation - wraps ctx.storage.delete(storageId) for rollback cleanup (requires API secret)

Blazor App

Convex Integration (Hybrid Approach)

Two Convex clients, each used where it fits best:

1. ConvexService.cs (HTTP API wrapper) - for one-shot request/response patterns:

  • Upload endpoints (POST), download endpoints, view page SSR loads
  • POST {CONVEX_URL}/api/query with Authorization: Convex {ADMIN_KEY} (queries)
  • POST {CONVEX_URL}/api/mutation with same auth header + apiSecret arg injected (mutations)
  • Body format: { "path": "module:functionName", "args": {...}, "format": "json" }
  • Methods: QueryAsync<T>(), MutateAsync<T>(), GenerateUploadUrlAsync(), UploadFileAsync(), DownloadStreamAsync(), DownloadStringAsync(), DeleteStorageObjectAsync()
  • Storage URL normalization for self-hosted Convex (localhost → configured host)

2. ConvexClient from NuGet (Convex.Client + Convex.Client.Blazor) - for real-time dashboard:

  • Registered as singleton in DI via ConvexClientBuilder
  • Used on admin overview and per-type admin pages
  • Subscribes to stats:getOverviewStats, stats:getImageStatsLive/getFileStatsLive/getTextStatsLive via Observe<T>()
  • Uses SubscribeToUI() Blazor integration so stats, view counts, and new uploads update live without page refreshes
  • Subscriptions disposed when leaving any admin stats page
  • No admin auth on WebSocket (public queries work without auth)
  • Prerender disabled on admin pages (InteractiveServerRenderMode(prerender: false))

Upload Flow (all 3 types)

1. Validate auth header against SHAREX_AUTH_KEY_HASH (PBKDF2-SHA256)
2. Validate content-type and file size against upload limits
3. For images: validate magic bytes (JPEG, PNG, GIF, BMP, WebP signatures)
4. Call storage:generateUploadUrl mutation → get presigned URL
5. Upload file bytes to presigned URL → get storageId
   (for text: convert string to UTF-8 byte stream, content-type text/plain)
6. Call create mutation:
   - images/files: `{ storageId, fileName, contentType, fileSize }`
   - texts: `{ storageId, contentType, fileSize }` and set stored filename inside `texts:create` to `text-{slug}.txt`
7. If step 6 fails, call storage:deleteStorageObject(storageId), return error, and stop
8. If step 6 partially succeeds then downstream server handling fails, call {type}:deleteById(id) and storage:deleteStorageObject(storageId), return error
9. Cleanup calls are best-effort; if cleanup also fails, log cleanup failure details and return 500
10. Return JSON URL built from PUBLIC_BASE_URL:
   - image upload (`POST /i/`): `{ "url": "{PUBLIC_BASE_URL}/i/{slug}" }`
   - file upload (`POST /f/`): `{ "url": "{PUBLIC_BASE_URL}/f/{slug}" }`
   - text upload (`POST /t/`): `{ "url": "{PUBLIC_BASE_URL}/t/{slug}" }`

Minimal API Endpoints

Upload (auth required, rate limited):

  • POST /i/ - multipart form-data → image upload (magic-byte validated)
  • POST /f/ - multipart form-data → file upload
  • POST /t/ - form post with text field (fallback to content field) → text upload (filename generated server-side as text-{slug}.txt)

Content (public, cached):

  • GET /content/i/{slug} - lookup slug; stream image with original content type, 1-year immutable cache, ETag, range request support (used by <img> tags on view pages)

Download (public, rate limited):

  • GET /download/i/{slug} - lookup slug; if found increment downloads once, stream as application/octet-stream
  • GET /download/f/{slug} - same pattern
  • GET /download/t/{slug} - lookup slug; if found increment downloads once, serves content as .txt file

Admin (session-protected):

  • POST /admin/login - validates password hash, sets session (anti-forgery validated)
  • POST /admin/logout - clears session (anti-forgery validated)

API docs (admin-protected):

  • GET /admin/api-reference - Scalar API reference UI
  • GET /admin/openapi/v1.json - OpenAPI spec

Blazor Pages

Home (/) - Landing page: site name, tagline "Personal file hosting", dark themed, minimal. No links to admin or upload routes.

ImageView (/i/{slug}) - Queries images:getBySlug; if found increments views once per request. Renders: modal-style centered image card, filename, upload date, file size, view/download counts, download button linking to /download/i/{slug}. 404 component if not found.

FileView (/f/{slug}) - Same pattern. Renders: file icon, filename, content type, size, dates, download button. No inline preview.

TextView (/t/{slug}) - Queries texts:getBySlug, fetches text content from storage URL; if found increments views once per request. Renders: gist-like styled card with monospace text in centered area, filename at top, copy button (JS interop for clipboard), download button linking to /download/t/{slug}. Dark background code block aesthetic.

AdminLogin (/admin) - Password input form. POST to /admin/login with anti-forgery token, on success set session and redirect to dashboard.

AdminDashboard (/admin/dashboard) - Checks session auth, redirects to /admin if not authenticated. Uses ConvexClient real-time subscriptions (via Convex.Client.Blazor) and shows:

  • Stat cards: image count, file count, text count, total views, total downloads, storage used
  • Cross-type top records table (top 10 by views/downloads)
  • Links to detailed pages: /admin/images, /admin/files, /admin/texts
  • Callouts auto-update in real-time when uploads, views, or downloads change

AdminImages (/admin/images) - Auth required. Uses AdminRecordList shared component. Shows image-specific totals (items, views, downloads, storage bytes) and cursor-paginated image records (50 per page, load more). Real-time updates via WebSocket subscription.

AdminFiles (/admin/files) - Auth required. Same pattern as AdminImages.

AdminTexts (/admin/texts) - Auth required. Same pattern as AdminImages.

AdminTools (/admin/tools) - Auth required. PBKDF2-SHA256 hash generator for credential rotation. Generates hashes for ADMIN_PASSWORD_HASH and SHAREX_AUTH_KEY_HASH env vars. Also supports ShareX uploader configuration regeneration.

Auth

  • ShareX uploads: Authorization header validated against SHAREX_AUTH_KEY_HASH using PBKDF2-SHA256. Supports both Bearer <key> and raw key formats. Returns 401 on failure.
  • Admin dashboard: Server-side session (DistributedMemoryCache, sufficient for single-instance Docker). Password validated against ADMIN_PASSWORD_HASH using PBKDF2-SHA256. Session hardened with HttpOnly, SameSite=Strict, Secure (non-dev), path restricted to /admin. Absolute session timeout of 8 hours.
  • Default credentials: If hash env vars are not set, random temporary passwords/keys are generated at startup and logged as warnings with sample hashes.
  • Admin IP allowlist: Optional ADMIN_IP_ALLOWLIST restricts /admin* access to specific IPs/CIDRs.
  • Public views: GET routes for /i/, /f/, /t/ require no auth.

Security

  • Rate limiting: ASP.NET Core AddRateLimiter with route-aware global partitions (upload per-IP, view/download per-IP+slug, admin login per-IP stricter). Returns 429 with JSON body.
  • Security headers: X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy, Content-Security-Policy.
  • Upload validation: Configurable max sizes per type; image magic-byte validation (JPEG, PNG, GIF, BMP, WebP).
  • Download safety: Forced application/octet-stream content type to prevent stored XSS.
  • Anti-forgery: Enabled on admin login/logout forms.
  • Proxy trust: TRUSTED_PROXY_IPS controls forwarded header trust; warning logged if unset in production.

Docker

Dockerfile (multi-stage)

Stage 1 (sdk:10.0): restore → build → publish
Stage 2 (aspnet:10.0): copy published output, expose 8080, ENTRYPOINT dotnet

CI/CD (GitHub Actions)

.github/workflows/ci.yml runs on pushes to master and PRs:

  1. build-and-test job — restore, build, and test the .NET solution
  2. push-image job (master pushes only, after tests pass) — builds the Docker image and pushes to ghcr.io/kyle-undefined/undefinedlabs.tech tagged with latest and the short commit SHA

Dokploy pulls the latest image from GHCR. The built-in GITHUB_TOKEN handles GHCR authentication (no additional secrets needed).

Convex deployment is separate

The convex/ directory functions are deployed via npx convex deploy (locally or CI), not inside the Docker build. The Docker container only runs the .NET Blazor app.

Verification

  1. Convex functions: Deploy to self-hosted instance with npx convex deploy, verify via Convex dashboard
  2. Upload test: curl -X POST -H "Authorization: Bearer <key>" -F "file=@test.png" http://localhost:8080/i/ - should return JSON with URL
  3. Text upload test: curl -X POST -H "Authorization: Bearer <key>" -F "text=hello" http://localhost:8080/t/
  4. View test: Open returned URL in browser, verify styled page with image/file/text content
  5. Download test: Click download button, verify file downloads correctly and download counter increments
  6. Auth test: POST without auth header → 401. Navigate to admin without session → redirect to login.
  7. Rate limit test: Exceed configured limits → 429 with JSON error body.
  8. Admin overview test: Login with correct password, verify dashboard callouts show total counts, top records, and links to each type page
  9. Per-type admin test: Verify /admin/images, /admin/files, /admin/texts each show type-specific totals and records with live updates
  10. Pagination test: Verify each type page shows latest 50 by default and can request older pages
  11. Rollback test: Force {type}:create failure after storage upload, verify uploaded storage object is deleted and no orphan record remains
  12. Cleanup-failure test: Simulate cleanup failure, verify error is logged and 500 is returned
  13. Docker test: docker build -t undefinedlabs . && docker run, run all above tests against container
  14. Unit tests: dotnet test passes all tests (auth, upload service, Convex service, rate limits, ShareX config)