Skip to content

dalal.md

Latest commit

 

History

History
202 lines (133 loc) · 18.4 KB

dalal.md

File metadata and controls

202 lines (133 loc) · 18.4 KB
Raw

🗺️ Project System Architecture Manifest & Safety Blueprint

This manifest maps the components of the Metaval project. It documents file boundaries, routing paths, and architectural vulnerabilities to enforce strict developer safeguards and eliminate technical clutter.

💻 1. Client-Side Workspace (React SPA Engine)

The frontend is a styled dashboard tracking real-time layout telemetry data and streaming administrative requests to the local server gateway.

📌 client/src/App.jsx

  • Primary Functionality: Core state lifecycle manager for the user interface. Coordinates modal toggles, table selection switches, background data extraction alerts, and global clock intervals.
  • Behavioral Rules: Runs synchronous useEffect intervals to handle background layout metrics checks.
  • [cite_start]🚨 Safety Safeguard Alert: When validating query dispatch updates via handleQuerySubmit, it must map to the original user input structure parameters (queryData.name, queryData.email, queryData.message)[cite: 71, 72]. Changing these fields without syncing your inline form properties (QueryForm.jsx) will throw layout blocking dialogues.

📌 client/src/api/axios.js

  • Primary Functionality: Central HTTP client instantiation. Employs a flat connection base URL targeting the local server port.
  • Network Mapping Mapping Rules: ```javascript baseURL: "http://localhost:5000"
  • 🚨 Safety Safeguard Alert: Do NOT append path folders like /api or /api/data inside this baseline client configuration file. Doing so while passing manual paths from your files will cause route duplications and trigger immediate 404 Not Found network crashes.

📌 client/src/components/WorkbookTable.jsx

  • Primary Functionality: High-performance tabular datagrid ledger viewer. Renders column variables dynamically based on table catalogs polled directly from the live database schema.

📌 client/src/components/NotificationBell.jsx

  • Primary Functionality: Dropdown interactive UI alert manager. Listens for incoming system admin tickets and handles real-time response inputs.

🛢️ 2. Backend Gateway Core (Node.js & Express API Server)

The application engine exposes endpoints that match front-end requests, discover database shapes dynamically, and call parsing binaries.

📌 server/app.js

  • Primary Functionality: Primary server entry point configuration loader. Initializes global CORS permissions, activates JSON serialization hooks, and mounts your primary application route modules under a unified folder prefix.
  • Middleware Route Map Rule: 
    app.use('/api/data', dataRoutes);

📌 server/routes/dataRoutes.js

  • Primary Functionality: Centralized route mapper. Matches frontend Axios client calls directly to your dataController methods using clean endpoints.
  • Active Route Maps:
    • GET /dashboard ➔ getDashboardData
    • POST /upload ➔ uploadPDF
    • GET /workbook/:tableName ➔ getWorkbookTableData
    • POST /query ➔ submitQuery
    • POST /reply ➔ replyQuery
    • GET /export ➔ exportExcel

📌 server/controllers/dataController.js

  • Primary Functionality: Core execution logic hub. Reads incoming payloads, inspects active database constraints dynamically, and commits parameters safely to your relational ledger tables.
  • 🚨 Safety Safeguard Alert (Response Duplication Bug): If you execute local sub-process routines inside your controllers (like spawning background scripts), you must explicitly run single return res.json(...) blocks inside your callbacks. Attempting to pass error text blocks followed by trailing fallback streams will throw an unhandled ERR_HTTP_HEADERS_SENT server crash.

📌 server/config/db.js

  • Primary Functionality: PostgreSQL client connection allocator. Pools credentials and manages high-concurrency transactional pipelines.

🔬 3. Independent Modular Core Services

Isolated procedural utilities that handle file conversion, file writing, and string positioning matching scripts.

📌 server/services/pdfService.js

  • Primary Functionality: Primary document text ingestion. Leverages layout parsers (pdf-parse) to convert selectable digital character strings into readable text vectors.

📌 server/services/strategyMatcher.js

  • Primary Functionality: Heuristic structural position layout matching. Maps loose text elements into correct database cells based on row index coordinates.

📌 server/services/excelService.js

  • Primary Functionality: Data download generation layer. Converts raw arrays straight into structured Excel sheets, avoiding unstable CSV modifications.

🚫 Critical Code of Conduct Safeguards (How to Prevent Disasters)

  1. [cite_start]Never Hard-Bind Database Schemas: Your worksheets are highly dynamic (over 129 columns for tables like ofa_upload)[cite: 462]. Your code must always run dynamic lookups against information_schema.columns to find active layout shapes rather than using static indices or hard-coded insertion strings[cite: 466].
  2. [cite_start]Protect the Event Loop on Large Files: In the future, as you scale toward processing 500-page logistics sheets, processing heavy visual calculations synchronously inside the main API execution thread will freeze your server[cite: 796]. [cite_start]Long-running document processes should be handled asynchronously via a background task framework[cite: 855].
  3. Validate Data Types Prior to Query Execution: If a string parser pulls text context (like "To be confirmed by vendor data") and tries to push it into a strict PostgreSQL NUMERIC table cell, the engine will drop the query. Enforce string filtering layers in your controller to keep your columns safe.

problems

Here's a full risk map and a fix order that won't leave you rebuilding the same thing twice. I've grouped issues by what breaks first if ignored, then sequenced the phases so each one is a stable foundation for the next.

Why this order matters

If you jump straight to "add tests" or "make it scale" while buildWorkbookSearchQuery is undefined and half the routes have no auth, you'll be debugging phantom failures that have nothing to do with what you're actually working on. The sequence below is: make it run correctly → make it safe → make it reliable → make it observable → make it fast/scalable → make it maintainable. Each phase assumes the previous one is done.

Phase 0 — Stop the bleeding (things that currently crash or silently fail)

Fix these before touching anything else — they'll sabotage your testing of every later phase.

Issue Where Fix
buildWorkbookSearchQuery is called but never defined dataController.js (exportExcel, getWorkbookTableData) Write it: build a parameterized WHERE clause from req.query against the known column list. Until this exists, /export and /workbook/:tableName throw.
pool used without import dataRoutes.js (/job-status/:jobId), emailStats.js Import pool from config/db, or move the logic into dataController.js where pool already exists.
strategy_core.py source is missing server/workers/pipeline_runner.py Recover it from wherever it was authored, or rewrite NumericField/DropdownField/StandardTextField from scratch — the pipeline cannot run without it.
aiController.js doesn't exist routes/airoutes.js Either implement it or delete the route file — a route pointing at nothing is a landmine for whoever touches it next.
Case-sensitive import mismatches deltaPoller.js → ./deltaService vs DeltaService.js; uislice.ts vs uiSlice import Rename files or fix imports to match exactly. This works today only because your OS filesystem is case-insensitive — it will break the moment you deploy to Linux/CI.
Two dashboard-fetch paths, two query clients features/dashboard/services/* (mocked) vs services/dashboardService.ts (real); app/queryClient.ts vs lib/queryClient.ts Delete the unused one of each pair now, before more code starts depending on the wrong one.

How to know you're done: every route in api-contracts.md marked 🔴 is either fixed or deliberately removed, and npm run dev at the root runs both apps with zero console errors on a cold start.

Phase 1 — Security (do this before any real data touches the system)

This is an enterprise project handling customer RFQs and email content — treat auth and injection risk as blocking, not "nice to have."

  1. Apply authenticate (and allowRoles where relevant) to every route that isn't public. Right now only /api/documents/* is protected. /api/data/* (including file upload!) and /api/review/* (approve/reject!) are wide open. Anyone who can reach port 5000 can upload arbitrary files or approve fabricated engineering specs into ofa_upload.
  2. Lock down CORS for real environments. app.js hard-codes localhost:5173/5174. Move this into an env-driven allowlist before you have a staging/prod domain.
  3. Re-audit the dynamic-table pattern. verifyTableExistsDynamically is a genuinely good idea — keep it — but double-check every place a table/column name gets string-interpolated after validation. Values must always go through $1, $2...; only validated identifiers get interpolated.
  4. Secrets hygiene. JWT_SECRET, GRAPH_CLIENT_SECRET, DB creds all live in .env files that are .gitignore'd — good. But confirm none of them were ever committed in an earlier commit (git log -p -- server/.env), and rotate anything that was.
  5. File-upload hardening. uploadMiddleware.js accepts any file type/size via Multer with no filter or size limit. Add MIME/extension whitelisting (.pdf, .zip only) and a limits: { fileSize } cap — otherwise /api/data/upload is an open door to disk-fill or malicious file attacks.
  6. Remove debug logging of secrets. config/msal.ts logs VITE_CLIENT_ID/VITE_TENANT_ID to the console — low risk but sloppy; strip before shipping.
  7. Rate-limit public-ish endpoints (/auth/login, /api/data/upload) — nothing currently stops a brute-force or flood.

How to know you're done: run through api-contracts.md and mark which endpoints require auth vs. are intentionally public — nothing should be "open" by accident.

Phase 2 — Data integrity & reliability

This is where "enterprise-grade" actually gets tested — money/engineering decisions depend on this data being correct.

  1. Consolidate the schema into migrations. Right now schema lives ad hoc: sql/ofa_upload.sql (drops and recreates!), an empty models/schema.sql, and several tables (document_jobs, review_queue, system_state, users, queries, status_ledger) that are only ever implied by query code, never defined anywhere. Introduce a migration tool (node-pg-migrate, Knex, or Prisma migrate) and check every table into version control. DROP TABLE ... CASCADE in a script that could run against a live DB is a production incident waiting to happen — remove that from any script that isn't explicitly a "reset dev DB" tool.
  2. Wrap multi-step DB writes in transactions. approveReview() does an INSERT into ofa_upload followed by an UPDATE on review_queue as two separate queries — if the second fails, you've committed a spec row with no corresponding review record. Wrap in BEGIN/COMMIT/ROLLBACK.
  3. Make the mail-delta polling idempotent and crash-safe. deltaPoller.js runs an infinite while(true) loop with no process supervision. If the Node process dies mid-cycle, you rely entirely on the saved Graph cursor to avoid re-processing — verify saveCursor only happens after a batch is fully queued, not before, or you can lose documents on a crash between "cursor saved" and "job queued."
  4. Add retry/dead-letter handling to the BullMQ worker. Currently a failed job just throws and BullMQ's defaults apply. For an enterprise pipeline, configure explicit attempts, backoff, and a dead-letter queue/alert so a permanently-failing PDF doesn't silently vanish from document_jobs as FAILED with nobody watching.
  5. Extend approveReview's column list. It only inserts 19 of ofa_upload's ~129 columns — confirm this is intentional (partial approval flow) or a gap that's silently dropping extracted data.
  6. Handle the Python service being down. documentQueue.js's worker POSTs to PYTHON_API_URL with no visible timeout/circuit-breaker — a hung external service will hang worker slots. Add a request timeout and treat timeout as a retryable failure, not a silent hang.

How to know you're done: you can kill the server mid-poll, mid-upload, and mid-review-approve, restart it, and nothing is duplicated, lost, or left in an inconsistent state.

Phase 3 — Observability (you can't fix what you can't see)

Do this before scaling, because scaling problems are invisible without it.

  1. Structured logging. Replace the emoji console.logs with a real logger (pino or winston) that outputs JSON, includes request IDs, and can be shipped to a log aggregator (Datadog, ELK, CloudWatch). Keep the operational clarity the current logs have — just make it machine-parseable too.
  2. Health/readiness endpoints. GET / exists as a liveness check; add a real readiness check that verifies DB, Redis, and (optionally) the Python service are reachable — this is what a load balancer/orchestrator (k8s, ECS) needs to route traffic correctly.
  3. Metrics. Track: queue depth/age (document_jobs stuck in PROCESSING too long), Graph API error rates per mailbox, upload volume, review-queue backlog. Even a simple /metrics Prometheus endpoint pays off fast.
  4. Alerting on the failure modes you just made visible in Phase 2 — a mailbox whose cursor keeps expiring, a worker that's failing every job, a review queue that's not being cleared.

Phase 4 — Testing & CI/CD

There is currently no test suite and no CI in either package.json. For an enterprise system this is the single biggest thing standing between you and confident deploys.

  1. Unit tests first, on the highest-risk logic: verifyTableExistsDynamically, buildWorkbookSearchQuery (once written), hasKeyword, the review approve/reject flow, normalizeValue.
  2. Integration tests for the mail pipeline against a mocked Graph client — this is the part most likely to silently regress since it only runs in a loop with no direct test coverage.
  3. API contract tests for every route in api-contracts.md — you already have the contract documented, turn it into supertest assertions.
  4. CI pipeline (GitHub Actions or similar): lint (frontend already has ESLint configured — wire it into CI), run tests, build both apps, and — importantly — run npm audit/pnpm audit for dependency vulnerabilities on every PR.
  5. Only after tests exist, start doing the case-sensitivity/dead-code/duplicate-module cleanup from Phase 0/coding-guidelines — tests are what let you refactor without fear.

Phase 5 — Scalability & performance

Now that it's correct, observable, and tested, make it handle real load.

  1. Horizontal scaling of the BullMQ worker. concurrency: 2 is hard-coded for one process — plan for running multiple worker instances/pods pointed at the same Redis, and make sure DB row locking (FOR UPDATE where needed) prevents two workers racing on the same job.
  2. Connection pool sizing. max: 20 on the Postgres pool is a reasonable default for one instance — revisit once you have multiple app instances, since pools multiply per instance.
  3. Machine-2 dependency. checkMachine2() pinging a hard-coded IP (192.168.11.239) is not production infra — replace with a proper service discovery mechanism or at minimum an env var, and add retry/backoff instead of a one-shot boot-time check.
  4. Pagination everywhere. GET /api/data/all and others return unbounded result sets — add LIMIT/OFFSET or cursor pagination before a table grows large enough to make that endpoint a liability.
  5. Frontend production build config — env-driven API base URL (already scaffolded via ENV.API_BASE_URL, just make sure it's actually set per environment), code-splitting for the feature folders as they get built out, and a CDN for static assets.

Phase 6 — Compliance & governance (enterprise-specific)

  • PII/data retention policy for email content and customer data flowing through document_jobs/ofa_upload — decide how long raw PDFs and email metadata are retained, and automate cleanup.
  • Audit trail for review approve/reject actions — you're currently storing reviewer_name as a free-text field the client sends; derive it from the authenticated JWT (req.user) instead, so it can't be spoofed, and log every state transition.
  • Backup/DR plan for Postgres — nothing in the repo currently addresses this; it needs to exist before this holds real customer commitments.
  • Role model completion. middleware/authenticate.js already sketches future roles (SUPER_ADMIN, AUDITOR, etc.) in a comment — formalize this into an actual permissions table/matrix rather than the current three flat roles, especially since approve/reject on customer specs is a sensitive action.

Phase 7 — Code quality / tech-debt cleanup

Do this last, or continuously in small batches — it's important but non-blocking, and premature cleanup on code you haven't stabilized yet just creates merge conflicts with the phases above.

  • Normalize CRLF→LF line endings across server/.
  • Standardize the error vs message response key.
  • Delete the large commented-out code blocks (app.js, App.tsx, DashboardPage.tsx, etc.) — trust Git history instead.
  • Finish or delete the empty scaffolded files (authMiddleware.js, permissionService.js, most store/slices/*, theme/mui/palette.ts, etc.) so it's clear what's "not built yet" vs. "abandoned."

The one-sentence version of the sequence

Fix what's broken → lock down who can do what → make writes correct and crash-safe → make failures visible → prove it with tests/CI → then and only then optimize for scale → clean up debt continuously in the gaps.

If you want, I can turn this into a tracked checklist (e.g. a markdown file or a project board format) you can check off as you go, or go deep on any one phase — Phase 1 (security) and Phase 2 (data integrity) are the two I'd start on literally today given this handles real customer RFQ data.