docs, api, other stuff update (#101)

* docs, api, other stuff update
* fixes

Author: HardMax71

backend/.env

@@ -29,14 +29,12 @@ KAFKA_HEARTBEAT_INTERVAL_MS=10000
 KAFKA_REQUEST_TIMEOUT_MS=40000
 KAFKA_MAX_POLL_RECORDS=500
 
-# WebSocket Configuration
-WEBSOCKET_PING_INTERVAL=30
-WEBSOCKET_PING_TIMEOUT=10
+# SSE Configuration
+SSE_CONSUMER_POOL_SIZE=10
+SSE_HEARTBEAT_INTERVAL=30
 
 # Logging Configuration
 LOG_LEVEL=DEBUG
-WEBSOCKET_MAX_CONNECTIONS_PER_USER=5
-WEBSOCKET_STALE_CONNECTION_TIMEOUT=300
 
 # Distributed Tracing
 ENABLE_TRACING=true

backend/.env.test

@@ -30,14 +30,12 @@ KAFKA_HEARTBEAT_INTERVAL_MS=3000
 KAFKA_REQUEST_TIMEOUT_MS=15000
 KAFKA_MAX_POLL_RECORDS=500
 
-# WebSocket Configuration
-WEBSOCKET_PING_INTERVAL=30
-WEBSOCKET_PING_TIMEOUT=10
+# SSE Configuration
+SSE_CONSUMER_POOL_SIZE=10
+SSE_HEARTBEAT_INTERVAL=30
 
 # Logging Configuration
 LOG_LEVEL=WARNING
-WEBSOCKET_MAX_CONNECTIONS_PER_USER=5
-WEBSOCKET_STALE_CONNECTION_TIMEOUT=300
 
 # Distributed Tracing
 ENABLE_TRACING=true

backend/app/settings.py

@@ -112,10 +112,6 @@ class Settings(BaseSettings):
     # App URL for notification links
     APP_URL: str = "https://integr8scode.cc"
 
-    # WebSocket configuration
-    WEBSOCKET_PING_INTERVAL: int = 30
-    WEBSOCKET_PING_TIMEOUT: int = 10
-
     # Redis Configuration
     REDIS_HOST: str = "redis"
     REDIS_PORT: int = 6379
@@ -131,8 +127,6 @@ class Settings(BaseSettings):
     RATE_LIMIT_BURST_MULTIPLIER: float = 1.5
     RATE_LIMIT_REDIS_PREFIX: str = "rate_limit:"
     RATE_LIMIT_ALGORITHM: str = "sliding_window"  # sliding_window or token_bucket
-    WEBSOCKET_MAX_CONNECTIONS_PER_USER: int = 5
-    WEBSOCKET_STALE_CONNECTION_TIMEOUT: int = 300
 
     # Service metadata
     SERVICE_NAME: str = "integr8scode-backend"

docs/SECURITY.md

@@ -1,62 +1,19 @@
 # Security Policy
 
-Integr8sCode takes the security of our software and our users' data seriously. We are committed to ensuring a secure
-environment and following best practices for vulnerability management and disclosure.
+Security patches go into `main` and the latest release. If you're running something older, upgrade.
 
-## Supported Versions
+## Reporting vulnerabilities
 
-We currently support security updates for the following versions of Integr8sCode:
+Found a security issue? Don't open a public GitHub issue - email [max.azatian@gmail.com](mailto:max.azatian@gmail.com) instead.
 
-| Version        | Supported          |
-|----------------|--------------------|
-| `main`         | :white_check_mark: |
-| Latest Release | :white_check_mark: |
+Include what you can: vulnerability type, where it occurs, reproduction steps, PoC if you have one. You'll get an acknowledgment within 48 hours. If confirmed, we'll patch it and credit you in the disclosure (unless you prefer to stay anonymous).
 
-If you are running an older version, we strongly recommend upgrading to the latest release to ensure you have the most
-recent security patches.
+## Automated scanning
 
-## Reporting a Vulnerability
+The CI pipeline runs [Bandit](https://bandit.readthedocs.io/) on the Python backend for static analysis, and [Dependabot](https://docs.github.com/en/code-security/dependabot) keeps dependencies patched across Python, npm, and Docker. For SBOM generation and vulnerability scanning, see [Supply Chain Security](security/supply-chain.md).
 
-If you discover a security vulnerability within Integr8sCode, please **DO NOT** create a public GitHub issue. Instead,
-please report it privately to our security team.
+## Runtime hardening
 
-### How to Report
+Executor pods run user code with non-root users, read-only filesystems, dropped capabilities, and no service account tokens. Network policies deny all traffic by default. Details in [Network Isolation](security/policies.md).
 
-1. **Email:** Send a detailed report to <mailto:max.azatian@gmail.com>.
-2. **Details:** Please include as much information as possible:
-    * Type of vulnerability (e.g., XSS, SQL Injection, RCE).
-    * Full path or URL where the vulnerability occurs.
-    * Step-by-step instructions to reproduce the issue.
-    * Proof of concept (PoC) code or screenshots, if available.
-    * Any specific configuration required to reproduce the issue.
-
-### Our Response Process
-
-1. **Acknowledgment:** We will acknowledge receipt of your report within 48 hours.
-2. **Assessment:** We will investigate the issue to confirm its validity and impact.
-3. **Resolution:** If confirmed, we will work on a patch. We will keep you updated on our progress.
-4. **Disclosure:** Once a fix is released, we will publicly disclose the vulnerability (with your permission, crediting
-   you for the discovery).
-
-## Security Measures
-
-We employ several automated tools and practices to maintain the security of our codebase:
-
-* **Static Application Security Testing (SAST):** We use **Bandit** to scan our Python backend code for common security
-  issues.
-* **Dependency Management:** We use **Dependabot** to automatically monitor and update vulnerable dependencies in our
-  `package.json`, `pyproject.toml`, and Docker files.
-* **Container Security:** We follow best practices for containerization, including using minimal base images and
-  non-root users where possible.
-* **Secrets Management:** We do not commit secrets to the repository. Please ensure `.env` files and other secrets are
-  properly managed in your deployment environment.
-
-## Software Bill of Materials (SBOM)
-
-We strive to maintain transparency regarding our dependencies. You can inspect our direct dependencies in:
-
-* `backend/pyproject.toml` (Python)
-* `frontend/package.json` (Node.js/Svelte)
-* `helm/integr8scode/Chart.yaml` (Kubernetes/Helm)
-
-Thank you for helping keep Integr8sCode safe!
+Secrets stay out of the repo - `.env` files and credentials are your responsibility to manage in deployment.

docs/components/sse/sse-architecture.md

@@ -83,6 +83,37 @@ Key metrics: `sse.connections.active`, `sse.messages.sent.total`, `sse.connectio
 
 WebSockets were initially implemented but removed because SSE is sufficient for server-to-client communication, simpler connection management, better proxy compatibility (many corporate proxies block WebSockets), excellent browser support with automatic reconnection, and works great with HTTP/2 multiplexing.
 
+## Testing SSE endpoints
+
+Testing SSE endpoints in E2E tests is tricky. Our tests use `httpx.AsyncClient` with `ASGITransport` to run against the FastAPI app in-process without starting a real server. The problem is that httpx's ASGITransport buffers entire responses before returning, which causes tests to hang indefinitely when streaming SSE — the response never "completes" because SSE streams are infinite by design.
+
+This is a [known limitation](https://github.com/encode/httpx/issues/2186) with no fix in sight (there's an open PR since January 2024 but it hasn't been merged).
+
+The solution is [async-asgi-testclient](https://github.com/vinissimus/async-asgi-testclient), an alternative ASGI test client that properly handles streaming. It's added as a dev dependency and used alongside httpx — httpx for regular requests, async-asgi-testclient for SSE endpoints.
+
+The test fixtures copy authentication cookies and CSRF tokens from the httpx client to the SSE client, so both share the same session:
+
+```python
+@pytest_asyncio.fixture
+async def sse_client(app: FastAPI, test_user: AsyncClient) -> SSETestClient:
+    client = SSETestClient(app)
+    for name, value in test_user.cookies.items():
+        client.cookie_jar.set(name, value)
+    if csrf := test_user.headers.get("X-CSRF-Token"):
+        client.headers["X-CSRF-Token"] = csrf
+    return client
+```
+
+Tests then verify that SSE endpoints return the correct content-type and stream data:
+
+```python
+async def test_notification_stream(self, sse_client: SSETestClient) -> None:
+    async with sse_client:
+        response = await sse_client.get("/api/v1/events/notifications/stream", stream=True)
+        assert response.status_code == 200
+        assert "text/event-stream" in response.headers.get("content-type", "")
+```
+
 ## Key files
 
 | File                                                                                                                              | Purpose                |

docs/getting-started.md

@@ -0,0 +1,114 @@
+# Getting Started
+
+This guide walks you through running Integr8sCode locally and executing your first script. You'll need Docker and Docker Compose installed - that's it.
+
+## What you're deploying
+
+The full stack includes a Svelte frontend, FastAPI backend, MongoDB, Redis, Kafka with Schema Registry, and seven background workers. Sounds like a lot, but `docker compose` handles all of it. First startup takes a few minutes to pull images and initialize services; subsequent starts are much faster.
+
+## Start the stack
+
+```bash
+git clone https://github.com/HardMax71/Integr8sCode.git
+cd Integr8sCode
+./deploy.sh dev
+```
+
+Wait for the services to come up. You can watch progress with `docker compose logs -f` in another terminal. When you see the backend responding to health checks, you're ready.
+
+Verify everything is running:
+
+```bash
+curl -k https://localhost/api/v1/health/live
+```
+
+If everything is up, you'll get:
+
+```json
+{"status":"ok","uptime_seconds":42,"timestamp":"2025-01-25T12:00:00Z"}
+```
+
+## Access the UI
+
+Open [https://localhost:5001](https://localhost:5001) in your browser. You'll get a certificate warning since it's a self-signed cert - accept it and continue.
+
+Log in with the default credentials:
+
+- **Username:** `user`
+- **Password:** `user123`
+
+For admin access (user management, system settings, event browser), use `admin` / `admin123`.
+
+!!! warning "Development only"
+    These credentials are seeded automatically for local development. For production, set `DEFAULT_USER_PASSWORD` and `ADMIN_USER_PASSWORD` environment variables before deploying.
+
+## Run your first script
+
+The editor opens with a Python example. Click **Run** to execute it. You'll see output streaming in real-time as the script runs in an isolated Kubernetes pod (or Docker container in dev mode).
+
+Try modifying the script or switching languages using the dropdown. Each execution gets its own isolated environment with configurable resource limits.
+
+## What's happening under the hood
+
+When you click Run:
+
+1. The frontend sends your script to the backend API
+2. Backend validates it, creates an execution record, and publishes an event to Kafka
+3. The coordinator worker picks up the event and starts a saga
+4. K8s worker spins up an isolated pod with the appropriate runtime
+5. Pod monitor watches for completion and captures output
+6. Result processor stores the result and publishes it to Redis
+7. Your browser receives the result via Server-Sent Events
+
+All of this happens in a few seconds. The event-driven architecture means you can scale each component independently and replay events for debugging.
+
+## Explore the stack
+
+With the stack running, you can poke around:
+
+| Service | URL | What it shows |
+|---------|-----|---------------|
+| Kafdrop | [http://localhost:9000](http://localhost:9000) | Kafka topics and messages |
+| Grafana | [http://localhost:3000](http://localhost:3000) | Metrics dashboards (admin/admin123) |
+| Jaeger | [http://localhost:16686](http://localhost:16686) | Distributed traces |
+
+## Troubleshooting
+
+**CPU/memory metrics show as `null`**
+
+The metrics server isn't installed by default in most local Kubernetes setups. Enable it:
+
+```bash
+kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
+```
+
+**Certificate warnings in browser**
+
+Expected - the stack uses self-signed certs for local development. Accept the warning and continue.
+
+**Services failing to start**
+
+Check logs for the specific service: `docker compose logs backend` or `docker compose logs kafka`. Most issues are either port conflicts (something else using 443, 5001, or 9092) or Docker running out of resources.
+
+**Kafka connection errors**
+
+Kafka takes longer to initialize than most services. Give it a minute after `deploy.sh dev` finishes, or watch `docker compose logs kafka` until you see "started" messages.
+
+## Stop the stack
+
+```bash
+./deploy.sh down
+```
+
+To also remove persistent data (MongoDB, Redis volumes):
+
+```bash
+docker compose down -v
+```
+
+## Next steps
+
+- [Architecture Overview](architecture/overview.md) - How the pieces fit together
+- [Deployment](operations/deployment.md) - Production deployment with Helm
+- [API Reference](reference/api-reference.md) - Full endpoint documentation
+- [Environment Variables](reference/environment-variables.md) - Configuration options