Skip to content
Merged
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- **Inbound webhook receivers** (#155): external systems (CI/CD, uptime monitors, arbitrary tools) can now push events into LogTide through per-receiver tokenized endpoints (`POST /api/v1/receivers/:id/:token`, `lr_`-prefixed tokens stored as SHA-256 hashes, timing-safe compare; the URL is the credential since GitHub/Uptime Robot cannot send custom headers). Three adapters normalize payloads into log entries: GitHub (workflow_run and deployment_status events with conclusion-to-level mapping; ping and unsupported events marked skipped), Uptime (shape-detects Uptime Robot alertType and Better Stack incident payloads), and Generic JSON (optional dot-path field mapping with levelMap and defaults, validated by a shared Zod schema; full payload always preserved in metadata). The endpoint answers 202 and processing is asynchronous via a new `receiver-events` queue job (queue abstraction, both backends), which runs the adapter, validates against `logSchema` and ingests through `ingestionService.ingestLogs`, so PII masking (fail-closed), usage quotas, Sigma detection, pipelines, metering and live tail all apply automatically; the outcome (processed/skipped/failed, normalized output, error) is recorded on a `receiver_events` row pruned to the last 100 per receiver. Management: project-scoped CRUD + recent-events API under `/api/v1/projects/:projectId/receivers` (session auth, audit-logged as `receiver.created`/`receiver.deleted`), a new `receivers.max` capability limit enforced with the canonical withLimitLock pattern (4-case + race tests), and a "Webhook Receivers" section in project settings (create dialog with adapter picker and generic field-mapping inputs, one-time URL reveal with copy, enable/disable switch, recent-events viewer with raw/normalized JSON). Migration 052 (`receivers`, `receiver_events`); public docs in `docs/receivers.md`
- **Soft-delete for projects with a 30-day grace window**: deleting a project now moves it to a recoverable "trash" state (`deleted_at`) instead of removing it immediately. Its logs, traces and metrics stay queryable and the project stays viewable read-only during the window (the project detail layout loads with `includeDeleted`, shows a "deleted (read-only)" banner with the permanent-deletion date, and hides the Settings tab); a "Restore" action brings it back. A daily purge worker (3 AM) hard-deletes projects past the grace window, calling `reservoir.purgeProject()` to clear logs/spans/metrics from every storage engine before deleting the row. Name and slug uniqueness moved to partial unique indexes (active projects only) so a deleted project's name/slug can be reused; restoring into a name or slug a new active project has since taken is rejected with a 409 instead of a raw constraint error. Soft-deleted projects are excluded from listings, data-availability widgets and API-key verification (ingestion is refused immediately, with the key-verification cache invalidated on delete). New `POST /api/v1/projects/:id/restore` and `GET /api/v1/projects?includeDeleted=true`. Migrations 050 (soft-delete column + partial indexes) and 051 (drop `ON DELETE CASCADE` from logs/spans/metrics so an out-of-band project delete cannot bulk-wipe data; the purge worker clears reservoir data explicitly first)

## [1.0.3] - 2026-06-26
Expand Down
122 changes: 122 additions & 0 deletions docs/receivers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# Webhook Receivers

Webhook receivers let external systems (CI/CD pipelines, uptime monitors, arbitrary tools) push events into LogTide. Each receiver exposes a unique tokenized URL; incoming payloads are normalized into standard log entries by an adapter and stored through the regular ingestion pipeline, so PII masking, usage quotas, Sigma detection and parsing rules all apply automatically.

Receivers are project-scoped and managed from **Project Settings > Webhook Receivers**, where you can create receivers, copy their URL, enable or disable them, and inspect the most recent received events with their normalized output.

## Endpoint contract

```
POST /api/v1/receivers/{receiverId}/{token}
Content-Type: application/json
```

The URL itself is the credential: `receiverId` identifies the receiver and `token` authenticates the request. No headers are required, which makes the endpoint compatible with systems that cannot send custom headers (GitHub webhooks, Uptime Robot).

The body must be a single JSON object of at most 256 KB.

### Responses

| Status | Meaning |
| --- | --- |
| `202 Accepted` | Event stored and queued for processing. Body: `{ "eventId": "..." }` |
| `400 Bad Request` | Body is not a JSON object (arrays, scalars and null are rejected) |
| `401 Unauthorized` | Token does not match the receiver |
| `403 Forbidden` | Receiver is disabled |
| `404 Not Found` | Unknown receiver id |
| `413 Payload Too Large` | Serialized payload exceeds 256 KB |

Processing is asynchronous: a `202` means the payload was accepted, not that it produced a log entry. The outcome (`processed`, `skipped` or `failed`, with the normalized output and any error) is visible in the receiver's recent-events view, which keeps the last 100 events per receiver.

## Tokens

Receiver tokens use the `lr_` prefix (distinct from `lp_` project API keys) and are shown exactly once, at creation time, embedded in the full webhook URL. Only a SHA-256 hash is stored. Treat the URL as a secret: anyone who has it can write logs into your project. To rotate a token, delete the receiver and create a new one, then update the external system with the new URL.

A receiver token cannot read data or call any other API endpoint; a leaked URL limits the blast radius to log ingestion into one project.

## Adapters

The adapter chosen at creation time decides how raw payloads become log entries.

### GitHub

Point a GitHub repository webhook (JSON content type) at the receiver URL. Supported events:

- **workflow_run** (only `action: completed`): `success` maps to `info`, `cancelled`/`skipped`/`neutral` to `warn`, `failure`/`timed_out`/`startup_failure`/`action_required` to `error`. The message looks like `Workflow CI completed: failure`.
- **deployment_status**: `success` maps to `info`, `failure`/`error` to `error`, anything else to `info`. The message looks like `Deployment to production: failure`.

The `service` field is the repository `full_name` (e.g. `acme/app`). Metadata includes the event type, workflow name, conclusion, run id and URL, branch and actor. Ping events and unsupported event types are marked `skipped`, not failed.

Example: a failed CI run

```json
{
"action": "completed",
"workflow_run": { "name": "CI", "conclusion": "failure", "html_url": "...", "head_branch": "main" },
"repository": { "full_name": "acme/app" },
"sender": { "login": "octocat" }
}
```

becomes an `error` log for service `acme/app` with message `Workflow CI completed: failure`.

### Uptime

Detects two payload shapes automatically:

- **Uptime Robot** (`alertType` field): `1` (down) maps to `error`, `2` (up) to `info`, `3` (SSL expiry) to `warn`. The monitor friendly name becomes the service.
- **Better Stack** (`data.attributes` incident shape): status `Started` maps to `error`, `Resolved` and `Acknowledged` to `info`. The monitor name becomes the service and the incident cause is part of the message.

Unrecognized shapes are marked `skipped`.

### Generic JSON

Accepts any JSON object. Without configuration it produces one `info` log with message `Received event`, the receiver name as service, and the full payload preserved under `metadata.payload`.

An optional **field mapping** extracts log fields from the payload using dot-paths:

```json
{
"message": "error.message",
"level": "severity",
"service": "source.app",
"timestamp": "ts",
"levelMap": { "crit": "critical", "sev1": "error" },
"defaults": { "level": "warn", "service": "external-system" }
}
```

| Key | Meaning |
| --- | --- |
| `message` | Dot-path to the log message. Missing or non-string values fall back to `Received event`. |
| `level` | Dot-path to the level value. Values are lowercased and matched against LogTide levels (`debug`, `info`, `warn`, `error`, `critical`); the synonyms `warning`, `fatal`, `err` and `crit` are also understood. |
| `service` | Dot-path to the service name (truncated to 100 chars). Falls back to `defaults.service`, then the receiver name. |
| `timestamp` | Dot-path to an ISO string or epoch value. Unparseable values fall back to the arrival time. |
| `levelMap` | Case-insensitive map applied to the extracted level value before the builtin matching (e.g. map `sev1` to `error`). |
| `defaults` | Fallback `level` and `service` used when the mapped values are missing or invalid. |

The create dialog exposes the four path fields; `levelMap` and `defaults` can be set through the API (`PATCH /api/v1/projects/{projectId}/receivers/{id}` with a `fieldMapping` object).

## Pipeline guarantees

Received events are ingested through the same path as `POST /api/v1/ingest`:

- **PII masking is fail-closed**: entries whose masking fails are rejected, and the event is marked `failed` with the rejection reason.
- **Usage quotas** (`ingestion.max_bytes_monthly`, `ingestion.max_events_monthly`, `storage.max_bytes`) apply; over-quota events fail with the quota error recorded.
- Sigma detection, exception parsing, log pipelines, metering and live tail all see receiver events like any other ingested log.

The number of receivers per organization can be capped with the `receivers.max` capability (unlimited by default in OSS).

## Management API

All management endpoints require session authentication and project access.

| Method and path | Purpose |
| --- | --- |
| `GET /api/v1/projects/{projectId}/receivers` | List receivers (never returns tokens) |
| `POST /api/v1/projects/{projectId}/receivers` | Create; returns `token` and `ingestPath` once |
| `PATCH /api/v1/projects/{projectId}/receivers/{id}` | Rename, enable/disable, update `fieldMapping` |
| `DELETE /api/v1/projects/{projectId}/receivers/{id}` | Delete (invalidates the URL immediately) |
| `GET /api/v1/projects/{projectId}/receivers/{id}/events?limit=50` | Recent events with raw payload, normalized output and status |

Receiver creation and deletion are recorded in the audit log (`receiver.created`, `receiver.deleted`).
30 changes: 30 additions & 0 deletions packages/backend/migrations/052_receivers.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
-- Inbound webhook receivers (#155): external systems POST events that get
-- normalized into log entries by per-receiver adapters.
CREATE TABLE IF NOT EXISTS receivers (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
project_id UUID NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
name TEXT NOT NULL,
adapter_type TEXT NOT NULL CHECK (adapter_type IN ('github', 'uptime', 'generic')),
token_hash TEXT NOT NULL UNIQUE,
field_mapping JSONB,
enabled BOOLEAN NOT NULL DEFAULT true,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
last_received_at TIMESTAMPTZ
);

CREATE INDEX IF NOT EXISTS idx_receivers_project ON receivers(project_id);

-- Recent raw/normalized events per receiver, capped at 100 rows per receiver
-- by the worker (pruneEvents). Powers the "recent events" UI.
CREATE TABLE IF NOT EXISTS receiver_events (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
receiver_id UUID NOT NULL REFERENCES receivers(id) ON DELETE CASCADE,
status TEXT NOT NULL CHECK (status IN ('pending', 'processed', 'skipped', 'failed')),
raw_payload JSONB NOT NULL,
normalized JSONB,
error TEXT,
received_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX IF NOT EXISTS idx_receiver_events_receiver
ON receiver_events(receiver_id, received_at DESC);
5 changes: 5 additions & 0 deletions packages/backend/src/capabilities/registry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,11 @@ export const CAPABILITIES = {
defaultLimit: null,
description: 'Maximum custom dashboards per organization',
},
'receivers.max': {
kind: 'limit',
defaultLimit: null,
description: 'Maximum inbound webhook receivers per organization',
},

// Consumption quotas (OSS-permissive: null = unlimited). signal maps to #212 metering types.
'ingestion.max_bytes_monthly': {
Expand Down
3 changes: 2 additions & 1 deletion packages/backend/src/database/tenant-tables.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ export const TENANT_TABLES = new Set<string>([
'pii_masking_rules', 'organization_pii_salts', 'audit_log', 'metrics_hourly_stats',
'metrics_daily_stats', 'metrics', 'metric_exemplars', 'custom_dashboards',
'log_pipelines', 'digest_configs', 'digest_recipients', 'projects',
'receivers',
]);

/**
Expand All @@ -23,7 +24,7 @@ export const CHILD_TABLES = new Set<string>([
'monitor_status', 'status_incident_updates', 'incident_alerts',
'incident_comments', 'incident_history', 'stack_frames',
'alert_rule_channels', 'sigma_rule_channels', 'monitor_channels',
'incident_channels', 'error_group_channels',
'incident_channels', 'error_group_channels', 'receiver_events',
]);

/** Intentionally global tables (no tenant scope). */
Expand Down
37 changes: 37 additions & 0 deletions packages/backend/src/database/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -924,6 +924,40 @@ export interface WebhookDeliveryAttemptsTable {
created_at: Generated<Timestamp>;
}

// ============================================================================
// INBOUND WEBHOOK RECEIVERS (#155)
// ============================================================================

export interface ReceiversTable {
id: Generated<string>;
project_id: string;
name: string;
adapter_type: string; // 'github' | 'uptime' | 'generic'
token_hash: string;
field_mapping: ColumnType<
Record<string, unknown> | null,
Record<string, unknown> | null,
Record<string, unknown> | null
>;
enabled: Generated<boolean>;
created_at: Generated<Timestamp>;
last_received_at: ColumnType<Date | null, Date | null, Date | null>;
}

export interface ReceiverEventsTable {
id: Generated<string>;
receiver_id: string;
status: string; // 'pending' | 'processed' | 'skipped' | 'failed'
raw_payload: ColumnType<
Record<string, unknown>,
Record<string, unknown>,
Record<string, unknown>
>;
normalized: ColumnType<unknown | null, unknown | null, unknown | null>;
error: string | null;
received_at: Generated<Timestamp>;
}

// ============================================================================
// PII MASKING TABLES
// ============================================================================
Expand Down Expand Up @@ -1209,4 +1243,7 @@ export interface Database {
// Outbound webhook delivery (#218)
webhook_deliveries: WebhookDeliveriesTable;
webhook_delivery_attempts: WebhookDeliveryAttemptsTable;
// Inbound webhook receivers (#155)
receivers: ReceiversTable;
receiver_events: ReceiverEventsTable;
}
3 changes: 3 additions & 0 deletions packages/backend/src/modules/audit-log/actions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ export const AUDIT_ACTIONS = {
// api keys
'apikey.created': 'config_change',
'apikey.revoked': 'config_change',
// inbound webhook receivers (#155)
'receiver.created': 'config_change',
'receiver.deleted': 'config_change',
// users and membership
'user.registered': 'user_management',
'user.created': 'user_management',
Expand Down
5 changes: 4 additions & 1 deletion packages/backend/src/modules/auth/plugin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,10 @@ const authPlugin: FastifyPluginAsync = async (fastify) => {
request.url.startsWith('/api/v1/invitations') ||
request.url.startsWith('/api/v1/status') ||
request.url.startsWith('/api/v1/status-incidents') ||
request.url.startsWith('/api/v1/maintenances')
request.url.startsWith('/api/v1/maintenances') ||
// Inbound webhook receivers (#155): the URL token is the credential,
// verified by the route itself.
request.url.startsWith('/api/v1/receivers')
) {
return;
}
Expand Down
84 changes: 84 additions & 0 deletions packages/backend/src/modules/receivers/adapters/generic.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
import type { LogLevel } from '@logtide/shared';
import { LOG_LEVELS } from '@logtide/shared';
import type { ReceiverAdapter } from './types.js';

/** Resolve a dot-path ("a.b.c") into a nested object. */
export function getPath(obj: unknown, path: string): unknown {
let current: unknown = obj;
for (const key of path.split('.')) {
if (current === null || typeof current !== 'object') return undefined;
current = (current as Record<string, unknown>)[key];
}
return current;
}

const LEVEL_SYNONYMS: Record<string, LogLevel> = {
warning: 'warn',
fatal: 'critical',
err: 'error',
crit: 'critical',
};

function coerceLevel(
value: unknown,
levelMap: Record<string, LogLevel> | undefined,
fallback: LogLevel
): LogLevel {
if (typeof value !== 'string' && typeof value !== 'number') return fallback;
const raw = String(value).toLowerCase();
if (levelMap) {
for (const [key, mapped] of Object.entries(levelMap)) {
if (key.toLowerCase() === raw) return mapped;
}
}
if ((LOG_LEVELS as readonly string[]).includes(raw)) return raw as LogLevel;
if (raw in LEVEL_SYNONYMS) return LEVEL_SYNONYMS[raw];
return fallback;
}

export const genericAdapter: ReceiverAdapter = (payload, receiver) => {
const mapping = receiver.fieldMapping ?? {};
const defaults = mapping.defaults ?? {};

const messageRaw = mapping.message ? getPath(payload, mapping.message) : undefined;
const message =
typeof messageRaw === 'string' && messageRaw.length > 0 ? messageRaw : 'Received event';

const serviceRaw = mapping.service ? getPath(payload, mapping.service) : undefined;
const service = (
typeof serviceRaw === 'string' && serviceRaw.length > 0
? serviceRaw
: (defaults.service ?? receiver.name)
).slice(0, 100);

const level = coerceLevel(
mapping.level ? getPath(payload, mapping.level) : undefined,
mapping.levelMap,
defaults.level ?? 'info'
);

let time = new Date().toISOString();
if (mapping.timestamp) {
const ts = getPath(payload, mapping.timestamp);
if (typeof ts === 'string' || typeof ts === 'number') {
const parsed = new Date(ts);
if (!Number.isNaN(parsed.getTime())) time = parsed.toISOString();
}
}

return {
kind: 'logs',
logs: [
{
time,
service,
level,
message,
metadata: {
receiver: { id: receiver.id, name: receiver.name, adapter: 'generic' },
payload,
},
},
],
};
};
Loading
Loading