diff --git a/.claude/commands/add-block-preview.md b/.claude/commands/add-block-preview.md new file mode 100644 index 00000000000..8345170767f --- /dev/null +++ b/.claude/commands/add-block-preview.md @@ -0,0 +1,60 @@ +--- +description: Gate a block's visibility — ship an unreleased block as a preview (hidden until revealed via AppConfig/env), reveal it to admins/orgs, GA it, or kill-switch a shipped block +argument-hint: +--- + +# Add Block Preview Skill + +You manage **block visibility gating** in Sim — hiding blocks from every discovery surface (toolbar, cmd+K search, copilot @-mentions, agent tool picker, mothership VFS/metadata/tools, Access Control list, public docs/catalog) while **never** gating execution of already-placed instances. + +## The model + +Three levers, evaluated in `apps/sim/lib/core/config/block-visibility.ts` and folded into the registry accessors (`apps/sim/blocks/registry.ts`): + +1. **`preview: true`** on the `BlockConfig` (static, in code) — the block is default-hidden EVERYWHERE (hosted, self-hosted, dev, SSR) until revealed. Fail-closed. +2. **The hosted `block-visibility` AppConfig document** — per-block rule keyed by the existing block type: + + ```jsonc + { + "": { + "enabled": false, // required. true = GA (visible to everyone) + "orgIds": ["org_..."], // optional allowlist clauses (any match reveals) + "userIds": ["user_..."], + "adminEnabled": true // platform admins (user.role === 'admin') + } + } + ``` + +3. **`PREVIEW_BLOCKS` env** (comma-separated block types) — the off-AppConfig reveal path for self-hosters and local dev. + +A revealed block that is not globally GA (`enabled !== true`, or env-revealed) renders with a **" (Preview)"** name suffix on discovery surfaces. `getBlock()` stays pure, so placed instances keep their canonical name and always execute. + +## Lifecycle of a preview block + +1. **Author** the block normally (`/add-block` etc.) and set `preview: true` on its `BlockConfig`. **Ship no `BlockMeta` and no docs until GA** — `check-block-registry` deliberately skips preview blocks in meta coverage, and `generate-docs` skips them at every gate. +2. **Local dev:** set `PREVIEW_BLOCKS=` in your env to see it (with the suffix). +3. **Merge/deploy.** The block's code is live everywhere but visible nowhere — no AppConfig rule exists and self-hosters have no env entry. +4. **Hosted preview:** add a rule to the `block-visibility` AppConfig document and start a deployment (no code deploy): + - Admins only: `{ "enabled": false, "adminEnabled": true }` + - Design-partner org: `{ "enabled": false, "orgIds": ["org_123"] }` + - GA via config (code cleanup pending): `{ "enabled": true }` — suffix disappears everywhere within ~30s (AppConfig TTL) + client refetch. + + Same runbook as `feature-flags`: edit the hosted document, `aws appconfig start-deployment` with the `sim--fast` strategy (see the infra README). +5. **GA cleanup:** delete `preview: true` from the block (now visible to self-hosters on their next upgrade), add its `BlockMeta` + regen docs, and drop the AppConfig entry. For a v2 upgrade, this is also when v1 gets `hideFromToolbar: true` (the superseded-version paradigm). + +## Kill switch (shipped blocks) + +To pull an already-GA block from discovery surfaces on hosted (incident, deprecation): add `{ "": { "enabled": false } }` to the document. Allowlist clauses can carve out exceptions. **Execution is NOT stopped** — workflows already using the block keep running; the kill switch only prevents new placement/discovery. + +## Invariants (do not violate) + +- **Execution is never gated.** The executor, serializer, drop-naming, and `isBlockTypeAccessControlExempt` resolve via pure `getBlock`. Do not add visibility checks to execution paths. +- **Clone-not-remove:** gated blocks stay in `getAllBlocks()` output as clones with `hideFromToolbar: true` — `.find`-by-type consumers rely on this. Never filter them out. +- **Keys are registry block types.** Never `custom_block_*` (parse drops them — custom blocks have their own enabled/disabled lifecycle). +- **The shared hidden-predicate is `isHiddenUnder`** (`apps/sim/blocks/visibility/context.ts`). Never restate the preview/disabled rule inline at a new consumer. +- **Process-global caches stay ungated.** `getStaticComponentFiles` (VFS) and `getExposedIntegrationTools` build the ungated universe; per-viewer filtering happens at stamp/consumer time. Never move gating into a shared builder. +- Gating is **surface hiding, not secrecy** — the full config ships in the client JS bundle. Anything truly secret cannot be a registered block. + +## Tests + +Evaluation semantics: `apps/sim/lib/core/config/block-visibility.test.ts`. Registry projection: `apps/sim/blocks/visibility/visibility.test.ts`. When gating behavior changes, extend those — mock `isPlatformAdmin` for the admin clause; use the local `withAppConfig` harness. diff --git a/.cursor/commands/add-block-preview.md b/.cursor/commands/add-block-preview.md new file mode 100644 index 00000000000..8345170767f --- /dev/null +++ b/.cursor/commands/add-block-preview.md @@ -0,0 +1,60 @@ +--- +description: Gate a block's visibility — ship an unreleased block as a preview (hidden until revealed via AppConfig/env), reveal it to admins/orgs, GA it, or kill-switch a shipped block +argument-hint: +--- + +# Add Block Preview Skill + +You manage **block visibility gating** in Sim — hiding blocks from every discovery surface (toolbar, cmd+K search, copilot @-mentions, agent tool picker, mothership VFS/metadata/tools, Access Control list, public docs/catalog) while **never** gating execution of already-placed instances. + +## The model + +Three levers, evaluated in `apps/sim/lib/core/config/block-visibility.ts` and folded into the registry accessors (`apps/sim/blocks/registry.ts`): + +1. **`preview: true`** on the `BlockConfig` (static, in code) — the block is default-hidden EVERYWHERE (hosted, self-hosted, dev, SSR) until revealed. Fail-closed. +2. **The hosted `block-visibility` AppConfig document** — per-block rule keyed by the existing block type: + + ```jsonc + { + "": { + "enabled": false, // required. true = GA (visible to everyone) + "orgIds": ["org_..."], // optional allowlist clauses (any match reveals) + "userIds": ["user_..."], + "adminEnabled": true // platform admins (user.role === 'admin') + } + } + ``` + +3. **`PREVIEW_BLOCKS` env** (comma-separated block types) — the off-AppConfig reveal path for self-hosters and local dev. + +A revealed block that is not globally GA (`enabled !== true`, or env-revealed) renders with a **" (Preview)"** name suffix on discovery surfaces. `getBlock()` stays pure, so placed instances keep their canonical name and always execute. + +## Lifecycle of a preview block + +1. **Author** the block normally (`/add-block` etc.) and set `preview: true` on its `BlockConfig`. **Ship no `BlockMeta` and no docs until GA** — `check-block-registry` deliberately skips preview blocks in meta coverage, and `generate-docs` skips them at every gate. +2. **Local dev:** set `PREVIEW_BLOCKS=` in your env to see it (with the suffix). +3. **Merge/deploy.** The block's code is live everywhere but visible nowhere — no AppConfig rule exists and self-hosters have no env entry. +4. **Hosted preview:** add a rule to the `block-visibility` AppConfig document and start a deployment (no code deploy): + - Admins only: `{ "enabled": false, "adminEnabled": true }` + - Design-partner org: `{ "enabled": false, "orgIds": ["org_123"] }` + - GA via config (code cleanup pending): `{ "enabled": true }` — suffix disappears everywhere within ~30s (AppConfig TTL) + client refetch. + + Same runbook as `feature-flags`: edit the hosted document, `aws appconfig start-deployment` with the `sim--fast` strategy (see the infra README). +5. **GA cleanup:** delete `preview: true` from the block (now visible to self-hosters on their next upgrade), add its `BlockMeta` + regen docs, and drop the AppConfig entry. For a v2 upgrade, this is also when v1 gets `hideFromToolbar: true` (the superseded-version paradigm). + +## Kill switch (shipped blocks) + +To pull an already-GA block from discovery surfaces on hosted (incident, deprecation): add `{ "": { "enabled": false } }` to the document. Allowlist clauses can carve out exceptions. **Execution is NOT stopped** — workflows already using the block keep running; the kill switch only prevents new placement/discovery. + +## Invariants (do not violate) + +- **Execution is never gated.** The executor, serializer, drop-naming, and `isBlockTypeAccessControlExempt` resolve via pure `getBlock`. Do not add visibility checks to execution paths. +- **Clone-not-remove:** gated blocks stay in `getAllBlocks()` output as clones with `hideFromToolbar: true` — `.find`-by-type consumers rely on this. Never filter them out. +- **Keys are registry block types.** Never `custom_block_*` (parse drops them — custom blocks have their own enabled/disabled lifecycle). +- **The shared hidden-predicate is `isHiddenUnder`** (`apps/sim/blocks/visibility/context.ts`). Never restate the preview/disabled rule inline at a new consumer. +- **Process-global caches stay ungated.** `getStaticComponentFiles` (VFS) and `getExposedIntegrationTools` build the ungated universe; per-viewer filtering happens at stamp/consumer time. Never move gating into a shared builder. +- Gating is **surface hiding, not secrecy** — the full config ships in the client JS bundle. Anything truly secret cannot be a registered block. + +## Tests + +Evaluation semantics: `apps/sim/lib/core/config/block-visibility.test.ts`. Registry projection: `apps/sim/blocks/visibility/visibility.test.ts`. When gating behavior changes, extend those — mock `isPlatformAdmin` for the admin clause; use the local `withAppConfig` harness. diff --git a/apps/docs/components/icons.tsx b/apps/docs/components/icons.tsx index c213134b0e1..fe869a5b77e 100644 --- a/apps/docs/components/icons.tsx +++ b/apps/docs/components/icons.tsx @@ -3687,6 +3687,62 @@ export const SakanaIcon = (props: SVGProps) => ( ) +export function MetaIcon(props: SVGProps) { + const id = useId() + const gradient1Id = `meta_gradient_1_${id}` + const gradient2Id = `meta_gradient_2_${id}` + + return ( + + Meta + + + + + + + + + + + + + + + + + ) +} + export function GeminiIcon(props: SVGProps) { const id = useId() const gradientId = `gemini_gradient_${id}` @@ -5572,6 +5628,27 @@ export function SshIcon(props: SVGProps) { ) } +export function SshTerminalIcon(props: SVGProps) { + return ( + + + + + + + ) +} + export function SftpIcon(props: SVGProps) { return ( ({ + mockGetSession: vi.fn(), + mockCheckWorkspaceAccess: vi.fn(), + mockIsPlatformAdmin: vi.fn(), + mockGetBlockVisibility: vi.fn(), + })) + +vi.mock('@/lib/auth', () => ({ + auth: { api: { getSession: vi.fn() } }, + getSession: mockGetSession, +})) + +vi.mock('@/lib/workspaces/permissions/utils', () => ({ + checkWorkspaceAccess: mockCheckWorkspaceAccess, +})) + +vi.mock('@/lib/permissions/super-user', () => ({ + isPlatformAdmin: mockIsPlatformAdmin, +})) + +vi.mock('@/lib/core/config/block-visibility', () => ({ + getBlockVisibility: mockGetBlockVisibility, +})) + +import { GET } from '@/app/api/blocks/visibility/route' + +const WORKSPACE_ID = '11111111-2222-4333-8444-555555555555' + +function request(workspaceId = WORKSPACE_ID) { + return new NextRequest(`http://localhost/api/blocks/visibility?workspaceId=${workspaceId}`) +} + +describe('GET /api/blocks/visibility', () => { + beforeEach(() => { + vi.clearAllMocks() + mockGetSession.mockResolvedValue({ user: { id: 'user-1' } }) + mockCheckWorkspaceAccess.mockResolvedValue({ + hasAccess: true, + workspace: { organizationId: 'org-1' }, + }) + mockIsPlatformAdmin.mockResolvedValue(false) + mockGetBlockVisibility.mockResolvedValue({ + revealed: new Set(['gmail_v2']), + disabled: new Set(['slack']), + previewTagged: new Set(['gmail_v2']), + }) + }) + + it('returns 401 without a session', async () => { + mockGetSession.mockResolvedValue(null) + const response = await GET(request()) + expect(response.status).toBe(401) + }) + + it('returns 403 without workspace access', async () => { + mockCheckWorkspaceAccess.mockResolvedValue({ hasAccess: false, workspace: null }) + const response = await GET(request()) + expect(response.status).toBe(403) + expect(mockGetBlockVisibility).not.toHaveBeenCalled() + }) + + it('evaluates visibility for the session user, workspace org, and pre-resolved admin', async () => { + mockIsPlatformAdmin.mockResolvedValue(true) + const response = await GET(request()) + expect(response.status).toBe(200) + expect(await response.json()).toEqual({ + revealed: ['gmail_v2'], + disabled: ['slack'], + previewTagged: ['gmail_v2'], + }) + expect(mockGetBlockVisibility).toHaveBeenCalledWith({ + userId: 'user-1', + orgId: 'org-1', + isAdmin: true, + }) + }) +}) diff --git a/apps/sim/app/api/blocks/visibility/route.ts b/apps/sim/app/api/blocks/visibility/route.ts new file mode 100644 index 00000000000..8aea5ad77b3 --- /dev/null +++ b/apps/sim/app/api/blocks/visibility/route.ts @@ -0,0 +1,46 @@ +import type { NextRequest } from 'next/server' +import { NextResponse } from 'next/server' +import { getBlockVisibilityContract } from '@/lib/api/contracts/block-visibility' +import { parseRequest } from '@/lib/api/server' +import { getSession } from '@/lib/auth' +import { getBlockVisibility } from '@/lib/core/config/block-visibility' +import { withRouteHandler } from '@/lib/core/utils/with-route-handler' +import { isPlatformAdmin } from '@/lib/permissions/super-user' +import { checkWorkspaceAccess } from '@/lib/workspaces/permissions/utils' + +/** + * Evaluates the viewer's block-visibility projection for a workspace: which + * preview blocks are revealed (and preview-tagged) and which shipped blocks are + * kill-switched. Consumed by the client visibility overlay + * (`BlockVisibilityLoader`); discovery-only — execution is never gated. + */ +export const GET = withRouteHandler(async (request: NextRequest) => { + const session = await getSession() + if (!session?.user?.id) { + return NextResponse.json({ error: 'Unauthorized' }, { status: 401 }) + } + + const parsed = await parseRequest(getBlockVisibilityContract, request, {}) + if (!parsed.success) return parsed.response + + const userId = session.user.id + const { workspaceId } = parsed.data.query + + const access = await checkWorkspaceAccess(workspaceId, userId) + if (!access.hasAccess) { + return NextResponse.json({ error: 'Access denied' }, { status: 403 }) + } + + const isAdmin = await isPlatformAdmin(userId) + const vis = await getBlockVisibility({ + userId, + orgId: access.workspace?.organizationId, + isAdmin, + }) + + return NextResponse.json({ + revealed: [...vis.revealed], + disabled: [...vis.disabled], + previewTagged: [...vis.previewTagged], + }) +}) diff --git a/apps/sim/app/playground/page.tsx b/apps/sim/app/playground/page.tsx index 9acd0f74367..4923394efa3 100644 --- a/apps/sim/app/playground/page.tsx +++ b/apps/sim/app/playground/page.tsx @@ -11,7 +11,6 @@ import { Button, ButtonGroup, ButtonGroupItem, - Card as CardIcon, Checkbox, ChevronDown, ChipDatePicker, @@ -1027,7 +1026,6 @@ export default function PlaygroundPage() { {[ { Icon: BubbleChatClose, name: 'BubbleChatClose' }, { Icon: BubbleChatPreview, name: 'BubbleChatPreview' }, - { Icon: CardIcon, name: 'Card' }, { Icon: ChevronDown, name: 'ChevronDown' }, { Icon: Connections, name: 'Connections' }, { Icon: Cursor, name: 'Cursor' }, diff --git a/apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/file-viewer.tsx b/apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/file-viewer.tsx index 3d4f4b5d3e9..9984e16b0df 100644 --- a/apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/file-viewer.tsx +++ b/apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/file-viewer.tsx @@ -99,8 +99,12 @@ interface FileViewerProps { previewMode?: PreviewMode autoFocus?: boolean onDirtyChange?: (isDirty: boolean) => void - onSaveStatusChange?: (status: 'idle' | 'saving' | 'saved' | 'error') => void + onSaveStatusChange?: ( + status: 'idle' | 'saving' | 'saved' | 'error', + retry?: () => Promise + ) => void saveRef?: React.MutableRefObject<(() => Promise) | null> + discardRef?: React.MutableRefObject<(() => void) | null> streamingContent?: string isAgentEditing?: boolean streamIsIncremental?: boolean @@ -131,6 +135,7 @@ function FileViewerContent({ onDirtyChange, onSaveStatusChange, saveRef, + discardRef, streamingContent, isAgentEditing, streamIsIncremental, @@ -174,6 +179,7 @@ function FileViewerContent({ onDirtyChange={onDirtyChange} onSaveStatusChange={onSaveStatusChange} saveRef={saveRef} + discardRef={discardRef} streamingContent={streamingContent} isAgentEditing={isAgentEditing} streamIsIncremental={streamIsIncremental} @@ -193,6 +199,7 @@ function FileViewerContent({ onDirtyChange={onDirtyChange} onSaveStatusChange={onSaveStatusChange} saveRef={saveRef} + discardRef={discardRef} streamingContent={streamingContent} isAgentEditing={isAgentEditing} disableStreamingAutoScroll={disableStreamingAutoScroll} diff --git a/apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/markdown-paste.test.ts b/apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/markdown-paste.test.ts index 9abccb28df6..b428f01a4da 100644 --- a/apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/markdown-paste.test.ts +++ b/apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/markdown-paste.test.ts @@ -240,4 +240,18 @@ describe('markdown paste', () => { expect(cleaned).toContain('a') expect(transformHtml(editor, 'ab')).toBe('ab') }) + + it('strips nested/repeated
diff --git a/apps/sim/app/workspace/[workspaceId]/providers/block-visibility-loader.tsx b/apps/sim/app/workspace/[workspaceId]/providers/block-visibility-loader.tsx new file mode 100644 index 00000000000..2b4e2513fc2 --- /dev/null +++ b/apps/sim/app/workspace/[workspaceId]/providers/block-visibility-loader.tsx @@ -0,0 +1,41 @@ +'use client' + +import { useEffect } from 'react' +import { useParams } from 'next/navigation' +import { hydrateBlockVisibility, resetBlockVisibilityForSwitch } from '@/blocks/visibility/client' +import { useBlockVisibility } from '@/hooks/queries/block-visibility' + +/** + * Hydrates the client block-visibility overlay for the active workspace so the + * registry accessors project the viewer's revealed/disabled preview blocks. + * Mounted once in the workspace layout, next to `CustomBlocksLoader`. + * + * First paint needs no prefetch: `preview: true` blocks are fail-closed until + * this hydrate lands, so the fetch only ever reveals (benign pop-in) or applies + * a kill switch to an already-public block. Identical refetches are absorbed by + * the deep-equal guard inside `hydrateBlockVisibility`. + */ +export function BlockVisibilityLoader() { + const params = useParams() + const workspaceId = params?.workspaceId as string | undefined + const { data } = useBlockVisibility(workspaceId) + + useEffect(() => { + // On a workspace switch the query key changes and `data` is undefined while + // the new projection loads — reset fail-closed so the previous workspace's + // preview reveals never linger across orgs, while carrying kill-switch + // entries over so disabled blocks don't flash back during the flight + // window. No-ops on first mount (nothing hydrated yet). + if (!data) { + resetBlockVisibilityForSwitch() + return + } + hydrateBlockVisibility({ + revealed: new Set(data.revealed), + disabled: new Set(data.disabled), + previewTagged: new Set(data.previewTagged), + }) + }, [data]) + + return null +} diff --git a/apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/copilot/components/user-input/hooks/use-mention-data.ts b/apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/copilot/components/user-input/hooks/use-mention-data.ts index 95508ac4102..d0d0a1a43a9 100644 --- a/apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/copilot/components/user-input/hooks/use-mention-data.ts +++ b/apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/copilot/components/user-input/hooks/use-mention-data.ts @@ -7,6 +7,7 @@ import { requestJson } from '@/lib/api/client/request' import { listCopilotChatsContract } from '@/lib/api/contracts/copilot' import { listKnowledgeBasesContract } from '@/lib/api/contracts/knowledge/base' import { listLogsContract } from '@/lib/api/contracts/logs' +import { useCustomBlockOverlayVersion } from '@/blocks/custom/client-overlay' import { type IntegrationDescriptor, listIntegrations } from '@/blocks/integration-matcher' import { useWorkflows } from '@/hooks/queries/workflows' import { usePermissionConfig } from '@/hooks/use-permission-config' @@ -130,9 +131,12 @@ export function useMentionData(props: UseMentionDataProps): MentionDataReturn { const [blocksList, setBlocksList] = useState([]) const [isLoadingBlocks, setIsLoadingBlocks] = useState(false) + // Reset on permission changes and on block-overlay bumps (custom-block or + // block-visibility hydrate) so late preview reveals refresh the folder. + const blockOverlayVersion = useCustomBlockOverlayVersion() useEffect(() => { setBlocksList([]) - }, [config.allowedIntegrations]) + }, [config.allowedIntegrations, blockOverlayVersion]) const [logsList, setLogsList] = useState([]) const [isLoadingLogs, setIsLoadingLogs] = useState(false) diff --git a/apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/toolbar/toolbar.tsx b/apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/toolbar/toolbar.tsx index e051f58379f..9118eef024a 100644 --- a/apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/toolbar/toolbar.tsx +++ b/apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/toolbar/toolbar.tsx @@ -35,6 +35,7 @@ import { CUSTOM_BLOCK_TILE_COLOR, isCustomBlockType, } from '@/blocks/custom/build-config' +import { useCustomBlockOverlayVersion } from '@/blocks/custom/client-overlay' import { getCustomBlockIcon } from '@/blocks/custom/custom-block-icon' import { getTileIconColorClass } from '@/blocks/icon-color' import { getCanonicalBlocksByCategory } from '@/blocks/registry' @@ -154,11 +155,29 @@ const ToolbarItem = memo(function ToolbarItem({ let cachedTriggers: BlockItem[] | null = null /** - * Gets triggers data, computing it once and caching for subsequent calls. - * Non-integration triggers (Start, Schedule, Webhook) are prioritized first, - * followed by all other triggers sorted alphabetically. + * Block-overlay version the caches below were built against. The registry's + * output is no longer static — a block-visibility hydrate (preview reveal / + * kill switch) bumps the shared overlay version — so the caches are keyed to + * it and dropped when it moves. -1 = never built. */ -function getTriggers(): BlockItem[] { +let cachedAtOverlayVersion = -1 + +/** Drop all three caches when the overlay version moved since they were built. */ +function syncCachesToOverlayVersion(version: number) { + if (cachedAtOverlayVersion === version) return + cachedAtOverlayVersion = version + cachedTriggers = null + cachedBlocks = null + cachedTools = null +} + +/** + * Gets triggers data, computing it once per overlay version and caching for + * subsequent calls. Non-integration triggers (Start, Schedule, Webhook) are + * prioritized first, followed by all other triggers sorted alphabetically. + */ +function getTriggers(overlayVersion: number): BlockItem[] { + syncCachesToOverlayVersion(overlayVersion) if (cachedTriggers === null) { const allTriggers = getTriggersForSidebar() const priorityOrder = ['Start', 'Schedule', 'Webhook'] @@ -250,12 +269,14 @@ function ensureBlockCaches() { cachedTools = toolItems } -function getBlocks(): BlockItem[] { +function getBlocks(overlayVersion: number): BlockItem[] { + syncCachesToOverlayVersion(overlayVersion) ensureBlockCaches() return cachedBlocks as BlockItem[] } -function getTools(): BlockItem[] { +function getTools(overlayVersion: number): BlockItem[] { + syncCachesToOverlayVersion(overlayVersion) ensureBlockCaches() return cachedTools as BlockItem[] } @@ -455,9 +476,12 @@ export const Toolbar = memo( const { data: whitelabel } = useWhitelabelSettings(customBlocksData?.[0]?.organizationId) const fallbackIconUrl = whitelabel?.logoUrl ?? null - const allTriggers = getTriggers() - const allBlocks = getBlocks() - const allTools = getTools() + // Re-read the block lists whenever the overlay version bumps (custom-block + // or block-visibility hydrate) — the module caches are keyed to it. + const blockOverlayVersion = useCustomBlockOverlayVersion() + const allTriggers = getTriggers(blockOverlayVersion) + const allBlocks = getBlocks(blockOverlayVersion) + const allTools = getTools(blockOverlayVersion) // Published custom blocks are their own section. Exclude disabled blocks (still // resolvable so placed instances survive, but not offered for new placement) and diff --git a/apps/sim/blocks/blocks/brex.ts b/apps/sim/blocks/blocks/brex.ts index 6a5feefbb83..086617b2615 100644 --- a/apps/sim/blocks/blocks/brex.ts +++ b/apps/sim/blocks/blocks/brex.ts @@ -4,6 +4,35 @@ import { AuthMode, IntegrationType } from '@/blocks/types' import { normalizeFileInput } from '@/blocks/utils' import type { BrexResponse } from '@/tools/brex/types' +/** Coerces a required money-amount field to a finite number, throwing on blank/non-numeric input rather than silently sending 0 or NaN to Brex. */ +function toRequiredAmount(value: unknown, fieldLabel: string): number { + if (value == null || (typeof value === 'string' && value.trim() === '')) { + throw new Error(`${fieldLabel} must be a valid number`) + } + const parsed = Number(value) + if (!Number.isFinite(parsed)) { + throw new Error(`${fieldLabel} must be a valid number`) + } + return parsed +} + +/** Coerces an optional numeric field to a finite number, throwing on non-numeric input instead of silently forwarding NaN. Preserves explicit 0. */ +function toOptionalFiniteNumber(value: unknown, fieldLabel: string): number | undefined { + if (value == null || (typeof value === 'string' && value.trim() === '')) return undefined + const parsed = Number(value) + if (!Number.isFinite(parsed)) { + throw new Error(`${fieldLabel} must be a valid number`) + } + return parsed +} + +/** Normalizes a boolean field that may arrive as a string (e.g. from a dynamic reference) instead of an actual boolean. */ +function toOptionalBoolean(value: unknown): boolean | undefined { + if (value == null) return undefined + if (typeof value === 'boolean') return value + return String(value).toLowerCase() === 'true' +} + const PAGINATED_OPERATIONS = new Set([ 'list_expenses', 'list_card_transactions', @@ -66,13 +95,19 @@ export const BrexBlock: BlockConfig = { // Budgets { label: 'List Budgets', id: 'list_budgets' }, { label: 'Get Budget', id: 'get_budget' }, + { label: 'Create Budget', id: 'create_budget' }, + { label: 'Archive Budget', id: 'archive_budget' }, { label: 'List Spend Limits', id: 'list_spend_limits' }, { label: 'Get Spend Limit', id: 'get_spend_limit' }, + { label: 'Create Spend Limit', id: 'create_spend_limit' }, // Payments { label: 'List Vendors', id: 'list_vendors' }, { label: 'Get Vendor', id: 'get_vendor' }, + { label: 'Create Vendor', id: 'create_vendor' }, + { label: 'Update Vendor', id: 'update_vendor' }, { label: 'List Transfers', id: 'list_transfers' }, { label: 'Get Transfer', id: 'get_transfer' }, + { label: 'Create Transfer', id: 'create_transfer' }, ], value: () => 'list_expenses', }, @@ -142,11 +177,16 @@ export const BrexBlock: BlockConfig = { placeholder: 'ID of the cash account (Get Cash Account defaults to primary)', condition: { field: 'operation', - value: ['list_cash_transactions', 'list_cash_statements', 'get_cash_account'], + value: [ + 'list_cash_transactions', + 'list_cash_statements', + 'get_cash_account', + 'create_transfer', + ], }, required: { field: 'operation', - value: ['list_cash_transactions', 'list_cash_statements'], + value: ['list_cash_transactions', 'list_cash_statements', 'create_transfer'], }, }, { @@ -162,8 +202,8 @@ export const BrexBlock: BlockConfig = { title: 'Budget ID', type: 'short-input', placeholder: 'ID of the budget', - condition: { field: 'operation', value: 'get_budget' }, - required: { field: 'operation', value: 'get_budget' }, + condition: { field: 'operation', value: ['get_budget', 'archive_budget'] }, + required: { field: 'operation', value: ['get_budget', 'archive_budget'] }, }, { id: 'spendLimitId', @@ -178,8 +218,8 @@ export const BrexBlock: BlockConfig = { title: 'Vendor ID', type: 'short-input', placeholder: 'ID of the vendor', - condition: { field: 'operation', value: 'get_vendor' }, - required: { field: 'operation', value: 'get_vendor' }, + condition: { field: 'operation', value: ['get_vendor', 'update_vendor'] }, + required: { field: 'operation', value: ['get_vendor', 'update_vendor'] }, }, { id: 'transferId', @@ -189,6 +229,305 @@ export const BrexBlock: BlockConfig = { condition: { field: 'operation', value: 'get_transfer' }, required: { field: 'operation', value: 'get_transfer' }, }, + { + id: 'companyName', + title: 'Company Name', + type: 'short-input', + placeholder: 'Name for the vendor (must be unique)', + condition: { field: 'operation', value: ['create_vendor', 'update_vendor'] }, + required: { field: 'operation', value: 'create_vendor' }, + }, + { + id: 'vendorEmail', + title: 'Email', + type: 'short-input', + placeholder: 'Email address for the vendor', + condition: { field: 'operation', value: ['create_vendor', 'update_vendor'] }, + }, + { + id: 'vendorPhone', + title: 'Phone', + type: 'short-input', + placeholder: 'Phone number for the vendor', + condition: { field: 'operation', value: ['create_vendor', 'update_vendor'] }, + }, + { + id: 'vendorPaymentInstrumentId', + title: 'Vendor Payment Instrument ID', + type: 'short-input', + placeholder: + "ID of the vendor's payment instrument to pay (from the vendor's payment accounts)", + condition: { field: 'operation', value: 'create_transfer' }, + required: { field: 'operation', value: 'create_transfer' }, + }, + { + id: 'amount', + title: 'Amount', + type: 'short-input', + placeholder: 'Amount in the smallest unit of the currency (e.g., cents for USD)', + condition: { field: 'operation', value: ['create_transfer', 'create_budget'] }, + required: { field: 'operation', value: ['create_transfer', 'create_budget'] }, + }, + { + id: 'currency', + title: 'Currency', + type: 'short-input', + placeholder: 'ISO 4217 currency code (defaults to USD)', + mode: 'advanced', + condition: { + field: 'operation', + value: ['create_transfer', 'create_budget', 'create_spend_limit'], + }, + }, + { + id: 'description', + title: 'Description', + type: 'long-input', + placeholder: 'Description of the transfer, budget, or spend limit', + condition: { + field: 'operation', + value: ['create_transfer', 'create_budget', 'create_spend_limit'], + }, + required: { field: 'operation', value: ['create_transfer', 'create_budget'] }, + }, + { + id: 'externalMemo', + title: 'External Memo', + type: 'short-input', + placeholder: 'Memo shown to the recipient (max 90 chars for ACH/Wire, 40 for Cheque)', + condition: { field: 'operation', value: 'create_transfer' }, + required: { field: 'operation', value: 'create_transfer' }, + }, + { + id: 'approvalType', + title: 'Approval Type', + type: 'dropdown', + options: [{ label: 'Manual approval required', id: 'MANUAL' }], + placeholder: 'Default policy applies if left blank', + mode: 'advanced', + condition: { field: 'operation', value: 'create_transfer' }, + }, + { + id: 'isPproEnabled', + title: 'Enable Principal Protection (PPRO)', + type: 'switch', + mode: 'advanced', + condition: { field: 'operation', value: 'create_transfer' }, + }, + { + id: 'resourceName', + title: 'Name', + type: 'short-input', + placeholder: 'Name for the budget or spend limit', + condition: { field: 'operation', value: ['create_budget', 'create_spend_limit'] }, + required: { field: 'operation', value: ['create_budget', 'create_spend_limit'] }, + }, + { + id: 'parentBudgetId', + title: 'Parent Budget ID', + type: 'short-input', + placeholder: 'ID of the parent budget', + condition: { field: 'operation', value: ['create_budget', 'create_spend_limit'] }, + required: { field: 'operation', value: 'create_budget' }, + }, + { + id: 'periodRecurrenceType', + title: 'Period Recurrence', + type: 'dropdown', + options: [ + { label: 'Weekly', id: 'WEEKLY' }, + { label: 'Monthly', id: 'MONTHLY' }, + { label: 'Quarterly', id: 'QUARTERLY' }, + { label: 'Yearly', id: 'YEARLY' }, + { label: 'One-time', id: 'ONE_TIME' }, + ], + condition: { field: 'operation', value: 'create_budget' }, + required: { field: 'operation', value: 'create_budget' }, + }, + { + id: 'startDate', + title: 'Start Date', + type: 'short-input', + placeholder: 'Date the budget/spend limit should start counting (YYYY-MM-DD)', + mode: 'advanced', + condition: { field: 'operation', value: ['create_budget', 'create_spend_limit'] }, + }, + { + id: 'endDate', + title: 'End Date', + type: 'short-input', + placeholder: 'Date the budget/spend limit should stop counting (YYYY-MM-DD)', + mode: 'advanced', + condition: { field: 'operation', value: ['create_budget', 'create_spend_limit'] }, + }, + { + id: 'ownerUserIds', + title: 'Owner User IDs', + type: 'short-input', + placeholder: 'Comma-separated user IDs of the budget/spend limit owners', + mode: 'advanced', + condition: { field: 'operation', value: ['create_budget', 'create_spend_limit'] }, + wandConfig: { + enabled: true, + prompt: + 'Generate a comma-separated list of Brex user IDs to set as owners based on the description.\n\nReturn ONLY the comma-separated user IDs - no explanations, no extra text.', + placeholder: 'Describe which users should own this budget/spend limit...', + }, + }, + { + id: 'spendLimitPeriodRecurrenceType', + title: 'Period Recurrence', + type: 'dropdown', + options: [ + { label: 'Per week', id: 'PER_WEEK' }, + { label: 'Per month', id: 'PER_MONTH' }, + { label: 'Per quarter', id: 'PER_QUARTER' }, + { label: 'Per year', id: 'PER_YEAR' }, + { label: 'One-time', id: 'ONE_TIME' }, + ], + condition: { field: 'operation', value: 'create_spend_limit' }, + required: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'spendType', + title: 'Spend Type', + type: 'dropdown', + options: [ + { label: 'Budget-provisioned cards only', id: 'BUDGET_PROVISIONED_CARDS_ONLY' }, + { + label: 'Non-budget-provisioned cards allowed', + id: 'NON_BUDGET_PROVISIONED_CARDS_ALLOWED', + }, + ], + condition: { field: 'operation', value: 'create_spend_limit' }, + required: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'expenseVisibility', + title: 'Expense Visibility', + type: 'dropdown', + options: [ + { label: 'Shared with all members', id: 'SHARED' }, + { label: 'Private', id: 'PRIVATE' }, + ], + condition: { field: 'operation', value: 'create_spend_limit' }, + required: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'authorizationVisibility', + title: 'Authorization Visibility', + type: 'dropdown', + options: [ + { label: 'Public to all members', id: 'PUBLIC' }, + { label: 'Private to controllers/owners', id: 'PRIVATE' }, + ], + condition: { field: 'operation', value: 'create_spend_limit' }, + required: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'limitIncreaseSetting', + title: 'Limit Increase Requests', + type: 'dropdown', + options: [ + { label: 'Enabled', id: 'ENABLED' }, + { label: 'Disabled', id: 'DISABLED' }, + ], + condition: { field: 'operation', value: 'create_spend_limit' }, + required: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'autoTransferCardsSetting', + title: 'Auto Transfer Cards', + type: 'dropdown', + options: [ + { label: 'Disabled', id: 'DISABLED' }, + { label: 'Enabled', id: 'ENABLED' }, + ], + condition: { field: 'operation', value: 'create_spend_limit' }, + required: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'autoCreateLimitCardsSetting', + title: 'Auto Create Limit Cards', + type: 'dropdown', + options: [ + { label: 'Disabled', id: 'DISABLED' }, + { label: 'All members', id: 'ALL_MEMBERS' }, + ], + condition: { field: 'operation', value: 'create_spend_limit' }, + required: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'expensePolicyId', + title: 'Expense Policy ID', + type: 'short-input', + placeholder: 'ID of the expense policy for this spend limit', + condition: { field: 'operation', value: 'create_spend_limit' }, + required: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'baseLimitAmount', + title: 'Base Limit Amount', + type: 'short-input', + placeholder: 'Base limit amount in the smallest unit of the currency (e.g., cents for USD)', + condition: { field: 'operation', value: 'create_spend_limit' }, + required: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'authorizationType', + title: 'Authorization Type', + type: 'dropdown', + options: [ + { label: 'Hard (declines over available balance)', id: 'HARD' }, + { label: 'Soft', id: 'SOFT' }, + ], + condition: { field: 'operation', value: 'create_spend_limit' }, + required: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'rolloverRefreshRate', + title: 'Rollover Refresh Rate', + type: 'dropdown', + options: [ + { label: 'Off', id: 'OFF' }, + { label: 'Never', id: 'NEVER' }, + { label: 'Per month', id: 'PER_MONTH' }, + { label: 'Per quarter', id: 'PER_QUARTER' }, + { label: 'Per year', id: 'PER_YEAR' }, + ], + condition: { field: 'operation', value: 'create_spend_limit' }, + required: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'limitBufferPercentage', + title: 'Limit Buffer Percentage', + type: 'short-input', + placeholder: 'Flexible buffer on the limit as a 0-100 percentage', + mode: 'advanced', + condition: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'transactionLimitAmount', + title: 'Transaction Limit Amount', + type: 'short-input', + placeholder: 'Per-transaction limit in the smallest unit of the currency', + mode: 'advanced', + condition: { field: 'operation', value: 'create_spend_limit' }, + }, + { + id: 'spendLimitMemberUserIds', + title: 'Member User IDs', + type: 'short-input', + placeholder: 'Comma-separated user IDs of the spend limit members', + mode: 'advanced', + condition: { field: 'operation', value: 'create_spend_limit' }, + wandConfig: { + enabled: true, + prompt: + 'Generate a comma-separated list of Brex user IDs to set as spend limit members based on the description.\n\nReturn ONLY the comma-separated user IDs - no explanations, no extra text.', + placeholder: 'Describe which users should be members of this spend limit...', + }, + }, { id: 'email', title: 'Email', @@ -388,12 +727,18 @@ export const BrexBlock: BlockConfig = { 'brex_get_company', 'brex_list_budgets', 'brex_get_budget', + 'brex_create_budget', + 'brex_archive_budget', 'brex_list_spend_limits', 'brex_get_spend_limit', + 'brex_create_spend_limit', 'brex_list_vendors', 'brex_get_vendor', + 'brex_create_vendor', + 'brex_update_vendor', 'brex_list_transfers', 'brex_get_transfer', + 'brex_create_transfer', ], config: { tool: (params) => `brex_${params.operation}`, @@ -459,15 +804,88 @@ export const BrexBlock: BlockConfig = { case 'get_budget': result.budgetId = params.budgetId break + case 'create_budget': { + result.name = params.resourceName + result.description = params.description + result.parentBudgetId = params.parentBudgetId + result.periodRecurrenceType = params.periodRecurrenceType + result.amount = toRequiredAmount(params.amount, 'Amount') + if (params.currency) result.currency = params.currency + if (params.ownerUserIds) result.ownerUserIds = params.ownerUserIds + if (params.startDate) result.startDate = params.startDate + if (params.endDate) result.endDate = params.endDate + break + } + case 'archive_budget': + result.budgetId = params.budgetId + break case 'get_spend_limit': result.spendLimitId = params.spendLimitId break + case 'create_spend_limit': { + result.name = params.resourceName + result.periodRecurrenceType = params.spendLimitPeriodRecurrenceType + result.spendType = params.spendType + result.expenseVisibility = params.expenseVisibility + result.authorizationVisibility = params.authorizationVisibility + result.limitIncreaseSetting = params.limitIncreaseSetting + result.autoTransferCardsSetting = params.autoTransferCardsSetting + result.autoCreateLimitCardsSetting = params.autoCreateLimitCardsSetting + result.expensePolicyId = params.expensePolicyId + result.baseLimitAmount = toRequiredAmount(params.baseLimitAmount, 'Base limit amount') + if (params.currency) result.currency = params.currency + result.authorizationType = params.authorizationType + result.rolloverRefreshRate = params.rolloverRefreshRate + const limitBufferPercentage = toOptionalFiniteNumber( + params.limitBufferPercentage, + 'Limit buffer percentage' + ) + if (limitBufferPercentage !== undefined) + result.limitBufferPercentage = limitBufferPercentage + if (params.description) result.description = params.description + if (params.parentBudgetId) result.parentBudgetId = params.parentBudgetId + if (params.startDate) result.startDate = params.startDate + if (params.endDate) result.endDate = params.endDate + const transactionLimitAmount = toOptionalFiniteNumber( + params.transactionLimitAmount, + 'Transaction limit amount' + ) + if (transactionLimitAmount !== undefined) + result.transactionLimitAmount = transactionLimitAmount + if (params.ownerUserIds) result.ownerUserIds = params.ownerUserIds + if (params.spendLimitMemberUserIds) + result.memberUserIds = params.spendLimitMemberUserIds + break + } case 'get_vendor': result.vendorId = params.vendorId break + case 'create_vendor': + result.companyName = params.companyName + if (params.vendorEmail) result.email = params.vendorEmail + if (params.vendorPhone) result.phone = params.vendorPhone + break + case 'update_vendor': + result.vendorId = params.vendorId + if (params.companyName) result.companyName = params.companyName + if (params.vendorEmail) result.email = params.vendorEmail + if (params.vendorPhone) result.phone = params.vendorPhone + break case 'get_transfer': result.transferId = params.transferId break + case 'create_transfer': { + result.cashAccountId = params.accountId + result.vendorPaymentInstrumentId = params.vendorPaymentInstrumentId + result.amount = toRequiredAmount(params.amount, 'Amount') + if (params.currency) result.currency = params.currency + result.description = params.description + result.externalMemo = params.externalMemo + if (params.approvalType) result.approvalType = params.approvalType + const pproEnabled = toOptionalBoolean(params.isPproEnabled) + if (pproEnabled !== undefined) result.isPproEnabled = pproEnabled + break + } default: break } @@ -508,6 +926,76 @@ export const BrexBlock: BlockConfig = { }, cursor: { type: 'string', description: 'Pagination cursor' }, limit: { type: 'string', description: 'Number of results to return' }, + companyName: { type: 'string', description: 'Vendor company name' }, + vendorEmail: { type: 'string', description: 'Vendor email address' }, + vendorPhone: { type: 'string', description: 'Vendor phone number' }, + vendorPaymentInstrumentId: { + type: 'string', + description: "Vendor's payment instrument ID for the transfer counterparty", + }, + amount: { type: 'string', description: 'Amount in the smallest unit of the currency' }, + currency: { type: 'string', description: 'ISO 4217 currency code' }, + description: { + type: 'string', + description: 'Description of the transfer, budget, or spend limit', + }, + externalMemo: { type: 'string', description: 'External memo shown to the transfer recipient' }, + approvalType: { type: 'string', description: 'Transfer approval type (MANUAL)' }, + isPproEnabled: { + type: 'boolean', + description: 'Whether to enable Principal Protection (PPRO) on the transfer', + }, + resourceName: { type: 'string', description: 'Name for the budget or spend limit' }, + parentBudgetId: { type: 'string', description: 'Parent budget ID' }, + periodRecurrenceType: { + type: 'string', + description: 'Budget period recurrence (WEEKLY, MONTHLY, QUARTERLY, YEARLY, ONE_TIME)', + }, + startDate: { type: 'string', description: 'Start date (YYYY-MM-DD)' }, + endDate: { type: 'string', description: 'End date (YYYY-MM-DD)' }, + ownerUserIds: { type: 'string', description: 'Comma-separated owner user IDs' }, + spendLimitPeriodRecurrenceType: { + type: 'string', + description: + 'Spend limit period recurrence (PER_WEEK, PER_MONTH, PER_QUARTER, PER_YEAR, ONE_TIME)', + }, + spendType: { type: 'string', description: 'Spend limit spend type' }, + expenseVisibility: { + type: 'string', + description: 'Spend limit expense visibility (SHARED, PRIVATE)', + }, + authorizationVisibility: { + type: 'string', + description: 'Spend limit authorization visibility (PUBLIC, PRIVATE)', + }, + limitIncreaseSetting: { + type: 'string', + description: 'Whether members can request limit increases', + }, + autoTransferCardsSetting: { + type: 'string', + description: 'Auto transfer setting for virtual cards', + }, + autoCreateLimitCardsSetting: { + type: 'string', + description: 'Auto limit card creation setting', + }, + expensePolicyId: { type: 'string', description: 'Expense policy ID for the spend limit' }, + baseLimitAmount: { + type: 'string', + description: 'Base spend limit amount before increases/rollovers', + }, + authorizationType: { + type: 'string', + description: 'Spend limit authorization type (HARD, SOFT)', + }, + rolloverRefreshRate: { type: 'string', description: 'Spend limit rollover refresh rate' }, + limitBufferPercentage: { type: 'string', description: 'Flexible buffer percentage (0-100)' }, + transactionLimitAmount: { type: 'string', description: 'Per-transaction limit amount' }, + spendLimitMemberUserIds: { + type: 'string', + description: 'Comma-separated member user IDs for a new spend limit', + }, }, outputs: { items: { type: 'json', description: 'Items returned by list operations' }, @@ -543,7 +1031,7 @@ export const BrexBlock: BlockConfig = { expenseId: { type: 'string', description: 'ID of the expense the receipt was attached to' }, firstName: { type: 'string', description: 'First name of the user' }, lastName: { type: 'string', description: 'Last name of the user' }, - email: { type: 'string', description: 'Email address of the user' }, + email: { type: 'string', description: 'Email address of the user or vendor' }, managerId: { type: 'string', description: 'Manager ID of the user' }, titleId: { type: 'string', description: 'Title ID of the user' }, legalName: { type: 'string', description: 'Legal name of the company' }, @@ -660,6 +1148,15 @@ export const BrexBlockMeta = { tags: ['automation'], alsoIntegrations: ['gmail'], }, + { + icon: BrexIcon, + title: 'Brex approved-invoice payment automation', + prompt: + 'Build a workflow that takes an approved invoice (vendor name, amount, and memo), creates the vendor in Brex if it does not already exist, and creates a transfer from the company cash account to pay it.', + modules: ['workflows'], + category: 'operations', + tags: ['automation'], + }, { icon: BrexIcon, title: 'Brex team directory assistant', @@ -717,5 +1214,11 @@ export const BrexBlockMeta = { content: '# Statement Reconciliation\n\nTie a card statement back to its underlying transactions.\n\n## Steps\n1. List card statements and pick the period to reconcile.\n2. List card transactions posted within that period using the posted-at filter.\n3. Compare transaction totals to the statement start and end balances and flag gaps.\n\n## Output\nReturn the statement period, its balances, the transaction total for the period, and any discrepancy that needs review.', }, + { + name: 'pay-vendor', + description: 'Pay a vendor from a Brex cash account, creating the vendor first if needed.', + content: + "# Pay a Vendor\n\nSend a payment to a vendor through Brex.\n\n## Steps\n1. List vendors and check if the target vendor already exists by name.\n2. If not, use Create Vendor with the company name (and email/phone if known).\n3. Use the vendor's payment_accounts entry to get the payment_instrument_id, then use Create Transfer with that ID, the cash account to pay from, the amount (in cents), a description, and an external memo.\n4. Confirm the transfer was created and note its status.\n\n## Output\nReturn the vendor used, the transfer ID and status, and the amount sent. Flag anything that requires manual approval (PENDING_APPROVAL status).", + }, ], } as const satisfies BlockMeta diff --git a/apps/sim/blocks/blocks/clickhouse.ts b/apps/sim/blocks/blocks/clickhouse.ts index 9372883faad..6fe93105867 100644 --- a/apps/sim/blocks/blocks/clickhouse.ts +++ b/apps/sim/blocks/blocks/clickhouse.ts @@ -1,4 +1,13 @@ -import { Bell, ClipboardList, Database, File, Search, Server, Trash, Wrench } from '@sim/emcn/icons' +import { + Bell, + ClipboardList, + Database, + File, + Search, + Server, + TrashOutline, + Wrench, +} from '@sim/emcn/icons' import { getErrorMessage } from '@sim/utils/errors' import { ClickHouseIcon } from '@/components/icons' import type { BlockConfig, BlockMeta } from '@/blocks/types' @@ -516,7 +525,7 @@ export const ClickHouseBlockMeta = { tags: ['data-warehouse', 'maintenance'], }, { - icon: Trash, + icon: TrashOutline, title: 'Partition retention cleanup', prompt: 'Build a scheduled workflow that enforces a retention policy on my ClickHouse events table: take an explicit cutoff date as input, list the table partitions, select only the partitions on that exact table whose range ends strictly before the cutoff, and drop just those. Never infer the cutoff and never drop a partition that is not clearly past it.', diff --git a/apps/sim/blocks/blocks/gmail.ts b/apps/sim/blocks/blocks/gmail.ts index d49483d7630..fc4761d5c7f 100644 --- a/apps/sim/blocks/blocks/gmail.ts +++ b/apps/sim/blocks/blocks/gmail.ts @@ -1,4 +1,4 @@ -import { Card } from '@sim/emcn/icons' +import { ClipboardList } from '@sim/emcn/icons' import { GmailIcon, LemlistIcon } from '@/components/icons' import { getScopesForService } from '@/lib/oauth/utils' import type { BlockConfig, BlockMeta } from '@/blocks/types' @@ -687,7 +687,7 @@ export const GmailBlockMeta = { alsoIntegrations: ['slack'], }, { - icon: Card, + icon: ClipboardList, title: 'Invoice processor', prompt: 'Build a workflow that processes invoice PDFs from Gmail, extracts vendor name, amount, due date, and line items, then logs everything to a tracking table and sends a Slack alert for invoices due within 7 days.', diff --git a/apps/sim/blocks/blocks/sftp.ts b/apps/sim/blocks/blocks/sftp.ts index f2c86991f13..16e19a81301 100644 --- a/apps/sim/blocks/blocks/sftp.ts +++ b/apps/sim/blocks/blocks/sftp.ts @@ -1,4 +1,12 @@ -import { ClipboardList, Download, File, Search, Server, Trash, Upload } from '@sim/emcn/icons' +import { + ClipboardList, + Download, + File, + Search, + Server, + TrashOutline, + Upload, +} from '@sim/emcn/icons' import { SftpIcon } from '@/components/icons' import type { BlockConfig, BlockMeta } from '@/blocks/types' import { AuthMode, IntegrationType } from '@/blocks/types' @@ -344,7 +352,7 @@ export const SftpBlockMeta = { tags: ['files', 'sync', 'sftp'], }, { - icon: Trash, + icon: TrashOutline, title: 'Archive and delete old files on a schedule', prompt: 'Build a workflow that runs nightly, lists an SFTP directory, downloads files older than a threshold to archive them, and then deletes the originals from the remote server.', diff --git a/apps/sim/blocks/blocks/ssh.ts b/apps/sim/blocks/blocks/ssh.ts index f3cecc0df1d..2b5b50a1879 100644 --- a/apps/sim/blocks/blocks/ssh.ts +++ b/apps/sim/blocks/blocks/ssh.ts @@ -1,13 +1,5 @@ -import { - ClipboardList, - Download, - File, - Search, - Server, - TerminalWindow, - Wrench, -} from '@sim/emcn/icons' -import { SshIcon } from '@/components/icons' +import { ClipboardList, Download, File, Search, Server, Wrench } from '@sim/emcn/icons' +import { SshIcon, SshTerminalIcon } from '@/components/icons' import type { BlockConfig, BlockMeta } from '@/blocks/types' import { AuthMode, IntegrationType } from '@/blocks/types' import type { SSHResponse } from '@/tools/ssh/types' @@ -616,7 +608,7 @@ export const SSHBlockMeta = { url: 'https://www.openssh.com', templates: [ { - icon: TerminalWindow, + icon: SshTerminalIcon, title: 'Scheduled server health check to Slack', prompt: 'Every morning, SSH into the production server and run a health check command (uptime, load average, and free memory). Summarize stdout with an agent and post the result to the #ops Slack channel so the team starts the day knowing the box is healthy.', diff --git a/apps/sim/blocks/custom/client-overlay.ts b/apps/sim/blocks/custom/client-overlay.ts index cc1bbff01e1..4dfab2aac94 100644 --- a/apps/sim/blocks/custom/client-overlay.ts +++ b/apps/sim/blocks/custom/client-overlay.ts @@ -24,11 +24,22 @@ registerBlockOverlayResolver({ all: () => [...map.values()], }) +/** + * Bump the overlay version and notify subscribers that block-registry-derived + * data changed. Shared signal for BOTH custom-block hydration and the + * block-visibility hydrate path — subscribers treat the version as an opaque + * "re-read `getAllBlocks()`" token, never as "custom blocks specifically + * changed". + */ +export function notifyBlockOverlayChanged(): void { + version += 1 + for (const listener of listeners) listener() +} + /** Replace the in-scope custom blocks and notify subscribers. */ export function hydrateClientCustomBlocks(configs: BlockConfig[]): void { map = new Map(configs.map((config) => [config.type, config])) - version += 1 - for (const listener of listeners) listener() + notifyBlockOverlayChanged() } function subscribe(listener: () => void): () => void { diff --git a/apps/sim/blocks/integration-matcher.ts b/apps/sim/blocks/integration-matcher.ts index 148d5cc6a3d..172d07e9be2 100644 --- a/apps/sim/blocks/integration-matcher.ts +++ b/apps/sim/blocks/integration-matcher.ts @@ -1,6 +1,7 @@ import { LandingPromptStorage } from '@/lib/core/utils/browser-storage' import { getCanonicalBlocksByCategory } from '@/blocks/registry' import type { BlockIcon } from '@/blocks/types' +import { registerBlockCacheInvalidator } from '@/blocks/visibility/context' /** * Public descriptor for a single integration block, exposed to UI surfaces @@ -46,6 +47,18 @@ function normalizeDisplayName(name: string): string { let cachedMatcher: IntegrationMatcher | null = null let cachedList: readonly IntegrationDescriptor[] | null = null +/** + * Drops the memoized matcher/list. Registered with the block cache-invalidator + * seam so a client block-visibility change (preview reveal / kill switch) + * rebuilds the matcher against the new canonical block set. + */ +function clearIntegrationMatcherCache(): void { + cachedMatcher = null + cachedList = null +} + +registerBlockCacheInvalidator(clearIntegrationMatcherCache) + function buildMatcher(): IntegrationMatcher { const byName = new Map() const names: string[] = [] diff --git a/apps/sim/blocks/registry.ts b/apps/sim/blocks/registry.ts index e7760a87974..d0019e28aaf 100644 --- a/apps/sim/blocks/registry.ts +++ b/apps/sim/blocks/registry.ts @@ -1,4 +1,5 @@ import { stripVersionSuffix } from '@sim/utils/string' +import type { BlockVisibilityState } from '@/lib/core/config/block-visibility' import { overlayBlocks, resolveOverlayBlock } from '@/blocks/custom/overlay' import { BLOCK_META_REGISTRY, BLOCK_REGISTRY } from '@/blocks/registry-maps' import type { @@ -8,6 +9,7 @@ import type { BlockTemplate, SuggestedSkill, } from '@/blocks/types' +import { isHiddenUnder, overlayVisibility } from '@/blocks/visibility/context' /** * Normalize an external block type to its registry key form: dashes become @@ -22,9 +24,58 @@ export function getBlock(type: string): BlockConfig | undefined { return BLOCK_REGISTRY[type] ?? BLOCK_REGISTRY[normalizeType(type)] ?? resolveOverlayBlock(type) } -/** All block configs, including any in-scope custom blocks from the overlay. */ +/** Whether any registered block is an unreleased `preview` block. Static — computed once. */ +const HAS_PREVIEW_BLOCKS = Object.values(BLOCK_REGISTRY).some((block) => block.preview) + +/** + * True when the visibility projection cannot change any block, so accessors can + * return raw arrays untouched: no `preview` blocks exist (they must be hidden + * even with a null state) and no kill-switch entries apply. + */ +function visibilityInert(vis: BlockVisibilityState | null): boolean { + if (HAS_PREVIEW_BLOCKS) return false + return vis === null || vis.disabled.size === 0 +} + +/** + * Effective hidden state for discovery surfaces: static `hideFromToolbar` + * (superseded versions, disabled custom blocks) plus the per-viewer visibility + * predicate ({@link isHiddenUnder}: unrevealed `preview` blocks — fail-closed + * even without a context — and kill-switched types). + */ +function effectiveHidden(block: BlockConfig, vis: BlockVisibilityState | null): boolean { + if (block.hideFromToolbar) return true + return isHiddenUnder(vis, block) +} + +/** + * Project a block through the viewer's visibility: gated blocks become shallow + * clones with `hideFromToolbar: true` (CLONE-NOT-REMOVE — gated blocks must stay + * in `getAllBlocks()` output because `.find`-by-type consumers rely on it), and + * revealed-but-not-GA preview blocks get a display " (Preview)" name suffix. + * The `!block.hideFromToolbar` guard keeps already-hidden blocks (including + * disabled custom blocks) un-cloned and never suffixed. + */ +function projectBlock(block: BlockConfig, vis: BlockVisibilityState | null): BlockConfig { + if (effectiveHidden(block, vis) && !block.hideFromToolbar) { + return { ...block, hideFromToolbar: true } + } + if (block.preview && vis?.previewTagged.has(block.type)) { + return { ...block, name: `${block.name} (Preview)` } + } + return block +} + +/** + * All block configs, including any in-scope custom blocks from the overlay, + * projected through the viewer's block visibility. Execution paths are + * unaffected: they resolve via the pure {@link getBlock}. + */ export function getAllBlocks(): BlockConfig[] { - return [...Object.values(BLOCK_REGISTRY), ...overlayBlocks()] + const all = [...Object.values(BLOCK_REGISTRY), ...overlayBlocks()] + const vis = overlayVisibility() + if (visibilityInert(vis)) return all + return all.map((block) => projectBlock(block, vis)) } /** Find the block whose `tools.access` contains the given tool id. */ @@ -77,14 +128,17 @@ export function getBlocksByCategory(category: BlockCategory): BlockConfig[] { * category. This is the single source of truth shared by every surface that * extracts blocks for presentation — the toolbar, the search/mention engine, * and the integrations catalog. A block is included when its `category` - * matches and it is not hidden from the toolbar (i.e. it is the latest - * version under the upgrade paradigm, since superseded versions set - * `hideFromToolbar: true`). + * matches and it is not effectively hidden: not `hideFromToolbar` (superseded + * versions), not an unrevealed `preview` block, and not kill-switched for the + * viewer. Visible blocks are projected so revealed preview blocks carry their + * " (Preview)" display suffix. */ export function getCanonicalBlocksByCategory(category: BlockCategory): BlockConfig[] { - return [...Object.values(BLOCK_REGISTRY), ...overlayBlocks()].filter( - (block) => block.category === category && !block.hideFromToolbar + const vis = overlayVisibility() + const blocks = [...Object.values(BLOCK_REGISTRY), ...overlayBlocks()].filter( + (block) => block.category === category && !effectiveHidden(block, vis) ) + return visibilityInert(vis) ? blocks : blocks.map((block) => projectBlock(block, vis)) } /** All registered block type identifiers. */ diff --git a/apps/sim/blocks/types.ts b/apps/sim/blocks/types.ts index 7e3f9636bef..6d9f63188ab 100644 --- a/apps/sim/blocks/types.ts +++ b/apps/sim/blocks/types.ts @@ -462,6 +462,16 @@ export interface BlockConfig { } } hideFromToolbar?: boolean + /** + * Marks an unreleased block. Preview blocks are hidden from every discovery + * surface (toolbar, search, mentions, copilot/VFS, docs) in every environment — + * hosted, self-hosted, dev, and SSR — until revealed via the hosted + * `block-visibility` AppConfig document or the `PREVIEW_BLOCKS` env allowlist. + * Fail-closed by design; distinct from {@link hideFromToolbar} (permanently + * hidden superseded versions). Execution of already-placed instances is never + * gated. Remove at GA. + */ + preview?: boolean triggers?: { enabled: boolean available: string[] // List of trigger IDs this block supports diff --git a/apps/sim/blocks/visibility/client.test.ts b/apps/sim/blocks/visibility/client.test.ts new file mode 100644 index 00000000000..426a099da54 --- /dev/null +++ b/apps/sim/blocks/visibility/client.test.ts @@ -0,0 +1,89 @@ +/** + * @vitest-environment node + */ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const { mockNotify } = vi.hoisted(() => ({ mockNotify: vi.fn() })) + +vi.mock('@/blocks/custom/client-overlay', () => ({ + notifyBlockOverlayChanged: mockNotify, +})) + +import type { BlockVisibilityState } from '@/lib/core/config/block-visibility' +// client-boundary-allow: vitest ignores the 'use client' directive; this node-env test exercises the module directly +import { hydrateBlockVisibility, resetBlockVisibilityForSwitch } from '@/blocks/visibility/client' +import { overlayVisibility, registerBlockCacheInvalidator } from '@/blocks/visibility/context' + +function state(revealed: string[], disabled: string[] = []): BlockVisibilityState { + return { + revealed: new Set(revealed), + disabled: new Set(disabled), + previewTagged: new Set(revealed), + } +} + +describe('hydrateBlockVisibility', () => { + beforeEach(() => vi.clearAllMocks()) + + // Must run FIRST: module state starts null and persists across tests. + it('treats an empty state while none is set as a no-op (null ≡ empty)', () => { + const invalidator = vi.fn() + const unregister = registerBlockCacheInvalidator(invalidator) + + hydrateBlockVisibility(state([])) + expect(overlayVisibility()).toBeNull() + expect(invalidator).not.toHaveBeenCalled() + expect(mockNotify).not.toHaveBeenCalled() + + unregister() + }) + + it('applies state, fires invalidators, and bumps the overlay version', () => { + const invalidator = vi.fn() + const unregister = registerBlockCacheInvalidator(invalidator) + + hydrateBlockVisibility(state(['gmail_v2'])) + expect(overlayVisibility()?.revealed.has('gmail_v2')).toBe(true) + expect(invalidator).toHaveBeenCalledTimes(1) + expect(mockNotify).toHaveBeenCalledTimes(1) + + unregister() + }) + + it('drops reveals but carries kill switches over on workspace switch', () => { + const invalidator = vi.fn() + const unregister = registerBlockCacheInvalidator(invalidator) + + // Reveal + kill-switch in workspace A, then switch (loader resets while the + // new projection loads). (notion_v3: distinct from prior tests' state — + // module state persists.) + hydrateBlockVisibility(state(['notion_v3'], ['slack'])) + resetBlockVisibilityForSwitch() + expect(overlayVisibility()?.revealed.size).toBe(0) + expect(overlayVisibility()?.previewTagged.size).toBe(0) + expect(overlayVisibility()?.disabled).toEqual(new Set(['slack'])) + expect(invalidator).toHaveBeenCalledTimes(2) + + // Repeated resets are no-ops (deep-equal). + resetBlockVisibilityForSwitch() + expect(invalidator).toHaveBeenCalledTimes(2) + + unregister() + }) + + it('no-ops on a deep-equal state (fresh objects, same content)', () => { + const invalidator = vi.fn() + const unregister = registerBlockCacheInvalidator(invalidator) + + hydrateBlockVisibility(state(['gmail_v2'], ['slack'])) + hydrateBlockVisibility(state(['gmail_v2'], ['slack'])) + expect(invalidator).toHaveBeenCalledTimes(1) + expect(mockNotify).toHaveBeenCalledTimes(1) + + hydrateBlockVisibility(state(['gmail_v2', 'notion_v3'], ['slack'])) + expect(invalidator).toHaveBeenCalledTimes(2) + expect(mockNotify).toHaveBeenCalledTimes(2) + + unregister() + }) +}) diff --git a/apps/sim/blocks/visibility/client.ts b/apps/sim/blocks/visibility/client.ts new file mode 100644 index 00000000000..4079c91a5e0 --- /dev/null +++ b/apps/sim/blocks/visibility/client.ts @@ -0,0 +1,70 @@ +'use client' + +import type { BlockVisibilityState } from '@/lib/core/config/block-visibility' +import { notifyBlockOverlayChanged } from '@/blocks/custom/client-overlay' +import { invalidateBlockCaches, registerBlockVisibilityResolver } from '@/blocks/visibility/context' + +/** + * Client-side visibility state, hydrated from `useBlockVisibility` by + * `BlockVisibilityLoader`. Registered at module load with `null` state so the + * very first render — including the SSR pass — is fail-closed for `preview` + * blocks; the post-mount fetch only ever reveals (benign pop-in) or applies a + * kill switch to an already-public block. + */ +let state: BlockVisibilityState | null = null + +registerBlockVisibilityResolver({ current: () => state }) + +function setsEqual(a: Set, b: Set): boolean { + if (a.size !== b.size) return false + for (const value of a) if (!b.has(value)) return false + return true +} + +function isEmptyState(vis: BlockVisibilityState): boolean { + return vis.revealed.size === 0 && vis.disabled.size === 0 && vis.previewTagged.size === 0 +} + +/** + * Replace the in-scope visibility state, reset registered module caches, and + * bump the shared block-overlay version so every subscribed consumer re-reads + * `getAllBlocks()`. + * + * No-ops when the change cannot alter the projection: an incoming state + * deep-equal to the current one (React Query refetches deliver + * fresh-but-identical objects on every poll — without this guard each poll + * would thundering-rebuild the toolbar, search, and matcher caches), or an + * empty state while none is set (`null` and empty are equivalent for + * `isHiddenUnder`, so the fail-closed reset on first mount is free). + */ +export function hydrateBlockVisibility(next: BlockVisibilityState): void { + if (state === null && isEmptyState(next)) return + if ( + state && + setsEqual(state.revealed, next.revealed) && + setsEqual(state.disabled, next.disabled) && + setsEqual(state.previewTagged, next.previewTagged) + ) { + return + } + state = next + invalidateBlockCaches() + notifyBlockOverlayChanged() +} + +/** + * Fail-closed reset while a workspace switch's visibility fetch is in flight: + * preview reveals are dropped immediately (they may not apply to the new + * workspace), but kill-switch entries are CARRIED OVER until the new + * projection arrives — dropping `disabled` would flash kill-switched blocks + * back into discovery for the flight window, while briefly over-hiding in the + * new workspace is benign in both directions. + */ +export function resetBlockVisibilityForSwitch(): void { + if (state === null) return + hydrateBlockVisibility({ + revealed: new Set(), + disabled: state.disabled, + previewTagged: new Set(), + }) +} diff --git a/apps/sim/blocks/visibility/context.ts b/apps/sim/blocks/visibility/context.ts new file mode 100644 index 00000000000..7c191603035 --- /dev/null +++ b/apps/sim/blocks/visibility/context.ts @@ -0,0 +1,73 @@ +import type { BlockVisibilityState } from '@/lib/core/config/block-visibility' +import type { BlockConfig } from '@/blocks/types' + +/** + * Resolver for the per-viewer block-visibility projection, mirroring the + * custom-block overlay seam (`@/blocks/custom/overlay`) but deliberately + * independent of it: visibility is a discovery concern with its own lifecycle, + * and a separate AsyncLocalStorage composes with `withCustomBlockOverlay` + * without `store.run` clobbering. + * + * Two environment-specific resolvers register here: + * - client: module state hydrated from `useBlockVisibility` (see `client.ts`) + * - server: an AsyncLocalStorage scoped per request (see `server-context.ts`) + * + * This module is isomorphic (no `'use client'`, no `node:` imports) so + * `@/blocks/registry` stays importable on both sides. When NO resolver state is + * active, `preview` blocks are still hidden ({@link isHiddenUnder} treats a null + * state as "nothing revealed") — fail-closed is the default everywhere, + * including SSR and server paths outside `withBlockVisibility`. + */ +export interface BlockVisibilityResolver { + current(): BlockVisibilityState | null +} + +let resolver: BlockVisibilityResolver | null = null + +/** Register (or clear with `null`) the active visibility resolver for this environment. */ +export function registerBlockVisibilityResolver(next: BlockVisibilityResolver | null): void { + resolver = next +} + +/** The visibility state in scope, or `null` when none (= nothing revealed, nothing disabled). */ +export function overlayVisibility(): BlockVisibilityState | null { + return resolver?.current() ?? null +} + +/** + * THE single hidden-predicate for block gating — every surface that hides + * blocks (registry projection, VFS stamp filter, exposed-integration-tools + * filter, `get_blocks_metadata`) calls this; never restate the rule inline. + * + * A block is hidden when it is an unrevealed `preview` block (fail-closed even + * with a null state) or when the kill switch (`disabled`) names it. Static + * `hideFromToolbar` is deliberately NOT part of this predicate — callers that + * need it check it separately. + */ +export function isHiddenUnder( + vis: BlockVisibilityState | null, + block: Pick +): boolean { + if (block.preview && !vis?.revealed.has(block.type)) return true + if (vis?.disabled.has(block.type)) return true + return false +} + +/** + * Registry of non-React module-cache resets (e.g. the tool-operations search + * index, the integration matcher) fired when the client visibility state + * changes. Lives here — not in the `'use client'` module — so plain `lib/` + * modules can register at load time on either side of the server boundary. + */ +const cacheInvalidators = new Set<() => void>() + +/** Register a cache reset to run on visibility changes. Returns an unregister fn. */ +export function registerBlockCacheInvalidator(fn: () => void): () => void { + cacheInvalidators.add(fn) + return () => cacheInvalidators.delete(fn) +} + +/** Fire every registered cache reset (called by the client hydrate path). */ +export function invalidateBlockCaches(): void { + for (const fn of cacheInvalidators) fn() +} diff --git a/apps/sim/blocks/visibility/registry-inert.test.ts b/apps/sim/blocks/visibility/registry-inert.test.ts new file mode 100644 index 00000000000..ddb7551737b --- /dev/null +++ b/apps/sim/blocks/visibility/registry-inert.test.ts @@ -0,0 +1,60 @@ +/** + * @vitest-environment node + */ +import { afterEach, describe, expect, it, vi } from 'vitest' + +vi.unmock('@/blocks/registry') + +const { synthRegistry } = vi.hoisted(() => ({ + synthRegistry: { + slack: { + type: 'slack', + name: 'Slack', + description: '', + category: 'tools', + bgColor: '#000', + icon: () => null, + subBlocks: [], + tools: { access: [] }, + inputs: {}, + outputs: {}, + }, + }, +})) + +vi.mock('@/blocks/registry-maps', () => ({ + BLOCK_REGISTRY: synthRegistry, + BLOCK_META_REGISTRY: {}, +})) + +import { getAllBlocks } from '@/blocks/registry' +import { registerBlockVisibilityResolver } from '@/blocks/visibility/context' + +afterEach(() => registerBlockVisibilityResolver(null)) + +describe('registry fast path (no preview blocks registered)', () => { + it('returns raw references with no context', () => { + expect(getAllBlocks()[0]).toBe(synthRegistry.slack) + }) + + it('returns raw references when the active state has no kill-switch entries', () => { + const state = { + revealed: new Set(['whatever']), + disabled: new Set(), + previewTagged: new Set(['whatever']), + } + registerBlockVisibilityResolver({ current: () => state }) + expect(getAllBlocks()[0]).toBe(synthRegistry.slack) + }) + + it('still projects when a kill-switch entry applies', () => { + const state = { + revealed: new Set(), + disabled: new Set(['slack']), + previewTagged: new Set(), + } + registerBlockVisibilityResolver({ current: () => state }) + expect(getAllBlocks()[0]?.hideFromToolbar).toBe(true) + expect(synthRegistry.slack).not.toHaveProperty('hideFromToolbar', true) + }) +}) diff --git a/apps/sim/blocks/visibility/server-context.ts b/apps/sim/blocks/visibility/server-context.ts new file mode 100644 index 00000000000..e713779c43c --- /dev/null +++ b/apps/sim/blocks/visibility/server-context.ts @@ -0,0 +1,25 @@ +import { AsyncLocalStorage } from 'node:async_hooks' +import type { BlockVisibilityState } from '@/lib/core/config/block-visibility' +import { registerBlockVisibilityResolver } from '@/blocks/visibility/context' + +/** + * Server-side visibility context: a per-request AsyncLocalStorage, independent + * of the custom-block overlay's ALS so `withBlockVisibility` and + * `withCustomBlockOverlay` nest in either order without clobbering each other. + * + * Only copilot/mothership discovery paths establish this scope. Execution entry + * points (execute route, trigger.dev tasks, schedules/webhooks) never do — so + * placed preview blocks always serialize and run, and `preview` blocks stay + * hidden on unscoped discovery reads purely via the static fail-closed default. + */ +const store = new AsyncLocalStorage() + +registerBlockVisibilityResolver({ current: () => store.getStore() ?? null }) + +/** Run `fn` with the given visibility state resolvable via the registry accessors. */ +export function withBlockVisibility( + vis: BlockVisibilityState, + fn: () => Promise +): Promise { + return store.run(vis, fn) +} diff --git a/apps/sim/blocks/visibility/visibility.test.ts b/apps/sim/blocks/visibility/visibility.test.ts new file mode 100644 index 00000000000..6cd95f22435 --- /dev/null +++ b/apps/sim/blocks/visibility/visibility.test.ts @@ -0,0 +1,149 @@ +/** + * @vitest-environment node + */ +import { afterEach, describe, expect, it, vi } from 'vitest' + +vi.unmock('@/blocks/registry') + +const { synthRegistry } = vi.hoisted(() => { + const block = (type: string, extra: Record = {}) => ({ + type, + name: type.toUpperCase(), + description: '', + category: 'tools', + bgColor: '#000', + icon: () => null, + subBlocks: [], + tools: { access: [] }, + inputs: {}, + outputs: {}, + ...extra, + }) + return { + synthRegistry: { + slack: block('slack'), + gmail_v2: block('gmail_v2', { preview: true }), + old_v1: block('old_v1', { hideFromToolbar: true }), + }, + } +}) + +vi.mock('@/blocks/registry-maps', () => ({ + BLOCK_REGISTRY: synthRegistry, + BLOCK_META_REGISTRY: {}, +})) + +import type { BlockVisibilityState } from '@/lib/core/config/block-visibility' +import { registerBlockOverlayResolver } from '@/blocks/custom/overlay' +import { getAllBlocks, getBlock, getCanonicalBlocksByCategory } from '@/blocks/registry' +import type { BlockConfig } from '@/blocks/types' +import { isHiddenUnder, registerBlockVisibilityResolver } from '@/blocks/visibility/context' + +function vis(partial: Partial = {}): BlockVisibilityState { + return { + revealed: new Set(), + disabled: new Set(), + previewTagged: new Set(), + ...partial, + } +} + +function withVisibility(state: BlockVisibilityState | null) { + registerBlockVisibilityResolver(state ? { current: () => state } : null) +} + +const byType = (blocks: BlockConfig[], type: string) => blocks.find((b) => b.type === type) + +afterEach(() => { + registerBlockVisibilityResolver(null) + registerBlockOverlayResolver(null) +}) + +describe('isHiddenUnder', () => { + it('hides unrevealed preview blocks even with a null state (fail-closed)', () => { + expect(isHiddenUnder(null, { type: 'gmail_v2', preview: true })).toBe(true) + expect(isHiddenUnder(null, { type: 'slack' })).toBe(false) + }) + + it('reveals preview blocks named in revealed', () => { + const state = vis({ revealed: new Set(['gmail_v2']) }) + expect(isHiddenUnder(state, { type: 'gmail_v2', preview: true })).toBe(false) + }) + + it('hides kill-switched types only with an active state', () => { + const state = vis({ disabled: new Set(['slack']) }) + expect(isHiddenUnder(state, { type: 'slack' })).toBe(true) + expect(isHiddenUnder(null, { type: 'slack' })).toBe(false) + }) +}) + +describe('registry projection', () => { + it('hides preview blocks without any context: clone-not-remove', () => { + withVisibility(null) + const all = getAllBlocks() + const gmail = byType(all, 'gmail_v2') + expect(gmail).toBeDefined() + expect(gmail?.hideFromToolbar).toBe(true) + // the underlying registry entry is untouched + expect(getBlock('gmail_v2')?.hideFromToolbar).toBeUndefined() + // canonical (filtered) set excludes it + expect(byType(getCanonicalBlocksByCategory('tools'), 'gmail_v2')).toBeUndefined() + }) + + it('reveals a preview block with a " (Preview)" display suffix when tagged', () => { + withVisibility(vis({ revealed: new Set(['gmail_v2']), previewTagged: new Set(['gmail_v2']) })) + expect(byType(getAllBlocks(), 'gmail_v2')?.name).toBe('GMAIL_V2 (Preview)') + const canonical = byType(getCanonicalBlocksByCategory('tools'), 'gmail_v2') + expect(canonical?.name).toBe('GMAIL_V2 (Preview)') + expect(canonical?.hideFromToolbar).toBeUndefined() + }) + + it('reveals a config-GA preview block without a suffix', () => { + withVisibility(vis({ revealed: new Set(['gmail_v2']) })) + expect(byType(getAllBlocks(), 'gmail_v2')?.name).toBe('GMAIL_V2') + }) + + it('kill-switches a shipped block only when a context is active', () => { + withVisibility(vis({ disabled: new Set(['slack']) })) + expect(byType(getAllBlocks(), 'slack')?.hideFromToolbar).toBe(true) + expect(byType(getCanonicalBlocksByCategory('tools'), 'slack')).toBeUndefined() + + withVisibility(null) + expect(byType(getAllBlocks(), 'slack')?.hideFromToolbar).toBeUndefined() + }) + + it('keeps getBlock pure regardless of visibility', () => { + withVisibility( + vis({ + revealed: new Set(['gmail_v2']), + previewTagged: new Set(['gmail_v2']), + disabled: new Set(['slack']), + }) + ) + expect(getBlock('gmail_v2')?.name).toBe('GMAIL_V2') + expect(getBlock('slack')?.hideFromToolbar).toBeUndefined() + }) + + it('returns untouched references for unaffected blocks', () => { + withVisibility(vis({ revealed: new Set(['gmail_v2']) })) + expect(byType(getAllBlocks(), 'slack')).toBe(synthRegistry.slack) + expect(byType(getAllBlocks(), 'old_v1')).toBe(synthRegistry.old_v1) + }) + + it('never re-clones or suffixes already-hidden custom blocks', () => { + const disabledCustom = { + ...synthRegistry.slack, + type: 'custom_block_abc', + name: 'My Custom', + hideFromToolbar: true, + } as BlockConfig + registerBlockOverlayResolver({ + get: (t) => (t === disabledCustom.type ? disabledCustom : undefined), + all: () => [disabledCustom], + }) + withVisibility(vis({ revealed: new Set(['gmail_v2']) })) + const projected = byType(getAllBlocks(), 'custom_block_abc') + expect(projected).toBe(disabledCustom) + expect(projected?.name).toBe('My Custom') + }) +}) diff --git a/apps/sim/components/icons.tsx b/apps/sim/components/icons.tsx index c2b6ed56e2e..fe869a5b77e 100644 --- a/apps/sim/components/icons.tsx +++ b/apps/sim/components/icons.tsx @@ -3696,7 +3696,7 @@ export function MetaIcon(props: SVGProps) { @@ -5628,6 +5628,27 @@ export function SshIcon(props: SVGProps) { ) } +export function SshTerminalIcon(props: SVGProps) { + return ( + + + + + + + ) +} + export function SftpIcon(props: SVGProps) { return ( { const blocks = getAllBlocks().filter((b) => !isBlockTypeAccessControlExempt(b.type)) return blocks.sort((a, b) => { @@ -628,6 +639,14 @@ export function GroupDetail({ }) }, [customBlockOverlayVersion]) + /** + * The RENDERED list: hides blocks gated for this viewer by reading the + * registry projection's effective flag off the clone (the single source of + * truth — never re-derive visibility here). Revealed viewers see preview + * blocks (with their " (Preview)" suffix) and can toggle them explicitly. + */ + const visibleBlocks = useMemo(() => allBlocks.filter((b) => !b.hideFromToolbar), [allBlocks]) + const allProviderIds = useMemo(() => { const allIds = getAllProviderIds() const blacklist = blacklistedProvidersData?.blacklistedProviders ?? [] @@ -830,10 +849,10 @@ export function GroupDetail({ }, [allProviderIds, providerSearchTerm]) const filteredBlocks = useMemo(() => { - if (!integrationSearchTerm.trim()) return allBlocks + if (!integrationSearchTerm.trim()) return visibleBlocks const query = integrationSearchTerm.toLowerCase() - return allBlocks.filter((b) => b.name.toLowerCase().includes(query)) - }, [allBlocks, integrationSearchTerm]) + return visibleBlocks.filter((b) => b.name.toLowerCase().includes(query)) + }, [visibleBlocks, integrationSearchTerm]) const filteredCoreBlocks = useMemo( () => filteredBlocks.filter((block) => block.category === 'blocks'), diff --git a/apps/sim/ee/access-control/utils/permission-check.test.ts b/apps/sim/ee/access-control/utils/permission-check.test.ts index be436ca0019..13352b9fb00 100644 --- a/apps/sim/ee/access-control/utils/permission-check.test.ts +++ b/apps/sim/ee/access-control/utils/permission-check.test.ts @@ -379,6 +379,19 @@ describe('validateBlockType', () => { await validateBlockType(undefined, undefined, 'notion') }) + it('does NOT treat preview blocks as exempt — preview is not legacy', async () => { + // A `preview: true` block has static hideFromToolbar unset, so it is a + // normal access-controlled block: visibility gating (discovery) and + // permission-group enforcement (execution) are deliberately independent. + mockGetBlock.mockImplementation((type) => + type === 'gmail_v2' ? ({ preview: true } as { hideFromToolbar?: boolean }) : undefined + ) + + await expect(validateBlockType(undefined, undefined, 'gmail_v2')).rejects.toThrow( + IntegrationNotAllowedError + ) + }) + it('matches case-insensitively', async () => { await validateBlockType(undefined, undefined, 'Slack') await validateBlockType(undefined, undefined, 'GOOGLE_DRIVE') diff --git a/apps/sim/ee/workspace-forking/lib/remap/block-identity.ts b/apps/sim/ee/workspace-forking/lib/remap/block-identity.ts index fd659d4a646..6360584cf69 100644 --- a/apps/sim/ee/workspace-forking/lib/remap/block-identity.ts +++ b/apps/sim/ee/workspace-forking/lib/remap/block-identity.ts @@ -15,11 +15,16 @@ function uuidToBytes(uuid: string): Buffer { /** * Deterministic UUIDv5 (SHA-1) of `name` within `namespace`. The same inputs * always yield the same UUID, which is how fork block identity stays stable. + * + * SHA-1 is mandated by RFC 4122 for UUIDv5 and is used here only for deterministic id derivation, + * never for secrecy or integrity — not a security use of the algorithm. Swapping it would change + * every derived id, breaking webhook URLs and stored block-id references across existing forks + * (see {@link FORK_BLOCK_NAMESPACE}). */ function uuidV5(name: string, namespace: string): string { const hash = createHash('sha1') - hash.update(uuidToBytes(namespace)) - hash.update(Buffer.from(name, 'utf8')) + hash.update(uuidToBytes(namespace)) // lgtm[js/weak-cryptographic-algorithm] + hash.update(Buffer.from(name, 'utf8')) // lgtm[js/weak-cryptographic-algorithm] const bytes = hash.digest().subarray(0, 16) bytes[6] = (bytes[6] & 0x0f) | 0x50 bytes[8] = (bytes[8] & 0x3f) | 0x80 diff --git a/apps/sim/hooks/queries/block-visibility.ts b/apps/sim/hooks/queries/block-visibility.ts new file mode 100644 index 00000000000..2f073cb08d4 --- /dev/null +++ b/apps/sim/hooks/queries/block-visibility.ts @@ -0,0 +1,31 @@ +import { useQuery } from '@tanstack/react-query' +import { requestJson } from '@/lib/api/client/request' +import { + type BlockVisibilityResponse, + getBlockVisibilityContract, +} from '@/lib/api/contracts/block-visibility' + +export const BLOCK_VISIBILITY_STALE_TIME = 60 * 1000 + +export const blockVisibilityKeys = { + all: ['block-visibility'] as const, + lists: () => [...blockVisibilityKeys.all, 'list'] as const, + list: (workspaceId?: string) => [...blockVisibilityKeys.lists(), workspaceId ?? ''] as const, +} + +async function fetchBlockVisibility( + workspaceId: string, + signal?: AbortSignal +): Promise { + return requestJson(getBlockVisibilityContract, { query: { workspaceId }, signal }) +} + +/** The viewer's block-visibility projection for a workspace (revealed/disabled/preview-tagged types). */ +export function useBlockVisibility(workspaceId?: string) { + return useQuery({ + queryKey: blockVisibilityKeys.list(workspaceId), + queryFn: ({ signal }) => fetchBlockVisibility(workspaceId as string, signal), + enabled: Boolean(workspaceId), + staleTime: BLOCK_VISIBILITY_STALE_TIME, + }) +} diff --git a/apps/sim/hooks/use-autosave.test.tsx b/apps/sim/hooks/use-autosave.test.tsx index e3c6e87ec70..0f035fee248 100644 --- a/apps/sim/hooks/use-autosave.test.tsx +++ b/apps/sim/hooks/use-autosave.test.tsx @@ -4,20 +4,40 @@ import { act } from 'react' import { createRoot, type Root } from 'react-dom/client' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +// jsdom has no real IndexedDB; fake idb-keyval with an in-memory map so draft persistence is +// deterministic and inspectable without depending on a browser implementation. +const { fakeDraftStore } = vi.hoisted(() => ({ fakeDraftStore: new Map() })) +vi.mock('idb-keyval', () => ({ + get: vi.fn((key: string) => Promise.resolve(fakeDraftStore.get(key))), + set: vi.fn((key: string, value: unknown) => { + fakeDraftStore.set(key, value) + return Promise.resolve() + }), + del: vi.fn((key: string) => { + fakeDraftStore.delete(key) + return Promise.resolve() + }), +})) + import { type SaveStatus, useAutosave } from '@/hooks/use-autosave' interface ProbeProps { content: string savedContent: string - onSave: () => Promise + onSave: (overrideContent?: string) => Promise delay?: number enabled?: boolean + draftKey?: string + onRestoreDraft?: (content: string) => void + onDiscardCorrectionFailed?: () => void } interface HookHandle { status: () => SaveStatus isDirty: () => boolean saveImmediately: () => Promise + discard: () => void rerender: (next: Partial) => void unmount: () => void } @@ -31,7 +51,12 @@ function renderAutosave(initial: ProbeProps): { handle: HookHandle; props: Probe const container = document.createElement('div') const root: Root = createRoot(container) const props = { ...initial } - let latest = { saveStatus: 'idle' as SaveStatus, isDirty: false, saveImmediately: async () => {} } + let latest = { + saveStatus: 'idle' as SaveStatus, + isDirty: false, + saveImmediately: async () => {}, + discard: () => {}, + } function Probe(p: ProbeProps) { latest = useAutosave(p) @@ -49,6 +74,7 @@ function renderAutosave(initial: ProbeProps): { handle: HookHandle; props: Probe status: () => latest.saveStatus, isDirty: () => latest.isDirty, saveImmediately: () => latest.saveImmediately(), + discard: () => latest.discard(), rerender: (next) => { Object.assign(props, next) render() @@ -69,6 +95,7 @@ async function flush() { describe('useAutosave', () => { beforeEach(() => { vi.useFakeTimers() + fakeDraftStore.clear() }) afterEach(() => { vi.runOnlyPendingTimers() @@ -257,6 +284,28 @@ describe('useAutosave', () => { expect(onSave).toHaveBeenCalledTimes(1) }) + it('writes a local backup on unmount even if the network flush fails', async () => { + const onSave = vi.fn(async () => { + throw new Error('offline') + }) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-unmount', + }) + + handle.rerender({ content: 'a1' }) + // Unmount before the 400ms local-draft debounce fires — the pending timer is cancelled, so + // the cleanup itself must persist the draft, not just attempt (and here, fail) the network flush. + handle.unmount() + await flush() + expect(fakeDraftStore.get('autosave-draft:file-unmount')).toEqual({ + content: 'a1', + savedContent: 'a', + }) + }) + it('does not flush on unmount when the document is clean', async () => { const onSave = vi.fn(async () => {}) const { handle } = renderAutosave({ content: 'a', savedContent: 'a', onSave }) @@ -264,4 +313,903 @@ describe('useAutosave', () => { await flush() expect(onSave).not.toHaveBeenCalled() }) + + describe('local draft persistence (draftKey)', () => { + it('mirrors dirty edits into IndexedDB on a short debounce, independent of the network save', async () => { + let resolveSave: (() => void) | undefined + const onSave = vi.fn( + () => + new Promise((resolve) => { + resolveSave = resolve + }) + ) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-1', + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(400) + }) + await flush() + // The local draft lands well before the (longer) network debounce fires. + expect(fakeDraftStore.get('autosave-draft:file-1')).toEqual({ + content: 'a1', + savedContent: 'a', + }) + expect(onSave).not.toHaveBeenCalled() + resolveSave?.() + }) + + it('clears the local draft once the network save succeeds and the caller advances savedContent', async () => { + let resolveSave: (() => void) | undefined + const onSave = vi.fn( + () => + new Promise((resolve) => { + resolveSave = resolve + }) + ) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-2', + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + // The network save is in flight; the local draft is still the only record of the edit. + expect(fakeDraftStore.has('autosave-draft:file-2')).toBe(true) + + handle.rerender({ savedContent: 'a1' }) + await act(async () => { + resolveSave?.() + }) + await flush() + expect(fakeDraftStore.has('autosave-draft:file-2')).toBe(false) + }) + + it('does not clear the local draft when a newer edit lands while the save is in flight', async () => { + let resolveSave: (() => void) | undefined + const onSave = vi.fn( + () => + new Promise((resolve) => { + resolveSave = resolve + }) + ) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-2b', + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + expect(fakeDraftStore.has('autosave-draft:file-2b')).toBe(true) + + // A further edit lands while the first save is still in flight. + handle.rerender({ content: 'a12' }) + await act(async () => { + vi.advanceTimersByTime(400) + }) + await flush() + + // The first save resolves; the caller advances savedContent only to the snapshot it saved. + handle.rerender({ savedContent: 'a1' }) + await act(async () => { + resolveSave?.() + }) + await flush() + // Still dirty ('a12' !== 'a1') — the local backup for the untransmitted edit must survive, + // even though `save()`'s success no longer explicitly clears it. + expect(fakeDraftStore.has('autosave-draft:file-2b')).toBe(true) + expect(fakeDraftStore.get('autosave-draft:file-2b')).toMatchObject({ content: 'a12' }) + }) + + it('flushes the draft to IndexedDB when the page becomes hidden', async () => { + const onSave = vi.fn(async () => {}) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-3', + }) + + handle.rerender({ content: 'a1' }) + // No timers advanced — simulates a tab close mid-keystroke, before either debounce fires. + Object.defineProperty(document, 'visibilityState', { value: 'hidden', configurable: true }) + document.dispatchEvent(new Event('visibilitychange')) + await flush() + expect(fakeDraftStore.get('autosave-draft:file-3')).toEqual({ + content: 'a1', + savedContent: 'a', + }) + }) + + it('restores a draft left behind by a prior session, once, on mount', async () => { + fakeDraftStore.set('autosave-draft:file-4', { content: 'recovered', savedContent: 'a' }) + const onSave = vi.fn(async () => {}) + const onRestoreDraft = vi.fn() + renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-4', + onRestoreDraft, + }) + + await flush() + expect(onRestoreDraft).toHaveBeenCalledTimes(1) + expect(onRestoreDraft).toHaveBeenCalledWith('recovered') + }) + + it('does not clobber a fresh edit made while the recovery read was still in flight', async () => { + fakeDraftStore.set('autosave-draft:file-6', { content: 'recovered', savedContent: 'a' }) + const onSave = vi.fn(async () => {}) + const onRestoreDraft = vi.fn() + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-6', + onRestoreDraft, + }) + + // The user types before the async IndexedDB read resolves. + handle.rerender({ content: 'user-typed' }) + await flush() + expect(onRestoreDraft).not.toHaveBeenCalled() + }) + + it('discards and purges a stale draft when the server baseline has moved on', async () => { + fakeDraftStore.set('autosave-draft:file-5', { + content: 'stale-edit', + savedContent: 'old-baseline', + }) + const onSave = vi.fn(async () => {}) + const onRestoreDraft = vi.fn() + renderAutosave({ + content: 'new-baseline', + savedContent: 'new-baseline', + onSave, + draftKey: 'file-5', + onRestoreDraft, + }) + + await flush() + expect(onRestoreDraft).not.toHaveBeenCalled() + // Purged, not merely skipped — otherwise a later open where the baseline coincidentally + // matches this stale snapshot again would silently resurrect the outdated edit. + expect(fakeDraftStore.has('autosave-draft:file-5')).toBe(false) + }) + + it('attempts recovery only once per mount, not every time draftKey toggles across a streaming lock', async () => { + fakeDraftStore.set('autosave-draft:file-once', { content: 'recovered', savedContent: 'a' }) + const onSave = vi.fn(async () => {}) + const onRestoreDraft = vi.fn() + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-once', + enabled: true, + onRestoreDraft, + }) + + await flush() + expect(onRestoreDraft).toHaveBeenCalledTimes(1) + + // Mirrors autosave being disabled during agent streaming (effectiveDraftKey -> undefined) + // and re-enabled once the stream settles (effectiveDraftKey -> defined again). A stale + // pre-stream draft left behind in the meantime must not be re-offered on the second pass. + fakeDraftStore.set('autosave-draft:file-once', { + content: 'pre-agent-edit', + savedContent: 'a', + }) + handle.rerender({ enabled: false }) + await flush() + handle.rerender({ enabled: true }) + await flush() + expect(onRestoreDraft).toHaveBeenCalledTimes(1) + }) + + it('discard clears the local draft immediately and blocks any further write, even mid-race with unmount', async () => { + const onSave = vi.fn(async () => {}) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard', + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(400) + }) + await flush() + expect(fakeDraftStore.has('autosave-draft:file-discard')).toBe(true) + + // Discard fires before the caller's content===savedContent reset has landed — the hook's + // own flag must block persistence regardless of that race, not just the IndexedDB delete. + act(() => handle.discard()) + await flush() + expect(fakeDraftStore.has('autosave-draft:file-discard')).toBe(false) + + // Simulate the unmount flush racing in right after discard, while still (from the hook's + // perspective) dirty: neither the local draft nor the network save must fire. + handle.unmount() + await flush() + expect(fakeDraftStore.has('autosave-draft:file-discard')).toBe(false) + expect(onSave).not.toHaveBeenCalled() + }) + + it('discard prevents a pending debounced network save from firing', async () => { + const onSave = vi.fn(async () => {}) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-2', + }) + + handle.rerender({ content: 'a1' }) + act(() => handle.discard()) + + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + expect(onSave).not.toHaveBeenCalled() + }) + + it('corrects the server with the captured baseline, even if the caller never resets content in time', async () => { + let resolveSave: (() => void) | undefined + const onSave = vi.fn( + () => + new Promise((resolve) => { + resolveSave = resolve + }) + ) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-3', + }) + + // A save is genuinely in flight (discardedRef can't stop it — it already started). + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(1) + + // The user discards while that save is still pending. Deliberately do NOT rerender content + // back to the baseline here — the correction must not depend on that caller-side reset + // landing before this continuation runs; it must push the baseline it captured at discard(). + act(() => handle.discard()) + + await act(async () => { + resolveSave?.() + }) + await flush() + // The stale in-flight write landed; a corrective save fires with the captured baseline ('a'), + // not whatever the still-dirty ambient content ('a1') happened to be. + expect(onSave).toHaveBeenCalledTimes(2) + expect(onSave).toHaveBeenNthCalledWith(2, 'a') + }) + + it('surfaces a corrective save failure instead of only logging it', async () => { + let resolveSave: (() => void) | undefined + const onSave = vi.fn( + () => + new Promise((resolve) => { + resolveSave = resolve + }) + ) + const onDiscardCorrectionFailed = vi.fn() + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-6', + onDiscardCorrectionFailed, + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + act(() => handle.discard()) + + // The corrective save (the second call) fails. + onSave.mockImplementationOnce(() => Promise.reject(new Error('offline'))) + await act(async () => { + resolveSave?.() + }) + await flush() + expect(onDiscardCorrectionFailed).toHaveBeenCalledTimes(1) + }) + + it('resumes normal autosave for a genuinely new edit made after discard', async () => { + const onSave = vi.fn(async () => {}) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-7', + }) + + handle.rerender({ content: 'a1' }) + act(() => handle.discard()) + // The caller resets content back to the baseline, mirroring discardChanges. + handle.rerender({ content: 'a' }) + + // The editor stays mounted a moment longer and the user starts a fresh edit. + handle.rerender({ content: 'b' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + // A brand-new edit made after discard must not be silently swallowed forever. + expect(onSave).toHaveBeenCalledTimes(1) + expect(onSave).toHaveBeenCalledWith() + }) + + it('does not issue a corrective save when discard finds nothing in flight', async () => { + const onSave = vi.fn(async () => {}) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-4', + }) + + handle.rerender({ content: 'a1' }) + act(() => handle.discard()) + await flush() + expect(onSave).not.toHaveBeenCalled() + }) + + it('does not treat a long-settled save as in flight when discard runs much later', async () => { + const onSave = vi.fn(async () => {}) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-5', + }) + + // A save fully completes well before discard is ever called. + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + handle.rerender({ savedContent: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(600) + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(1) + + // A later, unrelated edit is discarded. inFlightRef must have been cleared when the first + // save settled — otherwise this reads it as still "in flight" and fires a stale corrective + // save with whatever savedContent happened to be at that (now long-past) moment. + handle.rerender({ content: 'a12' }) + act(() => handle.discard()) + await flush() + expect(onSave).toHaveBeenCalledTimes(1) + }) + + it('does not let a stale discard correction clobber a genuinely new edit made afterward', async () => { + const resolvers: Array<() => void> = [] + const onSave = vi.fn( + () => + new Promise((resolve) => { + resolvers.push(resolve) + }) + ) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-8', + }) + + // A save is in flight when the user discards. + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(1) + act(() => handle.discard()) + + // The caller resets content to the baseline, then the user types something genuinely new + // before the in-flight save (and its scheduled correction) has settled. + handle.rerender({ content: 'a' }) + handle.rerender({ content: 'a2' }) + + // The in-flight save resolves; the correction runs but must defer, since content has + // moved on to something that's neither the discarded baseline nor what it was at discard. + await act(async () => { + resolvers[0]?.() + }) + await flush() + await act(async () => { + vi.advanceTimersByTime(600) + }) + await flush() + + // A fresh save for the new edit fires instead of a correction pushing the stale baseline. + const targets = onSave.mock.calls.map((call) => call[0]) + expect(targets).not.toContain('a') + expect(onSave.mock.calls.length).toBeGreaterThan(1) + + resolvers[resolvers.length - 1]?.() + await flush() + }) + + it('does not let the original save leftover status timer clobber a still-running correction', async () => { + const resolvers: Array<() => void> = [] + const onSave = vi.fn( + () => + new Promise((resolve) => { + resolvers.push(resolve) + }) + ) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-9', + }) + + // A save is in flight when the user discards. + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(1) + act(() => handle.discard()) + handle.rerender({ content: 'a' }) + + // The original save resolves; the correction starts (savingRef/inFlightRef now belong to it). + await act(async () => { + resolvers[0]?.() + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(2) + + // The original save's own MIN_SAVING_DISPLAY_MS timer fires while the correction is still + // unresolved. Without the inFlightRef guard, this used to unconditionally reset savingRef, + // letting a debounced save for a new edit start concurrently with the correction. + await act(async () => { + vi.advanceTimersByTime(600) + }) + await flush() + + // A new edit made in this window must not be able to schedule a concurrent save while the + // correction (still unresolved) rightfully holds the mutex. + handle.rerender({ content: 'a2' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(2) + + resolvers[1]?.() + await flush() + }) + + it('reaches a terminal status once the discard correction settles instead of sticking on saving', async () => { + const resolvers: Array<() => void> = [] + const onSave = vi.fn( + () => + new Promise((resolve) => { + resolvers.push(resolve) + }) + ) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-10', + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(1) + act(() => handle.discard()) + handle.rerender({ content: 'a' }) + + // The original save resolves; the correction starts. Its own MIN_SAVING_DISPLAY_MS timer + // must not resolve status to 'saved'/'idle' for content the user no longer sees. + await act(async () => { + resolvers[0]?.() + }) + await flush() + await act(async () => { + vi.advanceTimersByTime(600) + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(2) + expect(handle.status()).toBe('saving') + + // The correction itself settles — without this, status stayed on 'saving' forever. + await act(async () => { + resolvers[1]?.() + }) + await flush() + expect(handle.status()).toBe('idle') + }) + + it('surfaces an error if the discard correction itself fails, rather than drifting to idle', async () => { + const resolvers: Array<() => void> = [] + const rejecters: Array<(error: Error) => void> = [] + let callCount = 0 + const onSave = vi.fn( + () => + new Promise((resolve, reject) => { + callCount += 1 + if (callCount === 1) resolvers.push(resolve) + else rejecters.push(reject) + }) + ) + const onDiscardCorrectionFailed = vi.fn() + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-11', + onDiscardCorrectionFailed, + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + act(() => handle.discard()) + handle.rerender({ content: 'a' }) + + await act(async () => { + resolvers[0]?.() + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(2) + + await act(async () => { + rejecters[0]?.(new Error('network down')) + }) + await flush() + expect(onDiscardCorrectionFailed).toHaveBeenCalledTimes(1) + expect(handle.status()).toBe('error') + }) + + it('does not let discard suppression from a previous file block saves for a newly switched-to file', async () => { + const onSave = vi.fn(async () => {}) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-x', + }) + + handle.rerender({ content: 'a1' }) + act(() => handle.discard()) + handle.rerender({ content: 'a' }) + + // Switch to a different file whose content coincidentally equals the previous file's + // discard target — without keying discard suppression to draftKey, this would stay + // wrongly suppressed forever, since content would never differ from the stale target. + handle.rerender({ draftKey: 'file-y', savedContent: 'z', content: 'a' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(1) + }) + + it('autosaves an edit made while a discard correction was in flight, once the correction settles', async () => { + const resolvers: Array<() => void> = [] + const onSave = vi.fn( + () => + new Promise((resolve) => { + resolvers.push(resolve) + }) + ) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-12', + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(1) + act(() => handle.discard()) + handle.rerender({ content: 'a' }) + + // The original save resolves; the correction starts and holds the savingRef mutex. + await act(async () => { + resolvers[0]?.() + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(2) + + // The user keeps typing while the correction is still in flight. The debounce effect sees + // savingRef held and bails — without a re-chain once the mutex frees, this edit would never + // autosave until some unrelated future change happened to fire the effect again. + handle.rerender({ content: 'a2' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(2) + + await act(async () => { + resolvers[1]?.() + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(3) + expect(onSave).toHaveBeenLastCalledWith() + }) + + it('surfaces the newer edit’s own save outcome if a discard correction fails after being superseded', async () => { + const resolvers: Array<() => void> = [] + const rejecters: Array<(error: Error) => void> = [] + let callCount = 0 + const onSave = vi.fn(() => { + callCount += 1 + if (callCount === 1) return new Promise((resolve) => resolvers.push(resolve)) + if (callCount === 2) return new Promise((_, reject) => rejecters.push(reject)) + // The third call is the superseding edit's own save — never resolved, since we only + // need to observe that it was picked up at all. + return new Promise(() => {}) + }) + const onDiscardCorrectionFailed = vi.fn() + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-13', + onDiscardCorrectionFailed, + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + act(() => handle.discard()) + handle.rerender({ content: 'a' }) + + await act(async () => { + resolvers[0]?.() + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(2) + + // A newer edit supersedes the in-flight correction before it settles. + handle.rerender({ content: 'a2' }) + + await act(async () => { + rejecters[0]?.(new Error('network down')) + }) + await flush() + + // The failed correction still reports itself, but the superseding edit's own save must be + // picked up immediately instead of leaving the hook stalled on a dead mutex. + expect(onDiscardCorrectionFailed).toHaveBeenCalledTimes(1) + expect(onSave).toHaveBeenCalledTimes(3) + expect(onSave).toHaveBeenLastCalledWith() + }) + + it('does not lift discard suppression when only `enabled` toggles for the same document', async () => { + const resolvers: Array<() => void> = [] + const rejecters: Array<(error: Error) => void> = [] + const onSave = vi.fn( + () => + new Promise((resolve, reject) => { + resolvers.push(resolve) + rejecters.push(reject) + }) + ) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-14', + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(1) + act(() => handle.discard()) + handle.rerender({ content: 'a' }) + + // The correction starts once the original save settles. + await act(async () => { + resolvers[0]?.() + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(2) + + // A streaming lock (or any other reason `enabled` flips) toggles for the SAME document + // while the correction is still in flight — this must not be mistaken for switching files + // and clear discard's ownership of saveStatus. + handle.rerender({ enabled: false }) + handle.rerender({ enabled: true }) + + // Without keying off the raw draftKey, the `enabled` round-trip would have cleared + // discardedRef, so the correction's failure handler would skip setSaveStatus('error') and + // leave the hook stuck on 'saving' with no visible retry affordance. + await act(async () => { + rejecters[1]?.(new Error('network down')) + }) + await flush() + expect(handle.status()).toBe('error') + }) + + it('retries a failed discard correction via saveImmediately instead of silently no-opping', async () => { + const resolvers: Array<() => void> = [] + const rejecters: Array<(error: Error) => void> = [] + let callCount = 0 + const onSave = vi.fn(() => { + callCount += 1 + if (callCount === 1) return new Promise((resolve) => resolvers.push(resolve)) + if (callCount === 2) return new Promise((_, reject) => rejecters.push(reject)) + return Promise.resolve() + }) + const onDiscardCorrectionFailed = vi.fn() + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-discard-15', + onDiscardCorrectionFailed, + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(1500) + }) + await flush() + act(() => handle.discard()) + handle.rerender({ content: 'a' }) + + await act(async () => { + resolvers[0]?.() + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(2) + + await act(async () => { + rejecters[0]?.(new Error('network down')) + }) + await flush() + expect(onDiscardCorrectionFailed).toHaveBeenCalledTimes(1) + expect(handle.status()).toBe('error') + + // Content already equals savedContent (that's what discard reverted to), so a naive retry + // through save()'s dirty-check would be a silent no-op. saveImmediately must still push + // the reverted baseline again. + await act(async () => { + await handle.saveImmediately() + }) + await flush() + expect(onSave).toHaveBeenCalledTimes(3) + expect(onSave).toHaveBeenLastCalledWith('a') + expect(handle.status()).toBe('idle') + }) + + it('serializes IndexedDB writes and deletes so a slow write cannot resurrect a discarded draft', async () => { + let resolveWrite: (() => void) | undefined + const idbKeyval = await import('idb-keyval') + vi.mocked(idbKeyval.set).mockImplementationOnce( + (key: string, value: unknown) => + new Promise((resolve) => { + resolveWrite = () => { + fakeDraftStore.set(key, value) + resolve() + } + }) + ) + const onSave = vi.fn(async () => {}) + const { handle } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave, + draftKey: 'file-race', + }) + + handle.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(400) + }) + await flush() + // The write is now in flight (not yet resolved) when discard's delete is queued behind it. + act(() => handle.discard()) + await flush() + expect(fakeDraftStore.has('autosave-draft:file-race')).toBe(false) + + // The slow write finally resolves — it must not resurrect the entry the queued delete removed. + resolveWrite?.() + await flush() + expect(fakeDraftStore.has('autosave-draft:file-race')).toBe(false) + }) + + it('serializes IndexedDB ops across an unmount and a fresh remount of the same draft key', async () => { + let resolveWrite: (() => void) | undefined + const idbKeyval = await import('idb-keyval') + vi.mocked(idbKeyval.set).mockImplementationOnce( + (key: string, value: unknown) => + new Promise((resolve) => { + resolveWrite = () => { + fakeDraftStore.set(key, value) + resolve() + } + }) + ) + const onSaveA = vi.fn(async () => {}) + const { handle: handleA } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave: onSaveA, + draftKey: 'file-cross-mount', + }) + + handleA.rerender({ content: 'a1' }) + await act(async () => { + vi.advanceTimersByTime(400) + }) + await flush() + // The write is in flight when this instance unmounts — its own idbQueueRef dies with it, + // but the operation must still be ordered relative to whatever comes next for this key. + handleA.unmount() + await flush() + + // A fresh instance mounts for the same file and immediately discards. + const onSaveB = vi.fn(async () => {}) + const { handle: handleB } = renderAutosave({ + content: 'a', + savedContent: 'a', + onSave: onSaveB, + draftKey: 'file-cross-mount', + }) + act(() => handleB.discard()) + await flush() + + // The old instance's slow write finally resolves — it must not resurrect the entry the + // new instance's delete already removed, even though they belong to different mounts. + resolveWrite?.() + await flush() + expect(fakeDraftStore.has('autosave-draft:file-cross-mount')).toBe(false) + }) + }) }) diff --git a/apps/sim/hooks/use-autosave.ts b/apps/sim/hooks/use-autosave.ts index 824e2b415d0..aaf2d3d6b88 100644 --- a/apps/sim/hooks/use-autosave.ts +++ b/apps/sim/hooks/use-autosave.ts @@ -1,21 +1,63 @@ 'use client' import { useCallback, useEffect, useRef, useState } from 'react' +import { createLogger } from '@sim/logger' +import { del, get, set } from 'idb-keyval' + +const logger = createLogger('Autosave') export type SaveStatus = 'idle' | 'saving' | 'saved' | 'error' +interface LocalDraft { + content: string + savedContent: string +} + +const LOCAL_DRAFT_DELAY_MS = 400 +const MIN_SAVING_DISPLAY_MS = 600 + +function localDraftDbKey(draftKey: string) { + return `autosave-draft:${draftKey}` +} + +const draftOpQueues = new Map>() + +/** Serializes IndexedDB reads/writes for a given draft key across every `useAutosave` instance (including one that already unmounted), so a late write from a just-unmounted instance can't land after a newly-mounted instance's delete or recovery read. */ +function enqueueDraftOp(key: string, op: () => Promise): Promise { + const prev = draftOpQueues.get(key) ?? Promise.resolve() + const result = prev.then(op, op) + const settled = result.catch(() => {}) + draftOpQueues.set(key, settled) + void settled.then(() => { + if (draftOpQueues.get(key) === settled) draftOpQueues.delete(key) + }) + return result +} + interface UseAutosaveOptions { content: string savedContent: string - onSave: () => Promise + /** `overrideContent`, when passed, is what `discard()`'s corrective save pushes — the reverted baseline captured at discard time, not whatever the ambient content ref reads by the time the in-flight save it's correcting for has settled. */ + onSave: (overrideContent?: string) => Promise delay?: number enabled?: boolean + /** + * Uniquely identifies the document being edited (e.g. a file id). When set, the draft is + * mirrored into IndexedDB on a short debounce, independent of the network save, and recovered + * via `onRestoreDraft` on mount if newer than `savedContent`. + */ + draftKey?: string + onRestoreDraft?: (content: string) => void + /** Called if `discard()`'s corrective save fails — the only way that failure can surface, since it happens after the component may already have unmounted. */ + onDiscardCorrectionFailed?: () => void } interface UseAutosaveReturn { saveStatus: SaveStatus saveImmediately: () => Promise isDirty: boolean + /** Abandons the current draft: blocks any save/local-draft write not yet started, clears the local draft immediately, and corrects the server if a save already in flight lands afterward. */ + discard: () => void } /** @@ -29,29 +71,92 @@ export function useAutosave({ onSave, delay = 1500, enabled = true, + draftKey, + onRestoreDraft, + onDiscardCorrectionFailed, }: UseAutosaveOptions): UseAutosaveReturn { const [saveStatus, setSaveStatus] = useState('idle') + const timerRef = useRef>(undefined) const idleTimerRef = useRef>(undefined) const displayTimerRef = useRef>(undefined) const savingRef = useRef(false) + const savingStartRef = useRef(0) + const inFlightRef = useRef | null>(null) + const unmountedRef = useRef(false) const onSaveRef = useRef(onSave) onSaveRef.current = onSave const enabledRef = useRef(enabled) enabledRef.current = enabled + const savedContentRef = useRef(savedContent) savedContentRef.current = savedContent const contentRef = useRef(content) contentRef.current = content + const effectiveDraftKey = enabled ? draftKey : undefined + const draftKeyRef = useRef(effectiveDraftKey) + draftKeyRef.current = effectiveDraftKey + const onRestoreDraftRef = useRef(onRestoreDraft) + onRestoreDraftRef.current = onRestoreDraft + const onDiscardCorrectionFailedRef = useRef(onDiscardCorrectionFailed) + onDiscardCorrectionFailedRef.current = onDiscardCorrectionFailed + + const localDraftTimerRef = useRef>(undefined) + const lastPersistedContentRef = useRef(null) + + const discardedRef = useRef(false) + const discardTargetRef = useRef(null) + const failedCorrectionTargetRef = useRef(null) + // Keyed off the raw `draftKey`, not `effectiveDraftKey` — the latter also flips with `enabled` + // (e.g. a streaming lock toggling for the SAME document), which must not be mistaken for a + // hook instance being reused across documents (today's callers all remount per file instead). + const documentKeyRef = useRef(draftKey) + const documentChanged = documentKeyRef.current !== draftKey + documentKeyRef.current = draftKey + if (documentChanged) { + discardedRef.current = false + failedCorrectionTargetRef.current = null + } + const isDirty = content !== savedContent - const savingStartRef = useRef(0) - const inFlightRef = useRef | null>(null) - const unmountedRef = useRef(false) - const MIN_SAVING_DISPLAY_MS = 600 + if (discardedRef.current && content !== discardTargetRef.current) { + discardedRef.current = false + failedCorrectionTargetRef.current = null + } + + const persistLocalDraft = useCallback(() => { + const key = draftKeyRef.current + if (discardedRef.current || !key || contentRef.current === savedContentRef.current) return + if (contentRef.current === lastPersistedContentRef.current) return + const content = contentRef.current + const savedContentSnapshot = savedContentRef.current + void enqueueDraftOp(key, () => + set(localDraftDbKey(key), { + content, + savedContent: savedContentSnapshot, + } satisfies LocalDraft) + ) + .then(() => { + lastPersistedContentRef.current = content + }) + .catch((error) => { + logger.warn('IndexedDB draft write failed', { key, error }) + }) + }, []) + + const clearLocalDraft = useCallback(() => { + const key = draftKeyRef.current + lastPersistedContentRef.current = null + if (!key) return + void enqueueDraftOp(key, () => del(localDraftDbKey(key))).catch((error) => { + logger.warn('IndexedDB draft delete failed', { key, error }) + }) + }, []) const save = useCallback(async () => { if ( + discardedRef.current || !enabledRef.current || savingRef.current || contentRef.current === savedContentRef.current @@ -68,15 +173,22 @@ export function useAutosave({ } catch { nextStatus = 'error' } finally { + inFlightRef.current = null if (unmountedRef.current) { savingRef.current = false } else { const elapsed = Date.now() - savingStartRef.current const remaining = Math.max(0, MIN_SAVING_DISPLAY_MS - elapsed) displayTimerRef.current = setTimeout(() => { - setSaveStatus(nextStatus) - clearTimeout(idleTimerRef.current) - idleTimerRef.current = setTimeout(() => setSaveStatus('idle'), 2000) + // While discarded, status is owned by discard()'s corrective save instead — this + // save's outcome no longer reflects what the user is looking at, and letting the + // idle-timer fire anyway would prematurely clear a status the correction just set. + if (!discardedRef.current) { + setSaveStatus(nextStatus) + clearTimeout(idleTimerRef.current) + idleTimerRef.current = setTimeout(() => setSaveStatus('idle'), 2000) + } + if (inFlightRef.current) return savingRef.current = false if (nextStatus !== 'error' && contentRef.current !== savedContentRef.current) { save() @@ -106,23 +218,145 @@ export function useAutosave({ clearTimeout(timerRef.current) clearTimeout(idleTimerRef.current) clearTimeout(displayTimerRef.current) - if (!enabledRef.current || contentRef.current === savedContentRef.current) return + clearTimeout(localDraftTimerRef.current) + persistLocalDraft() + if ( + discardedRef.current || + !enabledRef.current || + contentRef.current === savedContentRef.current + ) { + return + } // Flush the latest content on unmount, but chain it AFTER any in-flight save rather than // firing a concurrent PUT: the in-flight save captured an older snapshot, so writing the // latest sequentially (last) prevents an out-of-order completion from clobbering it. void (async () => { await inFlightRef.current - if (contentRef.current !== savedContentRef.current) { - await onSaveRef.current().catch(() => {}) + if (!discardedRef.current) { + await onSaveRef.current().then(clearLocalDraft, () => {}) } })() } - }, []) + }, [clearLocalDraft, persistLocalDraft]) + + const wasDirtyRef = useRef(isDirty) + + useEffect(() => { + if (effectiveDraftKey && !isDirty && wasDirtyRef.current) clearLocalDraft() + wasDirtyRef.current = isDirty + }, [effectiveDraftKey, isDirty, clearLocalDraft]) + + useEffect(() => { + if (!effectiveDraftKey || !isDirty) return + clearTimeout(localDraftTimerRef.current) + localDraftTimerRef.current = setTimeout(persistLocalDraft, LOCAL_DRAFT_DELAY_MS) + return () => clearTimeout(localDraftTimerRef.current) + }, [content, effectiveDraftKey, isDirty, persistLocalDraft]) + + useEffect(() => { + if (!effectiveDraftKey) return + const handleVisibility = () => { + if (document.visibilityState === 'hidden') persistLocalDraft() + } + window.addEventListener('pagehide', persistLocalDraft) + document.addEventListener('visibilitychange', handleVisibility) + return () => { + window.removeEventListener('pagehide', persistLocalDraft) + document.removeEventListener('visibilitychange', handleVisibility) + } + }, [effectiveDraftKey, persistLocalDraft]) + + const recoveredForKeyRef = useRef(null) + + useEffect(() => { + if (!effectiveDraftKey || recoveredForKeyRef.current === effectiveDraftKey) return + recoveredForKeyRef.current = effectiveDraftKey + let cancelled = false + void enqueueDraftOp(effectiveDraftKey, () => + get(localDraftDbKey(effectiveDraftKey)) + ) + .then((draft) => { + if (cancelled || !draft) return + if (draft.savedContent !== savedContentRef.current) { + clearLocalDraft() + return + } + if (draft.content === draft.savedContent) return + if (contentRef.current !== savedContentRef.current) return + onRestoreDraftRef.current?.(draft.content) + }) + .catch((error) => { + logger.warn('IndexedDB draft read failed', { draftKey: effectiveDraftKey, error }) + }) + return () => { + cancelled = true + } + }, [effectiveDraftKey, clearLocalDraft]) + + /** Pushes `target` (the reverted baseline) to the server. Used both by `discard()`'s initial correction and by `saveImmediately`'s retry of a correction that previously failed — content already equals `target` in both cases, so this bypasses `save()`'s dirty-check entirely rather than special-casing it there. */ + const runCorrection = useCallback( + (target: string) => { + savingRef.current = true + const correctionRun = onSaveRef + .current(target) + .then( + () => { + failedCorrectionTargetRef.current = null + // Only ours to set if nothing has since un-suppressed discard (a newer edit) — that + // flow owns status once it takes over. + if (!unmountedRef.current && discardedRef.current) setSaveStatus('idle') + }, + (error) => { + failedCorrectionTargetRef.current = target + logger.warn('Corrective save after discard failed', { error }) + onDiscardCorrectionFailedRef.current?.() + if (!unmountedRef.current && discardedRef.current) setSaveStatus('error') + } + ) + .finally(() => { + savingRef.current = false + inFlightRef.current = null + // A newer edit made while the correction was in flight bailed out of the debounce + // effect (savingRef was held) and never got rescheduled — pick it up now that the + // mutex is free. This also gives that edit's own save cycle ownership of saveStatus, + // covering a correction that failed after a newer edit already un-suppressed discard. + if (!unmountedRef.current && contentRef.current !== savedContentRef.current) save() + }) + inFlightRef.current = correctionRun + return correctionRun + }, + [save] + ) const saveImmediately = useCallback(async () => { clearTimeout(timerRef.current) + // Retrying after a failed correction: content already equals savedContent (that's what + // discard reverted to), so save()'s own dirty-check would treat this as a no-op and the + // error's retry button would silently do nothing. + if (failedCorrectionTargetRef.current !== null) { + await runCorrection(failedCorrectionTargetRef.current) + return + } await save() - }, [save]) + }, [save, runCorrection]) + + const discard = useCallback(() => { + discardedRef.current = true + discardTargetRef.current = savedContentRef.current + failedCorrectionTargetRef.current = null + clearTimeout(timerRef.current) + clearTimeout(localDraftTimerRef.current) + clearLocalDraft() + const pendingSave = inFlightRef.current + if (!pendingSave) return + const target = discardTargetRef.current + const contentAtDiscard = contentRef.current + void pendingSave.then(() => { + const current = contentRef.current + if (inFlightRef.current || (current !== target && current !== contentAtDiscard)) return + runCorrection(target) + }) + }, [clearLocalDraft, runCorrection]) - return { saveStatus, saveImmediately, isDirty } + return { saveStatus, saveImmediately, isDirty, discard } } diff --git a/apps/sim/lib/api/contracts/block-visibility.ts b/apps/sim/lib/api/contracts/block-visibility.ts new file mode 100644 index 00000000000..505eec694c6 --- /dev/null +++ b/apps/sim/lib/api/contracts/block-visibility.ts @@ -0,0 +1,37 @@ +import { z } from 'zod' +import { defineRouteContract } from '@/lib/api/contracts' +import { workspaceIdSchema } from '@/lib/api/contracts/primitives' + +/** + * Per-viewer block visibility projection (see + * `@/lib/core/config/block-visibility`): which `preview: true` block types this + * viewer may see, which types are kill-switched, and which revealed types carry + * the " (Preview)" display tag. + */ + +const getBlockVisibilityQuerySchema = z.object({ + workspaceId: workspaceIdSchema, +}) + +export type GetBlockVisibilityQuery = z.input + +const blockVisibilityResponseSchema = z.object({ + /** Preview block types revealed to this viewer. */ + revealed: z.array(z.string()), + /** Block types kill-switched (hidden from discovery) for this viewer. */ + disabled: z.array(z.string()), + /** Revealed types not globally GA — displayed with a " (Preview)" suffix. */ + previewTagged: z.array(z.string()), +}) + +export type BlockVisibilityResponse = z.output + +export const getBlockVisibilityContract = defineRouteContract({ + method: 'GET', + path: '/api/blocks/visibility', + query: getBlockVisibilityQuerySchema, + response: { + mode: 'json', + schema: blockVisibilityResponseSchema, + }, +}) diff --git a/apps/sim/lib/billing/storage/index.ts b/apps/sim/lib/billing/storage/index.ts index 9203c24b455..8820af428ba 100644 --- a/apps/sim/lib/billing/storage/index.ts +++ b/apps/sim/lib/billing/storage/index.ts @@ -1,6 +1,8 @@ export { checkStorageQuota, getUserStorageLimit, getUserStorageUsage } from './limits' export { + checkAndIncrementStorageUsageInTx, decrementStorageUsage, decrementStorageUsageInTx, incrementStorageUsage, + maybeNotifyStorageLimit, } from './tracking' diff --git a/apps/sim/lib/billing/storage/tracking.ts b/apps/sim/lib/billing/storage/tracking.ts index 6311a881846..3c31bb8eb3b 100644 --- a/apps/sim/lib/billing/storage/tracking.ts +++ b/apps/sim/lib/billing/storage/tracking.ts @@ -7,11 +7,12 @@ import { db } from '@sim/db' import { organization, userStats } from '@sim/db/schema' import { createLogger } from '@sim/logger' -import { eq, sql } from 'drizzle-orm' +import { generateId } from '@sim/utils/id' +import { and, eq, sql } from 'drizzle-orm' import { maybeNotifyLimit } from '@/lib/billing/core/limit-notifications' import type { HighestPrioritySubscription } from '@/lib/billing/core/plan' import { getUserStorageLimit, getUserStorageUsage } from '@/lib/billing/storage/limits' -import { isOrgScopedSubscription } from '@/lib/billing/subscriptions/utils' +import { getFreeTierLimit, isOrgScopedSubscription } from '@/lib/billing/subscriptions/utils' import { isBillingEnabled } from '@/lib/core/config/env-flags' const logger = createLogger('StorageTracking') @@ -33,7 +34,7 @@ function formatGb(bytes: number, decimals: number): string { * @param rearmOnly - True on decrements, so a shrink that leaves usage above a * threshold re-arms but never sends (a drop is not a fresh crossing). */ -async function maybeNotifyStorageLimit( +export async function maybeNotifyStorageLimit( userId: string, workspaceId: string, sub: HighestPrioritySubscription | null, @@ -195,3 +196,94 @@ export async function decrementStorageUsageInTx( .where(eq(userStats.userId, userId)) } } + +/** + * Atomically check quota and increment a user's (or their org's) storage + * counter inside an existing transaction, using a pre-resolved subscription. + * The check and the increment are a single conditional `UPDATE`, so two + * concurrent callers can no longer both read the same pre-increment usage, + * both pass the check, and both commit past the limit — the second caller's + * `UPDATE` re-evaluates the WHERE clause against the first caller's already + * -committed-within-the-same-DB-round-trip row and correctly fails. Replaces + * the old read-then-decide-then-increment-after-commit split (`checkStorageQuota` + * + a fire-and-forget `incrementStorageUsage` after the transaction), which left + * a window between the read and the increment. + * + * On success, callers should best-effort call {@link maybeNotifyStorageLimit} + * after the transaction commits (mirrors the existing post-increment threshold + * check) — this helper doesn't do it itself since it runs mid-transaction. + * + * For a personal (non-org-scoped) `userId`, this first upserts the + * `userStats` row on `tx` — a documented possibility for OAuth account + * linking (see `ensureUserStatsExists` in `lib/billing/core/usage.ts`, whose + * insert values this mirrors) — because the conditional `UPDATE` below + * matches 0 rows, and therefore reads as "quota exceeded", if that row + * doesn't exist yet. `ensureUserStatsExists` itself isn't reused here since + * it writes through the standalone `db` client, which would open a second + * pooled connection while this transaction's is held. + */ +export async function checkAndIncrementStorageUsageInTx( + tx: StorageTransaction, + sub: HighestPrioritySubscription | null, + userId: string, + bytes: number +): Promise<{ allowed: boolean; currentUsage: number; limit: number; error?: string }> { + if (!isBillingEnabled) { + return { allowed: true, currentUsage: 0, limit: Number.MAX_SAFE_INTEGER } + } + + const limit = await getUserStorageLimit(userId, sub) + + if (bytes <= 0) { + return { allowed: true, currentUsage: await getUserStorageUsage(userId, sub), limit } + } + + const orgScoped = isOrgScopedSubscription(sub, userId) && sub + + if (!orgScoped) { + await tx + .insert(userStats) + .values({ + id: generateId(), + userId, + currentUsageLimit: getFreeTierLimit().toString(), + usageLimitUpdatedAt: new Date(), + }) + .onConflictDoNothing({ target: userStats.userId }) + } + + const [updated] = orgScoped + ? await tx + .update(organization) + .set({ storageUsedBytes: sql`${organization.storageUsedBytes} + ${bytes}` }) + .where( + and( + eq(organization.id, sub.referenceId), + sql`${organization.storageUsedBytes} + ${bytes} <= ${limit}` + ) + ) + .returning({ storageUsedBytes: organization.storageUsedBytes }) + : await tx + .update(userStats) + .set({ storageUsedBytes: sql`${userStats.storageUsedBytes} + ${bytes}` }) + .where( + and( + eq(userStats.userId, userId), + sql`${userStats.storageUsedBytes} + ${bytes} <= ${limit}` + ) + ) + .returning({ storageUsedBytes: userStats.storageUsedBytes }) + + if (updated) { + return { allowed: true, currentUsage: updated.storageUsedBytes - bytes, limit } + } + + const currentUsage = await getUserStorageUsage(userId, sub) + const newUsage = currentUsage + bytes + return { + allowed: false, + currentUsage, + limit, + error: `Storage limit exceeded. Used: ${(newUsage / (1024 * 1024 * 1024)).toFixed(2)}GB, Limit: ${(limit / (1024 * 1024 * 1024)).toFixed(0)}GB`, + } +} diff --git a/apps/sim/lib/copilot/block-visibility.ts b/apps/sim/lib/copilot/block-visibility.ts new file mode 100644 index 00000000000..90e3c04cea7 --- /dev/null +++ b/apps/sim/lib/copilot/block-visibility.ts @@ -0,0 +1,58 @@ +import { LRUCache } from 'lru-cache' +import { type BlockVisibilityState, getBlockVisibility } from '@/lib/core/config/block-visibility' +import { getWorkspaceWithOwner } from '@/lib/workspaces/permissions/utils' + +/** + * Copilot-side resolver for the viewer's block-visibility projection. + * + * A single mothership turn fans out into many Go→Sim tool callbacks; resolving + * visibility per callback would repeat the workspace→org lookup (and, for + * admin-gated rules, a replica read) N times. This memoizes the resolved state + * per (userId, workspaceId) for a short TTL matching the AppConfig cache + * cadence, so a turn costs at most one resolution. + */ +const VISIBILITY_CACHE_TTL_MS = 30_000 + +const visibilityCache = new LRUCache>({ + max: 1000, + ttl: VISIBILITY_CACHE_TTL_MS, +}) + +async function resolveVisibility( + userId: string, + workspaceId?: string +): Promise { + const orgId = workspaceId + ? (await getWorkspaceWithOwner(workspaceId, { includeArchived: true }))?.organizationId + : undefined + return getBlockVisibility({ userId, orgId }) +} + +/** The viewer's visibility state, memoized per (userId, workspaceId) for ~30s. */ +export function getBlockVisibilityForCopilot( + userId: string, + workspaceId?: string +): Promise { + const key = `${userId}:${workspaceId ?? ''}` + let promise = visibilityCache.get(key) + if (!promise) { + promise = resolveVisibility(userId, workspaceId).catch((error) => { + visibilityCache.delete(key) + throw error + }) + visibilityCache.set(key, promise) + } + return promise +} + +/** + * Stable signature of a visibility state, for keying caches whose contents + * depend on the gated projection (e.g. the integration tool-schema LRU). + */ +export function visibilitySignature(vis: BlockVisibilityState): string { + return JSON.stringify([ + [...vis.revealed].sort(), + [...vis.disabled].sort(), + [...vis.previewTagged].sort(), + ]) +} diff --git a/apps/sim/lib/copilot/chat/payload.test.ts b/apps/sim/lib/copilot/chat/payload.test.ts index 1afbfbbaac3..88d6abbf689 100644 --- a/apps/sim/lib/copilot/chat/payload.test.ts +++ b/apps/sim/lib/copilot/chat/payload.test.ts @@ -53,7 +53,17 @@ vi.mock('@/tools/utils', () => ({ stripVersionSuffix: vi.fn((toolId: string) => toolId), })) +vi.mock('@/lib/copilot/block-visibility', () => ({ + getBlockVisibilityForCopilot: vi.fn(async () => ({ + revealed: new Set(), + disabled: new Set(), + previewTagged: new Set(), + })), + visibilitySignature: vi.fn(() => 'vis:none'), +})) + vi.mock('@/lib/copilot/integration-tools', () => ({ + filterExposedIntegrationTools: vi.fn((tools: unknown[]) => tools), getExposedIntegrationTools: vi.fn(() => [ { toolId: 'gmail_send', diff --git a/apps/sim/lib/copilot/chat/payload.ts b/apps/sim/lib/copilot/chat/payload.ts index e718b33cce3..97943990f09 100644 --- a/apps/sim/lib/copilot/chat/payload.ts +++ b/apps/sim/lib/copilot/chat/payload.ts @@ -3,11 +3,16 @@ import { toError } from '@sim/utils/errors' import { LRUCache } from 'lru-cache' import { getHighestPrioritySubscription } from '@/lib/billing/core/subscription' import { isPaid } from '@/lib/billing/plan-helpers' +import { getBlockVisibilityForCopilot, visibilitySignature } from '@/lib/copilot/block-visibility' import type { VfsSnapshotV1 } from '@/lib/copilot/generated/vfs-snapshot-v1' -import { getExposedIntegrationTools } from '@/lib/copilot/integration-tools' +import { + filterExposedIntegrationTools, + getExposedIntegrationTools, +} from '@/lib/copilot/integration-tools' import { getToolEntry } from '@/lib/copilot/tool-executor/router' import { getCopilotToolDescription } from '@/lib/copilot/tools/descriptions' import { encodeVfsSegment } from '@/lib/copilot/vfs/path-utils' +import type { BlockVisibilityState } from '@/lib/core/config/block-visibility' import { isE2BDocEnabled, isHosted } from '@/lib/core/config/env-flags' import { buildUserSkillTool } from '@/lib/mothership/skills' import { trackChatUpload } from '@/lib/uploads/contexts/workspace/workspace-file-manager' @@ -73,9 +78,12 @@ const integrationToolSchemaCache = new LRUCache { const schemaSurface = options.schemaSurface ?? 'copilot' - const cacheKey = getIntegrationToolSchemaCacheKey(userId, workspaceId, schemaSurface) + const vis = await getBlockVisibilityForCopilot(userId, workspaceId) + const cacheKey = getIntegrationToolSchemaCacheKey( + userId, + workspaceId, + schemaSurface, + visibilitySignature(vis) + ) const cached = integrationToolSchemaCache.get(cacheKey) if (cached) { return cloneToolSchemas(await cached.promise) @@ -120,7 +134,8 @@ export async function buildIntegrationToolSchemas( userId, messageId, { schemaSurface }, - workspaceId + workspaceId, + vis ).catch((error) => { integrationToolSchemaCache.delete(cacheKey) throw error @@ -137,7 +152,8 @@ async function buildIntegrationToolSchemasUncached( userId: string, messageId: string | undefined, options: Required, - workspaceId?: string + workspaceId?: string, + vis: BlockVisibilityState | null = null ): Promise { const reqLogger = logger.withMetadata({ messageId }) const integrationTools: ToolSchema[] = [] @@ -186,7 +202,8 @@ async function buildIntegrationToolSchemasUncached( } } - for (const { toolId, config: toolConfig, service } of getExposedIntegrationTools()) { + const exposedTools = filterExposedIntegrationTools(getExposedIntegrationTools(), vis) + for (const { toolId, config: toolConfig, service } of exposedTools) { try { if (allowedIntegrations && toolIdToBlockType) { const owningBlock = toolIdToBlockType.get(stripVersionSuffix(toolId)) diff --git a/apps/sim/lib/copilot/integration-tools.ts b/apps/sim/lib/copilot/integration-tools.ts index c1c93cc536e..77559d78784 100644 --- a/apps/sim/lib/copilot/integration-tools.ts +++ b/apps/sim/lib/copilot/integration-tools.ts @@ -1,4 +1,6 @@ -import { getAllBlocks } from '@/blocks/registry' +import type { BlockVisibilityState } from '@/lib/core/config/block-visibility' +import { BLOCK_REGISTRY } from '@/blocks/registry-maps' +import { isHiddenUnder } from '@/blocks/visibility/context' import { tools as toolRegistry } from '@/tools/registry' import type { ToolConfig } from '@/tools/types' import { getLatestVersionTools, stripVersionSuffix } from '@/tools/utils' @@ -15,14 +17,24 @@ export interface ExposedIntegrationTool { service: string /** Operation stem within the service (used for the VFS path filename), e.g. "read". */ operation: string + /** Owning block's registry type — the key block-visibility rules gate on. */ + blockType: string + /** Owning block's static `preview` marker, for the per-viewer filter. */ + preview?: boolean } let cached: ExposedIntegrationTool[] | null = null /** - * Returns the canonical set of integration tools exposed to the copilot agent: - * the latest version of each operation owned by a visible (non-hideFromToolbar) - * block. + * Returns the UNGATED universe of integration tools exposable to the copilot + * agent: the latest version of each operation owned by a non-`hideFromToolbar` + * block — INCLUDING unreleased `preview` blocks. + * + * Deliberately sourced from the raw `BLOCK_REGISTRY` (never the visibility- + * projected `getAllBlocks`) so this process-global memo is deterministic and + * can never be poisoned by whichever viewer's gated projection ran first. + * Every per-viewer consumer MUST apply {@link filterExposedIntegrationTools} + * before exposing the set. * * This is the single source of truth shared by VFS discovery * (components/integrations/**) and the deferred callable-tool payload, so the @@ -33,15 +45,16 @@ export function getExposedIntegrationTools(): ExposedIntegrationTool[] { if (cached) return cached // Map the tool ids each visible block exposes (both the raw id and its - // version-stripped base name) to that block's service directory. - const toolToService = new Map() - for (const block of getAllBlocks()) { + // version-stripped base name) to that block's service directory + type. + const toolToBlock = new Map() + for (const block of Object.values(BLOCK_REGISTRY)) { if (block.hideFromToolbar) continue if (!block.tools?.access) continue const service = stripVersionSuffix(block.type) + const owner = { service, blockType: block.type, preview: block.preview } for (const toolId of block.tools.access) { - toolToService.set(toolId, service) - toolToService.set(stripVersionSuffix(toolId), service) + toolToBlock.set(toolId, owner) + toolToBlock.set(stripVersionSuffix(toolId), owner) } } @@ -49,19 +62,41 @@ export function getExposedIntegrationTools(): ExposedIntegrationTool[] { const seen = new Set() for (const [toolId, config] of Object.entries(getLatestVersionTools(toolRegistry))) { const baseName = stripVersionSuffix(toolId) - const service = toolToService.get(toolId) ?? toolToService.get(baseName) - if (!service) continue + const owner = toolToBlock.get(toolId) ?? toolToBlock.get(baseName) + if (!owner) continue if (seen.has(baseName)) continue seen.add(baseName) - const prefix = `${service}_` + const prefix = `${owner.service}_` const operation = baseName.startsWith(prefix) ? baseName.slice(prefix.length) : baseName - exposed.push({ toolId, config, service, operation }) + exposed.push({ + toolId, + config, + service: owner.service, + operation, + blockType: owner.blockType, + preview: owner.preview, + }) } cached = exposed return exposed } +/** + * Per-viewer projection of the exposed set: drops tools whose owning block is + * hidden under `vis` (unrevealed preview blocks — including with a null state — + * and kill-switched types). Apply at every surface that hands the set to a + * viewer: VFS stamping, the deferred tool payload, `list_integration_tools`. + */ +export function filterExposedIntegrationTools( + tools: ExposedIntegrationTool[], + vis: BlockVisibilityState | null +): ExposedIntegrationTool[] { + return tools.filter( + (tool) => !isHiddenUnder(vis, { type: tool.blockType, preview: tool.preview }) + ) +} + /** Test-only: clears the memoized set so registry changes are picked up. */ export function resetExposedIntegrationToolsCache(): void { cached = null diff --git a/apps/sim/lib/copilot/tools/handlers/integration-tools.ts b/apps/sim/lib/copilot/tools/handlers/integration-tools.ts index a3e307bac8a..465da57bc33 100644 --- a/apps/sim/lib/copilot/tools/handlers/integration-tools.ts +++ b/apps/sim/lib/copilot/tools/handlers/integration-tools.ts @@ -1,17 +1,24 @@ -import { getExposedIntegrationTools } from '@/lib/copilot/integration-tools' +import { getBlockVisibilityForCopilot } from '@/lib/copilot/block-visibility' +import { + filterExposedIntegrationTools, + getExposedIntegrationTools, +} from '@/lib/copilot/integration-tools' import type { ExecutionContext, ToolCallResult } from '@/lib/copilot/request/types' import { stripVersionSuffix } from '@/tools/utils' export async function executeListIntegrationTools( params: Record, - _context: ExecutionContext + context: ExecutionContext ): Promise { const raw = typeof params.integration === 'string' ? params.integration.trim() : '' if (!raw) { return { success: false, error: "Missing required parameter 'integration'" } } - const all = getExposedIntegrationTools() + // The exposed set is the ungated universe — project it for this viewer so + // gated (preview / kill-switched) integrations stay undiscoverable. + const vis = await getBlockVisibilityForCopilot(context.userId, context.workspaceId) + const all = filterExposedIntegrationTools(getExposedIntegrationTools(), vis) const service = stripVersionSuffix(raw.toLowerCase()) const matches = all.filter((tool) => tool.service === service) diff --git a/apps/sim/lib/copilot/tools/handlers/vfs.ts b/apps/sim/lib/copilot/tools/handlers/vfs.ts index ca1902692e1..fd4a515a8af 100644 --- a/apps/sim/lib/copilot/tools/handlers/vfs.ts +++ b/apps/sim/lib/copilot/tools/handlers/vfs.ts @@ -1,15 +1,28 @@ import { createLogger } from '@sim/logger' import { getErrorMessage, toError } from '@sim/utils/errors' +import { getBlockVisibilityForCopilot } from '@/lib/copilot/block-visibility' import { TOOL_RESULT_MAX_INLINE_CHARS } from '@/lib/copilot/constants' import type { ExecutionContext, ToolCallResult } from '@/lib/copilot/request/types' import { getOrMaterializeVFS } from '@/lib/copilot/vfs' import type { GrepCountEntry, GrepMatch } from '@/lib/copilot/vfs/operations' import { WorkspaceFileGrepError } from '@/lib/copilot/vfs/operations' import { encodeVfsSegment } from '@/lib/copilot/vfs/path-utils' +import { withBlockVisibility } from '@/blocks/visibility/server-context' import { grepChatUpload, listChatUploads, readChatUpload } from './upload-file-reader' const logger = createLogger('VfsTools') +/** + * Materialize the workspace VFS inside the viewer's block-visibility context so + * the static component files stamped into it exclude blocks gated for this + * viewer (unrevealed previews, kill-switched types). Visibility is memoized per + * (userId, workspaceId), so repeated tool calls in one turn resolve once. + */ +async function getGatedVFS(workspaceId: string, userId: string) { + const vis = await getBlockVisibilityForCopilot(userId, workspaceId) + return withBlockVisibility(vis, () => getOrMaterializeVFS(workspaceId, userId)) +} + /** * Encode a chat-upload display name as a single canonical VFS path segment so * `uploads/` paths follow the same percent-encoded convention as `files/`. @@ -119,7 +132,7 @@ export async function executeVfsGrep( } result = await grepChatUpload(filename, context.chatId, pattern, grepOptions) } else { - const vfs = await getOrMaterializeVFS(workspaceId, context.userId) + const vfs = await getGatedVFS(workspaceId, context.userId) result = isWorkspaceFileGrepPath(rawPath) ? await vfs.grepFile(rawPath, pattern, grepOptions) : await vfs.grep(pattern, rawPath, grepOptions) @@ -176,7 +189,7 @@ export async function executeVfsGlob( } try { - const vfs = await getOrMaterializeVFS(workspaceId, context.userId) + const vfs = await getGatedVFS(workspaceId, context.userId) let files = vfs.glob(pattern) if (context.chatId && (pattern === 'uploads/*' || pattern.startsWith('uploads/'))) { @@ -281,7 +294,7 @@ export async function executeVfsRead( } } - const vfs = await getOrMaterializeVFS(workspaceId, context.userId) + const vfs = await getGatedVFS(workspaceId, context.userId) // Plain canonical file leaves are metadata resources. Dynamic file content // and inspection paths use explicit suffixes like /content, /style, diff --git a/apps/sim/lib/copilot/tools/server/blocks/get-blocks-metadata-tool.ts b/apps/sim/lib/copilot/tools/server/blocks/get-blocks-metadata-tool.ts index 6dd2579711a..7828f89ec98 100644 --- a/apps/sim/lib/copilot/tools/server/blocks/get-blocks-metadata-tool.ts +++ b/apps/sim/lib/copilot/tools/server/blocks/get-blocks-metadata-tool.ts @@ -10,6 +10,7 @@ import { getServiceAccountProviderForProviderId } from '@/lib/oauth/utils' import { isCustomBlockType } from '@/blocks/custom/build-config' import { getBlock } from '@/blocks/registry' import { AuthMode, type BlockConfig, isHiddenFromDisplay } from '@/blocks/types' +import { isHiddenUnder, overlayVisibility } from '@/blocks/visibility/context' import { getUserPermissionConfig } from '@/ee/access-control/utils/permission-check' import { PROVIDER_DEFINITIONS } from '@/providers/models' import { tools as toolsRegistry } from '@/tools/registry' @@ -165,6 +166,15 @@ export const getBlocksMetadataServerTool: BaseServerTool< continue } + // getBlock is pure, so the viewer's visibility must be checked + // explicitly: unrevealed preview blocks and kill-switched types stay + // out of the agent's metadata (the router wraps this tool in + // withBlockVisibility). + if (isHiddenUnder(overlayVisibility(), blockConfig)) { + logger.debug('Skipping block gated by visibility', { blockId }) + continue + } + if (isCustomBlockType(blockId)) { // Custom (deploy-as-block) blocks run a bound workflow via an internal // `workflow_executor`; the agent never configures a workflowId/inputMapping. diff --git a/apps/sim/lib/copilot/tools/server/router.ts b/apps/sim/lib/copilot/tools/server/router.ts index 6754920bca4..9dd8c737dde 100644 --- a/apps/sim/lib/copilot/tools/server/router.ts +++ b/apps/sim/lib/copilot/tools/server/router.ts @@ -1,5 +1,6 @@ import { createLogger } from '@sim/logger' import { z } from 'zod' +import { getBlockVisibilityForCopilot } from '@/lib/copilot/block-visibility' import { CreateFile, CreateFileFolder, @@ -60,6 +61,7 @@ import { editWorkflowServerTool } from '@/lib/copilot/tools/server/workflow/edit import { queryLogsServerTool } from '@/lib/copilot/tools/server/workflow/query-logs' import { listCustomBlocksWithInputsForWorkspace } from '@/lib/workflows/custom-blocks/operations' import { withCustomBlockOverlay } from '@/blocks/custom/server-overlay' +import { withBlockVisibility } from '@/blocks/visibility/server-context' export type ExecuteResponseSuccess = z.output @@ -76,6 +78,16 @@ const logger = createLogger('ServerToolRouter') */ const CUSTOM_BLOCK_OVERLAY_TOOLS = new Set(['edit_workflow', 'get_blocks_metadata']) +/** + * DISCOVERY tools that must run inside the viewer's block-visibility context so + * gated (preview / kill-switched) blocks disappear from what the agent can + * list. Deliberately a DIFFERENT set from {@link CUSTOM_BLOCK_OVERLAY_TOOLS}: + * `edit_workflow` is excluded because its registry use is functional + * (find-by-type over clones, never a discovery listing) and gating it would + * only risk leaking display projections into persisted state. + */ +const VISIBILITY_GATED_TOOLS = new Set(['get_blocks_metadata', 'get_trigger_blocks']) + const WRITE_ACTIONS: Record = { [KnowledgeBase.id]: [ 'create', @@ -243,15 +255,22 @@ export async function routeExecution( // Execute. The registry-dependent tools resolve blocks via getBlock/getAllBlocks; // wrap them in the custom-block overlay for the workspace's org so `custom_block_*` // types resolve (metadata lookup + edit-workflow validation) instead of being - // rejected as unknown. Other tools skip the extra query. - const runTool = () => tool.execute(args, context) - const result = - CUSTOM_BLOCK_OVERLAY_TOOLS.has(toolName) && context?.workspaceId - ? await withCustomBlockOverlay( - await listCustomBlocksWithInputsForWorkspace(context.workspaceId), - runTool - ) - : await runTool() + // rejected as unknown, and wrap discovery tools in the viewer's block-visibility + // context so gated blocks stay hidden. The two ALS scopes are independent and + // nest in either order. Other tools skip the extra queries. + let run = () => tool.execute(args, context) + if (VISIBILITY_GATED_TOOLS.has(toolName) && context?.userId) { + // Memoized per (userId, workspaceId) ~30s — a multi-tool turn resolves once. + const vis = await getBlockVisibilityForCopilot(context.userId, context.workspaceId) + const inner = run + run = () => withBlockVisibility(vis, inner) + } + if (CUSTOM_BLOCK_OVERLAY_TOOLS.has(toolName) && context?.workspaceId) { + const rows = await listCustomBlocksWithInputsForWorkspace(context.workspaceId) + const inner = run + run = () => withCustomBlockOverlay(rows, inner) + } + const result = await run() // Validate output if tool declares a schema; otherwise fall back to the // generated JSON schema contract emitted from Go. diff --git a/apps/sim/lib/copilot/vfs/workspace-vfs.ts b/apps/sim/lib/copilot/vfs/workspace-vfs.ts index abfc3ad9508..f40def73f7e 100644 --- a/apps/sim/lib/copilot/vfs/workspace-vfs.ts +++ b/apps/sim/lib/copilot/vfs/workspace-vfs.ts @@ -88,6 +88,7 @@ import { workspacePlanBackingPath, workspacePlansBackingFolderPath, } from '@/lib/copilot/vfs/workflow-aliases' +import type { BlockVisibilityState } from '@/lib/core/config/block-visibility' import { isE2BDocEnabled } from '@/lib/core/config/env-flags' import { isFeatureEnabled } from '@/lib/core/config/feature-flags' import { @@ -124,8 +125,9 @@ import { } from '@/lib/workspaces/permissions/utils' import { computeNeedsRedeployment } from '@/app/api/workflows/utils' import { buildCustomBlockConfig, isCustomBlockType } from '@/blocks/custom/build-config' -import { getAllBlocks } from '@/blocks/registry' -import type { BlockIcon } from '@/blocks/types' +import { BLOCK_REGISTRY } from '@/blocks/registry-maps' +import type { BlockConfig, BlockIcon } from '@/blocks/types' +import { isHiddenUnder, overlayVisibility } from '@/blocks/visibility/context' import { CONNECTOR_REGISTRY } from '@/connectors/registry.server' import type { WorkflowState } from '@/stores/workflows/workflow/types' import { TRIGGER_REGISTRY } from '@/triggers/registry' @@ -137,9 +139,39 @@ const logger = createLogger('WorkspaceVFS') const PLACEHOLDER_BLOCK_ICON = (() => null) as unknown as BlockIcon const MAX_COMPILED_ATTACHMENT_BYTES = 5 * 1024 * 1024 -/** Static component files, computed once and shared across all VFS instances */ +/** + * Static component files, computed once and shared across all VFS instances. + * Built from the UNGATED registry universe (preview blocks included) so this + * process-global cache can never be poisoned by one viewer's gated projection; + * per-viewer gating is applied when the map is stamped into each fresh VFS + * (see {@link isStaticFileHidden}). + */ let staticComponentFiles: Map | null = null +/** + * Owning block for each `components/integrations/**` file, recorded at build + * time. Block/trigger schema files carry their owning type as the path + * basename, but integration paths use the version-stripped service name — so + * their owners need this lookup for the stamp-time visibility filter. + */ +const integrationPathOwners = new Map>() + +/** + * Per-request visibility filter for the shared static files: hides files whose + * owning block is gated for this viewer (unrevealed preview blocks — the + * default with no context — and kill-switched types). Non-registry paths + * (loop/parallel, connectors, overviews) are always visible. + */ +function isStaticFileHidden(path: string, vis: BlockVisibilityState | null): boolean { + const blockMatch = path.match(/^components\/(?:blocks|triggers\/sim)\/([^/]+)\.json$/) + if (blockMatch) { + const config = BLOCK_REGISTRY[blockMatch[1]!] + return config ? isHiddenUnder(vis, config) : false + } + const owner = integrationPathOwners.get(path) + return owner ? isHiddenUnder(vis, owner) : false +} + // On-the-fly doc reads (render/extract) download the binary into the Sim process // and base64-stage it to E2B, so bound the input like the compile path's staging // caps — otherwise an authenticated member could OOM the worker with a multi-GB @@ -170,7 +202,12 @@ function getStaticComponentFiles(): Map { const files = new Map() - const allBlocks = getAllBlocks() + // Raw registry, never the visibility-projected getAllBlocks: this map is a + // process-global shared cache, so it must hold the deterministic ungated + // universe. Preview blocks get schema files here (path-filterable at stamp + // time for revealed viewers) but are EXCLUDED from the shared aggregate + // files (overviews, oauth/api-key summaries) that all viewers receive. + const allBlocks = Object.values(BLOCK_REGISTRY) const visibleBlocks = allBlocks.filter((b) => !b.hideFromToolbar) let blocksFiltered = 0 @@ -188,11 +225,17 @@ function getStaticComponentFiles(): Map { // Integration tools come from the shared exposed-tool set (latest version of // each operation owned by a visible block), the same set used to build the // deferred callable tools — so discovery and execution can never drift. - for (const { config: tool, service, operation } of getExposedIntegrationTools()) { + for (const exposedTool of getExposedIntegrationTools()) { + const { config: tool, service, operation, blockType, preview } = exposedTool const path = `components/integrations/${service}/${operation}.json` files.set(path, serializeIntegrationSchema(tool)) + integrationPathOwners.set(path, { type: blockType, preview }) integrationCount++ + // Preview-owned tools stay out of the shared oauth/api-key aggregates — + // those files are identical for every viewer. + if (preview) continue + if (tool.oauth?.required) { const existing = oauthServices.get(service) if (existing) { @@ -320,12 +363,16 @@ function getStaticComponentFiles(): Map { files.set( 'components/triggers/triggers.md', serializeTriggerOverview( - builtinTriggerBlocks.map((b) => ({ - id: b.type, - name: b.name, - provider: 'sim', - description: b.description, - })), + // The overview is a shared file — preview trigger blocks stay out of it + // (their per-type schema file remains discoverable for revealed viewers). + builtinTriggerBlocks + .filter((b) => !b.preview) + .map((b) => ({ + id: b.type, + name: b.name, + provider: 'sim', + description: b.description, + })), Object.entries(TRIGGER_REGISTRY).map(([id, t]) => ({ id, name: t.name, @@ -642,7 +689,11 @@ export class WorkspaceVFS { await timed('recently_deleted', this.materializeRecentlyDeleted(workspaceId, userId)) + // Per-viewer gating happens HERE, not in the shared builder: files + // owned by blocks hidden for this viewer are skipped at stamp time. + const blockVisibility = overlayVisibility() for (const [path, content] of getStaticComponentFiles()) { + if (isStaticFileHidden(path, blockVisibility)) continue this.files.set(path, content) } diff --git a/apps/sim/lib/core/config/appconfig-rules.test.ts b/apps/sim/lib/core/config/appconfig-rules.test.ts new file mode 100644 index 00000000000..ab79359000d --- /dev/null +++ b/apps/sim/lib/core/config/appconfig-rules.test.ts @@ -0,0 +1,69 @@ +/** + * @vitest-environment node + */ +import { describe, expect, it } from 'vitest' +import { matchesRule, normalizeRule, parseGateConfig } from '@/lib/core/config/appconfig-rules' + +describe('normalizeRule', () => { + it('returns null for non-object values', () => { + expect(normalizeRule('nope')).toBeNull() + expect(normalizeRule(null)).toBeNull() + expect(normalizeRule(42)).toBeNull() + }) + + it('keeps only boolean enabled/adminEnabled', () => { + expect(normalizeRule({ enabled: 'true', adminEnabled: 1 })).toEqual({}) + expect(normalizeRule({ enabled: true, adminEnabled: false })).toEqual({ + enabled: true, + adminEnabled: false, + }) + }) + + it('trims, dedupes, and drops empty ids', () => { + expect(normalizeRule({ orgIds: ['Org_1', ' org_1 ', '', 'org_2'], userIds: 'nope' })).toEqual({ + orgIds: ['Org_1', 'org_1', 'org_2'], + }) + }) +}) + +describe('parseGateConfig', () => { + it('drops malformed entries and coerces the rest', () => { + const rules = parseGateConfig({ + a: { enabled: true }, + b: 'not-an-object', + c: { userIds: ['u1'] }, + }) + expect(rules.a).toEqual({ enabled: true }) + expect(rules.b).toBeUndefined() + expect(rules.c).toEqual({ userIds: ['u1'] }) + }) + + it('degrades to an empty map on a malformed document', () => { + expect(parseGateConfig('not-an-object')).toEqual({}) + expect(parseGateConfig(null)).toEqual({}) + }) +}) + +describe('matchesRule', () => { + it('returns false for a missing rule', () => { + expect(matchesRule(undefined, { userId: 'u1' }, true)).toBe(false) + }) + + it('matches the global enabled clause', () => { + expect(matchesRule({ enabled: true }, {}, false)).toBe(true) + expect(matchesRule({ enabled: false }, {}, false)).toBe(false) + }) + + it('matches the userId and orgId allowlists', () => { + expect(matchesRule({ userIds: ['u1'] }, { userId: 'u1' }, false)).toBe(true) + expect(matchesRule({ userIds: ['u1'] }, { userId: 'u2' }, false)).toBe(false) + expect(matchesRule({ orgIds: ['o1'] }, { orgId: 'o1' }, false)).toBe(true) + expect(matchesRule({ orgIds: ['o1'] }, {}, false)).toBe(false) + }) + + it('matches the admin clause only with the supplied isAdmin', () => { + expect(matchesRule({ adminEnabled: true }, { userId: 'u1' }, true)).toBe(true) + expect(matchesRule({ adminEnabled: true }, { userId: 'u1' }, false)).toBe(false) + expect(matchesRule({ enabled: false }, { userId: 'u1' }, true)).toBe(false) + }) +}) diff --git a/apps/sim/lib/core/config/appconfig-rules.ts b/apps/sim/lib/core/config/appconfig-rules.ts new file mode 100644 index 00000000000..34ceec5173a --- /dev/null +++ b/apps/sim/lib/core/config/appconfig-rules.ts @@ -0,0 +1,80 @@ +/** + * Shared parsing and clause evaluation for AppConfig gating documents + * (`feature-flags`, `block-visibility`). Both documents are maps of key → + * gate rule with identical rule shapes; this module is the single copy of the + * security-sensitive normalization that prevents a malformed document from + * granting access. Admin-resolution *scheduling* deliberately stays with the + * callers (feature-flags resolves lazily per rule; block-visibility resolves + * once per document), so {@link matchesRule} takes an explicit `isAdmin`. + */ + +/** + * A single gating rule. A gate is open for a context when ANY clause matches: + * the global `enabled` default, the org/user allowlists, or `adminEnabled` for + * platform admins. An absent clause never matches. + */ +export interface AppConfigGateRule { + enabled?: boolean + orgIds?: string[] + userIds?: string[] + adminEnabled?: boolean +} + +/** + * Per-request evaluation context. Pass only the ids you have — a missing id + * skips its clause. `isAdmin` is a fast-path override for callers that already + * resolved platform-admin status. + */ +export interface AppConfigGateContext { + userId?: string | null + orgId?: string | null + isAdmin?: boolean +} + +function normalizeIds(values: unknown): string[] | undefined { + if (!Array.isArray(values)) return undefined + const ids = Array.from(new Set(values.map((v) => String(v).trim()).filter(Boolean))) + return ids.length > 0 ? ids : undefined +} + +/** Coerce a single arbitrary JSON value into a rule, or `null` when malformed. */ +export function normalizeRule(value: unknown): AppConfigGateRule | null { + if (!value || typeof value !== 'object') return null + const obj = value as Record + const rule: AppConfigGateRule = {} + if (typeof obj.enabled === 'boolean') rule.enabled = obj.enabled + if (typeof obj.adminEnabled === 'boolean') rule.adminEnabled = obj.adminEnabled + const orgIds = normalizeIds(obj.orgIds) + if (orgIds) rule.orgIds = orgIds + const userIds = normalizeIds(obj.userIds) + if (userIds) rule.userIds = userIds + return rule +} + +/** Coerce an arbitrary AppConfig/JSON document into a rule map, dropping malformed entries. */ +export function parseGateConfig(json: unknown): Record { + const obj = (json && typeof json === 'object' ? json : {}) as Record + const rules: Record = {} + for (const [key, value] of Object.entries(obj)) { + const rule = normalizeRule(value) + if (rule) rules[key] = rule + } + return rules +} + +/** + * Pure OR-of-clauses check. The caller supplies `isAdmin` — pass `false` to + * evaluate only the non-admin clauses (for lazy admin resolution). + */ +export function matchesRule( + rule: AppConfigGateRule | undefined, + ctx: AppConfigGateContext, + isAdmin: boolean +): boolean { + if (!rule) return false + if (rule.enabled) return true + if (ctx.userId && rule.userIds?.includes(ctx.userId)) return true + if (ctx.orgId && rule.orgIds?.includes(ctx.orgId)) return true + if (rule.adminEnabled && isAdmin) return true + return false +} diff --git a/apps/sim/lib/core/config/block-visibility.test.ts b/apps/sim/lib/core/config/block-visibility.test.ts new file mode 100644 index 00000000000..ef7e44c1006 --- /dev/null +++ b/apps/sim/lib/core/config/block-visibility.test.ts @@ -0,0 +1,157 @@ +/** + * @vitest-environment node + */ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const { mockFetch, mockIsPlatformAdmin, envRef, flagRef } = vi.hoisted(() => ({ + mockFetch: vi.fn(), + mockIsPlatformAdmin: vi.fn(), + envRef: { + APPCONFIG_APPLICATION: 'sim-staging' as string | undefined, + APPCONFIG_ENVIRONMENT: 'staging' as string | undefined, + }, + flagRef: { isAppConfigEnabled: false, previewBlocks: [] as string[] }, +})) + +vi.mock('@/lib/core/config/appconfig', () => ({ + fetchAppConfigProfile: mockFetch, +})) + +vi.mock('@/lib/core/config/env', () => ({ + get env() { + return envRef + }, +})) + +vi.mock('@/lib/core/config/env-flags', () => ({ + get isAppConfigEnabled() { + return flagRef.isAppConfigEnabled + }, + getPreviewBlocksFromEnv: () => flagRef.previewBlocks, +})) + +vi.mock('@/lib/permissions/super-user', () => ({ + isPlatformAdmin: mockIsPlatformAdmin, +})) + +import { getBlockVisibility } from '@/lib/core/config/block-visibility' + +/** Make `getBlockVisibility` resolve `doc` via the AppConfig path (also exercises parsing). */ +function withAppConfig(doc: unknown) { + flagRef.isAppConfigEnabled = true + mockFetch.mockImplementation((_ids, parse) => Promise.resolve(parse(doc))) +} + +describe('getBlockVisibility', () => { + beforeEach(() => { + vi.clearAllMocks() + flagRef.isAppConfigEnabled = false + flagRef.previewBlocks = [] + }) + + describe('off-AppConfig (env fallback)', () => { + it('reveals and preview-tags the PREVIEW_BLOCKS types without fetching', async () => { + flagRef.previewBlocks = ['gmail_v2', 'notion_v3'] + const vis = await getBlockVisibility({ userId: 'u1' }) + expect(vis.revealed).toEqual(new Set(['gmail_v2', 'notion_v3'])) + expect(vis.previewTagged).toEqual(new Set(['gmail_v2', 'notion_v3'])) + expect(vis.disabled.size).toBe(0) + expect(mockFetch).not.toHaveBeenCalled() + }) + + it('returns empty state when PREVIEW_BLOCKS is unset', async () => { + const vis = await getBlockVisibility() + expect(vis.revealed.size).toBe(0) + expect(vis.disabled.size).toBe(0) + expect(vis.previewTagged.size).toBe(0) + }) + }) + + it('fetches the block-visibility profile', async () => { + withAppConfig({}) + await getBlockVisibility() + expect(mockFetch).toHaveBeenCalledWith( + { application: 'sim-staging', environment: 'staging', profile: 'block-visibility' }, + expect.any(Function) + ) + }) + + it('GA rule (enabled: true) reveals without a preview tag', async () => { + withAppConfig({ gmail_v2: { enabled: true } }) + const vis = await getBlockVisibility({ userId: 'u1' }) + expect(vis.revealed.has('gmail_v2')).toBe(true) + expect(vis.previewTagged.has('gmail_v2')).toBe(false) + expect(vis.disabled.has('gmail_v2')).toBe(false) + }) + + it('allowlist rule reveals with a preview tag; non-matching viewers get disabled', async () => { + withAppConfig({ gmail_v2: { enabled: false, orgIds: ['o1'], userIds: ['u9'] } }) + + const allowedOrg = await getBlockVisibility({ orgId: 'o1' }) + expect(allowedOrg.revealed.has('gmail_v2')).toBe(true) + expect(allowedOrg.previewTagged.has('gmail_v2')).toBe(true) + + const allowedUser = await getBlockVisibility({ userId: 'u9' }) + expect(allowedUser.revealed.has('gmail_v2')).toBe(true) + + const denied = await getBlockVisibility({ userId: 'u1', orgId: 'o2' }) + expect(denied.revealed.has('gmail_v2')).toBe(false) + expect(denied.disabled.has('gmail_v2')).toBe(true) + }) + + it('kill switch (enabled: false, no allowlists) disables for everyone', async () => { + withAppConfig({ slack: { enabled: false } }) + const vis = await getBlockVisibility({ userId: 'u1', orgId: 'o1' }) + expect(vis.disabled.has('slack')).toBe(true) + expect(vis.revealed.has('slack')).toBe(false) + }) + + it('drops custom_block_* keys so custom blocks can never be gated', async () => { + withAppConfig({ custom_block_abc123: { enabled: false }, gmail_v2: { enabled: true } }) + const vis = await getBlockVisibility({ userId: 'u1' }) + expect(vis.disabled.has('custom_block_abc123')).toBe(false) + expect(vis.revealed.has('custom_block_abc123')).toBe(false) + expect(vis.revealed.has('gmail_v2')).toBe(true) + }) + + it('drops malformed entries', async () => { + withAppConfig({ a: 'nope', b: { enabled: false, orgIds: [' o1 ', ''] } }) + const vis = await getBlockVisibility({ orgId: 'o1' }) + expect(vis.disabled.has('a')).toBe(false) + expect(vis.revealed.has('b')).toBe(true) + }) + + describe('admin resolution (once per call)', () => { + it('resolves admin exactly once for a document with multiple adminEnabled rules', async () => { + withAppConfig({ + a: { enabled: false, adminEnabled: true }, + b: { enabled: false, adminEnabled: true }, + c: { enabled: false }, + }) + mockIsPlatformAdmin.mockResolvedValue(true) + const vis = await getBlockVisibility({ userId: 'u1' }) + expect(mockIsPlatformAdmin).toHaveBeenCalledTimes(1) + expect(vis.revealed).toEqual(new Set(['a', 'b'])) + expect(vis.previewTagged).toEqual(new Set(['a', 'b'])) + expect(vis.disabled).toEqual(new Set(['c'])) + }) + + it('uses the isAdmin fast-path without querying', async () => { + withAppConfig({ a: { enabled: false, adminEnabled: true } }) + const vis = await getBlockVisibility({ userId: 'u1', isAdmin: true }) + expect(vis.revealed.has('a')).toBe(true) + expect(mockIsPlatformAdmin).not.toHaveBeenCalled() + }) + + it('does not query when no rule has adminEnabled or when userId is absent', async () => { + withAppConfig({ a: { enabled: false, orgIds: ['o1'] } }) + await getBlockVisibility({ userId: 'u1' }) + expect(mockIsPlatformAdmin).not.toHaveBeenCalled() + + withAppConfig({ a: { enabled: false, adminEnabled: true } }) + const vis = await getBlockVisibility({ orgId: 'o1' }) + expect(vis.disabled.has('a')).toBe(true) + expect(mockIsPlatformAdmin).not.toHaveBeenCalled() + }) + }) +}) diff --git a/apps/sim/lib/core/config/block-visibility.ts b/apps/sim/lib/core/config/block-visibility.ts new file mode 100644 index 00000000000..3a8cb96447a --- /dev/null +++ b/apps/sim/lib/core/config/block-visibility.ts @@ -0,0 +1,110 @@ +import { fetchAppConfigProfile } from '@/lib/core/config/appconfig' +import type { AppConfigGateContext, AppConfigGateRule } from '@/lib/core/config/appconfig-rules' +import { matchesRule, parseGateConfig } from '@/lib/core/config/appconfig-rules' +import { env } from '@/lib/core/config/env' +import { getPreviewBlocksFromEnv, isAppConfigEnabled } from '@/lib/core/config/env-flags' + +/** + * Name of the AppConfig configuration profile holding per-block visibility rules. + * Cross-repo contract: must match the `CfnConfigurationProfile` name created by + * the infra stack (`BLOCK_VISIBILITY_PROFILE_NAME`). + */ +const BLOCK_VISIBILITY_PROFILE = 'block-visibility' + +/** + * Custom (deploy-as-block) block types are org-scoped and managed by their own + * enabled/disabled lifecycle — the visibility document must never gate them. + * Literal mirrors `CUSTOM_BLOCK_TYPE_PREFIX` in `@/blocks/custom/build-config`, + * not imported to keep the blocks graph out of this config module. + */ +const CUSTOM_BLOCK_KEY_PREFIX = 'custom_block_' + +/** Per-request evaluation context; same shape as the feature-flag context. */ +export type BlockVisibilityContext = AppConfigGateContext + +/** + * The evaluated per-viewer visibility projection. + * + * - `revealed` — preview block types this viewer may see. + * - `disabled` — types whose rule exists but matched no clause; hides + * non-preview (shipped) blocks from discovery surfaces (the kill switch). + * - `previewTagged` — revealed types not globally GA (`enabled !== true`); + * the registry appends " (Preview)" to their names. + * + * All three are needed: `revealed \ previewTagged` is the "GA'd via config while + * `preview: true` is still in code" window, and `disabled` targets a disjoint + * (non-preview) population. + */ +export interface BlockVisibilityState { + revealed: Set + disabled: Set + previewTagged: Set +} + +function parseVisibilityConfig(json: unknown): Record { + const rules = parseGateConfig(json) + for (const key of Object.keys(rules)) { + if (key.startsWith(CUSTOM_BLOCK_KEY_PREFIX)) delete rules[key] + } + return rules +} + +/** + * Resolve platform-admin status lazily. Dynamically imported so the DB-backed + * helper (and `@sim/db`) stay out of this config module's load graph for callers + * that never reach an admin-gated rule. + */ +async function resolveAdmin(userId: string): Promise { + const { isPlatformAdmin } = await import('@/lib/permissions/super-user') + return isPlatformAdmin(userId) +} + +/** + * Evaluate the block-visibility document for a viewer. + * + * On hosted deployments the rules come from the AppConfig profile (cached, + * ~30s TTL); off-AppConfig the `PREVIEW_BLOCKS` env allowlist is the only + * reveal path and nothing is disabled. + * + * Unlike feature-flags (one rule per call, admin resolved lazily per rule), + * this evaluates the whole document, so platform-admin status is resolved at + * most ONCE per call — and only when some rule actually has `adminEnabled` and + * the caller didn't already supply `ctx.isAdmin`. + */ +export async function getBlockVisibility( + ctx: BlockVisibilityContext = {} +): Promise { + if (!isAppConfigEnabled) { + const revealed = new Set(getPreviewBlocksFromEnv()) + return { revealed, disabled: new Set(), previewTagged: new Set(revealed) } + } + + const rules = + (await fetchAppConfigProfile( + { + application: env.APPCONFIG_APPLICATION as string, + environment: env.APPCONFIG_ENVIRONMENT as string, + profile: BLOCK_VISIBILITY_PROFILE, + }, + parseVisibilityConfig + )) ?? {} + + const needsAdmin = + ctx.isAdmin === undefined && + Boolean(ctx.userId) && + Object.values(rules).some((rule) => rule.adminEnabled) + const isAdmin = ctx.isAdmin ?? (needsAdmin ? await resolveAdmin(ctx.userId as string) : false) + + const revealed = new Set() + const disabled = new Set() + const previewTagged = new Set() + for (const [type, rule] of Object.entries(rules)) { + if (matchesRule(rule, ctx, isAdmin)) { + revealed.add(type) + if (rule.enabled !== true) previewTagged.add(type) + } else { + disabled.add(type) + } + } + return { revealed, disabled, previewTagged } +} diff --git a/apps/sim/lib/core/config/env-flags.ts b/apps/sim/lib/core/config/env-flags.ts index b9f3478159f..92b7b26219c 100644 --- a/apps/sim/lib/core/config/env-flags.ts +++ b/apps/sim/lib/core/config/env-flags.ts @@ -293,6 +293,19 @@ export function getAllowedIntegrationsFromEnv(): string[] | null { return parsed.length > 0 ? parsed : null } +/** + * Returns the preview block types revealed via the environment variable — the + * off-AppConfig reveal path for self-hosters and local dev. If not set or empty, + * returns an empty array (all `preview: true` blocks stay hidden). Block types + * are already lowercase snake_case, so entries are trimmed but not lowercased. + */ +export function getPreviewBlocksFromEnv(): string[] { + if (!env.PREVIEW_BLOCKS) return [] + return env.PREVIEW_BLOCKS.split(',') + .map((t) => t.trim()) + .filter(Boolean) +} + /** * Returns the list of blacklisted provider IDs from the environment variable. * If not set or empty, returns an empty array (meaning no providers are blacklisted). diff --git a/apps/sim/lib/core/config/env.ts b/apps/sim/lib/core/config/env.ts index d7903b9a380..990ab7679d3 100644 --- a/apps/sim/lib/core/config/env.ts +++ b/apps/sim/lib/core/config/env.ts @@ -163,6 +163,7 @@ export const env = createEnv({ BLACKLISTED_MODELS: z.string().optional(), // Comma-separated model names/prefixes to hide (e.g., "gpt-4,claude-*") ALLOWED_MCP_DOMAINS: z.string().optional(), // Comma-separated domains for MCP servers (e.g., "internal.company.com,mcp.example.org"). Empty = all allowed. ALLOWED_INTEGRATIONS: z.string().optional(), // Comma-separated block types to allow (e.g., "slack,github,agent"). Empty = all allowed. + PREVIEW_BLOCKS: z.string().optional(), // Comma-separated preview block types to reveal off-AppConfig (e.g., "gmail_v2,notion_v3"). Empty = all preview blocks hidden. // Azure Configuration - Shared credentials with feature-specific models AZURE_OPENAI_ENDPOINT: z.string().url().optional(), // Shared Azure OpenAI service endpoint diff --git a/apps/sim/lib/core/config/feature-flags.ts b/apps/sim/lib/core/config/feature-flags.ts index 8a44624ba78..558707fa587 100644 --- a/apps/sim/lib/core/config/feature-flags.ts +++ b/apps/sim/lib/core/config/feature-flags.ts @@ -1,4 +1,6 @@ import { fetchAppConfigProfile } from '@/lib/core/config/appconfig' +import type { AppConfigGateContext, AppConfigGateRule } from '@/lib/core/config/appconfig-rules' +import { matchesRule, parseGateConfig } from '@/lib/core/config/appconfig-rules' import { env, isTruthy } from '@/lib/core/config/env' import { isAppConfigEnabled } from '@/lib/core/config/env-flags' @@ -11,15 +13,11 @@ const FEATURE_FLAGS_PROFILE = 'feature-flags' /** * A single flag's gating rule. A flag is ON for a context when ANY clause matches: - * the global `enabled` default, the org/user allowlists, or `admins` for platform - * admins. An absent clause never matches. + * the global `enabled` default, the org/user allowlists, or `adminEnabled` for + * platform admins. An absent clause never matches. Shape shared with the other + * AppConfig gating documents via {@link AppConfigGateRule}. */ -export interface FeatureFlagRule { - enabled?: boolean - orgIds?: string[] - userIds?: string[] - adminEnabled?: boolean -} +export type FeatureFlagRule = AppConfigGateRule export type FeatureFlagsConfig = Record @@ -28,11 +26,7 @@ export type FeatureFlagsConfig = Record * its clause. Admin status is resolved internally from `userId`; `isAdmin` is an * optional fast-path override for callers that already know it (e.g. admin routes). */ -export interface FeatureFlagContext { - userId?: string | null - orgId?: string | null - isAdmin?: boolean -} +export type FeatureFlagContext = AppConfigGateContext /** * Registry of known feature flags. Each maps to the secret consulted ONLY when @@ -137,36 +131,6 @@ function fallbackFlags(): FeatureFlagsConfig { return flags } -function normalizeIds(values: unknown): string[] | undefined { - if (!Array.isArray(values)) return undefined - const ids = Array.from(new Set(values.map((v) => String(v).trim()).filter(Boolean))) - return ids.length > 0 ? ids : undefined -} - -function normalizeRule(value: unknown): FeatureFlagRule | null { - if (!value || typeof value !== 'object') return null - const obj = value as Record - const rule: FeatureFlagRule = {} - if (typeof obj.enabled === 'boolean') rule.enabled = obj.enabled - if (typeof obj.adminEnabled === 'boolean') rule.adminEnabled = obj.adminEnabled - const orgIds = normalizeIds(obj.orgIds) - if (orgIds) rule.orgIds = orgIds - const userIds = normalizeIds(obj.userIds) - if (userIds) rule.userIds = userIds - return rule -} - -/** Coerce an arbitrary AppConfig/JSON value into a config, dropping malformed entries. */ -function parseConfig(json: unknown): FeatureFlagsConfig { - const obj = (json && typeof json === 'object' ? json : {}) as Record - const flags: FeatureFlagsConfig = {} - for (const [name, value] of Object.entries(obj)) { - const rule = normalizeRule(value) - if (rule) flags[name] = rule - } - return flags -} - /** * Resolve platform-admin status lazily. Dynamically imported so the DB-backed * helper (and `@sim/db`) stay out of this config module's load graph for callers @@ -179,17 +143,15 @@ async function resolveAdmin(userId: string): Promise { /** * The admin clause is resolved last and lazily: a global/userId/orgId match - * short-circuits before any DB read, a rule without `admins` never queries, and a - * missing `userId` resolves to `false` without a query. + * short-circuits before any DB read, a rule without `adminEnabled` never queries, + * and a missing `userId` resolves to `false` without a query. */ async function evaluate( rule: FeatureFlagRule | undefined, ctx: FeatureFlagContext ): Promise { if (!rule) return false - if (rule.enabled) return true - if (ctx.userId && rule.userIds?.includes(ctx.userId)) return true - if (ctx.orgId && rule.orgIds?.includes(ctx.orgId)) return true + if (matchesRule(rule, ctx, false)) return true if (rule.adminEnabled) { const admin = ctx.isAdmin ?? (ctx.userId ? await resolveAdmin(ctx.userId) : false) if (admin) return true @@ -211,7 +173,7 @@ export async function getFeatureFlags(): Promise { environment: env.APPCONFIG_ENVIRONMENT as string, profile: FEATURE_FLAGS_PROFILE, }, - parseConfig + parseGateConfig ) return value ?? fallbackFlags() diff --git a/apps/sim/lib/integrations/integrations.json b/apps/sim/lib/integrations/integrations.json index 8e741d71e57..16dd3fc2a81 100644 --- a/apps/sim/lib/integrations/integrations.json +++ b/apps/sim/lib/integrations/integrations.json @@ -11780,9 +11780,14 @@ "id": "microsoftteams_webhook", "name": "Microsoft Teams Channel", "description": "Trigger workflow from Microsoft Teams channel messages via outgoing webhooks" + }, + { + "id": "microsoftteams_chat_subscription", + "name": "Microsoft Teams Chat", + "description": "Trigger workflow from new messages in Microsoft Teams chats via Microsoft Graph subscriptions" } ], - "triggerCount": 1, + "triggerCount": 2, "authType": "oauth", "oauthServiceId": "microsoft-teams", "category": "tools", diff --git a/apps/sim/lib/knowledge/chunks/service.ts b/apps/sim/lib/knowledge/chunks/service.ts index fe34a40b543..6660027d470 100644 --- a/apps/sim/lib/knowledge/chunks/service.ts +++ b/apps/sim/lib/knowledge/chunks/service.ts @@ -17,6 +17,8 @@ import { estimateTokenCount } from '@/lib/tokenization/estimators' const logger = createLogger('ChunksService') +const KB_CHUNK_LOCK_TIMEOUT_MS = 5_000 + /** * Query chunks for a document with filtering and pagination */ @@ -101,7 +103,23 @@ export async function queryChunks( } /** - * Create a new chunk for a document + * Create a new chunk for a document. + * + * Assigns `chunkIndex` as `max(chunkIndex) + 1` under a transactional + * `pg_advisory_xact_lock` keyed on the document, so concurrent calls for the + * same document serialize instead of computing the same index and colliding + * on the `(document_id, chunk_index)` unique constraint. A `SELECT ... FOR + * UPDATE` on the current max row doesn't prevent that collision, since the + * row it would lock is unrelated to the not-yet-inserted next row. A row + * lock on `document` instead of an advisory lock would also work, but would + * invert the embedding-before-document lock order every other chunk + * mutation path uses (see lock-order.test.ts) — the advisory lock is a + * separate namespace, so it can't deadlock against that convention. + * + * `pg_advisory_xact_lock` auto-releases at transaction end, so there's no + * session lock to leak onto a pooled connection, and `lock_timeout` bounds + * the wait (it raises SQLSTATE 55P03 instead of hanging a pooled connection) + * if a same-document holder is stuck. */ export async function createChunk( knowledgeBaseId: string, @@ -135,8 +153,14 @@ export async function createChunk( const chunkId = generateId() const now = new Date() - // Use transaction to atomically get next index and insert chunk const newChunk = await db.transaction(async (tx) => { + await tx.execute( + sql`select set_config('lock_timeout', ${`${KB_CHUNK_LOCK_TIMEOUT_MS}ms`}, true)` + ) + await tx.execute( + sql`select pg_advisory_xact_lock(hashtextextended(${`kb_chunk_seq:${documentId}`}, 0))` + ) + const activeDocument = await tx .select({ id: document.id }) .from(document) @@ -156,7 +180,6 @@ export async function createChunk( throw new Error('Document not found') } - // Get the next chunk index atomically within the transaction const lastChunk = await tx .select({ chunkIndex: embedding.chunkIndex }) .from(embedding) diff --git a/apps/sim/lib/knowledge/documents/service.ts b/apps/sim/lib/knowledge/documents/service.ts index 56300b1d837..88b33ff7a62 100644 --- a/apps/sim/lib/knowledge/documents/service.ts +++ b/apps/sim/lib/knowledge/documents/service.ts @@ -18,9 +18,9 @@ import type { HighestPrioritySubscription } from '@/lib/billing/core/plan' import { getHighestPrioritySubscription } from '@/lib/billing/core/subscription' import { recordUsage } from '@/lib/billing/core/usage-log' import { - checkStorageQuota, + checkAndIncrementStorageUsageInTx, decrementStorageUsageInTx, - incrementStorageUsage, + maybeNotifyStorageLimit, } from '@/lib/billing/storage' import { checkAndBillOverageThreshold } from '@/lib/billing/threshold-billing' import type { ChunkingStrategy, StrategyOptions } from '@/lib/chunkers/types' @@ -864,6 +864,33 @@ export function isTriggerAvailable(): boolean { return Boolean(env.TRIGGER_SECRET_KEY) && isTriggerDevEnabled } +/** + * Resolves the subscription to bill for a storage-metered upload, ahead of + * the FOR UPDATE transaction that will insert the document row. + * + * Must run before that transaction opens: `knowledgeBase.userId` is + * immutable once a KB is created, so resolving it here can't go stale + * relative to the transaction's own read of it, and resolving it inside the + * transaction would open a second pooled-database-connection checkout while + * the first is held. + */ +async function resolveQuotaSubscription( + knowledgeBaseId: string, + uploadedBy: string | null +): Promise { + const billedUserId = + uploadedBy ?? + ( + await db + .select({ userId: knowledgeBase.userId }) + .from(knowledgeBase) + .where(eq(knowledgeBase.id, knowledgeBaseId)) + .limit(1) + )[0]?.userId + + return billedUserId ? getHighestPrioritySubscription(billedUserId) : null +} + export async function createDocumentRecords( documents: Array<{ filename: string @@ -883,7 +910,15 @@ export async function createDocumentRecords( requestId: string, uploadedBy: string | null = null ): Promise { - let storageBilling: { userId: string; workspaceId: string | null; bytes: number } | null = null + let storageBilling: { + userId: string + workspaceId: string | null + bytes: number + sub: HighestPrioritySubscription | null + } | null = null + + const totalBytes = documents.reduce((sum, docData) => sum + (docData.fileSize || 0), 0) + const sub = totalBytes > 0 ? await resolveQuotaSubscription(knowledgeBaseId, uploadedBy) : null const returnData = await db.transaction(async (tx) => { await tx.execute(sql`SELECT 1 FROM knowledge_base WHERE id = ${knowledgeBaseId} FOR UPDATE`) @@ -912,13 +947,12 @@ export async function createDocumentRecords( ) const billedUserId = uploadedBy ?? kb[0].userId - const totalBytes = documents.reduce((sum, docData) => sum + (docData.fileSize || 0), 0) if (totalBytes > 0) { - const quotaCheck = await checkStorageQuota(billedUserId, totalBytes) + const quotaCheck = await checkAndIncrementStorageUsageInTx(tx, sub, billedUserId, totalBytes) if (!quotaCheck.allowed) { throw new Error(quotaCheck.error || 'Storage limit exceeded') } - storageBilling = { userId: billedUserId, workspaceId: kbWorkspaceId, bytes: totalBytes } + storageBilling = { userId: billedUserId, workspaceId: kbWorkspaceId, bytes: totalBytes, sub } } // One load per batch (was N+1); skip entirely if no doc carries tags. @@ -1011,11 +1045,14 @@ export async function createDocumentRecords( }) if (storageBilling) { - const billing: { userId: string; workspaceId: string | null; bytes: number } = storageBilling - try { - await incrementStorageUsage(billing.userId, billing.bytes, billing.workspaceId ?? undefined) - } catch (storageError) { - logger.error(`[${requestId}] Failed to update storage tracking:`, storageError) + const billing: { + userId: string + workspaceId: string | null + bytes: number + sub: HighestPrioritySubscription | null + } = storageBilling + if (billing.workspaceId) { + void maybeNotifyStorageLimit(billing.userId, billing.workspaceId, billing.sub) } } @@ -1338,7 +1375,15 @@ export async function createSingleDocument( ...processedTags, } - let storageBilling: { userId: string; workspaceId: string | null; bytes: number } | null = null + let storageBilling: { + userId: string + workspaceId: string | null + bytes: number + sub: HighestPrioritySubscription | null + } | null = null + + const sub = + documentData.fileSize > 0 ? await resolveQuotaSubscription(knowledgeBaseId, uploadedBy) : null await db.transaction(async (tx) => { await tx.execute(sql`SELECT 1 FROM knowledge_base WHERE id = ${knowledgeBaseId} FOR UPDATE`) @@ -1367,7 +1412,12 @@ export async function createSingleDocument( const billedUserId = uploadedBy ?? kb[0].userId if (documentData.fileSize > 0) { - const quotaCheck = await checkStorageQuota(billedUserId, documentData.fileSize) + const quotaCheck = await checkAndIncrementStorageUsageInTx( + tx, + sub, + billedUserId, + documentData.fileSize + ) if (!quotaCheck.allowed) { throw new Error(quotaCheck.error || 'Storage limit exceeded') } @@ -1375,6 +1425,7 @@ export async function createSingleDocument( userId: billedUserId, workspaceId: kb[0].workspaceId, bytes: documentData.fileSize, + sub, } } @@ -1387,11 +1438,14 @@ export async function createSingleDocument( }) if (storageBilling) { - const billing: { userId: string; workspaceId: string | null; bytes: number } = storageBilling - try { - await incrementStorageUsage(billing.userId, billing.bytes, billing.workspaceId ?? undefined) - } catch (storageError) { - logger.error(`[${requestId}] Failed to update storage tracking:`, storageError) + const billing: { + userId: string + workspaceId: string | null + bytes: number + sub: HighestPrioritySubscription | null + } = storageBilling + if (billing.workspaceId) { + void maybeNotifyStorageLimit(billing.userId, billing.workspaceId, billing.sub) } } diff --git a/apps/sim/lib/search/tool-operations.ts b/apps/sim/lib/search/tool-operations.ts index 25640935deb..c4c2cfa4fd9 100644 --- a/apps/sim/lib/search/tool-operations.ts +++ b/apps/sim/lib/search/tool-operations.ts @@ -1,6 +1,7 @@ import type { ComponentType } from 'react' import { getAllBlocks } from '@/blocks' import type { BlockConfig, SubBlockConfig } from '@/blocks/types' +import { registerBlockCacheInvalidator } from '@/blocks/visibility/context' /** * Represents a searchable tool operation extracted from block configurations. @@ -169,13 +170,17 @@ export function buildToolOperationsIndex(): ToolOperationItem[] { /** * Cached operations index to avoid rebuilding on every search. - * The index is built lazily on first access. + * The index is built lazily on first access and reset when the client + * block-visibility state changes (a preview reveal / kill switch alters the + * `hideFromToolbar` projection this index is filtered on). */ let cachedOperations: ToolOperationItem[] | null = null +registerBlockCacheInvalidator(() => clearToolOperationsCache()) + /** * Returns the tool operations index, building it if necessary. - * The index is cached after first build since block registry is static. + * The index is cached after first build and dropped on visibility changes. */ export function getToolOperationsIndex(): ToolOperationItem[] { if (!cachedOperations) { diff --git a/apps/sim/lib/uploads/core/storage-service.blob-connection-string.test.ts b/apps/sim/lib/uploads/core/storage-service.blob-connection-string.test.ts new file mode 100644 index 00000000000..dffb79607eb --- /dev/null +++ b/apps/sim/lib/uploads/core/storage-service.blob-connection-string.test.ts @@ -0,0 +1,82 @@ +/** + * Regression tests: Azure Blob storage must be fully usable with ONLY + * AZURE_CONNECTION_STRING set (no AZURE_ACCOUNT_NAME/AZURE_ACCOUNT_KEY) — this + * is the connection-string auth mode documented as a standalone alternative + * across .env.example, helm/sim/values.yaml, and env.ts. + * + * @vitest-environment node + */ +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const CONNECTION_STRING = + 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;' + +const { mockHeadBlobObject, mockGetBlobServiceClient, mockGenerateBlobSASQueryParameters } = + vi.hoisted(() => ({ + mockHeadBlobObject: vi.fn(), + mockGetBlobServiceClient: vi.fn(), + mockGenerateBlobSASQueryParameters: vi.fn(() => ({ toString: () => 'sig=fake' })), + })) + +vi.mock('@/lib/uploads/config', () => ({ + USE_S3_STORAGE: false, + USE_BLOB_STORAGE: true, + // Connection-string-only: accountName/accountKey intentionally absent. + getStorageConfig: () => ({ + containerName: 'workspace-files', + accountName: undefined, + accountKey: undefined, + connectionString: CONNECTION_STRING, + }), +})) + +vi.mock('@/lib/uploads/providers/blob/client', () => ({ + headBlobObject: mockHeadBlobObject, + getBlobServiceClient: mockGetBlobServiceClient, + parseConnectionString: (connectionString: string) => { + const accountName = connectionString.match(/AccountName=([^;]+)/)?.[1] + const accountKey = connectionString.match(/AccountKey=([^;]+)/)?.[1] + if (!accountName || !accountKey) throw new Error('cannot parse') + return { accountName, accountKey } + }, +})) + +vi.mock('@azure/storage-blob', () => ({ + StorageSharedKeyCredential: vi.fn(), + BlobSASPermissions: { parse: vi.fn(() => 'w') }, + generateBlobSASQueryParameters: mockGenerateBlobSASQueryParameters, +})) + +import { generatePresignedUploadUrl, headObject } from '@/lib/uploads/core/storage-service' + +describe('Azure Blob storage — connection-string-only auth', () => { + beforeEach(() => { + vi.clearAllMocks() + mockHeadBlobObject.mockResolvedValue({ size: 42, contentType: 'text/plain' }) + mockGetBlobServiceClient.mockResolvedValue({ + getContainerClient: () => ({ + getBlockBlobClient: () => ({ url: 'https://devstoreaccount1.blob.core.windows.net/c/k' }), + }), + }) + }) + + it('headObject does not throw when only connectionString is configured', async () => { + await expect(headObject('some-key', 'workspace')).resolves.toEqual({ + size: 42, + contentType: 'text/plain', + }) + expect(mockHeadBlobObject).toHaveBeenCalled() + }) + + it('generatePresignedUploadUrl derives SAS credentials from connectionString when accountName/accountKey are absent', async () => { + const result = await generatePresignedUploadUrl({ + fileName: 'report.csv', + contentType: 'text/csv', + context: 'workspace', + fileSize: 100, + }) + + expect(mockGenerateBlobSASQueryParameters).toHaveBeenCalled() + expect(result.url).toContain('sig=fake') + }) +}) diff --git a/apps/sim/lib/uploads/core/storage-service.ts b/apps/sim/lib/uploads/core/storage-service.ts index 9584bb89968..d88f931db67 100644 --- a/apps/sim/lib/uploads/core/storage-service.ts +++ b/apps/sim/lib/uploads/core/storage-service.ts @@ -29,13 +29,13 @@ const logger = createLogger('StorageService') * @throws Error if required properties are missing */ function createBlobConfig(config: StorageConfig): BlobConfig { - if (!config.containerName || !config.accountName) { - throw new Error('Blob configuration missing required properties: containerName and accountName') + if (!config.containerName) { + throw new Error('Blob configuration missing required property: containerName') } - if (!config.connectionString && !config.accountKey) { + if (!config.connectionString && !(config.accountName && config.accountKey)) { throw new Error( - 'Blob configuration missing authentication: either connectionString or accountKey must be provided' + 'Blob configuration missing authentication: either connectionString or both accountName and accountKey must be provided' ) } @@ -635,7 +635,9 @@ async function generateBlobPresignedUrl( }, expirationSeconds: number ): Promise { - const { getBlobServiceClient } = await import('@/lib/uploads/providers/blob/client') + const { getBlobServiceClient, parseConnectionString } = await import( + '@/lib/uploads/providers/blob/client' + ) const { BlobSASPermissions, generateBlobSASQueryParameters, StorageSharedKeyCredential } = await import('@azure/storage-blob') @@ -650,27 +652,30 @@ async function generateBlobPresignedUrl( const startsOn = new Date() const expiresOn = new Date(startsOn.getTime() + expirationSeconds * 1000) - let sasToken: string + let accountName = config.accountName + let accountKey = config.accountKey + if ((!accountName || !accountKey) && config.connectionString) { + ;({ accountName, accountKey } = parseConnectionString(config.connectionString)) + } - if (config.accountName && config.accountKey) { - const sharedKeyCredential = new StorageSharedKeyCredential( - config.accountName, - config.accountKey + if (!accountName || !accountKey) { + throw new Error( + 'Azure Blob SAS generation requires accountName/accountKey or a connectionString' ) - sasToken = generateBlobSASQueryParameters( - { - containerName: config.containerName, - blobName: key, - permissions: BlobSASPermissions.parse('w'), // write permission for upload - startsOn, - expiresOn, - }, - sharedKeyCredential - ).toString() - } else { - throw new Error('Azure Blob SAS generation requires accountName and accountKey') } + const sharedKeyCredential = new StorageSharedKeyCredential(accountName, accountKey) + const sasToken = generateBlobSASQueryParameters( + { + containerName: config.containerName, + blobName: key, + permissions: BlobSASPermissions.parse('w'), // write permission for upload + startsOn, + expiresOn, + }, + sharedKeyCredential + ).toString() + return { url: `${blobClient.url}?${sasToken}`, key, diff --git a/apps/sim/lib/uploads/providers/blob/client.test.ts b/apps/sim/lib/uploads/providers/blob/client.test.ts index 024fa2d794f..19b06c93a91 100644 --- a/apps/sim/lib/uploads/providers/blob/client.test.ts +++ b/apps/sim/lib/uploads/providers/blob/client.test.ts @@ -54,6 +54,7 @@ import { deleteFromBlob, downloadFromBlob, getPresignedUrl, + parseConnectionString, uploadToBlob, } from '@/lib/uploads/providers/blob/client' import { sanitizeFilenameForMetadata } from '@/lib/uploads/utils/file-utils' @@ -207,6 +208,27 @@ describe('Azure Blob Storage Client', () => { }) }) + describe('parseConnectionString', () => { + it('extracts accountName and accountKey from a well-formed connection string', () => { + const result = parseConnectionString( + 'DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey123;EndpointSuffix=core.windows.net' + ) + expect(result).toEqual({ accountName: 'myaccount', accountKey: 'mykey123' }) + }) + + it('throws when AccountName is missing', () => { + expect(() => + parseConnectionString('DefaultEndpointsProtocol=https;AccountKey=mykey123') + ).toThrow('Cannot extract account name from connection string') + }) + + it('throws when AccountKey is missing', () => { + expect(() => + parseConnectionString('DefaultEndpointsProtocol=https;AccountName=myaccount') + ).toThrow('Cannot extract account key from connection string') + }) + }) + describe('sanitizeFilenameForMetadata', () => { const testCases = [ { input: 'test file.txt', expected: 'test file.txt' }, diff --git a/apps/sim/lib/uploads/providers/blob/client.ts b/apps/sim/lib/uploads/providers/blob/client.ts index f2fa5925220..eda490a8b82 100644 --- a/apps/sim/lib/uploads/providers/blob/client.ts +++ b/apps/sim/lib/uploads/providers/blob/client.ts @@ -30,7 +30,7 @@ interface ParsedCredentials { * Extract account name and key from an Azure connection string. * Connection strings have the format: DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=... */ -function parseConnectionString(connectionString: string): ParsedCredentials { +export function parseConnectionString(connectionString: string): ParsedCredentials { const accountNameMatch = connectionString.match(/AccountName=([^;]+)/) if (!accountNameMatch) { throw new Error('Cannot extract account name from connection string') diff --git a/apps/sim/lib/uploads/providers/blob/types.ts b/apps/sim/lib/uploads/providers/blob/types.ts index b024e2a6363..a24a87e384e 100644 --- a/apps/sim/lib/uploads/providers/blob/types.ts +++ b/apps/sim/lib/uploads/providers/blob/types.ts @@ -1,6 +1,6 @@ export interface BlobConfig { containerName: string - accountName: string + accountName?: string accountKey?: string connectionString?: string } diff --git a/apps/sim/lib/workflows/sanitization/subblocks.test.ts b/apps/sim/lib/workflows/sanitization/subblocks.test.ts new file mode 100644 index 00000000000..a9a714a400a --- /dev/null +++ b/apps/sim/lib/workflows/sanitization/subblocks.test.ts @@ -0,0 +1,137 @@ +/** + * @vitest-environment node + */ +import { describe, expect, it, vi } from 'vitest' + +vi.unmock('@/blocks/registry') + +import { migrateSubblockIds } from '@/lib/workflows/migrations/subblock-migrations' +import { sanitizeMalformedSubBlocks } from '@/lib/workflows/sanitization/subblocks' +import type { BlockState } from '@/stores/workflows/workflow/types' + +const FIELD_ID = 'cd7e4a16-c608-4087-8f2d-61f9672baeda' + +/** + * A placed custom block whose stored structure only has the wiring sub-blocks. + * `getBlock` cannot resolve `custom_block_*` types outside the org overlay — + * exactly the draft-load context where sanitization runs. + */ +function makeCustomBlock(subBlocks: Record) { + return { id: 'block-1', type: 'custom_block_abc123', subBlocks } +} + +describe('sanitizeMalformedSubBlocks', () => { + describe('custom blocks (schema-agnostic)', () => { + it('keeps and repairs a consumer-typed field value stored with the realtime "unknown" fallback', () => { + const { subBlocks, changed } = sanitizeMalformedSubBlocks( + makeCustomBlock({ + workflowId: { id: 'workflowId', type: 'short-input', value: 'wf-1' }, + inputMapping: { id: 'inputMapping', type: 'code', value: '{}' }, + [FIELD_ID]: { id: FIELD_ID, type: 'unknown', value: 'theo' }, + }) + ) + + expect(changed).toBe(true) + expect(subBlocks[FIELD_ID]).toEqual({ id: FIELD_ID, type: 'short-input', value: 'theo' }) + expect(subBlocks.workflowId.value).toBe('wf-1') + expect(subBlocks.inputMapping.value).toBe('{}') + }) + + it('keeps a raw non-record field value by wrapping it in a repaired entry', () => { + const { subBlocks, changed } = sanitizeMalformedSubBlocks( + makeCustomBlock({ [FIELD_ID]: 'theo' }) + ) + + expect(changed).toBe(true) + expect(subBlocks[FIELD_ID]).toEqual({ id: FIELD_ID, type: 'short-input', value: 'theo' }) + }) + + it('keeps an entry with missing metadata, keying it by the map key', () => { + const { subBlocks, changed } = sanitizeMalformedSubBlocks( + makeCustomBlock({ [FIELD_ID]: { value: 'theo' } }) + ) + + expect(changed).toBe(true) + expect(subBlocks[FIELD_ID]).toEqual({ id: FIELD_ID, type: 'short-input', value: 'theo' }) + }) + + it('leaves well-formed sub-blocks untouched and reports no change', () => { + const input = { + workflowId: { id: 'workflowId', type: 'short-input', value: 'wf-1' }, + [FIELD_ID]: { id: FIELD_ID, type: 'short-input', value: 'theo' }, + } + const { subBlocks, changed } = sanitizeMalformedSubBlocks(makeCustomBlock(input)) + + expect(changed).toBe(false) + expect(subBlocks).toBe(input) + }) + + it('still drops the literal "undefined" key', () => { + const { subBlocks, changed } = sanitizeMalformedSubBlocks( + makeCustomBlock({ undefined: { id: 'undefined', type: 'unknown', value: 'x' } }) + ) + + expect(changed).toBe(true) + expect(subBlocks).toEqual({}) + }) + }) + + describe('through migrateSubblockIds (the draft-load migration pipeline)', () => { + it('a typed custom-block field value survives the load-time migration pass', () => { + const blocks: Record = { + b1: { + id: 'b1', + name: 'Update Internal Allowlist 1', + position: { x: 0, y: 0 }, + type: 'custom_block_abc123', + subBlocks: { + workflowId: { id: 'workflowId', type: 'short-input', value: 'wf-child' }, + inputMapping: { id: 'inputMapping', type: 'code', value: '{}' }, + [FIELD_ID]: { id: FIELD_ID, type: 'unknown', value: 'test' }, + }, + outputs: {}, + enabled: true, + } as BlockState, + } + + const { blocks: migrated, migrated: changed } = migrateSubblockIds(blocks) + + expect(changed).toBe(true) + expect(migrated.b1.subBlocks[FIELD_ID]).toEqual({ + id: FIELD_ID, + type: 'short-input', + value: 'test', + }) + }) + }) + + describe('regular blocks (config is the schema)', () => { + it('still drops an "unknown"-typed entry that matches no configured sub-block', () => { + const { subBlocks, changed } = sanitizeMalformedSubBlocks({ + id: 'block-1', + type: 'function', + subBlocks: { + code: { id: 'code', type: 'code', value: 'return 1' }, + stale: { id: 'stale', type: 'unknown', value: 'x' }, + }, + }) + + expect(changed).toBe(true) + expect(subBlocks.stale).toBeUndefined() + expect(subBlocks.code.value).toBe('return 1') + }) + + it('repairs an "unknown"-typed entry that matches a configured sub-block', () => { + const { subBlocks, changed } = sanitizeMalformedSubBlocks({ + id: 'block-1', + type: 'function', + subBlocks: { + code: { id: 'code', type: 'unknown', value: 'return 1' }, + }, + }) + + expect(changed).toBe(true) + expect(subBlocks.code).toEqual({ id: 'code', type: 'code', value: 'return 1' }) + }) + }) +}) diff --git a/apps/sim/lib/workflows/sanitization/subblocks.ts b/apps/sim/lib/workflows/sanitization/subblocks.ts index a97bb7b479b..1f0d222ce2a 100644 --- a/apps/sim/lib/workflows/sanitization/subblocks.ts +++ b/apps/sim/lib/workflows/sanitization/subblocks.ts @@ -2,6 +2,7 @@ import { createLogger } from '@sim/logger' import { isPlainRecord } from '@sim/utils/object' import { DEFAULT_SUBBLOCK_TYPE } from '@sim/workflow-persistence/subblocks' import { getBlock } from '@/blocks' +import { isCustomBlockType } from '@/blocks/custom/build-config' import type { BlockState } from '@/stores/workflows/workflow/types' const logger = createLogger('WorkflowSubblockSanitization') @@ -19,6 +20,15 @@ interface SanitizableBlock { /** * Repairs legacy subBlock metadata when the map key identifies a real field, * and drops entries that cannot be associated with a stable subBlock. + * + * Custom blocks are schema-agnostic here: their server-side config never + * declares the per-field input sub-blocks (the execution overlay passes bare + * wiring rows, and this may run with no overlay at all), so "not in config" + * carries no signal for them. A consumer-typed field value stored via the + * realtime `type: 'unknown'` fallback must be repaired to a concrete type and + * kept — dropping it would delete user input from the draft. Values for fields + * the source workflow no longer has are filtered at serialization/execution + * (`customBlockHasDeclaredInputs`, `remapCustomBlockInputKeys`), never at rest. */ export function sanitizeMalformedSubBlocks( block: SanitizableBlock, @@ -26,6 +36,7 @@ export function sanitizeMalformedSubBlocks( ): { subBlocks: Record; changed: boolean } { let changed = false const blockConfig = getBlock(block.type) + const schemaAgnostic = isCustomBlockType(block.type) const result: Record = {} for (const [subBlockId, subBlock] of Object.entries(block.subBlocks || {})) { @@ -38,7 +49,7 @@ export function sanitizeMalformedSubBlocks( const configuredType = blockConfig?.subBlocks?.find((config) => config.id === subBlockId)?.type if (!isPlainRecord(subBlock)) { - if (!configuredType) { + if (!configuredType && !schemaAgnostic) { logger.warn('Skipping malformed subBlock: unrecognized value entry', { blockId: block.id, subBlockId, @@ -57,7 +68,7 @@ export function sanitizeMalformedSubBlocks( continue } - if (subBlock.type === 'unknown' && !configuredType) { + if (subBlock.type === 'unknown' && !configuredType && !schemaAgnostic) { logger.warn('Skipping malformed subBlock: type is "unknown"', { blockId: block.id, subBlockId, @@ -75,7 +86,7 @@ export function sanitizeMalformedSubBlocks( typeof subBlock.type !== 'string' || subBlock.type.length === 0 - if (missingMetadata && !typeFromConfig) { + if (missingMetadata && !typeFromConfig && !schemaAgnostic) { logger.warn('Skipping malformed subBlock: unrecognized metadata entry', { blockId: block.id, subBlockId, diff --git a/apps/sim/scripts/check-block-registry.ts b/apps/sim/scripts/check-block-registry.ts index a1a36ed9bae..13089512f14 100644 --- a/apps/sim/scripts/check-block-registry.ts +++ b/apps/sim/scripts/check-block-registry.ts @@ -255,7 +255,12 @@ function checkIntegrationMetaCoverage(): CheckResult { const errors: string[] = [] for (const block of getAllBlocks()) { - const isCatalogIntegration = block.category === 'tools' && !block.hideFromToolbar + // Unreleased preview blocks ship no BlockMeta until GA (they are absent + // from every catalog surface), so meta coverage must not force one. The + // registry projection already hides them here (no visibility context in a + // script), but the explicit check keeps this true regardless. + const isCatalogIntegration = + block.category === 'tools' && !block.hideFromToolbar && !block.preview if (!isCatalogIntegration) continue if (!getBlockMeta(block.type)) { diff --git a/apps/sim/tools/brex/archive_budget.ts b/apps/sim/tools/brex/archive_budget.ts new file mode 100644 index 00000000000..1749e455d2f --- /dev/null +++ b/apps/sim/tools/brex/archive_budget.ts @@ -0,0 +1,71 @@ +import type { BrexArchiveBudgetParams, BrexArchiveBudgetResponse } from '@/tools/brex/types' +import { BREX_API_BASE, buildBrexHeaders, parseBrexJson } from '@/tools/brex/utils' +import type { ToolConfig } from '@/tools/types' + +export const brexArchiveBudgetTool: ToolConfig = + { + id: 'brex_archive_budget', + name: 'Brex Archive Budget', + description: + 'Archive a Brex budget, making any spend limits beneath it unusable for future expenses and removing it from the UI', + version: '1.0.0', + + params: { + apiKey: { + type: 'string', + required: true, + visibility: 'user-only', + description: 'Brex user token (generated from Developer Settings in the Brex dashboard)', + }, + budgetId: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'ID of the budget to archive', + }, + }, + + request: { + url: (params) => + `${BREX_API_BASE}/v2/budgets/${encodeURIComponent(params.budgetId.trim())}/archive`, + method: 'POST', + headers: (params) => buildBrexHeaders(params.apiKey), + }, + + transformResponse: async (response, params) => { + if (!response.ok) { + // parseBrexJson throws a descriptive error for non-2xx responses; it never + // returns in this branch since the body cannot be a successful JSON payload. + await parseBrexJson(response) + } + + // Brex's archive endpoint does not document a response body schema; fall back + // to the request's budget ID and an ARCHIVED status when the body is empty. + let data: Record = {} + const text = await response.text() + if (text) { + try { + data = JSON.parse(text) + } catch { + data = {} + } + } + + return { + success: true, + output: { + budgetId: (data.budget_id as string) ?? params?.budgetId ?? '', + spendBudgetStatus: (data.spend_budget_status as string) ?? 'ARCHIVED', + }, + } + }, + + outputs: { + budgetId: { type: 'string', description: 'ID of the archived budget' }, + spendBudgetStatus: { + type: 'string', + description: 'Status of the budget after archiving', + optional: true, + }, + }, + } diff --git a/apps/sim/tools/brex/create_budget.ts b/apps/sim/tools/brex/create_budget.ts new file mode 100644 index 00000000000..113e84c4dc9 --- /dev/null +++ b/apps/sim/tools/brex/create_budget.ts @@ -0,0 +1,145 @@ +import { generateId } from '@sim/utils/id' +import type { BrexCreateBudgetParams, BrexCreateBudgetResponse } from '@/tools/brex/types' +import { BREX_MONEY_PROPERTIES } from '@/tools/brex/types' +import { BREX_API_BASE, buildBrexHeaders, parseBrexJson, splitBrexIdList } from '@/tools/brex/utils' +import type { ToolConfig } from '@/tools/types' + +export const brexCreateBudgetTool: ToolConfig = { + id: 'brex_create_budget', + name: 'Brex Create Budget', + description: 'Create a new budget in the Brex account', + version: '1.0.0', + + params: { + apiKey: { + type: 'string', + required: true, + visibility: 'user-only', + description: 'Brex user token (generated from Developer Settings in the Brex dashboard)', + }, + name: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'Name for the budget', + }, + description: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'Description of what the budget is used for', + }, + parentBudgetId: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'ID of the parent budget', + }, + periodRecurrenceType: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'Period type of the budget (WEEKLY, MONTHLY, QUARTERLY, YEARLY, ONE_TIME)', + }, + amount: { + type: 'number', + required: true, + visibility: 'user-or-llm', + description: 'Budget amount, in the smallest unit of the currency (e.g., cents for USD)', + }, + currency: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'ISO 4217 currency code (defaults to USD)', + }, + ownerUserIds: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'Comma-separated user IDs of the budget owners', + }, + startDate: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'Date the budget should start counting (YYYY-MM-DD)', + }, + endDate: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'Date the budget should stop counting (YYYY-MM-DD)', + }, + }, + + request: { + url: () => `${BREX_API_BASE}/v2/budgets`, + method: 'POST', + headers: (params) => ({ + ...buildBrexHeaders(params.apiKey), + 'Idempotency-Key': generateId(), + }), + body: (params) => { + const body: Record = { + name: params.name, + description: params.description, + parent_budget_id: params.parentBudgetId, + period_recurrence_type: params.periodRecurrenceType, + amount: { + amount: params.amount, + currency: params.currency || 'USD', + }, + } + const ownerUserIds = splitBrexIdList(params.ownerUserIds) + if (ownerUserIds) body.owner_user_ids = ownerUserIds + if (params.startDate) body.start_date = params.startDate + if (params.endDate) body.end_date = params.endDate + return body + }, + }, + + transformResponse: async (response) => { + const data = await parseBrexJson(response) + return { + success: true, + output: { + budgetId: data.budget_id ?? '', + accountId: data.account_id ?? '', + name: data.name ?? '', + description: data.description ?? null, + parentBudgetId: data.parent_budget_id ?? null, + ownerUserIds: data.owner_user_ids ?? [], + periodRecurrenceType: data.period_recurrence_type ?? '', + startDate: data.start_date ?? null, + endDate: data.end_date ?? null, + amount: data.amount ?? null, + spendBudgetStatus: data.spend_budget_status ?? '', + limitType: data.limit_type ?? null, + }, + } + }, + + outputs: { + budgetId: { type: 'string', description: 'Unique budget ID' }, + accountId: { type: 'string', description: 'Account ID the budget belongs to' }, + name: { type: 'string', description: 'Budget name' }, + description: { type: 'string', description: 'Budget description', optional: true }, + parentBudgetId: { type: 'string', description: 'Parent budget ID', optional: true }, + ownerUserIds: { type: 'array', description: 'User IDs of the budget owners' }, + periodRecurrenceType: { + type: 'string', + description: 'Budget period recurrence (WEEKLY, MONTHLY, QUARTERLY, YEARLY, ONE_TIME)', + }, + startDate: { type: 'string', description: 'Budget start date', optional: true }, + endDate: { type: 'string', description: 'Budget end date', optional: true }, + amount: { + type: 'json', + description: 'Budget amount', + optional: true, + properties: BREX_MONEY_PROPERTIES, + }, + spendBudgetStatus: { type: 'string', description: 'Status of the created budget' }, + limitType: { type: 'string', description: 'Budget limit type', optional: true }, + }, +} diff --git a/apps/sim/tools/brex/create_spend_limit.ts b/apps/sim/tools/brex/create_spend_limit.ts new file mode 100644 index 00000000000..fde9df62d41 --- /dev/null +++ b/apps/sim/tools/brex/create_spend_limit.ts @@ -0,0 +1,256 @@ +import { generateId } from '@sim/utils/id' +import type { BrexCreateSpendLimitParams, BrexCreateSpendLimitResponse } from '@/tools/brex/types' +import { BREX_SPEND_LIMIT_PERIOD_BALANCE_PROPERTIES } from '@/tools/brex/types' +import { BREX_API_BASE, buildBrexHeaders, parseBrexJson, splitBrexIdList } from '@/tools/brex/utils' +import type { ToolConfig } from '@/tools/types' + +export const brexCreateSpendLimitTool: ToolConfig< + BrexCreateSpendLimitParams, + BrexCreateSpendLimitResponse +> = { + id: 'brex_create_spend_limit', + name: 'Brex Create Spend Limit', + description: 'Create a new spend limit (hard-authorization card program) in the Brex account', + version: '1.0.0', + + params: { + apiKey: { + type: 'string', + required: true, + visibility: 'user-only', + description: 'Brex user token (generated from Developer Settings in the Brex dashboard)', + }, + name: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'Name for the spend limit', + }, + periodRecurrenceType: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: + 'Period type of the spend limit (PER_WEEK, PER_MONTH, PER_QUARTER, PER_YEAR, ONE_TIME)', + }, + spendType: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: + 'Whether the spend limit can only be spent from cards it provisions (BUDGET_PROVISIONED_CARDS_ONLY, NON_BUDGET_PROVISIONED_CARDS_ALLOWED)', + }, + expenseVisibility: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: + 'Whether expenses on this spend limit are viewable by all members (SHARED, PRIVATE)', + }, + authorizationVisibility: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: + 'Whether the limit amount is visible to all members, or just controllers/bookkeepers/owners (PUBLIC, PRIVATE)', + }, + limitIncreaseSetting: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'Whether members can request limit increases (ENABLED, DISABLED)', + }, + autoTransferCardsSetting: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: + 'How auto transfer works for virtual cards on this spend limit (DISABLED, ENABLED)', + }, + autoCreateLimitCardsSetting: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'How auto limit card creation works for members (DISABLED, ALL_MEMBERS)', + }, + expensePolicyId: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'ID of the expense policy corresponding to this spend limit', + }, + baseLimitAmount: { + type: 'number', + required: true, + visibility: 'user-or-llm', + description: + 'Base spend limit amount, without increases/rollovers, in the smallest unit of the currency (e.g., cents for USD)', + }, + currency: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'ISO 4217 currency code for the base limit (defaults to USD)', + }, + authorizationType: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'Whether authorizations decline based on available balance (HARD, SOFT)', + }, + rolloverRefreshRate: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: + 'Recurrence at which rolled-over unused funds stop rolling over (OFF, NEVER, PER_MONTH, PER_QUARTER, PER_YEAR)', + }, + limitBufferPercentage: { + type: 'number', + required: false, + visibility: 'user-or-llm', + description: 'Flexible buffer on the limit as a 0-100 percentage', + }, + description: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'Description of what the spend limit is used for', + }, + parentBudgetId: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'ID of the parent budget', + }, + startDate: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'Date the spend limit should start counting (YYYY-MM-DD)', + }, + endDate: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'Date the spend limit should expire (YYYY-MM-DD)', + }, + transactionLimitAmount: { + type: 'number', + required: false, + visibility: 'user-or-llm', + description: + 'Per-transaction limit this spend limit enforces, in the smallest unit of the currency', + }, + ownerUserIds: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'Comma-separated user IDs of the spend limit owners', + }, + memberUserIds: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'Comma-separated user IDs of the spend limit members', + }, + }, + + request: { + url: () => `${BREX_API_BASE}/v2/spend_limits`, + method: 'POST', + headers: (params) => ({ + ...buildBrexHeaders(params.apiKey), + 'Idempotency-Key': generateId(), + }), + body: (params) => { + const currency = params.currency || 'USD' + const body: Record = { + name: params.name, + period_recurrence_type: params.periodRecurrenceType, + spend_type: params.spendType, + expense_visibility: params.expenseVisibility, + authorization_visibility: params.authorizationVisibility, + limit_increase_setting: params.limitIncreaseSetting, + auto_transfer_cards_setting: params.autoTransferCardsSetting, + auto_create_limit_cards_setting: params.autoCreateLimitCardsSetting, + expense_policy_id: params.expensePolicyId, + authorization_settings: { + base_limit: { + amount: params.baseLimitAmount, + currency, + }, + authorization_type: params.authorizationType, + rollover_refresh_rate: params.rolloverRefreshRate, + ...(params.limitBufferPercentage !== undefined + ? { limit_buffer_percentage: params.limitBufferPercentage } + : {}), + }, + } + if (params.description) body.description = params.description + if (params.parentBudgetId) body.parent_budget_id = params.parentBudgetId + if (params.startDate) body.start_date = params.startDate + if (params.endDate) body.end_date = params.endDate + if (params.transactionLimitAmount !== undefined) { + body.transaction_limit = { amount: params.transactionLimitAmount, currency } + } + const ownerUserIds = splitBrexIdList(params.ownerUserIds) + if (ownerUserIds) body.owner_user_ids = ownerUserIds + const memberUserIds = splitBrexIdList(params.memberUserIds) + if (memberUserIds) body.member_user_ids = memberUserIds + return body + }, + }, + + transformResponse: async (response) => { + const data = await parseBrexJson(response) + return { + success: true, + output: { + id: data.id ?? '', + accountId: data.account_id ?? '', + name: data.name ?? '', + description: data.description ?? null, + parentBudgetId: data.parent_budget_id ?? null, + status: data.status ?? '', + periodRecurrenceType: data.period_recurrence_type ?? '', + spendType: data.spend_type ?? '', + startDate: data.start_date ?? null, + endDate: data.end_date ?? null, + ownerUserIds: data.owner_user_ids ?? [], + memberUserIds: data.member_user_ids ?? [], + currentPeriodBalance: data.current_period_balance ?? null, + authorizationSettings: data.authorization_settings ?? null, + }, + } + }, + + outputs: { + id: { type: 'string', description: 'Unique spend limit ID' }, + accountId: { type: 'string', description: 'Account ID the spend limit belongs to' }, + name: { type: 'string', description: 'Spend limit name' }, + description: { type: 'string', description: 'Spend limit description', optional: true }, + parentBudgetId: { type: 'string', description: 'Parent budget ID', optional: true }, + status: { type: 'string', description: 'Spend limit status' }, + periodRecurrenceType: { + type: 'string', + description: 'Period recurrence (PER_WEEK, PER_MONTH, PER_QUARTER, PER_YEAR, ONE_TIME)', + }, + spendType: { type: 'string', description: 'Spend type of the limit' }, + startDate: { type: 'string', description: 'Spend limit start date', optional: true }, + endDate: { type: 'string', description: 'Spend limit end date', optional: true }, + ownerUserIds: { type: 'array', description: 'User IDs of the spend limit owners' }, + memberUserIds: { type: 'array', description: 'User IDs of the spend limit members' }, + currentPeriodBalance: { + type: 'json', + description: 'Spend and rollover amounts for the current period', + optional: true, + properties: BREX_SPEND_LIMIT_PERIOD_BALANCE_PROPERTIES, + }, + authorizationSettings: { + type: 'json', + description: 'Authorization settings (base limit, authorization type, rollover refresh)', + optional: true, + }, + }, +} diff --git a/apps/sim/tools/brex/create_transfer.ts b/apps/sim/tools/brex/create_transfer.ts new file mode 100644 index 00000000000..5c4dc7a9689 --- /dev/null +++ b/apps/sim/tools/brex/create_transfer.ts @@ -0,0 +1,189 @@ +import { generateId } from '@sim/utils/id' +import type { BrexCreateTransferParams, BrexCreateTransferResponse } from '@/tools/brex/types' +import { BREX_MONEY_PROPERTIES } from '@/tools/brex/types' +import { BREX_API_BASE, buildBrexHeaders, parseBrexJson } from '@/tools/brex/utils' +import type { ToolConfig } from '@/tools/types' + +export const brexCreateTransferTool: ToolConfig< + BrexCreateTransferParams, + BrexCreateTransferResponse +> = { + id: 'brex_create_transfer', + name: 'Brex Create Transfer', + description: 'Create a money transfer from a Brex cash account to a vendor', + version: '1.0.0', + + params: { + apiKey: { + type: 'string', + required: true, + visibility: 'user-only', + description: 'Brex user token (generated from Developer Settings in the Brex dashboard)', + }, + cashAccountId: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: + 'ID of the Brex cash account to send the transfer from (found via the /accounts endpoint)', + }, + vendorPaymentInstrumentId: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: + "ID of the vendor's payment instrument to send the transfer to (from the vendor's payment_accounts)", + }, + amount: { + type: 'number', + required: true, + visibility: 'user-or-llm', + description: 'Amount to transfer, in the smallest unit of the currency (e.g., cents for USD)', + }, + currency: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'ISO 4217 currency code (defaults to USD)', + }, + description: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'Description of the transfer for internal use (not exposed externally)', + }, + externalMemo: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: + 'External memo shown to the recipient (max 90 characters for ACH/Wire, 40 for Cheque)', + }, + approvalType: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'Set to MANUAL to require cash admin approval before the transfer is sent', + }, + isPproEnabled: { + type: 'boolean', + required: false, + visibility: 'user-or-llm', + description: + 'Enable Principal Protection (PPRO) to have Brex cover intermediary/receiving bank fees (international wires only)', + }, + }, + + request: { + url: () => `${BREX_API_BASE}/v1/transfers`, + method: 'POST', + headers: (params) => ({ + ...buildBrexHeaders(params.apiKey), + // Brex requires a fresh Idempotency-Key per transfer creation to prevent duplicate money movement. + 'Idempotency-Key': generateId(), + }), + body: (params) => { + const body: Record = { + counterparty: { + type: 'VENDOR', + payment_instrument_id: params.vendorPaymentInstrumentId, + }, + amount: { + amount: params.amount, + currency: params.currency || 'USD', + }, + description: params.description, + external_memo: params.externalMemo, + originating_account: { + type: 'BREX_CASH', + id: params.cashAccountId, + }, + } + if (params.approvalType) body.approval_type = params.approvalType + if (params.isPproEnabled !== undefined) body.is_ppro_enabled = params.isPproEnabled + return body + }, + }, + + transformResponse: async (response) => { + const data = await parseBrexJson(response) + return { + success: true, + output: { + id: data.id ?? '', + counterparty: data.counterparty ?? null, + description: data.description ?? null, + paymentType: data.payment_type ?? '', + amount: data.amount ?? null, + processDate: data.process_date ?? null, + originatingAccount: data.originating_account ?? null, + status: data.status ?? '', + cancellationReason: data.cancellation_reason ?? null, + estimatedDeliveryDate: data.estimated_delivery_date ?? null, + creatorUserId: data.creator_user_id ?? null, + createdAt: data.created_at ?? null, + displayName: data.display_name ?? null, + externalMemo: data.external_memo ?? null, + isPproEnabled: data.is_ppro_enabled ?? null, + }, + } + }, + + outputs: { + id: { type: 'string', description: 'Unique transfer ID' }, + counterparty: { type: 'json', description: 'Transfer counterparty details', optional: true }, + description: { type: 'string', description: 'Description of the transfer', optional: true }, + paymentType: { + type: 'string', + description: + 'Payment type (ACH, DOMESTIC_WIRE, CHEQUE, INTERNATIONAL_WIRE, BOOK_TRANSFER, STABLECOIN)', + }, + amount: { + type: 'json', + description: 'Transfer amount', + optional: true, + properties: BREX_MONEY_PROPERTIES, + }, + processDate: { type: 'string', description: 'Transaction processing date', optional: true }, + originatingAccount: { + type: 'json', + description: 'Originating account details for the transfer', + optional: true, + }, + status: { + type: 'string', + description: 'Transfer status (PROCESSING, SCHEDULED, PENDING_APPROVAL, FAILED, PROCESSED)', + }, + cancellationReason: { + type: 'string', + description: 'Reason the transfer was canceled', + optional: true, + }, + estimatedDeliveryDate: { + type: 'string', + description: 'Estimated delivery date for the transfer', + optional: true, + }, + creatorUserId: { + type: 'string', + description: 'ID of the user who created the transfer', + optional: true, + }, + createdAt: { + type: 'string', + description: 'Creation timestamp of the transfer', + optional: true, + }, + displayName: { + type: 'string', + description: 'Human-readable name of the transfer', + optional: true, + }, + externalMemo: { type: 'string', description: 'External memo of the transfer', optional: true }, + isPproEnabled: { + type: 'boolean', + description: 'Whether Principal Protection (PPRO) is enabled for the transfer', + optional: true, + }, + }, +} diff --git a/apps/sim/tools/brex/create_vendor.ts b/apps/sim/tools/brex/create_vendor.ts new file mode 100644 index 00000000000..9b485eddba1 --- /dev/null +++ b/apps/sim/tools/brex/create_vendor.ts @@ -0,0 +1,75 @@ +import { generateId } from '@sim/utils/id' +import type { BrexCreateVendorParams, BrexCreateVendorResponse } from '@/tools/brex/types' +import { BREX_API_BASE, buildBrexHeaders, parseBrexJson } from '@/tools/brex/utils' +import type { ToolConfig } from '@/tools/types' + +export const brexCreateVendorTool: ToolConfig = { + id: 'brex_create_vendor', + name: 'Brex Create Vendor', + description: 'Create a new vendor in the Brex account', + version: '1.0.0', + + params: { + apiKey: { + type: 'string', + required: true, + visibility: 'user-only', + description: 'Brex user token (generated from Developer Settings in the Brex dashboard)', + }, + companyName: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'Name for the vendor (must be unique)', + }, + email: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'Email address for the vendor', + }, + phone: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'Phone number for the vendor', + }, + }, + + request: { + url: () => `${BREX_API_BASE}/v1/vendors`, + method: 'POST', + headers: (params) => ({ + ...buildBrexHeaders(params.apiKey), + 'Idempotency-Key': generateId(), + }), + body: (params) => { + const body: Record = { company_name: params.companyName } + if (params.email) body.email = params.email + if (params.phone) body.phone = params.phone + return body + }, + }, + + transformResponse: async (response) => { + const data = await parseBrexJson(response) + return { + success: true, + output: { + id: data.id ?? '', + companyName: data.company_name ?? null, + email: data.email ?? null, + phone: data.phone ?? null, + paymentAccounts: data.payment_accounts ?? [], + }, + } + }, + + outputs: { + id: { type: 'string', description: 'Unique vendor ID' }, + companyName: { type: 'string', description: 'Vendor company name', optional: true }, + email: { type: 'string', description: 'Vendor email address', optional: true }, + phone: { type: 'string', description: 'Vendor phone number', optional: true }, + paymentAccounts: { type: 'array', description: 'Payment accounts associated with the vendor' }, + }, +} diff --git a/apps/sim/tools/brex/get_budget.ts b/apps/sim/tools/brex/get_budget.ts index 9496eac14cc..d00bebd1b48 100644 --- a/apps/sim/tools/brex/get_budget.ts +++ b/apps/sim/tools/brex/get_budget.ts @@ -72,7 +72,7 @@ export const brexGetBudgetTool: ToolConfig | null + description: string | null + paymentType: string + amount: BrexMoney | null + processDate: string | null + originatingAccount: Record | null + status: string + cancellationReason: string | null + estimatedDeliveryDate: string | null + creatorUserId: string | null + createdAt: string | null + displayName: string | null + externalMemo: string | null + isPproEnabled: boolean | null + } +} + +export interface BrexCreateBudgetResponse extends ToolResponse { + output: { + budgetId: string + accountId: string + name: string + description: string | null + parentBudgetId: string | null + ownerUserIds: string[] + periodRecurrenceType: string + startDate: string | null + endDate: string | null + amount: BrexMoney | null + spendBudgetStatus: string + limitType: string | null + } +} + +export interface BrexArchiveBudgetResponse extends ToolResponse { + output: { + budgetId: string + spendBudgetStatus: string | null + } +} + +export interface BrexCreateSpendLimitResponse extends ToolResponse { + output: { + id: string + accountId: string + name: string + description: string | null + parentBudgetId: string | null + status: string + periodRecurrenceType: string + spendType: string + startDate: string | null + endDate: string | null + ownerUserIds: string[] + memberUserIds: string[] + currentPeriodBalance: BrexSpendLimitPeriodBalance | null + authorizationSettings: Record | null + } +} + +export interface BrexCreateVendorResponse extends ToolResponse { + output: { + id: string + companyName: string | null + email: string | null + phone: string | null + paymentAccounts: unknown[] + } +} + +export interface BrexUpdateVendorResponse extends ToolResponse { + output: { + id: string + companyName: string | null + email: string | null + phone: string | null + paymentAccounts: unknown[] + } +} + export type BrexResponse = | BrexListExpensesResponse | BrexGetExpenseResponse @@ -585,6 +738,12 @@ export type BrexResponse = | BrexGetSpendLimitResponse | BrexGetVendorResponse | BrexGetTransferResponse + | BrexCreateTransferResponse + | BrexCreateBudgetResponse + | BrexArchiveBudgetResponse + | BrexCreateSpendLimitResponse + | BrexCreateVendorResponse + | BrexUpdateVendorResponse export const BREX_MONEY_PROPERTIES: Record = { amount: { diff --git a/apps/sim/tools/brex/update_vendor.ts b/apps/sim/tools/brex/update_vendor.ts new file mode 100644 index 00000000000..713e4db7fc8 --- /dev/null +++ b/apps/sim/tools/brex/update_vendor.ts @@ -0,0 +1,88 @@ +import { generateId } from '@sim/utils/id' +import type { BrexUpdateVendorParams, BrexUpdateVendorResponse } from '@/tools/brex/types' +import { BREX_API_BASE, buildBrexHeaders, parseBrexJson } from '@/tools/brex/utils' +import type { ToolConfig } from '@/tools/types' + +export const brexUpdateVendorTool: ToolConfig = { + id: 'brex_update_vendor', + name: 'Brex Update Vendor', + description: 'Update an existing vendor in the Brex account', + version: '1.0.0', + + params: { + apiKey: { + type: 'string', + required: true, + visibility: 'user-only', + description: 'Brex user token (generated from Developer Settings in the Brex dashboard)', + }, + vendorId: { + type: 'string', + required: true, + visibility: 'user-or-llm', + description: 'ID of the vendor to update', + }, + companyName: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'New name for the vendor', + }, + email: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'New email address for the vendor', + }, + phone: { + type: 'string', + required: false, + visibility: 'user-or-llm', + description: 'New phone number for the vendor', + }, + }, + + request: { + url: (params) => `${BREX_API_BASE}/v1/vendors/${encodeURIComponent(params.vendorId.trim())}`, + method: 'PUT', + headers: (params) => ({ + ...buildBrexHeaders(params.apiKey), + // Optional per Brex's spec for this endpoint, but included for safe-retry semantics. + 'Idempotency-Key': generateId(), + }), + body: (params) => { + const body: Record = {} + if (params.companyName) body.company_name = params.companyName + if (params.email) body.email = params.email + if (params.phone) body.phone = params.phone + if (Object.keys(body).length === 0) { + throw new Error( + 'At least one of company name, email, or phone must be provided to update the vendor' + ) + } + return body + }, + }, + + transformResponse: async (response) => { + const data = await parseBrexJson(response) + return { + success: true, + output: { + id: data.id ?? '', + companyName: data.company_name ?? null, + email: data.email ?? null, + phone: data.phone ?? null, + paymentAccounts: data.payment_accounts ?? [], + }, + } + }, + + outputs: { + id: { type: 'string', description: 'Unique vendor ID' }, + companyName: { type: 'string', description: 'Vendor company name', optional: true }, + email: { type: 'string', description: 'Vendor email address', optional: true }, + phone: { type: 'string', description: 'Vendor phone number', optional: true }, + paymentAccounts: { type: 'array', description: 'Payment accounts associated with the vendor' }, + }, +} diff --git a/apps/sim/tools/brex/utils.ts b/apps/sim/tools/brex/utils.ts index 19edd9bb79b..2530557f231 100644 --- a/apps/sim/tools/brex/utils.ts +++ b/apps/sim/tools/brex/utils.ts @@ -51,6 +51,19 @@ export function appendBrexPagination( if (params.limit) query.append('limit', params.limit) } +/** + * Splits a comma-separated string of IDs into a trimmed, non-empty array for + * use in a JSON request body (as opposed to repeated query parameters). + */ +export function splitBrexIdList(value?: string): string[] | undefined { + if (!value) return undefined + const ids = value + .split(',') + .map((id) => id.trim()) + .filter(Boolean) + return ids.length > 0 ? ids : undefined +} + /** * Converts a timestamp to the timezone-less date-time form the Brex Transactions * API requires (e.g., 2026-01-01T00:00:00). Brex rejects timezone-suffixed diff --git a/apps/sim/tools/registry.ts b/apps/sim/tools/registry.ts index 4c2ce2bf0d9..e9423a2837b 100644 --- a/apps/sim/tools/registry.ts +++ b/apps/sim/tools/registry.ts @@ -345,6 +345,11 @@ import { } from '@/tools/box_sign' import { brandfetchGetBrandTool, brandfetchSearchTool } from '@/tools/brandfetch' import { + brexArchiveBudgetTool, + brexCreateBudgetTool, + brexCreateSpendLimitTool, + brexCreateTransferTool, + brexCreateVendorTool, brexGetBudgetTool, brexGetCashAccountTool, brexGetCompanyTool, @@ -372,6 +377,7 @@ import { brexListVendorsTool, brexMatchReceiptTool, brexUpdateExpenseTool, + brexUpdateVendorTool, brexUploadReceiptTool, } from '@/tools/brex' import { @@ -4632,6 +4638,11 @@ export const tools: Record = { athena_stop_query: athenaStopQueryTool, brandfetch_get_brand: brandfetchGetBrandTool, brandfetch_search: brandfetchSearchTool, + brex_archive_budget: brexArchiveBudgetTool, + brex_create_budget: brexCreateBudgetTool, + brex_create_spend_limit: brexCreateSpendLimitTool, + brex_create_transfer: brexCreateTransferTool, + brex_create_vendor: brexCreateVendorTool, brex_get_budget: brexGetBudgetTool, brex_get_cash_account: brexGetCashAccountTool, brex_get_company: brexGetCompanyTool, @@ -4659,6 +4670,7 @@ export const tools: Record = { brex_list_vendors: brexListVendorsTool, brex_match_receipt: brexMatchReceiptTool, brex_update_expense: brexUpdateExpenseTool, + brex_update_vendor: brexUpdateVendorTool, brex_upload_receipt: brexUploadReceiptTool, brightdata_cancel_snapshot: brightDataCancelSnapshotTool, brightdata_discover: brightDataDiscoverTool, diff --git a/helm/sim/README.md b/helm/sim/README.md index 355aa5109c9..affa22fc2c7 100644 --- a/helm/sim/README.md +++ b/helm/sim/README.md @@ -219,7 +219,15 @@ Before installing in production, confirm each of the following: * **Secrets management** — provide secrets via External Secrets Operator (ESO) or pre-created Kubernetes Secrets. Never commit secrets to `values.yaml`. * **TLS / Ingress** — set the `cert-manager.io/cluster-issuer` annotation on the ingress and tune `proxy-body-size` / `proxy-read-timeout` for your workload. See commented examples in `values.yaml`. * **Network policy egress** — review `networkPolicy.egressExceptCidrs`. Defaults block cloud metadata endpoints (`169.254.169.254/32`, `169.254.170.2/32`); add your cluster's API server CIDR for stronger isolation. Custom egress rules go in `networkPolicy.egress` (a list). -* **Namespace hardening** — label the install namespace with Pod Security Standards `restricted` enforcement (`pod-security.kubernetes.io/enforce=restricted`). +* **Network policy ingress** — `networkPolicy.ingressFrom` defaults to `[{}]` (an empty peer selector), which allows ingress traffic from **any pod in the cluster**, not just your ingress controller. This is a deliberate simple default, not a locked-down one. On a shared or multi-tenant cluster, scope it down, e.g. to the ingress-nginx namespace: + ```yaml + networkPolicy: + ingressFrom: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: ingress-nginx + ``` +* **Namespace hardening** — label the install namespace with Pod Security Standards `restricted` enforcement (`pod-security.kubernetes.io/enforce=restricted`). All workloads set `runAsNonRoot`, drop all Linux capabilities, disable privilege escalation, and set `seccompProfile: RuntimeDefault` — the four controls the Restricted profile requires. `readOnlyRootFilesystem` is intentionally **not** defaulted anywhere (Postgres/Ollama genuinely need a writable root; the stateless services — `realtime`, `pii`, `copilot` — could tolerate it but aren't pre-wired with a `/tmp` `emptyDir`). If your policy requires it, set `.securityContext.readOnlyRootFilesystem: true` and mount an `emptyDir` at `/tmp` yourself via `extraVolumes`/`extraVolumeMounts`. * **Env validation** — keys under `app.env`, `realtime.env`, and `copilot.env` are passed through to the application and validated at startup. The JSON Schema intentionally does not enforce `additionalProperties: false` (would break custom user envs), so typos like `OPENA_API_KEY` (instead of `OPENAI_API_KEY`) surface as missing-key errors at runtime, not at `helm install` time. Review your env block carefully. * **Set public URLs** — `app.env.NEXT_PUBLIC_APP_URL` and `app.env.BETTER_AUTH_URL` must match your public origin (e.g. `https://sim.example.com`). Leaving them as `localhost` breaks sign-in. @@ -287,6 +295,17 @@ externalSecrets: INTERNAL_API_SECRET: sim/app/internal-api-secret postgresql: password: sim/postgresql/password + # Only needed when copilot.enabled=true and copilot.server.secret.create=true. + # Every non-empty copilot.server.env key must have a matching entry here — + # template rendering fails with a clear message naming the missing key otherwise. + copilot: + AGENT_API_DB_ENCRYPTION_KEY: sim/copilot/agent-api-db-encryption-key + INTERNAL_API_SECRET: sim/copilot/internal-api-secret + LICENSE_KEY: sim/copilot/license-key + SIM_BASE_URL: sim/copilot/sim-base-url + SIM_AGENT_API_KEY: sim/copilot/sim-agent-api-key + REDIS_URL: sim/copilot/redis-url + OPENAI_API_KEY_1: sim/copilot/openai-api-key ``` See `examples/values-external-secrets.yaml`. diff --git a/helm/sim/examples/values-azure.yaml b/helm/sim/examples/values-azure.yaml index 9de564a8923..2404088b221 100644 --- a/helm/sim/examples/values-azure.yaml +++ b/helm/sim/examples/values-azure.yaml @@ -98,6 +98,16 @@ app: AZURE_ANTHROPIC_API_KEY: "" # Azure Anthropic API key AZURE_ANTHROPIC_API_VERSION: "" # Azure Anthropic API version (e.g., 2023-06-01) NEXT_PUBLIC_AZURE_CONFIGURED: "true" # Set to "true" once credentials are configured above + KB_OPENAI_MODEL_NAME: "" # Azure deployment name serving the KB embedding model (used only when AZURE_OPENAI_* is set) + WAND_OPENAI_MODEL_NAME: "" # Wand generation model deployment name + + # Azure Mistral OCR (optional — Azure-hosted document OCR for knowledge base processing) + OCR_AZURE_ENDPOINT: "" # Azure Mistral OCR service endpoint + OCR_AZURE_MODEL_NAME: "" # Azure Mistral OCR model name + OCR_AZURE_API_KEY: "" # Azure Mistral OCR API key + + # Email — Azure Communication Services (recommended on Azure; alternative to Resend/SES/SMTP) + AZURE_ACS_CONNECTION_STRING: "" # Azure Communication Services connection string # Azure Blob Storage Configuration (RECOMMENDED for production) # Create a storage account and containers in your Azure subscription @@ -308,5 +318,13 @@ ingressInternal: # AZURE_ACCOUNT_KEY: "sim-azure-storage-account-key" # # Use AZURE_CONNECTION_STRING instead if you prefer a single connection string: # # AZURE_CONNECTION_STRING: "sim-azure-storage-connection-string" +# OCR_AZURE_API_KEY: "sim-azure-ocr-api-key" +# AZURE_ACS_CONNECTION_STRING: "sim-azure-acs-connection-string" +# # Non-secret config (deployment names, endpoints) — still required here: ESO mode requires +# # every non-empty app.env key to be mapped, even ones that aren't sensitive. +# KB_OPENAI_MODEL_NAME: "sim-kb-openai-model-name" +# WAND_OPENAI_MODEL_NAME: "sim-wand-openai-model-name" +# OCR_AZURE_ENDPOINT: "sim-azure-ocr-endpoint" +# OCR_AZURE_MODEL_NAME: "sim-azure-ocr-model-name" # postgresql: # password: "sim-postgresql-password" diff --git a/helm/sim/examples/values-copilot.yaml b/helm/sim/examples/values-copilot.yaml index 62fbb87b1a6..d95e1aa2e91 100644 --- a/helm/sim/examples/values-copilot.yaml +++ b/helm/sim/examples/values-copilot.yaml @@ -60,9 +60,6 @@ copilot: # Required secrets (set via values or provide your own secret) env: - PORT: "8080" - SERVICE_NAME: "copilot" - ENVIRONMENT: "production" AGENT_API_DB_ENCRYPTION_KEY: "" # openssl rand -hex 32 INTERNAL_API_SECRET: "" # reuse Sim INTERNAL_API_SECRET LICENSE_KEY: "" # Provided by Sim team @@ -72,9 +69,16 @@ copilot: SIM_AGENT_API_KEY: "" # Must match SIM-side COPILOT_API_KEY REDIS_URL: "redis://default:password@redis:6379" # Optional configuration - LOG_LEVEL: "info" CORS_ALLOWED_ORIGINS: "https://sim.example.com" OTEL_EXPORTER_OTLP_ENDPOINT: "" + + # Non-secret operational tunables — inlined as plain container env, never go through + # the Secret/ExternalSecret, so they don't need externalSecrets.remoteRefs.copilot entries + envDefaults: + PORT: "8080" + SERVICE_NAME: "copilot" + ENVIRONMENT: "production" + LOG_LEVEL: "info" # Create a Secret from the values above. Set create=false to reference an existing secret instead. secret: diff --git a/helm/sim/examples/values-external-secrets.yaml b/helm/sim/examples/values-external-secrets.yaml index 1ba38e1b960..23aa6c5cfbc 100644 --- a/helm/sim/examples/values-external-secrets.yaml +++ b/helm/sim/examples/values-external-secrets.yaml @@ -36,6 +36,16 @@ externalSecrets: API_ENCRYPTION_KEY: "sim/app/api-encryption-key" postgresql: password: "sim/postgresql/password" + # Only needed when copilot.enabled=true and copilot.server.secret.create=true + # (uncomment and pre-populate the referenced paths in your secret store): + # copilot: + # AGENT_API_DB_ENCRYPTION_KEY: "sim/copilot/agent-api-db-encryption-key" + # INTERNAL_API_SECRET: "sim/copilot/internal-api-secret" + # LICENSE_KEY: "sim/copilot/license-key" + # SIM_BASE_URL: "sim/copilot/sim-base-url" + # SIM_AGENT_API_KEY: "sim/copilot/sim-agent-api-key" + # REDIS_URL: "sim/copilot/redis-url" + # OPENAI_API_KEY_1: "sim/copilot/openai-api-key" app: enabled: true diff --git a/helm/sim/templates/_helpers.tpl b/helm/sim/templates/_helpers.tpl index 627b49b57a2..32d917a33b6 100644 --- a/helm/sim/templates/_helpers.tpl +++ b/helm/sim/templates/_helpers.tpl @@ -141,6 +141,38 @@ Migrations specific labels app.kubernetes.io/component: migrations {{- end }} +{{/* +Copilot specific labels +*/}} +{{- define "sim.copilot.labels" -}} +{{ include "sim.labels" . }} +app.kubernetes.io/component: copilot +{{- end }} + +{{/* +Copilot selector labels +*/}} +{{- define "sim.copilot.selectorLabels" -}} +{{ include "sim.selectorLabels" . }} +app.kubernetes.io/component: copilot +{{- end }} + +{{/* +Copilot PostgreSQL specific labels +*/}} +{{- define "sim.copilotPostgresql.labels" -}} +{{ include "sim.labels" . }} +app.kubernetes.io/component: copilot-postgresql +{{- end }} + +{{/* +Copilot PostgreSQL selector labels +*/}} +{{- define "sim.copilotPostgresql.selectorLabels" -}} +{{ include "sim.selectorLabels" . }} +app.kubernetes.io/component: copilot-postgresql +{{- end }} + {{/* Create the name of the service account to use */}} @@ -609,16 +641,33 @@ Validate Copilot configuration {{- end -}} {{- if .Values.copilot.server.secret.create -}} {{- $env := .Values.copilot.server.env -}} + {{- $useExternalSecrets := and .Values.externalSecrets .Values.externalSecrets.enabled -}} + {{- $remoteRefs := default (dict) (default (dict) .Values.externalSecrets.remoteRefs).copilot -}} {{- $required := list "AGENT_API_DB_ENCRYPTION_KEY" "INTERNAL_API_SECRET" "LICENSE_KEY" "SIM_BASE_URL" "SIM_AGENT_API_KEY" "REDIS_URL" -}} {{- range $key := $required -}} - {{- if not (and $env (index $env $key) (ne (index $env $key) "")) -}} - {{- fail (printf "copilot.server.env.%s is required when copilot is enabled" $key) -}} + {{- $inEnv := and $env (index $env $key) (ne (index $env $key) "") -}} + {{- $mapped := index $remoteRefs $key -}} + {{- if not (or $inEnv (and $useExternalSecrets $mapped)) -}} + {{- if $useExternalSecrets -}} + {{- fail (printf "Required key '%s' is missing: externalSecrets.enabled=true but the key is neither set in copilot.server.env nor mapped in externalSecrets.remoteRefs.copilot. Map it via externalSecrets.remoteRefs.copilot.%s='path/in/store' so it is synced into the copilot Secret." $key $key) -}} + {{- else -}} + {{- fail (printf "copilot.server.env.%s is required when copilot is enabled" $key) -}} + {{- end -}} + {{- end -}} + {{- end -}} + {{- if $useExternalSecrets -}} + {{- range $key, $value := $env -}} + {{- if and (ne (toString $value) "") (ne (toString $value) "") -}} + {{- if not (index $remoteRefs $key) -}} + {{- fail (printf "Key '%s' is set in copilot.server.env but externalSecrets.enabled=true and externalSecrets.remoteRefs.copilot.%s is not configured. When ESO is enabled the chart-managed copilot Secret is not rendered, so the container would start with no value. Either map it via externalSecrets.remoteRefs.copilot.%s='path/in/store' or remove it from copilot.server.env." $key $key $key) -}} + {{- end -}} + {{- end -}} {{- end -}} {{- end -}} - {{- $hasOpenAI := and $env (ne (default "" (index $env "OPENAI_API_KEY_1")) "") -}} - {{- $hasAnthropic := and $env (ne (default "" (index $env "ANTHROPIC_API_KEY_1")) "") -}} + {{- $hasOpenAI := or (and $env (ne (default "" (index $env "OPENAI_API_KEY_1")) "")) (and $useExternalSecrets (index $remoteRefs "OPENAI_API_KEY_1")) -}} + {{- $hasAnthropic := or (and $env (ne (default "" (index $env "ANTHROPIC_API_KEY_1")) "")) (and $useExternalSecrets (index $remoteRefs "ANTHROPIC_API_KEY_1")) -}} {{- if not (or $hasOpenAI $hasAnthropic) -}} - {{- fail "Set at least one of copilot.server.env.OPENAI_API_KEY_1 or copilot.server.env.ANTHROPIC_API_KEY_1" -}} + {{- fail "Set at least one of copilot.server.env.OPENAI_API_KEY_1 or copilot.server.env.ANTHROPIC_API_KEY_1 (or map one via externalSecrets.remoteRefs.copilot when externalSecrets.enabled=true)" -}} {{- end -}} {{- end -}} {{- if .Values.copilot.postgresql.enabled -}} diff --git a/helm/sim/templates/deployment-copilot.yaml b/helm/sim/templates/deployment-copilot.yaml index 6bab3f27ec2..985a323433e 100644 --- a/helm/sim/templates/deployment-copilot.yaml +++ b/helm/sim/templates/deployment-copilot.yaml @@ -6,8 +6,7 @@ metadata: name: {{ include "sim.fullname" . }}-copilot namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.labels" . | nindent 4 }} spec: type: {{ .Values.copilot.server.service.type }} ports: @@ -16,9 +15,7 @@ spec: protocol: TCP name: http selector: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.selectorLabels" . | nindent 4 }} --- apiVersion: apps/v1 kind: Deployment @@ -26,26 +23,30 @@ metadata: name: {{ include "sim.fullname" . }}-copilot namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.labels" . | nindent 4 }} spec: replicas: {{ .Values.copilot.server.replicaCount }} selector: matchLabels: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.selectorLabels" . | nindent 6 }} template: metadata: annotations: - checksum/secret: {{ include (print $.Template.BasePath "/secrets-copilot.yaml") . | sha256sum }} + {{- /* + Hash both the inline Secret and the ExternalSecret template — whichever mode is + active, only one renders non-empty content, but concatenating both means a mode + switch or a remoteRefs.copilot mapping change still changes the checksum and + forces a rollout. This can't see the live value ESO syncs from the external + store (Helm only has the ExternalSecret manifest at render time, not the + store's payload) — that's an inherent ESO limitation, not something a Helm + checksum can close; ESO's own reconciliation handles picking up rotated values. + */}} + checksum/secret: {{ printf "%s%s" (include (print $.Template.BasePath "/secrets-copilot.yaml") .) (include (print $.Template.BasePath "/external-secret-copilot.yaml") .) | sha256sum }} {{- with .Values.podAnnotations }} {{- toYaml . | nindent 8 }} {{- end }} labels: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.selectorLabels" . | nindent 8 }} {{- with .Values.podLabels }} {{- toYaml . | nindent 8 }} {{- end }} @@ -85,9 +86,35 @@ spec: {{- with .Values.copilot.server.extraEnvFrom }} {{- toYaml . | nindent 12 }} {{- end }} - {{- with .Values.copilot.server.extraEnv }} + {{- /* + copilot.server.env flows through envFrom (the Secret/ExternalSecret) above. + copilot.server.envDefaults are inlined here so they don't leak into the + chart-managed Secret and don't need to be mapped when + externalSecrets.enabled=true — same rationale as app.envDefaults. Skipped + entirely in existingSecret mode: the pre-created Secret is the source of + truth, and inlining defaults would silently shadow keys (PORT, LOG_LEVEL, + etc.) the user stored there via envFrom, since explicit env wins over + envFrom. Keyed purely on copilot.server.secret.create, NOT the global + externalSecrets.enabled flag — envFrom always points at the user-provided + Secret name whenever secret.create=false, regardless of whether ESO is used + for other components, so the global flag is irrelevant here. + */}} + {{- $useExistingSecret := not .Values.copilot.server.secret.create }} + {{- if or (not $useExistingSecret) .Values.copilot.server.extraEnv }} env: + {{- $copilotEnv := .Values.copilot.server.env | default dict }} + {{- if not $useExistingSecret }} + {{- range $key, $value := .Values.copilot.server.envDefaults | default dict }} + {{- $override := index $copilotEnv $key }} + {{- if and (ne (toString $value) "") (ne (toString $value) "") (or (not $override) (eq (toString $override) "") (eq (toString $override) "")) }} + - name: {{ $key }} + value: {{ $value | quote }} + {{- end }} + {{- end }} + {{- end }} + {{- with .Values.copilot.server.extraEnv }} {{- toYaml . | nindent 12 }} + {{- end }} {{- end }} {{- if .Values.copilot.server.startupProbe }} startupProbe: @@ -106,5 +133,23 @@ spec: {{- toYaml . | nindent 12 }} {{- end }} {{- include "sim.containerSecurityContext" .Values.copilot.server | nindent 10 }} + {{- if or .Values.extraVolumeMounts .Values.copilot.server.extraVolumeMounts }} + volumeMounts: + {{- with .Values.extraVolumeMounts }} + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.copilot.server.extraVolumeMounts }} + {{- toYaml . | nindent 12 }} + {{- end }} + {{- end }} + {{- if or .Values.extraVolumes .Values.copilot.server.extraVolumes }} + volumes: + {{- with .Values.extraVolumes }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.copilot.server.extraVolumes }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} {{- end }} diff --git a/helm/sim/templates/external-secret-copilot.yaml b/helm/sim/templates/external-secret-copilot.yaml new file mode 100644 index 00000000000..a7b78da0310 --- /dev/null +++ b/helm/sim/templates/external-secret-copilot.yaml @@ -0,0 +1,38 @@ +{{- if and .Values.externalSecrets.enabled .Values.copilot.enabled .Values.copilot.server.secret.create }} +# ExternalSecret for copilot server credentials (syncs from external secret managers) +# +# Mirrors external-secret-app.yaml: the data list is generated by iterating every +# non-empty entry in externalSecrets.remoteRefs.copilot. sim.copilot.validate fails +# template rendering if a required key (or any non-empty copilot.server.env key) is +# missing a matching remoteRefs.copilot entry, so a misconfiguration surfaces at +# `helm template` time instead of a container starting with an empty value. +apiVersion: external-secrets.io/{{ .Values.externalSecrets.apiVersion | default "v1beta1" }} +kind: ExternalSecret +metadata: + name: {{ include "sim.copilot.envSecretName" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "sim.copilot.labels" . | nindent 4 }} +spec: + refreshInterval: {{ .Values.externalSecrets.refreshInterval | quote }} + secretStoreRef: + name: {{ required "externalSecrets.secretStoreRef.name is required when externalSecrets.enabled=true" .Values.externalSecrets.secretStoreRef.name }} + kind: {{ .Values.externalSecrets.secretStoreRef.kind | default "ClusterSecretStore" }} + target: + name: {{ include "sim.copilot.envSecretName" . }} + creationPolicy: Owner + data: + {{- range $secretKey, $ref := .Values.externalSecrets.remoteRefs.copilot }} + {{- if $ref }} + {{- if kindIs "string" $ref }} + - secretKey: {{ $secretKey }} + remoteRef: + key: {{ $ref }} + {{- else if kindIs "map" $ref }} + - secretKey: {{ $secretKey }} + remoteRef: + {{- toYaml $ref | nindent 8 }} + {{- end }} + {{- end }} + {{- end }} +{{- end }} diff --git a/helm/sim/templates/poddisruptionbudget.yaml b/helm/sim/templates/poddisruptionbudget.yaml index 49c5015e786..7981fbda931 100644 --- a/helm/sim/templates/poddisruptionbudget.yaml +++ b/helm/sim/templates/poddisruptionbudget.yaml @@ -75,8 +75,7 @@ metadata: name: {{ include "sim.fullname" $ }}-copilot-pdb namespace: {{ $.Release.Namespace }} labels: - {{- include "sim.labels" $ | nindent 4 }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.labels" $ | nindent 4 }} spec: {{- if .minAvailable }} minAvailable: {{ .minAvailable }} @@ -90,8 +89,6 @@ spec: {{- end }} selector: matchLabels: - app.kubernetes.io/name: {{ include "sim.name" $ }} - app.kubernetes.io/instance: {{ $.Release.Name }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.selectorLabels" $ | nindent 6 }} {{- end }} {{- end }} diff --git a/helm/sim/templates/secrets-copilot.yaml b/helm/sim/templates/secrets-copilot.yaml index 55e397fdf3d..3dd9170055f 100644 --- a/helm/sim/templates/secrets-copilot.yaml +++ b/helm/sim/templates/secrets-copilot.yaml @@ -1,12 +1,11 @@ -{{- if and .Values.copilot.enabled .Values.copilot.server.secret.create }} +{{- if and .Values.copilot.enabled .Values.copilot.server.secret.create (not (and .Values.externalSecrets .Values.externalSecrets.enabled)) }} apiVersion: v1 kind: Secret metadata: name: {{ include "sim.copilot.envSecretName" . }} namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.labels" . | nindent 4 }} {{- with .Values.copilot.server.secret.annotations }} annotations: {{- toYaml . | nindent 4 }} @@ -25,8 +24,7 @@ metadata: name: {{ include "sim.copilot.databaseSecretName" . }} namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot + {{- include "sim.copilot.labels" . | nindent 4 }} {{- with .Values.copilot.server.secret.annotations }} annotations: {{- toYaml . | nindent 4 }} diff --git a/helm/sim/templates/statefulset-copilot-postgres.yaml b/helm/sim/templates/statefulset-copilot-postgres.yaml index 91dd5bad195..fcb2e797774 100644 --- a/helm/sim/templates/statefulset-copilot-postgres.yaml +++ b/helm/sim/templates/statefulset-copilot-postgres.yaml @@ -5,8 +5,7 @@ metadata: name: {{ include "sim.fullname" . }}-copilot-postgresql-secret namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.labels" . | nindent 4 }} type: Opaque stringData: POSTGRES_USER: {{ .Values.copilot.postgresql.auth.username | quote }} @@ -21,8 +20,7 @@ metadata: name: {{ include "sim.fullname" . }}-copilot-postgresql namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.labels" . | nindent 4 }} spec: type: {{ .Values.copilot.postgresql.service.type }} ports: @@ -31,9 +29,7 @@ spec: protocol: TCP name: postgresql selector: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.selectorLabels" . | nindent 4 }} --- # Headless Service for the StatefulSet (stable per-pod DNS) apiVersion: v1 @@ -42,8 +38,7 @@ metadata: name: {{ include "sim.fullname" . }}-copilot-postgresql-headless namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.labels" . | nindent 4 }} spec: clusterIP: None publishNotReadyAddresses: true @@ -53,9 +48,7 @@ spec: protocol: TCP name: postgresql selector: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.selectorLabels" . | nindent 4 }} --- apiVersion: apps/v1 kind: StatefulSet @@ -63,8 +56,7 @@ metadata: name: {{ include "sim.fullname" . }}-copilot-postgresql namespace: {{ .Release.Namespace }} labels: - {{- include "sim.labels" . | nindent 4 }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.labels" . | nindent 4 }} spec: # Must remain {{ include "sim.fullname" . }}-copilot-postgresql (not the # -headless name) — spec.serviceName is immutable on a StatefulSet, and @@ -77,15 +69,11 @@ spec: type: RollingUpdate selector: matchLabels: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.selectorLabels" . | nindent 6 }} template: metadata: labels: - app.kubernetes.io/name: {{ include "sim.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.selectorLabels" . | nindent 8 }} {{- with .Values.podLabels }} {{- toYaml . | nindent 8 }} {{- end }} @@ -143,8 +131,7 @@ spec: - metadata: name: data labels: - {{- include "sim.labels" . | nindent 10 }} - app.kubernetes.io/component: copilot-postgresql + {{- include "sim.copilotPostgresql.labels" . | nindent 10 }} spec: accessModes: {{- range .Values.copilot.postgresql.persistence.accessModes }} diff --git a/helm/sim/templates/telemetry.yaml b/helm/sim/templates/telemetry.yaml index aa78f8acd5c..71bfea320cf 100644 --- a/helm/sim/templates/telemetry.yaml +++ b/helm/sim/templates/telemetry.yaml @@ -125,10 +125,8 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} serviceAccountName: {{ include "sim.serviceAccountName" . }} - securityContext: - runAsNonRoot: true - runAsUser: 10001 - fsGroup: 10001 + automountServiceAccountToken: false + {{- include "sim.podSecurityContext" .Values.telemetry | nindent 6 }} containers: - name: otel-collector image: {{ include "sim.image" (dict "imageRoot" .Values.telemetry.image "global" .Values.global "chartAppVersion" .Chart.AppVersion) }} @@ -177,6 +175,7 @@ spec: failureThreshold: 3 resources: {{- toYaml .Values.telemetry.resources | nindent 12 }} + {{- include "sim.containerSecurityContext" .Values.telemetry | nindent 10 }} volumes: - name: otel-config configMap: diff --git a/helm/sim/tests/chart-computed-env_test.yaml b/helm/sim/tests/chart-computed-env_test.yaml index 50ac5206405..8d78fec65ab 100644 --- a/helm/sim/tests/chart-computed-env_test.yaml +++ b/helm/sim/tests/chart-computed-env_test.yaml @@ -7,12 +7,12 @@ tests: - it: app pod uses chart-computed DATABASE_URL even when user tries to override template: deployment-app.yaml set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x app.env.DATABASE_URL: "should-be-ignored" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - notContains: path: spec.template.spec.containers[0].env @@ -27,7 +27,7 @@ tests: app.secrets.existingSecret.name: my-secret app.env.DATABASE_URL: "postgres://evil-1:5432/x" app.env.SOCKET_SERVER_URL: "https://evil-2.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - notContains: path: spec.template.spec.containers[0].env @@ -43,7 +43,7 @@ tests: app.secrets.existingSecret.name: my-secret app.env.DATABASE_URL: "postgres://evil-1:5432/x" app.env.SOCKET_SERVER_URL: "https://evil-2.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - notContains: path: spec.template.spec.containers[0].env diff --git a/helm/sim/tests/copilot-secret-modes_test.yaml b/helm/sim/tests/copilot-secret-modes_test.yaml new file mode 100644 index 00000000000..dd881a27195 --- /dev/null +++ b/helm/sim/tests/copilot-secret-modes_test.yaml @@ -0,0 +1,349 @@ +suite: copilot secret modes — inline / existingSecret / ESO routing (best-practices fix regression net) +release: + name: t + namespace: sim + +tests: + - it: checksum/secret changes when an ESO remoteRefs.copilot mapping changes, forcing a rollout (Cursor round 4 finding) + template: deployment-copilot.yaml + documentSelector: + path: kind + value: Deployment + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + externalSecrets.remoteRefs.copilot.AGENT_API_DB_ENCRYPTION_KEY: path/to/agentdbkey + externalSecrets.remoteRefs.copilot.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.copilot.LICENSE_KEY: path/to/license-v1 + externalSecrets.remoteRefs.copilot.SIM_BASE_URL: path/to/baseurl + externalSecrets.remoteRefs.copilot.SIM_AGENT_API_KEY: path/to/agentkey + externalSecrets.remoteRefs.copilot.REDIS_URL: path/to/redis + externalSecrets.remoteRefs.copilot.OPENAI_API_KEY_1: path/to/openai + asserts: + - isNotEmpty: + path: spec.template.metadata.annotations["checksum/secret"] + - matchRegex: + path: spec.template.metadata.annotations["checksum/secret"] + pattern: "^[a-f0-9]{64}$" + + - it: ESO mode with only the required copilot secrets mapped renders cleanly (envDefaults regression net — Greptile round 1 finding) + template: deployment-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + # Only the genuinely-secret keys are mapped — PORT/SERVICE_NAME/ENVIRONMENT/LOG_LEVEL + # live in copilot.server.envDefaults now, not copilot.server.env, so they must NOT + # need a remoteRefs.copilot entry. + externalSecrets.remoteRefs.copilot.AGENT_API_DB_ENCRYPTION_KEY: path/to/agentdbkey + externalSecrets.remoteRefs.copilot.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.copilot.LICENSE_KEY: path/to/license + externalSecrets.remoteRefs.copilot.SIM_BASE_URL: path/to/baseurl + externalSecrets.remoteRefs.copilot.SIM_AGENT_API_KEY: path/to/agentkey + externalSecrets.remoteRefs.copilot.REDIS_URL: path/to/redis + externalSecrets.remoteRefs.copilot.OPENAI_API_KEY_1: path/to/openai + documentSelector: + path: kind + value: Deployment + asserts: + - isKind: { of: Deployment } + - contains: + path: spec.template.spec.containers[0].env + content: + name: PORT + value: "8080" + - contains: + path: spec.template.spec.containers[0].env + content: + name: SERVICE_NAME + value: copilot + + - it: inline mode renders the chart-managed copilot Secret + template: secrets-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: true + copilot.postgresql.auth.password: xxxxxxxx + copilot.server.env.AGENT_API_DB_ENCRYPTION_KEY: x + copilot.server.env.INTERNAL_API_SECRET: x + copilot.server.env.LICENSE_KEY: x + copilot.server.env.SIM_BASE_URL: x + copilot.server.env.SIM_AGENT_API_KEY: x + copilot.server.env.REDIS_URL: x + copilot.server.env.OPENAI_API_KEY_1: x + asserts: + - hasDocuments: { count: 1 } + - isKind: { of: Secret } + - equal: { path: metadata.name, value: t-sim-copilot-env } + + - it: existingSecret mode does not shadow the pre-created Secret's values with envDefaults (Greptile + Cursor round 2 finding) + template: deployment-copilot.yaml + documentSelector: + path: kind + value: Deployment + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + copilot.server.secret.create: false + copilot.server.secret.name: my-existing-copilot-secret + asserts: + - notExists: + path: spec.template.spec.containers[0].env + + - it: existingSecret mode still renders extraEnv even though envDefaults are skipped + template: deployment-copilot.yaml + documentSelector: + path: kind + value: Deployment + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + copilot.server.secret.create: false + copilot.server.secret.name: my-existing-copilot-secret + copilot.server.extraEnv: + - name: CUSTOM_VAR + value: custom-value + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: CUSTOM_VAR + value: custom-value + - notContains: + path: spec.template.spec.containers[0].env + content: + name: PORT + value: "8080" + + - it: copilot's own existingSecret is still the source of truth even when externalSecrets.enabled=true globally for other components (Greptile round 3 finding) + template: deployment-copilot.yaml + documentSelector: + path: kind + value: Deployment + set: + app.env.BETTER_AUTH_SECRET: "" + app.env.ENCRYPTION_KEY: "" + app.env.INTERNAL_API_SECRET: "" + app.env.CRON_SECRET: "" + postgresql.auth.password: "" + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + copilot.server.secret.create: false + copilot.server.secret.name: my-existing-copilot-secret + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + asserts: + - notExists: + path: spec.template.spec.containers[0].env + - equal: + path: spec.template.spec.containers[0].envFrom[0].secretRef.name + value: my-existing-copilot-secret + + - it: existingSecret mode skips the chart-managed copilot Secret and the ExternalSecret + templates: + - secrets-copilot.yaml + - external-secret-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: true + copilot.postgresql.auth.password: xxxxxxxx + copilot.server.secret.create: false + copilot.server.secret.name: my-existing-copilot-secret + asserts: + - hasDocuments: { count: 0 } + + - it: ESO mode renders an ExternalSecret instead of the chart-managed copilot Secret + template: external-secret-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + externalSecrets.remoteRefs.copilot.PORT: path/to/port + externalSecrets.remoteRefs.copilot.SERVICE_NAME: path/to/name + externalSecrets.remoteRefs.copilot.ENVIRONMENT: path/to/env + externalSecrets.remoteRefs.copilot.LOG_LEVEL: path/to/log + externalSecrets.remoteRefs.copilot.AGENT_API_DB_ENCRYPTION_KEY: path/to/agentdbkey + externalSecrets.remoteRefs.copilot.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.copilot.LICENSE_KEY: path/to/license + externalSecrets.remoteRefs.copilot.SIM_BASE_URL: path/to/baseurl + externalSecrets.remoteRefs.copilot.SIM_AGENT_API_KEY: path/to/agentkey + externalSecrets.remoteRefs.copilot.REDIS_URL: path/to/redis + externalSecrets.remoteRefs.copilot.OPENAI_API_KEY_1: path/to/openai + asserts: + - isKind: { of: ExternalSecret } + - equal: { path: metadata.name, value: t-sim-copilot-env } + - equal: { path: spec.secretStoreRef.name, value: sim-store } + + - it: ESO mode skips the chart-managed copilot Secret + template: secrets-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: true + copilot.postgresql.auth.password: xxxxxxxx + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + externalSecrets.remoteRefs.copilot.PORT: path/to/port + externalSecrets.remoteRefs.copilot.SERVICE_NAME: path/to/name + externalSecrets.remoteRefs.copilot.ENVIRONMENT: path/to/env + externalSecrets.remoteRefs.copilot.LOG_LEVEL: path/to/log + externalSecrets.remoteRefs.copilot.AGENT_API_DB_ENCRYPTION_KEY: path/to/agentdbkey + externalSecrets.remoteRefs.copilot.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.copilot.LICENSE_KEY: path/to/license + externalSecrets.remoteRefs.copilot.SIM_BASE_URL: path/to/baseurl + externalSecrets.remoteRefs.copilot.SIM_AGENT_API_KEY: path/to/agentkey + externalSecrets.remoteRefs.copilot.REDIS_URL: path/to/redis + externalSecrets.remoteRefs.copilot.OPENAI_API_KEY_1: path/to/openai + asserts: + - hasDocuments: { count: 0 } + + - it: ESO validator fails when a required copilot key is neither in env nor mapped + template: deployment-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + asserts: + - failedTemplate: + errorPattern: "Required key 'AGENT_API_DB_ENCRYPTION_KEY' is missing" + + - it: ESO validator fails when copilot.server.env contains an unmapped key + template: deployment-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + externalSecrets.enabled: true + externalSecrets.secretStoreRef.name: sim-store + externalSecrets.remoteRefs.app.BETTER_AUTH_SECRET: path/to/auth + externalSecrets.remoteRefs.app.ENCRYPTION_KEY: path/to/enc + externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron + externalSecrets.remoteRefs.postgresql.password: path/to/pgpw + externalSecrets.remoteRefs.copilot.PORT: path/to/port + externalSecrets.remoteRefs.copilot.SERVICE_NAME: path/to/name + externalSecrets.remoteRefs.copilot.ENVIRONMENT: path/to/env + externalSecrets.remoteRefs.copilot.LOG_LEVEL: path/to/log + externalSecrets.remoteRefs.copilot.AGENT_API_DB_ENCRYPTION_KEY: path/to/agentdbkey + externalSecrets.remoteRefs.copilot.INTERNAL_API_SECRET: path/to/iapi + externalSecrets.remoteRefs.copilot.LICENSE_KEY: path/to/license + externalSecrets.remoteRefs.copilot.SIM_BASE_URL: path/to/baseurl + externalSecrets.remoteRefs.copilot.SIM_AGENT_API_KEY: path/to/agentkey + externalSecrets.remoteRefs.copilot.REDIS_URL: path/to/redis + externalSecrets.remoteRefs.copilot.OPENAI_API_KEY_1: path/to/openai + copilot.server.env.UNMAPPED_KEY: "would-go-nowhere" + asserts: + - failedTemplate: + errorPattern: "Key 'UNMAPPED_KEY' is set in copilot.server.env but externalSecrets.enabled=true" + + - it: non-ESO mode still requires OPENAI_API_KEY_1 or ANTHROPIC_API_KEY_1 directly in env + template: deployment-copilot.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + copilot.enabled: true + copilot.postgresql.enabled: false + copilot.database.url: "postgresql://x:x@x:5432/x" + copilot.server.env.AGENT_API_DB_ENCRYPTION_KEY: x + copilot.server.env.INTERNAL_API_SECRET: x + copilot.server.env.LICENSE_KEY: x + copilot.server.env.SIM_BASE_URL: x + copilot.server.env.SIM_AGENT_API_KEY: x + copilot.server.env.REDIS_URL: x + asserts: + - failedTemplate: + errorPattern: "Set at least one of copilot.server.env.OPENAI_API_KEY_1 or copilot.server.env.ANTHROPIC_API_KEY_1" diff --git a/helm/sim/tests/env-defaults_test.yaml b/helm/sim/tests/env-defaults_test.yaml index a658f3dbfb2..7e3fdcb4898 100644 --- a/helm/sim/tests/env-defaults_test.yaml +++ b/helm/sim/tests/env-defaults_test.yaml @@ -3,12 +3,11 @@ release: name: t namespace: sim defaults: &defaults - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x - + postgresql.auth.password: xxxxxxxx tests: - it: inline mode renders localhost envDefaults on the app pod template: deployment-app.yaml @@ -37,7 +36,7 @@ tests: set: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-secret - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - notContains: path: spec.template.spec.containers[0].env @@ -50,7 +49,7 @@ tests: set: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-secret - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - notContains: path: spec.template.spec.containers[0].env @@ -64,7 +63,7 @@ tests: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-secret app.env.NEXT_PUBLIC_APP_URL: "https://prod.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - contains: path: spec.template.spec.containers[0].env @@ -78,7 +77,7 @@ tests: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-secret app.env.NEXT_PUBLIC_APP_URL: "https://prod.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - contains: path: spec.template.spec.containers[0].env @@ -93,7 +92,7 @@ tests: app.secrets.existingSecret.name: my-secret app.env.NEXT_PUBLIC_APP_URL: "https://prod.example.com" realtime.env.NEXT_PUBLIC_APP_URL: "https://realtime.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - contains: path: spec.template.spec.containers[0].env diff --git a/helm/sim/tests/helm-test-hook_test.yaml b/helm/sim/tests/helm-test-hook_test.yaml index 4240d584ec2..ee19f4f7776 100644 --- a/helm/sim/tests/helm-test-hook_test.yaml +++ b/helm/sim/tests/helm-test-hook_test.yaml @@ -3,12 +3,11 @@ release: name: t namespace: sim defaults: &defaults - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x - + postgresql.auth.password: xxxxxxxx tests: - it: renders the test pod with helm.sh/hook=test by default template: tests/test-connection.yaml diff --git a/helm/sim/tests/networkpolicy_test.yaml b/helm/sim/tests/networkpolicy_test.yaml index d6e0b260ec4..355fab6d27d 100644 --- a/helm/sim/tests/networkpolicy_test.yaml +++ b/helm/sim/tests/networkpolicy_test.yaml @@ -5,11 +5,11 @@ release: name: t namespace: sim defaults: &defaults - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx networkPolicy.enabled: true ingress.enabled: true ingress.app.host: a.test diff --git a/helm/sim/tests/pdb-hpa_test.yaml b/helm/sim/tests/pdb-hpa_test.yaml index 70f3b0903a2..3960f2e4006 100644 --- a/helm/sim/tests/pdb-hpa_test.yaml +++ b/helm/sim/tests/pdb-hpa_test.yaml @@ -3,12 +3,11 @@ release: name: t namespace: sim defaults: &defaults - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x - + postgresql.auth.password: xxxxxxxx tests: - it: PDB does not render when replicaCount=1 and HPA is off (tri-state null) template: poddisruptionbudget.yaml diff --git a/helm/sim/tests/pii_test.yaml b/helm/sim/tests/pii_test.yaml index 3a416237207..f35c87a5759 100644 --- a/helm/sim/tests/pii_test.yaml +++ b/helm/sim/tests/pii_test.yaml @@ -3,12 +3,11 @@ release: name: t namespace: sim set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x - + postgresql.auth.password: xxxxxxxx tests: - it: does not render the pii deployment when disabled template: deployment-pii.yaml diff --git a/helm/sim/tests/pod-rollout_test.yaml b/helm/sim/tests/pod-rollout_test.yaml index 39b0c8c72fc..999ba85c434 100644 --- a/helm/sim/tests/pod-rollout_test.yaml +++ b/helm/sim/tests/pod-rollout_test.yaml @@ -3,12 +3,11 @@ release: name: t namespace: sim defaults: &defaults - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x - + postgresql.auth.password: xxxxxxxx tests: - it: app pod template has checksum/secret annotation template: deployment-app.yaml @@ -30,7 +29,7 @@ tests: template: deployment-app.yaml set: <<: *defaults - app.env.BETTER_AUTH_SECRET: original-value + app.env.BETTER_AUTH_SECRET: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy asserts: - notEqual: path: spec.template.metadata.annotations["checksum/secret"] diff --git a/helm/sim/tests/schema-secret-length_test.yaml b/helm/sim/tests/schema-secret-length_test.yaml new file mode 100644 index 00000000000..b2255d93853 --- /dev/null +++ b/helm/sim/tests/schema-secret-length_test.yaml @@ -0,0 +1,59 @@ +suite: values.schema.json — secret length enforcement (blocker fix regression net) +release: + name: t + namespace: sim + +tests: + - it: rejects a BETTER_AUTH_SECRET shorter than 32 characters + set: + app.env.BETTER_AUTH_SECRET: short + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + asserts: + - failedTemplate: + errorPattern: "BETTER_AUTH_SECRET.*minLength" + + - it: rejects an ENCRYPTION_KEY shorter than 32 characters + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: short + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + asserts: + - failedTemplate: + errorPattern: "ENCRYPTION_KEY.*minLength" + + - it: rejects a postgresql.auth.password shorter than 8 characters + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: short + asserts: + - failedTemplate: + errorPattern: "password.*minLength" + + - it: accepts an empty BETTER_AUTH_SECRET (deferred to existingSecret/ESO) + templates: + - secrets-app.yaml + set: + app.secrets.existingSecret.enabled: true + app.secrets.existingSecret.name: my-existing-secret + postgresql.auth.password: xxxxxxxx + asserts: + - hasDocuments: { count: 0 } + + - it: accepts a valid 32-character BETTER_AUTH_SECRET and ENCRYPTION_KEY + template: deployment-app.yaml + set: + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.INTERNAL_API_SECRET: x + app.env.CRON_SECRET: x + postgresql.auth.password: xxxxxxxx + asserts: + - isKind: { of: Deployment } diff --git a/helm/sim/tests/secret-modes_test.yaml b/helm/sim/tests/secret-modes_test.yaml index d1e73c3fdab..6b60cf56693 100644 --- a/helm/sim/tests/secret-modes_test.yaml +++ b/helm/sim/tests/secret-modes_test.yaml @@ -7,11 +7,11 @@ tests: - it: inline mode renders the chart-managed Secret template: secrets-app.yaml set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - isKind: { of: Secret } - equal: { path: metadata.name, value: t-sim-app-secrets } @@ -23,7 +23,7 @@ tests: set: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-existing-secret - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - hasDocuments: { count: 0 } @@ -38,7 +38,7 @@ tests: externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron externalSecrets.remoteRefs.postgresql.password: path/to/pgpw - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - isKind: { of: ExternalSecret } - equal: { path: metadata.name, value: t-sim-app-secrets } @@ -55,7 +55,7 @@ tests: externalSecrets.remoteRefs.app.INTERNAL_API_SECRET: path/to/iapi externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron externalSecrets.remoteRefs.postgresql.password: path/to/pgpw - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - hasDocuments: { count: 0 } @@ -69,7 +69,7 @@ tests: externalSecrets.remoteRefs.app.CRON_SECRET: path/to/cron externalSecrets.remoteRefs.postgresql.password: path/to/pgpw app.env.UNMAPPED_KEY: "would-go-nowhere" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorPattern: "Key 'UNMAPPED_KEY' is set in app.env but externalSecrets.enabled=true" @@ -79,7 +79,7 @@ tests: set: app.secrets.existingSecret.enabled: true app.secrets.existingSecret.name: my-existing-secret - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - contains: path: spec.template.spec.containers[0].envFrom @@ -90,13 +90,13 @@ tests: - it: realtime.env-only value reaches the Secret when app.env entry is empty (cursor bugbot fix) template: secrets-app.yaml set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x app.env.BETTER_AUTH_URL: "" realtime.env.BETTER_AUTH_URL: "https://realtime.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - equal: path: stringData.BETTER_AUTH_URL @@ -105,13 +105,13 @@ tests: - it: app.env wins over realtime.env on collision when both are non-empty template: secrets-app.yaml set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x app.env.NEXT_PUBLIC_APP_URL: "https://app.example.com" realtime.env.NEXT_PUBLIC_APP_URL: "https://realtime.example.com" - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - equal: path: stringData.NEXT_PUBLIC_APP_URL diff --git a/helm/sim/tests/smoke_test.yaml b/helm/sim/tests/smoke_test.yaml index 71bf0888236..0e50a2799be 100644 --- a/helm/sim/tests/smoke_test.yaml +++ b/helm/sim/tests/smoke_test.yaml @@ -9,7 +9,7 @@ release: namespace: sim set: app.env.BETTER_AUTH_SECRET: test-better-auth-secret-32-bytes - app.env.ENCRYPTION_KEY: test-encryption-key-32-bytes + app.env.ENCRYPTION_KEY: test-encryption-key-32-bytes-long app.env.INTERNAL_API_SECRET: test-internal-api-secret-32-bytes app.env.CRON_SECRET: test-cron-secret-32-bytes postgresql.auth.password: testpassword diff --git a/helm/sim/tests/telemetry-security-context_test.yaml b/helm/sim/tests/telemetry-security-context_test.yaml new file mode 100644 index 00000000000..4d3f7233899 --- /dev/null +++ b/helm/sim/tests/telemetry-security-context_test.yaml @@ -0,0 +1,53 @@ +suite: telemetry — Restricted Pod Security Standard hardening (best-practices fix regression net) +release: + name: t + namespace: sim +templates: + - telemetry.yaml + +tests: + - it: pod-level security context preserves the collector's original UID/GID/fsGroup and adds seccompProfile + set: + telemetry.enabled: true + documentSelector: + path: kind + value: Deployment + asserts: + - equal: + path: spec.template.spec.securityContext.runAsUser + value: 10001 + - equal: + path: spec.template.spec.securityContext.runAsGroup + value: 10001 + - equal: + path: spec.template.spec.securityContext.fsGroup + value: 10001 + - equal: + path: spec.template.spec.securityContext.runAsNonRoot + value: true + - equal: + path: spec.template.spec.securityContext.seccompProfile.type + value: RuntimeDefault + - equal: + path: spec.template.spec.automountServiceAccountToken + value: false + + - it: container-level security context matches the Restricted profile (same as every other workload) + set: + telemetry.enabled: true + documentSelector: + path: kind + value: Deployment + asserts: + - equal: + path: spec.template.spec.containers[0].securityContext.allowPrivilegeEscalation + value: false + - equal: + path: spec.template.spec.containers[0].securityContext.capabilities.drop[0] + value: ALL + - equal: + path: spec.template.spec.containers[0].securityContext.runAsNonRoot + value: true + - equal: + path: spec.template.spec.containers[0].securityContext.seccompProfile.type + value: RuntimeDefault diff --git a/helm/sim/tests/validators_test.yaml b/helm/sim/tests/validators_test.yaml index b53800b2d14..aff09b4386b 100644 --- a/helm/sim/tests/validators_test.yaml +++ b/helm/sim/tests/validators_test.yaml @@ -9,52 +9,52 @@ tests: - it: fails when BETTER_AUTH_SECRET is missing set: app.env.BETTER_AUTH_SECRET: "" - app.env.ENCRYPTION_KEY: x + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorMessage: "app.env.BETTER_AUTH_SECRET is required for production deployment" - it: fails when ENCRYPTION_KEY is missing set: - app.env.BETTER_AUTH_SECRET: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.ENCRYPTION_KEY: "" app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorMessage: "app.env.ENCRYPTION_KEY is required for production deployment" - it: fails when INTERNAL_API_SECRET is missing (1.0.0 requirement) set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: "" app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorMessage: "app.env.INTERNAL_API_SECRET is required for production deployment (shared auth between sim-app and sim-realtime pods). Generate one with: openssl rand -hex 32" - it: fails when cronjobs are enabled but CRON_SECRET is empty set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: "" cronjobs.enabled: true - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorMessage: "app.env.CRON_SECRET is required when cronjobs.enabled=true (every cron pod authenticates with this token). Generate one with: openssl rand -hex 32, or set cronjobs.enabled=false." - it: fails when postgresql.auth.password is missing set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x postgresql.auth.password: "" @@ -64,8 +64,8 @@ tests: - it: rejects postgres passwords with URL-unsafe characters set: - app.env.BETTER_AUTH_SECRET: x - app.env.ENCRYPTION_KEY: x + app.env.BETTER_AUTH_SECRET: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x postgresql.auth.password: "pa$$word" @@ -76,10 +76,10 @@ tests: - it: rejects placeholder BETTER_AUTH_SECRET set: app.env.BETTER_AUTH_SECRET: "CHANGE-ME-32-CHAR-SECRET-FOR-PRODUCTION-USE" - app.env.ENCRYPTION_KEY: x + app.env.ENCRYPTION_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx app.env.INTERNAL_API_SECRET: x app.env.CRON_SECRET: x - postgresql.auth.password: x + postgresql.auth.password: xxxxxxxx asserts: - failedTemplate: errorPattern: "must not use the default placeholder value" diff --git a/helm/sim/values.schema.json b/helm/sim/values.schema.json index 37f518542b2..4b1e86c0a75 100644 --- a/helm/sim/values.schema.json +++ b/helm/sim/values.schema.json @@ -108,11 +108,13 @@ "properties": { "BETTER_AUTH_SECRET": { "type": "string", - "description": "Auth secret (minimum 32 characters required when not using existingSecret)" + "anyOf": [{ "minLength": 32 }, { "const": "" }], + "description": "Auth secret (minimum 32 characters required when not using existingSecret; leave empty when the value comes from existingSecret/ESO instead)" }, "ENCRYPTION_KEY": { "type": "string", - "description": "Encryption key (minimum 32 characters required when not using existingSecret)" + "anyOf": [{ "minLength": 32 }, { "const": "" }], + "description": "Encryption key (minimum 32 characters required when not using existingSecret; leave empty when the value comes from existingSecret/ESO instead)" }, "NEXT_PUBLIC_APP_URL": { "type": "string", @@ -186,6 +188,10 @@ "type": "string", "description": "AWS region for SES (e.g., 'us-east-1'). Credentials are resolved via the standard AWS provider chain (env vars, IRSA, EC2/ECS task role, SSO)." }, + "AZURE_ACS_CONNECTION_STRING": { + "type": "string", + "description": "Azure Communication Services connection string (email provider — used when Resend/SES/SMTP are not configured)." + }, "SMTP_HOST": { "type": "string", "description": "SMTP server hostname. When set together with SMTP_PORT, enables the SMTP mail provider." @@ -270,6 +276,26 @@ "type": "string", "description": "ElevenLabs API key for text-to-speech in deployed chat" }, + "KB_OPENAI_MODEL_NAME": { + "type": "string", + "description": "Azure deployment name serving the configured KB embedding model (used only when AZURE_OPENAI_* credentials are set)." + }, + "WAND_OPENAI_MODEL_NAME": { + "type": "string", + "description": "Wand generation model deployment name (works with both regular OpenAI and Azure OpenAI)." + }, + "OCR_AZURE_ENDPOINT": { + "type": "string", + "description": "Azure Mistral OCR service endpoint." + }, + "OCR_AZURE_MODEL_NAME": { + "type": "string", + "description": "Azure Mistral OCR model name." + }, + "OCR_AZURE_API_KEY": { + "type": "string", + "description": "Azure Mistral OCR API key." + }, "RATE_LIMIT_WINDOW_MS": { "type": "string", "description": "Rate limit window duration in milliseconds" @@ -408,7 +434,8 @@ "properties": { "BETTER_AUTH_SECRET": { "type": "string", - "description": "Auth secret (minimum 32 characters required when not using existingSecret)" + "anyOf": [{ "minLength": 32 }, { "const": "" }], + "description": "Auth secret (minimum 32 characters required when not using existingSecret; leave empty when the value comes from existingSecret/ESO instead)" }, "NEXT_PUBLIC_APP_URL": { "type": "string", @@ -504,7 +531,8 @@ }, "password": { "type": "string", - "description": "PostgreSQL password (minimum 8 characters when not using existingSecret)" + "anyOf": [{ "minLength": 8 }, { "const": "" }], + "description": "PostgreSQL password (minimum 8 characters when not using existingSecret; leave empty when the value comes from existingSecret/ESO instead)" }, "existingSecret": { "type": "object", diff --git a/helm/sim/values.yaml b/helm/sim/values.yaml index a12713a5bec..c7a39c81e92 100644 --- a/helm/sim/values.yaml +++ b/helm/sim/values.yaml @@ -117,6 +117,7 @@ app: FROM_EMAIL_ADDRESS: "" # Complete from address (e.g., "Sim " or "DoNotReply@domain.com") EMAIL_DOMAIN: "" # Domain for sending emails (fallback when FROM_EMAIL_ADDRESS not set) AWS_SES_REGION: "" # AWS region for SES (e.g., "us-east-1"); credentials resolved via the standard AWS provider chain (env, IRSA, instance profile) + AZURE_ACS_CONNECTION_STRING: "" # Azure Communication Services connection string (email provider — used when Resend/SES/SMTP are not configured) SMTP_HOST: "" # SMTP server hostname (alternative to Resend/SES) SMTP_PORT: "" # SMTP server port (465 for TLS, 587 for STARTTLS, 25 for plain) SMTP_USER: "" # SMTP username (optional — omit for unauthenticated relays like MailHog) @@ -145,6 +146,13 @@ app: AZURE_ANTHROPIC_ENDPOINT: "" # Azure AI Foundry endpoint for Anthropic models AZURE_ANTHROPIC_API_KEY: "" # Azure Anthropic API key AZURE_ANTHROPIC_API_VERSION: "" # Azure Anthropic API version (e.g., 2023-06-01) + KB_OPENAI_MODEL_NAME: "" # Azure deployment name serving the configured KB embedding model (used only when AZURE_OPENAI_* credentials are set) + WAND_OPENAI_MODEL_NAME: "" # Wand generation model deployment name (works with both regular OpenAI and Azure OpenAI) + + # Azure Mistral OCR Configuration (leave empty if not using Azure-hosted OCR for document processing) + OCR_AZURE_ENDPOINT: "" # Azure Mistral OCR service endpoint + OCR_AZURE_MODEL_NAME: "" # Azure Mistral OCR model name + OCR_AZURE_API_KEY: "" # Azure Mistral OCR API key # AI Provider API Keys (leave empty if not using) OPENAI_API_KEY: "" # Primary OpenAI API key @@ -1429,6 +1437,15 @@ telemetry: tls: enabled: false + # Pod security context (preserves the collector's original UID/GID/fsGroup) + podSecurityContext: + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + + # Container security context (sim.containerSecurityContext restricted defaults apply) + securityContext: {} + # Copilot service configuration (optional microservice) copilot: # Enable/disable the copilot service @@ -1470,11 +1487,12 @@ copilot: runAsNonRoot: true runAsUser: 1001 - # Environment variables (required and optional) + # Environment variables that go through the chart-managed Secret (or ExternalSecret + # when externalSecrets.enabled=true). Every non-empty key here must have a matching + # externalSecrets.remoteRefs.copilot entry under ESO — reserve this block for values + # that are genuinely sensitive or user-provided; static operational config belongs in + # envDefaults below instead. env: - PORT: "8080" - SERVICE_NAME: "copilot" - ENVIRONMENT: "production" AGENT_API_DB_ENCRYPTION_KEY: "" INTERNAL_API_SECRET: "" LICENSE_KEY: "" @@ -1483,17 +1501,31 @@ copilot: SIM_BASE_URL: "" SIM_AGENT_API_KEY: "" REDIS_URL: "" - # Optional configuration - LOG_LEVEL: "info" CORS_ALLOWED_ORIGINS: "" OTEL_EXPORTER_OTLP_ENDPOINT: "" - + + # Operational tunables shipped with the chart. Rendered as inline `env:` on the + # copilot container — NOT written into the chart-managed Secret and NOT required to + # be mapped when externalSecrets.enabled=true. Override any key by setting + # copilot.server.envDefaults.KEY in your values file. Move a key into + # copilot.server.env above only if it must be treated as secret. + envDefaults: + PORT: "8080" + SERVICE_NAME: "copilot" + ENVIRONMENT: "production" + LOG_LEVEL: "info" + # Optional: additional static environment variables extraEnv: [] - + # Optional: references to existing ConfigMaps/Secrets extraEnvFrom: [] - + + # Optional: additional volumes/mounts (e.g. to back an emptyDir /tmp when + # enabling securityContext.readOnlyRootFilesystem) + extraVolumes: [] + extraVolumeMounts: [] + # Secret generation configuration (set create=false to use an existing secret) secret: create: true @@ -1719,6 +1751,27 @@ externalSecrets: # Path to external database password in external store password: "" + # Copilot server secrets (used when copilot.enabled=true and copilot.server.secret.create=true). + # Every non-empty copilot.server.env key must have a matching entry here, or template + # rendering fails with a clear "map it or remove it" error — same rule as remoteRefs.app. + copilot: + # Path to AGENT_API_DB_ENCRYPTION_KEY in external store + AGENT_API_DB_ENCRYPTION_KEY: "" + # Path to INTERNAL_API_SECRET in external store (shared with app/realtime) + INTERNAL_API_SECRET: "" + # Path to LICENSE_KEY in external store + LICENSE_KEY: "" + # Path to SIM_BASE_URL in external store + SIM_BASE_URL: "" + # Path to SIM_AGENT_API_KEY in external store + SIM_AGENT_API_KEY: "" + # Path to REDIS_URL in external store + REDIS_URL: "" + # Path to OPENAI_API_KEY_1 in external store (one of OPENAI_API_KEY_1/ANTHROPIC_API_KEY_1 required) + OPENAI_API_KEY_1: "" + # Path to ANTHROPIC_API_KEY_1 in external store (one of OPENAI_API_KEY_1/ANTHROPIC_API_KEY_1 required) + ANTHROPIC_API_KEY_1: "" + # cert-manager configuration # Prerequisites: Install cert-manager in your cluster first # See: https://cert-manager.io/docs/installation/ diff --git a/packages/emcn/src/icons/card.tsx b/packages/emcn/src/icons/card.tsx deleted file mode 100644 index ac0461deae8..00000000000 --- a/packages/emcn/src/icons/card.tsx +++ /dev/null @@ -1,26 +0,0 @@ -import type { SVGProps } from 'react' - -/** - * Card icon component - * @param props - SVG properties including className, fill, etc. - */ -export function Card(props: SVGProps) { - return ( - - ) -} diff --git a/packages/emcn/src/icons/index.ts b/packages/emcn/src/icons/index.ts index 1af0459200b..6cf6d1bbf20 100644 --- a/packages/emcn/src/icons/index.ts +++ b/packages/emcn/src/icons/index.ts @@ -12,7 +12,6 @@ export { BubbleChatDelay } from './bubble-chat-delay' export { BubbleChatPreview } from './bubble-chat-preview' export { Bug } from './bug' export { Calendar } from './calendar' -export { Card } from './card' export { Check } from './check' export { ChevronDown } from './chevron-down' export { CircleAlert } from './circle-alert' diff --git a/scripts/check-api-validation-contracts.ts b/scripts/check-api-validation-contracts.ts index 23c532a1dea..3fb1e9e1c1b 100644 --- a/scripts/check-api-validation-contracts.ts +++ b/scripts/check-api-validation-contracts.ts @@ -9,8 +9,8 @@ const QUERY_HOOKS_DIR = path.join(ROOT, 'apps/sim/hooks/queries') const SELECTOR_HOOKS_DIR = path.join(ROOT, 'apps/sim/hooks/selectors') const BASELINE = { - totalRoutes: 921, - zodRoutes: 921, + totalRoutes: 922, + zodRoutes: 922, nonZodRoutes: 0, } as const diff --git a/scripts/generate-docs.ts b/scripts/generate-docs.ts index 3ad1c9e3938..984dd89a76e 100755 --- a/scripts/generate-docs.ts +++ b/scripts/generate-docs.ts @@ -165,6 +165,17 @@ interface BlockConfig { [key: string]: any } +/** + * True when a block's source text marks it as an unreleased `preview: true` + * block. THE single preview gate for this script — every surface it emits + * (docs .mdx, integrations.json, icon mapping) must consult this, because a + * missed gate publishes an unreleased block to docs.sim.ai, the catalog, the + * sitemap, and OG images. Mirrors the `hideFromToolbar` source-text checks. + */ +function isPreviewSource(blockContent: string): boolean { + return /preview\s*:\s*true/.test(blockContent) +} + /** * Find the position after the matching close delimiter for an opening delimiter. * Assumes `content[openPos]` is the opening char (e.g. `{` or `[`). @@ -298,6 +309,11 @@ async function generateIconMapping(options: { // Check hideFromToolbar - skip hidden blocks for docs but NOT for icon mapping const hideFromToolbar = /hideFromToolbar\s*:\s*true/.test(blockContent) + // Unreleased preview blocks never reach any public surface, icon map included. + if (isPreviewSource(blockContent)) { + continue + } + // Get block type const blockType = extractStringPropertyFromContent(blockContent, 'type') || blockName.toLowerCase() @@ -983,6 +999,14 @@ function extractAllBlockConfigs(fileContent: string): BlockConfig[] { continue } + // Unreleased preview blocks stay out of every generated surface: docs + // .mdx pages, integrations.json (landing + workspace catalog + sitemap + + // OG images), and the icon mapping. + if (isPreviewSource(blockContent)) { + console.log(`Skipping ${blockName}Block - preview is true`) + continue + } + // Pass fileContent to enable spread inheritance resolution const config = extractBlockConfigFromContent(blockContent, blockName, fileContent) if (config) { @@ -1139,8 +1163,12 @@ function extractBlockConfigFromContent( * also naturally selects the canonical version. Recategorizing a block to * `'blocks'` or `'triggers'` removes it from all integration surfaces. */ -function isIntegrationBlock(config: { category?: string; hideFromToolbar?: boolean }): boolean { - return config.category === 'tools' && !config.hideFromToolbar +function isIntegrationBlock(config: { + category?: string + hideFromToolbar?: boolean + preview?: boolean +}): boolean { + return config.category === 'tools' && !config.hideFromToolbar && !config.preview } /**