[Feature]: image 2 已经够厉害了，最需要的是如何把生成好的UI 变成组件化，再到原型的过程！

[chenjunyu-1990]

Problem / pain point

image 2 已经够厉害了，最需要的是如何把生成好的UI 拆解变成组件化，再到交互原型的过程！市面上都没有完整的方案，求求

Proposed solution

UI 拆解变成组件化，目前lovart 只能做到一点点。但是再到下一步他就完全不支持了

Alternatives considered

No response

Scope

UI

Hard-constraints check

• I have read the hard constraints in CLAUDE.md and this proposal does not conflict with any of them.

[github-actions]

感谢反馈，这个方向很重要。按当前公开仓库来看，我确认现在已经有的是“提示词/附件 -> HTML 或 JSX/React 原型”的生成与局部改写能力；但没有在公开代码里找到“把现有 UI 自动拆成可复用组件，再继续串成交互原型流程”的完整实现。

目前更接近的是：你可以附带文件/图片作为上下文，让模型生成或改写原型；生成后还会自动补一轮交互深化，例如 tab 切换、卡片展开、hover/press、空状态等。但这和“结构化拆组件 -> 组件树/状态机 -> 原型流”还是两件事。

这个项目现在仍是 pre-alpha，README 里的路线也显示很多导入/设计系统相关能力还在后续阶段，公开代码里我只看到更丰富的导入能力排在 v0.2、Figma 图层导出排在 1.0 之后，所以你提的这条链路目前更像是待规划能力，不是已经内置但没暴露的功能。

Relevant code:

• README.zh-CN.md:224 — 当前公开描述是“提示词 → HTML 或 JSX/React 组件原型”
• README.zh-CN.md:228 — 已有 Comment mode，支持只重写预览中的局部区域
• apps/desktop/src/renderer/src/components/Sidebar.tsx:167 — 当前 UI 支持把文件作为上下文附加到会话里
• apps/desktop/src/main/prompt-context.ts:148 — 图片附件会被作为 image input 传给模型
• packages/core/src/index.ts:682 — 生成时会把附件转成 userImages 送入模型
• packages/core/src/index.ts:736 — 局部改写时同样会把附件转成 userImages
• apps/desktop/src/renderer/src/hooks/polishPrompt.ts:14 — 现有“二轮深化”主要是在已有页面上补交互深度，不是做组件拆解流水线
• README.zh-CN.md:255 — roadmap 里 v0.2 提到的是更丰富的导入能力
• README.zh-CN.md:267 — roadmap 里写了 Figma 图层导出在 1.0 之后

Need more info:

• 你更希望输入是什么：截图、生成后的 HTML/JSX、还是 Figma 设计稿？
• 你更希望输出是什么：React 组件树、设计系统 token、还是可点击的多页面原型？
• 你说的“到原型的过程”更偏向单页状态交互，还是页面之间的 flow（跳转、返回、分支）？

---

open-codesign Bot

[HomenShum]

这条思路我也很想看到落地，稍微构思一下， 如果我们把gpt image 2 生成过程加到prompt -> composition 的pi-ai agent tool call kits, 那是否可以用生成出来的image当作context 来reference and enhance the final composition output of each ui component. We can also then show the gpt image 2 生成的原型图 next to or as a popover for the contexts referenced within the UI component display on a sidebar

[HomenShum]

这条思路我也很想看到落地，稍微构思一下， 如果我们把gpt image 2 生成过程加到prompt -> composition 的pi-ai agent tool call kits, 那是否可以用生成出来的image当作context 来reference and enhance the final composition output of each ui component. We can also then show the gpt image 2 生成的原型图 next to or as a popover for the contexts referenced within the UI component display on a sidebar

Quick update on my last comment. After checking PR #193 ("Gpt image 2 support"), generate_image_asset is already a registered agent tool in packages/core/src/tools/generate-image-asset.ts with 7 purposes (hero, product, poster, background, illustration, logo, other), defaulted off in Settings. So that idea is more accurately framed as "build a new orchestration loop on top of the existing tool" rather than "add a new tool", smaller change, fully reuses PR #193's infrastructure. Saving that for Phase 2 as a separate Discussion once Phase 1 lands.

Answers to the bot's three open questions, focused on Phase 1:

• Input: AI-generated UI mockup images (gpt image 2 / lovart / midjourney / plain screenshots). Plumbing already exists at prompt-context.ts:148 and index.ts:682 via userImages, what's missing is structured output.

• Output: a ui_kits/<slug>/ folder shaped like this (mock data shown for clarity):

  ui_kits/
  └── saas-dashboard/
      ├── index.html          # full-page HTML, visually parity-matched to source mockup
      ├── components/
      │   ├── Sidebar.tsx     # <Sidebar nav={[...]} active="dashboard" />
      │   ├── MetricCard.tsx  # <MetricCard label="MRR" value="$12.4k" delta="+8.2%" />
      │   ├── ChartPanel.tsx  # <ChartPanel data={...} timeRange="30d" />
      │   └── ActivityTable.tsx
      ├── tokens.css          # :root { --color-brand: #c96442; --space-md: 16px; ... }
      ├── manifest.json       # { schemaVersion: 1, generator, components, tokens }
      └── README.md           # handoff notes for the downstream coding agent
  

  So Claude Code / Cursor can pick the bundle up and implement it into real product code. This is the segment OP's title is calling out ("image 2 已经够厉害了, 最需要的是怎么变成组件再到原型").

• Granularity: Phase 1 is single image → single ui_kit only. Multi-page flow / routing / state machines → Phase 3, separate issue.

Implementation plans to follow existing patterns strictly: decompose-to-ui-kit.ts mirrors done.ts / generate-image-asset.ts (factory + AgentTool + typebox params + adjacent .test.ts), trigger hook mirrors polishPrompt.ts but user-triggered instead of auto-fired, no new dependencies, decomposition is prompt-driven, output carries schemaVersion so downstream coding agents can evolve safely.

Next step: I'll prototype on my fork, render a before/after side-by-side video via Remotion, reply here with the MP4 before opening any PR.

One binary I'd like to align on first: should the emitted ui_kits/<slug>/ be (a) in-memory only, surfaced via the Files panel + existing ZIP export (downstream coding agent unzips the bundle, no SQLite schema change), or (b) persisted to disk in the design workspace (requires schemaVersion bump per CLAUDE.md)? My lean is (a), smaller blast radius, consistent with how polishPrompt.ts's second-pass only modifies the in-memory artifact.

中文版本 (Chinese version)

接着上一条更新一下, 查了 PR #193 ("Gpt image 2 support") 才发现 generate_image_asset 已经是 agent 的 tool 了 (packages/core/src/tools/generate-image-asset.ts), 7 个 purpose, Settings 里默认关。所以那条思路准确说是在已有 tool 之上做 orchestration loop, 不是加新工具, 改动更小, 完全复用 PR #193 的基础设施。这条放 Phase 2, 等 Phase 1 落了再单开 Discussion。

回 bot 那三个澄清, 聚焦 Phase 1:

• 输入: AI 生成的 UI mockup 图 (gpt image 2 / lovart / 截图都算)。地基已经在 prompt-context.ts:148 跟 index.ts:682 通过 userImages 通路上, 缺产出端的结构化。

• 输出: 一个 ui_kits/<slug>/ 目录, 形态如上 (mock 数据)。这样 Claude Code / Cursor 拿过去就能直接接到真实代码里。

• 粒度: Phase 1 只做单图 → 单 ui_kit。多页 flow / 路由 / 状态机 → Phase 3 单开 issue。

实现严格抄现有 pattern: decompose-to-ui-kit.ts 跟 done.ts / generate-image-asset.ts 一致 (factory + AgentTool + typebox + 邻近 .test.ts), trigger 抄 polishPrompt.ts 但用户显式触发不 auto-fire, 不引入新依赖, decomposition 走 prompt-driven, 输出带 schemaVersion 方便下游 coding agent。

下一步: 先在 fork 上跑 PoC, Remotion 渲个 before/after side-by-side 视频回贴这条 thread 再开 PR。

想先确认一个二选一: ui_kits/<slug>/ 是 (a) 内存里只在 Files panel + 走现有 ZIP export (编码 agent 解压就用, 不动 SQLite schema), 还是 (b) 持久化到磁盘 design workspace (走 schemaVersion bump 按 CLAUDE.md 要求)。我倾向 (a), blast radius 小, 跟 polishPrompt.ts 二轮深化只改 in-memory artifact 是一致的。

[HomenShum]

Status update on the Phase 1 PoC I committed to in the previous comment.

TL;DR: built it on a fork, validated cross-tier with real runs, opened PR #241. Self-answering my own open binary in option (a) (in-memory + Files panel + ZIP export, no SQLite schema bump). Surfaced 3 systemic dependencies during dogfood that I'm filing as separate Discussions, not bundling into this PR.

What landed

3 new agent tools in packages/core/src/tools/, all mirroring existing patterns (done.ts / generate-image-asset.ts):

• decompose-to-ui-kit.ts — emits ui_kits/<slug>/{index.html, components/*.tsx, tokens.css, manifest.json, README.md} with schemaVersion: 1
• verify-ui-kit-parity.ts — deterministic verifier (element count, visible-text coverage, token coverage) — no LLM, no floats
• verify-ui-kit-visual-parity.ts — vision judge wrapped in 12-question boolean rubric across 5 dimensions; parityScore = passCount / totalChecks derived deterministically; bounded-enum status (verified | needs_review | needs_iteration | failed | unavailable)

Plus host-side BrowserWindow.capturePage() renderer + judge dispatcher in apps/desktop/src/main/, sidebar AddMenu trigger, locale-aware EN/ZH prompt walking the agent through decompose → verify → reconcile → iterate (max 2) → done. Per-tool-call cost surfaces inline as a toast.

PR #241 has the full breakdown.

Visual proof (real runs, not mocks)

Side-by-side hero — e2e-opus-final run, parityScore 0.90:

4-frame reconcile reel — e2e-nodebench-iter showing iter-0 (0.82) → iter-1 (0.78) honest score drift. Agent fixed some gaps but introduced new layout drift; the boolean rubric exposes the regression instead of hiding it. This is the intended behavior, not a bug:

BENCHMARKS.md has the full cross-tier methodology + per-run results (Opus / Pro+Pro+iterate / Kimi+Gemini3 / NodeBench iter), reproducibility instructions, and research citations (WebDevJudge, Prometheus-Vision, Trust-but-Verify ICCV 2025).

A live-recorded session demo (real Electron app, no slideshow stitching) is in progress and will be added to the PR description when ready.

Self-answer to my own open binary

in-memory + Files panel + ZIP export (option a) vs persisted-to-disk + schemaVersion bump (option b)

Going with (a). Rationale held up under implementation: smaller blast radius, consistent with polishPrompt.ts second-pass mutating only in-memory artifact, no SQLite migration, downstream coding agents (Claude Code / Cursor) unzip the bundle and treat it as filesystem input regardless of how Open CoDesign stored it internally. Easy to upgrade to (b) later if a multi-design "library of ui_kits" use case emerges.

Three systemic dependencies surfaced during dogfood (filing as Discussions)

End-to-end usage in a real session exposed three failure modes that this PR doesn't fix and shouldn't try to:

1. Spiral failures — agent kept retrying find-and-insert against fragile whitespace, broke the file structure, then tried to fix the breakage and broke it more. ~14min of wasted compute. No /rollback exists.
2. Rate-limit death — 4 consecutive 429 Provider returned error... is temporarily rate-limited upstream on deepseek-v4-flash and deepseek-v4-pro (free tier limits). Agent gave up. No auto-failover to next capable model.
3. Lesson amnesia — even after correction, the agent has no way to remember "last time find-and-insert against this file's whitespace failed, prefer read-then-replace" on the next turn.

Each deserves its own design conversation, not a bundled implementation. Filed 3 separate Discussions in Ideas:

• #242 — Time-travel chat: snapshot + rollback + lessons subsystem
• #243 — Capability-aware multi-provider failover with budget gates
• #244 — Spiral detector: auto-pause after N consecutive non-progressive turns

These are explicitly not blocking for the Phase 1 decompose PR — that ships standalone. They're prereqs for the decompose flow to be robust under hands-off async use, which is a different bar.

What I'd value most from review

1. Whether the boolean rubric design (12 checks, 5 dimensions, threshold-derived status enum) is the shape you'd want, or if you'd push toward fewer/more checks
2. Whether the host-injection pattern for RenderUiKitFn + JudgeVisualParityFn matches your preference vs an alternative coupling
3. Whether option (a) (in-memory) for ui_kits/<slug>/ output is right, or you'd push for (b) (persisted) before merge
4. PR size feedback — happy to split into 3 separate PRs (per-tool) if the ~1500 LOC feels too big for one review

Branch is 11 commits behind upstream main (mostly chore/deps including the pi-agent-core 0.67.68 → 0.70.2 bump). I held off on rebasing to keep the as-built state visible for review; will rebase + adjust for the pi-ai 0.70.2 changes on request.

中文版本 (Chinese version)

接 上一条 Phase 1 PoC 的状态更新。

TL;DR：fork 上跑通了，跨 tier 真数据验证完，开了 PR #241。自答上次留的二选一是 (a) 内存 + Files panel + ZIP export，不动 SQLite schema。Dogfood 过程中暴露 3 条系统级依赖，单独开 Discussion 不塞这个 PR。

已落地 — 3 个 agent tool，全部抄现有 pattern (done.ts / generate-image-asset.ts)：

• decompose-to-ui-kit.ts — 输出 ui_kits/<slug>/{index.html, components/*.tsx, tokens.css, manifest.json, README.md}，带 schemaVersion: 1
• verify-ui-kit-parity.ts — 确定性 verifier（元素数对齐、可见文本覆盖、token 覆盖），无 LLM 无浮点
• verify-ui-kit-visual-parity.ts — vision judge 包一层 12 项 boolean rubric / 5 个维度；parityScore = passCount / totalChecks 确定性推导；状态用 bounded enum (verified | needs_review | needs_iteration | failed | unavailable)

加上 host 侧的 BrowserWindow.capturePage() renderer + judge dispatcher、sidebar AddMenu 入口、EN/ZH 双语 prompt 走 decompose → verify → reconcile → iterate (max 2) → done 流程。每次 tool call 的成本 toast 实时显示。

详细 breakdown 在 PR #241。

视觉证据（真实跑出来的，不是 mock）：

• 并排 hero — e2e-opus-final run，parityScore 0.90（见上面英文版图）
• iter reel — e2e-nodebench-iter iter-0 (0.82) → iter-1 (0.78)，agent 修了一些 gap 但引入新的 layout drift，boolean rubric 把 regression 直接 surface 不藏，这是设计意图不是 bug

BENCHMARKS.md 有完整方法论 + 跨 tier 单跑结果 + 复现说明 + 研究引用。

实景录制（真 Electron app 不是切片拼接）正在录，好了贴到 PR 描述里。

Dogfood 暴露的 3 条系统级依赖（单独开 Discussion，不塞这个 PR）：

1. Spiral 失败 — agent 反复对脆弱空白做 find-and-insert，破坏文件结构后试图修复又破坏更多，浪费 ~14 分钟。没有 /rollback
2. Rate-limit 死锁 — deepseek-v4-flash 和 deepseek-v4-pro 连续 4 次 429（免费 tier 限制），agent 直接放弃。没有自动 failover
3. Lesson 失忆 — 即使被纠正，agent 没法记住「上次对这个文件做 find-and-insert 失败了，下次先 read 再 replace」

每条都值得单独讨论，不要打包实现。已开 3 条 Discussion 在 Ideas：

• #242 — Time-travel chat: snapshot + rollback + lessons
• #243 — Capability-aware multi-provider failover
• #244 — Spiral detector: auto-pause

这些 不阻塞 Phase 1 decompose PR — 那个独立 ship。它们是 decompose 在 无人值守 async 场景下要 robust 的前置依赖，是另一个 bar。

最希望评审的方向：

1. boolean rubric 的形态（12 项 / 5 维 / 阈值推导 status）是不是你想要的，要不要增减
2. host 注入模式（RenderUiKitFn + JudgeVisualParityFn）跟你的偏好是否一致
3. ui_kits/<slug>/ 走 (a) 内存 vs (b) 持久化的判断 — merge 前需不需要切到 (b)
4. PR 大小（~1500 LOC）是不是过大 — 可以拆成 3 个 PR（每个 tool 一个）

分支落后 upstream main 11 commits（主要是 chore/deps 包括 pi-agent-core 0.67.68 → 0.70.2）。先没 rebase 是想保留 as-built 状态便于评审，需要的话立刻 rebase + 适配 pi-ai 0.70.2 的改动。

[chenjunyu-1990]

大佬，牛逼！我去试试

[github-actions]

感谢你的反馈！目前主干的公开代码中已有 generate_image_asset 工具（packages/core/src/tools/generate-image-asset.ts），支持基于提示生成图像（hero、产品图等），但尚未包含自动将截图/生成图像拆解为可复用组件并串联成交互原型的功能。

好消息是，贡献者 @HomenShum 已经提交了 PR #241，在其中实现了 3 个新工具：

• decompose-to-ui-kit — 将 UI 截图分解为组件、token、workflow
• verify-ui-kit-parity / verify-ui-kit-visual-parity — 验证与迭代

PR 目前处于 review 阶段，合并后将拥有你描述的“UI 图片 → 组件化 → 原型”核心能力。

如果你想现在就尝试，可以关注 PR #241 的进展，或直接在 fork 上体验。未来路线图还包括多页面流程、状态机等更高级的原型编排（Phase 3）。

再次感谢提出这个方向！

---

Open-CoDesign Bot