Merge pull request #5 from ssdeanx/develop

ssdeanx · web-flow · commit cefa8f11dd28 · 2025-12-05T12:48:01.000-05:00
feat: update AGENTS and README metadata, enhance chat API
diff --git a/AGENTS.md b/AGENTS.md
@@ -1,9 +1,9 @@
-<!-- AGENTS-META {"title":"Mastra Root","version":"2.2.0","applies_to":"/","last_updated":"2025-11-27T00:00:00Z","status":"stable"} -->
+<!-- AGENTS-META {"title":"Mastra Root","version":"2.3.0","applies_to":"/","last_updated":"2025-12-05T00:00:00Z","status":"stable"} -->
 # AGENTS
 
 ## Project Overview
 
-Mastra is a production-grade multi-agent framework for building agent-driven applications and RAG (retrieval-augmented generation) workflows. It provides **30+ enterprise tools**, **22+ specialized agents**, **10 workflows**, **4 agent networks**, **A2A/MCP orchestration**, and now a **complete UI component library** (49 components) for scalable AI systems. Key capabilities include **financial intelligence**, **RAG pipelines**, **observability**, **secure governance**, and **AI chat interfaces**.
+Mastra is a production-grade multi-agent framework for building agent-driven applications and RAG (retrieval-augmented generation) workflows. It provides **34+ enterprise tools**, **38 specialized agents**, **10 workflows**, **4 agent networks**, **A2A/MCP orchestration**, and a **complete UI component library** (64 components: 30 AI Elements + 34 base UI) for scalable AI systems. Key capabilities include **financial intelligence**, **RAG pipelines**, **observability**, **secure governance**, and **AI chat interfaces**.
 
 This repo is structured to keep tools, agents, workflows, networks, UI components, and configs separated, with strict Zod schemas for tool inputs/outputs and strong environment-based configuration in `src/mastra/config`.
 
@@ -89,9 +89,9 @@ NEXT_PUBLIC_MASTRA_API_URL=http://localhost:4111
 
 ## Architecture & conventions
 
-- **Frontend** (`app/`, `ui/`, `src/components/ai-elements/`): Next.js 16 App Router with React 19. AI Elements (30 components) for chat/reasoning/canvas UIs. shadcn/ui base (19 components) in `ui/`. Tailwind CSS 4 with oklch color variables.
-- **Tools** (`src/mastra/tools`): 30+ tools implementing `createTool({ id, inputSchema, outputSchema, execute })` with strict Zod schemas. Categories: Financial (Polygon, Finnhub, AlphaVantage), Research (SerpAPI, ArXiv), Data (CSV, JSON), RAG (chunking, embeddings).
-- **Agents** (`src/mastra/agents`): 22+ agents composing tools into specialized behaviors (research, stock analysis, content creation, data processing).
+- **Frontend** (`app/`, `ui/`, `src/components/ai-elements/`): Next.js 16 App Router with React 19. AI Elements (30 components) for chat/reasoning/canvas UIs. shadcn/ui base (34 components) in `ui/`. Tailwind CSS 4 with oklch color variables.
+- **Tools** (`src/mastra/tools`): 34+ tools implementing `createTool({ id, inputSchema, outputSchema, execute })` with strict Zod schemas. Categories: Financial (Polygon, Finnhub, AlphaVantage), Research (SerpAPI, ArXiv), Data (CSV, JSON), RAG (chunking, embeddings).
+- **Agents** (`src/mastra/agents`): 38 agents composing tools into specialized behaviors (research, stock analysis, content creation, data processing, business/legal, charting, image processing).
 - **Networks** (`src/mastra/networks`): 4 agent networks for routing and orchestration (agentNetwork, dataPipelineNetwork, reportGenerationNetwork, researchPipelineNetwork).
 - **Workflows** (`src/mastra/workflows`): 10 multi-step workflows using Mastra DSL (weather, content, financial reports, document processing, research synthesis).
 - **Config** (`src/mastra/config`): Centralized provider clients (Google, OpenAI, Anthropic, OpenRouter, Vertex), pg-storage with PgVector, and role hierarchy.
@@ -120,11 +120,11 @@ NEXT_PUBLIC_MASTRA_API_URL=http://localhost:4111
 ## Where to look for more info
 
 - `app/AGENTS.md`: Next.js App Router pages and layouts
-- `ui/AGENTS.md`: shadcn/ui base components (19 components)
+- `ui/AGENTS.md`: shadcn/ui base components (34 components)
 - `src/components/ai-elements/AGENTS.md`: AI Elements library (30 components)
 - `src/mastra/AGENTS.md`: top-level code-agent focused docs (this file is mirrored to subfolders)
-- `src/mastra/tools/AGENTS.md`: 30+ tools and their patterns
-- `src/mastra/agents/AGENTS.md`: 22+ agents catalog
+- `src/mastra/tools/AGENTS.md`: 34+ tools and their patterns
+- `src/mastra/agents/AGENTS.md`: 38 agents catalog
 - `src/mastra/workflows/AGENTS.md`: 10 workflow definitions
 - `src/mastra/networks/AGENTS.md`: 4 agent networks
 - `src/mastra/config/AGENTS.md`: configuration and storage guidance
@@ -135,4 +135,4 @@ NEXT_PUBLIC_MASTRA_API_URL=http://localhost:4111
 If you need more details for a subdirectory, open the folder-specific `AGENTS.md` which contains persona, purpose, and actionable commands.
 
 ---
-Last updated: 2025-11-27
+Last updated: 2025-12-05
diff --git a/README.md b/README.md
@@ -1,4 +1,4 @@
-<!-- AGENTS-META {"title":"AgentStack README","version":"3.2.0","applies_to":"/","last_updated":"2025-11-27T00:00:00Z","status":"stable"} -->
+<!-- AGENTS-META {"title":"AgentStack README","version":"3.3.0","applies_to":"/","last_updated":"2025-12-05T00:00:00Z","status":"stable"} -->
 
 <div align="center">
 
@@ -13,7 +13,7 @@
 [![wakatime](https://wakatime.com/badge/user/7a2fb9a0-188b-4568-887f-7645f9249e62/project/e44412f3-9bcc-4661-b79d-23160d90dfe0.svg)](https://wakatime.com/badge/user/7a2fb9a0-188b-4568-887f-7645f9249e62/project/e44412f3-9bcc-4661-b79d-23160d90dfe0)
 [![GitMCP](https://img.shields.io/endpoint?url=https://gitmcp.io/badge/ssdeanx/AgentStack)](https://gitmcp.io/ssdeanx/AgentStack)
 
-**AgentStack** is a **production-grade multi-agent framework** built on Mastra, delivering **30+ enterprise tools**, **25+ specialized agents**, **10 workflows**, **4 agent networks**, **49 UI components**, and **A2A/MCP orchestration** for scalable AI systems. Focuses on **financial intelligence**, **RAG pipelines**, **observability**, **secure governance**, and **AI chat interfaces**.
+**AgentStack** is a **production-grade multi-agent framework** built on Mastra, delivering **34+ enterprise tools**, **38 specialized agents**, **10 workflows**, **4 agent networks**, **64 UI components** (30 AI Elements + 34 base), and **A2A/MCP orchestration** for scalable AI systems. Focuses on **financial intelligence**, **RAG pipelines**, **observability**, **secure governance**, and **AI chat interfaces**.
 
 [![@mastra/core](https://img.shields.io/npm/v/@mastra/core.svg)](https://www.npmjs.com/package/@mastra/core)
 [![@mastra/pg](https://img.shields.io/npm/v/@mastra/pg.svg)](https://www.npmjs.com/package/@mastra/pg)
@@ -44,7 +44,7 @@
 | **Multi-Agent**              | ✅ **A2A MCP + parallel orchestration**              | ⚠️ Sequential | ✅ Sequential | ✅ Custom  |
 | **Governance**               | ✅ **JWT/RBAC + path traversal + HTML sanitization** | ❌ Custom     | ❌ None       | ❌ None    |
 | **TypeScript**               | ✅ **Zod schemas everywhere**                        | ⚠️ JS/TS mix  | ⚠️ JS focus   | ❌ Python  |
-| **UI Components**            | ✅ **49 AI Elements + shadcn/ui**                    | ❌ None       | ❌ None       | ❌ None    |
+| **UI Components**            | ✅ **64 AI Elements + shadcn/ui**                    | ❌ None       | ❌ None       | ❌ None    |
 | **Tests**                    | ✅ **97% Vitest coverage**                           | ⚠️ Partial    | ❌ Sparse     | ⚠️ Partial |
 
 **Built for production**: Secure, observable, testable agents with **zero-config** PgVector RAG + **enterprise financial APIs**.
@@ -53,11 +53,11 @@
 
 - **💰 Financial Intelligence**: 20+ tools (Polygon quotes/aggs/fundamentals, Finnhub analysis, AlphaVantage indicators)
 - **🔍 Semantic RAG**: PgVector (3072D embeddings) + MDocument chunking + rerank + graph traversal
-- **🤖 25+ Agents**: Research → Learn → Report → Edit → Analyze (stock/crypto/copywriter/evaluator/data pipeline)
+- **🤖 38 Agents**: Research → Learn → Report → Edit → Analyze (stock/crypto/copywriter/evaluator/data pipeline/business-legal/charting/image)
 - **📋 10 Workflows**: Weather, content, financial reports, document processing, research synthesis, learning extraction
 - **🌐 4 Agent Networks**: Primary routing, data pipeline, report generation, research pipeline
 - **🔌 A2A/MCP**: MCP server coordinates parallel agents (research+stock→report)
-- **🎨 49 UI Components**: AI Elements (30 chat/reasoning/canvas components) + shadcn/ui (19 base primitives)
+- **🎨 64 UI Components**: AI Elements (30 chat/reasoning/canvas components) + shadcn/ui (34 base primitives)
 - **📊 Full Observability**: Arize/Phoenix traces + 10+ custom scorers (diversity/quality/completeness)
 - **🛡️ Enterprise Security**: JWT auth, RBAC, path validation, HTML sanitization, secrets masking
 - **⚡ Extensible**: Model registry (Gemini/OpenAI/Anthropic), Zod schemas everywhere
@@ -67,7 +67,7 @@
 ```mermaid
 graph TB
     subgraph "� Frontend (Next.js 16)"
-        UI[AI Elements + shadcn/ui<br/>• 30 AI Components<br/>• 19 Base Primitives]
+        UI[AI Elements + shadcn/ui<br/>• 30 AI Components<br/>• 34 Base Primitives]
         App[App Router<br/>• React 19<br/>• Tailwind CSS 4]
     end
 
@@ -76,8 +76,8 @@ graph TB
     end
 
     subgraph "🎯 AgentStack Runtime"
-        Coord --> Agents[22+ Agents<br/>• ResearchAgent<br/>• StockAnalysis<br/>• Copywriter<br/>• ReportAgent]
-        Agents --> Tools[30+ Tools<br/>• Polygon/Finnhub<br/>• SerpAPI 10+<br/>• PgVector RAG<br/>• PDF→MD]
+        Coord --> Agents[38 Agents<br/>• ResearchAgent<br/>• StockAnalysis<br/>• Copywriter<br/>• ReportAgent]
+        Agents --> Tools[34+ Tools<br/>• Polygon/Finnhub<br/>• SerpAPI 10+<br/>• PgVector RAG<br/>• PDF→MD]
         Agents --> Workflows[Research→Report<br/>Weather→Activities]
     end
 
@@ -212,14 +212,14 @@ npm run start
 ```bash
 # Frontend
 app/                      # 📱 Next.js 16 App Router (layouts, pages)
-ui/                       # 🎨 shadcn/ui base components (19 primitives)
+ui/                       # 🎨 shadcn/ui base components (34 primitives)
 src/components/ai-elements/  # 🤖 AI Elements (30 chat/reasoning/canvas components)
 
 # Backend
 src/mastra/
-├── index.ts              # 🎯 Mastra bootstrap (25+ agents, 10 workflows, 4 networks, MCP)
-├── agents/               # 🤖 22+ agents (research/stock/copywriter/report/data pipeline...)
-├── tools/                # 🔧 30+ tools (financial/RAG/scrape/PDF/SerpAPI...)
+├── index.ts              # 🎯 Mastra bootstrap (38 agents, 10 workflows, 4 networks, MCP)
+├── agents/               # 🤖 38 agents (research/stock/copywriter/report/data pipeline/business-legal/charting...)
+├── tools/                # 🔧 34+ tools (financial/RAG/scrape/PDF/SerpAPI...)
 ├── workflows/            # 📋 10 workflows (weather, content, financial, document, research)
 ├── networks/             # 🌐 4 agent networks (routing/coordination)
 ├── config/               # ⚙️ Models/PgVector/Logging/Auth
@@ -333,10 +333,10 @@ npm run mcp-server
 
 ## 📚 **Resources**
 
-- **[UI Components](ui/AGENTS.md)**: 19 shadcn/ui base components
+- **[UI Components](ui/AGENTS.md)**: 34 shadcn/ui base components
 - **[AI Elements](src/components/ai-elements/AGENTS.md)**: 30 AI chat/reasoning/canvas components
-- **[Agents Catalog](src/mastra/agents/AGENTS.md)**: 22+ agents
-- **[Tools Matrix](src/mastra/tools/AGENTS.md)**: 30+ tools
+- **[Agents Catalog](src/mastra/agents/AGENTS.md)**: 38 agents
+- **[Tools Matrix](src/mastra/tools/AGENTS.md)**: 34+ tools
 - **[Workflows](src/mastra/workflows/AGENTS.md)**: 10 multi-step workflows
 - **[Networks](src/mastra/networks/AGENTS.md)**: 4 agent networks
 - **[Config Guide](src/mastra/config/AGENTS.md)**: Setup + env vars
@@ -350,7 +350,7 @@ npm run mcp-server
 - [x] **A2A MCP**: Parallel orchestration (✅ Live)
 - [x] **10 Workflows**: Sequential, parallel, branch, loop, foreach, suspend/resume (✅ Live)
 - [x] **4 Agent Networks**: Routing and coordination (✅ Live)
-- [x] **UI Components**: AI Elements + shadcn/ui (49 components) (✅ Live)
+- [x] **UI Components**: AI Elements + shadcn/ui (64 components) (✅ Live)
 - [ ] **Chat Interface**: Full agent chat UI with AI Elements
 - [ ] **LangSmith/Phoenix**: Eval dashboards
 - [ ] **Docker/Helm**: K8s deploy
@@ -362,4 +362,4 @@ npm run mcp-server
 🐦 **Follow [@ssdeanx](https://x.com/ssdeanx)**
 📘 **[Docs](https://agentstack.ai)** (Coming Q1 2026)
 
-_Last updated: 2025-11-27 | v3.2.0_
+_Last updated: 2025-12-05 | v3.3.0_
diff --git a/app/api/chat/route.ts b/app/api/chat/route.ts
@@ -1,64 +1,21 @@
-import { mastra } from "@/src/mastra";
-import { createAgentStreamResponse } from "@/lib/client-stream-to-ai-sdk";
-import type { UIMessage } from "ai";
-
-export const maxDuration = 60;
-
-interface ChatRequestBody {
-  messages: UIMessage[];
-  agentId?: string;
-  threadId?: string;
-  resourceId?: string;
-  memory?: {
-    thread?: string | { id: string; resourceId?: string };
-    resource?: string;
-    options?: {
-      lastMessages?: number;
-      semanticRecall?: boolean;
-      workingMemory?: { enabled?: boolean };
-    };
-  };
-  maxSteps?: number;
-}
+import { mastra } from "../../../src/mastra";
+import { RuntimeContext } from "@mastra/core/runtime-context";
 
 export async function POST(req: Request) {
-  const body: ChatRequestBody = await req.json();
-  
-  // Get available agents dynamically from mastra
-  const agentsMap = await mastra.getAgents();
-  const availableAgents = Object.keys(agentsMap);
-  
-  // Use first available agent if none specified
-  const agentId = body.agentId || availableAgents[0];
-  
-  if (!agentId || !availableAgents.includes(agentId)) {
-    return Response.json(
-      { error: `Invalid or missing agentId. Available: ${availableAgents.join(", ")}` },
-      { status: 400 }
-    );
-  }
+  const { messages, data } = await req.json();
+  const myAgent = mastra.getAgent("weatherAgent");
 
-  if (!body.messages?.length) {
-    return Response.json({ error: "messages required" }, { status: 400 });
-  }
-  
-  try {
-    return await createAgentStreamResponse(mastra as Parameters<typeof createAgentStreamResponse>[0], agentId, body.messages, {
-      threadId: body.threadId,
-      resourceId: body.resourceId,
-      memory: body.memory,
-      maxSteps: body.maxSteps ?? 50,
-    });
-  } catch (error) {
-    return Response.json(
-      { error: error instanceof Error ? error.message : "Stream failed" },
-      { status: 500 }
-    );
+  const runtimeContext = new RuntimeContext();
+
+  if (data) {
+    for (const [key, value] of Object.entries(data)) {
+      runtimeContext.set(key, value);
+    }
   }
-}
 
-export async function GET() {
-  const agentsMap = await mastra.getAgents();
-  const availableAgents = Object.keys(agentsMap);
-  return Response.json({ agents: availableAgents, count: availableAgents.length });
-}
+  const stream = await myAgent.stream(messages, {
+    runtimeContext,
+    format: "aisdk",
+  });
+  return stream.toUIMessageStreamResponse();
+}
diff --git a/lib/client-stream-to-ai-sdk.ts b/lib/client-stream-to-ai-sdk.ts
@@ -3,6 +3,8 @@ import {
   createUIMessageStreamResponse,
 } from "ai";
 import { toAISdkFormat } from "@mastra/ai-sdk";
+import type { Mastra } from "@mastra/core/mastra";
+import type { AgentExecutionOptions } from "@mastra/core/agent";
 import type { MastraModelOutput } from "@mastra/core/stream";
 
 export interface StreamToAISdkOptions {
@@ -16,34 +18,34 @@ export interface AgentStreamOptions {
   format?: "aisdk" | "mastra";
   threadId?: string;
   resourceId?: string;
-  memory?: {
-    thread?: string | { id: string; resourceId?: string };
-    resource?: string;
-    options?: {
-      lastMessages?: number;
-      semanticRecall?: boolean;
-      workingMemory?: { enabled?: boolean };
-    };
-  };
+  memory?: AgentExecutionOptions["memory"];
   maxSteps?: number;
 }
 
-type MastraAgent = {
-  stream: (
-    messages: unknown,
-    options?: {
-      format?: string;
-      threadId?: string;
-      resourceId?: string;
-      memory?: AgentStreamOptions["memory"];
-      maxSteps?: number;
-    }
-  ) => Promise<MastraModelOutput & { toUIMessageStreamResponse?: () => Response }>;
-};
+type StreamResult = MastraModelOutput & { toUIMessageStreamResponse?: () => Response };
+
+function isReadableStream<T>(value: unknown): value is ReadableStream<T> {
+  return (
+    typeof value === "object" &&
+    value !== null &&
+    typeof (value as ReadableStream<T>).getReader === "function"
+  );
+}
 
-type MastraInstance = {
-  getAgent: (id: string) => MastraAgent;
-} & Record<string, any>;
+async function* asyncIterableFromReadableStream<T>(
+  stream: ReadableStream<T>
+): AsyncIterable<T> {
+  const reader = stream.getReader();
+  try {
+    while (true) {
+      const { done, value } = await reader.read();
+      if (done) break;
+      yield value;
+    }
+  } finally {
+    reader.releaseLock();
+  }
+}
 
 /**
  * Creates a streaming Response for Next.js API routes using server-side Mastra agent.
@@ -55,7 +57,7 @@ type MastraInstance = {
  * ```ts
  * // app/api/chat/route.ts
  * import { mastra } from "@/src/mastra";
- * import { createAgentStreamResponse } from "@/lib/client-stream-to-ai-sdk";
+ * import { createAgentStreamResponse } from "@/lib/server/agent-stream";
  * 
  * export async function POST(req: Request) {
  *   const { messages, agentId, threadId, resourceId, memory } = await req.json();
@@ -70,7 +72,7 @@ type MastraInstance = {
  * @see https://mastra.ai/docs/frameworks/agentic-uis/ai-sdk
  */
 export async function createAgentStreamResponse(
-  mastra: MastraInstance,
+  mastra: Mastra,
   agentId: string,
   messages: unknown,
   options?: AgentStreamOptions
@@ -85,34 +87,27 @@ export async function createAgentStreamResponse(
     maxSteps: options?.maxSteps,
   };
 
-  // Preferred: Use built-in AI SDK format
-  if (streamOptions.format === "aisdk") {
-    const stream = await agent.stream(messages, streamOptions);
-    if (stream.toUIMessageStreamResponse) {
-      return stream.toUIMessageStreamResponse();
-    }
+  // Call agent.stream once and reuse the result
+  const stream = await agent.stream(messages, streamOptions) as StreamResult;
+
+  // Preferred: Use built-in AI SDK format when available
+  if (streamOptions.format === "aisdk" && stream.toUIMessageStreamResponse) {
+    return stream.toUIMessageStreamResponse();
   }
 
   // Fallback: Manual transformation with toAISdkFormat
-  const stream = await agent.stream(messages, {
-    threadId: options?.threadId,
-    resourceId: options?.resourceId,
-    memory: options?.memory,
-    maxSteps: options?.maxSteps,
-  });
-  
+  // Handles both ReadableStream and AsyncIterable return types
   const uiMessageStream = createUIMessageStream({
     execute: async ({ writer }) => {
-      const aiSdkStream = toAISdkFormat(stream, { from: "agent" });
-      const reader = aiSdkStream.getReader();
-      try {
-        while (true) {
-          const { done, value } = await reader.read();
-          if (done) break;
-          writer.write(value);
-        }
-      } finally {
-        reader.releaseLock();
+      const aiSdkResult = toAISdkFormat(stream, { from: "agent" });
+      
+      // Handle both ReadableStream and AsyncIterable
+      const iterable: AsyncIterable<unknown> = isReadableStream(aiSdkResult)
+        ? asyncIterableFromReadableStream(aiSdkResult)
+        : aiSdkResult;
+      
+      for await (const value of iterable) {
+        writer.write(value as Parameters<typeof writer.write>[0]);
       }
     },
   });
diff --git a/lib/mastra-client.ts b/lib/mastra-client.ts
@@ -1,5 +1,12 @@
 import { MastraClient } from "@mastra/client-js";
 
+/**
+ * Client-side Mastra SDK instance for frontend use.
+ * Use this in React components to interact with the Mastra API.
+ * 
+ * For server-side streaming in API routes, import createAgentStreamResponse
+ * directly from "@/lib/client-stream-to-ai-sdk" instead.
+ */
 export const mastraClient = new MastraClient({
   baseUrl: process.env.NEXT_PUBLIC_MASTRA_API_URL || "http://localhost:4111",
   retries: 3,
@@ -9,6 +16,3 @@ export const mastraClient = new MastraClient({
   headers: {},
   credentials: "same-origin",
 });
-
-export { createAgentStreamResponse } from "./client-stream-to-ai-sdk";
-export type { StreamToAISdkOptions } from "./client-stream-to-ai-sdk";
