Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/grok-imagine-video-adapter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@tanstack/ai-grok': minor
---

Add a `grokVideo` adapter for the grok-imagine video models (`grok-imagine-video`, `grok-imagine-video-1.5-preview`) via xAI's Imagine API. Follows the experimental `generateVideo()` jobs/polling architecture: `createVideoJob` posts to `/v1/videos/generations`, status polling reads `/v1/videos/{request_id}`, and the completed result carries the hosted video URL plus usage (`unitsBilled` seconds and exact `cost` in USD). Sizing uses the aspect-ratio template consistent with the grok-imagine image models (`size: '16:9_720p'` β†’ `aspect_ratio` / `resolution`), durations are 1–15 integer seconds, and image-to-video starting frames can be passed via `modelOptions.image: { url }`.
62 changes: 60 additions & 2 deletions docs/adapters/grok.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,20 @@
title: Grok (xAI)
id: grok-adapter
order: 5
description: "Use xAI Grok models with TanStack AI β€” Grok 4.1, Grok 4, Grok 3, and Grok 2 Image generation via @tanstack/ai-grok."
description: "Use xAI Grok models with TanStack AI β€” Grok 4.1, Grok 4, Grok 3, Grok 2 Image generation, and Grok Imagine video generation via @tanstack/ai-grok."
keywords:
- tanstack ai
- grok
- xai
- grok 4
- grok 4.1
- image generation
- video generation
- grok imagine
- adapter
---

The Grok adapter provides access to xAI's Grok models, including Grok 4.1, Grok 4, Grok 3, and image generation with Grok 2 Image.
The Grok adapter provides access to xAI's Grok models, including Grok 4.1, Grok 4, Grok 3, image generation with Grok 2 Image, and video generation with the Grok Imagine video models.

## Installation

Expand Down Expand Up @@ -160,6 +162,58 @@ const result = await generateImage({
console.log(result.images);
```

## Video Generation (Experimental)

Generate short video clips (1–15 seconds, with audio) with the Grok Imagine video models via xAI's asynchronous jobs/polling API:

```typescript
import { generateVideo, getVideoJobStatus } from "@tanstack/ai";
import { grokVideo } from "@tanstack/ai-grok";

const adapter = grokVideo("grok-imagine-video");

// 1. Create the job
const { jobId } = await generateVideo({
adapter,
prompt: "A red panda balancing on a bamboo stalk in the rain",
size: "16:9_720p", // "aspectRatio" or "aspectRatio_resolution"
duration: 5, // integer seconds, 1–15
});

// 2. Poll until complete, then read the video URL
let status = await getVideoJobStatus({ adapter, jobId });
while (status.status !== "completed" && status.status !== "failed") {
await new Promise((r) => setTimeout(r, 5000));
status = await getVideoJobStatus({ adapter, jobId });
}

console.log(status.url); // hosted .mp4 URL
```

Available models:

- `grok-imagine-video` β€” text-to-video and image-to-video, $0.05 per second of video
- `grok-imagine-video-1.5-preview` β€” preview of the next model generation, $0.08 per second

Like the Grok Imagine image models, sizing is aspect-ratio based: the `size` option takes an `aspectRatio_resolution` template. Supported aspect ratios are `1:1`, `16:9`, `9:16`, `4:3`, `3:4`, `3:2`, and `2:3`; supported resolutions are `480p`, `720p`, and `1080p` (e.g. `"9:16_1080p"`). The resolution suffix is optional.

For image-to-video, pass a starting frame via `modelOptions.image` β€” a public URL or a base64 data URI:

```typescript
const { jobId } = await generateVideo({
adapter: grokVideo("grok-imagine-video"),
prompt: "Make the waterfall crash down and slowly pan out the camera",
duration: 10,
modelOptions: {
image: { url: "https://example.com/waterfall-still.png" },
},
});
```

When the job completes, the adapter reports usage on the result: `usage.unitsBilled` carries the billed seconds of video and `usage.cost` the exact cost in USD, both as returned by the xAI API.

See [Video Generation](../media/video-generation) for the full jobs/polling flow, streaming mode, and the `useGenerateVideo` hook.

## Text-to-Speech

Generate speech with Grok TTS:
Expand Down Expand Up @@ -263,6 +317,10 @@ Creates a Grok summarization adapter with an explicit API key.

Creates a Grok image generation adapter.

### `grokVideo(model, config?)` / `createGrokVideo(model, apiKey, config?)`

Creates a Grok video generation adapter (experimental) for the Grok Imagine video models (`'grok-imagine-video'`, `'grok-imagine-video-1.5-preview'`).

### `grokSpeech(model, config?)` / `createGrokSpeech(model, apiKey, config?)`

Creates a Grok text-to-speech adapter.
Expand Down
5 changes: 3 additions & 2 deletions docs/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@
"label": "Video Generation",
"to": "media/video-generation",
"addedAt": "2026-04-15",
"updatedAt": "2026-06-08"
"updatedAt": "2026-06-10"
},
{
"label": "Generation Hooks",
Expand Down Expand Up @@ -420,7 +420,8 @@
{
"label": "Grok (xAI)",
"to": "adapters/grok",
"addedAt": "2026-04-15"
"addedAt": "2026-04-15",
"updatedAt": "2026-06-10"
},
{
"label": "Groq",
Expand Down
32 changes: 31 additions & 1 deletion docs/media/video-generation.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,13 @@
title: Video Generation
id: video-generation
order: 6
description: "Generate video from text prompts with OpenAI Sora using TanStack AI's experimental generateVideo() jobs/polling API."
description: "Generate video from text prompts with OpenAI Sora, xAI Grok Imagine, or fal.ai using TanStack AI's experimental generateVideo() jobs/polling API."
keywords:
- tanstack ai
- video generation
- sora
- grok imagine
- fal
- generateVideo
- jobs api
- experimental
Expand Down Expand Up @@ -36,6 +38,8 @@ TanStack AI provides experimental support for video generation through dedicated

Currently supported:
- **OpenAI**: Sora-2 and Sora-2-Pro models (when available)
- **Grok (xAI)**: grok-imagine-video and grok-imagine-video-1.5-preview models
- **fal.ai**: MiniMax, Luma, Kling, Hunyuan, and other hosted video models

## Basic Usage

Expand Down Expand Up @@ -406,6 +410,32 @@ const { jobId } = await generateVideo({
})
```

### Grok (xAI Imagine) Model Options

Based on the [xAI video generation API](https://docs.x.ai/docs/guides/video-generations). The Grok Imagine models are aspect-ratio sized β€” the generic `size` option takes an `aspectRatio_resolution` template (like the Grok Imagine image models), and clips can be 1–15 seconds long:

```typescript
import { generateVideo } from '@tanstack/ai'
import { grokVideo } from '@tanstack/ai-grok'

const { jobId } = await generateVideo({
adapter: grokVideo('grok-imagine-video'),
prompt: 'A beautiful sunset over the ocean',
size: '16:9_720p', // aspect ratio: '1:1' | '16:9' | '9:16' | '4:3' | '3:4' | '3:2' | '2:3'
// resolution (optional suffix): '480p' | '720p' | '1080p'
duration: 5, // integer seconds, 1-15
modelOptions: {
aspect_ratio: '16:9', // Alternative way to specify the aspect ratio
resolution: '720p', // Alternative way to specify the resolution
duration: 5, // Alternative way to specify the duration
// Image-to-video: animate a starting frame (public URL or base64 data URI)
image: { url: 'https://example.com/still.png' },
},
})
```

Generated clips include an audio track. When the job completes, the adapter reports `usage.unitsBilled` (billed seconds of video) and `usage.cost` (exact USD cost as returned by the API) on the result.

## Response Types

> **Note:** The interfaces below are the underlying adapter-level types. The `getVideoJobStatus()` helper returns a single merged object, `{ status, progress?, url?, error?, usage? }` β€” it does not return `jobId` or `expiresAt`.
Expand Down
4 changes: 4 additions & 0 deletions examples/ts-react-media/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,7 @@ FAL_KEY=

# Get a Google API key at https://aistudio.google.com/apikey
GOOGLE_API_KEY=

# Get an xAI API key at https://console.x.ai β€” used by the "xAI Direct"
# Grok Imagine video models (the other Grok Imagine entries go through fal).
XAI_API_KEY=
1 change: 1 addition & 0 deletions examples/ts-react-media/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
"@tanstack/ai": "workspace:*",
"@tanstack/ai-fal": "workspace:*",
"@tanstack/ai-gemini": "workspace:*",
"@tanstack/ai-grok": "workspace:*",
"@tanstack/react-router": "^1.158.4",
"@tanstack/react-start": "^1.159.0",
"@tanstack/router-plugin": "^1.158.4",
Expand Down
46 changes: 35 additions & 11 deletions examples/ts-react-media/src/components/VideoGenerator.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ type JobState =
model: string
progress?: number | undefined
}
| { status: 'completed'; url: string; unitsBilled?: number }
| { status: 'completed'; url: string; unitsBilled?: number; cost?: number }
| { status: 'error'; message: string }

interface VideoGeneratorProps {
Expand All @@ -41,6 +41,8 @@ export default function VideoGenerator({
const pollingRefs = useRef<Map<string, NodeJS.Timeout>>(new Map())

const filteredModels = VIDEO_MODELS.filter((m) => m.mode === mode)
const falModels = filteredModels.filter((m) => m.provider === 'fal')
const xaiModels = filteredModels.filter((m) => m.provider === 'xai')

useEffect(() => {
if (initialImageUrl) {
Expand Down Expand Up @@ -99,6 +101,7 @@ export default function VideoGenerator({
status: 'completed',
url: url,
unitsBilled: urlResult.usage?.unitsBilled,
cost: urlResult.usage?.cost,
},
}))
} else if (status.status === 'processing') {
Expand Down Expand Up @@ -159,8 +162,11 @@ export default function VideoGenerator({
},
}))

// Poll keyed by the UI model id, not result.model: the direct-xAI
// entries share one adapter model ('grok-imagine-video-1.5-preview'),
// so result.model wouldn't identify the card (or the adapter) uniquely.
const interval = setInterval(() => {
pollStatus(result.jobId, result.model)
pollStatus(result.jobId, modelId)
}, 4000)
pollingRefs.current.set(modelId, interval)
} catch (err) {
Expand Down Expand Up @@ -244,11 +250,20 @@ export default function VideoGenerator({
className="w-full px-4 py-3 bg-gray-800 border border-gray-700 rounded-lg text-white focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-transparent disabled:opacity-50"
>
<option value="all">All Models</option>
{filteredModels.map((model) => (
<option key={model.id} value={model.id}>
{model.name}
</option>
))}
<optgroup label="fal.ai">
{falModels.map((model) => (
<option key={model.id} value={model.id}>
{model.name}
</option>
))}
</optgroup>
<optgroup label="xAI (direct)">
{xaiModels.map((model) => (
<option key={model.id} value={model.id}>
{model.name}
</option>
))}
</optgroup>
</select>
</div>

Expand Down Expand Up @@ -401,12 +416,21 @@ export default function VideoGenerator({
className="w-full h-auto"
/>
</div>
{state.unitsBilled != null && (
{state.cost != null ? (
<p className="text-xs text-gray-500">
Billed {state.unitsBilled} fal unit
{state.unitsBilled === 1 ? '' : 's'} β€” multiply by the
endpoint unit price for USD cost
Billed ${state.cost.toFixed(3)}
{state.unitsBilled != null
? ` for ${state.unitsBilled} second${state.unitsBilled === 1 ? '' : 's'} of video`
: ''}
</p>
) : (
state.unitsBilled != null && (
<p className="text-xs text-gray-500">
Billed {state.unitsBilled} fal unit
{state.unitsBilled === 1 ? '' : 's'} β€” multiply by the
endpoint unit price for USD cost
</p>
)
)}
</>
)}
Expand Down
22 changes: 22 additions & 0 deletions examples/ts-react-media/src/lib/models.ts
Original file line number Diff line number Diff line change
Expand Up @@ -79,48 +79,70 @@ export const VIDEO_MODELS = [
name: 'Kling 3 Pro (Text-to-Video)',
description: 'High-quality text-to-video generation',
mode: 'text-to-video' as const,
provider: 'fal' as const,
},
{
id: 'fal-ai/kling-video/v3/pro/image-to-video',
name: 'Kling 3 Pro (Image-to-Video)',
description: 'Animate images with Kling',
mode: 'image-to-video' as const,
provider: 'fal' as const,
},
{
id: 'fal-ai/veo3.1',
name: 'Veo 3.1 (Text-to-Video)',
description: 'Google Veo text-to-video',
mode: 'text-to-video' as const,
provider: 'fal' as const,
},
{
id: 'fal-ai/veo3.1/image-to-video',
name: 'Veo 3.1 (Image-to-Video)',
description: 'Google Veo image-to-video',
mode: 'image-to-video' as const,
provider: 'fal' as const,
},
{
id: 'xai/grok-imagine-video/text-to-video',
name: 'Grok Imagine Video (Text-to-Video)',
description: 'xAI video generation from text',
mode: 'text-to-video' as const,
provider: 'fal' as const,
},
{
id: 'xai/grok-imagine-video/image-to-video',
name: 'Grok Imagine Video (Image-to-Video)',
description: 'xAI animate images to video',
mode: 'image-to-video' as const,
provider: 'fal' as const,
},
{
id: 'grok-imagine-video-1.5-preview',
name: 'Grok Imagine Video 1.5 (Text-to-Video)',
description: 'xAI Imagine API via the native grokVideo adapter',
mode: 'text-to-video' as const,
provider: 'xai' as const,
},
{
id: 'grok-imagine-video-1.5-preview/image-to-video',
name: 'Grok Imagine Video 1.5 (Image-to-Video)',
description: 'Animate a starting frame via the native grokVideo adapter',
mode: 'image-to-video' as const,
provider: 'xai' as const,
},
{
id: 'fal-ai/ltx-2.3/text-to-video/fast',
name: 'LTX-2.3 Fast (Text-to-Video)',
description: 'Fast text-to-video generation',
mode: 'text-to-video' as const,
provider: 'fal' as const,
},
{
id: 'fal-ai/ltx-2.3/image-to-video/fast',
name: 'LTX-2.3 Fast (Image-to-Video)',
description: 'Fast image-to-video animation',
mode: 'image-to-video' as const,
provider: 'fal' as const,
},
] as const

Expand Down
Loading
Loading