Skip to content

docs: fix agent test generation page — Antigravity, API Testing section, optional schema #813

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
slayerjain merged 3 commits into from
Mar 30, 2026
Merged

docs: fix agent test generation page — Antigravity, API Testing section, optional schema #813

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
15c8995
docs: fix agent test generation page per review feedback
slayerjain Mar 29, 2026
ae7f8ed
fix: add assertion type names to Vale vocabulary
slayerjain Mar 29, 2026
8dbb70b
fix: accept test-gen not keploy-test-gen, fix respository typo
slayerjain Mar 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 67 additions & 69 deletions ...0.0/keploy-cloud/agent-test-generation.md → ...0/running-keploy/agent-test-generation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
---
id: agent-test-generation
title: Agent-Based Test Generation
sidebar_label: Agent Test Generation
description: Generate API tests using AI coding agents (Claude Code, Cursor, Windsurf) with the keploy-runner CLI
title: Agent-Based API Test Generation
sidebar_label: AI Agent Testing
description: Generate API tests using AI coding agents like Claude Code, Cursor, and Antigravity with Keploy Enterprise
tags:
- AI Agent
- Test Generation
- Claude Code
- Cursor
- Windsurf
- keploy-runner
- CLI
- Antigravity
- API Testing
keywords:
- AI agent test generation
- Claude Code
- Cursor
- keploy-runner
- Antigravity
- Keploy Enterprise
- API testing
- coverage feedback loop
---
Expand All @@ -26,56 +26,52 @@

## Overview

Keploy's agent-based test generation lets AI coding assistants (Claude Code, Cursor, Windsurf, GitHub Copilot) generate and iterate on API tests directly in your IDE. Instead of using a separate web UI, the agent:
Keploy Enterprise supports agent-based test generation, allowing AI coding assistants (Claude Code, Cursor, Antigravity, GitHub Copilot) to generate and iterate on API tests directly in your IDE.

1. **Generates test YAML** using the Keploy test format (taught via CLAUDE.md or .cursorrules)
2. **Runs tests** via the `keploy-runner` CLI with structured JSON output
Instead of switching to a separate web UI, the agent:

1. **Generates test YAML** using the Keploy test format
2. **Runs tests** via Keploy Enterprise with structured JSON output
3. **Reads coverage gaps** from the output and generates targeted tests
4. **Syncs results** to the Keploy platform for team-wide visibility

This eliminates the "double LLM tax"—the agent writes tests directly instead of calling a second LLM.
The Keploy platform automatically builds and refines the API schema over time from your test runs, so providing an OpenAPI spec upfront is **optional**.

## Prerequisites

- Go 1.26+ installed (the version required by the enterprise module)
- Keploy Enterprise installed ([installation guide](/docs/keploy-cloud/cloud-installation/))
- A running API server to test against
- An OpenAPI spec (optional, enables coverage feedback loop)

## Installation
- An OpenAPI spec (optional; the platform generates schema coverage over time)

```bash
# From the keploy/enterprise repository
go install github.com/keploy/enterprise/v3/cmd/keploy-runner@latest
```
## Setup for AI Agents

Or build from source:
To teach your AI agent the Keploy test format, add a context file to your project root. The agent reads this file and learns how to generate valid test suites.

```bash
git clone https://github.com/keploy/enterprise.git
cd enterprise
go build -o keploy-runner ./cmd/keploy-runner/
```
### Claude Code

## Setup for AI Agents
Create a `CLAUDE.md` file in your project root with the Keploy test format reference. Claude Code automatically reads project-level markdown files. The file should document:

### Claude Code
- The YAML test suite schema (name, description, steps)
- Available assertion types (status_code, json_equal, json_contains, etc.)

Check failure on line 55 in versioned_docs/version-4.0.0/running-keploy/agent-test-generation.md

View workflow job for this annotation

GitHub Actions / vale

[vale] versioned_docs/version-4.0.0/running-keploy/agent-test-generation.md#L55 

[Vale.Spelling] Did you really mean 'status_code'?
Raw output 
{"message": "[Vale.Spelling] Did you really mean 'status_code'?", "location": {"path": "versioned_docs/version-4.0.0/running-keploy/agent-test-generation.md", "range": {"start": {"line": 55, "column": 30}}}, "severity": "ERROR"}
- Variable extraction and substitution rules
- The CLI commands to run and validate tests

Copy `CLAUDE.md` from the [agent-docs directory](https://github.com/keploy/enterprise/tree/main/agent-docs) to your project root. Claude Code will automatically read it and learn the test format.
See the [Test Suite YAML Format](#test-suite-yaml-format) section below for the complete schema to include.

### Cursor

Copy the `cursorrules` file from agent-docs to `.cursorrules` in your project root.
Create a `.cursorrules` file in your project root with the same test format reference.

### Windsurf / Other MCP-compatible agents
### Antigravity

Use the same CLAUDE.md content as system instructions or project context.
Use the same test format documentation as system instructions or project context.

## Workflow

### 1. Initialize the test directory

```bash
keploy-runner init --dir ./keploy
keploy test-gen init --dir ./keploy
```

This creates:
Expand All @@ -85,36 +81,37 @@

### 2. Generate tests with your AI agent

Ask your agent to generate tests. It will use the format from CLAUDE.md:
Ask your agent to generate tests:

> "Generate API tests for my user CRUD endpoints based on the OpenAPI spec"
> "Generate API tests for my user CRUD endpoints"

The agent writes YAML test files directly to `keploy/tests/`.

### 3. Run tests

```bash
keploy-runner run --base-url http://localhost:8080 --output json
keploy test-gen run --base-url http://localhost:8080 --output json
```

The JSON output includes pass/fail status, assertion failures, and extracted variables—the agent reads this to fix failing tests.
The JSON output includes pass/fail status, assertion failures, and extracted variables. The agent reads this to fix failing tests.

### 4. Coverage feedback loop
### 4. Coverage feedback loop (optional)

Add `--spec` to include coverage analysis:
If you have an OpenAPI spec, add `--spec` for coverage analysis:

```bash
keploy-runner run --base-url http://localhost:8080 --spec openapi.yaml --output json
keploy test-gen run --base-url http://localhost:8080 --spec openapi.yaml --output json
```

The output now includes a `coverage` section with `next_steps`—prioritized suggestions for uncovered endpoints. The agent reads these and generates targeted tests automatically.
The output includes a `coverage` section with `next_steps`—prioritized suggestions for uncovered endpoints. The agent reads these and generates targeted tests automatically.

Even without a spec, the Keploy platform builds schema coverage over time from your test runs. You can view this on the Keploy dashboard.

### 5. Sync results to platform

```bash
keploy-runner run \
keploy test-gen run \
--base-url http://localhost:8080 \
--spec openapi.yaml \
--sync --app-id your-app-id --api-key $KEPLOY_API_KEY \
--output json
```
Expand All @@ -123,7 +120,7 @@

## CLI Reference

### `keploy-runner run`
### `keploy test-gen run`

Execute test suites against a running API server.

Expand All @@ -132,10 +129,10 @@
| `--base-url` | API server URL (required) | — |
| `--test-dir` | Directory with test YAML files | `./keploy/tests` |
| `--output` | Output format: `text`, `json`, `junit` | `text` |
| `--spec` | OpenAPI spec for coverage analysis | — |
| `--spec` | OpenAPI spec for coverage analysis (optional) | — |
| `--suite` | Run specific suites by name | all |
| `--ci` | Exit 1 on failure or low coverage | `false` |
| `--min-coverage` | Coverage target (with `--ci`) | `80` |
| `--min-coverage` | Coverage target (with `--ci` and `--spec`) | `80` |
| `--flaky-runs` | Re-run failed suites N times | `0` |
| `--sync` | Push results to Keploy platform | `false` |
| `--app-id` | Keploy app ID (with `--sync`) | — |
Expand All @@ -144,9 +141,9 @@
| `--rate-limit` | Max requests per second | unlimited |
| `--timeout` | Per-request timeout (seconds) | `30` |

### `keploy-runner coverage`
### `keploy test-gen coverage`

Standalone coverage analysis without executing tests.
Standalone coverage analysis without executing tests. Requires an OpenAPI spec.

| Flag | Description | Default |
| ---------------- | ------------------------------ | ---------------- |
Expand All @@ -155,7 +152,7 @@
| `--output` | Output format: `text`, `json` | `text` |
| `--min-coverage` | Coverage target percentage | `80` |

### `keploy-runner init`
### `keploy test-gen init`

Scaffold a test directory with examples.

Expand Down Expand Up @@ -202,24 +199,24 @@

### Assertion Types

| Type | Description | Example |
| ------------------- | -------------------------- | ---------------------------------------------------------------------------- |
| `status_code` | Exact HTTP status match | `expected_string: "200"` |
| `status_code_class` | Status class match | `expected_string: "2xx"` |
| `status_code_in` | Status in list | `expected_string: "200,201,204"` |
| `json_equal` | Exact match at gjson path | `key: $.id`, `expected_string: '"abc"'` |
| `json_contains` | Subset match at gjson path | `key: $.data`, `expected_string: '{"name":"John"}'` |
| `json_path` | Alias for json_equal | `key: $.id`, `expected_string: '"abc"'` |
| `header_equal` | Exact header match | `key: Content-Type`, `expected_string: "application/json"` |
| `header_contains` | Header substring | `key: Content-Type`, `expected_string: "json"` |
| `header_exists` | Header presence | `key: X-Request-Id`, `expected_string: "true"` |
| `header_matches` | Header regex match | `key: Content-Type`, `expected_string: "application/.*json"` |
| `schema` | JSON Schema validation | `expected_string: '{"type":"object","properties":{"id":{"type":"string"}}}'` |
| `custom_functions` | JS function assertion | `expected_string: 'function(req, res) { return res.status === 200; }'` |
| Type | Description | Example |
| ------------------- | ---------------------------------- | ---------------------------------------------------------------------------- |
| `status_code` | Exact HTTP status match | `expected_string: "200"` |
| `status_code_class` | Status class match | `expected_string: "2xx"` |
| `status_code_in` | Status in list | `expected_string: "200,201,204"` |
| `json_equal` | Exact match at a JSON path | `key: $.id`, `expected_string: '"abc"'` |
| `json_contains` | Subset match at a JSON path | `key: $.data`, `expected_string: '{"name":"John"}'` |
| `json_path` | Alias for json_equal | `key: $.id`, `expected_string: '"abc"'` |
| `header_equal` | Exact header match | `key: Content-Type`, `expected_string: "application/json"` |
| `header_contains` | Header value contains a string | `key: Content-Type`, `expected_string: "json"` |
| `header_exists` | Header key is present | `key: X-Request-Id`, `expected_string: "true"` |
| `header_matches` | Header matches a regex pattern | `key: Content-Type`, `expected_string: "application/.*json"` |
| `schema` | Validate response against a schema | `expected_string: '{"type":"object","properties":{"id":{"type":"string"}}}'` |
| `custom_functions` | Custom JS function assertion | `expected_string: 'function(req, res) { return res.status === 200; }'` |

### Variable Extraction

Use gjson paths to extract values from responses and use them in subsequent steps:
Use JSON paths to extract values from responses and use them in subsequent steps:

```yaml
extract:
Expand All @@ -233,8 +230,8 @@

- Variables are scoped to the test suite (not shared between suites)
- Must be extracted before use (by a previous step)
- gjson path syntax: `$.field` or `field`, `$.nested.field` or `nested.field` (the `$.` prefix is optional and stripped automatically)
- Use dot notation for arrays: `$.users.0.id` (correct) instead of `$.users[0].id` (incorrect—brackets are not supported by gjson)
- Path syntax: `$.field` or `field`, `$.nested.field` or `nested.field` (the `$.` prefix is optional)
- Use dot notation for arrays: `$.users.0.id` (correct) instead of `$.users[0].id` (brackets are not supported)

## CI/CD Integration

Expand All @@ -243,10 +240,9 @@
```yaml
- name: Run API tests
run: |
keploy-runner run \
keploy test-gen run \
--base-url http://localhost:8080 \
--spec openapi.yaml \
--ci --min-coverage 80 \
--ci \
--output junit > test-results.xml

- name: Upload test results
Expand All @@ -260,7 +256,7 @@

| Code | Meaning |
| ---- | -------------------------------------------------------------------------------------------- |
| 0 | All tests pass, coverage above threshold |
| 0 | All tests pass, coverage above threshold (if `--spec` provided) |
| 1 | Test failures, validation errors, coverage below threshold, or sync failure (in `--ci` mode) |

## Coverage Feedback Loop
Expand All @@ -287,3 +283,5 @@
```

The agent reads `next_steps` and generates targeted tests for uncovered endpoints. This loop continues automatically until coverage targets are met.

Without an OpenAPI spec, you can still run tests and sync results. The Keploy platform builds schema coverage progressively from your test execution data over time.
2 changes: 1 addition & 1 deletion versioned_sidebars/version-4.0.0-sidebars.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@
"keploy-cloud/mock-registry",
"keploy-cloud/keploy-console",
"keploy-cloud/auto-test-generation",
"keploy-cloud/agent-test-generation",
"keploy-cloud/deduplication",
{
"type": "category",
Expand Down Expand Up @@ -165,6 +164,7 @@
"items": [
"running-keploy/api-test-generator",
"running-keploy/generate-api-tests-using-ai",
"running-keploy/agent-test-generation",
"running-keploy/api-testing-chrome-extension",
"running-keploy/api-testing-local-agent",
"running-keploy/api-testing-auth-setup",
Expand Down
Loading