Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions .github/ISSUE_TEMPLATE/bug.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
name: Bug report
description: Report a reproducible problem in the CLI, generated checks, or reports
title: "[Bug]: "
labels:
- bug
body:
- type: markdown
attributes:
value: |
Thanks for reporting a problem. Use a local or synthetic target where possible, and remove credentials or confidential payloads before submitting.

- type: dropdown
id: workflow
attributes:
label: Affected workflow
options:
- Python function analysis and execution
- OpenAPI parsing and REST execution
- JSON or HTML reporting
- Optional AI supplementation
- Java sample
- Other
validations:
required: true

- type: textarea
id: description
attributes:
label: What happened?
description: Describe the observed behavior and why it is a problem.
validations:
required: true

- type: textarea
id: reproduction
attributes:
label: Minimal reproduction
description: Include the smallest safe command, function, or OpenAPI fragment that reproduces the problem.
render: shell
validations:
required: true

- type: textarea
id: expected
attributes:
label: Expected behavior
validations:
required: true

- type: input
id: version
attributes:
label: Version or commit
placeholder: "0.1.0 or git commit SHA"
validations:
required: true

- type: input
id: environment
attributes:
label: Environment
placeholder: "Python 3.13, Windows 11"
validations:
required: true

- type: textarea
id: logs
attributes:
label: Relevant output
description: Paste only the relevant output after removing tokens, credentials, personal data, and private endpoints.
render: shell

- type: checkboxes
id: checks
attributes:
label: Submission checks
options:
- label: I searched existing issues for the same problem.
required: true
- label: I removed credentials, confidential data, and private service details.
required: true
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
blank_issues_enabled: false
49 changes: 49 additions & 0 deletions .github/ISSUE_TEMPLATE/feature.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: Feature request
description: Propose a focused improvement backed by a concrete testing workflow
title: "[Feature]: "
labels:
- enhancement
body:
- type: markdown
attributes:
value: |
Explain the testing problem first. Focused proposals with a safe example and expected behavior are easiest to evaluate.

- type: textarea
id: problem
attributes:
label: Problem to solve
description: Who encounters this problem, and in which Python or REST testing workflow?
validations:
required: true

- type: textarea
id: proposal
attributes:
label: Proposed behavior
description: Describe the smallest useful outcome rather than only an implementation preference.
validations:
required: true

- type: textarea
id: example
attributes:
label: Example
description: Show a safe input, CLI command, or expected report fragment when possible.
render: shell

- type: textarea
id: alternatives
attributes:
label: Alternatives considered
description: Note any current workaround or smaller alternative.

- type: checkboxes
id: scope
attributes:
label: Scope checks
options:
- label: This proposal is relevant to Python-function or OpenAPI/REST testing.
required: true
- label: The example does not require credentials or access to an unauthorized service.
required: true
18 changes: 18 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
## Summary

- What changed?
- Which user or developer problem does it solve?

## Validation

- [ ] Python tests pass: `python -m unittest discover -s tests -v`
- [ ] Python sources compile: `python -m compileall -q src sample_targets tests`
- [ ] Java sample verifies when affected: `mvn -f java-springboot-sample/pom.xml -B -ntp verify`
- [ ] Documentation and generated examples match the implemented behavior

## Safety and scope

- [ ] The change contains no credentials, private endpoints, personal data, or generated reports
- [ ] REST examples target only local, synthetic, or explicitly authorized systems
- [ ] New behavior includes a focused test or executable example, or the change is documentation-only
- [ ] Known limitations and compatibility impact are documented
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Changelog

All notable user-visible changes to this project are documented in this file. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## Unreleased

### Added

- Dependency-free local REST target and OpenAPI document for the first-run workflow.
- End-to-end coverage for OpenAPI parsing, real HTTP execution, validation, and JSON/HTML report output.
- Optional `--fail-on-failure` exit behavior for CI pipelines.
- Contributor guidance, structured issue forms, and a pull request checklist.

### Changed

- README now leads with the offline value proposition and a three-minute path to both supported workflows.

### Fixed

- Negative REST cases no longer pass when the target never returns an HTTP response.
- Expected response fields now fail validation when the response is not a JSON object.

## 0.1.0 - 2026-07-13

### Added

- Rule-based test generation and execution for Python functions.
- OpenAPI JSON parsing and REST request execution.
- Standalone JSON and HTML reports.
- Optional OpenAI-compatible supplemental case generation.
- Spring Boot sample target and GitHub Actions verification.
28 changes: 28 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Contributing

Thanks for helping improve Agentic API Testkit. Small, focused changes with an executable example or regression test are especially useful.

## Local setup

```bash
python -m venv .venv
python -m pip install -e .
python -m unittest discover -s tests -v
python -m compileall -q src sample_targets tests
```

To verify the optional Java target:

```bash
mvn -f java-springboot-sample/pom.xml -B -ntp verify
```

## Pull requests

- Open an issue first for broad behavior or public API changes.
- Keep generated reports, virtual environments, credentials, and build output out of commits.
- Add a regression test for fixes and an executable example for new workflows.
- Run the Python suite and the checks relevant to changed samples.
- Describe the supported OpenAPI subset and any known limitations clearly.

Only exercise services that you own or are explicitly authorized to test. Examples must use local or synthetic endpoints and placeholder credentials.
71 changes: 58 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,24 @@
# Agentic API Testkit

An extensible command-line testkit that turns Python function signatures or OpenAPI documents into executable test cases, then writes review-friendly JSON and HTML reports.
[![CI](https://github.com/device-kunkun/agentic-api-testkit/actions/workflows/ci.yml/badge.svg)](https://github.com/device-kunkun/agentic-api-testkit/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python 3.9+](https://img.shields.io/badge/Python-3.9%2B-blue.svg)](pyproject.toml)

Turn a Python function or OpenAPI document into executable checks and review-friendly JSON/HTML reports. The default workflow is deterministic, offline, and does not require an API key.

> 面向 Python 函数与 REST API 的智能测试工具:规则生成基础用例,可选使用大模型补充业务边界用例。

**Try it in under three minutes:** use the [Python-function workflow](#3-minute-quick-start) or run the [zero-dependency REST demo](#zero-dependency-openapi--rest-demo).

## Highlights

- Analyze Python signatures, annotations, source code, and simple constraints.
- Parse OpenAPI 3 JSON documents and exercise REST endpoints.
- Generate deterministic baseline cases locally with no external service.
- Optionally ask an OpenAI-compatible model for additional high-value cases.
- Validate outcomes and summarize possible defects.
- Validate status codes and JSON response shapes without hiding transport failures.
- Export machine-readable JSON and standalone HTML reports.
- Return a CI-friendly nonzero exit code on failed checks when requested.

```text
Python function / OpenAPI JSON
Expand All @@ -25,7 +32,7 @@ Python function / OpenAPI JSON
JSON + HTML reports
```

## Quick start
## 3-minute quick start

Agentic API Testkit requires Python 3.9 or newer.

Expand All @@ -34,7 +41,7 @@ python -m venv .venv
python -m pip install -e .
```

Run the included Python example:
Run the included Python-function example:

```bash
api-testkit \
Expand All @@ -45,16 +52,46 @@ api-testkit \

The command writes `reports/report.json` and `reports/report.html`.

## OpenAPI / REST workflow
## Zero-dependency OpenAPI / REST demo

The fastest REST workflow uses a deterministic local server from Python's standard library. It does not need Docker, a database, credentials, or network access.

Start the target in one terminal:

```bash
python -m sample_targets.rest_demo_server
```

Run the generated checks in a second terminal:

```bash
api-testkit \
--spec sample_targets/rest_demo_openapi.json \
--base-url http://127.0.0.1:8765 \
--output-dir rest_reports \
--fail-on-failure
```

This exercises `GET /health`, a valid `POST /echo`, and an invalid request missing its required `message`. The command writes `rest_reports/rest_report.json` and `rest_reports/rest_report.html`.

```text
Completed REST spec sample_targets/rest_demo_openapi.json: 3/3 passed.
rest_reports/rest_report.json
rest_reports/rest_report.html
```

`--fail-on-failure` returns exit code `1` after reports are written when any generated check fails, making the command suitable for CI. Without the flag, report failures do not change the process exit code.

## Spring Boot sample

The repository includes a small Spring Boot service and its OpenAPI document. Configure a local MySQL database with environment variables before starting it; no database credentials are committed.
The repository also includes a larger Spring Boot service and its OpenAPI document. Configure a local MySQL database with environment variables before starting it; no database credentials are committed.

```bash
cd java-springboot-sample
mvn spring-boot:run
```

In another terminal, from the repository root:
Then run the testkit from the repository root:

```bash
api-testkit \
Expand All @@ -75,7 +112,7 @@ python -m pip install -e ".[ai]"

Set `OPENAI_API_KEY` and, when needed, `OPENAI_MODEL`. `OPENAI_SUPPLEMENT_MODEL` can select a separate model for generation; if omitted, the primary model is reused. See [`.env.example`](.env.example) for the complete variable list. The project does not automatically load `.env` files.

Be aware that enabling this feature sends the analyzed function source, docstring, and generated inputs to the configured model provider. Do not enable it for confidential code unless that data flow is approved.
Enabling this feature sends the analyzed function source, docstring, and generated inputs to the configured model provider. Do not enable it for confidential code unless that data flow is approved.

## Configuration

Expand All @@ -90,10 +127,11 @@ Be aware that enabling this feature sends the analyzed function source, docstrin

## Development

Run the Python test suite:
Run the Python checks:

```bash
python -m unittest discover -s tests -v
python -m compileall -q src sample_targets tests
```

Compile the Java example without starting it:
Expand All @@ -108,13 +146,20 @@ GitHub Actions verifies the supported Python versions and compiles the Java samp

```text
src/ai_agent_test_framework/ Python package and CLI
sample_targets/ Local Python examples
java-springboot-sample/ REST service and OpenAPI example
sample_targets/ Offline Python and REST examples
java-springboot-sample/ Larger REST service and OpenAPI example
tests/ Offline unit and integration tests
```

Generated reports, virtual environments, local credentials, Python caches, and Java build output are intentionally ignored.

## Current scope
## Community

Found a bug or have a focused feature idea? [Open a structured issue](https://github.com/device-kunkun/agentic-api-testkit/issues/new/choose). Contributions are welcome; read [CONTRIBUTING.md](CONTRIBUTING.md) for the local checks and safety boundaries before opening a pull request. User-visible changes are tracked in [CHANGELOG.md](CHANGELOG.md).

## Limitations and safety

The OpenAPI parser intentionally targets a practical subset of OpenAPI 3 JSON. It does not yet resolve external `$ref` documents, authentication schemes, or every schema composition keyword. Treat generated expectations as a starting point for review, not as a replacement for domain-specific assertions.
- The OpenAPI parser targets a practical subset of OpenAPI 3 JSON; YAML input is not supported yet.
- External `$ref` documents, authentication schemes, and some schema composition keywords are not resolved.
- Generated expectations are a reviewable baseline, not a replacement for domain-specific assertions.
- Run REST checks only against systems you own or are explicitly authorized to test.
Loading
Loading