Agentic API Testkit

An extensible command-line testkit that turns Python function signatures or OpenAPI documents into executable test cases, then writes review-friendly JSON and HTML reports.

面向 Python 函数与 REST API 的智能测试工具：规则生成基础用例，可选使用大模型补充业务边界用例。

Highlights

• Analyze Python signatures, annotations, source code, and simple constraints.
• Parse OpenAPI 3 JSON documents and exercise REST endpoints.
• Generate deterministic baseline cases locally with no external service.
• Optionally ask an OpenAI-compatible model for additional high-value cases.
• Validate outcomes and summarize possible defects.
• Export machine-readable JSON and standalone HTML reports.

Python function / OpenAPI JSON
            |
      analyzer / parser
            |
   rule cases + optional AI cases
            |
      executor + validator
            |
       JSON + HTML reports

Quick start

Agentic API Testkit requires Python 3.9 or newer.

python -m venv .venv
python -m pip install -e .

Run the included Python example:

api-testkit \
  --module sample_targets.order_service \
  --function create_order \
  --output-dir reports

The command writes reports/report.json and reports/report.html.

OpenAPI / REST workflow

The repository includes a small Spring Boot service and its OpenAPI document. Configure a local MySQL database with environment variables before starting it; no database credentials are committed.

cd java-springboot-sample
mvn spring-boot:run

In another terminal, from the repository root:

api-testkit \
  --spec java-springboot-sample/openapi.json \
  --base-url http://localhost:8080 \
  --output-dir rest_reports

Only run REST tests against systems you own or are explicitly authorized to test.

Optional AI case supplementation

The default workflow is local and does not make model calls. To enable AI-generated supplemental cases:

python -m pip install -e ".[ai]"

Set OPENAI_API_KEY and, when needed, OPENAI_MODEL. OPENAI_SUPPLEMENT_MODEL can select a separate model for generation; if omitted, the primary model is reused. See .env.example for the complete variable list. The project does not automatically load .env files.

Be aware that enabling this feature sends the analyzed function source, docstring, and generated inputs to the configured model provider. Do not enable it for confidential code unless that data flow is approved.

Configuration

Variable	Purpose	Required
OPENAI_API_KEY	Enables model-backed explanations and supplemental cases	No
OPENAI_MODEL	Primary reasoning model	Only for model mode when the default is unsuitable
OPENAI_SUPPLEMENT_MODEL	Optional separate case-generation model	No
SPRING_DATASOURCE_URL	JDBC URL for the Java sample	For running the Java sample
SPRING_DATASOURCE_USERNAME	Database user for the Java sample	For running the Java sample
SPRING_DATASOURCE_PASSWORD	Database password for the Java sample	For running the Java sample

Development

Run the Python test suite:

python -m unittest discover -s tests -v

Compile the Java example without starting it:

mvn -f java-springboot-sample/pom.xml -B -ntp verify

GitHub Actions verifies the supported Python versions and compiles the Java sample on every push and pull request.

Repository layout

src/ai_agent_test_framework/  Python package and CLI
sample_targets/               Local Python examples
java-springboot-sample/       REST service and OpenAPI example
tests/                        Offline unit and integration tests

Generated reports, virtual environments, local credentials, Python caches, and Java build output are intentionally ignored.

Current scope

The OpenAPI parser intentionally targets a practical subset of OpenAPI 3 JSON. It does not yet resolve external $ref documents, authentication schemes, or every schema composition keyword. Treat generated expectations as a starting point for review, not as a replacement for domain-specific assertions.