From 6cead22b2d1f65ce50dc0ff76b5a80678ddb6a91 Mon Sep 17 00:00:00 2001 From: device-kunkun <175306042+device-kunkun@users.noreply.github.com> Date: Tue, 14 Jul 2026 00:11:20 +0800 Subject: [PATCH 1/2] Add local REST demo and CI exit handling --- CONTRIBUTING.md | 28 ++++ README.md | 52 ++++++-- sample_targets/rest_demo_openapi.json | 70 ++++++++++ sample_targets/rest_demo_server.py | 80 ++++++++++++ src/ai_agent_test_framework/cli.py | 15 ++- src/ai_agent_test_framework/rest_support.py | 19 ++- tests/test_rest_workflow.py | 136 ++++++++++++++++++++ 7 files changed, 382 insertions(+), 18 deletions(-) create mode 100644 CONTRIBUTING.md create mode 100644 sample_targets/rest_demo_openapi.json create mode 100644 sample_targets/rest_demo_server.py create mode 100644 tests/test_rest_workflow.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..9fd847d --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,28 @@ +# Contributing + +Thanks for helping improve Agentic API Testkit. Small, focused changes with an executable example or regression test are especially useful. + +## Local setup + +```bash +python -m venv .venv +python -m pip install -e . +python -m unittest discover -s tests -v +python -m compileall -q src sample_targets tests +``` + +To verify the optional Java target: + +```bash +mvn -f java-springboot-sample/pom.xml -B -ntp verify +``` + +## Pull requests + +- Open an issue first for broad behavior or public API changes. +- Keep generated reports, virtual environments, credentials, and build output out of commits. +- Add a regression test for fixes and an executable example for new workflows. +- Run the Python suite and the checks relevant to changed samples. +- Describe the supported OpenAPI subset and any known limitations clearly. + +Only exercise services that you own or are explicitly authorized to test. Examples must use local or synthetic endpoints and placeholder credentials. diff --git a/README.md b/README.md index c80d649..b4fcd65 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # Agentic API Testkit +[![CI](https://github.com/device-kunkun/agentic-api-testkit/actions/workflows/ci.yml/badge.svg)](https://github.com/device-kunkun/agentic-api-testkit/actions/workflows/ci.yml) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) +[![Python 3.9+](https://img.shields.io/badge/Python-3.9%2B-blue.svg)](pyproject.toml) + An extensible command-line testkit that turns Python function signatures or OpenAPI documents into executable test cases, then writes review-friendly JSON and HTML reports. > 面向 Python 函数与 REST API 的智能测试工具：规则生成基础用例，可选使用大模型补充业务边界用例。 @@ -10,8 +14,9 @@ An extensible command-line testkit that turns Python function signatures or Open - Parse OpenAPI 3 JSON documents and exercise REST endpoints. - Generate deterministic baseline cases locally with no external service. - Optionally ask an OpenAI-compatible model for additional high-value cases. -- Validate outcomes and summarize possible defects. +- Validate status codes and JSON response shapes without hiding transport failures. - Export machine-readable JSON and standalone HTML reports. +- Return a CI-friendly nonzero exit code on failed checks when requested. ```text Python function / OpenAPI JSON @@ -34,7 +39,7 @@ python -m venv .venv python -m pip install -e . ``` -Run the included Python example: +Run the included Python-function example: ```bash api-testkit \ @@ -45,16 +50,40 @@ api-testkit \ The command writes `reports/report.json` and `reports/report.html`. -## OpenAPI / REST workflow +## Zero-dependency OpenAPI / REST demo + +The fastest REST workflow uses a deterministic local server from Python's standard library. It does not need Docker, a database, credentials, or network access. + +Start the target in one terminal: + +```bash +python -m sample_targets.rest_demo_server +``` + +Run the generated checks in a second terminal: + +```bash +api-testkit \ + --spec sample_targets/rest_demo_openapi.json \ + --base-url http://127.0.0.1:8765 \ + --output-dir rest_reports \ + --fail-on-failure +``` + +This exercises `GET /health`, a valid `POST /echo`, and an invalid request missing its required `message`. The command writes `rest_reports/rest_report.json` and `rest_reports/rest_report.html`. -The repository includes a small Spring Boot service and its OpenAPI document. Configure a local MySQL database with environment variables before starting it; no database credentials are committed. +`--fail-on-failure` returns exit code `1` after reports are written when any generated check fails, making the command suitable for CI. Without the flag, report failures do not change the process exit code. + +## Spring Boot sample + +The repository also includes a larger Spring Boot service and its OpenAPI document. Configure a local MySQL database with environment variables before starting it; no database credentials are committed. ```bash cd java-springboot-sample mvn spring-boot:run ``` -In another terminal, from the repository root: +Then run the testkit from the repository root: ```bash api-testkit \ @@ -75,7 +104,7 @@ python -m pip install -e ".[ai]" Set `OPENAI_API_KEY` and, when needed, `OPENAI_MODEL`. `OPENAI_SUPPLEMENT_MODEL` can select a separate model for generation; if omitted, the primary model is reused. See [`.env.example`](.env.example) for the complete variable list. The project does not automatically load `.env` files. -Be aware that enabling this feature sends the analyzed function source, docstring, and generated inputs to the configured model provider. Do not enable it for confidential code unless that data flow is approved. +Enabling this feature sends the analyzed function source, docstring, and generated inputs to the configured model provider. Do not enable it for confidential code unless that data flow is approved. ## Configuration @@ -90,10 +119,11 @@ Be aware that enabling this feature sends the analyzed function source, docstrin ## Development -Run the Python test suite: +Run the Python checks: ```bash python -m unittest discover -s tests -v +python -m compileall -q src sample_targets tests ``` Compile the Java example without starting it: @@ -108,13 +138,17 @@ GitHub Actions verifies the supported Python versions and compiles the Java samp ```text src/ai_agent_test_framework/ Python package and CLI -sample_targets/ Local Python examples -java-springboot-sample/ REST service and OpenAPI example +sample_targets/ Offline Python and REST examples +java-springboot-sample/ Larger REST service and OpenAPI example tests/ Offline unit and integration tests ``` Generated reports, virtual environments, local credentials, Python caches, and Java build output are intentionally ignored. +## Community + +Found a bug or have a focused feature idea? [Open an issue](https://github.com/device-kunkun/agentic-api-testkit/issues). Contributions are welcome; read [CONTRIBUTING.md](CONTRIBUTING.md) for the local checks and safety boundaries before opening a pull request. + ## Current scope The OpenAPI parser intentionally targets a practical subset of OpenAPI 3 JSON. It does not yet resolve external `$ref` documents, authentication schemes, or every schema composition keyword. Treat generated expectations as a starting point for review, not as a replacement for domain-specific assertions. diff --git a/sample_targets/rest_demo_openapi.json b/sample_targets/rest_demo_openapi.json new file mode 100644 index 0000000..c7fcc9e --- /dev/null +++ b/sample_targets/rest_demo_openapi.json @@ -0,0 +1,70 @@ +{ + "openapi": "3.0.3", + "info": { + "title": "Agentic API Testkit local demo", + "version": "1.0.0" + }, + "paths": { + "/health": { + "get": { + "operationId": "getHealth", + "summary": "Return service health", + "responses": { + "200": { + "description": "Service is healthy", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "status": { "type": "string" }, + "service": { "type": "string" } + } + } + } + } + } + } + } + }, + "/echo": { + "post": { + "operationId": "echoMessage", + "summary": "Echo a required message", + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "required": ["message"], + "properties": { + "message": { "type": "string" } + } + } + } + } + }, + "responses": { + "200": { + "description": "Message echoed", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "message": { "type": "string" }, + "length": { "type": "integer" } + } + } + } + } + }, + "400": { + "description": "Message is missing" + } + } + } + } + } +} diff --git a/sample_targets/rest_demo_server.py b/sample_targets/rest_demo_server.py new file mode 100644 index 0000000..c052a02 --- /dev/null +++ b/sample_targets/rest_demo_server.py @@ -0,0 +1,80 @@ +"""Dependency-free local REST target for the OpenAPI quick start.""" + +from __future__ import annotations + +import argparse +import json +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from typing import Any +from urllib.parse import urlsplit + + +class DemoRequestHandler(BaseHTTPRequestHandler): + """Serve two deterministic JSON endpoints without external services.""" + + server_version = "AgenticApiTestkitDemo/1.0" + + def do_GET(self) -> None: # noqa: N802 + if urlsplit(self.path).path == "/health": + self._write_json( + 200, + {"status": "ok", "service": "agentic-api-testkit-demo"}, + ) + return + self._write_json(404, {"error": "not found"}) + + def do_POST(self) -> None: # noqa: N802 + if urlsplit(self.path).path != "/echo": + self._write_json(404, {"error": "not found"}) + return + + try: + content_length = int(self.headers.get("Content-Length", "0")) + payload = json.loads(self.rfile.read(content_length) or b"{}") + except (ValueError, json.JSONDecodeError): + self._write_json(400, {"error": "request body must be valid JSON"}) + return + + message = payload.get("message") if isinstance(payload, dict) else None + if not isinstance(message, str) or not message.strip(): + self._write_json(400, {"error": "message is required"}) + return + + self._write_json(200, {"message": message, "length": len(message)}) + + def _write_json(self, status: int, payload: dict[str, Any]) -> None: + body = json.dumps(payload, ensure_ascii=False).encode("utf-8") + self.send_response(status) + self.send_header("Content-Type", "application/json; charset=utf-8") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, format: str, *args: Any) -> None: + """Keep the demo quiet so testkit output remains readable.""" + + +def create_server(host: str = "127.0.0.1", port: int = 8765) -> ThreadingHTTPServer: + """Build a server; passing port 0 lets the OS choose a free test port.""" + + return ThreadingHTTPServer((host, port), DemoRequestHandler) + + +def main() -> None: + parser = argparse.ArgumentParser(description="Run the local REST demo target") + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", default=8765, type=int) + args = parser.parse_args() + server = create_server(args.host, args.port) + host, port = server.server_address[:2] + print(f"REST demo listening on http://{host}:{port}") + try: + server.serve_forever() + except KeyboardInterrupt: + pass + finally: + server.server_close() + + +if __name__ == "__main__": + main() diff --git a/src/ai_agent_test_framework/cli.py b/src/ai_agent_test_framework/cli.py index 0820ae9..60f66d5 100644 --- a/src/ai_agent_test_framework/cli.py +++ b/src/ai_agent_test_framework/cli.py @@ -3,6 +3,7 @@ import argparse import os import sys +from typing import Optional, Sequence from .rest_support import RestOrchestrator from .orchestrator import TestOrchestrator @@ -17,14 +18,19 @@ def build_parser() -> argparse.ArgumentParser: parser.add_argument("--spec", help="OpenAPI JSON file path") parser.add_argument("--base-url", help="REST service base URL, e.g. http://localhost:8080") parser.add_argument("--output-dir", default="reports", help="Directory for generated reports") + parser.add_argument( + "--fail-on-failure", + action="store_true", + help="Return exit code 1 when one or more generated checks fail", + ) return parser -def main() -> None: +def main(argv: Optional[Sequence[str]] = None) -> int: cwd = os.getcwd() if cwd not in sys.path: sys.path.insert(0, cwd) - args = build_parser().parse_args() + args = build_parser().parse_args(argv) output_dir = args.output_dir if not os.path.isabs(output_dir): output_dir = os.path.join(cwd, output_dir) @@ -38,7 +44,7 @@ def main() -> None: f"{report.summary['passed']}/{report.summary['total']} passed. " f"Report directory: {output_dir}" ) - return + return 1 if args.fail_on_failure and report.summary["failed"] else 0 if not args.module or not args.function: raise SystemExit("--module and --function are required when --spec is not provided") @@ -49,7 +55,8 @@ def main() -> None: f"{report.summary['passed']}/{report.summary['total']} passed. " f"Report directory: {output_dir}" ) + return 1 if args.fail_on_failure and report.summary["failed"] else 0 if __name__ == "__main__": - main() + raise SystemExit(main()) diff --git a/src/ai_agent_test_framework/rest_support.py b/src/ai_agent_test_framework/rest_support.py index 9648cef..02c20e5 100644 --- a/src/ai_agent_test_framework/rest_support.py +++ b/src/ai_agent_test_framework/rest_support.py @@ -239,12 +239,21 @@ def validate(self, test_case: RestTestCase, execution: RestExecutionResult) -> R reasons.append( f"expected status {test_case.expected_status}, got {execution.status_code}" ) - if test_case.expected_response_fields and isinstance(execution.response_json, dict): - missing = [k for k in test_case.expected_response_fields if k not in execution.response_json] - if missing: - reasons.append(f"missing response fields: {', '.join(missing)}") + if test_case.expected_response_fields: + if not isinstance(execution.response_json, dict): + reasons.append("expected a JSON object response") + else: + missing = [ + key + for key in test_case.expected_response_fields + if key not in execution.response_json + ] + if missing: + reasons.append(f"missing response fields: {', '.join(missing)}") else: - if execution.status_code < 400 and execution.error is None: + if execution.status_code == 0: + reasons.append("no HTTP response received") + elif execution.status_code < 400: reasons.append("expected failure but request succeeded") return RestValidationResult(passed=not reasons, reasons=reasons) diff --git a/tests/test_rest_workflow.py b/tests/test_rest_workflow.py new file mode 100644 index 0000000..4d36ef9 --- /dev/null +++ b/tests/test_rest_workflow.py @@ -0,0 +1,136 @@ +from __future__ import annotations + +import io +import json +import tempfile +import threading +import unittest +from contextlib import redirect_stdout +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import patch + +from ai_agent_test_framework.cli import main as cli_main +from ai_agent_test_framework.models import ( + ApiEndpointAnalysis, + RestExecutionResult, + RestTestCase, +) +from ai_agent_test_framework.rest_support import RestOrchestrator, RestValidator +from sample_targets.rest_demo_server import create_server + + +ROOT = Path(__file__).resolve().parents[1] +DEMO_SPEC = ROOT / "sample_targets" / "rest_demo_openapi.json" + + +class RestWorkflowTest(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.server = create_server(port=0) + cls.server_thread = threading.Thread( + target=cls.server.serve_forever, + daemon=True, + ) + cls.server_thread.start() + cls.base_url = f"http://127.0.0.1:{cls.server.server_port}" + + @classmethod + def tearDownClass(cls) -> None: + cls.server.shutdown() + cls.server.server_close() + cls.server_thread.join(timeout=5) + + def test_local_openapi_demo_runs_end_to_end_and_writes_reports(self) -> None: + with tempfile.TemporaryDirectory() as output_dir: + report = RestOrchestrator().run( + str(DEMO_SPEC), + self.base_url, + output_dir, + ) + + self.assertEqual( + report.summary, + {"total": 3, "passed": 3, "failed": 0, "pass_rate": 100.0}, + ) + json_report = Path(output_dir) / "rest_report.json" + self.assertTrue(json_report.is_file()) + self.assertTrue((Path(output_dir) / "rest_report.html").is_file()) + saved_report = json.loads(json_report.read_text(encoding="utf-8")) + self.assertEqual(saved_report["summary"]["failed"], 0) + self.assertEqual( + [record["execution"]["status_code"] for record in saved_report["records"]], + [200, 200, 400], + ) + + def test_negative_case_does_not_pass_without_an_http_response(self) -> None: + test_case = RestTestCase( + case_id="REST-001", + endpoint=ApiEndpointAnalysis( + path="/echo", + method="POST", + operation_id="echoMessage", + summary="", + ), + description="missing body field message", + expected_success=False, + expected_status=400, + ) + + validation = RestValidator().validate( + test_case, + RestExecutionResult( + status_code=0, + response_text="connection refused", + error="connection refused", + ), + ) + + self.assertFalse(validation.passed) + self.assertIn("no HTTP response received", validation.reasons) + + def test_expected_response_fields_require_a_json_object(self) -> None: + test_case = RestTestCase( + case_id="REST-001", + endpoint=ApiEndpointAnalysis( + path="/health", + method="GET", + operation_id="getHealth", + summary="", + ), + description="valid request", + expected_response_fields=["status"], + ) + + validation = RestValidator().validate( + test_case, + RestExecutionResult(status_code=200, response_text="ok"), + ) + + self.assertFalse(validation.passed) + self.assertIn("expected a JSON object response", validation.reasons) + + def test_fail_on_failure_returns_nonzero_for_ci(self) -> None: + failed_report = SimpleNamespace( + summary={"total": 2, "passed": 1, "failed": 1} + ) + with patch( + "ai_agent_test_framework.cli.RestOrchestrator.run", + return_value=failed_report, + ): + with redirect_stdout(io.StringIO()): + exit_code = cli_main( + [ + "--spec", + str(DEMO_SPEC), + "--base-url", + self.base_url, + "--fail-on-failure", + ] + ) + + self.assertEqual(exit_code, 1) + + +if __name__ == "__main__": + unittest.main() From 403d46078195497c0ed23df3d2f91d186324a39a Mon Sep 17 00:00:00 2001 From: device-kunkun <175306042+device-kunkun@users.noreply.github.com> Date: Tue, 14 Jul 2026 00:48:37 +0800 Subject: [PATCH 2/2] Add release and contribution guidance --- .github/ISSUE_TEMPLATE/bug.yml | 81 ++++++++++++++++++++++++++++++ .github/ISSUE_TEMPLATE/config.yml | 1 + .github/ISSUE_TEMPLATE/feature.yml | 49 ++++++++++++++++++ .github/PULL_REQUEST_TEMPLATE.md | 18 +++++++ CHANGELOG.md | 31 ++++++++++++ README.md | 21 ++++++-- 6 files changed, 196 insertions(+), 5 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/bug.yml create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/feature.yml create mode 100644 .github/PULL_REQUEST_TEMPLATE.md create mode 100644 CHANGELOG.md diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml new file mode 100644 index 0000000..a568aff --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug.yml @@ -0,0 +1,81 @@ +name: Bug report +description: Report a reproducible problem in the CLI, generated checks, or reports +title: "[Bug]: " +labels: + - bug +body: + - type: markdown + attributes: + value: | + Thanks for reporting a problem. Use a local or synthetic target where possible, and remove credentials or confidential payloads before submitting. + + - type: dropdown + id: workflow + attributes: + label: Affected workflow + options: + - Python function analysis and execution + - OpenAPI parsing and REST execution + - JSON or HTML reporting + - Optional AI supplementation + - Java sample + - Other + validations: + required: true + + - type: textarea + id: description + attributes: + label: What happened? + description: Describe the observed behavior and why it is a problem. + validations: + required: true + + - type: textarea + id: reproduction + attributes: + label: Minimal reproduction + description: Include the smallest safe command, function, or OpenAPI fragment that reproduces the problem. + render: shell + validations: + required: true + + - type: textarea + id: expected + attributes: + label: Expected behavior + validations: + required: true + + - type: input + id: version + attributes: + label: Version or commit + placeholder: "0.1.0 or git commit SHA" + validations: + required: true + + - type: input + id: environment + attributes: + label: Environment + placeholder: "Python 3.13, Windows 11" + validations: + required: true + + - type: textarea + id: logs + attributes: + label: Relevant output + description: Paste only the relevant output after removing tokens, credentials, personal data, and private endpoints. + render: shell + + - type: checkboxes + id: checks + attributes: + label: Submission checks + options: + - label: I searched existing issues for the same problem. + required: true + - label: I removed credentials, confidential data, and private service details. + required: true diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..3ba13e0 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1 @@ +blank_issues_enabled: false diff --git a/.github/ISSUE_TEMPLATE/feature.yml b/.github/ISSUE_TEMPLATE/feature.yml new file mode 100644 index 0000000..0a4c7f6 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature.yml @@ -0,0 +1,49 @@ +name: Feature request +description: Propose a focused improvement backed by a concrete testing workflow +title: "[Feature]: " +labels: + - enhancement +body: + - type: markdown + attributes: + value: | + Explain the testing problem first. Focused proposals with a safe example and expected behavior are easiest to evaluate. + + - type: textarea + id: problem + attributes: + label: Problem to solve + description: Who encounters this problem, and in which Python or REST testing workflow? + validations: + required: true + + - type: textarea + id: proposal + attributes: + label: Proposed behavior + description: Describe the smallest useful outcome rather than only an implementation preference. + validations: + required: true + + - type: textarea + id: example + attributes: + label: Example + description: Show a safe input, CLI command, or expected report fragment when possible. + render: shell + + - type: textarea + id: alternatives + attributes: + label: Alternatives considered + description: Note any current workaround or smaller alternative. + + - type: checkboxes + id: scope + attributes: + label: Scope checks + options: + - label: This proposal is relevant to Python-function or OpenAPI/REST testing. + required: true + - label: The example does not require credentials or access to an unauthorized service. + required: true diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..93efbfd --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,18 @@ +## Summary + +- What changed? +- Which user or developer problem does it solve? + +## Validation + +- [ ] Python tests pass: `python -m unittest discover -s tests -v` +- [ ] Python sources compile: `python -m compileall -q src sample_targets tests` +- [ ] Java sample verifies when affected: `mvn -f java-springboot-sample/pom.xml -B -ntp verify` +- [ ] Documentation and generated examples match the implemented behavior + +## Safety and scope + +- [ ] The change contains no credentials, private endpoints, personal data, or generated reports +- [ ] REST examples target only local, synthetic, or explicitly authorized systems +- [ ] New behavior includes a focused test or executable example, or the change is documentation-only +- [ ] Known limitations and compatibility impact are documented diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..6ed1ffe --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,31 @@ +# Changelog + +All notable user-visible changes to this project are documented in this file. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## Unreleased + +### Added + +- Dependency-free local REST target and OpenAPI document for the first-run workflow. +- End-to-end coverage for OpenAPI parsing, real HTTP execution, validation, and JSON/HTML report output. +- Optional `--fail-on-failure` exit behavior for CI pipelines. +- Contributor guidance, structured issue forms, and a pull request checklist. + +### Changed + +- README now leads with the offline value proposition and a three-minute path to both supported workflows. + +### Fixed + +- Negative REST cases no longer pass when the target never returns an HTTP response. +- Expected response fields now fail validation when the response is not a JSON object. + +## 0.1.0 - 2026-07-13 + +### Added + +- Rule-based test generation and execution for Python functions. +- OpenAPI JSON parsing and REST request execution. +- Standalone JSON and HTML reports. +- Optional OpenAI-compatible supplemental case generation. +- Spring Boot sample target and GitHub Actions verification. diff --git a/README.md b/README.md index b4fcd65..de8f34f 100644 --- a/README.md +++ b/README.md @@ -4,10 +4,12 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) [![Python 3.9+](https://img.shields.io/badge/Python-3.9%2B-blue.svg)](pyproject.toml) -An extensible command-line testkit that turns Python function signatures or OpenAPI documents into executable test cases, then writes review-friendly JSON and HTML reports. +Turn a Python function or OpenAPI document into executable checks and review-friendly JSON/HTML reports. The default workflow is deterministic, offline, and does not require an API key. > 面向 Python 函数与 REST API 的智能测试工具：规则生成基础用例，可选使用大模型补充业务边界用例。 +**Try it in under three minutes:** use the [Python-function workflow](#3-minute-quick-start) or run the [zero-dependency REST demo](#zero-dependency-openapi--rest-demo). + ## Highlights - Analyze Python signatures, annotations, source code, and simple constraints. @@ -30,7 +32,7 @@ Python function / OpenAPI JSON JSON + HTML reports ``` -## Quick start +## 3-minute quick start Agentic API Testkit requires Python 3.9 or newer. @@ -72,6 +74,12 @@ api-testkit \ This exercises `GET /health`, a valid `POST /echo`, and an invalid request missing its required `message`. The command writes `rest_reports/rest_report.json` and `rest_reports/rest_report.html`. +```text +Completed REST spec sample_targets/rest_demo_openapi.json: 3/3 passed. +rest_reports/rest_report.json +rest_reports/rest_report.html +``` + `--fail-on-failure` returns exit code `1` after reports are written when any generated check fails, making the command suitable for CI. Without the flag, report failures do not change the process exit code. ## Spring Boot sample @@ -147,8 +155,11 @@ Generated reports, virtual environments, local credentials, Python caches, and J ## Community -Found a bug or have a focused feature idea? [Open an issue](https://github.com/device-kunkun/agentic-api-testkit/issues). Contributions are welcome; read [CONTRIBUTING.md](CONTRIBUTING.md) for the local checks and safety boundaries before opening a pull request. +Found a bug or have a focused feature idea? [Open a structured issue](https://github.com/device-kunkun/agentic-api-testkit/issues/new/choose). Contributions are welcome; read [CONTRIBUTING.md](CONTRIBUTING.md) for the local checks and safety boundaries before opening a pull request. User-visible changes are tracked in [CHANGELOG.md](CHANGELOG.md). -## Current scope +## Limitations and safety -The OpenAPI parser intentionally targets a practical subset of OpenAPI 3 JSON. It does not yet resolve external `$ref` documents, authentication schemes, or every schema composition keyword. Treat generated expectations as a starting point for review, not as a replacement for domain-specific assertions. +- The OpenAPI parser targets a practical subset of OpenAPI 3 JSON; YAML input is not supported yet. +- External `$ref` documents, authentication schemes, and some schema composition keywords are not resolved. +- Generated expectations are a reviewable baseline, not a replacement for domain-specific assertions. +- Run REST checks only against systems you own or are explicitly authorized to test.