GraphQL Conformance

View the dashboard

What is this?

This project tests whether open-source GraphQL implementations behave the same way. It runs the same queries against each implementation and compares the results to a reference (graphql-js).

Implementations tested:

• graphql-js 17 (reference)
• graphql-js 16
• graphql-java
• Viaduct
• graphql-go
• graphql-php
• graphql-core
• gqlgen
• Hot Chocolate
• graphql-dotnet
• async-graphql
• Absinthe
• Juniper

How it works

Each implementation is wrapped in a small harness that accepts a schema file and a query file, builds the schema, runs the query, and prints the result as JSON. All harnesses use the same deterministic resolvers (the Wiring Spec) so that the only differences come from the GraphQL engine itself.

Test cases are generated randomly using the Viaduct Arbitrary toolkit. This produces arbitrary GraphQL schemas, documents, and variables, which are stored in corpus.

A coordinator runs every test case against the reference first. If the reference produces a result, the test is runnable and every implementation is compared against that result. If the reference crashes, times out, or emits invalid JSON, the test is excluded from scoring for that run and is not attempted on any other implementation.

Requirements

• mise — manages tool versions (Node.js, Go, Java, .NET, Rust, Python)

Then, in the repository root, run:

mise trust          # Tell mise you trust this repo
mise install        # Install the various runtimes

That's it. mise handles installing the right versions of everything else.

MacOS

On macOS, mise install php may require a newer bison on PATH than the system default.

Ubuntu

On Ubuntu you might need some additional dependencies; here's some common deps required to build the common programming language runtimes from source (quite a lot of these are a belt-and-braces list for PHP):

sudo apt update
sudo apt install \
  build-essential build-dep autoconf automake libtool pkg-config \
  libffi-dev libssl-dev zlib1g-dev \
  libbz2-dev libreadline-dev libsqlite3-dev \
  libncurses-dev libncursesw5-dev \
  libyaml-dev libxml2-dev libcurl4-openssl-dev re2c \
  curl git bison plocate libgd-dev libicu-dev libzip-dev \
  libonig-dev libpq-dev libzip-dev libjpeg-dev libpng-dev \
  libxpm-dev libmysqlclient-dev libfreetype6-dev libldap2-dev \
  libxslt-dev libldb-dev

Quick start

make build          # clone libraries and build all implementations
make test           # run all tests
make ci-smoke       # cross-platform smoke target used by CI
make run-conformer  # run conformance suite and update the dashboard
make serve-site     # serve the dashboard locally

Other commands

make gen-corpus                                        # regenerate test cases
make run-impl IMPL=graphql-go TEST=corpus/0/0          # run one impl on one test
make diff-impl IMPL=graphql-go TEST=corpus/0/0         # diff an impl against the reference
make clean-corpus                                      # delete generated test cases (keeps corpus/0)
make clean-results                                     # delete stored results
make clean                                             # clean all build artifacts

Project layout

corpus/           test cases (schema + query + optional variables)
corpus-gen/       test case generator (Kotlin)
conformer/        coordinator that runs tests and compares results (Node.js)
impls/            one directory per GraphQL implementation
results/          results store (writes to results/data/)
site/             static dashboard (reads from site/data/)

Adding an implementation

1. Create impls/<name>/ with code that implements the Wiring Spec
2. Add a Makefile with build, test, clean targets
3. Add an entry to config.json
4. Add any new tool versions to .mise.toml
5. Run make build && make test