README.md

GitHub Action Rollout Processor Hello World

Run distributed LLM evaluations using GitHub Actions as your rollout processor with automatic trace collection. Scale your evaluations in the cloud without managing infrastructure.

📖 New to GitHub Action Rollout Processing? Read the complete GitHub Actions Rollout Tutorial first - this repository serves as the working example for that tutorial.

Quick Start

Repository Setup

1. Fork this repository to your GitHub account, then clone your fork:

   git clone https://github.com/YOUR-USERNAME/github-action-rollout-processor-hello-world.git
   cd github-action-rollout-processor-hello-world

2. Update the repository configuration in quickstart.py:

   rollout_processor=GithubActionRolloutProcessor(
       owner="YOUR-USERNAME",  # Change this to your GitHub username
       repo="github-action-rollout-processor-hello-world",
       # ... rest of configuration
   )

3. Set up your local environment:

   For the GitHub token, create a Personal Access Token (classic) with permissions for repo and workflow.

   export GITHUB_TOKEN="ghp_..."
   export FIREWORKS_API_KEY="your_fireworks_key"

Installation

pip install eval-protocol

Running the Example

pytest quickstart.py -vs

What's Happening

1. /init triggers one rollout: Eval Protocol dispatches a GitHub Actions workflow with completion_params, metadata (incl. rollout_id), and model_base_url.
2. Polling to check rollout status: The processor finds the rollout:<rollout_id> run and polls GitHub Actions until it completes.
3. Send chat completions and store as trace: The workflow executes your agent and sends completions/logs to Fireworks with the rollout’s correlation tags.
4. Once rollout finished, pull full trace and evaluate: Eval Protocol fetches the Fireworks trace by rollout_id and scores the result.

Example: See a successful rollout execution to understand what the GitHub Actions workflow looks like in practice.

Viewing Results

After running the test, start the local UI server to view results:

ep logs

Visit http://localhost:8000 to explore your evaluation results with detailed traces and metrics.

Learn More

• Eval Protocol Documentation
• GitHub Actions Documentation
• Fireworks AI Platform
• GitHub Actions Rollout Tutorial