Skip to content

Commit ecb8538

Browse files
assassin808assassin808
committed
Initial commit: HumanStudy-Bench community contribution hub
12 foundational studies, contributor tutorial docs, Next.js website, CI validation pipeline with contributor attribution checks.
0 parents  commit ecb8538

283 files changed

Lines changed: 94174 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/sync-studies-index.yml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
name: Sync studies index
2+
on:
3+
push:
4+
branches: [dev, main]
5+
paths:
6+
- "studies/**"
7+
- "scripts/build_studies_index.py"
8+
9+
jobs:
10+
rebuild:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v4
14+
- uses: actions/setup-python@v5
15+
with:
16+
python-version: "3.11"
17+
- name: Build studies index
18+
run: python scripts/build_studies_index.py
19+
- name: Commit updated index
20+
uses: stefanzweifel/git-auto-commit-action@v5
21+
with:
22+
commit_message: "chore: rebuild studies_index.json"
23+
file_pattern: co_website/data/studies_index.json

‎.github/workflows/validate-studies.yml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
name: Validate studies
2+
on:
3+
pull_request:
4+
branches: [dev, main]
5+
paths:
6+
- "studies/**"
7+
- "scripts/build_studies_index.py"
8+
- "scripts/verify_study.sh"
9+
10+
jobs:
11+
validate:
12+
runs-on: ubuntu-latest
13+
steps:
14+
- uses: actions/checkout@v4
15+
- uses: actions/setup-python@v5
16+
with:
17+
python-version: "3.11"
18+
- name: Install dependencies
19+
run: pip install -r requirements.txt
20+
- name: Build studies index
21+
run: python scripts/build_studies_index.py
22+
- name: Verify changed studies
23+
run: |
24+
# Find study directories changed in this PR
25+
CHANGED_STUDIES=$(git diff --name-only origin/${{ github.base_ref }}...HEAD -- studies/ | grep -oP 'studies/[^/]+' | sort -u)
26+
if [ -z "$CHANGED_STUDIES" ]; then
27+
echo "No study directories changed."
28+
exit 0
29+
fi
30+
for study_path in $CHANGED_STUDIES; do
31+
study_id=$(basename "$study_path")
32+
echo "--- Verifying $study_id ---"
33+
bash scripts/verify_study.sh "$study_id"
34+
echo ""
35+
done
36+
- name: Verify contributor attribution
37+
env:
38+
PR_AUTHOR: ${{ github.event.pull_request.user.login }}
39+
run: |
40+
CHANGED_STUDIES=$(git diff --name-only origin/${{ github.base_ref }}...HEAD -- studies/ | grep -oP 'studies/[^/]+' | sort -u)
41+
if [ -z "$CHANGED_STUDIES" ]; then
42+
exit 0
43+
fi
44+
FAILED=0
45+
for study_path in $CHANGED_STUDIES; do
46+
INDEX="$study_path/index.json"
47+
if [ ! -f "$INDEX" ]; then
48+
continue
49+
fi
50+
# Check if any contributor github field contains the PR author's username
51+
MATCH=$(python3 -c "
52+
import json, sys
53+
with open('$INDEX') as f:
54+
data = json.load(f)
55+
contributors = data.get('contributors', [])
56+
if not contributors:
57+
print('MISSING')
58+
sys.exit(0)
59+
author = '$PR_AUTHOR'.lower()
60+
for c in contributors:
61+
gh = c.get('github', '').lower()
62+
# Match 'username' or 'https://github.com/username'
63+
if gh.rstrip('/').split('/')[-1] == author:
64+
print('MATCH')
65+
sys.exit(0)
66+
print('MISMATCH')
67+
")
68+
study_id=$(basename "$study_path")
69+
if [ "$MATCH" = "MISSING" ] || [ "$MATCH" = "MISMATCH" ]; then
70+
echo "::error file=$INDEX::Contributor check failed for $study_id. Your GitHub username is '$PR_AUTHOR'. Please add the following to your index.json:%0A%0A\"contributors\": [%0A { \"name\": \"Your Name\", \"github\": \"$PR_AUTHOR\" }%0A]"
71+
FAILED=1
72+
else
73+
echo "$study_id: Contributor @$PR_AUTHOR verified."
74+
fi
75+
done
76+
if [ "$FAILED" -eq 1 ]; then
77+
exit 1
78+
fi

‎.gitignore‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Python
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
*.so
6+
.Python
7+
build/
8+
develop-eggs/
9+
dist/
10+
downloads/
11+
eggs/
12+
.eggs/
13+
lib/
14+
lib64/
15+
parts/
16+
sdist/
17+
var/
18+
wheels/
19+
*.egg-info/
20+
.installed.cfg
21+
*.egg
22+
23+
24+
# Virtual environments
25+
venv/
26+
env/
27+
ENV/
28+
.venv
29+
30+
# IDEs
31+
.vscode/
32+
.idea/
33+
*.swp
34+
*.swo
35+
*~
36+
.DS_Store
37+
38+
# Jupyter Notebook
39+
.ipynb_checkpoints
40+
*.ipynb_checkpoints/
41+
42+
# Testing
43+
.pytest_cache/
44+
.coverage
45+
htmlcov/
46+
coverage.xml
47+
.tox/
48+
.nox/
49+
50+
# Type checking
51+
.mypy_cache/
52+
.dmypy.json
53+
dmypy.json
54+
.pytype/
55+
56+
# Results and logs
57+
results/*
58+
!results/benchmark/
59+
results/benchmark/study_*/
60+
results/benchmark/full_benchmark_*.json
61+
logs/
62+
*.log
63+
64+
# Data (except schemas and registry)
65+
# Uncomment if you want to exclude large study materials
66+
# studies/*/materials/large_files/
67+
68+
# Environment variables
69+
.env
70+
.env.local
71+
72+
# OS
73+
Thumbs.db
74+
75+
# AI assistants
76+
.claude/
77+
.cursor/
78+
CLAUDE.md
79+
claude.md
80+
81+
# co_website and build artifacts (lite branch)
82+
co_website/.history/
83+
co_website/node_modules/
84+
co_website/.next/
85+
humanstudybench.egg-info/
86+
dist/
87+
site/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Xuan Liu
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
<div align="center">
2+
<img src="docs/img/new-HS-bench_logo.png" alt="HumanStudy-Bench Logo" width="300">
3+
4+
<h1>HumanStudy-Bench: Towards AI Agent Design for Participant Simulation</h1>
5+
6+
<h3><a href="https://www.hs-bench.clawder.ai">📊 See Leaderboard & Results</a> | <a href="https://arxiv.org/abs/2602.00685">📖 Read the Paper</a></h3>
7+
8+
[![Release v1.0.0](https://img.shields.io/github/v/release/AISmithLab/HumanStudy-Bench)](https://github.com/AISmithLab/HumanStudy-Bench/releases/tag/v1.0.0)
9+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
10+
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
11+
[![Docs](https://img.shields.io/badge/docs-website-blue)](https://www.hs-bench.clawder.ai)
12+
13+
</div>
14+
15+
---
16+
17+
> HumanStudy-Bench is a standardized testbed for replaying human-subject experiments with LLM agents. We currently include **12 foundational studies** — and we need your help to grow the benchmark. This repository is the community contribution hub.
18+
19+
## How to Contribute a Study
20+
21+
### 1. Fork and clone
22+
23+
```bash
24+
git clone https://github.com/<your-github-id>/HumanStudy-Bench.git
25+
cd HumanStudy-Bench
26+
git checkout -b contrib-<yourgithubid>-013
27+
```
28+
29+
### 2. Create your study folder
30+
31+
Add a new directory under `studies/` with the required folders:
32+
33+
```
34+
studies/<yourgithubid>_013/
35+
├── index.json
36+
├── source/
37+
├── scripts/
38+
└── README.md
39+
```
40+
41+
See the docs below for what goes inside each folder and the exact schemas:
42+
43+
| # | Guide | Description |
44+
|---|-------|-------------|
45+
| 1 | [What Should I Submit?](https://www.hs-bench.clawder.ai/docs/what_to_submit) | Overview of contribution, required folders and files |
46+
| 2 | [How to Extract Data from a Paper](https://www.hs-bench.clawder.ai/docs/extract_from_paper) | Paper hierarchy, AI extraction prompt, walkthrough example |
47+
| 3 | [How to Build Your Study Files](https://www.hs-bench.clawder.ai/docs/build_study_files) | Schemas, code examples, and contracts for each file |
48+
| 4 | [How to Submit Your Study](https://www.hs-bench.clawder.ai/docs/submit_study) | Fork, verify, push, and open a PR |
49+
50+
### 3. Verify locally
51+
52+
```bash
53+
bash scripts/verify_study.sh <yourgithubid>_013
54+
```
55+
56+
### 4. Commit and push
57+
58+
```bash
59+
git add studies/<yourgithubid>_013/
60+
git commit -m "Add study: <Your Study Title>"
61+
git push origin contrib-<yourgithubid>-013
62+
```
63+
64+
### 5. Open a Pull Request
65+
66+
Open a PR on GitHub targeting the `dev` branch. Maintainers assign final `study_XXX` numbering by merge order. CI runs validation automatically; confirmation is by human review.
67+
68+
You can also submit a study via **web upload** at [hs-bench.clawder.ai/contribute](https://www.hs-bench.clawder.ai/contribute).
69+
70+
## Existing Studies
71+
72+
The 12 foundational studies (cognition, strategic interaction, social psychology) serve as reference examples. Browse them on the [website](https://www.hs-bench.clawder.ai/contribute#studies) or locally under `studies/`.
73+
74+
## Citation & Hugging Face
75+
76+
If you use HumanStudy-Bench, please cite:
77+
78+
```bibtex
79+
@misc{liu2026humanstudybenchaiagentdesign,
80+
title={HumanStudy-Bench: Towards AI Agent Design for Participant Simulation},
81+
author={Xuan Liu and Haoyang Shang and Zizhang Liu and Xinyan Liu and Yunze Xiao and Yiwen Tu and Haojian Jin},
82+
year={2026},
83+
eprint={2602.00685},
84+
archivePrefix={arXiv},
85+
primaryClass={cs.AI},
86+
url={https://arxiv.org/abs/2602.00685},
87+
}
88+
```
89+
90+
**Hugging Face:** Benchmark and resources are available on the [Hugging Face Hub](https://huggingface.co/) — `fuyyckwhy/HS-Bench-results`.
91+
92+
## License
93+
94+
MIT License. See [LICENSE](LICENSE) for details.

‎co_website/.gitignore‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
2+
3+
# dependencies
4+
/node_modules
5+
/.pnp
6+
.pnp.*
7+
.yarn/*
8+
!.yarn/patches
9+
!.yarn/plugins
10+
!.yarn/releases
11+
!.yarn/versions
12+
13+
# testing
14+
/coverage
15+
16+
# next.js
17+
/.next/
18+
/out/
19+
20+
# production
21+
/build
22+
23+
# misc
24+
.DS_Store
25+
*.pem
26+
27+
# debug
28+
npm-debug.log*
29+
yarn-debug.log*
30+
yarn-error.log*
31+
.pnpm-debug.log*
32+
33+
# env files (can opt-in for committing if needed)
34+
.env*
35+
36+
# vercel
37+
.vercel
38+
39+
# typescript
40+
*.tsbuildinfo
41+
next-env.d.ts

‎co_website/README.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
## co_website (HumanStudy-Bench web app)
2+
3+
Next.js app for:
4+
- Homepage sections (`Overview`, `Leaderboard`, `Dataset`)
5+
- Study catalog and study detail pages
6+
- Contribute flow (upload + PR creation)
7+
- Docs tab (`/docs`)
8+
9+
## Local preview
10+
11+
```bash
12+
cd co_website
13+
npm install
14+
npm run dev
15+
```
16+
17+
Open `http://localhost:3000`.
18+
19+
## Build checks
20+
21+
```bash
22+
cd co_website
23+
npm run lint
24+
npm run build
25+
```
26+
27+
## Study ZIP download
28+
29+
ZIP downloads are served by:
30+
- `GET /api/studies/:studyId/zip`
31+
32+
This endpoint packages `studies/:studyId` directly from the repository.
33+
34+
## Refreshing effect-size plot data
35+
36+
Homepage effect-size scatter reads:
37+
- `public/data/effects/gemini_flash_v4_effect_data.json`
38+
39+
To refresh:
40+
1. From repo root: `python scripts/plot_single_fig3_effects.py --export-data /absolute/path/to/co_website/public/data/effects/gemini_flash_v4_effect_data.json`
41+
2. Rebuild/redeploy the site.

0 commit comments

Comments
 (0)