Skip to content

feat: add benchmark results for TurboPuffer#743

Merged
liliu-z merged 5 commits intozilliztech:mainfrom
jamesgao-jpg:turbopuffer
Apr 3, 2026
Merged

feat: add benchmark results for TurboPuffer#743
liliu-z merged 5 commits intozilliztech:mainfrom
jamesgao-jpg:turbopuffer

Conversation

@jamesgao-jpg
Copy link
Copy Markdown
Collaborator

@jamesgao-jpg jamesgao-jpg commented Apr 1, 2026

Summary

  • Add --region CLI parameter (required) for region-based API routing, making --api-base-url an optional override for private networking
  • Fix SDK compatibility: rename write(columns=...) to write(upsert_columns=...) per current TurboPuffer SDK
  • Fix pickle error: defer tpuf.Turbopuffer client creation to init() to avoid serialization errors with ProcessPoolExecutor(spawn)
  • Fix recall comparison: cast search result IDs to int for ground truth matching
  • Fix docstring: correct copy-paste references to Pinecone/Milvus
  • Add benchmark results: consolidated TurboPuffer filter performance results (20 entries) and streaming benchmark entries (500/1000 rows/s)

Changes

Client fixes (backend/clients/turbopuffer/)

File Change
cli.py Add --region option (required); make --api-base-url optional with no default
config.py Add region field; make api_base_url optional (None default)
turbopuffer.py Defer client creation to _create_client() called in init() (fixes pickle); pass region to SDK; use upsert_columns instead of columns; cast result IDs to int; fix docstring

Benchmark results

File Change
results/TurboPuffer/result_20260331_standard_2025_turbopuffer.json Consolidated results from 22 individual TurboPuffer benchmark runs
results/leaderboard_v2.json +20 TurboPuffer filter performance entries
results/leaderboard_v2_streaming.json +2 TurboPuffer streaming benchmark entries (500/1000 rows/s)

Test plan

  • Run TurboPuffer benchmarks with --region aws-us-east-1 and verify client connects correctly
  • Verify --api-base-url override still works when provided
  • Confirm search recall scores match expected values (ID type casting fix)
  • Verify no pickle errors when running with multiprocess executor

…client

- Add --region CLI parameter (required) for region-based API routing
- Change --api-base-url to optional override for private networking
- Rename write(columns=...) to write(upsert_columns=...) per current SDK
- Fix docstring referencing wrong database name

Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
…ults

- Defer tpuf.Turbopuffer client creation to init() to avoid pickle
  errors with ProcessPoolExecutor(spawn)
- Cast search result IDs to int for ground truth recall comparison
- Update leaderboard_v2.json with 20 TurboPuffer filter performance entries

Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
…rboard

Merge 22 individual TurboPuffer result files into single consolidated
result file. Add streaming benchmark entries (500/1000 rows/s) to
leaderboard_v2_streaming.json. Normalize TurboPuffer db_name and label
in both leaderboard files.

Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
@jamesgao-jpg jamesgao-jpg changed the title Turbopuffer feat: add region parameter, fix SDK compatibility, and add benchmark results for TurboPuffer Apr 1, 2026
@jamesgao-jpg jamesgao-jpg changed the title feat: add region parameter, fix SDK compatibility, and add benchmark results for TurboPuffer feat: add benchmark results for TurboPuffer Apr 1, 2026
Ruff 0.15.8 flags these as RUF100 since S608 is not enabled in the
project's ruff config.

Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Copy link
Copy Markdown
Collaborator

@liliu-z liliu-z left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@sre-ci-robot
Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jamesgao-jpg, liliu-z
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@liliu-z liliu-z merged commit b39689b into zilliztech:main Apr 3, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants