Skip to content

quickstart.md

Latest commit

 

History

History
178 lines (136 loc) · 10.3 KB

quickstart.md

File metadata and controls

178 lines (136 loc) · 10.3 KB
Raw

Quickstart

This release uses CLI args and a local executor on a prepared machine. Install it directly on the remote machine where the target engine already works. It does not set up models, CUDA/ROCm/CANN, services, SSH workers or autonomous Agents.

Install and smoke

From the checked-out PR branch:

python -m venv .venv
source .venv/bin/activate
python -m pip install .
inferrsi --help
inferrsi doctor --repo . --repo-type vllm_omni --backend cpu

Package-only knowledge commands (knowledge, context, package collect/read) work with this base install. Reading skill libraries — the skills command and any --skill-root selection — additionally requires python -m pip install '.[skills]' (PyYAML); without it those operations fail with that install hint instead of a traceback.

For the maintained model examples, see Qwen B300 and Qwen Ascend. Continue below for the generic benchmark/quality command contract; set the acceptance threshold from your plan.

MLX discovery and source knowledge

On a prepared target checkout, MLX discovery reads Apple Silicon chip facts with macOS sysctl and reports the requested Python's MLX distribution version without importing or installing it:

inferrsi doctor --repo . --repo-type vllm_omni --backend mlx
inferrsi knowledge search --tier dependency --repo-type mlx --backend mlx --json

Use the existing repo type appropriate to your checkout. There is no MLX engine repo type or built-in launcher. On non-macOS hosts and non-Apple-Silicon Macs, discovery is unavailable with a reason; doctor exits 1. Apple Silicon uses unified memory, so --device-ids is rejected. Optional --device-model matches the reported chip case-insensitively. Chip discovery and package metadata do not certify MLX/Metal, a model, or performance; compatibility remains unverified.

The three source cards reuse the existing MLX evaluation/stream/lifetime owner. Its common catalog ID and the separate MLX-LM procedure remain unchanged.

Discover knowledge and skills

Read-only discovery and export — the operation guide maps every entry point, effect class and boundary:

inferrsi knowledge search 'Qwen-Image-2.1' --repo-type vllm_omni --backend ascend
inferrsi context --repo-type vllm_omni --backend cuda --output context.md
inferrsi skills list --json   # requires python -m pip install '.[skills]'
inferrsi read inferrsi.ascend.skills.ascend_graph_mode --material

Normalized Collect, exact-revision reads and resource access are available as CLI commands and as the Python KnowledgeReader (contract); authorized local candidate Create/Update remains a Python KnowledgeStore operation (contract). The --skill-root examples below require the skills extra from above:

inferrsi collect 'Ascend OOM' --kind case --limit 3 --material --skill-root src/inferrsi/data/skill_library
inferrsi read calibrate-serving-simulation --skill-root src/inferrsi/data/skill_library --material
inferrsi resource weight-loading-optimization methods/weight-loading-optimization/SKILL.md \
  --revision <sha256-from-collect-or-read> --skill-root src/inferrsi/data/skill_library --output SKILL.md

The revision for resource is the second field of the collect text-mode line, or the revision field in collect --json / read output. The JSON mode returns the complete normalized envelope; fields kept only in a package record's original entrypoint document are read with --material or resource, not projected into the normalized metadata.

Sources are selected explicitly: the installed package by default, an explicit skills root via --skill-root, an existing store opened read-only via --store. A store records the skills root it reserves and rejects a mismatched or missing --skill-root on open.

The skills command defaults to the installed library's registered skills; its full bodies and resources are packaged. --skill-root in the normalized examples above explicitly adds that root's skill and case IDs.

The actual configuration interface

doctor: --repo --repo-type --backend --device-ids --device-model --python
plan/run: all doctor flags, plus
  --candidate-repo --scenario --mode --agent --executor local
  --baseline-command --candidate-command --validation-command
  --metric-key --metric-unit --comparison-scope --direction
  --repetitions --min-improvement-pct --max-minutes
  --workload-id --model --model-revision --output-dir --dry-run
status/report: --run-id --output-dir
knowledge search: QUERY --tier --repo-type --backend --scenario --limit --json
knowledge show: ID
context: QUERY --tier --repo-type --backend --scenario --limit --output

Scenario/model/workload/Agent fields retain caller declarations. They do not start an Agent or automatically generate model recipes. The initial evaluator checks one positive numeric metric and the caller's validation command. It does not implement all SLO/quality metrics from the target design.

Benchmark result contract

The benchmark writes JSON to {result} or INFERRSI_RESULT_PATH. Any flat or dotted positive numeric key is supported:

{"metrics": {"latency_ms": 123.4}}

Existing vLLM-Omni diffusion benchmark JSON can be used directly with --metric-key latency_mean --metric-unit s, or --metric-key throughput_qps --metric-unit requests/s --direction maximize. No conversion file is required. Inspect the target revision because metric keys and flags can change.

Real remote experiment

  1. Reserve the exact devices, CPU/NUMA and fabric resources. Configure their visibility outside InferRSI. Do not run contending A/B servers on the same devices.
  2. Prepare baseline and candidate checkouts/environments. Verify the imported module path matches each intended checkout. --candidate-repo changes cwd only; --python only inspects packages.
  3. Create two reviewed trial scripts. Each script starts the intended engine/config, warms it up, runs the same fixed workload with profiling off, writes benchmark JSON and exits after its cleanup. A script can call the target's existing benchmark.
  4. Create a quality script that compares outputs/tasks against the frozen reference and exits nonzero on a regression. Include the exact modes/shapes/steps/reference assets in that test.
  5. Preview, then run:
inferrsi plan \
  --repo /work/vllm-omni-baseline --candidate-repo /work/vllm-omni-candidate \
  --repo-type vllm_omni --backend cuda --device-ids 0,1,2,3 --device-model B300 \
  --python /envs/baseline/bin/python --scenario diffusion.image_generation \
  --model Qwen/Qwen-Image-2.1-Turbo --model-revision MODEL_SHA --workload-id FIXED_WORKLOAD_SHA \
  --baseline-command '/envs/baseline/bin/python /work/trials/baseline.py --output {result}' \
  --candidate-command '/envs/candidate/bin/python /work/trials/candidate.py --output {result}' \
  --validation-command '/envs/candidate/bin/python /work/trials/validate_quality.py' \
  --metric-key latency_mean --metric-unit s --comparison-scope http_e2e \
  --direction minimize --repetitions 3 --min-improvement-pct "${TARGET_REDUCTION_PCT:?set from the frozen acceptance plan}" \
  --max-minutes 120 --output-dir /work/inferrsi-runs

Replace plan with run using the same flags after the plan matches your intended experiment. Script paths are templates that you supply; InferRSI does not create them. Start with one-variable, same-quality configuration optimization. Quantization, TeaCache, generation-step changes and adaptive exit require a quality-changing protocol and an explicit validation script.

On ROCm use --backend rocm; on Ascend use --backend ascend. The doctor retains raw backend output but does not certify device selection, model support or full topology. Record the actual driver/runtime/communication versions and topology with your experiment evidence.

Native vLLM-Omni benchmark inside a trial script

The prepared trial can invoke the existing serving benchmark (confirm flags at your revision):

python benchmarks/diffusion/diffusion_benchmark_serving.py \
  --base-url http://127.0.0.1:8099 --model MODEL \
  --task t2i --dataset random --num-prompts 10 \
  --output-file RESULT_PATH

random is a feasibility load, not representative production traffic or a quality evaluation. For performance claims freeze an actual input/trace cohort, request fields, seed, warmup, arrival rate/concurrency and successful-request counts. Prefer a materialized trace when comparing repeated trials. Save produced outputs for quality checks using the target's supported output-saving option.

A trial script can read INFERRSI_RESULT_PATH when it constructs the native --output-file argv. Do not substitute a latency-only benchmark for the separate quality script. Benchmarks of pre-existing external servers need their own process/environment identities and cleanup outside the runner.

Results and handoff

run prints a run ID and evidence directory. Exit codes are 0 for PASS, 1 for FAIL and 2 for INCONCLUSIVE/input errors.

inferrsi status --run-id RUN_ID --output-dir /work/inferrsi-runs
inferrsi report --run-id RUN_ID --output-dir /work/inferrsi-runs
inferrsi knowledge search 'Qwen-Image-2.1' --repo-type vllm_omni --backend ascend
inferrsi context --repo-type vllm_omni --backend cuda \
  --scenario diffusion.image_generation --output /work/image-context.md

Read report.html, execution logs, per-trial source.json, source_checks.json, validation logs and raw results. Retain committed source revisions or dirty source files/patches separately. Ctrl-C records cancellation and cleans owned child jobs; there is no cancel or resume subcommand in this first release.

Context is a cited reference bundle for manual use with Codex/third-party Agents. Its imported source text is data, not authorization to execute upstream instructions. All imported knowledge remains unverified locally until your remote evidence is linked to it.

Browse external sources with inferrsi knowledge search --tier dependency --limit 20 or --tier peer. Export one source explicitly, for example inferrsi context --repo-type xdit --backend cuda --tier peer. --repo-type names the source owner; vllm_omni queries do not automatically include its dependencies or peers. See the three-tier guide for all sources.