Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,16 @@ MODEL_NAME=google/gemini-2.5-flash
# LAYA_SERVED_TIMEOUT_S=5 # per decision, the retry and any /health refresh included
# LAYA_SERVED_MAX_LEN=512 # the served checkpoint's window: english 512, multilingual 1024

# ---- Content desk (python -m s1a.content; draft Omni workers, not the shared --model flag) ----
# Unset OMNI_BASE_URL and the desk uses the lexical rule. No Clef URL: that draft worker is refused.
# OMNI_BASE_URL=http://127.0.0.1:8000
# OMNI_MODEL=StartLux-Decision-v1.1-4B-Instant
# OMNI_API_KEY=
# OMNI_TIMEOUT_S=30
# OMNI_FACTCHECK_BASE_URL=http://127.0.0.1:8001 # required for --fact-check-worker pplx
# OMNI_FACTCHECK_MODEL=perplexity-ai/pplx-decider-v1.1-27b
# OMNI_FACTCHECK_API_KEY=

# ---- CLM (behind --model clm; `clm-serve` from the CLM checkout, no extra needed) ----
# CLM owns everything after a frozen Qwen3-8B encoder, so clm-serve needs an /v1/embeddings server beside it;
# system1-omni's recipe/clm/README.md runs the pair. There is no cloud key.
Expand Down
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,14 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver

### Added

- Content desk workflow (`python -m s1a.content`): RSS and fixture ingest, an optional X fixture or adapter
hook, a keep/drop screen with a relevance score, a human pick of 3–5 items, a fact-check choice and score,
and a human publish decision. Nothing is posted. `OMNI_BASE_URL` sends short `/v1/systemone` prefills to a
draft StartLux Instant 4B worker (system1-omni #164); fact-check can switch to draft PPLX (#166) on
`OMNI_FACTCHECK_BASE_URL`. Clef (#165) is refused. With the URL unset, a lexical rule runs offline.
`--allow-unsafe-skip-gates` is unsafe and still does not publish. `latency-report` is the unattached hook
for a later Public231-C1-style ceiling. Recipe: `recipes/content-workflow`.

- `s1a probe --order reverse|shuffle:N` (with `--seed` and `--json PATH`): the option-order check. Every case is
asked in file order and again with the same keys and texts in another dict order; the probe reports the flip
rate, accuracy in each order and a row per case with every order asked. Verdict and exit code are unchanged
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@ The gates and the templates: [docs/skills.md](docs/skills.md#build-a-system-1-ag
- `flights`: browser use, a Google Flights search over `@playwright/mcp`.
- `desktop`: computer use, any Windows or macOS app window through [Cua Driver](https://cua.ai/docs/cua-driver).
- `ticket_router`: 30 labelled support tickets to five queues.
- Content desk: RSS and fixture ingest, a screen, a human pick, a fact-check, and a human publish decision. Nothing is auto-published. Recipe: [recipes/content-workflow](recipes/content-workflow/README.md).
- `alfworld`: household tasks in text, with the AI2-THOR scene in the replays.
- `game2048`, `millionaire`, `blackjack`: games with a score per episode.
- `injection_guard`: a rail that answers one question at a hook of a running agent and fails closed.
Expand Down
12 changes: 12 additions & 0 deletions docs/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,15 @@ shipped set is 30 tickets in `s1a/agents/_data/ticket_router_eval.jsonl`, each w
order status; the label is read by the scorer alone. `--dataset` points the agent at a
JSONL of your own with the same fields, and `--batch-size` caps the tickets per episode. The rules text the model
reads is `RULES` in `s1a/agents/ticket_router.py`.

## The content desk

The news desk is not a row in the table above. It is a five-stage command, `python -m s1a.content run`, because
the human pick and the human publish decision have to stop the run. The shared tool loop would act on every
choice. Screen and fact-check are still enumerated decisions: `keep`/`drop` plus a relevance score, then
`supported`/`unsupported`/`needs_human` plus a support score. With `OMNI_BASE_URL` unset, a lexical rule
answers, the same kind of baseline as the ticket router's keywords. With the variable set, a draft
System1-Omni worker answers over `POST /v1/systemone`. The default worker is StartLux Instant 4B. Fact-check
can switch to PPLX on `OMNI_FACTCHECK_BASE_URL`. Clef is refused. The run records a publish decision and
does not post. Details: [content-workflow.md](content-workflow.md) and the
[recipe](../recipes/content-workflow/README.md).
2 changes: 1 addition & 1 deletion docs/capabilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ For issue trackers and serving support, see [System1-Omni model tracker #83](htt
| Cua-S1 4B 0.2 text adapter (`Qwen/Qwen3.5-4B` + `cua-ai/cua-s1-4b-0.2/text`, `CUA_S1_VARIANT=4b`, `CUA_S1_MODALITY=text`) | Text | `choice` only | Tool decisions and text-only client paths; desktop option grounding is implemented | In-process Transformers/PEFT scorer | Contract tests use a fake scorer; real checkpoint/device validation **not present in this checkout**. |
| Cua-S1 4B 0.2 multimodal adapter (`Qwen/Qwen3.5-4B` + `cua-ai/cua-s1-4b-0.2/multimodal`, variant `4b`) | One current screenshot plus text task/context | `choice` only; maximum 26 options in text and multimodal modes; multimodal requires exactly one screenshot | Screenshot-dependent desktop decision adapter; this does not itself complete the Omni capture/execute bridge | In-process Transformers/PEFT scorer with screenshot passed as a temporary file | Contract tests use a fake scorer; separate historical external evidence reports a bounded MPS fixture run (see below). Current-main reproduction and CUDA validation remain unverified. |

`random` is a seeded uniform-random choice baseline; `rule` is a hand-written one-hot choice policy. Neither is a model checkpoint. Both implement choice-only selection. `llm` refers to the separate chat model, not a decision-model backend. The `score` answer shape in Omni is not evidence that an Agents client can issue a score question.
`random` is a seeded uniform-random choice baseline; `rule` is a hand-written one-hot choice policy. Neither is a model checkpoint. Both implement choice-only selection. `llm` refers to the separate chat model, not a decision-model backend. The `score` answer shape in Omni is not evidence that an Agents client can issue a score question. The content desk (`python -m s1a.content`) is a separate client: when `OMNI_BASE_URL` is set it posts one short `choice` prefill and one short `score` prefill per item to a draft System1-Omni worker. That does not add `score` to `s1a decide`, and a live worker has not been measured here. See [content-workflow.md](content-workflow.md).

## Real validation evidence

Expand Down
7 changes: 7 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,13 @@ Variables can be exported in your shell or placed in a `.env` file at the root o
| `LAYA_SERVED_API_KEY` | `laya-served` model | *(unset)* | Bearer token, the server's `LAYA_API_KEY` when it sets one. |
| `LAYA_SERVED_TIMEOUT_S` | `laya-served` model | `5` | Deadline per decision in seconds, the one retry and any `/health` refresh included. A finite number above 0: `nan`, `inf` and `0` are configuration errors. |
| `LAYA_SERVED_MAX_LEN` | `laya-served` model | `512` | The served checkpoint's token window per question (`english` 512, `multilingual` 1024); a request that fills it raises. The server takes the window from the checkpoint, so set this to match and nothing higher. |
| `OMNI_BASE_URL` | content desk, screen (and fact-check unless a second URL is set) | *(unset)* | Base URL of a draft System1-Omni worker, with no path or query. The client appends `/v1/systemone` and `/health`. Unset, `python -m s1a.content run --backend auto` uses the lexical rule and does not open a decision socket. |
| `OMNI_MODEL` | content desk screen | `StartLux-Decision-v1.1-4B-Instant` | Model alias sent to the screen worker. The default is the StartLux Instant 4B draft (system1-omni pull request 164). The thinking flag stays `think_mode: never`. |
| `OMNI_API_KEY` | content desk | *(unset)* | Bearer token for `OMNI_BASE_URL`, sent only when non-empty. |
| `OMNI_TIMEOUT_S` | content desk | `30` | Per-request timeout in seconds. A finite number above 0. There is no retry, so the latency hook can read one wall time. |
| `OMNI_FACTCHECK_BASE_URL` | content desk fact-check | *(same as `OMNI_BASE_URL` for StartLux)* | Required when `--fact-check-worker pplx`. PPLX (system1-omni pull request 166) is not assumed to share the StartLux port. |
| `OMNI_FACTCHECK_MODEL` | content desk fact-check | StartLux alias, or `perplexity-ai/pplx-decider-v1.1-27b` for PPLX | Model alias for the fact-check worker. |
| `OMNI_FACTCHECK_API_KEY` | content desk fact-check | `OMNI_API_KEY` | Bearer token for the fact-check URL when it differs. |
| `CUA_S1_CHECKPOINT` | `cua` model | `cua-ai/cua-s1-nano-0.1` | Hugging Face checkpoint ID or local directory for Cua-S1 Nano option scorer. |
| `CUA_S1_SUBFOLDER` | `cua` model | `text` | Subfolder within checkpoint directory containing text option scoring weights. |
| `CUA_S1_DEVICE` | `cua` model | `auto` | PyTorch device used for Cua-S1 Nano evaluation (`auto`, `cpu`, `cuda`, or `mps`). |
Expand Down
91 changes: 91 additions & 0 deletions docs/content-workflow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Content workflow

Jev screens. A person decides. The desk path is five stages:

1. Ingest news from RSS and, when an adapter is supplied, Twitter/X.
2. Screen each item to keep or drop, and score relevance.
3. A person picks 3 to 5 kept items for deep research.
4. Fact-check those items' claims against the sources the person supplied, as a choice and a score.
5. A person decides publish or hold. The run records that decision and posts nothing.

System1-Agents owns ingest, the gates, and the report. A draft System1-Omni worker is used only for stage 2 and stage 4, and only when `OMNI_BASE_URL` is set. The shared tool loop is not used: that loop would act on every choice, and these two gates must be able to stop the run.

The diagram that asks for about 200 items down to 30–50, then 3–5, is the shape of a live desk. The shipped fixture is 50 labelled items and keeps 32. It does not measure the diagram's 15 seconds or ¥0.01.

## Where each stage lives

| Stage | Code | What it refuses to do |
|---|---|---|
| Ingest RSS | `load_rss_file` / `load_rss_url` in `s1a/content/ingest.py` | A path is a file. A URL is opt-in via `--rss-url`. |
| Ingest X | `load_x_fixture`, `posts_to_items`, `UnconfiguredXAdapter` | No request is sent to `api.x.com`. `X_BEARER_TOKEN` is not read. |
| Screen | `RuleDecider.screen` or `OmniDecider.screen` | Labels and drafts are not copied into the prefill. |
| Human pick | `run_workflow` in `s1a/content/workflow.py` | Missing picks raise `GateRequired` before fact-check. |
| Fact-check | `RuleDecider.fact_check` or `OmniDecider.fact_check` | No draft means `needs_human` and no model call. The worker does not write the research. |
| Human publish | the same `run_workflow` | `human_publish.executed` is always `[]`. `auto_publish` is always false. |

A live X adapter is a `fetch()` callable passed to `load_inputs` as `x_fetch`. It should `GET https://api.x.com/2/tweets/search/recent` with `Authorization: Bearer $X_BEARER_TOKEN`, `query`, and `max_results`, then return posts that `posts_to_items` can map. That call is the operator's code. `UnconfiguredXAdapter.fetch` raises instead.

## Decision shape

Screen choices are `keep` and `drop`. The relevance score uses two levels, "Not relevant" then "Relevant hotspot". Fact-check choices are `supported`, `unsupported`, and `needs_human`. The support score uses three levels, "Unsupported", "Needs a human", then "Supported".

Those are the same kind of enumerated questions as `ticket_router` and `s1a decide`. `s1a decide` still answers one `choice`. It does not grow a `score` question. Score is asked only by `s1a/content/omni_client.py`, one short prefill per question, Public231-style: a string `state`, one entry in `questions`, and no images.

The offline decider is a lexical rule, the same role as the ticket router's keyword baseline. On the shipped fixture it matches the labels by construction. That is a wiring check, not a Jev or Omni quality result. The rule keeps a whole word such as `model` or `agent`. It also keeps "model trains", and it drops "supermodel", "chipotle", and "agency". The fixture simply does not include the "model trains" trap.

## Draft workers

Nothing in pull requests 164–166 has to be merged. The client needs a base URL that already speaks `POST /v1/systemone` and `GET /health`.

| Role | Worker | Draft PR | Request flag |
|---|---|---|---|
| Screen, and fact-check by default | StartLux Instant 4B (`StartLux-Decision-v1.1-4B-Instant`) | [system1-omni #164](https://github.com/InferMatrixLab/system1-omni/pull/164) | `think_mode: never`. The `thinking` object is not sent. |
| Optional heavier fact-check | PPLX decider (`perplexity-ai/pplx-decider-v1.1-27b`) | [system1-omni #166](https://github.com/InferMatrixLab/system1-omni/pull/166) | `thinking: {"enabled": false}` on `OMNI_FACTCHECK_BASE_URL`. |
| Not on this path | Clef / Clef-flash | [system1-omni #165](https://github.com/InferMatrixLab/system1-omni/pull/165) | Refused. |

`OMNI_MODEL` overrides the StartLux alias. `OMNI_FACTCHECK_MODEL` overrides the fact-check alias. PPLX is not assumed to share StartLux's port: `--fact-check-worker pplx` without `OMNI_FACTCHECK_BASE_URL` is an error. Clef is an error on either step.

With `OMNI_BASE_URL` unset, `--backend auto` is the lexical rule and opens no decision socket. `--backend omni` without that variable is a usage error.

## Human gates

Pass `--picks` (`{"ids": [...]}`) and `--publish` (`{"decisions": {"id": "publish"|"hold"}}`). On a terminal, omitting them prompts. Otherwise the process exits 2 and writes a packet only when `--packet` is set. The packet's `stopped_at` is `human_pick` or `human_publish`. It is not a finished run.

`--allow-unsafe-skip-gates` is unsafe. It invents a pick (the first three kept ids, or every kept id when fewer than three were kept) and holds every picked item. It does not publish. An explicit file or hook wins over the flag: the flag does not replace a decision you already passed.

A decision of `publish` is still only a record. There is no publisher, no CMS write, and no MCP tool that posts. Hosts that speak MCP should collect the human files and run this command. `run_agent` is the wrong tool: it finishes a loop, and this desk has to stop for a person.

Python hooks, for a caller that already has the person in the loop:

```python
from s1a.content.rules import RuleDecider
from s1a.content.workflow import run_workflow

run_workflow(items, decider=RuleDecider(), pick_hook=ask_for_ids, publish_hook=ask_publish_or_hold)
```

`pick_hook` sees kept items without labels. `publish_hook` sees the fact-check choice and score name. Both results are checked: 3 to 5 kept ids when at least three were kept, otherwise every kept id; publish decisions must name each picked id as `publish` or `hold`.

## Latency gate, not attached

Every report carries `latency_gate.attached: false`. Per-request client wall times are stored on Omni calls (`calls[].latency_ms`). Rule rows are omitted from the gate so a 0 ms baseline cannot clear a ceiling.

Hang a Public231-C1-style ceiling later, after a worker exists:

1. Serve the draft worker. Confirm `GET $OMNI_BASE_URL/health`.
2. Discard a warmup of 5 requests. The workflow itself does not drop them; run a throwaway command first, or strip those calls from the report.
3. Keep concurrency at 1. This client already sends one POST at a time.
4. Repeat three measured rounds on the same fixture.
5. Pool the successful request latencies and run:

```bash
python -m s1a.content latency-report \
--report runs/recipes/content-workflow/report.json \
--p50-max-ms <ceiling> \
--p95-max-ms <ceiling> \
--min-samples <measured-requests>
```

No ceiling is pinned. A p50 from Public231 on another checkpoint is not this fixture's budget. The clock here is `client_request_wall_ms` (POST start through the response body). System1-Omni's bench clock is `post_to_body_json_and_schema_validation`. Do not treat them as one number. With a ceiling set and too few Omni samples, the command fails closed and exits 1. With no ceiling, it prints the summary and exits 0.

The runnable fixture path is [the recipe](../recipes/content-workflow/README.md).
3 changes: 3 additions & 0 deletions docs/skills.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,9 @@ runs `s1a run flights --model jev --goal "..."` and reads the answer from `final
- Keep arithmetic, constraint deduction, search over move trees, free-text generation and any value the page never
shows with the chat model. The three probe misses, Minesweeper, Wordle and a severity rubric, needed deduction. A
plain page fetch needs no agent; use `curl`.
- A news desk is `python -m s1a.content run`, not a chat draft and not `run_agent`. Jev or a draft Omni worker
screens and fact-checks. A person picks 3 to 5 items and decides publish or hold. The command does not post.
See [content-workflow.md](content-workflow.md).

### What comes back

Expand Down
1 change: 1 addition & 0 deletions recipes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ backend, and the required application + System1-Agents + System1-Omni video for
| Use case | Agent / front | Execution profile | Validation |
|---|---|---|---|
| [Route support tickets](ticket-routing/README.md) | `ticket_router` / tool | Local CPU rule baseline; Laya/Jev alternatives, System1-Omni served Laya, or CLM over HTTP | Baseline smoke run; this fixture's required served-model video remains pending, and CLM measured 18/90 under this loop's framing ([evidence](../evals/ticket_router/CLM.md)) |
| [Screen a news desk](content-workflow/README.md) | `s1a.content` orchestration, not a tool-loop agent | Local CPU lexical rule; optional draft StartLux/PPLX `/v1/systemone` workers | Fixture smoke run with human pick and publish files; nothing is posted. Served-worker video remains pending |
| [Play Snake](snake/README.md) | snake game client (`evals/snake`) / tool-style loop | System1-Omni native worker (H800); MPS dev path | 3×2400-step (0 deaths) and 16×600-step multigrid (0 deaths) recorded runs, playback-1 GIFs |

The recipe's validation section identifies what was actually run. A supported model or a passing unit test
Expand Down
Loading
Loading