Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .copier-answers.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
_commit: main
_src_path: https://github.com/ONEcampaign/bblocks-projects.git
author_email: jorge.rivera@one.org
author_email: data@one.org
author_name: ONE Campaign
include_notebooks: false
include_pytest: true
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ Only published pages are valid link targets. The link check fails for a page abs

A source repository keeps its pages in a docs folder and declares them in a nav sidecar. The build looks for the sidecar in three places, in this order: `docs/mkdocs.yml`, a root `mkdocs.yml`, and a root `zensical.toml`. The docs root is the sidecar's own `docs_dir`, resolved from the folder the sidecar sits in.

Use a root `mkdocs.yml` when user-facing pages sit in a subfolder separate from a repository's internal docs. dfo-apis publishes `docs/guide/` this way:
Use a root `mkdocs.yml` when user-facing pages sit in a subfolder separate from a repository's internal docs. `apis` publishes `docs/guide/` this way:

```yaml
site_name: API
Expand Down Expand Up @@ -129,7 +129,7 @@ Then register the repository in `sources.yaml`:

In `sources.yaml`, `docs_dir` sets which folder the build checks out. Match it to the sidecar's `docs_dir`. Set `path` to `extra.dfo.path`. The build fails if they differ. Add a `mount` entry in `hub.yaml` that names the source row. To register a repository before publication, set `enabled: false` with a `reason` and leave it out of `hub.yaml`. A mount for a disabled source fails the build. Every registered repository must be public. The README covers `exclude`, `kind: external`, `kind: skill`, and each build failure.

Before opening a pull request in the source repository, check its nav against its pages from a dfo-docs checkout. Then check links from the source repository's docs folder:
Before opening a pull request in the source repository, check its nav against its pages from a `docs` checkout. Then check links from the source repository's docs folder:

```
uv run python scripts/check_nav.py /path/to/source-repo
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ To write a page or publish a repository's docs, read [CONTRIBUTING.md](CONTRIBUT

Add a row to `sources.yaml` with `name`, `repo`, `ref`, `docs_dir`, and the `path` the documentation should publish at. A row with `kind: external` points at a package that publishes its own documentation elsewhere, and carries a `url` instead of a `docs_dir`.

A row with `kind: skill` publishes a repository of agent skills, such as dfo-skills. It needs no sidecar. The build walks `docs_dir` and publishes each directory that holds a `SKILL.md` as a skill page at `<path>/<directory>`, with each file in its `references/` as a page at `<path>/<directory>/references/<name>`. Directories whose names start with `_` stay out. The row must set `docs_dir`, `path`, and `title`, and it may not set `exclude`. It may set `internal_dir`, a second directory the build walks the same way. Skills under `docs_dir` are offered for install. Skills under `internal_dir` are published at the same `<path>/<directory>` and are not offered. A directory name may appear under only one of the two. Until the repository has the `internal_dir` directory, the build warns and reads `docs_dir` alone. Its `hub.yaml` mount becomes one group titled by `title`. A `categories.yaml` in `docs_dir` lists skills under `sources` or under a domain in `domains`, and each category becomes a group inside the mount, titled in sentence case: Sources, and Domains with a group such as Sovereign debt inside it. Categories and the skills in them follow the file's order. Skills the file leaves out come first, in directory order, and without the file that is every skill. Each skill page opens with a header the build writes from the front matter: the skill's name and the first sentence of its `description`. For a skill under `docs_dir`, whatever its category, the build writes a zip to `corpus/<name>/<directory>.zip`, holding `<directory>/SKILL.md` and `<directory>/references/*.md` as committed, records the zip on the page's manifest entry as `bundle_path`, and adds a line to the header that links to the hub page on using skills in your own agent and to the zip. A skill under `internal_dir` gets no zip, no `bundle_path`, and no such line.
A row with `kind: skill` publishes a repository of agent skills, such as `skills`. It needs no sidecar. The build walks `docs_dir` and publishes each directory that holds a `SKILL.md` as a skill page at `<path>/<directory>`, with each file in its `references/` as a page at `<path>/<directory>/references/<name>`. Directories whose names start with `_` stay out. The row must set `docs_dir`, `path`, and `title`, and it may not set `exclude`. It may set `internal_dir`, a second directory the build walks the same way. Skills under `docs_dir` are offered for install. Skills under `internal_dir` are published at the same `<path>/<directory>` and are not offered. A directory name may appear under only one of the two. Until the repository has the `internal_dir` directory, the build warns and reads `docs_dir` alone. Its `hub.yaml` mount becomes one group titled by `title`. A `categories.yaml` in `docs_dir` lists skills under `sources` or under a domain in `domains`, and each category becomes a group inside the mount, titled in sentence case: Sources, and Domains with a group such as Sovereign debt inside it. Categories and the skills in them follow the file's order. Skills the file leaves out come first, in directory order, and without the file that is every skill. Each skill page opens with a header the build writes from the front matter: the skill's name and the first sentence of its `description`. For a skill under `docs_dir`, whatever its category, the build writes a zip to `corpus/<name>/<directory>.zip`, holding `<directory>/SKILL.md` and `<directory>/references/*.md` as committed, records the zip on the page's manifest entry as `bundle_path`, and adds a line to the header that links to the hub page on using skills in your own agent and to the zip. A skill under `internal_dir` gets no zip, no `bundle_path`, and no such line.

If the repository needs to sit under a grouping heading, or nest alongside another package, add it to `hub.yaml`'s nav as a `mount` (or `external`) entry. A grouping heading such as `bblocks` or `ODA` lives only in `hub.yaml`, since no source repository declares it.

Expand Down
2 changes: 1 addition & 1 deletion content/hub/get-started/use-studio.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ This example asks for an overview of health spending in Kenya and produces a pin

The walkthrough takes about ten minutes and uses two of the five queries available without an account. Sign in first if you want to save your canvas or ask more than five queries.

<!-- TODO: verify the chart title, plan and answer against the staging site; they come from the recorded Studio session dfo-ai docs/transcripts/studio_transcript_plan.jsonl. -->
<!-- TODO: verify the chart title, plan and answer against the staging site; they come from the recorded Studio session in the `ai` repository, docs/transcripts/studio_transcript_plan.jsonl. -->

## Step 1: Ask a question

Expand Down
10 changes: 5 additions & 5 deletions content/hub/skills/use-in-your-agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ These steps cover Claude on the web, the Claude desktop app, and Cowork.

To install one skill on its own, download its zip, then open Customize, then Skills, click +, and choose "Upload a skill". Uploaded skills need code execution switched on in Claude's settings.

<!-- TODO: verify by uploading oda-measures.zip in claude.ai: the path Customize > Skills > + > "Upload a skill", that Claude accepts the zip dfo-docs builds (one top folder `oda-measures/` holding `SKILL.md` and `references/`), and the name and location of the code execution setting. From Anthropic's help article 12512180, not run. -->
<!-- TODO: verify by uploading oda-measures.zip in claude.ai: the path Customize > Skills > + > "Upload a skill", that Claude accepts the zip the `docs` repository builds (one top folder `oda-measures/` holding `SKILL.md` and `references/`), and the name and location of the code execution setting. From Anthropic's help article 12512180, not run. -->

::tab{label="ChatGPT"}

Expand All @@ -56,7 +56,7 @@ Whether you can upload a skill depends on your ChatGPT plan, and in a ChatGPT wo
1. Wait for ChatGPT to finish scanning the skill.
1. Start a new chat.

<!-- TODO: confirm these steps with one real upload of oda-measures.zip in ChatGPT. They come from search snippets of OpenAI's help article 20001066 and from a review citing it. The article returned 403 to every fetch, so no primary source has been read. Confirm the labels (Plugins, Skills, Create, "Upload from your computer"), what the scan shows while it runs (the snippets report "Needs Review" and "Blocked" states), that the dfo-docs zip layout (one top folder `oda-measures/`) passes the scan, and whether the skill loads only in a new chat. -->
<!-- TODO: confirm these steps with one real upload of oda-measures.zip in ChatGPT. They come from search snippets of OpenAI's help article 20001066 and from a review citing it. The article returned 403 to every fetch, so no primary source has been read. Confirm the labels (Plugins, Skills, Create, "Upload from your computer"), what the scan shows while it runs (the snippets report "Needs Review" and "Blocked" states), that the zip layout from `docs` (one top folder `oda-measures/`) passes the scan, and whether the skill loads only in a new chat. -->

<!-- TODO: confirm which ChatGPT plans can upload a skill, then name them in the sentence above the steps. Business, Enterprise, Healthcare and Edu are reported, with uploads off by default on Enterprise and Edu. Free, Go, Plus and Pro are unconfirmed. Confirm also that a workspace admin has to enable skill uploads, which the same snippets report. -->

Expand All @@ -75,7 +75,7 @@ Gemini takes skills on a personal Google Account, for people aged 18 or over, wi

<!-- TODO: confirm in the Gemini app (gemini.google.com) that the Skills page is under Settings. Google's help page 17094296 gives the steps from the Skills page onwards (Upload, select the .zip, Open, review, Create) and the account requirements, and does not say how to reach the page. Confirm also whether the skill loads only in a new chat. -->

<!-- TODO: upload a zip built by dfo-docs to Gemini, which nobody has done. Google's help page 17094296 says `SKILL.md` must be in the zip's root (main) folder, and the dfo-docs zip holds it one level down, in `oda-measures/`. If Gemini rejects that layout, change the zip builder in dfo-docs (`src/dfo_docs/skills.py`) and leave this page as it is. -->
<!-- TODO: upload a zip built by `docs` to Gemini, which nobody has done. Google's help page 17094296 says `SKILL.md` must be in the zip's root (main) folder, and the `docs` zip holds it one level down, in `oda-measures/`. If Gemini rejects that layout, change the zip builder in `docs` (`src/dfo_docs/skills.py`) and leave this page as it is. -->

::tab{label="Claude Code"}

Expand Down Expand Up @@ -114,7 +114,7 @@ An agent that reads the Agent Skills format loads a skill from a folder on your

1. Start a new session.

<!-- TODO: verify `~/.agents/skills/` against the documentation of each agent named (Cursor, GitHub Copilot, Gemini CLI) and load oda-measures from it in at least one of them. The folder comes from vendor documentation, not from a test. Run the unzip command on a zip built by dfo-docs and confirm it leaves `~/.agents/skills/oda-measures/SKILL.md`. -->
<!-- TODO: verify `~/.agents/skills/` against the documentation of each agent named (Cursor, GitHub Copilot, Gemini CLI) and load oda-measures from it in at least one of them. The folder comes from vendor documentation, not from a test. Run the unzip command on a zip built by `docs` and confirm it leaves `~/.agents/skills/oda-measures/SKILL.md`. -->

Check your agent's documentation if it reads a different folder. Keep the unzipped folder whole, because `SKILL.md` points to the files in `references/` by relative path.

Expand Down Expand Up @@ -162,7 +162,7 @@ Download the zip again from the skill's page. In Claude, ChatGPT, or Gemini, del

:::

<!-- TODO: add a "Programmatically" route. dfo-apis includes a Skills API that lists skills and serves each skill's body and reference files, with no public deployment yet. Document the Skills API (GET /api/skills, GET /api/skills/{skill_id}, GET /api/skills/{skill_id}/resources/{path}) and link the API pages once a public deployment serves it with DFO_SKILLS_SURFACE on. -->
<!-- TODO: add a "Programmatically" route. `apis` includes a Skills API that lists skills and serves each skill's body and reference files, with no public deployment yet. Document the Skills API (GET /api/skills, GET /api/skills/{skill_id}, GET /api/skills/{skill_id}/resources/{path}) and link the API pages once a public deployment serves it with DFO_SKILLS_SURFACE on. -->

## Next

Expand Down
10 changes: 5 additions & 5 deletions content/hub/skills/what-a-skill-is.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ oda-measures/

`SKILL.md` holds YAML front matter with three fields. `name` matches the directory name. `description` says what the skill covers and when to load it, and it's the text an agent matches a question against. `license` names the licence, `CC-BY-4.0` for every DFO skill. Below the front matter, the body lists the reference files and says when to read each one.

`references/` holds the long material: definitions, method notes, defaults. The dfo-skills repository adds two rules of its own, checked in CI by `scripts/validate_skills.py`:
`references/` holds the long material: definitions, method notes, defaults. The `skills` repository adds two rules of its own, checked in CI by `scripts/validate_skills.py`:

- Pointers stay one level deep. `SKILL.md` names every reference file, and no reference file sends the agent on to another.
- Keep each paragraph and list item on one line.
Expand All @@ -59,19 +59,19 @@ Some skills point to others. [`oda-dataset-selection`](../../dfo-skills/oda-data

## Where skills live in the DFO

The dfo-skills repository holds each skill in its own directory, and each `description` says what its skill covers. `skills/` holds the skills offered for install. `internal/skills/` holds the skills the DFO's own assistant uses and that are not offered for install, such as `dfo-faq`. Each skill also has a page in this section, which shows its `SKILL.md` with its reference files as pages beneath it. Sources lists the skills that each explain one dataset, such as [`oda-dac1`](../../dfo-skills/oda-dac1/SKILL.md). Domains lists the skills that explain a subject across datasets, grouped by domain, such as Sovereign debt.
The `skills` repository holds each skill in its own directory, and each `description` says what its skill covers. `skills/` holds the skills offered for install. `internal/skills/` holds the skills the DFO's own assistant uses and that are not offered for install, such as `dfo-faq`. Each skill also has a page in this section, which shows its `SKILL.md` with its reference files as pages beneath it. Sources lists the skills that each explain one dataset, such as [`oda-dac1`](../../dfo-skills/oda-dac1/SKILL.md). Domains lists the skills that explain a subject across datasets, grouped by domain, such as Sovereign debt.

dfo-pipelines checks dfo-skills for new commits and publishes each skill to the [knowledge graph](../data/knowledge-graph.md) as an `AgentSkill` node. Each reference file is an `AgentSkillReference` node.
The `pipelines` repository checks the `skills` repository for new commits and publishes each skill to the [knowledge graph](../data/knowledge-graph.md) as an `AgentSkill` node. Each reference file is an `AgentSkillReference` node.

Install the skills under `skills/` in your own assistant to use them.

## How the DFO assistant uses skills

The DFO assistant is an agent too, and it reads the same skills from the knowledge graph. dfo-apis serves them through its Skills API, and the DFO MCP server gives the assistant two tools over it: `get_skill` loads a skill's `SKILL.md` and lists its reference files, and `get_skill_file` reads one of those files.
The DFO assistant is an agent too, and it reads the same skills from the knowledge graph. The `apis` repository serves them through its Skills API, and the DFO MCP server gives the assistant two tools over it: `get_skill` loads a skill's `SKILL.md` and lists its reference files, and `get_skill_file` reads one of those files.

The descriptions the assistant matches against come from its data search. Skills are attached to datasets, and the search result names up to five skills attached to the datasets it found. A question about DFO itself goes straight to [`dfo-faq`](../../dfo-skills/dfo-faq/SKILL.md).

An edit to a skill reaches the assistant after dfo-pipelines publishes it to the graph and the Skills API refreshes its index, which it does every 15 minutes.
An edit to a skill reaches the assistant after `pipelines` publishes it to the graph and the Skills API refreshes its index, which it does every 15 minutes.

## Next

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ version = "0.1.0"
description = "Fetches, normalises, and publishes documentation aggregated from DFO source repositories."

authors = [
{ name = "ONE Campaign", email = "jorge.rivera@one.org" }
{ name = "ONE Campaign", email = "data@one.org" },
]

readme = "README.md"
Expand Down
7 changes: 4 additions & 3 deletions src/dfo_docs/docsync/edits.py
Original file line number Diff line number Diff line change
Expand Up @@ -1884,9 +1884,10 @@ def _same_list_kind(block: Block, other: Block) -> bool:
def heading_anchor(heading: str) -> str:
"""The anchor the docs site gives a heading with Markdown text `heading`.

The site (dfo-website `docsSlug`) takes the heading's rendered text, code spans
included, lowercases and trims it, drops every character but ASCII letters,
digits, `_`, `-` and whitespace, and turns each run of whitespace into `-`.
The site (`docsSlug` in the website repository) takes the heading's rendered
text, code spans included, lowercases and trims it, drops every character but
ASCII letters, digits, `_`, `-` and whitespace, and turns each run of whitespace
into `-`.
"""
text = "".join(_text_content(_MARKDOWN.parseInline(heading)[0].children or []))
dropped = _SLUG_DROP_RE.sub("", text.lower().strip())
Expand Down
2 changes: 1 addition & 1 deletion tests/docsync/fixtures/edits/sampler.edited.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ and its second line.
1. Clone the skills repo.

```sh
git clone --depth 1 https://github.com/ONEcampaign/dfo-skills.git
git clone --depth 1 https://github.com/DevelopmentFinanceObservatory/skills.git
```

2. Copy the skills into a skills folder.
Expand Down
2 changes: 1 addition & 1 deletion tests/docsync/fixtures/edits/sampler.expected.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ An unindented callout body.
1. Clone the skills repo.

```bash
git clone --depth 1 https://github.com/ONEcampaign/dfo-skills.git
git clone --depth 1 https://github.com/DevelopmentFinanceObservatory/skills.git
```

1. Copy the skills into a skills folder.
Expand Down
2 changes: 1 addition & 1 deletion tests/docsync/fixtures/pages/conventions-live.baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ A proxy in front of every route enforces three limits before the request reaches
| `413` | Request body too large. |
| `415` | The media type is unsupported. Send `Content-Type: application/json`. |

**Author note:** TODO: rate-limit thresholds; set in proxy config, not recorded in dfo-apis.
**Author note:** TODO: rate-limit thresholds; set in proxy config, not recorded in the apis repository.

**Author note:** TODO: byte ceiling; set in proxy config.

Expand Down
2 changes: 1 addition & 1 deletion tests/docsync/fixtures/pages/conventions-live.edited.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ A proxy in front of every route enforces three limits before the request reaches
| `413` | Request body too large. |
| `415` | The media type is unsupported. Send `Content-Type: application/json`. |

**Author note:** TODO: rate-limit thresholds; set in proxy config, not recorded in dfo-apis.
**Author note:** TODO: rate-limit thresholds; set in proxy config, not recorded in the apis repository.

**Author note:** TODO: byte ceiling; set in proxy config.

Expand Down
Loading
Loading