diff --git a/RELEASE.md b/RELEASE.md index 5d15f88fb9..276a2e51d5 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -110,9 +110,9 @@ Once these are verified working, you can move on to publishing the release. ### Publishing to PyPI -The final step is to publish the release to PyPI. **You will need special permissions from Plotly leadership to do this.**. +The final step is to publish the release to PyPI. **You will need special permissions from Plotly leadership to do this.** -You must install first install [Twine](https://pypi.org/project/twine/) (`pip install twine`) if not already installed. +You must first install [Twine](https://pypi.org/project/twine/) (`pip install twine`) if not already installed. Publishing to PyPI: ```bash @@ -132,10 +132,21 @@ Your account must have permissions to publish to the `plotly` project on PyPI. start by doing it first if not. Then merge `main` into `doc-prod` to deploy the doc related to features in the release. 3. in a clone of the [`graphing-library-docs` repo](https://github.com/plotly/graphing-library-docs): - 1. bump the version of plotly.py in `_data/pyversion.json` + 1. bump the version of plotly.py in `_data/pyversion.json` 2. bump the version of plotly.js with `cd _data && python get_plotschema.py ` fixing any errors that come up. - - If plotly.js contains any new traces or trace or layout attributes, you'll get a warning `“missing key in attributes: `. To resolve, add the attribute to the relevant section in `/_data/orderings.json` in the position you want it to appear in the reference docs. - 3. rebuild the Algolia `schema` index with `ALGOLIA_API_KEY= make update_ref_search` + + **About `_data/orderings.json`:** `get_plotschema.py` downloads the raw `plot-schema.json` from the specified plotly.js release and uses `_data/orderings.json` (which lives in `graphing-library-docs`, not plotly.js) to determine the order in which traces, trace attributes, and layout attributes appear in the [reference documentation](https://plotly.com/python/reference/). The file has three sections: + - `layout` — top-level layout attributes (e.g. `hovermode`, `clickmode`, `xaxis`) + - `traces` — order of trace types (e.g. `scatter`, `bar`, `pie`) + - `trace_attr_order` — order of attributes shared across traces + + If plotly.js adds new traces or trace/layout attributes that aren't listed in `orderings.json`, `get_plotschema.py` prints a warning like `missing key in attributes: ` and appends the missing entry to the end of its section. To resolve: + + - Add each missing attribute to the appropriate section of `/_data/orderings.json` in the position you want it to appear in the reference docs. + - When in doubt about trace-attribute placement, match plotly.js's native order: open the regenerated `_data/plotschema.json` and find where plotly.js itself places the attribute (e.g. inspect a representative trace's `attributes` keys in order). Following the native order keeps related attributes grouped (for example, `texttemplate` → `texttemplatefallback` → `texttemplatesrc`). + - For new top-level layout attributes, group them with semantically related entries rather than appending to the end (for example, a new click-behavior attribute like `clickanywhere` belongs next to `clickmode`). + - Re-run `python get_plotschema.py ` after editing `orderings.json` and confirm the warnings are gone. + 3. Rebuild the Algolia `schema` index with `ALGOLIA_API_KEY= make update_ref_search` 4. Rebuild the Algolia `python` index with `ALGOLIA_API_KEY= make update_python_search` 5. Commit and push the changes to `master` in that repo @@ -158,5 +169,5 @@ PyPI RC (no special flags, just the `rc1` suffix): (plotly_dev) $ twine upload dist/plotly-X.Y.Zrc1* ``` -The `--tag next` part ensures that users won't install this version unless -they explicitly ask for the version or for the version with the `next` tag. +The `rc1` suffix ensures that users won't install this version by default — +they must explicitly request it (e.g., `pip install plotly==X.Y.Zrc1`). diff --git a/doc/README.md b/doc/README.md index cb66bb726f..4ccfaf4f65 100644 --- a/doc/README.md +++ b/doc/README.md @@ -5,23 +5,54 @@ The `doc` directory contains the source files of the documentation of plotly.py. It is composed of two parts: -- inside the [`python/` directory](python), tutorials corresponding to https://plot.ly/python/ +- inside the [`python/` directory](python), tutorials corresponding to https://plotly.com/python/ - inside the [`apidoc/` directory](apidoc), configuration files for generating - the API reference documentation (hosted on https://plot.ly/python-api-reference/) + the API reference documentation (hosted on https://plotly.com/python-api-reference/) -Python packages required to build the doc are listed in +Python packages required to build the docs are listed in [`requirements.txt`](requirements.txt) in the `doc` directory. +## Local environment setup + +Before building the documentation locally, you need to set up a dedicated +environment with the doc-specific dependencies. + +```bash +cd doc +uv venv --python 3.9 +source .venv/bin/activate +uv pip install -r requirements.txt +``` + +If you are documenting a feature that has not yet been released, you also need +an editable install of plotly so your local changes are reflected: + +```bash +uv pip uninstall plotly # remove the PyPI version installed by requirements.txt +uv pip install -e .. # install from your local checkout +``` + +### Mapbox token + +Several geographic examples require a free Mapbox public token. Without it, +those specific pages will fail to build. + +1. Create an account at https://account.mapbox.com/auth/signup +2. Navigate to https://account.mapbox.com/ and copy your "Default public token" +3. Save it to the file `doc/python/.mapbox_token` + +The Makefile symlinks this token into the build directory automatically. + ## Tutorials (`python` directory) -Each tutorial is a markdown (`.md`) file, which can be opened in the Jupyter -notebook or in Jupyterlab by installing [jupytext](https://jupytext.readthedocs.io/en/latest/install.html). +Each tutorial is a markdown (`.md`) file, which can be opened in Jupyter +Notebook or in JupyterLab by installing [jupytext](https://jupytext.readthedocs.io/en/latest/install.html). For small edits (e.g., correcting typos) to an existing tutorial, you can simply click on the "edit this -page on Github" link at the top right of the page (e.g. clicking on this link -on https://plot.ly/python/bar-charts/ will take you to +page on GitHub" link at the top right of the page (e.g. clicking on this link +on https://plotly.com/python/bar-charts/ will take you to https://github.com/plotly/plotly.py/edit/doc-prod/doc/python/bar-charts.md, -where you can edit the page on Github). +where you can edit the page on GitHub). For more important edits where you need to run the notebook to check the output, clone the repository and setup an environment as described in the [main @@ -31,19 +62,126 @@ install (`pip install -e`, as described in [main contributing notes](../CONTRIBUTING.md)), so that you only need to restart the Jupyter kernel when you have changed the source code of the feature. -### Branches +### Branching strategy + +- The `doc-prod` branch contains the live docs which are available on the website. As soon as a change is merged into `doc-prod`, the updated docs are deployed and made publicly available. +- The `main` branch contains docs which have been written but are not ready to be released, such as for an upcoming feature in the next plotly.py release. + +When updating the docs, two workflows are possible: + +- In the case of updating the docs for an already-released feature: your changes can be deployed to the documentation website as soon as they have been merged, so you should branch off of the `doc-prod` branch and open your pull request into the `doc-prod` branch. +- In the case of writing docs for a new (not-yet-released) feature: you should branch off `main` and open your pull request into `main`, so that the documentation of the feature is only deployed when it is available in a released version of `plotly.py`. The `main` branch will be merged into `doc-prod` at release time, as described below. + +#### Keeping `main` and `doc-prod` in sync + +Changes to `doc-prod` are **not** automatically merged back into `main`. To +prevent the branches from diverging, `doc-prod` should be merged into `main` on +a regular basis via a pull request (e.g., a branch named +`merge-doc-prod-to-main-branch` merged into `main`). + +At release time the synchronization is bidirectional (see also +[`RELEASE.md`](../RELEASE.md)): + +1. **`doc-prod` → `main`** — merge any outstanding doc-only fixes into `main` + (if not already done recently). +2. **`main` → `doc-prod`** — merge `main` into `doc-prod` so that documentation + for newly released features is deployed to the live site. +3. **Publish the site** — update the + [`graphing-library-docs`](https://github.com/plotly/graphing-library-docs) + repo to bump the plotly.py and plotly.js versions and rebuild the Algolia + search indexes. See the + [Update documentation site](../RELEASE.md#update-documentation-site) + section of `RELEASE.md` for the full procedure. + +> **Release prep:** When synchronizing `main` into `doc-prod` for a new +> release, update the `plotly==` version pin in `doc/requirements.txt` to match +> the newly released version. The `doc-prod` build uses this pinned version +> (not an editable install), so examples that rely on new features will fail +> if the pin is stale. -Two different cases exist, whether you are documenting a feature already -released, or which has just been included but not yet released. +### Tutorial file format -- Case of an already released feature: your changes can be deployed to the - documentation website as soon as they have been merged, and you should start - your branch off the `doc-prod` branch and open your pull request against this - `doc-prod` branch. -- Case of a new (not released yet) feature: start your branch / pull request - against the `main` branch. `main` and `doc-prod` will be synchronized at - release time, so that the documentation of the feature is only deployed when - it is available in a released version of `plotly.py`. +Tutorial files are Markdown files with a YAML frontmatter block that contains +Jupyter notebook metadata (used by jupytext) and plotly-specific metadata (used +by the documentation site for navigation, SEO, and categorization). + +Here is an annotated example of the frontmatter: + +```yaml +--- +jupyter: + jupytext: + notebook_metadata_filter: all + text_representation: + extension: .md + format_name: markdown + format_version: '1.3' + jupytext_version: 1.17.3 + kernelspec: + display_name: Python 3 (ipykernel) + language: python + name: python3 + language_info: + codemirror_mode: + name: ipython + version: 3 + file_extension: .py + mimetype: text/x-python + name: python + nbconvert_exporter: python + pygments_lexer: ipython3 + version: 3.9.0 + plotly: + description: Short description for SEO and page previews. + display_as: basic # Category: basic, statistical, scientific, maps, 3d, etc. + language: python + layout: base + name: Page Title # Displayed in the navigation sidebar + order: 3 # Position within the display_as category + page_type: example_index + permalink: python/my-page/ # URL path on the documentation site + thumbnail: thumbnail/my-page.jpg +--- +``` + +The `plotly` metadata fields are the most important to enter correctly when creating/updating a tutorial: + +| Field | Description | +|-------|-------------| +| `name` | Page title shown in the sidebar and browser tab | +| `permalink` | URL slug — must match the filename (e.g., `bar-charts.md` → `python/bar-charts/`) | +| `description` | Short description used by search engines | +| `display_as` | Category grouping (e.g., `basic`, `statistical`, `scientific`, `maps`, `3d_charts`, `file_settings`) | +| `order` | Numeric sort order within the category | +| `page_type` | Typically `example_index` for tutorial pages | +| `thumbnail` | Path to the thumbnail image | + +Code cells are written as fenced code blocks with the `python` language tag. +Each code cell is separated by a blank line and starts with ` ```python `. +Markdown cells are written as regular Markdown text between code blocks. + +### Creating a new tutorial page + +1. **Copy an existing tutorial** from `doc/python/` as a starting point to get + the frontmatter structure right. +2. **Update the frontmatter** — at minimum, change `name`, `permalink`, + `description`, `display_as`, and `order`. Make sure `permalink` matches the + filename (e.g., `my-feature.md` → `python/my-feature/`). +3. **Write examples** using fenced `python` code blocks. Each block becomes a + separate Jupyter cell when the file is converted. +4. **Test in Jupyter** — open the file directly in JupyterLab (with jupytext + installed) and run all cells to verify the examples work: + ```bash + jupyter lab doc/python/my-feature.md + ``` +5. **Build the single page** (optional) — you can build just your page instead + of the entire doc set: + ```bash + cd doc + make build/html/2019-07-03-my-feature.html + ``` +6. **Check that CI passes** — push your branch and open a pull request. The CI + will build all pages and run validation on frontmatter and page ordering. ### Guidelines @@ -65,17 +203,60 @@ Checklist ### Build process -This build process requries a free personal public mapbox token to work. +#### Building all tutorials + +From the `doc` directory, with the virtual environment activated: + +```bash +cd doc +source .venv/bin/activate +make +``` + +This runs through every `.md` file in `python/` and: + +1. Appends the "What About Dash?" footer from `what_about_dash.md` +2. Converts each Markdown file to a Jupyter notebook using **jupytext** +3. Executes the notebook and converts it to HTML using **nbconvert** (with a + 10-minute timeout per notebook) +4. Outputs HTML files to `build/html/` with a `2019-07-03-` date prefix +5. Generates redirect pages for v3 backward compatibility and "next version" + previews + +To build in parallel (as CI does): + +```bash +make -kj8 +``` + +The `-k` flag continues past failures and `-j8` runs 8 jobs in parallel. + +#### Building a single tutorial + +To build only one page (useful during development): + +```bash +make build/html/2019-07-03-bar-charts.html +``` -First, create an account at https://account.mapbox.com/auth/signup Once that is done, navigate to https://account.mapbox.com/ and copy your "Default public token" to the file `doc/python/.mapbox_token` +The filename follows the pattern `2019-07-03-.html`. -Next, run `make` to build html pages for the tutorials. This uses `jupytext` to -execute the notebooks and `nbconvert` to convert notebook files to static html -pages. Note that the CI will build the doc, so you don't have to build it -yourself, it is enough to check that the markdown file runs correctly in -Jupyter. +> **Why the `2019-07-03-` prefix?** The downstream `graphing-library-docs` +> site uses Jekyll, whose `_posts/` collection only processes files matching +> the pattern `YYYY-MM-DD-title.ext` and silently ignores anything else. +> The specific date is an arbitrary placeholder — its value is never +> displayed; it just satisfies Jekyll's filename parser. -The output of the `Makefile` is stored by CI in the `built` branch of the `plotly.py-docs` repo which is then used by the `documentation` repo to generate https://plot.ly/python. +#### Build output + +| Directory | Contents | +|-----------|----------| +| `build/ipynb/` | Intermediate Jupyter notebook files | +| `build/html/` | Final HTML tutorial pages | +| `build/html/redir/` | Redirect pages (v3 and next-version) | +| `build/failures/` | Stderr logs for pages that failed to build | + +If a build fails, check `build/failures/` for the error output. ## API reference documentation (`apidoc` directory) @@ -84,16 +265,156 @@ extension](http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html) in order to generate the documentation of the API. Sphinx uses the [reST markup language](https://www.sphinx-doc.org/en/2.0/usage/restructuredtext/basics.html). -Run `make html` inside `apidoc` to build the API doc in the `_build/html` -directory. +### Building the API docs + +The API docs require an editable install of plotly because the build process +temporarily modifies source files (renaming `graph_objects` references to the +internal `graph_objs` name for Sphinx, then reverting afterward). + +```bash +cd doc +source .venv/bin/activate +uv pip uninstall plotly +uv pip install -e .. +cd apidoc +make html +``` + +The output is written to `apidoc/_build/html/`. + +### How the API doc build works + +The `apidoc/Makefile` performs several steps: + +1. Temporarily rewrites `:class:` cross-references in `plotly/graph_objs/` to + use the internal `graph_objs` name so Sphinx can resolve them. +2. Copies color module files into `plotly/colors/` and `plotly/express/colors/` + so their docstrings are picked up. +3. Runs `sphinx-apidoc` to auto-generate `.rst` stubs from the Python source, + excluding `validators/`, `tests/`, `matplotlylib/`, `offline/`, and `api/`. +4. Runs `sphinx-build` to produce HTML from the `.rst` files. +5. Reverts the `graph_objs` source changes with `git checkout`. +6. Cleans up the temporarily copied color files. +7. Renames all `graph_objs` references back to `graph_objects` in the generated + HTML and related files. + +### Adding new API objects -Lists of objects to be documented are found in files corresponding to -submodules, such as [`plotly.express.rst`](plotly.express.rst). When a new -object is added to the exposed API, it needs to be added to the corresponding -file to appear in the API doc. +Lists of objects to be documented are found in `.rst` files corresponding to +submodules: -Other files +| File | Module | +|------|--------| +| `plotly.express.rst` | `plotly.express` (high-level API) | +| `plotly.graph_objects.rst` | `plotly.graph_objects` (traces, layout) | +| `plotly.io.rst` | `plotly.io` (display, read, write) | +| `plotly.subplots.rst` | `plotly.subplots` (subplot helpers) | +| `plotly.figure_factory.rst` | `plotly.figure_factory` | +| `basefigure.rst` | `BaseFigure` class | + +When a new object is added to the exposed API, it needs to be added to the +corresponding `.rst` file to appear in the API doc. + +### Other files - `css` files are found in `_static` - Template files are found in `_templates`. `.rst` templates describe how the autodoc of the different objects should look like. +- `conf.py` contains the Sphinx configuration (theme, extensions, etc.) + +## CI/CD pipeline + +Documentation is built and deployed automatically by the GitHub Actions workflow +defined in `.github/workflows/build-doc.yml`. + +### What triggers a build + +| Event | What happens | +|-------|-------------| +| Pull request (any branch) | Tutorials (markdown files in doc/python) are built and validated. The build artifact is uploaded but not deployed. | +| Push to `doc-prod` | Full build: tutorials are built, validated, and deployed. API docs are also built and deployed. | + +### Build steps + +1. **Environment setup** — Python 3.9, `uv`, and system dependencies (`rename` + utility) are installed. +2. **Install doc dependencies** — `uv pip install -r requirements.txt` inside + `doc/`. +3. **Install editable plotly** (non-`doc-prod` branches only) — Replaces the + PyPI plotly with the local checkout so that in-development features are + available. +4. **Build HTML tutorials** — Runs `make -kj8` (twice, to retry transient + failures). Then downloads and runs validation scripts from + `plotly/graphing-library-docs`: + - `front-matter-ci.py` validates the YAML frontmatter of all built pages. + - `check-or-enforce-order.py` verifies page ordering within categories. +5. **Upload build artifact** — The built HTML is uploaded as a GitHub Actions + artifact named `doc-html` for inspection. + +### Deployment (doc-prod only) + +When changes are pushed to `doc-prod`, the workflow deploys to three branches +of the [`plotly/plotly.py-docs`](https://github.com/plotly/plotly.py-docs) +repository: + +| Target branch | Contents | +|---------------|----------| +| `built` | Final HTML tutorial pages | +| `built_ipynb` | Intermediate Jupyter notebook files | +| `gh-pages` | API reference HTML (built by Sphinx) | + +After deploying, the workflow triggers a downstream build in +[`plotly/graphing-library-docs`](https://github.com/plotly/graphing-library-docs) +by pushing an empty commit. That repository generates the final +https://plotly.com/python site. + +### Summary of the full deployment path + +``` +doc/python/*.md (source) + ↓ make (jupytext + nbconvert) +doc/build/html/*.html (built tutorials) + ↓ CI deploys to plotly/plotly.py-docs@built +plotly/graphing-library-docs (triggered rebuild) + ↓ Jekyll site generation +https://plotly.com/python/ (live site) +``` + +## Troubleshooting + +### A single page fails to build + +Check `build/failures/` for the full error output. Common causes: + +- **Missing import or dataset** — make sure all imports and remote data URLs + are correct. +- **Timeout** — the default is 600 seconds (10 minutes). If your example + legitimately needs more time, discuss in an issue before increasing the + timeout. +- **Missing Mapbox token** — geographic examples will fail if + `doc/python/.mapbox_token` does not exist. + +### `make` fails immediately + +- Confirm you are running from the `doc/` directory with the virtual + environment activated. +- Check that `jupytext` and `nbconvert` are installed: `jupytext --version` + and `jupyter nbconvert --version`. + +### API doc build fails on `graph_objs` references + +The API doc build temporarily modifies files under `plotly/graph_objs/`. If a +previous build was interrupted, those files may be in a dirty state. Reset them +with: + +```bash +git checkout -- plotly/graph_objs +``` + +### CI frontmatter validation fails + +The CI runs `front-matter-ci.py` and `check-or-enforce-order.py` against the +built HTML. Ensure your tutorial's YAML frontmatter includes all required +fields (`name`, `permalink`, `description`, `display_as`, `order`, `layout`, +`language`) and that the `order` value does not conflict with existing pages in +the same `display_as` category.