Skip to content

Commit b0c5b28

Browse files
ArBridgemanArBridgeman
authored
Documentation/676 move and improve workflows (#677)
* Move workflows to be in features * Move section to configuration * Add dependency for mermaid diagrams in sphinx * Update with a description & flowcharts * Add custom css
1 parent 58da5de commit b0c5b28

10 files changed

Lines changed: 221 additions & 23 deletions

File tree

‎doc/_static/css/custom.css‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#merge-diagram svg {
2+
max-height: 225px !important;
3+
}
4+
#release-diagram svg {
5+
max-height: 325px !important;
6+
}

‎doc/_static/github-workflows.png‎

-60.1 KB
Binary file not shown.

‎doc/changes/unreleased.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,7 @@
55
## Feature
66

77
* #673: Switched `checks.yml` to get Python versions for matrix from `BaseConfig`
8+
9+
## Documentation
10+
11+
* #676: Move GitHub Workflows to be inside features & updated

‎doc/conf.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
"sphinx.ext.napoleon",
3434
"sphinx.ext.intersphinx",
3535
"sphinx.ext.autosummary",
36+
"sphinxcontrib.mermaid",
3637
"sphinx_toolbox.collapse",
3738
"sphinx_copybutton",
3839
"myst_parser",
@@ -72,6 +73,8 @@
7273
# relative to this directory. They are copied after the builtin static files,
7374
# so a file named "default.css" will overwrite the builtin "default.css".
7475
html_static_path = ["_static"]
76+
html_css_files = ["css/custom.css"]
77+
7578
html_title = "Toolbox"
7679
html_theme_options = {
7780
"light_logo": "_static/light-exasol-logo.svg",

doc/user_guide/workflows.rst renamed to doc/user_guide/features/github_workflows/configuration.rst

Lines changed: 4 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,14 @@
1-
.. _GitHub Workflows:
2-
3-
GitHub Workflows
4-
================
5-
6-
.. figure:: ../_static/github-workflows.png
7-
:alt: GitHub Workflow Example
8-
9-
The exasol-toolbox ships with various GitHub workflow templates. To leverage the full feature set of the toolbox, you should use them.
10-
11-
.. attention::
12-
13-
Generally, it is advised to install/use all workflows provided by the toolbox as a whole due to their interdependencies.
14-
15-
However, if you know what you are doing and are well-versed in GitHub workflows and actions, you can use just select individual ones or use them as inspiration. Still, an individual approach is likely to be more error-prone.
16-
17-
.. note::
18-
19-
The toolbox command itself, :code:`tbx`, provides various CLI functions to help you maintain those workflows.
20-
For further help, run the command :code:`tbx workflow --help`.
1+
.. _github_workflows_configuration:
212

3+
Configuration
4+
=============
225

236
1. Configure the GitHub project
247
+++++++++++++++++++++++++++++++
258

269
* Make sure your GitHub project has access to a deployment token for PyPi with the following name: **PYPI_TOKEN**. It should be available to the repository either as an Organization-, Repository-, or Environment-secret.
2710

28-
* If your CI workflow involves slow or expensive steps you can guard these to be executed only after manual approval. The CI workflow will automaticall create a GitHub environment named :code:`manual-approval`. You only need to add reviewers in (:code:`Settings/Environments/manual-approval`) and move the steps to be guarded into the related section in job :code:`slow-checks` in file :code:`.github/workflows/merge-gate.yml`.
11+
* If your CI workflow involves slow or expensive steps you can guard these to be executed only after manual approval. The CI workflow will automatically create a GitHub environment named :code:`manual-approval`. You only need to add reviewers in (:code:`Settings/Environments/manual-approval`) and move the steps to be guarded into the related section in job :code:`slow-checks` in file :code:`.github/workflows/merge-gate.yml`.
2912

3013
2. Add all workflows to your project
3114
++++++++++++++++++++++++++++++++++++

‎doc/user_guide/features/github_workflows/index.rst‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
.. _GitHub Workflows:
2+
3+
Enabling GitHub Workflows
4+
=========================
5+
6+
.. toctree::
7+
:maxdepth: 2
8+
:hidden:
9+
10+
configuration
11+
12+
The PTB ships with configurable GitHub workflow templates covering the most common
13+
CI/CD setup variants for Python projects. The templates are defined in:
14+
`exasol/toolbox/templates/github/workflows <https://github.com/exasol/python-toolbox/tree/main/exasol/toolbox/templates/github/workflows>`__.
15+
16+
The PTB provides a command line interface (CLI) for generating and updating actual
17+
workflows from the templates.
18+
19+
.. code-block:: bash
20+
21+
poetry run -- tbx workflow --help
22+
23+
.. attention::
24+
25+
In most cases, we recommend using _all_ workflows without change to ensure
26+
consistent interdependencies. For more details, see :ref:`ci_actions`.
27+
28+
Underlying the CLI, the PTB uses Jinja to dynamically generate project-specific
29+
workflows. The rendering process is supported by the ``github_template_dict`` found in
30+
your ``noxconfig.py::PROJECT_CONFIG``. This dictionary is inherited by default from
31+
``BaseConfig.py``, ensuring a standardized baseline that can be easily overridden, if
32+
necessary.
33+
34+
.. literalinclude:: ../../../../exasol/toolbox/config.py
35+
:language: python
36+
:start-at: github_template_dict
37+
38+
39+
Workflows
40+
---------
41+
42+
.. list-table::
43+
:widths: 25 25 50
44+
:header-rows: 1
45+
46+
* - Filename
47+
- Run on
48+
- Description
49+
* - ``build-and-publish.yml``
50+
- Workflow call
51+
- Packages the distribution and publishes it to PyPi and GitHub.
52+
* - ``cd.yml``
53+
- Push with new tag
54+
- Manages continuous delivery by calling ``check-release-tag.yml``,
55+
``build-and-publish.yml``, and ``gh-pages.yml``. See :ref:`cd_yml`
56+
for a graph of workflow calls.
57+
* - ``check-release-tag.yml``
58+
- Workflow call
59+
- Verifies that the release tag matches the project's internal versioning.
60+
* - ``checks.yml``
61+
- Workflow call
62+
- Executes many small & fast checks: builds documentation and validates
63+
cross-references (AKA. "links") to be valid,runs various linters
64+
(security, type checks, etc.), and unit tests.
65+
* - ``ci.yml``
66+
- Pull request and monthly
67+
- Executes the continuous integration suite by calling ``merge-gate.yml`` and
68+
``report.yml``. See :ref:`ci_yml` for a graph of workflow calls.
69+
* - ``gh-pages.yml``
70+
- Workflow call
71+
- Builds the documentation and deploys it to GitHub Pages.
72+
* - ``matrix-all.yml``
73+
- Workflow call
74+
- Calls Nox session ``matrix:all``, which typically evaluates ``exasol_versions``
75+
and ``python_versions`` from the ``PROJECT_CONFIG``.
76+
* - ``matrix-exasol.yml``
77+
- Workflow call
78+
- Calls Nox session ``matrix:exasol`` to get the ``exasol_versions`` from the
79+
``PROJECT_CONFIG``.
80+
* - ``matrix-python.yml``
81+
- Workflow call
82+
- Calls Nox session ``matrix:python`` to get the ``python_versions`` from the
83+
``PROJECT_CONFIG``.
84+
* - ``merge-gate.yml``
85+
- Workflow call
86+
- Acts as a final status check (gatekeeper) to ensure all required CI steps have
87+
passed before allowing to merge the branch of your pull request to the
88+
default branch of the repository. e.g. ``main``.
89+
* - ``pr-merge.yml``
90+
- Push to main
91+
- Runs ``checks.yml``, ``gh-pages.yml``, and ``report.yml``. See
92+
:ref:`pr_merge_yml` for a graph of called workflows.
93+
* - ``report.yml``
94+
- Workflow call
95+
- Downloads results from code coverage analysis and linting,
96+
creates a summary displayed by GitHub as result of running
97+
the action, and uploads the results to Sonar.
98+
* - ``slow-checks.yml``
99+
- Workflow call
100+
- Runs long-running checks, which typically involve an Exasol database instance.
101+
102+
.. _ci_actions:
103+
104+
CI Actions
105+
----------
106+
107+
.. _ci_yml:
108+
109+
Pull Request
110+
^^^^^^^^^^^^
111+
112+
When any pull request is opened, synchronized, or reopened, then the ``ci.yml`` will be
113+
triggered.
114+
115+
When configured as described on :ref:`github_workflows_configuration`, the
116+
``run-slow-tests`` job requires a developer to manually approve executing the slower
117+
workflows, like ``slow-checks.yml``. This allows developers to update their pull
118+
request more often and to only periodically run the more time-expensive tests.
119+
120+
If one of the jobs in the chain fails (or if ``run-slow-tests`` is not approved),
121+
then the subsequent jobs will not be started.
122+
123+
.. mermaid::
124+
125+
graph TD
126+
%% Workflow Triggers (Solid Lines)
127+
ci[ci.yml] --> merge-gate[merge-gate.yml]
128+
ci --> metrics[report.yml]
129+
130+
merge-gate --> fast-checks[checks.yml]
131+
merge-gate --> run-slow-tests[run-slow-tests]
132+
run-slow-tests -.->|needs| slow-checks[slow-checks.yml]
133+
134+
%% Dependencies / Waiting (Dotted Lines)
135+
fast-checks -.->|needs| approve-merge[approve-merge]
136+
slow-checks -.->|needs| approve-merge
137+
138+
%% Final Dependency
139+
approve-merge -.->|needs| metrics
140+
141+
%% Visual Styling to distinguish jobs
142+
style approve-merge fill:#fff,stroke:#333,stroke-dasharray: 5 5
143+
style run-slow-tests fill:#fff,stroke:#333,stroke-dasharray: 5 5
144+
145+
146+
.. _pr_merge_yml:
147+
148+
Merge
149+
^^^^^
150+
151+
When a pull request is merged to main, then the ``pr-merge.yml`` workflow is activated.
152+
153+
.. mermaid::
154+
:name: merge-diagram
155+
156+
graph TD
157+
%% Workflow Triggers (Solid Lines)
158+
pr-merge[pr-merge.yml] --> checks[checks.yml]
159+
pr-merge --> publish-docs[publish-docs.yml]
160+
161+
%% Dependencies / Waiting (Dotted Lines)
162+
checks -.->|needs| report[report.yml]
163+
164+
.. _cd_yml:
165+
166+
Release
167+
^^^^^^^
168+
169+
When the nox session ``release:trigger`` is used, a new tag is created & pushed
170+
to main. This starts the release process by activating the ``cd.yml`` workflow.
171+
172+
.. mermaid::
173+
:name: release-diagram
174+
175+
graph TD
176+
%% Workflow Triggers (Solid Lines)
177+
cd[cd.yml] --> check-release-tag[check-release-tag.yml]
178+
179+
%% Dependencies / Waiting (Dotted Lines)
180+
check-release-tag -.->|needs| build-and-publish[build-and-publish.yml]
181+
build-and-publish -.->|needs| gh-pages[gh-pages.yml]

‎doc/user_guide/features/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ Features
1010
creating_a_release
1111
documentation/index
1212
git_hooks/index
13+
github_workflows/index
1314
formatting_code/index
1415
managing_dependencies
1516

‎doc/user_guide/user_guide.rst‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,5 @@
1010
getting_started
1111
configuration
1212
features/index
13-
workflows
1413
customization
1514
migrating

‎poetry.lock‎

Lines changed: 21 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pyproject.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ dependencies = [
5757
"sphinx-toolbox>=4.0.0,<5",
5858
"typer[all]>=0.7.0",
5959
"twine>=6.1.0,<7",
60+
"sphinxcontrib-mermaid (>=2.0.0,<3.0.0)",
6061
]
6162

6263
[project.scripts]

0 commit comments

Comments
 (0)