Skip to content

Commit c5358cb

Browse files
romeokienzlerromeokienzler
authored
Merge pull request #310 from claimed-framework/add_doc_improve_some_code
Add doc improve some code
2 parents 465c924 + 330c494 commit c5358cb

18 files changed

Lines changed: 969 additions & 56 deletions

‎.github/workflows/docs.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: Publish Docs to GitHub Pages
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
paths:
8+
- 'docs/**'
9+
- 'mkdocs.yml'
10+
- 'src/**'
11+
- '.github/workflows/docs.yml'
12+
workflow_dispatch:
13+
14+
permissions:
15+
contents: write
16+
17+
jobs:
18+
deploy:
19+
runs-on: ubuntu-latest
20+
21+
steps:
22+
- name: Checkout repository
23+
uses: actions/checkout@v4
24+
with:
25+
fetch-depth: 0 # full history needed for git-committers plugin
26+
27+
- name: Set up Python
28+
uses: actions/setup-python@v5
29+
with:
30+
python-version: '3.11'
31+
cache: pip
32+
33+
- name: Install doc dependencies
34+
run: |
35+
pip install \
36+
mkdocs-material \
37+
mkdocstrings[python] \
38+
mkdocs-git-revision-date-localized-plugin \
39+
griffe
40+
41+
- name: Install claimed package (for mkdocstrings introspection)
42+
run: pip install -e .
43+
44+
- name: Build & deploy docs
45+
run: mkdocs gh-deploy --force --clean --verbose

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,4 @@ dist
1212
*.bak
1313
*.swp
1414
.DS_Store
15+
site/

‎docs/assets/logo.png‎

48.5 KB
Loading

‎docs/c3/create-gridwrapper.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# create_gridwrapper
2+
3+
Wraps an existing component to run in parallel over a collection of inputs
4+
using one of several storage backends.
5+
6+
## CLI
7+
8+
```bash
9+
c3_create_gridwrapper <source_file> [options]
10+
```
11+
12+
| Option | Type | Default | Description |
13+
|---|---|---|---|
14+
| `source_file` | path | *required* | `.ipynb` or `.py` component to wrap |
15+
| `--backend` | str | `local` | Storage backend (see table below) |
16+
| `--component-inputs` | str | `''` | Comma-separated parameter names that vary per grid cell |
17+
| `--component-dependencies` | str | `''` | Pip dependencies to inject |
18+
| `--repository` | str | | Container registry namespace |
19+
| `--log-level` | str | `WARNING` | Python logging level |
20+
21+
## Backends
22+
23+
| Key | Description |
24+
|---|---|
25+
| `local` | Local filesystem, simple parallelism |
26+
| `cos` | IBM COS – iterate over objects in a bucket prefix |
27+
| `s3kv` | MLX S3 key-value store backend |
28+
| `simple_grid_wrapper` | Source-only, minimal overhead |
29+
| `folder_grid_wrapper` | Separate source and target folder |
30+
| `legacy_cos_grid_wrapper` | Older COS format |
31+
32+
## Python API
33+
34+
::: claimed.c3.create_gridwrapper
35+
options:
36+
members:
37+
- wrap_component
38+
- create_gridwrapper
39+
40+
## Example
41+
42+
```bash
43+
# Wrap a training script to process every CSV in a COS bucket in parallel
44+
c3_create_gridwrapper train_model.py \
45+
--backend cos \
46+
--component-inputs input_file \
47+
--repository docker.io/myuser
48+
```
49+
50+
This emits `gw_train_model.py` which, when containerised, launches one worker per input file.

‎docs/c3/create-operator.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# create_operator
2+
3+
Builds a Docker image and generates KFP, CWL, and Kubernetes descriptors
4+
from a Jupyter notebook, Python script, or R script.
5+
6+
## CLI
7+
8+
```bash
9+
c3_create_operator <source_file> [options]
10+
```
11+
12+
| Option | Type | Default | Description |
13+
|---|---|---|---|
14+
| `source_file` | path | *required* | `.ipynb`, `.py`, or `.R` file |
15+
| `--repository` | str | *required* | Container registry namespace, e.g. `docker.io/myuser` |
16+
| `--version` | str | auto | Image tag; auto-detected from `image_version` variable in source |
17+
| `--additional-files` | list | `[]` | Extra files to `ADD` into the image |
18+
| `--dockerfile` | path | auto | Custom Dockerfile template |
19+
| `--log-level` | str | `WARNING` | Python logging level |
20+
21+
## Python API
22+
23+
::: claimed.c3.create_operator
24+
options:
25+
members:
26+
- create_operator
27+
- create_dockerfile
28+
29+
## Output Files
30+
31+
After a successful run you will find:
32+
33+
| File | Description |
34+
|---|---|
35+
| `<name>.dockerfile` | Generated Dockerfile |
36+
| `<name>.yaml` | KubeFlow Pipelines component spec |
37+
| `<name>.job.yaml` | Kubernetes Job spec |
38+
| `<name>.cwl` | CWL component descriptor |

‎docs/c3/index.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# C3 – CLAIMED Component Compiler
2+
3+
C3 automates the transformation of arbitrary code assets into fully portable, executable AI components.
4+
5+
---
6+
7+
## What C3 does
8+
9+
```
10+
┌──────────────────────┐
11+
│ .ipynb / .py / .R │ ← your code
12+
└──────────┬───────────┘
13+
│ c3_create_operator
14+
▼
15+
┌──────────────────────────────────────────┐
16+
│ Dockerfile (build + push) │
17+
│ KubeFlow component YAML │
18+
│ Kubernetes Job YAML │
19+
│ CWL component descriptor │
20+
└──────────────────────────────────────────┘
21+
```
22+
23+
C3 reads **parameter declarations** from the top of your source file:
24+
25+
```python
26+
import os
27+
28+
# description of my_param
29+
my_param = os.environ.get('my_param', 'default_value')
30+
```
31+
32+
Each `os.environ.get(...)` line is parsed into a typed, documented parameter
33+
that appears in the generated YAML descriptors and KFP UI.
34+
35+
---
36+
37+
## Modules
38+
39+
| Module | CLI entry-point | Purpose |
40+
|---|---|---|
41+
| [`create_operator`](create-operator.md) | `c3_create_operator` | Build container images and component descriptors |
42+
| [`create_gridwrapper`](create-gridwrapper.md) | `c3_create_gridwrapper` | Wrap a component for parallel grid execution |
43+
| [`create_containerless_operator`](create-operator.md) | `c3_create_containerless_operator` | Containerless variant (runs in-process) |
44+
| [`operator_utils`](operator-utils.md) | – | Shared helpers (connection strings, logging) |
45+
| `parser` | – | Source-file parameter parser |
46+
| `notebook` | – | Jupyter notebook handler |
47+
| `pythonscript` | – | Python script handler |
48+
| `rscript` | – | R script handler |
49+
50+
---
51+
52+
## Grid Compute Backends
53+
54+
| Backend key | Description |
55+
|---|---|
56+
| `local` | Plain local filesystem |
57+
| `cos` / `cos_grid_wrapper` | IBM Cloud Object Storage |
58+
| `s3kv` | S3-backed key-value store (MLX) |
59+
| `simple_grid_wrapper` | Minimal wrapper – source folder only |
60+
| `folder_grid_wrapper` | Source **and** target folder variant |
61+
| `legacy_cos_grid_wrapper` | Older COS format, kept for backwards compatibility |

‎docs/c3/operator-utils.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# operator_utils
2+
3+
Shared utility helpers used across C3 and the component library.
4+
5+
## Python API
6+
7+
::: claimed.c3.operator_utils
8+
9+
## Connection String Format
10+
11+
Many CLAIMED components accept a `cos_connection` parameter in the following URI format:
12+
13+
```
14+
[cos|s3]://access_key_id:secret_access_key@endpoint_host/bucket/path
15+
```
16+
17+
**Examples:**
18+
19+
```
20+
s3://AKIAIOSFODNN7EXAMPLE:wJalrXUtnFEMI@s3.us-east-1.amazonaws.com/my-bucket/data/
21+
cos://mykey:mysecret@s3.eu-de.cloud-object-storage.appdomain.cloud/my-bucket/models/
22+
```
23+
24+
### `explode_connection_string(cs)`
25+
26+
Parses the URI into its components:
27+
28+
```python
29+
from claimed.c3.operator_utils import explode_connection_string
30+
31+
access_key_id, secret_access_key, endpoint, path = explode_connection_string(
32+
's3://KEY:SECRET@s3.eu-de.cloud-object-storage.appdomain.cloud/my-bucket/prefix'
33+
)
34+
# endpoint → 'https://s3.eu-de.cloud-object-storage.appdomain.cloud'
35+
# path → 'my-bucket/prefix'
36+
```
37+
38+
If the string does not start with `cos://` or `s3://`, the input is returned as-is in the `path` field
39+
(useful when passing a plain local path or a Kubernetes secret reference).

‎docs/cli.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# CLI Reference
2+
3+
The `claimed` command is the single entry-point for the CLAIMED framework.
4+
5+
---
6+
7+
## Synopsis
8+
9+
```
10+
claimed <subcommand> [options]
11+
```
12+
13+
---
14+
15+
## Subcommands
16+
17+
### `claimed run`
18+
19+
Directly invoke the `run()` function of any CLAIMED Python module.
20+
21+
```
22+
claimed run <module.path> [--param-name value ...] [--help]
23+
```
24+
25+
**Arguments**
26+
27+
| Argument | Description |
28+
|---|---|
29+
| `module.path` | Fully-qualified Python module containing a `run()` function (e.g. `claimed.components.util.cosutils`) |
30+
| `--<param-name>` | Any parameter accepted by `run()`. Hyphens are converted to underscores. |
31+
| `--help` | Print the function signature, docstring, and parameter list, then exit. |
32+
33+
**Type coercion**
34+
35+
String values from the command line are automatically cast to the type declared in the function signature
36+
(annotation or default-value type).
37+
For example, `--recursive true` is cast to `bool` if the parameter is annotated as `bool`.
38+
39+
**Examples**
40+
41+
```bash
42+
# List objects in a COS bucket
43+
claimed run claimed.components.util.cosutils \
44+
--cos-connection s3://KEY:SECRET@endpoint/bucket \
45+
--operation ls \
46+
--local-path .
47+
48+
# Download a file
49+
claimed run claimed.components.util.cosutils \
50+
--cos-connection s3://KEY:SECRET@endpoint/bucket/file.zip \
51+
--operation get \
52+
--local-path .
53+
54+
# Show help for any module
55+
claimed run claimed.components.util.cosutils --help
56+
57+
# CPU benchmark
58+
claimed run claimed.components.util.gpu_performance_test \
59+
--mode cpu \
60+
--matrix-size 4096 \
61+
--iterations 100
62+
```
63+
64+
---
65+
66+
### `claimed create operator`
67+
68+
Generate a container image + KFP/CWL/Kubernetes descriptors from a script or notebook.
69+
70+
```
71+
claimed create operator <script_or_notebook> [options]
72+
```
73+
74+
| Option | Description |
75+
|---|---|
76+
| `--repository` | Container registry namespace, e.g. `docker.io/myuser` |
77+
| `--version` | Image tag (default: auto-detected from script) |
78+
| `--additional-files` | Space-separated list of extra files to bundle |
79+
80+
Example:
81+
82+
```bash
83+
claimed create operator my_script.py --repository docker.io/myuser
84+
```
85+
86+
---
87+
88+
### `claimed create gridwrapper`
89+
90+
Wrap a component so it executes in parallel over a collection of inputs.
91+
92+
```
93+
claimed create gridwrapper <script_or_notebook> [options]
94+
```
95+
96+
| Option | Description |
97+
|---|---|
98+
| `--backend` | Storage backend: `local` \| `cos` \| `s3kv` \| `simple_grid_wrapper` \| `folder_grid_wrapper` |
99+
| `--component-inputs` | Comma-separated parameter names that vary across grid cells |
100+
| `--repository` | Container registry namespace |
101+
102+
Example:
103+
104+
```bash
105+
claimed create gridwrapper my_script.py \
106+
--backend cos \
107+
--component-inputs input_file \
108+
--repository docker.io/myuser
109+
```
110+
111+
---
112+
113+
### `claimed --component` *(legacy)*
114+
115+
Run a component image via Docker.
116+
117+
```
118+
claimed --component <image> [--param-name value ...]
119+
```
120+
121+
| Option | Description |
122+
|---|---|
123+
| `--component` | Docker image reference, e.g. `docker.io/claimed/my-op:latest` |
124+
| `--<param-name>` | Environment variable to pass into the container |
125+
126+
Set `CLAIMED_DATA_PATH` to mount a local directory as `/opt/app-root/src/data` inside the container.
127+
128+
---
129+
130+
## Environment Variables
131+
132+
| Variable | Effect |
133+
|---|---|
134+
| `CLAIMED_DATA_PATH` | Local path mounted as `/opt/app-root/src/data` when using `--component` |
135+
| `CLAIMED_CONTAINERLESS_OPERATOR_PATH` | Root path for containerless operator resolution |

0 commit comments

Comments
 (0)