Skip to content

Commit 4e28ab9

Browse files
committed
Update README.md and export interface through __init__.py
1 parent f882ac1 commit 4e28ab9

4 files changed

Lines changed: 42 additions & 14 deletions

File tree

README.md

Lines changed: 19 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,27 @@
11
# Pipeline Component System (PCS)
22

3-
TODO
4-
53
A strange programming framework
64

75
The name is inspired by Entity Component System (ECS)
86

97
## Why?
108

11-
Why create this? I often find myself not liking the programs I create, and then end up rewriting them to be better, but they still end up quite brittle. This is a programming framework to make code cleaner and hopefully more maintainable. Have I succeeded in my goal? I have been using this framework for some time now and it has definitely helped my development speed and mental fatigue a lot! So yes, it has helped me achieve my goal!
9+
I often find myself not liking the programs I create, and then end up rewriting them to be better, but they still end up quite brittle. This is a programming framework to make code cleaner and hopefully more maintainable. Have I succeeded in my goal? I have been using this framework for some time now and it has definitely helped my development speed and mental fatigue a lot! So yes, it has helped me achieve my goal!
1210

1311
## Introduction
1412

15-
I will first discuss the few simple components which make up this framework, then connect them together, explaining choices I took along the way. If you wish to see an example of how this all ties together, look at `examples/example.py`.
13+
I will first discuss the few simple elements which make up this framework, then connect them together, explaining choices I took along the way. If you wish to see an example of how this all ties together, look at `examples/example.py`.
1614

1715
### Component
1816

19-
Think of the component as your global database. Each piece of persistent data (literal or object) is stored here. It is a dataclass, and it is the only dataclass (unless you want to nest them ofcourse). The reason for this design choice is that this way, we ALWAYS know where the data is. We do not have to guess which class owns what, like in those OOP messes.
17+
Think of the component as your global database. Each piece of persistent data (static or dyamic) is stored here. It is a dataclass, and it is the only dataclass (unless you want to nest them ofcourse). The reason for this design choice is that this way, we ALWAYS know where the data is. We do not have to guess which class owns what, unlike OOP soups.
2018

21-
The Component distinguishes between 2 data types: config (constant / defined at the start then not changed after initialization) and runtime (dynamic data which changes during runtime). The config variables can only be of primitive types (a restriction which comes from omegaconf, which this project depends on). Whereas the config class is
19+
The Component distinguishes between 2 data types: `conf` (constant / static / defined at the start then remains read-only after initialization) and `runtime` (dynamic data which changes during runtime). The config variables can only be of primitive types (a restriction which comes from omegaconf, which this project depends on).
2220

2321
```python
2422
@dataclass
2523
class Config: # Note: the name is not important
26-
i: int # Only primitive types in the config class
24+
i: int # Only primitive types in the config class (whatever OmegaConf is capable of)
2725
f: float
2826
s: str
2927
result: float
@@ -33,6 +31,7 @@ class Dynamic:
3331
di: int # Dynamic class can also take complex types
3432

3533
data = parse_arguments_cli(Config, Dynamic)
34+
data.seal() # Makes the `Config` part of the component read-only
3635

3736
print(data.i) # Print's the Config class' 'i'
3837
print(data.di) # Print's the Runtime class' 'i'
@@ -67,25 +66,34 @@ A pipeline takes a component, and a list of systems, then automatically passes t
6766
An example pipeline looks like this:
6867

6968
```python
70-
component = Component(1, 2, 'hello', -1)
7169
pipeline = Pipeline(
72-
component, [print_add_system, result_add_system, result_add_system2]
70+
component,
71+
[
72+
print_add_system,
73+
result_add_system,
74+
result_add_system2,
75+
]
7376
)
7477
pipeline.execute()
7578
pipeline.execute() # Execute pipeline a second time
7679
```
7780

7881
When a system returns a dictionary, the keys of the dict are interpreted to be the names of the component variables to replace with the value of the respective key. So the final 2 systems in the Systems examples will replace the `result` field.
7982

80-
Note that this helps us avoid having to pass parameters around, as it is done automatically for us, which cleans up the code base tremendously, as we have a concise pipeline definition, and when we call `Pipeline.execute`, we execute the 3 functions.
83+
Note that this helps us avoid having to pass parameters around, as it is done automatically for us, which cleans up the code base tremendously, as we have a concise pipeline definition, and when we call `Pipeline.execute`, we execute the 3 functions sequentially.
8184

8285
### Other handy tools
8386

84-
`parse_arguments_cli` will read your argvs using argparse and give you a component object ready to use. So you may run your file like so: `file.py --args-files="file1.yaml,file2.yaml" --rest a=1 --rest b=2`. Consecutive files will overwrite the previous entries, and `--rest` take precendence always, but each `--rest` takes precedence over the previous.
87+
`parse_arguments_cli` will read your `argv`s using argparse and give you a component object ready to use. So you may run your file like so: `file.py --args-files="file1.yaml,file2.yaml" --rest a=1 -r b=2`. Consecutive files will overwrite the previous entries, and `--rest/-r` take precendence always, but each `--rest` takes precedence over the previous.
8588

8689
* `--args-files` can be shortened to `-f`
8790
* `--rest` can be shortened to `-r`
8891

8992
## Some pattern ideas
9093

9194
* Nested pipelines
95+
* Pipeline in loops
96+
97+
# Example
98+
99+
Look at and run `example.py` for better usage

examples/example.py

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,7 @@
77

88
import omegaconf
99

10-
from pcs.argument_parser import parse_arguments_cli
11-
from pcs.pipeline import Pipeline
10+
from pcs import Pipeline, parse_arguments_cli
1211

1312

1413
@dataclass

pcs/__init__.py

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
from .argument_parser import (
2+
do_parse_arguments,
3+
parse_arguments_cli,
4+
parse_arguments_from_files,
5+
update_dict_with_comma_separated_file_list,
6+
update_dict_with_files,
7+
)
8+
from .component import Component
9+
from .init import initialize_object_nones
10+
from .pipeline import Pipeline
11+
12+
__all__ = [
13+
"Component",
14+
"Pipeline",
15+
"do_parse_arguments",
16+
"initialize_object_nones",
17+
"parse_arguments_cli",
18+
"parse_arguments_from_files",
19+
"update_dict_with_comma_separated_file_list",
20+
"update_dict_with_files",
21+
]

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[project]
22
name = "pipeline-component-system"
3-
version = "0.4.5"
3+
version = "0.5.0"
44
authors = [
55
{ name="Daniel Cauchi" },
66
]

0 commit comments

Comments
 (0)