Skip to content

Commit ec2aa29

Browse files
CowKeyManCowKeyMan
committed
Add minimal docs, component.py assert no duplicate keys
1 parent 07fc617 commit ec2aa29

2 files changed

Lines changed: 120 additions & 120 deletions

File tree

‎README.md‎

Lines changed: 88 additions & 113 deletions
Original file line numberDiff line numberDiff line change
@@ -1,116 +1,91 @@
11
# Pipeline Component System (PCS)
22

33
TODO
4-
<!---->
5-
<!-- A strange programming framework -->
6-
<!---->
7-
<!-- The name is inspired by Entity Component System (ECS) -->
8-
<!---->
9-
<!-- ## Why? -->
10-
<!---->
11-
<!-- Why create this? I often find myself not liking the programs I create, and then end up rewriting them to be better, but they still end up quite brittle. This is a programming framework to make code cleaner and hopefully more maintainable. Have I succeeded in my goal? I'm not sure, but I will try it out myself more and update this documentation here if it is good or not. -->
12-
<!---->
13-
<!-- ## Introduction -->
14-
<!---->
15-
<!-- I will first discuss the few simple components which make up this framework, then connect them together, explaining choices I took along the way. If you wish to see an example of how this all ties together, look at `examples/basic.py`. -->
16-
<!---->
17-
<!-- ### Component -->
18-
<!---->
19-
<!-- Think of the component as your global database. Each piece of persistent data (literal or object) is stored here. It is a dataclass, and it is the only dataclass (unless you want to nest them ofcourse). The reason for this design choice is that this way, we ALWAYS know where the data is. We do not have to guess which class owns what. The Component owns everything. A simple component looks like the following: -->
20-
<!---->
21-
<!-- ```python -->
22-
<!-- @dataclass -->
23-
<!-- class Component: # Note: the name is not important. I like to use `Data` as well -->
24-
<!-- i: int -->
25-
<!-- f: float -->
26-
<!-- s: str -->
27-
<!-- result: float -->
28-
<!---->
29-
<!-- component = Component(1, 2, 'hello', -1) -->
30-
<!-- ``` -->
31-
<!---->
32-
<!-- Components can store anything. Note: the types are recommended (for static analysers) but not enforced. -->
33-
<!---->
34-
<!-- ### Systems -->
35-
<!---->
36-
<!-- Systems are functions, with parameter names equivalent to the fields in the component. That's it. An example system may look like the following: -->
37-
<!---->
38-
<!-- ```python -->
39-
<!-- def print_add_system(i: int, f: float): # Note: the variable names match those in the component exactly -->
40-
<!-- print("Add System:", i + f) -->
41-
<!---->
42-
<!---->
43-
<!-- def result_add_system(i: int, f: float, result: float): -->
44-
<!-- result = result + i + f -->
45-
<!-- return {"result": result} -->
46-
<!---->
47-
<!---->
48-
<!-- def result_add_system2(i: int, f: float, result: float): -->
49-
<!-- return {"result": result + i + f} # Note: the key matches the variable names in the component exactly -->
50-
<!-- ``` -->
51-
<!---->
52-
<!-- Take note of the return at the end of the last 2 systems. We will discuss this syntax in the `Pipeline` section. -->
53-
<!---->
54-
<!-- ### Pipeline -->
55-
<!---->
56-
<!-- A pipeline takes a component, and a list of systems, then automatically passes the fields of the component to the systems, and writes results back to the component. -->
57-
<!---->
58-
<!-- An example pipeline looks like this: -->
59-
<!---->
60-
<!-- ```python -->
61-
<!-- component = Component(1, 2, 'hello', -1) -->
62-
<!-- pipeline = Pipeline( -->
63-
<!-- component, [print_add_system, result_add_system, result_add_system2] -->
64-
<!-- ) -->
65-
<!-- pipeline.execute() -->
66-
<!-- pipeline.execute() # Execute pipeline a second time -->
67-
<!-- ``` -->
68-
<!---->
69-
<!-- When a system returns a dictionary, the keys of the dict are interpreted to be the names of the component variables to replace with the value of the respective key. So the final 2 systems in the Systems examples will replace the `result` field. -->
70-
<!---->
71-
<!-- Note that this helps us avoid having to pass parameters around, as it is done automatically for us, which cleans up the code base tremendously, as we have a concise pipeline definition, and when we call `Pipeline.execute`, we execute the 3 functions. -->
72-
<!---->
73-
<!-- ### Other handy tools -->
74-
<!---->
75-
<!-- #### `initialize_object_nones` -->
76-
<!---->
77-
<!-- If we want to initialize all the component variables to `None`, instead of: -->
78-
<!---->
79-
<!-- ```python -->
80-
<!-- component = Component(None, None, None, None) -->
81-
<!-- ``` -->
82-
<!---->
83-
<!-- We can do: -->
84-
<!---->
85-
<!-- ```python -->
86-
<!-- from pcs.init import initialize_object_nones -->
87-
<!---->
88-
<!-- component = initialize_object_nones(Component) -->
89-
<!-- ``` -->
90-
<!---->
91-
<!-- Why would we want to do this? Well sometimes we want to initialize only some of our variables in the Component object, but not all. So we initialize everything initially to `None`s, and then replace the `None`s with the actual value we want. Look at the `parse_arguments` section to find out a more useful reason for this! -->
92-
<!---->
93-
<!-- #### `parse_arguments` -->
94-
<!---->
95-
<!-- This function is here to replace all your argument parsing forever! By specifying default variables for some of your arguments in a yaml file, these will be loaded into your component. All you need to do is call: -->
96-
<!---->
97-
<!-- ```python -->
98-
<!-- from pcs.init import initialize_object_nones -->
99-
<!-- from pcs.argument_parser import parse_arguments -->
100-
<!---->
101-
<!-- component = initialize_object_nones(Component) -->
102-
<!-- parse_arguments(component) -->
103-
<!-- ``` -->
104-
<!---->
105-
<!-- An example of such a yaml file is in `examples/configs/default.yaml` -->
106-
<!---->
107-
<!-- Then to run your application, you can call (for the example): `python examples/basic.py --args-files=examples/configs/default.yaml,examples/configs/override1.yaml --rest s="hello world" -r s2="hello world2"` -->
108-
<!---->
109-
<!-- Note: config files specified later will override earlier ones, and 'rest' options will override options in the config files. You can also use `-r` as a shorthand for `--rest`. -->
110-
<!---->
111-
<!-- You will need to call `initialize_object_nones` beforehand. -->
112-
<!---->
113-
<!-- ## Some pattern ideas -->
114-
<!---->
115-
<!-- * Use `parse_arguments` on a different component than your main component, and maybe pass the 'args' component to the main component. -->
116-
<!-- * Nested pipelines -->
4+
5+
A strange programming framework
6+
7+
The name is inspired by Entity Component System (ECS)
8+
9+
## Why?
10+
11+
Why create this? I often find myself not liking the programs I create, and then end up rewriting them to be better, but they still end up quite brittle. This is a programming framework to make code cleaner and hopefully more maintainable. Have I succeeded in my goal? I have been using this framework for some time now and it has definitely helped my development speed and mental fatigue a lot! So yes, it has helped me achieve my goal!
12+
13+
## Introduction
14+
15+
I will first discuss the few simple components which make up this framework, then connect them together, explaining choices I took along the way. If you wish to see an example of how this all ties together, look at `examples/example.py`.
16+
17+
### Component
18+
19+
Think of the component as your global database. Each piece of persistent data (literal or object) is stored here. It is a dataclass, and it is the only dataclass (unless you want to nest them ofcourse). The reason for this design choice is that this way, we ALWAYS know where the data is. We do not have to guess which class owns what, like in those OOP messes.
20+
21+
The Component distinguishes between 2 data types: config (constant / defined at the start then not changed after initialization) and runtime (dynamic data which changes during runtime). The config variables can only be of primitive types (a restriction which comes from omegaconf, which this project depends on). Whereas the config class is
22+
23+
```python
24+
@dataclass
25+
class Config: # Note: the name is not important
26+
i: int # Only primitive types in the config class
27+
f: float
28+
s: str
29+
result: float
30+
31+
@dataclass
32+
class Dynamic:
33+
di: int # Dynamic class can also take complex types
34+
35+
data = parse_arguments_cli(Config, Dynamic)
36+
37+
print(data.i) # Print's the Config class' 'i'
38+
print(data.di) # Print's the Runtime class' 'i'
39+
```
40+
41+
We can print this data object and we can also serialize it with pickle. The types are necessary for the Config dataclass, and recommended for the Dynamic class. They are enfored in the Config class, but not the Dynamic.
42+
43+
### Systems
44+
45+
Systems are functions, with parameter names equivalent to the fields in the component. That's it. An example system may look like the following:
46+
47+
```python
48+
def print_add_system(i: int, f: float): # Note: the variable names match those in the component exactly
49+
print("Add System:", i + f)
50+
51+
52+
def result_add_system(i: int, f: float, result: float):
53+
result = result + i + f
54+
return {"result": result}
55+
56+
57+
def result_add_system2(i: int, f: float, result: float):
58+
return {"result": result + i + f} # Note: the key matches the variable names in the component exactly
59+
```
60+
61+
Take note of the return at the end of the last 2 systems. We will discuss this syntax in the `Pipeline` section.
62+
63+
### Pipeline
64+
65+
A pipeline takes a component, and a list of systems, then automatically passes the fields of the component to the systems, and writes results back to the component.
66+
67+
An example pipeline looks like this:
68+
69+
```python
70+
component = Component(1, 2, 'hello', -1)
71+
pipeline = Pipeline(
72+
component, [print_add_system, result_add_system, result_add_system2]
73+
)
74+
pipeline.execute()
75+
pipeline.execute() # Execute pipeline a second time
76+
```
77+
78+
When a system returns a dictionary, the keys of the dict are interpreted to be the names of the component variables to replace with the value of the respective key. So the final 2 systems in the Systems examples will replace the `result` field.
79+
80+
Note that this helps us avoid having to pass parameters around, as it is done automatically for us, which cleans up the code base tremendously, as we have a concise pipeline definition, and when we call `Pipeline.execute`, we execute the 3 functions.
81+
82+
### Other handy tools
83+
84+
`parse_arguments_cli` will read your argvs using argparse and give you a component object ready to use. So you may run your file like so: `file.py --args-files="file1.yaml,file2.yaml" --rest a=1 --rest b=2`. Consecutive files will overwrite the previous entries, and `--rest` take precendence always, but each `--rest` takes precedence over the previous.
85+
86+
* `--args-files` can be shortened to `-f`
87+
* `--rest` can be shortened to `-r`
88+
89+
## Some pattern ideas
90+
91+
* Nested pipelines

‎pcs/component.py‎

Lines changed: 32 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
from typing import Generic, TypeVar
22

33
from omegaconf import DictConfig, OmegaConf
4+
45
from pcs.init import initialize_object_nones
56

67
T = TypeVar("T")
@@ -15,14 +16,38 @@ def init_with_conf(
1516
return cls(conf, runtime)
1617

1718
def __init__(self, conf: DictConfig, runtime: T):
18-
duplicate_keys = set(conf.keys()).intersection(set(runtime.__dict__))
19-
assert len(duplicate_keys) == 0, (
20-
f"Error initializing component - duplicate keys found: {duplicate_keys}"
21-
)
19+
internal_attr_to_value = {
20+
"conf": conf,
21+
"runtime": runtime,
22+
"sealed": False,
23+
}
24+
conf_set = set(conf.keys())
25+
runtime_set = set(runtime.__dict__)
26+
internal_set = set(internal_attr_to_value.keys())
27+
28+
def assert_no_duplicate_keys(set1: set, set2: set, error_message):
29+
duplicate_keys = set1.intersection(set2)
30+
assert (
31+
len(duplicate_keys) == 0
32+
), f"Error initializing component - {error_message} - {duplicate_keys=}"
2233

23-
super().__setattr__("conf", conf)
24-
super().__setattr__("runtime", runtime)
25-
super().__setattr__("sealed", False)
34+
assert_no_duplicate_keys(
35+
conf_set,
36+
runtime_set,
37+
"conf and runtime set cannot have keys which are named the same",
38+
)
39+
assert_no_duplicate_keys(
40+
internal_set,
41+
conf_set,
42+
"conf set has keys which are equal to internal component keys",
43+
)
44+
assert_no_duplicate_keys(
45+
internal_set,
46+
runtime_set,
47+
"runtime set has keys which are equal to internal component keys",
48+
)
49+
for k, v in internal_attr_to_value.items():
50+
super().__setattr__(k, v)
2651

2752
def seal(self):
2853
OmegaConf.set_readonly(super().__getattribute__("conf"), True)

0 commit comments

Comments
 (0)