decide on test granularity (#104)

A decision is suggested to cope with currently experienced limitations
(platformio/platformio-core#4849 (comment))
of PlatformIO.

Author: dhebbeker

doc/decisions/dr-001.md

@@ -1,4 +1,4 @@
-DR001 Represent Software Architecture in Directory Structure
+DR001 Represent Software Architecture in Directory Structure {#dr_001}
 ============================================================
 
 Context

doc/decisions/dr-003.md

@@ -0,0 +1,141 @@
+# DR003 Test Granularity and Stubbing Challenges {#dr_003}
+
+## Context
+
+Our software development project encounters challenges in designing unit tests due to issues with test granularity and stubbing dependencies.
+
+Note that we structure our software into packages (see 
+[Implementing a Plug-in Architecture](<\ref plugin_architecture> "Implementing a Plug-in Architecture"))
+which are technically implemented as [(private) libraries][3] (see 
+[Root Directory for the Package's source files](<\ref dr_002> "Root Directory for the Package's source files")
+).
+
+[3]: https://docs.platformio.org/en/latest/projectconf/sections/platformio/options/directory/lib_dir.html#lib-dir "documentation of `lib_dir`"
+
+## Challenges with Stubbing
+
+Our initial assumption was to write unit tests for individual units.
+However, challenges arose, particularly regarding the granularity of "unit" tests envisioned.
+
+### Option 0: Testing individual units {#dr_003_o0}
+
+A unit in this context is, for example, a class or a translation unit.
+A test would be compiled along with the unit under test (UUT), linked together, and executed.
+The build configuration should be defined within the [`platformio.ini` configuration file][1].
+
+A limitation of PlatformIO [has been experienced][7] specifically with test builds, where not only the UUT but the whole library in which it resides is included in the build (see [issue #4849][2]).
+This prevents stubbing dependencies of the UUT to other translation units within the same library, making it impossible to test individual units isolated from others this way.
+
+[1]: https://docs.platformio.org/en/latest/projectconf/#platformio-ini-project-configuration-file
+[2]: https://github.com/platformio/platformio-core/issues/4849 "Library Dependency Finder (LDF) adds too many files when testing (pio test)"
+[7]: https://community.platformio.org/t/partial-compilation-of-private-libraries-components-while-testing-for-different-environments/37079 "Partial compilation of private libraries/components (while testing) for different environments"
+
+### Option 1: Filter source files {#dr_003_o1}
+
+To overcome the limitations from 
+[option 0](<\ref dr_003_o0> "option 0"),
+one can use a [`library.json`][JSON] manifest file for each library in combination with the [`extraScript` option][4] to filter files.
+This way, individual files could be filtered for specific build targets, although this approach is expected to be effortful, as filters have to be implemented for each individual unit.
+
+[JSON]: https://docs.platformio.org/en/latest/manifests/library-json/index.html "PlatformIO documentation of manifest file"
+[4]: https://docs.platformio.org/en/latest/manifests/library-json/fields/build/extrascript.html "documentation of `extraScript`"
+
+### Option 2: Split units into separate libraries {#dr_003_o2}
+
+Another way to circumvent the limitations from 
+[option 0](<\ref dr_003_o0> "option 0")
+would be to separate each unit into an individual library.
+This way, only the UUT would be added to the test script for the test build.
+However, a major disadvantage would be that the planned structuring of the software packages into directories would be lost, making the analysis of the software structure more difficult.
+
+### Option 3: Test integrated units {#dr_003_o3}
+
+Test features using units and their dependencies to a certain degree:
+
+Either include (using [`lib_deps`][5] in [the project configuration][1]):
+
+- **Option 3a):** all dependencies compatible with the test environment
+- **Option 3b):** all dependencies compatible with the test environment, but no 
+  [3rd party library adapters](<\ref third_party_adapters> "3rd party library adapters")
+- **Option 3c):** only those dependencies which inevitably come as part of the library in which the UUT resides
+
+Omitted dependencies need to be substituted with stubs.
+
+[5]: https://docs.platformio.org/en/latest/projectconf/sections/env/options/library/lib_deps.html "documentation of `lib_deps`"
+
+## Comparison of Options {#dr_003_c1}
+
+- **Option 0:** Testing individual units without additional build configuration files
+  - Pros:
+    - would enable keeping the test design as originally envisioned
+    - errors in one unit would not affect other units
+  - Cons:
+    - currently **impossible** to implement
+- **Option 1:** Testing individual units with filters defined in additional files
+  - Pros: same as Option 0
+  - Cons:
+    - implementation is **effortful and more complex**
+- **Option 2:** Testing individual units by separating them into individual libraries
+  - Pros: same as Option 0
+  - Cons:
+    - the structuring of the software into packages could not be represented by directories anymore
+- **Option 3:** Testing integrated units
+  - Pros:
+    - fewer individual tests necessary
+    - less prone to changes (of details irrelevant for the test scope)
+    - fits better with the **Common Closure Principle**:
+      Tests are defined outside of the libraries but in a [dedicated test directory][6].
+      This is required by the build management tool currently used (PlatformIO).  
+      Interfaces of some units may change for other reasons than changes to the application requirements.
+      For example when interfaces within a layer or to lower level layers change due to refactoring.
+      These changes would require to adjust the tests.
+      This means that the change in one layer would lead to a change in the "global" test directory.
+      This effectively violates Common Closure Principle.
+  - Cons:
+    - to cover the same amount of control paths within the software with the integrated tests as with testing individual units, integrated test may require more complex or additional test cases.
+
+[6]: https://docs.platformio.org/en/stable/projectconf/sections/platformio/options/directory/test_dir.html "documentation of `test_dir`"
+
+### Comparing Options 3a)-3c) {#dr_003_c2}
+
+- **Option 3a):** Testing integrated units including all compatible dependencies
+  - Pros:
+    - highest test coverage of options 3a)-3c)
+  - Cons:
+    - requires stubs of the 3rd party dependencies incompatible with the test environment
+- **Option 3b):** Testing integrated units including all compatible dependencies, excluding 3rd party adapters
+  - Pros:
+    - Does not depend on the interface of the 3rd party dependencies incompatible with the test environment.
+  - Cons:
+    - requires stubs of all 3rd party adapters
+- **Option 3c):** Testing integrated units of the libraries each
+  - Pros:
+    - Does not depend on the interface of the 3rd party dependencies incompatible with the test environment.
+  - Cons:
+    - requires the highest amount of stubs from the options 3a)-3c)
+
+## Decision & Rationale {#dr_003_decision}
+
+What do we want to achieve with the tests?
+To check if the observable behavior of the overall software is as expected.
+
+As long as libraries are not delivered separately, it is not necessary to test them individually.
+
+After careful consideration, we've decided on:
+
+> **Option 3b):** Testing integrated units including all compatible dependencies, excluding 3rd party adapters
+
+This decision aligns with the test objective, aiming to achieve good software quality while avoiding excessive effort.
+
+This design decision shall be regarded as a minimum requirement.
+Additional tests or stubbing may be defined at one's liberty.
+
+## Next Steps {#dr_003_next}
+
+To implement this decision:
+
+- Future tests are only required for the application layer.
+  Units from lower layers will be tested indirectly.
+- Future tests may test several units at once.
+- Legacy tests that require modification must be checked if they satisfy the new test design and modified or removed accordingly.
+