Make testcase docs more consistent, introduce testcase url

Signed-off-by: Matthias Büchse <matthias.buechse@alasca.cloud>

Author: mbuechse

Standards/scs-0003-v1-sovereign-cloud-standards-yaml.md

@@ -226,7 +226,8 @@ TBD
 | ----------------- | --------------- | ------------------------------------------------------------------------------------------------- | ----------------- |
 | `id`              | String          | Identifier for this test case (immutable and unique within this module)                           | `image-md-check`  |
 | `lifetime`        | String          | One of: `day`, `week` (_default_), `month`, `quarter`, `year`                                     | `day`             |
-| `description`     | String          | Short description of the test case                                                                |                   |
+| `description`     | String          | Short description of the test case (markdown allowed, but keep it short for CLI users)            |                   |
+| `url`             | String          | URL pointing to the relevant SCS documentation for the testcase                                   |                   |
 
 A test result is valid until the end of the next period, except when lifetime is `year`: then the result is
 valid until the end of the following month plus one year.

Standards/scs-0100-w1-flavor-naming-implementation-testing.md

@@ -154,30 +154,12 @@ generation `x` for the time being.
 
 ## Automated tests
 
-### Errors
-
-The following items MUST be detected and reported as an error:
-
-- any syntax error in a name starting with `SCS-`,
-- any mismatch between any immediately discoverable property of a flavor (currently, CPU, RAM and disk size)
-  and the meaning of its name (which is usually a lower bound), such as the CPU generation or hypervisor.
-
-In addition, the following items MAY be reported as an error:
-
-- any mismatch between any non-immediately discoverable property of flavor and the meaning of its name.
-
-### Warnings
-
-None so far.
-
-### Implementation
-
-We implemented two testcases, paralleling the two items in the "Errors" section above:
+The following testcases [are implemented](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/openstack_test.py):
 
 - `scs-0100-syntax-check` ensures that any name starting with `SCS-` adheres to the standard;
-- `scs-0100-semantics-check` ensures that any such name is telling the truth as specified in the standard.
-
-These testcases can be checked using [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/openstack_test.py).
+- `scs-0100-semantics-check` ensures that any such name is telling the truth as specified in the standard;
+  specifically: any immediately discoverable property of a flavor (currently, CPU, RAM and disk size)
+  matches the meaning of its name (which is usually a lower bound), such as the CPU generation or hypervisor.
 
 ## Manual tests
 

Standards/scs-0101-w1-entropy-implementation-testing.md

@@ -19,57 +19,26 @@ Only the flavor and image attributes required by the standard have to be set:
 
 ## Automated tests
 
-### Images sample
+The following testcases [are implemented](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py):
 
-Some checks need to be performed on a live instance. For these checks, it is
-necessary to choose a sample of VM images to test on.
-
-For the time being, the sample MUST contain at least one public image reported
-by OpenStack. This may be extended in the future.
-
-### Errors
-
-For every image in the chosen sample, the following items MUST be detected and
-reported as an error:
-
-- the service `rngd` is not running,
-- the special file `/proc/sys/kernel/random/entropy_avail` does not contain
-  the value 256 (pinned since kernel 5.18),
-- the number of FIPS 140-2 failures exceeds 5 out of 1000 blocks
-  tested, as determined by `cat /dev/random | rngtest -c 1000` .
+- `scs-0101-image-property` ensures that each flavor has the extra spec `hw_rng:allowed=True`,
+- `scs-0101-flavor-property` ensures that each public image has the property `hw_rng_model: virtio`,
+- `scs-0101-rngd` ensures that the service `rngd` is present on a sample VM,
+- `scs-0101-entropy-avail` ensures that the special file `/proc/sys/kernel/random/entropy_avail` contains
+  the value 256 (pinned since kernel 5.18) on a sample VM,
+- `scs-0101-fips-test` ensures that the number of FIPS 140-2 failures is below 5 out of 1000 blocks
+  tested, as determined by `cat /dev/random | rngtest -c 1000` on a sample VM.
 
 Note: The latter two items act as surrogates for the following item, which
 cannot be detected directly:
 
-- CPU instructions for accessing entropy are not available to the VMs.
-
-### Warnings
-
-The following items MUST be detected and reported as a warning:
-
-- any flavor missing the attribute `hw_rng:allowed=True`,
-- any image missing the attribute `hw_rng_model: virtio`,
-- the number of FIPS 140-2 failures exceeds 3 out of 1000 blocks
-  tested (compare with errors).
+- CPU instructions for accessing entropy are available to the VMs.
 
 Note that the requirement regarding the kernel patch level will not be
 checked, because of two reasons: (a) we already check the file `entropy_avail`
 (see subsection on Errors), and (b) users can always choose a recent image,
 as ensured by the image metadata standard.
 
-### Implementation
-
-We implemented the following testcases that reflect the items in the above section
-on automated tests:
-
-- `scs-0101-image-property`,
-- `scs-0101-flavor-property`,
-- `scs-0101-entropy-avail`,
-- `scs-0101-rngd`,
-- `scs-0101-fips-test` (covers both the error and warning case).
-
-These testcases can be checked using [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).
-
 ## Manual tests
 
 None.

Standards/scs-0102-w1-image-metadata-implementation-testing.md

@@ -15,8 +15,10 @@ for these images.
 
 ## Automated tests
 
-We implemented a host of testcases to reflect the requirements and recommendations of the standard. The following
-testcases ensure that fields have proper values:
+We [implemented](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
+a host of testcases to reflect the requirements and recommendations of the standard.
+
+The following testcases ensure that fields have proper values:
 
 - `scs-0102-prop-architecture`,
 - `scs-0102-prop-hash_algo`,
@@ -42,9 +44,6 @@ The following testcase ensures that each image is as recent as claimed by its `r
 
 - `scs-0102-image-recency`
 
-The script [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
-can be used to check these testcases.
-
 ## Manual tests
 
 None.

Standards/scs-0103-w1-standard-flavors-implementation.md

@@ -17,13 +17,11 @@ You can generate input for it using the tool
 
 ## Automated tests
 
-We implemented a set of testcases corresponding 1:1 to the standard flavors:
+We [implemented](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
+a set of testcases corresponding 1:1 to the standard flavors:
 
 - `scs-0103-flavor-X` with varying `X`: ensures that flavor `SCS-X` is present and has the
   required `extra_specs`.
 
 _NOTE_: We still need to add testcases to ensure that the `extra_specs` of non-standard
 flavors are correct as well.
-
-The testcases can be run using the script
-[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0104-w1-standard-images-implementation.md

@@ -61,7 +61,8 @@ Please see [its documentation](https://docs.scs.community/docs/iaas/components/i
 
 ## Automated tests
 
-We implemented testcases reflecting the information given in the YAML file(s). Be advised that
+We [implemented](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
+testcases reflecting the information given in the YAML file(s). Be advised that
 the approach with the YAML file has considerable drawbacks, and it will be abandoned in
 future versions of the standard in favor of a more classical approach. The testcases already
 anticipate this future development.
@@ -72,6 +73,3 @@ There are two classes of testcases:
   these ensure that certain images have the correct `image_source`;
 - `scs-0104-image-X` with varying `X`:
   these ensure that certain images can be found in the list of public images.
-
-The testcases can be run using the script
-[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0114-w1-volume-type-implementation.md

@@ -8,12 +8,10 @@ supplements:
 
 ## Automated tests
 
-We implemented the following testcases:
+We [implemented](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
+the following testcases:
 
 - `scs-0114-syntax-check` ensures that, for every volume type description,
   the list of aspects, if present, is formatted according to the standard.
 - `scs-0114-encrypted-type` ensures that a volume type featuring encryption is present.
 - `scs-0114-replicated-type` ensures that a volume type featuring replication is present.
-
-The testcases can be run using the script
-[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0115-w1-security-groups-implementation.md

@@ -8,14 +8,11 @@ supplements:
 
 ## Automated tests
 
-We implemented a single test case,
+We [implemented](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py) a single test case,
 
 - `scs-0115-default-rules`,
 
 which ensures
 
 1. the absence of any ingress traffic rules except traffic from the same Security Group in the `openstack default security group rule list`,
 2. the presence of any egress traffic rules.
-
-The testcase can be run using the script
-[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0116-w1-key-manager-implementation-testing.md

@@ -40,14 +40,12 @@ This can be done with a small change in the policy.yaml file. The `creator` has
 
 ## Automated Tests
 
-We implemented the following testcases, in accordance with the standard:
+We [implemented](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
+the following testcases, in accordance with the standard:
 
 - `scs-0116-presence` ensures that a service of type "key-manager" occurs in the service catalog;
 - `scs-0116-permissions` ensures that a regular user has suitable access to the key-manager API.
 
-The testcases can be run using the script
-[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).
-
 ## Manual Tests
 
 It is not possible to check a deployment for a correctly protected Master KEK automatically from the outside.

Standards/scs-0117-w1-volume-backup-service-implementation.md

@@ -8,17 +8,15 @@ supplements:
 
 ## Automated tests
 
-We implemented a single testcase,
+We [implemented](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
+a single testcase,
 
 - `scs-0117-test-backup`,
 
 which verifies that a non-admin user can backup and restore volumes.
 
 To this end, the testcase uses the `/v3/{project_id}/backups` Block Storage API endpoint to create a volume and a backup of it and subsequently restores the backup on a new volume while verifying the success of each operation.
 
-The testcase can be run using the script
-[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).
-
 ## Manual tests
 
 Note that the automated tests don't verify the optional part of the standard: providing a separate storage backend for Cinder volume backups.