ci: drop pytest in favor of Sprocket test

.github/instructions/wdl.instructions.md

@@ -0,0 +1,13 @@
+---
+applyTo: "**/*.wdl"
+---
+# This file contains instructions for the WDL code style in the workflows directory.
+
+# WDL Development Instructions
+
+- Use the latest version of the WDL spec from https://github.com/openwdl/wdl/
+- Use the CONTRIBUTING.md guide in this repository for general coding style and best practices.
+- Use the best-practices.md guide in this repository for WDL-specific best practices.
+- Prefer the strings in template/common-parameter-meta.txt when writing parameter meta sections.
+- Use the templates in template/task-examples.wdl when writing new tasks.
+- All WDL code should be checked with Sprocket lint and formatted with Sprocket format before committing.

.github/workflows/build-and-test.yaml

@@ -7,6 +7,3 @@ jobs:
     call-build:
         uses: ./.github/workflows/docker-build.yaml
         secrets: inherit
-    call-test:
-        uses: ./.github/workflows/pytest.yaml
-        needs: call-build

.github/workflows/docker-build.yaml

@@ -70,47 +70,6 @@ jobs:
           tags: ${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.TOOL }}:local
           cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.TOOL }}:buildcache
           cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.TOOL }}:buildcache,mode=max
-      - name: container diff
-        run: |
-          docker pull ${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.TOOL }}:${{ env.TAG_TO_CHECK }} || true
-          if [ -z "$(docker images -q ${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.TOOL }}:${{ env.TAG_TO_CHECK }})" ]; then
-            echo "Prior image not found in registry, skipping container-diff"
-            exit 0
-          fi
-          curl -LO https://storage.googleapis.com/container-diff/latest/container-diff-linux-amd64 && \
-          chmod +x container-diff-linux-amd64 && \
-          ./container-diff-linux-amd64 diff --json daemon://${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.TOOL }}:${{ env.TAG_TO_CHECK }} daemon://${{ env.REGISTRY }}/${{ github.repository_owner }}/${{ env.TOOL }}:local --type=history --type=file --type=size | tee comparison.json
-          # check overall size result
-          changed=0
-          size_diff=$(jq '.[] | select(.DiffType == "Size") | if .Diff[0].Size1 == .Diff[0].Size2 then 1 else 0 end' comparison.json)
-          if [ $size_diff -eq 0 ]
-          then
-            echo "Size of the image has changed"
-            changed=1
-          fi
-          # check file diff result
-          additions=$(jq '.[] | select(.DiffType == "File") | if .Diff.Adds != null then .Diff.Adds[].Name else 0 end' comparison.json)
-          deletions=$(jq '.[] | select(.DiffType == "File") | if .Diff.Dels != null then .Diff.Dels[].Name else 0 end' comparison.json)
-          modifications=$(jq '.[] | select(.DiffType == "File") | if .Diff.Mods != null then .Diff.Mods[].Name else 0 end' comparison.json)
-          if [ "$additions" != "0" ]
-          then
-            echo "Files have been added"
-            echo $additions
-            changed=1
-          fi
-          if [ "$deletions" != "0" ]
-          then
-            echo "Files have been deleted"
-            echo $deletions
-            changed=1
-          fi
-          if [ "$modifications" != "0" ]
-          then
-            echo "Files have been modified"
-            echo $modifications
-            changed=1
-          fi
-          exit $changed
       - name: Run Snyk to check Docker image for vulnerabilities
         # Snyk can be used to break the build when it detects vulnerabilities.
         # In this case we want to upload the issues to GitHub Code Scanning

.github/workflows/sprocket-lint.yaml

@@ -3,12 +3,17 @@ name: Workflows Sprocket Lint
 on: [push]
 
 jobs:
-    sprocket_lint:
-        runs-on: ubuntu-latest
-        steps:
-        - uses: actions/checkout@v2
-        - name: Run sprocket
-          uses: stjude-rust-labs/sprocket-action@main
-          with:
-            lint: true
-            except: KnownRules
+  sprocket_lint:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Run 'sprocket lint'
+      uses: stjude-rust-labs/sprocket-action@main
+      with:
+        lint: true
+        except: KnownRules
+        ignore-patterns: methylation
+    - name: Run 'sprocket format'
+      uses: stjude-rust-labs/sprocket-action@feat/format
+      with:
+        action: format

.gitignore

@@ -1,3 +1,4 @@
+pytest/
 # Ignore common bioinformatics formats used in these workflows.
 # But only if they are in the root of this repo
 /*.fastq.gz
@@ -14,6 +15,7 @@
 **/womtool*
 *.jar
 runs/
+out/
 miniwdl_call_cache/
 miniwdl_singularity_cache/
 _LAST
@@ -29,7 +31,7 @@ _LAST
 results*/
 output*/
 
-# Ignore JSONs in the root (probably being used for input to sprocket run)
+# Ignore JSONs in the root (probably being used for input)
 /*.json
 
 # Ignore `sprocket doc` dir

CONTRIBUTING.md

@@ -31,6 +31,50 @@ Our pull request template has an extensive checklist that must be completed prio
 
 Note that the maintainers reserve the right to close any submission without review for any reason.
 
+## Expectations for WDL contributions
+
+We have some opinionated rules and guidelines we use while writing WDL for this repository. These include:
+
+- See `template/common-parameter-meta.txt` for common description strings.
+  - If applicable, use the same parameter name and help text as the underlying tool called by the task.
+- All requirement values are overridable at runtime. However, tasks should have easily configurable memory and disk space allocations.
+  - See the various tasks in the template directory for possible ways to allocate resources.
+    - Contributors can mix and match the available templates, copy and pasting subsections as appropriate.
+    - A task may contain both statically and dynamically allocated resources.
+- Multi-core tasks should *always* follow the conventions laid out in the `use_all_cores_task` example (see `template/task-examples.wdl`).
+  - This is catering to cloud users, who may be allocated a machine with more cores than are specified by the `ncpu` parameter.
+  - Note that future versions of WDL will likely cause a change to this convention.
+    - We plan to deprecate the `ncpu` param in favor of accessing the runtime section directly (`n_cores=~{task.runtime.cpu}`).
+- Output file names should *always* be determined with either the `outfile_name` parameter or the `prefix` parameter.
+  - `outfile_name` should be preferred if no downstream tasks/tools rely on the file name/extension.
+  - Tasks with multiple outputs should always use the `prefix` convention.
+- After the input sorting rules in `sprocket lint` have been applied, follow the below rules for further sorting.
+  - "sample" files come before "reference" files.
+  - If present, `use_all_cores` should be the last `Boolean` in its block.
+  - The `ncpu` parameter comes before inputs that allocate memory, which come before inputs that allocate disk space.
+    - This block of 2-3 inputs should come after all other inputs.
+- If a task uses multiple cores or is multithreaded, then at least 2 cpu should be specified.
+- Use the `as` keyword sparingly; only in the case of increased readability or to avoid name collisions.
+  - Prefer using `as` in the import block rather than at the task/workflow call level.
+  - When using `as` to rename an invalid URI, attempt to make as few changes to the filename as possible (i.e. try not to abbreviate).
+  - To disambiguate a task or workflow file from it's contents, you can respectively add the `_tasks` or `_wf` suffix in the import section.
+- Whenever possible, prefer a Docker image maintained by an external source (such as BioContainers) rather than creating your own image.
+- When adding a `Dockerfile` to this repository, follow the below conventions:
+  - Create a directory under the `docker/` directory and choose an appropriate name (likely shared with the underlying tool). The `Dockerfile` should be nested under this new directory. Then create a `package.json` alongside the `Dockerfile`. The `package.json` file is required to contain two JSON fields (`name` and `version`). It can optionally contain a `revision` field.
+  - Docker images should be versioned according to the following convention
+    - The `version` should be shared with whatever underlying tool is being used
+      - If no specific tool is named (e.g. the `util` image), default to SemVer. Ignore the next 3 bullet points.
+    - The revision should start with zero (`0`).
+      - If the Docker image gets updated, *without* updating the base tool's version, increment the number by one.
+      - If the Docker image gets updated, *including* updating the base tool's version, revert back to zero.
+- Any tasks which are deprecated should have a `deprecated: true` key in their `meta` section.
+  - Never include a `deprecated: false` key in any production tasks. All tasks are assumed to not be deprecated unless otherwise noted.
+  - In addition, there should be a `warning` key which starts with the text `**[DEPRECATED]**`.
+    - No other text or explanation is required after the above text, but it can be added for further context.
+  - These two conventions allow for a task's deprecated status to be communicated in multiple ways, ensuring no user misses the notice.
+  - Deprecated tasks should be placed at the end of their file.
+- While WDL allows embedded scripts in the `command` block sections, this repository requires scripts (e.g. R, Python) to be separate and placed in the `scripts` folder. The relevant Docker image build for your task should then include the script during the build so the task can access it. This separation of concerns improves the developer experience by improving syntax highlighting in the WDL document and enabling linting and formatting checks for the scripting languages.
+
 ## FAQs
 
 ### Can I use Artificial Intelligence (AI)?