Implement best practices, refactor tests (#16)

* Implement best practices + mypy-ify
* Refactor filesystem tests
* Update python-multipart with vulnerability patch
* Refactor endpoint integration tests

Author: nolan1999

.github/workflows/code-checks.yaml

@@ -0,0 +1,60 @@
+name: Code checks
+
+on:
+  pull_request:
+    branches:
+      - main
+  push:
+    branches:
+      - main
+
+jobs:
+  static_code_checks:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install poetry
+      run: pipx install poetry
+    - uses: actions/setup-python@v5
+      with:
+        python-version: '3.11.10'
+        cache: 'poetry'
+    - name: Install dependencies
+      run: poetry install
+    - name: Ruff check
+      run: poetry run ruff check .
+    - name: Ruff format check
+      run: poetry run ruff format --check .
+    - name: Mypy check
+      run: poetry run mypy .
+  tests:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    steps:
+    - uses: actions/checkout@v4
+    - name: Install poetry
+      run: pipx install poetry
+    - uses: actions/setup-python@v5
+      with:
+        python-version: '3.11.10'
+        cache: 'poetry'
+    - name: Install dependencies
+      run: poetry install
+    - name: configure AWS
+      uses: aws-actions/configure-aws-credentials@v4
+      with:
+        aws-access-key-id: ${{ secrets.TESTS_AWS_ACCESS_KEY_ID }}
+        aws-secret-access-key: ${{ secrets.TESTS_AWS_SECRET_ACCESS_KEY }}
+        aws-region: eu-central-1
+    - name: Run tests
+      run: |
+        poetry run pytest -m "aws or not(aws)" --junitxml=pytest.xml --cov-report=term-missing --cov=api | tee pytest-coverage.txt
+        echo "test_exit_code=${PIPESTATUS[0]}" >> $GITHUB_ENV
+    - name: Coverage comment
+      uses: MishaKav/pytest-coverage-comment@main
+      with:
+        pytest-coverage-path: ./pytest-coverage.txt
+        junitxml-path: ./pytest.xml
+    - name: Fail on test failure
+      run: exit ${{ env.test_exit_code }}

.pre-commit-config.yaml

@@ -0,0 +1,21 @@
+repos:
+  - repo: local
+    hooks:
+      - id: ruff-check
+        name: ruff check
+        entry: poetry run ruff check --fix .
+        language: system
+        types: [python]
+      - id: ruff-format
+        name: ruff format
+        entry: poetry run ruff format --check .
+        language: system
+        types: [python]
+
+  - repo: local
+    hooks:
+      - id: mypy
+        name: mypy
+        entry: poetry run mypy
+        language: system
+        types: [python]

README.md

@@ -18,7 +18,25 @@ The authenticated users can:
 Behind the scenes, the API communicates with the [worker-facing API](https://github.com/ries-lab/DECODE_Cloud_WorkerAPI) of DECODE OpenCloud.
 When a user starts a job, it sends it to the [worker-facing API](https://github.com/ries-lab/DECODE_Cloud_WorkerAPI) and gets job updates from it.
 
-## Run
+## Development guide
+
+### Prepare the development environment
+We use [poetry](https://python-poetry.org/) for dependency tracking.
+See online guides on how to use it, but this setup should work:
+ - `conda create -n "3-11-10" python=3.11.10`
+ - `conda activate 3-11-10`
+ - `pip install pipx`
+ - `pipx install poetry`
+ - `poetry env use /path/to/conda/env/bin/python`
+ - `poetry install`
+Afterwards, when you need a new package, use `poetry add [--group dev] <package>` to add new dependencies.
+The `--group dev` option adds the dependency only for development (e.g., pre-commit hooks, pytest, ...).
+
+Install the pre-commit hooks: `pre-commit install`.
+These currently include ruff and mypy.
+
+### Run locally
+
 #### Define the environment variables
 Copy the `.env.example` file to a `.env` file at the root of the directory and define its fields appropriately:
  - Deployment settings:
@@ -46,13 +64,27 @@ Copy the `.env.example` file to a `.env` file at the root of the directory and d
    - `EMAIL_SENDER_SECRET_KEY`: API key secret to use the email sender. Can also be the ARN of an AWS SecretsManager secret.
 
 #### Start the user-facing API
-`uvicorn api.main:app --reload --port 8000`
+`poetry run serve`
 
 #### View the API documentation
 You can find it at `<API_URL>/docs` (if running locally, `<API_URL>=localhost:8000`).
 
+#### Docker
+Alternatively, you can build a Docker image.
+For this, run `poetry run docker-build`:
+This will create a Docker image named `api:<branch_name>`.  
+To run the Docker container, use `poetry run docker-serve`.  
+To stop and delete all containers for this package, use `poetry run docker-stop`.
+If you want to additionally remove all images for this package and prune dangling images, run `poetry run docker-cleanup`.
+
+### Tests
+Run them with `poetry run pytest`.
+
+Note that tests marked with `aws` are skipped by default, to avoid the need for an AWS setup.
+They are however ran in the GitHub Action.
+For this to work, they must have been ran once locally with an account with sufficient permissions (`poetry run pytest -m "aws"`), since for security reasons, the AWS account used on GitHub does not have permissions to create RDS instances.
 
-## Add/modify runnable applications
+### Add/modify runnable applications
 #### Dockerize the application
 See for example [DECODE](https://github.com/ries-lab/DECODE_Internal/blob/dockerfile_stable/Dockerfile) and [Comet](https://github.com/nolan1999/Comet/blob/docker/Python_interface/Dockerfile).  
 The image should:

api/core/aws.py

@@ -1,11 +1,13 @@
 import base64
-import hmac
 import hashlib
+import hmac
 
 
-def calculate_secret_hash(email, client_id, key):
-    key = bytes(key, "utf-8")
-    message = bytes(email + client_id, "utf-8")
+def calculate_secret_hash(email: str, client_id: str, key: str) -> str:
     return base64.b64encode(
-        hmac.new(key, message, digestmod=hashlib.sha256).digest()
+        hmac.new(
+            bytes(key, "utf-8"),
+            bytes(email + client_id, "utf-8"),
+            digestmod=hashlib.sha256,
+        ).digest()
     ).decode()