docs(guides): improve Crypt4GH + proTES tutorial with a detailed use case

Author: Valentin Schneider-Lunitz

docs/guides/guide-admin/crypt4gh_to_protes.md

@@ -1,46 +1,44 @@
 # Setting up Crypt4GH encryption/decryption in Funnel
 
-This guide explains how to configure and deploy an environment that enables encryption and decryption of sensitive data files using TES/[Funnel](https://github.com/ohsu-comp-bio/funnel) with [proTES](https://github.com/elixir-cloud-aai/proTES) as a stable and scalable [GA4GH TES](https://github.com/ga4gh/task-execution-schemas) gateway.
+This guide explains how to configure and deploy an environment that enables collaborative research on sensitive genomic data. Data holders can securely provide encrypted data for analysis while researchers process it through TES/[Funnel](https://github.com/ohsu-comp-bio/funnel) and [proTES](https://github.com/elixir-cloud-aai/proTES), where automatic decryption occurs within secure containers without granting researchers direct access to the sensitive data. This setup leverages [GA4GH TES](https://github.com/ga4gh/task-execution-schemas) standards for scalable and secure task execution.
 
 ## Use Case
 
-Imagine you are a researcher who needs to analyse sensitive data in a cloud environment. You need to ensure:
+A data holder needs to provide sensitive genomic data for analysis to researchers in a cloud environment. The data must remain encrypted during storage and transfer, with decryption occurring only within a secure computational environment (container), without granting direct data access to the researcher.
 
-- **Your data is encrypted during transfer**: Your files are encrypted for transfer. Raw sensitive data remains located at your storage.
-- **Only authorized researcher can decrypt the data**: Data can only be decrypted with specific private keys. Data theft is useless without specific keys. 
-- **Automatic decryption**: Your setup does automatic decryption given `.c4gh` files and the correct private key.
-- **Secure collaboration**: Data exchange between collaborators is not restricted, as long as the correct key is available.
+1. The data holder encrypts sensitive data using Crypt4GH and stores them at a secure storage (e.g. S3 buckets).
+2. The researcher submits a GA4GH TES task to `proTES` for analysis of the encrypted data. 
+3. The installed `proTES middleware` automatically detects the encrypted data and decrypts them using Crypt4GH keys that are managed by `proTES`.
+4. The researcher's task command is executed on the decrypted data.
+5. The analysis results are stored at a dedicated storage accessible to the researcher
 
-This tutorial presents a solution where:
+`Note` all computational steps are done in a secure containerized environment.
 
-1. A data provider encrypts sensitive data using Crypt4GH before uploading them to storage.
-2. Encrypted data is sent to a `Task Execution Service (TES)` instance via `proTES` and a `proTES middleware` for processing.
-3. A researcher (recipient) can process these files in a secure containerized environment where automatic decryption happens using the `proTES middleware`.
+This approach allows collaborative research where sensitive data can be processed in cloud environments without provisioning data access to the researcher but instead utilizing a combination of `Crypt4GH` and `proTES` for data encryption, decryption, and analysis.
+Additionally, the researcher can repeat the analysis with adjusted parameters anytime without further action of the data holder.
 
-This approach allows collaborative research where sensitive data can be processed in cloud environments while maintaining strict access controls and encryption throughout the data lifecycle.
 
 ## Overview
 
 [Crypt4GH](https://crypt4gh.readthedocs.io/) is a standard for encrypting sensitive genomic data. This setup demonstrates:
 
-- Generating cryptographic key pairs for data exchange between parties (sender and recipient)
-- Encrypting files using the sender's private key and recipient's public key
+- Generating cryptographic key pairs for data exchange between parties (data holder and researcher)
+- Encrypting files using the data holder's private key and researcher's public key
 - Automatically decrypting `.c4gh` encrypted files during task execution using [protes-middleware-crypt4gh](https://github.com/elixir-cloud-aai/protes-middleware-crypt4gh)
 - Securely processing sensitive data in containerized environments
 
-**Security Note:** Private keys should be stored in secure locations and used only for decryption. Consider using signed URLs for transferring private keys to the TES instance. 
+**Security Note:** Private keys should be stored in secure locations and used only for encryption/decryption. Consider using signed URLs for transferring private keys to the TES instance. 
 
-**Goal of this tutorial:** You'll have a setup where you can submit encrypted files via task inputs, and they will be automatically decrypted and processed, ensuring that sensitive data remains protected.
+**Goal of this tutorial:** You'll have a setup which encrypts sensitive data, stores them in a secure storage, automatic detection of encrypted data triggers decryption followed by processing, ensuring that sensitive data remains protected.
 
 ## Setup
 
 The complete setup consists of three main tasks:
 
-1. **Key Generation**: Generate Crypt4GH key pairs for the sender and recipient parties (optional).
-2. **File Encryption**: Encrypt sensitive data using the generated keys.
-3. **File Decryption**: Decrypt and process encrypted files in a secure environment.
+1. **Key Generation**: Generate Crypt4GH key pairs for the data holder and researcher parties (optional).
+2. **File Encryption**: Encrypt sensitive data using the Crypt4GH keys.
+3. **File Decryption**: automatic detection of encrypted data, their decryption and processing in a secure computing environment.
 
-All keys are generated inside containers and exported to configured storage via TES outputs. The encrypted files (with `.c4gh` extension) are automatically decrypted during task execution using the proTES middleware.
 
 ## Prerequisites
 
@@ -52,7 +50,7 @@ Before starting, ensure you have:
   - ProTES deployment VM
 - [Docker](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) installed on all VMs
 - Network connectivity between all VMs
-- Sufficient storage space for encrypted/decrypted files
+- Sufficient storage space for encrypted/decrypted files and results. 
 
 ## Installation and Configuration
 
@@ -200,49 +198,49 @@ The following examples demonstrate the complete encryption/decryption workflow u
 
 ### Task 1: Generate Crypt4GH Key Pairs
 
-This task generates cryptographic key pairs for both the sender and recipient parties. This step is independent of the following steps and may have happened a while ago. Your private keys may already be in a secure place. If you have crypt4gh keys, feel free to skip this step.
+This task generates cryptographic key pairs for both the data holder and researcher. This step is independent of the following steps and may have happened a while ago. Your private keys may already be in a secure place. If you have crypt4gh keys, feel free to skip this step.
 
 Create a file named `task1_keygen.json`:
 
 ```json
 {
   "name": "Generate crypt4gh key pairs",
-  "description": "Generate sender and recipient key pairs locally in container",
+  "description": "Generate data holder and researcher key pairs locally in container",
   "inputs": [],
   "outputs": [
     {
-      "name": "sender_sk",
-      "description": "Sender secret key",
-      "url": "file:///tmp/funnel-storage/keys/sender/sender.sec",
-      "path": "/outputs/keys/sender/sender.sec",
+      "name": "data_holder_sk",
+      "description": "Data holder secret key",
+      "url": "file:///tmp/funnel-storage/keys/data_holder/data_holder.sec",
+      "path": "/outputs/keys/data_holder/data_holder.sec",
       "type": "FILE"
     },
     {
-      "name": "sender_pk",
-      "description": "Sender public key",
-      "url": "file:///tmp/funnel-storage/keys/sender/sender.pub",
-      "path": "/outputs/keys/sender/sender.pub",
+      "name": "data_holder_pk",
+      "description": "data_holder public key",
+      "url": "file:///tmp/funnel-storage/keys/data_holder/data_holder.pub",
+      "path": "/outputs/keys/data_holder/data_holder.pub",
       "type": "FILE"
     },
     {
-      "name": "recipient_sk",
-      "description": "Recipient secret key",
-      "url": "file:///tmp/funnel-storage/keys/recipient/recipient.sec",
-      "path": "/outputs/keys/recipient/recipient.sec",
+      "name": "researcher_sk",
+      "description": "researcher secret key",
+      "url": "file:///tmp/funnel-storage/keys/researcher/researcher.sec",
+      "path": "/outputs/keys/researcher/researcher.sec",
       "type": "FILE"
     },
     {
-      "name": "recipient_pk",
-      "description": "Recipient public key",
-      "url": "file:///tmp/funnel-storage/keys/recipient/recipient.pub",
-      "path": "/outputs/keys/recipient/recipient.pub",
+      "name": "researcher_pk",
+      "description": "researcher public key",
+      "url": "file:///tmp/funnel-storage/keys/researcher/researcher.pub",
+      "path": "/outputs/keys/researcher/researcher.pub",
       "type": "FILE"
     },
     {
-      "name": "recipient_pk_copy",
-      "description": "Copy of recipient public key",
-      "url": "file:///tmp/funnel-storage/keys/sender/recipient.pub",
-      "path": "/outputs/keys/sender/recipient.pub",
+      "name": "researcher_pk_copy",
+      "description": "Copy of researcher public key",
+      "url": "file:///tmp/funnel-storage/keys/data_holder/researcher.pub",
+      "path": "/outputs/keys/data_holder/researcher.pub",
       "type": "FILE"
     }
   ],
@@ -252,7 +250,7 @@ Create a file named `task1_keygen.json`:
       "command": [
         "/bin/bash",
         "-c",
-        "crypt4gh-keygen --sk /outputs/keys/sender/sender.sec --pk /outputs/keys/sender/sender.pub -f --nocrypt && crypt4gh-keygen --sk /outputs/keys/recipient/recipient.sec --pk /outputs/keys/recipient/recipient.pub -f --nocrypt && cp /outputs/keys/recipient/recipient.pub /outputs/keys/sender/recipient.pub"
+        "crypt4gh-keygen --sk /outputs/keys/data_holder/data_holder.sec --pk /outputs/keys/data_holder/data_holder.pub -f --nocrypt && crypt4gh-keygen --sk /outputs/keys/researcher/researcher.sec --pk /outputs/keys/researcher/researcher.pub -f --nocrypt && cp /outputs/keys/researcher/researcher.pub /outputs/keys/data_holder/researcher.pub"
       ],
       "workdir": "/tmp"
     }
@@ -267,32 +265,32 @@ Create a file named `task1_keygen.json`:
 
 **Key Details:**
 
-- Generates two key pairs: one for the sender and one for the recipient
+- Generates two key pairs: one for the data holder and one for the researcher
 - Keys are generated without encryption (`--nocrypt`) for demonstration purposes
-- The recipient's public key is copied to the sender's directory for use in encryption
+- The researcher's public key is copied to the data holder's directory for use in encryption
 - All keys are exported to local storage via TES outputs
 
 ### Task 2: Encrypt a File
 
-This task downloads a file, encrypts it using Crypt4GH, and stores both the encrypted file and metadata. Create a file named `task2_encrypt_file.json`:
+This task retrieves a file, encrypts it using Crypt4GH, and stores both the encrypted file and metadata. Create a file named `task2_encrypt_file.json`:
 
 ```json
 {
   "name": "Encrypt file with crypt4gh",
-  "description": "Download a file, record its size, and encrypt it locally using sender and recipient keys",
+  "description": "Retrieve a file, record its size, and encrypt it using data holder and researcher keys",
   "inputs": [
     {
-      "name": "sender_sk",
-      "description": "Sender secret key",
-      "url": "file:///tmp/funnel-storage/keys/sender/sender.sec",
-      "path": "/inputs/keys/sender/sender.sec",
+      "name": "data_holder_sk",
+      "description": "data_holder secret key",
+      "url": "file:///tmp/funnel-storage/keys/data_holder/data_holder.sec",
+      "path": "/inputs/keys/data_holder/data_holder.sec",
       "type": "FILE"
     },
     {
-      "name": "recipient_pk",
-      "description": "Recipient public key",
-      "url": "file:///tmp/funnel-storage/keys/recipient/recipient.pub",
-      "path": "/inputs/keys/recipient/recipient.pub",
+      "name": "researcher_pk",
+      "description": "researcher public key",
+      "url": "file:///tmp/funnel-storage/keys/researcher/researcher.pub",
+      "path": "/inputs/keys/researcher/researcher.pub",
       "type": "FILE"
     }
   ],
@@ -318,7 +316,7 @@ This task downloads a file, encrypts it using Crypt4GH, and stores both the encr
       "command": [
         "/bin/bash",
         "-c",
-        "curl -L -o /tmp/file.png http://britishfamily.co.uk/wp-content/uploads/2015/02/MADE_IN_BRITAIN_web_300x300.png && stat -c %s /tmp/file.png > /outputs/raw/united_kingdom_logo_size.txt && crypt4gh encrypt --sk /inputs/keys/sender/sender.sec --recipient_pk /inputs/keys/recipient/recipient.pub < /outputs/raw/united_kingdom_logo_size.txt > /outputs/encrypted/united_kingdom_logo_size.txt.c4gh"
+        "curl -L -o /tmp/file.png http://britishfamily.co.uk/wp-content/uploads/2015/02/MADE_IN_BRITAIN_web_300x300.png && stat -c %s /tmp/file.png > /outputs/raw/united_kingdom_logo_size.txt && crypt4gh encrypt --sk /inputs/keys/data_holder/data_holder.sec --recipient_pk /inputs/keys/researcher/researcher.pub < /outputs/raw/united_kingdom_logo_size.txt > /outputs/encrypted/united_kingdom_logo_size.txt.c4gh"
       ],
       "workdir": "/tmp"
     }
@@ -333,15 +331,15 @@ This task downloads a file, encrypts it using Crypt4GH, and stores both the encr
 
 **Key Details:**
 
-- Takes the sender's private key and recipient's public key as inputs
+- Takes the data holder's private key and researcher's public key as inputs
 - Downloads a sample file from a URL
 - Records the original file size for verification
 - Encrypts the file using Crypt4GH, producing a `.c4gh` encrypted file
 - Stores both the encrypted file and size metadata
 
 ### Task 3: Decrypt and Process File
 
-This task decrypts the encrypted file using the recipient's private key and processes it.
+This task decrypts the encrypted file using the researcher's private key and processes it.
 
 **Note:** The different paths indicate isolated storage paths that do not necessarily see each other. For example, distinct S3 buckets.
 
@@ -350,7 +348,7 @@ Create a file named `task3_decrypt_and_write_size.json`:
 ```json
 {
   "name": "Decrypt crypt4gh file",
-  "description": "Decrypt an encrypted file using recipient key locally",
+  "description": "Decrypt an encrypted file using researcher key locally",
   "volumes": ["/outputs/test"],
   "inputs": [
     {
@@ -361,10 +359,10 @@ Create a file named `task3_decrypt_and_write_size.json`:
       "type": "FILE"
     },
     {
-      "name": "recipient_sk",
-      "description": "Recipient secret key",
-      "url": "file:///tmp/funnel-storage/keys/recipient/recipient.sec",
-      "path": "/inputs/keys/recipient/recipient.sec",
+      "name": "researcher_sk",
+      "description": "researcher secret key",
+      "url": "file:///tmp/funnel-storage/keys/researcher/researcher.sec",
+      "path": "/inputs/keys/researcher/researcher.sec",
       "type": "FILE"
     }
   ],
@@ -398,7 +396,7 @@ Create a file named `task3_decrypt_and_write_size.json`:
 
 **Key Details:**
 
-- Takes the encrypted `.c4gh` file and recipient's private key as inputs
+- Takes the encrypted `.c4gh` file and researcher's private key as inputs
 - The proTES middleware automatically decrypts the file during task execution
 - Computes an MD5 checksum of the decrypted data for verification
 - Stores the checksum in the output directory
@@ -432,15 +430,6 @@ Each task submission returns a task ID that you can use to monitor progress:
 curl http://localhost:8080/ga4gh/tes/v1/tasks/<task-id>
 ```
 
-## How It Works
-
-1. **Task Submission**: Tasks are submitted to proTES via the GA4GH TES API
-2. **Task Distribution**: ProTES distributes tasks to available Funnel TES endpoints
-3. **Automatic Decryption**: The Crypt4GH middleware automatically detects `.c4gh` files and injects a decryption step
-4. **Container Execution**: Funnel worker nodes execute tasks in isolated containers
-5. **Result Storage**: All results are stored in the configured local storage directory (`/tmp/funnel-storage`)
-6. **Data Persistence**: Task and scheduler metadata is stored in the Funnel BoltDB database
-
 ## Troubleshooting
 
 ### Common Issues