Merge pull request #103 from elixir-europe/update-README

Updated readme

Author: bedroesb

MARS_overview.svg

@@ -2,9 +2,9 @@
 
 ## Introduction
 
-MARS is a data brokering initiative for submitting multi-omics life sciences studies to multiple specialized repositories.
-It is setup to be modular and enables data producers and multiple data repositories to exchange information seamlessly using the same standardized ISA-JSON format. Unlike a centralized platform, MARS operates as a common framework, allowing for decentralized data submissions while ensuring consistent interpretation and validation of ISA-JSON containing metadata across various repositories.
-The initiative ensures mutual understanding and accurate interpretation of the data, preserving the important links between multi-omics data generated from the same biological source.
+MARS is a data brokering initiative designed to facilitate the submission of multi-omics life sciences studies to [multiple specialized repositories](#isa-json-support-by-repositories). Built as a modular system, MARS enables seamless data exchange between data producers and repositories using the standardized ISA-JSON format.
+
+Unlike centralized platforms, MARS functions as a common framework for decentralized data submissions while ensuring consistent interpretation and validation of ISA-JSON metadata across various repositories. This approach preserves important links between multi-omics datasets derived from the same biological source, ensuring mutual understanding and accurate data interpretation.
 
 ## Stakeholders
 
@@ -13,7 +13,6 @@ MARS is comprised of multiple stakeholders: the end-user, the platform that gene
 
 ## Components
 
-
 ![MARS overview](/MARS_overview.svg)
 
 
@@ -26,81 +25,89 @@ We use [ISA-JSON](https://isatools.readthedocs.io/en/latest/isamodel.html) to st
 - **Interoperability**: Since ISA-JSON follows a standard format, it facilitates interoperability between different software tools and platforms that support the ISA standard. 
 - **Community Adoption**: Widely adopted within the life sciences research community for metadata standardization.
 
-It is produced by the [ISA-JSON producing platforms](/stakeholders.md#isa-json-producing-platforms) and will be the metadata input of the Data broker platform, see below.
+ISA-JSON is generated by [ISA-JSON producing platforms](/stakeholders.md#isa-json-producing-platforms) and serves as the metadata input for the Data Broker platform, as outlined below.
 
 ### Data broker platform
 
-A platform operated by the [Data broker](/stakeholders.md#data-broker) should:
+A platform operated by the [Data broker](/stakeholders.md#data-broker) should:  
 
-* Accept an ISA-JSON as input and submit it to the repositories without any loss of information.
-* Extend the ISA-JSON with additional information provided by the target repositories. For example, the accessions assigned to the submitted objects.
-* Process reporting errors
-* Enable secure credential management and the possibility to set brokering accounts.
-* Supports data transfer through various protocols (e.g. FTP). This would include the verification of the checksums associated to the data files. 
-* Allows the Data broker to set up a brokering account or the end-user a personal account.
-* To ensure that the brokering account is not used beyond the purposes defined by the producer. In other words, not to modify or submit in the name of the producer without their consent.
+- Accept ISA-JSON as input and submit it to target repositories without any loss of information.  
+- Extend the ISA-JSON with additional information from the repositories, such as accession numbers assigned to submitted objects.  
+- Handle error reporting efficiently.  
+- Maintain an active submission process throughout its duration (up to multiple days), including waiting for repository-side validation steps to complete.  
+- Enable secure credential management.  
+- Support data transfer via various protocols (e.g., FTP), ensuring checksum verification for data integrity.  
+- Allow the Data broker to set up a brokering account or enable end-users to create personal accounts.  
 
-> **To be discussed**:
-> Handling brokering accounts: who creates it? Same for all repositories? Who handles requests to broker data? Can it be done automatically? Are all namespaces for submissions would be shared? Check: https://ena-docs.readthedocs.io/en/latest/faq/data_brokering.html
+Examples of Data broker platforms include ARC, Galaxy, and others.  
 
 
-Examples of Data broker platforms are ARC, Galaxy, ...
-
 ### MARS-CLI
 
-This command line tool (CLI) will be used by the Data broker platform and will perform the actual submission of the ISA-JSON to the repositories. Based on receipts repositories give back as response, the ISA-JSON will be updated with accession numbers. The application is build as a Python library which can be integrated in a web application, ARC, Galaxy and others. Source code and documentation can be found in the [mars-cli folder](/mars-cli/) in this repo.
+MARS-CLI is a command-line tool (CLI) used by the Data Broker platform to handle the submission of ISA-JSON metadata to multiple repositories. It automates the submission process, updates ISA-JSON with accession numbers based on repository responses, and ensures smooth data integration.  
 
-The main steps of MARS-CLI are:
+Built as a Python library, MARS-CLI can be integrated into web applications, ARC, Galaxy, and other platforms. The source code and documentation are available in the [mars-cli folder](/mars-cli/) in this repository.  
 
-1. **Ingesting and validating the ISA-JSON**: Compared to the vanilla ISA specification, the MARS-CLI has certain fields required (for example `target repository` as comment) in order to function properly. Upon ISA-JSON ingestion the information gets loaded in memory and validated at the same time using Pydantic.
+#### Main Steps of MARS-CLI  
 
-2. **Identifying the target repositories**: The order of submission can be depended on the target repositories specified in the ISA-JSON.
+1. **Ingesting and Validating the ISA-JSON**  
+MARS-CLI requires certain mandatory fields beyond the standard ISA specification (e.g., `target_repository` as a comment). Upon ingestion, the ISA-JSON is loaded into memory and validated using Pydantic to ensure it meets these requirements.  
 
-3. **Registering samples in BioSamples**: Submitting an ISA-JSON to a newly developed API at BioSamples. The BioSamples accession will be reused by the other repositories and thus needs to be done first.
-After a successful submission, BioSamples sends back a receipt containing BioSamples accession numbers for `Source` and `Sample` as `Source characteristics` and `Sample characteristics`, respectively. 
+2. **Identifying the Target Repositories**  
+The order of submission depends on the repositories specified in the ISA-JSON. MARS-CLI determines the correct sequence for submitting metadata and data.  
 
-    > The source code for the ISA-JSON API for BioSamples can be found in the [repository-services repo](/repository-services/isajson-biosamples/) and can be used for testing
+3. **Registering Samples in BioSamples**  
+MARS-CLI first submits the ISA-JSON to BioSamples via a newly developed API. BioSamples accessions are crucial since other repositories reuse them.  
 
-4. **Filtering the ISA-JSON**: The ISA-JSON (updated with BioSamples IDs) has to be filtered for every target repository so it only contains information relevant for that repo. This will be facilitated by the `target repository` attribute present in the ISA-JSON assays.
+- After submission, BioSamples returns a receipt containing accessions for `Source` and `Sample`, mapped to `Source characteristics` and `Sample characteristics`, respectively.  
+- Source Code: The BioSamples ISA-JSON API can be found in the [repository-services repo](/repository-services/isajson-biosamples/) and is available for testing.  
 
-5. **Submitting data to target repositories**: Since some repositories have the requirement that the actual data is already present in their upload space, the MARS-CLI could optionally take care of the data submission. This would guarantee the presence of the the data upon the metadata (ISA-JSON) submission and a checksum check.
+4. **Filtering the ISA-JSON**  
+Once updated with BioSamples accessions, the ISA-JSON is filtered for each target repository. This ensures that only relevant metadata is submitted to each repository. The filtering is based on the `target repository` attribute present in the ISA-JSON assays.  
 
-6. **Registering ISA-JSON at the target repositories**: Sending the filtered ISA-JSON to the endpoints of the repositories who accept ISA-JSON, for example ENA.
+5. **Submitting Data to Target Repositories**  
+MARS-CLI uploads data using FTP. Some repositories require that data files are present in their upload space before metadata submission. This step ensures that data availability and checksum validation are completed before sending metadata.  
 
-    > The source code for the ISA-JSON API for ENA can be found in the [repository-services repo](/repository-services/isajson-ena/) and can be used for testing
+6. **Registering ISA-JSON at the Target Repositories**  
+The filtered ISA-JSON is submitted to repositories that accept ISA-JSON metadata, such as ENA. The MARS project helps out with the adaptation of ISA-JSON by the repositories using the so called adapters. 
 
-7. **Processing the receipts and errors from the repositories**: After a successful submission, each repository sends back a receipt in a standard format defined for MARS (see [repository-api info](/repository-services/repository-api.md)). The receipt contains the path of the objects in the ISA-JSON for which an accession number has been generated, and the related accession number. 
+- Source Code: The ISA-JSON API for ENA is available in the [repository-services repo](/repository-services/isajson-ena/) and can be used for testing.  
 
-8. **Updating the BioSamples External References**: Data broker uses the BioSamples accession numbers to download the submitted BioSamples JSON and extend the `External References` schema by adding the accession numbers provided by the other target archives.
+7. **Processing Repository Receipts and Errors**  
+After submission, each repository returns a receipt in a standardized format defined for MARS (see [repository-api info](/repository-services/repository-api.md)).  
 
-9. **Dumping back an updated ISA-JSON with repositories' information**: Based on the information in the receipts, the ISA-JSON is populated with accession numbers linked to the submitted metadata objects and can be given back as output.
+- The receipt includes the paths of objects within the ISA-JSON and their assigned accession numbers.  
+- Errors encountered during submission are processed and reported accordingly.  
 
+8. **Updating BioSamples External References**  
+The Data Broker retrieves the BioSamples JSONs of the submitted samples using its accession numbers and updates the `External References` schema by adding the accession numbers assigned by other target repositories.  
 
-#### Credential management
+9. **Generating an Updated ISA-JSON with Repository Information**  
+Based on the information in repository receipts, the ISA-JSON is updated with accession numbers linked to submitted metadata objects. This final, enriched ISA-JSON can then be provided as output.  
 
-MARS-CLI is not responsible for storing and managing credentials, used for submission to the target repositories. Therefor, credentials should be managed by the [Data broker platform](#data-broker-platform).
+#### Credential management
 
+MARS-CLI comes with the functionality to interact with your device's keychain backend in order to fetch the necessary credentials. This allows the user to set credentials in a safe way. 
 
 #### Data submission
 
-MARS-CLI is not to be used as a platform to host data and will not store the data after submission to the target repository. This should be handled by the [Data broker platform](#data-broker-platform). The ISA-JSON provided to the application will be updated and stored in the BioSamples repository as an External Reference, but is otherwise considered as ephemeral.
+MARS-CLI is not a platform to host data and will not store the data after submission to the target repository. This should be handled by the [Data broker platform](#data-broker-platform) where MARS-CLI is installed. The ISA-JSON provided to the application will be updated and stored in the BioSamples repository as an External Reference, but is otherwise considered as ephemeral.
 
-=> Data submission could be added to MARS-CLI?
 
 ### ISA-JSON support by repositories
 
-ISA-JSON API services are being developed and deployed by the repositories that are part of the MARS initiative. This includes programmatic submission, the ingestion of ISA-JSON in order to register the metadata objects and the creation of a receipt according to the MARS [repository-api](/repository-services/repository-api.md) standard.
+ISA-JSON API services, also known as adapters, are being developed and deployed by the repositories that are part of the MARS initiative. This includes programmatic submission, the ingestion of ISA-JSON in order to register the metadata objects and the creation of a receipt according to the MARS [repository-api](/repository-services/repository-api.md) standard.
 
 Track the status of each repository here:
 
-| Repository | Programmatic submission | Development status | Deployed? | Source code |
+| Repository | Programmatic submission | Development status | Deployed | Source code |
 |---|---|---|---|---|
-| [BioSamples](https://www.ebi.ac.uk/biosamples/) | yes | Ready to be tested | no | [GitHub](repository-services/isajson-biosamples) |
-| [ENA](https://www.ebi.ac.uk/ena/browser/) | yes | Ready to be tested | no | [GitHub](repository-services/isajson-ena) |
-| [MetaboLights](https://www.ebi.ac.uk/metabolights/) | NA | Not started | no |  |
+| [BioSamples](https://www.ebi.ac.uk/biosamples/) | yes | PoC being improved | no | [GitHub](repository-services/isajson-biosamples) |
+| [ENA](https://www.ebi.ac.uk/ena/browser/) | yes | PoC being improved | no | [GitHub](repository-services/isajson-ena) |
+| [MetaboLights](https://www.ebi.ac.uk/metabolights/) | yes | Proof of concept | no |  |
 | [BioStudies/ArrayExpress](https://www.ebi.ac.uk/biostudies/arrayexpress) | yes, in dev | Not started | no |  |
 | [e!DAL-PGP](https://edal-pgp.ipk-gatersleben.de/) | NA | Not started | no |  |
-| Your repository here? Join MARS!  |
+| Your repository here? Join MARS!  |  |  |  |  | 
 
 ## File structure in this repo
 
@@ -126,3 +133,7 @@ Track the status of each repository here:
     - **repository-api.md**: Describing the receipt standard for repository APIs to follow.
 - **test-data**: Test data to be used in a submission.
 - **README.md**: This file
+
+## Acknowledgements
+
+This project was initiated during the ELIXIR Europe BioHackathon 2022 and has since received continued support through subsequent ELIXIR Hackathons and the ELIXIR Data Platform WP2.