Skip to content

Commit 0afa6a2

Browse files
wihetooesteban
wiheto
authored and
oesteban
committed
draft 1 of documentation
1 parent 5eb2f3a commit 0afa6a2

4 files changed

Lines changed: 407 additions & 0 deletions

File tree

‎docs/contributing_tutorials/adding_a_new_template.rst‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
Creating a new template space
2+
###############################################
3+
4+
Who is this tutorial for
5+
============================
6+
7+
First, this is intended for those wishing to add templates to TemplateFlow.
8+
Second, this is for people who want to add a template directory that does not already exists.
9+
TemplateFlow consists of multiple templates sorted by the space the template is in.
10+
This tutorial tells you how to add a new template space.
11+
12+
If the space for you template already exists, then you should follow the tutorial: :ref: `uploading_to_existing_templates`.
13+
14+
This tutorial assumes you have done all the steps in the preceding tutorial: :ref: `prerequisites_to_contributing`.
15+
16+
__Note__ at present, this tutorial will require writing access to the TemplateFlow repo.
17+
If you do not have access here, it may be best to open up an issue asking for a template space to be created.
18+
19+
Big picture
20+
===============
21+
22+
Image files (e.g. nifti files) are hosted on OSF and linked to Github.
23+
24+
You will need to publish the directory to OSF and create a link to it to Github.
25+
This link can be done with DataLad, DataLad tracks a dataset and only downloads files when they are needed.
26+
You can read more about DataLad [`Link <FIXLINK>`_].
27+
28+
Step 1: create a new dataset
29+
=============================
30+
31+
First make sure you are in your local templateflow directory.
32+
If you do not have a local templateflow copy, run:
33+
34+
::.
35+
git clone https://github.com/templateflow/templateflow/
36+
cd templateflow
37+
38+
Now set the variable ``TEMPLATENAME`` to whatever your template will be called.
39+
Also set your Github username to the variable ``GITHUBUSERNAME``.
40+
Finally, write a description of your template.
41+
42+
::.
43+
TEMPALTENAME='tpl-test'
44+
GITHUBUSERNAME='yourusername'
45+
TEMPALTEDESCRIPTION="This is a test template"
46+
GITHUBREPO='templateflow'
47+
48+
At the moment, always keep templateflow as GITHUBREPO, this may be changed in the future
49+
50+
With these variables set you can then run the following code with no modifications:
51+
52+
::.
53+
datalad create -d . -D "$TEMPALTEDESCRIPTION" $TEMPALTENAME
54+
cd $TEMPALTENAME
55+
datalad create-sibling-github --github-organization $GITHUBREPO --github-login $GITHUBUSERNAME --access-protocol ssh $TEMPALTENAME
56+
cd ..
57+
sed -i -e "s/url = .\/$TEMPALTENAME/url = https:\/\/github.com\/$GITHUBREPO\/$TEMPLATENAME/g" .gitmodules
58+
datalad save -m "set the github repo url for new template ``$TEMPLATENAME``"
59+
datalad publish
60+
61+
You will be asked to enter your github username and password while running this code.
62+
63+
Explanation of above code
64+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
65+
66+
After running, this code will create an empty datalad folder called ``tpl-test`` (or whatever TEMPLATENAME is set to).
67+
It will then change into that directory, upload the template to github.
68+
It will then return to the templateflow directory.
69+
70+
The 5th line in the above code edits the file .gitmodules to replace:
71+
72+
::.
73+
[submodule "tpl-test"]
74+
path = tpl-test
75+
url = ./tpl-test
76+
77+
with:
78+
79+
::.
80+
[submodule "tpl-test"]
81+
path = tpl-test
82+
url = https://github.com/templateflow/tpl-test
83+
84+
I.e. it adds a full url link to the.
85+
The final two lines upload this change.
86+
87+
Step 2: Add a template_description.json
88+
========================================
89+
90+
Within this directory we place a template_description.json which is needed in all templates.
91+
The json file contains the following:
92+
93+
::.
94+
{
95+
"Authors": ["Noone"],
96+
"Acknowledgements": "Curated and added to TemplateFlow by Thompson WH",
97+
"BIDSVersion": "1.1.0",
98+
"HowToAcknowledge": "You should not use this template",
99+
"Identifier": "test",
100+
"License": "See LICENSE file",
101+
"Name": "A test template to for testing.",
102+
"RRID": "SCR_002823",
103+
"ReferencesAndLinks": [
104+
"Link to article if there is one"],
105+
"TemplateFlowVersion": "1.0.0",
106+
"res": {
107+
"01": {
108+
"origin": [-91.0, -126.0, -72.0],
109+
"shape": [182, 218, 182],
110+
"zooms": [1.0, 1.0, 1.0]
111+
}
112+
}
113+
}
114+
115+
Add all the necessary information into the .json file.
116+
Then open a pull request on github to submit this information.

‎docs/contributing_tutorials/prerequisites_to_contributing.rst‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
2+
Prerequisites to contributing templates to TemplateFlow
3+
############################################################
4+
5+
Who is this tutorial for?
6+
=================================
7+
8+
Have a template that you would like to see on TemplateFlow? Great!
9+
Contributions are welcome.
10+
This document goes through some of the prerequisites needed to submit a template.
11+
Once you have these prerequisites achieved.
12+
13+
Are you allowed to share the template?
14+
==========================================
15+
16+
Templates have a licence which specifies the terms that they can be shared.
17+
TemplateFlow can only include templates that allow for redistribution.
18+
It is okay if the template requires attribution, but you need to make sure to add the attribution information.
19+
20+
What type of contribution are you making?
21+
============================================
22+
23+
There are three different types of contributions you can make to TemplateFlow.
24+
25+
**A new template space**.
26+
This contribution involves adding a new space that does not currently exist.
27+
Let us say you have made a new pediatric space that you transform your images to; this would be a new template space.
28+
All the different MNI templates are each considered their own template space.
29+
30+
**Nifti images within an existing template space**.
31+
This contribution involves adding to a template space that currently exists.
32+
An example of this would be adding a nifti file that is an atlas.
33+
You need to know which template space your atlas is in (Note: there are multiple MNI spaces).
34+
35+
**Meta information**.
36+
This contribution involves additional information about existing templates.
37+
These will generally be in .json or .tsv files.
38+
There are also transform files which help translate between templates.
39+
40+
There are tutorials for each of these three different types of contributions.
41+
42+
Prerequisites for uploading nifti files
43+
=====================================================
44+
45+
Aside from typical requirements (python3, git, pip), some prerequisites need addressing before you can upload a template (i.e. any image file).
46+
47+
1. Getting access to the OSF repository. Create an issue stating you would like to access to the OSF repo. You need an account at: `osf.io <https://osf.io>`_
48+
2. Download the OSF client: ``pip install osfclient``.
49+
3. Install `DataLad <https://www.datalad.org>`_. See the `installation page <https://www.datalad.org/get_DataLad.html>`_ for more details.
50+
4. You also need TemplateFlows DataLad/osf tools. To install these type in a terminal: ``pip install git+https://github.com/TemplateFlow/DataLad-osf``.
51+
52+
After installing these tools, you are ready to upload a template.
53+
54+
If you only plan to contribute metadata (e.g. json files and tsv files), then these three steps are not needed.
55+
56+
Initializing the TemplateFlow OSF directory
57+
==================================================
58+
59+
Once you have the prerequisites set up, you can initialize the OSF directory onto your computer.
60+
61+
In a new directory type:
62+
63+
> osf init
64+
65+
This will prompt you for your username and TemplateFlow project number.
66+
This project number is ``ue5gx``.
67+
68+
You can check that your directory has been correctly set up by typing:
69+
70+
> osf ls
71+
72+
You should see the contents of the OSF project folder appear in the console.

‎docs/contributing_tutorials/uploading_to_existing_template.rst‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
Uploading files to an existing template space
2+
###############################################
3+
4+
Who is this tutorial for
5+
============================
6+
7+
First, this is intended for those wishing to upload templates to TemplateFlow.
8+
Second, this is for people who want to add to a template directory that already exists.
9+
TemplateFlow consists of multiple templates sorted by the space the template is in.
10+
11+
If the space for you template does not exist, then you should follow this tutorial: :ref: `adding_a_new_template`.
12+
13+
This tutorial assumes you have done all the steps in the preceding tutorial: :ref: `prerequisites_to_contributing`.
14+
15+
Big picture
16+
===============
17+
18+
Image files (e.g. nifti files) are hosted on OSF.
19+
All other information (e.g. tsv, json files) will be hosted on github as regular files.
20+
21+
When you use TemplateFlow, the images are only downloaded when they are needed.
22+
This can be done with DataLad, DataLad tracks a dataset and only downloads files when they are needed.
23+
You can read more about DataLad [`Link <FIXLINK>`_].
24+
25+
Thus, when uploading to an existing template, there are two different steps:
26+
27+
1. You need to place them on the OSF server.
28+
2. you need to tell DataLad about the new items so the Github repo can track the new files.
29+
30+
Step 1: Placing the template on the OSF server
31+
================================================
32+
33+
Go to the TemplateFlow directory you previously initialized.
34+
Place the file you want to upload in directory and give them appropriate .
35+
36+
Let us say you want to upload:
37+
`tpl-test/tpl-test_atlas-test1_dseg.nii.gz`, `tpl-test/tpl-test_atlas-test2_dseg.nii.gz`.
38+
39+
>>> export OSF_PASSWORD='<some-password>'
40+
>>> osf upload tpl-test_atlas-test1_dseg.nii.gz tpl-test_atlas-test1_dseg.nii.gz
41+
42+
If you are overwriting an existing image, use the -f flag.
43+
44+
If you want to upload multiple images (let us say we have multiple atlases), use a for bash loop:
45+
46+
>>> ls tpl-test/*_atlas-test*_dseg.nii.gz | parallel -j8 'osf upload {} {}'
47+
48+
And this will iteratively upload all instances of the atlas to OSF.
49+
50+
Step 2: Telling Datalad where the files are
51+
================================================
52+
53+
Github is using Datalad to track the files.
54+
You need to tell Datalad where the new files are.
55+
56+
To do this, run the following python script, changing the tqo required variables:
57+
58+
.. code-block:: python
59+
60+
from pathlib import Path
61+
from datalad_osf.utils import url_from_key, json_from_url,
62+
get_osf_recursive, osf_to_csv
63+
64+
## VARIABLES THAT NEED TO BE CHANGED
65+
66+
# THe template you are adding too
67+
subset = 'tpl-test'
68+
# A string that identifies your new nifit files
69+
# Leave as blank if you want all files added
70+
strinfile = 'atlas-test'
71+
72+
## OPTIONAL VARIABLES
73+
74+
# Name of output file
75+
output_filename = 'new_files.csv'
76+
# templateflow project name
77+
key = 'ue5gx'
78+
79+
## Rest of script
80+
81+
toplevel = json_from_url(url_from_key(key))
82+
for folder in toplevel['data']:
83+
if folder['attributes']['name'] == subset:
84+
url = folder['links']['move']
85+
break
86+
87+
data = json_from_url(url)
88+
hits = ['name,link']
89+
for item in data['data']:
90+
name = item['attributes']['name']
91+
ext = ''.join(Path(name).suffixes)
92+
if strinfile in name and '.nii' in ext:
93+
print(item)
94+
link = item['links']['download']
95+
path = item['attributes']['materialized']
96+
hits.append(','.join((name, link)))
97+
Path(output_filename).write_text('\n'.join(hits))
98+
99+
Either run this script interactively or save the script as a python file (e.g. 'get_datalad_urls.py')
100+
then run the file with `python get_datalad_urls.py`.
101+
102+
Note, the above script assumes there are no subdirectories within the template folder.
103+
See end of tutorial for an example script when there are subdirectories within the template folder.
104+
105+
This script will produce a file called new_files.csv.
106+
107+
Finally, the contents of new_files.csv need to be uploaded via DataLad.
108+
109+
To do this first, move the local image file into a tmp folder.
110+
111+
> mv tpl-test/*_atlas-test*.nii.gz ~/tmp/
112+
113+
Then you add the new urls to DataLad. Add a message
114+
115+
> datalad addurls new_files.csv '{link}' '{name}' --message 'My test atlases'
116+
> datalad publish
117+
118+
Example script when subdirectories are presents
119+
================================================
120+
121+
.. code-block:: python
122+
123+
from pathlib import Path
124+
from datalad_osf.utils import url_from_key, json_from_url, get_osf_recursive, osf_to_csv
125+
126+
## VARIABLES THAT NEED TO BE CHANGED
127+
128+
# THe template you are adding too
129+
subset = 'tpl-test'
130+
# A string that identifies your new files
131+
strinfile = 'atlas-test'
132+
133+
## OPTIONAL VARIABLES
134+
135+
# Name of output file
136+
output_filename = 'new_files.csv'
137+
# templateflow project name
138+
key = 'ue5gx'
139+
140+
## REST OF SCRIPT
141+
142+
toplevel = json_from_url(url_from_key(key))
143+
for folder in toplevel['data']:
144+
if folder['attributes']['name'] == subset:
145+
url = folder['links']['move']
146+
break
147+
148+
data = json_from_url(url)
149+
hits = ['name,link']
150+
for item in data['data']:
151+
if item['attributes']['kind'] == 'folder':
152+
subdata = json_from_url(item['links']['move'])
153+
for subitem in subdata['data']:
154+
if subitem['attributes']['kind'] == 'file':
155+
name = subitem['attributes']['name']
156+
ext = ''.join(Path(name).suffixes)
157+
if strinfile in name and '.nii' in ext:
158+
print(name)
159+
link = subitem['links']['download']
160+
path = subitem['attributes']['materialized']
161+
hits.append(','.join((name, link)))
162+
Path(output_filename).write_text('\n'.join(hits))

0 commit comments

Comments
 (0)