You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+16-31Lines changed: 16 additions & 31 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -109,38 +109,28 @@ sequencers:
109
109
110
110
## How It Works
111
111
112
-
### Run Processing Pipeline
113
-
114
112
1.**Discovery**: Scans configured sequencing directories for run folders
115
113
2.**Validation**: Confirms run ID matches expected format for the sequencer type
116
114
3.**Transfer Phases**:
117
-
-**Sequencing Phase**: Starts continuous background rsync transfer while sequencing is ongoing (when the final sequencing file doesn't exist). Uploads status and metadata files (`metadata_for_statusdb`) to database.
115
+
-**Sequencing Phase**: Starts continuous background rsync transfer while sequencing is ongoing (when the final sequencing file doesn't exist). Uploads status and metadata files (specified for each sequencer type in the config with `metadata_for_statusdb`) to database.
118
116
-**Final Transfer**: After sequencing completes (final sequencing file appears), initiates final rsync transfer and captures exit code.
119
-
-**Completion**: Updates database when transfer succeeds.
117
+
-**Completion**: Updates database when transfer was successful.
120
118
121
119
### Status Tracking
122
120
123
121
Run status is tracked in CouchDB with events including:
124
122
125
-
**Status:**`sequencing_started`
126
-
**Meaning:** Sequencing is ongoing
127
-
**Occurs when:** A run folder exists but the final sequencing file has not been created yet
128
-
129
-
**Status:**`transfer_started`
130
-
**Meaning:** Intermediate transfer was initiated
131
-
**Occurs when:** A run folder exists but the final sequencing file has not been created yet.
132
-
133
-
**Status:**`sequencing_finished`
134
-
**Meaning:** Sequencing has completed
135
-
**Occurs when:** A run folder exists and the final sequencing file has been created
|`sequencing_started`| Sequencing is ongoing | A run folder exists but the final sequencing file has not been created yet |
126
+
|`transfer_started`| Intermediate transfer was initiated | Sequencing is ongoing and an rsync has been started |
127
+
|`sequencing_finished`| Sequencing has completed | A run folder exists and the final sequencing file has been created |
128
+
|`final_transfer_started`| Final sync has started | A run folder exists and the final sequencing file has been created, but the final rsync exit code file has not yet been created or contains a non-zero exit code |
129
+
|`transferred_to_hpc`| Transfer completed successfully | A run folder exists, the final sequencing file has been created, and the final rsync exit code file contains a 0 exit code |
136
130
137
-
**Status:**`final_transfer_started`
138
-
**Meaning:** Final sync has started
139
-
**Occurs when:** A run folder exists and the final sequencing file has been created, but the final rsync exit code file has not yet been created or contains a non-zero exit code
131
+
### Flow chart
140
132
141
-
**Status:**`transferred_to_hpc`
142
-
**Meaning:** Transfer completed successfully
143
-
**Occurs when:** A run folder exists, the final sequencing file has been created, and the final rsync exit code file contains a 0 exit code
133
+

144
134
145
135
### Key Components
146
136
@@ -161,13 +151,8 @@ Run status is tracked in CouchDB with events including:
161
151
162
152
The logic of the script relies on the following status files:
163
153
164
-
- run.final_file
165
-
- The final file written by each sequencing machine. Used to indicate when the sequencing has completed.
166
-
- final_rsync_exitcode - Used to indicate when the final rsync is done, so that the final rsync can be run in the background. This is especially useful for restarts after long pauses of the cronjob.
167
-
168
-
### Flow chart
169
-
170
-

154
+
-`run.final_file` - The final file written by each sequencing machine. Used to indicate when the sequencing has completed.
155
+
-`final_rsync_exitcode` - Used to indicate when the final rsync is done, so that the final rsync can be run in the background. This is especially useful for restarts after long pauses of the cronjob.
171
156
172
157
## Development
173
158
@@ -209,9 +194,9 @@ dataflow_transfer/
209
194
To add support for a new sequencer, add the following to dataflow_transfer:
210
195
211
196
1. Add a new class for the sequencer in one of the run classes files below. Make sure it inherits from the manufacturer class (IlluminaRun, ElementRun, ONTRun)
0 commit comments