Skip to content

Commit ea7a3ac

Browse files
JoeBatt1989JoeBatt1989
authored
Merge pull request #79 from Project-MONAI/perf-pipeline-v2
Perf pipeline v2
2 parents 47d7d8b + 4842763 commit ea7a3ac

10 files changed

Lines changed: 609 additions & 180 deletions

File tree

‎performance-testing/README.md‎

Lines changed: 167 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -1,74 +1,174 @@
11
# Description #
2-
Solution is a basic .net webapp for sending DICOMs and Rabbit messages. This will be used during performance testing by hitting the relevant endpoints to generate traffic to MIG and MWM via Rabbit. This will allow components to be tested in isolation or as a complete E2E process.
2+
Performance tests have been written to simulate real life load on the MONAI deploy solution stack. This includes benchmark, average and peak load (soak) configuration. Because of the asynchronous architecture of MONAI Deploy, k6 will be used as the load generator but logs from elastic will be used to measure the performance of the individual components.
33

44
# Getting Started #
55
## Dependencies ##
6-
- .net 6
7-
- Docker
86
- MWM and its dependencies
97
- MIG and its dependencies
8+
- Docker
9+
- Orthanc
1010
- Dummy models (to be included as part of MWM development)
11-
12-
## Running Tests ##
13-
### MONAI Informatics Gateway ###
14-
You will need to:
15-
16-
- Deploy MIG to a suitable environment including all of its dependencies (Minio, RabbitMQ, ELK).
17-
- Configure dotnet-performance-app to point to the MIG.
18-
- Configure k6 app to specify relevant load profile.
19-
- Perform GET requests on the dicom/{modality} endpoints.
20-
- Review logs in logging DB for time metrics.
21-
22-
### MONAI Workflow Manager ###
23-
You will need to:
24-
25-
- Deploy MWM to a suitable environment including all of its dependencies (Mongo, RabbitMQ, ELK, dummy models).
26-
- Configure dotnet-performance-app to point to the event message broker deployed.
27-
- Seed Mongo with Workflow Revisions
28-
- Configure k6 app to specify relevant load profile.
29-
- Perform GET requests on the rabbit/{workflow} endpoints.
30-
- Review logs in logging DB for time metrics.
31-
32-
### MIG and MWM ###
33-
You will need to:
34-
35-
- Deploy MIG to a suitable environment including all of its dependencies (Minio, RabbitMQ, ELK).
36-
- Deploy the MWM to a suitable environment including all of its dependencies (Mongo, RabbitMQ, ELK, dummy models).
37-
- Configure dotnet-performance-app to point to the event message broker deployed.
38-
- Seed Mongo with Workflow Revisions
39-
- Configure k6 app to specify relevant load profile.
40-
- Perform GET requests on the dicom/{modality} endpoints.
41-
- Review logs in logging DB for time metrics.
42-
43-
---
44-
45-
## Running locally ##
46-
47-
## Prerequisites ##
48-
49-
* .net 6.0
50-
* [K6](https://k6.io/docs/getting-started/installation/)
51-
* Docker & docker-compose
52-
53-
## Installation ##
54-
55-
* Clone the repo
56-
* Run `dotnet build` in the terminal
57-
58-
## Start Docker services ##
59-
60-
* Run `docker-compose up -d`
61-
62-
## Start the server ##
63-
64-
* Run `dotnet run`
65-
66-
## Start the performance tests ##
67-
68-
### RabbitMQ ###
69-
70-
* Run `k6 run -e CONFIG=config/{appropriate-config-file} tests/rabbit.js`
71-
72-
### Dicom ###
73-
74-
* Run `k6 run -e CONFIG=config/{appropriate-config-file} tests/dicom.js`
11+
- ELK Stack
12+
- Prometheus and Grafana
13+
- Test data (CT, MR, US, RF)
14+
15+
## Architecture ##
16+
- k6
17+
- Load generator written in Go and can be executed via Docker.
18+
- Scripting is done in JS or TS.
19+
- Configuration of load throughput are held in config files [here](./k6/dicom/config/)
20+
- Will send HTTP STORE requests to Orthanc for a given modality.
21+
- k6 Scripts
22+
- dicom_benchmark.js - Sends MR study store requests with a 2 minute sleep between each iteration.
23+
- dicom_peak_avg.js - Sends CT, MR, US and RF study store requests based on the configuration
24+
- Orthanc
25+
- is an open-source lightweight DICOM server.
26+
- Will perform C-STORE requests to MIG.
27+
- 2 modalities set up (MONAI, NOTMONAI) which will send C-STORE with an AET which will either trigger or workflow or wont depending on the Clinical Workflows set up.
28+
- ELK Stack
29+
- A log aggregator (i.e ELK) will be used for capturing all logs so that investigation of run time metrics can be achieved.
30+
- Grafana and Prometheus
31+
- Monitoring and visualization platforms to monitor memory, CPU and GPU usage from the applications
32+
- Models
33+
- Dummy models created will simulate real model usage to stress the system.
34+
35+
| Name | CPU Cores | RAM | GPU | Disk Space |
36+
| ------ | --------- | ---- | ---- | ---------- |
37+
| Small | 2 | 1GB | 1GB | 2GB |
38+
| Medium | 8 | 10GB | 6GB | 15GB |
39+
| Large | 12 | 16GB | 12GB | 25GB |
40+
41+
![link](./k6/static/Perf%20Architecture.png)
42+
43+
### Clinical Workflow Example ##
44+
```
45+
AET: MONAI
46+
Tasks [
47+
{
48+
name: router
49+
type: router
50+
task-destinations{
51+
if CT run ct-argo
52+
if MR run mr-argo
53+
if US run us-argo
54+
if RF run rf-argo
55+
}
56+
},
57+
{
58+
name: ct-argo
59+
type: argo
60+
args{
61+
argo-template: large-model
62+
}
63+
},
64+
{
65+
name: mr-argo
66+
type: argo
67+
args{
68+
argo-template: medium-model
69+
}
70+
},
71+
{
72+
name: us-argo
73+
type: argo
74+
args{
75+
argo-template: medium-model
76+
}
77+
}
78+
{
79+
name: rf-argo
80+
type: argo
81+
args{
82+
argo-template: small-model
83+
}
84+
}
85+
]
86+
```
87+
88+
## Tests ##
89+
### Baseline/ Benchmark ###
90+
Baseline/ Benchmark tests will be used to measuring the best performance of the MONAI stack. This is a low throughput test which put no stress on the system. These stats will be used to measure any degradation.
91+
92+
| Modality | Iterations | Typical Image Size | \# of Images / Study | Size (Raw) |
93+
| -------- | ---------- | ------------------ | -------------------- | ---------- |
94+
| MRI | 10 | (256, 256, 30, 1) | 200 | 26mb |
95+
96+
#### Set Up ####
97+
- Deploy MIG and MWM to an envrironment including all its dependencies.
98+
- Set up MIG with AET and Destinations scripts found [here](TBD)
99+
- Seed Orthanc with Test Data from [here](TBD)
100+
- Set up Orthanc with a Remote Modality, configuration can be found [here](https://book.orthanc-server.com/users/configuration.html#configuration)
101+
- MONAI - This will be send C-STORE requests to MIG with an AET "MONAI"
102+
- Seed MongoDB with Clinical Workflows found [here](TBD)
103+
- Seed Argo with the Argo Workflow Templates found [here](TBD)
104+
- Install k6 from [here](https://k6.io/docs/getting-started/installation/)
105+
- Update Orthanc details (i.e url) in the config/benchmarkConfig.json
106+
107+
#### Running Tests ####
108+
```bash
109+
cd k6
110+
```
111+
112+
```bash
113+
k6 run -e CONFIG=config/benchmarkConfig.json dicom/dicom_benchmark.js --insecure-skip-tls-verify
114+
```
115+
116+
#### Investigating Metrics ####
117+
##### MONAI Informatics Gateway #####
118+
- Informatics gateway will output logs detailing the time when an association was made and when a WorkflowRequest was sent. This can be seen by | grep "Payload took" which will give a hh:mm:ss between the 2 events.
119+
- Export request times can be seen by checking the time the export request was send and the time an export
120+
121+
##### MONAI Workflow Manager #####
122+
- Logs TBC
123+
124+
##### MONAI Task Manager #####
125+
- Logs TBC
126+
127+
##### Argo #####
128+
- Grafana will be used for visualization of the hardware resources.
129+
130+
### Average and Peak Load ###
131+
Average and Peak load times are displayed as below. These tests are most valuable running on production like hardware to measure performance metrics including processing times as well as system metrics such as CPU, Memory and GPU usage.
132+
133+
| Modality | Peak 1 hour | Avg 1 hour (8-5) | Typical Image Size | \# of Images / Study | Size (Raw) |
134+
| --------------------------- | ---------------- | ---------------- | ------------------ | -------------------- | ---------- |
135+
| X-ray | 120 | 60 | (2000, 2500, 1, 1) | 3 | 30mb |
136+
| Ultrasound | 50 | 28 | (640, 480, 1, 1) | 30 | 9.2mb |
137+
| CT | 30 | 10 | (512, 512, 1, 1) | 60 | 32mb |
138+
| Multi Slice CT | split with above | split with above | (512, 512, 200, 1) | 500 | 262mb |
139+
| MRI | 25 | 13 | (256, 256, 30, 1) | 200 | 26mb |
140+
| ALL (Inc. other modalities) | 250 | 140 | \- | \- | \- |
141+
142+
#### Set Up ####
143+
- Deploy MIG and MWM to an envrironment including all its dependencies.
144+
- Set up MIG with AET and Destinations scripts found [here](TBD)
145+
- Seed Orthanc with Test Data from [here](TBD)
146+
- Set up Orthanc with 2 Remote Modalities, configuration can be found [here](https://book.orthanc-server.com/users/configuration.html#configuration)
147+
- MONAI - This will be send C-STORE requests to MIG with an AET "MONAI"
148+
- NOTMONAI - This will be send C-STORE requests to MIG with an AET "NOTMONAI"
149+
- Seed MongoDB with Clinical Workflows found [here](TBD)
150+
- Seed Argo with the Argo Workflow Templates found [here](TBD)
151+
- Install k6 from [here](https://k6.io/docs/getting-started/installation/)
152+
- Update Orthanc details (i.e url) in the config/benchmarkConfig.json
153+
154+
#### Running Tests ####
155+
```bash
156+
cd k6
157+
```
158+
```bash
159+
k6 run -e CONFIG=config/{config}.json dicom/dicom_peak_avg.js --insecure-skip-tls-verify
160+
```
161+
162+
#### Investigating Metrics ####
163+
##### MONAI Informatics Gateway #####
164+
- Informatics gateway will output logs detailing the time when an association was made and when a WorkflowRequest was sent. This can be seen by | grep "Payload took" which will give a hh:mm:ss between the 2 events.
165+
- Export request times can be seen by checking the time the export request was send and the time an export
166+
167+
##### MONAI Workflow Manager #####
168+
- Logs TBC
169+
170+
##### MONAI Task Manager #####
171+
- Logs TBC
172+
173+
##### Argo #####
174+
- Grafana will be used for visualization of the hardware resources.

‎performance-testing/build.sh‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright 2021 MONAI Consortium
2+
# Licensed under the Apache License, Version 2.0 (the "License");
3+
# you may not use this file except in compliance with the License.
4+
# You may obtain a copy of the License at
5+
# http://www.apache.org/licenses/LICENSE-2.0
6+
# Unless required by applicable law or agreed to in writing, software
7+
# distributed under the License is distributed on an "AS IS" BASIS,
8+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
9+
# See the License for the specific language governing permissions and
10+
# limitations under the License.
11+
12+
SCRIPT_DIR=$(dirname "$(readlink -f "$0")")
13+
14+
VERSION=$(cat $SCRIPT_DIR/VERSION)
15+
FILEVERSION=$VERSION
16+
17+
# pass in pre-releae tags as argument
18+
if [ ! -z "$1" ]; then
19+
VERSION=$VERSION-$1
20+
fi
21+
22+
echo "Building Workflow Manager Docker Image. VERSION=$VERSION, FILEVERSION=$FILEVERSION"
23+
pushd $SCRIPT_DIR
24+
docker build --tag monai/performance-app:$VERSION -f ./dotnet-performance-app.Dockerfile --build-arg Version=$VERSION --build-arg FileVersion=$FILEVERSION .
25+
popd

‎performance-testing/dotnet-performance-app/appsettings.Development.json‎

Lines changed: 0 additions & 8 deletions
This file was deleted.

‎performance-testing/k6/dicom/config/avgConfig.json‎

Lines changed: 24 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,31 @@
11
{
2+
"orthanc":{
3+
"url": "https://preprod-kch-aide.answerdigital.io:8042",
4+
"username": "orthanc",
5+
"password": "orthanc",
6+
"workflow_modality": "monai",
7+
"no_workflow_modality": "monai_none"
8+
},
9+
"lowerThinkTime": 10,
10+
"upperThinkTime": 30,
211
"ct": {
3-
"rate": 10,
4-
"timeUnit": "1h",
5-
"duration": "1h",
6-
"preAllocatedVUs": 5
12+
"vus": 1,
13+
"iterations": 1,
14+
"maxDuration": "60s"
715
},
8-
"mri": {
9-
"rate": 13,
10-
"timeUnit": "1h",
11-
"duration": "1h",
12-
"preAllocatedVUs": 5
16+
"mr": {
17+
"vus": 1,
18+
"iterations": 1,
19+
"maxDuration": "60s"
1320
},
14-
"ultrasound": {
15-
"rate": 28,
16-
"timeUnit": "1h",
17-
"duration": "1h",
18-
"preAllocatedVUs": 5
21+
"us": {
22+
"vus": 1,
23+
"iterations": 1,
24+
"maxDuration": "60s"
1925
},
20-
"xray": {
21-
"rate": 60,
22-
"timeUnit": "1h",
23-
"duration": "1h",
24-
"preAllocatedVUs": 5
26+
"rf": {
27+
"vus": 1,
28+
"iterations": 1,
29+
"maxDuration": "60s"
2530
}
2631
}

‎performance-testing/k6/dicom/config/benchmarkConfig.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"orthanc":{
3+
"url": "https://preprod-kch-aide.answerdigital.io:8042",
4+
"username": "orthanc",
5+
"password": "orthanc",
6+
"workflow_modality": "monai",
7+
"no_workflow_modality": "monai_none"
8+
},
9+
"mr": {
10+
"vus": 1,
11+
"iterations": 1,
12+
"maxDuration": "60s"
13+
}
14+
}

‎performance-testing/k6/dicom/config/peakConfig.json‎

Lines changed: 24 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,31 @@
11
{
2+
"orthanc":{
3+
"url": "https://preprod-kch-aide.answerdigital.io:8042",
4+
"username": "orthanc",
5+
"password": "orthanc",
6+
"workflow_modality": "monai",
7+
"no_workflow_modality": "monai_none"
8+
},
9+
"lowerThinkTime": 10,
10+
"upperThinkTime": 30,
211
"ct": {
3-
"rate": 30,
4-
"timeUnit": "1h",
5-
"duration": "1h",
6-
"preAllocatedVUs": 5
12+
"vus": 1,
13+
"iterations": 1,
14+
"maxDuration": "60s"
715
},
8-
"mri": {
9-
"rate": 25,
10-
"timeUnit": "1h",
11-
"duration": "1h",
12-
"preAllocatedVUs": 5
16+
"mr": {
17+
"vus": 1,
18+
"iterations": 1,
19+
"maxDuration": "60s"
1320
},
14-
"ultrasound": {
15-
"rate": 50,
16-
"timeUnit": "1h",
17-
"duration": "1h",
18-
"preAllocatedVUs": 5
21+
"us": {
22+
"vus": 1,
23+
"iterations": 1,
24+
"maxDuration": "60s"
1925
},
20-
"xray": {
21-
"rate": 120,
22-
"timeUnit": "1h",
23-
"duration": "1h",
24-
"preAllocatedVUs": 5
26+
"rf": {
27+
"vus": 1,
28+
"iterations": 1,
29+
"maxDuration": "60s"
2530
}
2631
}

0 commit comments

Comments
 (0)