Update MDE container images to latest

- include ELK as optional services

Signed-off-by: Victor Chang <vicchang@nvidia.com>

Author: mocsharp

deploy/monai-deploy-express/.env

@@ -1,15 +1,20 @@
 # Container Images
-INFORMATICS_GATEWAY_IMAGE=ghcr.io/project-monai/monai-deploy-informatics-gateway:0.3.1
+INFORMATICS_GATEWAY_IMAGE=ghcr.io/project-monai/monai-deploy-informatics-gateway:0.3.2
 INFORMATICS_GATEWAY_CONFIG_IMAGE=dwdraju/alpine-curl-jq:latest
-WORKFLOW_MANAGER_IMAGE=ghcr.io/project-monai/monai-deploy-workflow-manager:develop-nightly-2022-09-26
-TASK_MANAGER_IMAGE=ghcr.io/project-monai/monai-deploy-task-manager:develop-nightly-2022-09-26
+WORKFLOW_MANAGER_IMAGE=ghcr.io/project-monai/monai-deploy-workflow-manager:0.1.0
+TASK_MANAGER_IMAGE=ghcr.io/project-monai/monai-deploy-task-manager:0.1.0
 
 MINIO_IMAGE=minio/minio:latest
 MINIO_MC_IMAGE=minio/mc
 MONGODB_IMAGE=mongo:5.0.12
 ORTHANC_IMAGE=osimis/orthanc:22.9.0
 RABBITMQ_IMAGE=rabbitmq:3.10-management
 
+ELK_VERSION=8.4.2
+ELASTICSEARCH_IMAGE=docker.elastic.co/elasticsearch/elasticsearch:${ELK_VERSION}
+LOGSTASH_IMAGE=docker.elastic.co/logstash/logstash:${ELK_VERSION}
+KIBANA_IMAGE=docker.elastic.co/kibana/kibana:${ELK_VERSION}
+KIBANA_CONFIG_IMAGE=dwdraju/alpine-curl-jq:latest
 
 # Default Listening AE Title
 INFORMATICS_GATEWAY_AE_TITLE=MONAI-DEPLOY
@@ -33,6 +38,12 @@ ORTHANC_UI_PORT=8042
 RABBITMQ_MGT_PORT=15672
 RABBITMQ_PORT=5672
 
+ELASTIC_PORT=9200
+ELASTIC_PORT_COMM=9300
+LOGSTASH_PORT=50000
+LOGSTASH_PORT_API=9600
+KIBANA_PORT=5601
+
 # Static IP allocations & hostnames
 # Changing of IPs/host names require updates to the service configuration files
 SUBNET=172.29.0.0/16
@@ -70,6 +81,22 @@ RABBITMQ_HOSTNAME=rabbitmq
 RABBITMQ_CONTAINERNAME=mdl-rabbitmq
 RABBITMQ_IP=172.29.0.20
 
+# change the following values may require updating the configuration files in configs/logging/*
+ELASTICSEARCH_HOSTNAME=elasticsearch
+ELASTICSEARCH_CONTAINERNAME=mdl-es
+ELASTICSEARCH_PASSWORD=monai
+ELASTICSEARCH_IP=172.29.0.101
+
+LOGSTASH_HOSTNAME=logstash
+LOGSTASH_CONTAINERNAME=mdl-logstash
+LOGSTASH_IP=172.29.0.102
+
+KIBANA_HOSTNAME=kibana
+KIBANA_CONTAINERNAME=mdl-kibana
+KIBANA_CFG_CONTAINERNAME=mdl-kibana-cfg
+KIBANA_IP=172.29.0.103
+KIBANA_CFG_IP=172.29.0.104
+
 # Host Data Volume Paths
 INFORMATICS_GATEWAY_PAYLOADS=.md/mdig/payloads/
 INFORMATICS_GATEWAY_DATABASE=.md/mdig/database/

deploy/monai-deploy-express/README.md

@@ -49,13 +49,23 @@ The [Docker compose file](docker-compose.yml) spins up the following services. S
 Execute one of the following commands in the directory where the `docker-compose.yml` resides to start, stop all the services or to view the logs.
 
 ```bash
-docker compose up # start all services
+docker compose up # start MONAI Deploy Express
 docker compose up -d # or run detached
 docker compose logs -t -f # view output from all containers
 docker compose down # stop all services
 ```
 
-The first time calling `docker compose up` may take longer as it needs to pull all the container images. However, once all container images are pulled and available on the local system, all the services should spin up within 20 to 60 seconds.
+Use the following commands to launch MONAI Deploy Express with ELK (ElasticSearch-LogStash-Kibana):
+
+```bash
+./init.sh # required to setup Elastic volume mapping permissions on Linux
+docker compose --profile elk up # start MONAI Deploy Express & ELK
+docker compose --profile elk up -d # or run detached
+docker compose logs -t -f # view output from all containers
+docker compose --profile elk down # stop all services
+```
+
+The first time calling `docker compose up` may take longer as it needs to pull all the container images. However, once all container images are pulled and available on the local system, all the services should spin up between 20 to 65 seconds.
 
 ### MONAI Deploy Services
 
@@ -75,6 +85,9 @@ The first time calling `docker compose up` may take longer as it needs to pull a
 | MongoDB (default database for Worklflow Manager & Task Manager) | http://localhost:27017 | http://172.29.0.30:27017 |
 | Orthanc UI (optional)                                           | http://localhost:8042  | http://172.29.0.100:8042 |
 | Orthanc SCP (optional)                                          | 4242                   | 4242                     |
+| Elastic Search                                                  | 9200, 9300             | 9200,9300                |
+| Log Stash                                                       | 9600, 50000            | 9600, 50000              |
+| Kibana                                                          | http://localhost:5601  | http://172.29.0.103:5601 |
 
 *Note: Orthanc is included for convenience and to demo end-to-end workflow execution. It may be disabled and have MONAI Deploy Express integrated with external Orthanc, PACS or other DICOM devices.*
 
@@ -166,12 +179,17 @@ This example uses the `alpine` container image instead of a MAP to print all fil
 In this section, we will download a DICOM dataset, upload it to Orthanc and then run the [Liver Segmentation MAP](https://github.com/Project-MONAI/monai-deploy-app-sdk/tree/main/examples/apps/ai_livertumor_seg_app) from the
 [MONAI Deploy App SDK](https://github.com/Project-MONAI/monai-deploy-app-sdk). Finally, we can expect the AI-generated segmentation result to appear in Orthanc.
 
+
 #### Steps
 
 1. Download the [Abdomen CT dataset](#running-a-monai-deploy-workflow) dataset
-2. Upload the dataset as described in [Uploading Data](#upload-dicom-datasets)
-3. Deploy the workflow definition to MONAI Deploy Workflow Manager:
-4. 
+2. Download the AI Liver Tumor MAP
+   ```bash
+   docker pull ghcr.io/mmelqin/monai_ai_livertumor_seg_app:1.0
+   ```
+3. Upload the dataset as described in [Uploading Data](#upload-dicom-datasets)
+4. Deploy the workflow definition to MONAI Deploy Workflow Manager:
+5. 
    ```bash
    curl --request POST --header 'Content-Type: application/json'  --data "@sample-workflows/liver-seg.json"  http://localhost:5001/workflows
    ```
@@ -182,10 +200,10 @@ In this section, we will download a DICOM dataset, upload it to Orthanc and then
    {"workflow_id":"811620da-381f-4daa-854d-600948e67228"}
    ```
 
-5. Navigate to Orthanc, select any study and then click *Send to DICOM Modality* from the menu on the left.
+6. Navigate to Orthanc, select any study and then click *Send to DICOM Modality* from the menu on the left.
    In the popup dialog, select **MONAI-DEPLOY** to start a C-STORE request to the Informatics Gateway.
-6. Wait for the workflow to complete; the entire workflow takes roughly one minute and thirty seconds to complete. To see the AI-generated segmentation object, reload the Orthanc study page.
-7. To see the output of the container, run the following commands:
+7. Wait for the workflow to complete; the entire workflow takes roughly one minute and thirty seconds to complete. To see the AI-generated segmentation object, reload the Orthanc study page.
+8. To see the output of the container, run the following commands:
    ```bash
    > docker container list -a | grep monai_ai_livertumor_seg_app
    # locate the container ID and run the following
@@ -202,9 +220,13 @@ In this section, we will download a DICOM dataset, upload it to Orthanc and then
 #### Steps
 
 1. Download the [Chest CT dataset](#running-a-monai-deploy-workflow) dataset
-2. Upload the dataset as described in [Uploading Data](#upload-dicom-datasets)
-3. Deploy the workflow definition to MONAI Deploy Workflow Manager:
-4. 
+2. Download the AI Lung Segmentation MAP
+   ```bash
+   docker pull ghcr.io/mmelqin/monai_ai_lung_seg_app:1.0
+   ```
+3. Upload the dataset as described in [Uploading Data](#upload-dicom-datasets)
+4. Deploy the workflow definition to MONAI Deploy Workflow Manager:
+5. 
    ```
    curl --request POST --header 'Content-Type: application/json'  --data "@sample-workflows/lung-seg.json"  http://localhost:5001/workflows
    ```
@@ -215,10 +237,10 @@ In this section, we will download a DICOM dataset, upload it to Orthanc and then
    {"workflow_id":"811620da-381f-4daa-854d-600948e67228"}
    ```
 
-5. Navigate to Orthanc, select any study and then click *Send to DICOM Modality* from the menu on the left.
+6. Navigate to Orthanc, select any study and then click *Send to DICOM Modality* from the menu on the left.
    In the popup dialog, select **MONAI-DEPLOY** to start a C-STORE request to the Informatics Gateway.
-6. Wait for the workflow to complete; the entire workflow takes roughly one minute to complete. To see the AI-generated segmentation object, reload the Orthanc study page.
-7. To see the output from the container, run the following commands:
+7. Wait for the workflow to complete; the entire workflow takes roughly one minute to complete. To see the AI-generated segmentation object, reload the Orthanc study page.
+8. To see the output from the container, run the following commands:
    ```bash
    > docker container list -a | grep monai_ai_lung_seg_app
    # locate the container ID and run the following
@@ -230,9 +252,14 @@ In this section, we will download a DICOM dataset, upload it to Orthanc and then
 In the `liver-lung-seg.json` workflow definition, we combined the AI Lung MAP and the AI Liver MAP into one single workflow definition. With this workflow definition, the Workflow Manager would route the incoming data based on the routing rules defined.
 
    1. Download one or both of the [datasets](#running-a-monai-deploy-workflow) provided above
-   2. Upload the dataset as described in [Uploading Data](#upload-dicom-datasets)
-   3. Deploy the workflow definition to MONAI Deploy Workflow Manager:
-   4. 
+   2. Download the AI Liver Tumor & the AI Lung Segmentation MAPs
+      ```bash
+      docker pull ghcr.io/mmelqin/monai_ai_lung_seg_app:1.0
+      docker pull ghcr.io/mmelqin/monai_ai_livertumor_seg_app:1.0
+      ```
+   3. Upload the dataset as described in [Uploading Data](#upload-dicom-datasets)
+   4. Deploy the workflow definition to MONAI Deploy Workflow Manager:
+   5. 
       ```
       curl --request POST --header 'Content-Type: application/json'  --data "@sample-workflows/liver-lung-seg.json"  http://localhost:5001/workflows
       ```
@@ -243,19 +270,24 @@ In the `liver-lung-seg.json` workflow definition, we combined the AI Lung MAP an
       {"workflow_id":"6d5e1e73-bd07-4e71-b1fa-b66408d43b82"}
       ```
 
-   5. Navigate to Orthanc, select one of the studies and then click *Send to DICOM Modality* from the menu on the left.
+   6. Navigate to Orthanc, select one of the studies and then click *Send to DICOM Modality* from the menu on the left.
       In the popup dialog, choose **MONAI-DEPLOY** to start a C-STORE request to the Informatics Gateway.
-   6. Wait for the workflow to complete and reload the Orthanc study page and expect a new series to be added.
-   7. To see the output from the container, run the following commands:
+   7. Wait for the workflow to complete and reload the Orthanc study page and expect a new series to be added.
+   8. To see the output from the container, run the following commands:
       ```bash
       > docker container list -a | grep monai_ai_
       # locate the container ID and run the following
       > docker logs {CONTAINER ID}
       ```
-   8. Repeat the steps with the other dataset.
+   9. Repeat the steps with the other dataset.
 
 In this example, the [Chest CT dataset](https://drive.google.com/file/d/1IGXUgZ7NQCwsix57cdSgr-iYErevqETO/view?usp=sharing) should only launch the AI Lung MAP, while the [Abdomen CT dataset](https://drive.google.com/file/d/1d8Scm3q-kHTqr_-KfnXH0rPnCgKld2Iy/view?usp=sharing) would launch the AI Liver MAP.
 
+## Using Kibana
+
+A default dataview is imported to Kibana at startup. To load the saved search, navigate to http://localhost:5601/, click on Analytics > Discover from the 🍔 menu. From the top right click *Open* and select *MONAI-Default*.
+
+
 ## Tips
 
 - If you are using your DICOM dataset, please remove the router task or modify the routing conditions to meet your needs.

deploy/monai-deploy-express/configs/informatics-gateway.json

@@ -10,7 +10,7 @@
         "rejectUnknownSources": false
       },
       "scu": {
-        "aeTitle": "MonaiSCU",
+        "aeTitle": "MONAISCU",
         "logDimseDatasets": false,
         "logDataPDUs": false
       }
@@ -38,12 +38,15 @@
       }
     },
     "storage": {
-      "bufferRootPath": "/temp",
+      "bufferRootPath": "/payloads",
       "tempStorageRootPath": "/incoming",
       "bucketName": "monaideploy",
       "storageRootPath": "/payloads",
       "temporaryBucketName": "monaideploy",
       "serviceAssemblyName": "Monai.Deploy.Storage.MinIO.MinIoStorageService, Monai.Deploy.Storage.MinIO",
+      "watermarkPercent": 75,
+      "reserveSpaceGB": 5,
+      "concurrentUploads": 4,
       "settings": {
         "endpoint": "minio:9000",
         "accessKey": "minioadmin",
@@ -61,42 +64,6 @@
       "sendAck": true
     }
   },
-  "Logging": {
-    "LogLevel": {
-      "Default": "Information",
-      "Dicom": "Warning",
-      "System": "Warning",
-      "Microsoft": "Warning",
-      "Microsoft.EntityFrameworkCore": "Warning",
-      "Microsoft.Hosting.Lifetime": "Warning",
-      "Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker": "Error",
-      "Monai": "Information"
-    },
-    "Console": {
-      "FormatterOptions": {
-        "ColorBehavior": "Enabled",
-        "IncludeScopes": true,
-        "SingleLine": false,
-        "TimestampFormat": " HH:mm:ss ",
-        "UseUtcTimestamp": true
-      }
-    },
-    "File": {
-      "BasePath": "logs",
-      "FileEncodingName": "utf-8",
-      "DateFormat": "yyyyMMdd",
-      "CounterFormat": "000",
-      "MaxFileSize": 10485760,
-      "IncludeScopes": true,
-      "MaxQueueSize": 100,
-      "TextBuilderType": "Monai.Deploy.InformaticsGateway.Logging.FileLoggingTextFormatter, Monai.Deploy.InformaticsGateway",
-      "Files": [
-        {
-          "Path": "MIG-<date>-<counter>.log"
-        }
-      ]
-    }
-  },
   "Kestrel": {
     "EndPoints": {
       "Http": {
@@ -114,4 +81,4 @@
     "InformaticsGatewayServerEndpoint": "http://localhost:5000",
     "DockerImagePrefix": "ghcr.io/project-monai/monai-deploy-informatics-gateway"
   }
-}
+}

deploy/monai-deploy-express/configs/logging/elasticsearch.yml

@@ -0,0 +1,27 @@
+# Copyright 2022 MONAI Consortium
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+
+## Default Elasticsearch configuration from Elasticsearch base image.
+## https://github.com/elastic/elasticsearch/blob/main/distribution/docker/src/docker/config/elasticsearch.yml
+#
+cluster.name: "docker-cluster"
+network.host: 0.0.0.0
+
+## X-Pack settings
+## see https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html
+#
+xpack.security.enabled: false
+xpack.license.self_generated.type: basic

deploy/monai-deploy-express/configs/logging/kibana.ndjson

@@ -0,0 +1,3 @@
+{"attributes":{"fieldAttrs":"{\"@version\":{\"count\":1},\"EventId\":{\"count\":3},\"Level\":{\"count\":4},\"LoggerName\":{\"count\":3},\"MachineName\":{\"count\":1},\"Message\":{\"count\":2},\"ServiceName\":{\"count\":1},\"ServiceVersion\":{\"count\":1}}","fields":"[]","name":"monai-deploy","runtimeFieldMap":"{}","sourceFilters":"[]","timeFieldName":"@timestamp","title":"monai-deploy-*","typeMeta":"{}"},"coreMigrationVersion":"8.4.2","id":"6c4ef8fa-f382-4dad-8c82-f180f09fc88d","migrationVersion":{"index-pattern":"8.0.0"},"references":[],"type":"index-pattern","updated_at":"2022-10-11T21:54:09.450Z","version":"Wzc2MiwxXQ=="}
+{"attributes":{"columns":["EventId","Level","LoggerName","Message"],"description":"","grid":{"columns":{"EventId":{"width":124},"Level":{"width":67.66666666666663},"LoggerName":{"width":668.1666666666667}}},"hideChart":false,"isTextBasedQuery":false,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[{\"meta\":{\"type\":\"phrases\",\"key\":\"Level\",\"params\":[\"TRACE\"],\"alias\":null,\"negate\":true,\"disabled\":false,\"value\":[\"TRACE\"],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index\"},\"query\":{\"bool\":{\"should\":[{\"match_phrase\":{\"Level\":\"TRACE\"}}],\"minimum_should_match\":1}},\"$state\":{\"store\":\"appState\"}}],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[["@timestamp","desc"]],"title":"MONAI-Default"},"coreMigrationVersion":"8.4.2","id":"48ce7db0-49af-11ed-bde2-430e63791a09","migrationVersion":{"search":"8.0.0"},"references":[{"id":"6c4ef8fa-f382-4dad-8c82-f180f09fc88d","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"},{"id":"6c4ef8fa-f382-4dad-8c82-f180f09fc88d","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index","type":"index-pattern"}],"type":"search","updated_at":"2022-10-11T21:54:30.795Z","version":"Wzc3NSwxXQ=="}
+{"excludedObjects":[],"excludedObjectsCount":0,"exportedCount":2,"missingRefCount":0,"missingReferences":[]}

deploy/monai-deploy-express/configs/logging/kibana.yml

@@ -0,0 +1,27 @@
+# Copyright 2022 MONAI Consortium
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+---
+## Default Kibana configuration from Kibana base image.
+## https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/templates/kibana_yml.template.ts
+#
+server.name: kibana
+server.host: 0.0.0.0
+elasticsearch.hosts: [ "http://elasticsearch:9200" ]
+monitoring.ui.container.elasticsearch.enabled: true
+
+## X-Pack security credentials
+#
+elasticsearch.username: kibana_system
+elasticsearch.password: monai

deploy/monai-deploy-express/configs/logging/logstash.yml

@@ -0,0 +1,18 @@
+# Copyright 2022 MONAI Consortium
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+---
+## Default Logstash configuration from Logstash base image.
+## https://github.com/elastic/logstash/blob/main/docker/data/logstash/config/logstash-full.yml
+#
+http.host: "0.0.0.0"

deploy/monai-deploy-express/configs/logging/pipeline.yml

@@ -0,0 +1,16 @@
+# Copyright 2022 MONAI Consortium
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+- pipeline.id: monei-deploy
+  path.config: "/usr/share/logstash/pipeline/monai-deploy.conf"
+  queue.type: persisted