Merge pull request #151 from dsajdak/isc23

ISC23

Author: aebruno

README.md

@@ -1,12 +1,11 @@
 # HPC Toolset Tutorial
 
-Tutorial for installing and configuring [OnDemand](https://openondemand.org/), [Open XDMoD](https://open.xdmod.org), and [ColdFront](http://coldfront.io): an HPC center management toolset.  
+Tutorial for installing and configuring [ColdFront](http://coldfront.io), [OnDemand](https://openondemand.org/), and [Open XDMoD](https://open.xdmod.org): an HPC center management toolset.  
 
 ### Presented by:
 
 [![OSC Logo https://osc.edu](docs/osc_logo.png)](https://osc.edu)  
 [![CCR logo](docs/ccr_logo.jpg)](https://buffalo.edu/ccr)  
-[![VT logo](docs/vt_logo.jpg)](https://arc.vt.edu/)  
 
 
 This tutorial aims to demonstrate how three open source applications work in concert to provide a toolset for high performance computing (HPC) centers. ColdFront is an allocations management portal that provides users an easy way to request access to allocations for a Center's resources.  HPC systems staff configure the data center’s resources with attributes that tie ColdFront’s plug-ins to systems such as job schedulers, authentication/account management systems, system monitoring, and Open XDMoD.  Once the user's allocation is activated in ColdFront, they are able to access the resource using OnDemand, a web-based portal for accessing HPC services that removes the complexities of HPC system environments from the end-user.  Through OnDemand, users can upload and download files, create, edit, submit and monitor jobs, create and share apps, run GUI applications and connect to a terminal, all via a web browser, with no client software to install and configure.  The Open XDMoD portal provides a rich set of features, which are tailored to the role of the user.  Sample metrics provided by Open XDMoD include: number of jobs, CPUs consumed, wait time, and wall time, with minimum, maximum and the average of these metrics. Performance and quality of service metrics of the HPC infrastructure are also provided, along with application specific performance metrics (flop/s, IO rates, network metrics, etc) for all user applications running on a given resource.  
@@ -27,6 +26,8 @@ This tutorial aims to demonstrate how three open source applications work in con
 ## Workshops
 This tutorial will be presented at the following conferences:
 
+[PEARC23](https://pearc.acm.org/pearc23/)  
+[ISC23](https://www.isc-hpc.com/)  
 [PEARC22](https://pearc.acm.org/pearc22)  
 [PEARC21](https://pearc.acm.org/pearc21)  
 [PEARC20](https://pearc.acm.org/pearc20/)  

coldfront/README.md

@@ -6,7 +6,7 @@ Thank you to the staff of the [Ohio Supercomputer Center](https://osc.edu), [Uni
 Thank you to the [National Science Foundation](https://nsf.gov) for the supporting the development of [Open OnDemand](https://openondemand.org) and [Open XDMoD](https://open.xdmod.org/)  
 - Open OnDemand NSF award numbers: [NSF#1534949](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1534949) and [NSF#1935725](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1835725)  
 
-- XDMoD NSF award numbers: [ACI 1025159](https://nsf.gov/awardsearch/showAward?AWD_ID=1025159) and [ACI 1445806](https://nsf.gov/awardsearch/showAward?AWD_ID=1445806)
+- XDMoD NSF award numbers: [ACI 1025159](https://nsf.gov/awardsearch/showAward?AWD_ID=1025159), [ACI 1445806](https://nsf.gov/awardsearch/showAward?AWD_ID=1445806), and [OAC 2137603](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2137603)  
 
 #### Publications
 - Andrew E. Bruno, Doris J. Sajdak. 2021. ColdFront: Resource Allocation Management System (PEARC ’21). Association for Computing Machinery, New York, NY, USA. DOI:[10.1145/3437359.3465585](https://doi.org/10.1145/3437359.3465585)  
@@ -18,12 +18,15 @@ Thank you to the [National Science Foundation](https://nsf.gov) for the supporti
 #### Workshops
 This tutorial was first presented at the PRACTICE AND EXPERIENCE IN ADVANCED RESEARCH COMPUTING 2020 Virtual Conference (PEARC20) - https://pearc.acm.org/pearc20/  
 
-A condensed version of this tutorial will be presented at [Gateways 2020](https://sciencegateways.org/web/gateways2020)  
+A condensed version of this tutorial was presented at [Gateways 2020](https://sciencegateways.org/web/gateways2020)  
 
 An updated version of this tutorial was presented at the PRACTICE AND EXPERIENCE IN ADVANCED RESEARCH COMPUTING 2021 Virtual Conference (PEARC21) - https://pearc.acm.org/pearc21/  
 
 An updated version of this tutorial was presented at the PRACTICE AND EXPERIENCE IN ADVANCED RESEARCH COMPUTING 2022 Conference (PEARC22), Boston, MA - https://pearc.acm.org/pearc22/ 
 
+A condensed, half day version of this tutorial will be presented at ISC High Performance Conference 2023, Hamburg, Germany - https://www.isc-hpc.com/  
+
+An updated version of this tutorial will be presented at the PRACTICE AND EXPERIENCE IN ADVANCED RESEARCH COMPUTING 2023 Conference (PEARC23), Portland, OR - https://pearc.acm.org/pearc23/ 
 
 #### Container Development
 

docs/acknowledgments.md

@@ -46,30 +46,28 @@ $ ./hpcts start
 
 - [Docker](https://docs.docker.com)
 - [Install & Start Docker](https://docs.docker.com/engine/install/)
-- [Install Docker Compose](https://docs.docker.com/compose/install/)
-- [Linux](https://docs.docker.com/engine/install/linux-postinstall/) 
+- [Linux & WSL](https://docs.docker.com/engine/install/linux-postinstall/) 
 - [MacOS Docker Desktop](https://docs.docker.com/docker-for-mac/troubleshoot/)  
-- [Windows Docker Desktop](https://docs.docker.com/docker-for-windows/troubleshoot/)
 
 ### Helpful Docker commands
 
 ```
 # Start all HPC Toolset Containers manually
-$ docker-compose up -d
+$ docker compose up -d
 
 # Display Tutorial Container Logs
-$ docker-compose logs -f
-$ docker-compose logs -f coldfront
-$ docker-compose logs -f xdmod
+$ docker compose logs -f
+$ docker compose logs -f coldfront
+$ docker compose logs -f xdmod
 
 # Stop containers 
-$ docker-compose stop
+$ docker compose stop
 
 # Stop containers and remove them
-$ docker-compose down
+$ docker compose down
 
 # Stop containers,remove them and all volumes
-$ docker-compose down -v
+$ docker compose down -v
 
 # Display Docker processes
 $ docker ps -a
@@ -113,28 +111,6 @@ NOTE: this is only necessary on some systems so don't use it if the previous com
 
 **Sometimes restarting your operating system is the only solution.**
 
-#### Windows Errors  
-
-NOTE: Windows users should get several pop-up messages from Docker Desktop during this process asking to allow local system access to the Docker containers.  Please click the "Share it" button:
-![](windows_sharing.PNG)
-
-If you have notifications blocked, you may not see these pop-ups and the authorization will eventually time out.  If this happens, you will get this type of error message:    
-
-```
-Error response from daemon: user declined directory sharing C:\Users\path_to_my_folder
-```
-Open Docker Desktop, navigate to Settings - Resources, and click on File Sharing.  Then add the directory where you've cloned the HPC Toolset Tutorial and click "Apply & Restart"
-
-Re-run:  
-```
-./hpcts start  
-```
-
-If this doesn't work, please run:  
-```
-./hpcts cleanup  
-./hpcts start  
-```
 
 #### Deleting Docker containers/images/volumes manually
 

docs/docker_tips.md

@@ -12,13 +12,8 @@ If you haven't already installed and tested the required packages, please refer
 
 ## Getting started
 
-You will need to clone the tutorial repo and then run the helper script.  The initial clone of the repo may take 5-10 minutes.  The first time running the helper script, you'll be downloading all the containers from Docker Hub.  This can take quite a long time depending on your network speed.  The images total approximately 13GB in size.  Once the containers are downloaded, they are started and the services launched.  For point of reference: on a recent test from a home fiber optic network with client connected over wifi this download and container startup process took 12 minutes.  
+You will need to clone the tutorial repo and then run the helper script.  The initial clone of the repo may take 5-10 minutes.  The first time running the helper script, you'll be downloading all the containers from Docker Hub.  This can take quite a long time depending on your network speed.  The images total approximately 20GB in size.  Once the containers are downloaded, they are started and the services launched.  For point of reference: on a recent test from a home fiber optic network with client connected over wifi this download and container startup process took 12 minutes.  
 
-NOTE: For Windows, if you haven't already done so, you will need to configure git not to convert line endings into Windows format.  Run this command using the git-bash shell application before cloning the tutorial repo:
-
-```
-git config --global core.autocrlf input
-```
 
 ### Clone Repo and Start Containers
 
@@ -84,28 +79,7 @@ Starting HPC Toolset Cluster..
 
 **NOTE:  Despite seeing this output with URLs, the processes on these containers may not be fully running yet.  Depending on the speed of your computer, starting up the processes may take a few minutes (or even up to 10 minutes).  Use the command below to check the docker logs if the websites are not yet displaying.**
 
-### Windows Errors  
-
-NOTE: Windows users should get several pop-up messages from Docker Desktop during this process asking to allow local system access to the Docker containers.  Please click the "Share it" button:
-![](windows_sharing.PNG)
 
-If you have notifications blocked, you may not see these pop-ups and the authorization will eventually time out.  If this happens, you will get this type of error message:    
-
-```
-Error response from daemon: user declined directory sharing C:\Users\path_to_my_folder
-```
-Open Docker Desktop, navigate to Settings - Resources, and click on File Sharing.  Then add the directory where you've cloned the HPC Toolset Tutorial and click "Apply & Restart"
-
-Re-run:  
-```
-./hpcts start  
-```
-
-If this doesn't work, please run:  
-```
-./hpcts destroy  
-./hpcts start  
-```
 
 ### Docker Logs
 

docs/getting_started.md

@@ -1,16 +1,10 @@
 ## Requirements
 
-For this tutorial you will need to have **20GB of free disk space** and git, docker, docker-compose and a web browser installed on your local machine.  This tutorial has been tested on various versions of Linux, MacOS, and Windows 10 with the following package versions:
+For this tutorial you will need to have **20GB of free disk space** and git, docker, docker-compose and a web browser installed on your local machine.  This tutorial has been tested on various versions of Linux, MacOS, and Windows 10/11 (using Windows subsystem for Linux) with the following package versions:
 
-- git 2.17+ (Windows users we recommend:  https://gitforwindows.org/)
+- git 2.17+ 
 - docker engine version 20.10.12+
-- docker-compose 2.6.0+
-
-NOTE: For Windows, if you haven't already done so, you will need to configure git not to convert line endings into Windows format.  Run this command using the git-bash shell application before cloning the tutorial repo:
-
-```
-git config --global core.autocrlf input
-```
+- docker compose 2.6.0+ (this is distributed with newer versions of docker and not necessary to install separately)  
 
 The following ports must be open and available:
 
@@ -27,11 +21,9 @@ If they are not you might see an error like:
 
 https://docs.docker.com/engine/install/
 
-**NOTE: You'll need to make sure the account you're running docker with is in the 'docker' group**
-
-### Install Docker Compose  
+**NOTE: Make sure the account you're running docker with is in the 'docker' group**
 
-https://docs.docker.com/compose/install/
+**Windows users:** We do NOT recommend using Docker Desktop with this container environment.  Instead, use Windows subsystem for Linux (WSL) and install Docker within an Ubuntu virtual machine.  [This site](https://nickjanetakis.com/blog/install-docker-in-wsl-2-without-docker-desktop) provides useful information on this method, except docker-compose no longer needs to be installed separately.  Using WSL allows you to follow the docker installation instructions for Linux and this is what is tested by the HPC Toolset Tutorial team.  
 
 ### Verify working Docker
 
@@ -42,28 +34,6 @@ docker info
 **This should display your system info along with Docker-specific info.  If there are any errors, stop/start Docker.  Do NOT proceed with the tutorial until you are sure you have a working Docker setup**
 
 
-### Error when running 'docker info' or when starting up tutorial containers
-
-If you get this error when starting the tutorial   
-
-```
-ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?
-
-or  
-
-ERROR: Couldn't connect to Docker daemon at http+docker://localhost - is it running?
-```
-
-Try stopping and starting Docker (restart doesn't usually fix the problem).  Commands for this differ depending on operating system.
-
-If the error persists, try:  
-
-```
-export DOCKER_HOST=127.0.0.1
-```
-
-NOTE: this is only necessary on some systems so don't use it if the previous command works
-
 ## Docker Tips
 
 Some useful info on installing Docker, navigating this tutorial and learning a bit about docker-compose

docs/requirements.md

@@ -1647,6 +1647,6 @@ Review Job Composer links - access Job Composer
 </details>
 
 ## Tutorial Navigation
-[Next - Acknowledgments](../docs/acknowledgments.md)  
-[Previous Step - Open XDMoD](../xdmod/README.md)  
-[Back to Start](../README.md)
+[Next Step - Open XDMoD](../xdmod/README.md)  
+[Previous Step - ColdFront](../coldfront/README.md)  
+[Back to Start](../README.md)  

ondemand/README.md

@@ -1486,6 +1486,6 @@ Admin Dashboard:
 ![Admin Dashboard](./tutorial-screenshots/admin-dashboard.png)
 
 ## Tutorial Navigation
-[Next - OnDemand](../ondemand/README.md)
-[Previous Step - ColdFront](../coldfront/README.md)
-[Back to Start](../README.md)
+[Next - Acknowledgements](../docs/acknowledgments.md)  
+[Previous Step - Open OnDemand](../ondemand/README.md)  
+[Back to Start](../README.md)