26.07

DeepOps 26.07 is the second release of the modern CalVer train, following 26.05 on a six-week cadence.

General

• New DGX software stack documentation and refreshed nvidia-dgx role covering DGX OS 6 (Ubuntu 22.04), DGX OS 7 (Ubuntu 24.04), DGX Software for RHEL 8, and Enterprise Linux 9, while preserving legacy Ubuntu 18.04/20.04 and RHEL 7 paths (#1346).
• Clarified current release validation targets versus legacy/community-supported OS paths across the README and docs (#1354).
• Script debug output now reports a meaningful DeepOps version derived from the checkout's git tag description, with a config/env.sh override (#1357).

Container runtime and OS support

• New nvidia_container_toolkit role: Ubuntu 24.04+ hosts install the NVIDIA Container Toolkit from the current libnvidia-container repository and configure Docker with nvidia-ctk (#1347).
• Red Hat family 8+ hosts now use the same role via the official NVIDIA Container Toolkit RPM repository; RPM airgap docs refreshed for EL8/EL9 (#1350).
• Container Toolkit airgap docs moved from the legacy nvidia-docker wrapper to mirrored package repositories plus nvidia-ctk configuration (#1352).

Provisioning

• Retired DeepOps-owned DGXIE, minimal PXE/Pixiecore, and Ubuntu 16.04/18.04 preseed assets. MAAS owns machine lifecycle, DHCP/PXE, images, and deployment; DeepOps consumes MAAS tags and dynamic inventory (#1351).

Kubernetes

• GPU Operator chart default updated to v26.3.3; Kubernetes GPU device plugin and GPU feature discovery charts updated to 0.19.3; MIG manager packages updated to mig-parted v0.14.2; Network Operator default updated to 26.4.0 (#1357).
• Kubernetes storage helper support boundaries clarified: the NFS subdir provisioner remains the simple default path (chart 4.0.18), NetApp Trident defaults/docs refreshed to Trident 26.02.1 with opt-in backend creation, and the deprecated Rook/Ceph helper now requires explicit opt-in (#1358).

Slurm

• Slurm default version updated to 26.05.1, the current upstream stable line (#1357).
• DCGM exporter refreshed to 4.5.3-4.8.2 with current upstream metrics CSVs and a distroless-image-compatible DaemonSet (#1348).

Monitoring

• Prometheus v3.13.0, Alertmanager v0.33.0, Grafana 13.1.0, node exporter v1.11.1, and kube-prometheus-stack chart 87.5.1 (#1348, #1357).

Examples and documentation

• CUDA/NGC example images refreshed to current tags across airgap docs, Kubernetes workload examples, and smoke scripts (#1349).
• RAPIDS/Dask, rootless Docker, RoCE/OFED, NGC-Ready workload, Slurm validation, and registry cache examples refreshed to current NGC and Kubernetes references; new Kubernetes RDMA deployments are pointed at the NVIDIA Network Operator with the DeepOps RoCE role retained as a legacy direct-host path (#1356).
• Jenkins-era CI harness and Vagrant virtual lab documented as legacy/community-supported reference material (#1353).

Agent operability

• New AGENTS.md operating guide: repository map, first-time setup, Slurm and Kubernetes golden paths, operating rules, and known gotchas that read as failures but are not (#1360).
• New scripts/validation/ machine-readable validation tools for automation and AI agents: deepops_doctor.py provisioning preflight, validate_slurm.py, and validate_k8s.py, each with --json flat output, explicit failure lists, and strict exit codes (#1359).
• New skills/ directory with four reusable procedures (deploy-slurm-cluster, deploy-k8s-gpu-cluster, validate-gpu-cluster, diagnose-driver-install) in the cross-tool agent-skills format (#1360).

Fixes

• parse_manifest.py firmware version parsing no longer aborts on a non-dict parseable manifest line; adds a regression test (#1355).

Known issues and validation notes

• Component support boundaries recorded for this release: registry/cache and the refreshed workload examples are supported paths; the RoCE backend role is a documented legacy direct-host path; Open OnDemand, the OpenShift helper role, and the Singularity wrapper are community-supported, with ownership calls planned next release.
• Release validation ran on single-node GPU hardware: fresh Slurm deployment with a GPU job, fresh Kubernetes deployment with GPU Operator and a CUDA pod, and a MAAS provisioning smoke. Multi-node scheduling and Network Operator install behavior on RDMA-capable networking hardware were not covered in this cycle.
• Direct in-place Kubernetes upgrades from 23.08 remain blocked by Kubespray's Calico minimum-version prechecks; use the documented staged upgrade or redeploy path for much older clusters.

26.05

DeepOps 26.05 Release Notes

DeepOps 26.05 is the first DeepOps release since 23.08. It refreshes the Kubernetes, Slurm, GPU, monitoring, provisioning, and supporting dependency stack while preserving the existing DeepOps release model of tagging master.

General

• Updated software component pins and documentation for the 26.05 release.
• Updated Ansible role dependencies for current ansible-core compatibility.
• Updated documentation references for current Kubespray inventory group names.
• Continued to keep release artifacts public-facing and free of environment-specific infrastructure details.

Kubernetes

• Updated Kubespray to v2.31.0.
• Updated Kubernetes inventory group references from legacy kube-master / kube-node names to current Kubespray group names.
• Fixed the NFS client provisioner playbook include path for current Ansible variable typing.
• Updated Helm installation handling so the installer runs with Bash and leaves the installed binary executable.
• Updated ingress and monitoring chart pins.

GPU Stack

• Updated GPU Operator to v26.3.1.
• Updated the default GPU Operator driver version to 580.126.20.
• Added Ubuntu open-kernel-module package selection through nvidia_driver_ubuntu_use_open_kernel_modules.
• Updated NVIDIA device plugin and GPU feature discovery chart versions to 0.19.1.
• Updated NVIDIA Network Operator to 26.1.1 and adjusted the role for the current chart behavior.
• Updated MIG Manager package URLs.

Slurm

• Updated Slurm to 25.11.6.
• Improved Slurm controller idempotency for repeated sacctmgr account and user creation.
• Ran login/compute GPU visibility setup with Bash so pipefail handling works consistently.

MAAS

• Updated the MAAS role dependency to the maintained upstream role revision used by this release.
• Updated MAAS examples for MAAS 3.5.
• Added maas_python_reqs defaults so pip packages do not shadow the packaged MAAS CLI.

Monitoring And Supporting Components

• Updated Prometheus, Alertmanager, Grafana, node exporter, cache, Spack, Helm chart, and container image defaults.
• Updated NetApp Trident documentation references.

Upgrading

• Re-run ./scripts/setup.sh after updating to this release.
• Compare your existing config/ directory against config.example/ and carry forward new or changed variables.
• Kubernetes upgrades from much older DeepOps releases may need staged upgrades through compatible intermediate Kubespray and network-plugin versions. Direct upgrades from 23.08-era Kubernetes deployments can stop in Kubespray prechecks when the deployed Calico version is older than the current minimum. Use staged upgrades or redeploy for older clusters instead of assuming a direct jump will work.
• For Slurm, consult SchedMD upgrade guidance before upgrading existing accounting and controller state across major releases.
• For GPU Operator and Network Operator upgrades, consult upstream release notes for CRD and chart behavior changes before upgrading existing clusters.

Validation

• Public CI passed for the 26.05 version-bump PR.
• Release validation covered role linting, selected playbook syntax checks, fresh Slurm deployment, fresh Kubernetes/GPU Operator deployment, MAAS provisioning, and upgrade-path behavior from 23.08-era Kubernetes.

Changes

• #1344 - chore(release): bump component versions for 26.05
• #1345 - release: 26.05 notes and README tag

23.08

Merge pull request #1296 from dholt/release-23.08

Release 23.08

22.08

DeepOps 22.08 Release Notes

Known Issues

• Kubeflow deployment is currently broken due to incompatibility between current Kubeflow and Kubernetes 1.22. Kubeflow deployment will be updated to add support when Kubeflow releases 1.6.

General

• Re-work of large portion of documentation
• Updates to NCCL tests
• Various bug fixes

Slurm

• Update to Slurm 22.05.2
• Add Alertmanager integration
• Option to share Slurm configuration among nodes via NFS
• Enhancements to Slurm re-install/re-build tasks

Kubernetes

• Update to Kubernetes 1.24.4
• Update to GPU Operator 1.11.1 (GPU driver branch 515)

Changes

Bugs/Enhancements

• Update NVIDIA driver role (#1216)
• Update Kubespray submodule URL (#1200)
• Add Alertmanager to Slurm cluster deployment (#1198)
• Fix Slurm configuration GRES syntax (#1196)
• Update Pyxis image cache size (#1191)
• Updates to documentation (#1188)
• Fix Slurm reinstall/rebuild tasks (#1187)
• Update MetalLB helm repo (#1185)
• Update EPEL GPG key (#1184)
• Add option to share Slurm configuration among nodes (#1182)
• Update NCCL tests (#1180, #1209)
• Netapp Trident fix PATH (#1176)
• Update default Slurm version to 21.08.8 (#1169, #1171)
• Update NVIDIA signing key (#1166, #1167)
• Update Ansible (#1165)

Upgrade Steps

If you are upgrading to this version of DeepOps from a previous release you will need to follow the upgrade section of the Slurm or Kubernetes Deployment Guides. In addition to this, the ./scripts/setup.sh script must be re-run and any new variables in the config.example files should be added to the existing config. For a full diff from release 22.04 run git diff 22.04 22.08 -- config.example/. If you encounter problem please open a GitHub issue. See the update guide for additional guidance.

Notes

22.04.2

DeepOps 22.04.2 Release Notes

Bugfix release - 22.04.2

SchedMD released Slurm 21.08.8 in order to address multiple CVEs. As part of this process, they un-published earlier versions of Slurm. This prevents earlier releases of DeepOps from installing with the default value of slurm_version.

Release 22.04.2 updates the default version of slurm_version to point to the latest available version, which should install successfully. (#1171)

Bugfix release - 22.04.1

NVIDIA rotated the signing keys for the CUDA repositories on April 27, breaking installs from DeepOps 22.04 released a few days prior.

This release, 22.04.1, starts from 22.04 and adds PR #1167 to handle the updated key.

The previous release notes from 22.04 appear below.

Known Issues

• Kubeflow deployment is currently broken due to incompatibility between current Kubeflow and Kubernetes 1.22. Kubeflow deployment will be updated to add support when Kubeflow releases 1.6. See #1147.

General

• Extensive improvements to automated testing with Jenkins, Ansible Molecule, and ansible-lint
• Update MIG playbook to use the new nvidia-mig-manager systemd service
• Updates to roles for nvidia-docker and GPU driver
• Various bug fixes

Slurm

• Enhanced NCCL tests for Slurm cluster validation
• Make use of pam_slurm_adopt optional
• Break out multiple sections in Slurm inventory file

Kubernetes

• Update to Kubernetes 1.22.6
• Update default container runtime from dockershim to containerd
• Add support for NVIDIA Network Operator
• Add support to deploy NVIDIA Deep Learning Examples on Kubernetes clusters
• Update to GPU Operator 1.9

Changes

Bugs/Enhancements

• Fixes for rsyslog server role (#1096, #1098)
• Update NetApp Trident default version number and branding (#1105)
• Introduce a common script library (#953)
• Update versions of monitoring stack components (#1107)
• Updates to Jenkins testing (#1112, #1127, #1133, #1137, #1138, #1139, #1150, #1151)
• Fixes for setup script (#1114)
• Automated testing of DeepOps roles using Molecule (#1094, #1116, #1158)
• Update nvidia.nvidia_docker role to v1.2.4 (#1121)
• Automated deployment of Deep Learning Examples (#1083, #1145)
• Make it optional to use pam_slurm_adopt (#1111)
• Convert MIG playbook to use nvidia-mig-manager service (#1106)
• Update to GPU Operator 1.9 (#1074)
• Automatically run ansible-lint on each role (#1129)
• Update Kubeflow deployment script to Kubeflow 1.4 (#1104)
• Remove old build dirs during Slurm upgrade (#1101)
• Fixes to ood-wrapper role (#1125)
• Documentation of network ports (#1126)
• Set missing defaults in playbooks (#1134)
• Update to Kubespray v2.18.1 and containerd (#1043, #1141)
• Fix GPU Operator config (#1136)
• Break out functional host groups in Slurm inventory (#1087)
• Fix ordering in k8s cluster deployment (#1128)
• Update nvidia.nvidia_driver role to v2.2.0 (#1143, #1160)
• Add support for NVIDIA Network Operator (#1113, #1156)
• Enhanced NCCL tests for Slurm validation (#1042)
• Fix git.io shortlinks (#1163)
• Check for SELinux disabled in SELinux tasks (#1162)

Upgrade Steps

If you are upgrading to this version of DeepOps from a previous release you will need to follow the upgrade section of the Slurm or Kubernetes Deployment Guides. In addition to this, the ./scripts/setup.sh script must be re-run and any new variables in the config.example files should be added to the existing config. For a full diff from release 22.01 run git diff 22.01 22.04 -- config.example/. If you encounter problem please open a GitHub issue. See the update guide for additional guidance.

Notes

22.04.1

DeepOps 22.04.1 Release Notes

Bugfix release

NVIDIA rotated the signing keys for the CUDA repositories on April 27, breaking installs from DeepOps 22.04 released a few days prior.

This release, 22.04.1, starts from 22.04 and adds PR #1167 to handle the updated key.

The previous release notes from 22.04 appear below.

Known Issues

• Kubeflow deployment is currently broken due to incompatibility between current Kubeflow and Kubernetes 1.22. Kubeflow deployment will be updated to add support when Kubeflow releases 1.6. See #1147.

General

• Extensive improvements to automated testing with Jenkins, Ansible Molecule, and ansible-lint
• Update MIG playbook to use the new nvidia-mig-manager systemd service
• Updates to roles for nvidia-docker and GPU driver
• Various bug fixes

Slurm

• Enhanced NCCL tests for Slurm cluster validation
• Make use of pam_slurm_adopt optional
• Break out multiple sections in Slurm inventory file

Kubernetes

• Update to Kubernetes 1.22.6
• Update default container runtime from dockershim to containerd
• Add support for NVIDIA Network Operator
• Add support to deploy NVIDIA Deep Learning Examples on Kubernetes clusters
• Update to GPU Operator 1.9

Changes

Bugs/Enhancements

• Fixes for rsyslog server role (#1096, #1098)
• Update NetApp Trident default version number and branding (#1105)
• Introduce a common script library (#953)
• Update versions of monitoring stack components (#1107)
• Updates to Jenkins testing (#1112, #1127, #1133, #1137, #1138, #1139, #1150, #1151)
• Fixes for setup script (#1114)
• Automated testing of DeepOps roles using Molecule (#1094, #1116, #1158)
• Update nvidia.nvidia_docker role to v1.2.4 (#1121)
• Automated deployment of Deep Learning Examples (#1083, #1145)
• Make it optional to use pam_slurm_adopt (#1111)
• Convert MIG playbook to use nvidia-mig-manager service (#1106)
• Update to GPU Operator 1.9 (#1074)
• Automatically run ansible-lint on each role (#1129)
• Update Kubeflow deployment script to Kubeflow 1.4 (#1104)
• Remove old build dirs during Slurm upgrade (#1101)
• Fixes to ood-wrapper role (#1125)
• Documentation of network ports (#1126)
• Set missing defaults in playbooks (#1134)
• Update to Kubespray v2.18.1 and containerd (#1043, #1141)
• Fix GPU Operator config (#1136)
• Break out functional host groups in Slurm inventory (#1087)
• Fix ordering in k8s cluster deployment (#1128)
• Update nvidia.nvidia_driver role to v2.2.0 (#1143, #1160)
• Add support for NVIDIA Network Operator (#1113, #1156)
• Enhanced NCCL tests for Slurm validation (#1042)
• Fix git.io shortlinks (#1163)
• Check for SELinux disabled in SELinux tasks (#1162)

Upgrade Steps

If you are upgrading to this version of DeepOps from a previous release you will need to follow the upgrade section of the Slurm or Kubernetes Deployment Guides. In addition to this, the ./scripts/setup.sh script must be re-run and any new variables in the config.example files should be added to the existing config. For a full diff from release 22.01 run git diff 22.01 22.04 -- config.example/. If you encounter problem please open a GitHub issue. See the update guide for additional guidance.

Notes

22.01

DeepOps 22.01 Release Notes

General

• Updates for Slurm and Kubernetes
• Bug fixes

Slurm

• Slurm version 21.08.5
• HPC SDK 22.1
• Open OnDemand v2.0.9
• CUDA toolkit 11.5
• Slurm Pyxis plugin 0.11.1
• Enroot container runtime v3.2.0
• Hwloc 2.5.0, pmix 3.2.3
• Spack v0.16.2

K8s

• Kubernetes version v1.20.7 (kubespray v2.17.1)
• Helm version v3.7.1
• GPU Operator v1.8.2 (GPU driver 470.57.02)
• GPU Device Plugin v0.9.0
• GPU Feature Discovery v0.4.1
• NFS Client Provisioner v4.0.13

Changes

Bugs/Enhancements

• Add new HPL files for DGX A100 (#1047)
• Fix vagrant_startup.sh on Ubuntu 20.04 (#1049)
• Improve documentation and playbook for DGX firmware upgrade (#1058)
• Update firmware docs (#1063)
• Fix python interpreter (#1061)
• GPU Operator automation with NVIDIA AI Enterprise (#1059)
• [Open OnDemand] Remove task for ood_auth_map.regex permisisons (#1068)
• Change default Interpreter in Ansible system default instead of Python3 (#1078)
• Add Log4Shell mitigation to ES statefulset example (#1080)
• Default to testing in Ubuntu 20.04 (#1051)
• Update k8s logging doc to use Elastic stack (#1081)
• Rewrite of DeepOps update documentation (#1050)
• Update Slurm ElasticSearch logging playbook for log4shell (#1079)
• Introduce a common script library, config for env vars, and inject these into all scripts (#953)
• Add proxy config to standalone container registry (#1090)
• Stop systemd-resolved on Ubuntu 20.04 (#1089)
• Add Molecule testing for Singularity, plus infra for more roles (#1088)

Upgrade steps

If you are upgrading to this version of DeepOps from a previous release you will need to follow the upgrade section of the Slurm or Kubernetes Deployment Guides. In addition to this, the ./scripts/setup.sh script must be re-run and any new variables in the config.example files should be added to the existing config. For a full diff from release 21.09 run git diff 21.09 22.01 -- config.example/. If you encounter problem please open a GitHub issue. See the update guide for additional guidance.

Notes

21.09

DeepOps 21.09 Release Notes

What's New

Release 21.09 is mostly a bug fix release

General

• Support for DGX OS 5 in nvidia-dgx role

Slurm

• Slurm version 21.08.1
• HPC SDK 21.9
• Open OnDemand v2.0.9
• CUDA toolkit 11.4
• Slurm Pyxis plugin 0.11.1
• Enroot container runtime v3.2.0
• Hwloc 2.5.0, pmix 3.2.3
• Spack v0.16.2

K8s

• Kubernetes version v1.20.7 (kubespray v2.16.0)
• Helm version v3.5.4
• GPU Operator v1.8.2 (GPU driver 470.57.02)
• GPU Device Plugin v0.9.0
• GPU Feature Discovery v0.4.1
• NFS Client Provisioner v4.0.13

Changes

• Docker version 20.10

Bugs/Enhancements

• Improved cleanup in Slurm epilog (#965)
• Fix disabling NVIDIA driver install on Slurm cluster install (#948)
• Permit SFTP in default SSHD config (#980)
• Address different possible DCGM service names depending on version (#983)
• Fix PAM Slurm adopt/login (#989)
• Enroot: adjust cache directory to be per-user (#997)
• Adding proxy support for downloading of hwloc, pmix, nhc and slurm (#1002)
• Remove broken offline deployment support and clarify documentation (#1012)
• Grafana: add var for custom config template (#994)
• EasyBuild: Enable both shells on all distros (#993)
• Default to building Slurm with dynamic libs (#1021)
• ood-wrapper: Don't install python3-passlib on CentOS 7 (#995)
• Update ansible-role-enroot to 0.5.0 (#1030)

Upgrade steps

If you are upgrading to this version of DeepOps from a previous release you will need to follow the upgrade section of the Slurm or Kubernetes Deployment Guides. In addition to this, the ./scripts/setup.sh script must be re-run and any new variables in the config.example files should be added to the existing config. For a full diff from release 21.06 run git diff 21.06 21.09 -- config.example/. If you encounter problem please open a GitHub issue. See the update guide for additional guidance.

Notes

21.06

DeepOps 21.06 Release Notes

What's New

General

• Documentation-based support for NGC Ready configuration in offline environment
• New setup script (see notes below [1])
• Rootless Docker support
• Update burn-in test to use NGC container (v21.4)
• UFM OS added to packer-maas repo

Slurm

• Slurm version 20.11.7 (previously 20.11.3) (NOTE: Slurm versions prior to 20.11.7 are affected by CVE-2021-31215 [2])
• HPC SDK 21.3 (previously 21.2)
• Open OnDemand support for Ubuntu 20.04
• Open OnDemand v1.8.20
• Playbook for single-node Slurm cluster
• CUDA toolkit 11.3
• Singularity 3.7.3
• Slurm Pyxis plugin 0.9.1
• Hwloc 2.4.1, pmix 3.2.3
• Spack v0.16.1

K8s

• Kubernetes version v1.19.9 (kubespray v2.15.1)
• Helm version v3.5.3 (previously v3.4.1)
• GPU Operator v1.6.0(previously v1.5.2)
• GPU Device Plugin v0.9.0 (previously v0.8.2)
• GPU Feature Discovery v0.4.1
• Update Trident deployment role to use Helm chart
• Update k8s examples to run on A100

Changes

• Move nvidia-peer-memory logic into a role
• Option to allow force install of GPU driver
• Install DCGM via CUDA repos
• Change namespace of k8s ingress controller
• Simplify GPU Operator support and change vGPU deployment method
• Update Triton Kubeflow pipeline to leverage nfs-client and download examples
• Ansible version 2.9.21

Bugs/Enhancements

• Update MIG playbook to enable MIG per device rather than all
• Fixes for DGX firmware update role
• Reorganize Slurm config file
• Various fixes to QA tests
• Documentation updates
• Improve NHC checks for DGX A100
• Skip automatic re-installation of NV HPC SDK
• Use correct ns when checking helm status of metallb
• Install python-docker/docker-py via yum vs. pip
• Updates/fixes for DCGM exporter
• Correctly install docker python SDK

Upgrade steps

If you are upgrading to this version of DeepOps from a previous release you will need to follow the upgrade section of the Slurm or Kubernetes Deployment Guides. In addition to this, the ./scripts/setup.sh script must be re-run and any new variables in the config.example files should be added to the existing config. For a full diff from release 21.03 run git diff 21.03 21.06 -- config.example/. If you encounter problem please open a GitHub issue. See the update guide for additional guidance.

Notes

This
[1] This release makes significant modifications to the control machine setup script (scripts/setup.sh) with the goal of making fewer modifications to the system and conflicting less with existing software. Re-running the setup script will create a Python virtual environment where Ansible and other dependencies are installed (except for any required system packages). The script attempts to make this new virtual environment part of the user's path, but some manual intervention may be required to use the newly installed Ansible version.

[2] https://www.schedmd.com/archives.php

21.03

DeepOps 21.03 Release Notes

What's New

General

• Rsyslog client/server for K8s & Slurm deployments
• Examples for running Ansible and configuring Inventory file
• Improved support for Ubuntu 20.04 and CentOS 8
• Docker login convenience playbook
• Marked air-gap as "experimental"
• Vagrant/virtual 2.2.14 (previously 2.2.3)

Slurm

• Slurm version 20.11.3 (previously 20.02.4)
• HPC SDK 21.2 (previously 2020_207)

K8s

• Helm version v3.4.1 (previously v3.1.2)
• NFS Client Provisioner as K8s Default StorageClass
• GPU Operator v1.5.2(previously v1.1.7)
• GPU Device Plugin v0.8.2 (previously v0.7.0)
• GPU Feature Discovery v0.4.1 (previously v0.2.0)
• Example NGC Dockerfiles bumped to 20.12 with improved documentation
• New example yaml files for launching single node/multi node training and jupyter notebooks
• RoCE perfromance playbook

Changes

• Deprecation of Rook-Ceph deployment script
• Removed default MPI Operator install for K8s
• NFS server is now deployed on kube-master[0] by default with path /export/deepops_nfs
• New log bundling tool (debug.sh) for K8s
• Enroot marked as "not fully automated" for CentOS (simple workaround is to bump enroot Ansible Galaxy role from v0.3.2 to v0.4.0 and re-run setup.sh)

Bugs/Enhancements

• K8s monitoring metrics now persist by default using NFS-backed PVs.
• Additional testing for Ubuntu 20.04, CentOS 8, GPU Operator, enroot, mpi, and testing.md
• Addressed firewall issues in CentOS
• Add vGPU support for GPU Operator installs
• Address intermittent download failures in Slurm install

Upgrade steps

If you are upgrading to this version of DeepOps from a previous release you will need to follow the upgrade section of the Slurm or Kubernetes Deployment Guides. In addition to this, the ./scripts/setup.sh script must be re-run and any new variables in the config.example files should be added to the existing config. For a full diff from release 20.12 run git diff 20.12 21.03 -- config.example/. Note, the majority of the config changes are around new functionality such as nfs-client-provisioner, rsyslog, and persistent monitoring metrics in K8s. If you encounter problem please open a GitHub issue. See the update guide for additional guidance.

Notes