Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
[id="persistent-storage-csi-vol-detach-non-graceful-overview_{context}"]
= Overview

[role="_abstract"]
A graceful node shutdown occurs when the kubelet's node shutdown manager detects the upcoming node shutdown action. Non-graceful shutdowns occur when the kubelet does not detect a node shutdown action, which can occur because of system or hardware failures. Also, the kubelet may not detect a node shutdown action when the shutdown command does not trigger the Inhibitor Locks mechanism used by the kubelet on Linux, or because of a user error, for example, if the shutdownGracePeriod and shutdownGracePeriodCriticalPods details are not configured correctly for that node.

With this feature, when a non-graceful node shutdown occurs, you can manually add an `out-of-service` taint on the node to allow volumes to automatically detach from the node.
Original file line number Diff line number Diff line change
Expand Up @@ -7,23 +7,25 @@
[id="persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure_{context}"]
= Adding an out-of-service taint manually for automatic volume detachment

[role="_abstract"]
To allow volumes to detach automatically from a node after a non-graceful node shutdown, you can add an out-of-service taint.

.Prerequisites

* Access to the cluster with cluster-admin privileges.

.Procedure

To allow volumes to detach automatically from a node after a non-graceful node shutdown:

. After a node is detected as unhealthy, shut down the worker node.

. Ensure that the node is shutdown by running the following command and checking the status:
+
[source,terminal]
----
$ oc get node <node_name> <1>
$ oc get node <node_name>
----
<1> <node_name> = name of the node that shut down non-gracefully
+
* `<node_name>` = name of the node that shut down non-gracefully

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per the SSG: Use a simple sentence to explain or describe a single line of command, variable, option, or parameter.

Suggested change
* `<node_name>` = name of the node that shut down non-gracefully
Use the `<node_name>` to specify the node that shut down non-gracefully.

+
[IMPORTANT]
====
Expand All @@ -34,22 +36,21 @@ If the node is not completely shut down, do not proceed with tainting the node.
+
[IMPORTANT]
====
Tainting a node this way deletes all pods on that node. This also causes any pods that are backed by
statefulsets to be evicted, and replacement pods to be created on a different node.
Tainting a node this way deletes all pods on that node. This also causes any pods that are backed by statefulsets to be evicted, and replacement pods to be created on a different node.
====
+
[source,terminal]
----
$ oc adm taint node <node_name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute <1>
$ oc adm taint node <node_name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
----
<1> <node_name> = name of the node that shut down non-gracefully
+
After the taint is applied, the volumes detach from the shutdown node allowing their disks to be attached to a different node.
* `<node_name>` = name of the node that shut down non-gracefully

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* `<node_name>` = name of the node that shut down non-gracefully
Use the `<node_name>` to specify the node that shut down non-gracefully.

+
.Example
After the taint is applied, the volumes detach from the shutdown node allowing their disks to be attached to a different node.
+
The resulting YAML file resembles the following:
The resulting YAML file resembles the following example file:
+
.Example node YAML file with out-of-service taint applied
[source, yaml]
----
spec:
Expand All @@ -65,6 +66,7 @@ spec:
+
[source, terminal]
----
$ oc adm taint node <node_name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute- <1>
$ oc adm taint node <node_name> node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-
----
<1> <node_name> = name of the node that shut down non-gracefully
+
* `<node_name>` = name of the node that shut down non-gracefully

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* `<node_name>` = name of the node that shut down non-gracefully
Use the `<node_name>` to specify the node that shut down non-gracefully.

Original file line number Diff line number Diff line change
@@ -1,13 +1,16 @@
:_mod-docs-content-type: ASSEMBLY
[id="ephemeral-storage-csi-vol-detach-non-graceful-shutdown"]
= Detach volumes after non-graceful node shutdown
include::_attributes/common-attributes.adoc[]
:toc:
:toc-title:
:context: ephemeral-storage-csi-vol-detach-non-graceful-shutdown
include::_attributes/common-attributes.adoc[]

toc::[]

[role="_abstract"]
This feature allows drivers to automatically detach volumes when a node goes down non-gracefully.

toc::[]

include::modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-overview.adoc[leveloffset=+1]

include::modules/persistent-storage-csi-vol-detach-non-graceful-shutdown-procedure.adoc[leveloffset=+1]