Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions docs/labs/clab.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,15 +270,19 @@ Alternatively, add the same settings to the [user defaults file](defaults-user-f

During the **netlab up** process, *containerlab* starts the containers and reports success. The virtual machines in those containers may take minutes to start, and until they're ready and have initial device configuration (including a management IP address and the SSH server configuration), _netlab_ cannot continue the initial configuration process.

The *vrnetlab* containers usually report their *health state*, changing from `starting` to `healthy` once the VM running inside the container is configured. _netlab_ ignores the container health state by default (it always checks the SSH server availability, though), but you can set the **defaults.providers.clab.healthy_timeout** parameter to the maximum time you expect the containers to need to become healthy.
The *vrnetlab* containers usually report their *health state*, changing from `starting` to `healthy` once the VM running inside the container is configured. _netlab_ checks the container health state[^CHS] and the SSH server availability[^CSSH] by default, but you can set the **defaults.providers.clab.healthy_timeout** [topology default](topo-defaults) to zero to disable the health check, or to the maximum time you expect the containers to need to become healthy if you're starting large topologies on an underpowered server.

[^CHS]: For containers that provide it

[^CSSH]: For containers that run the SSH server

However, having the *vrnetlab* containers in *healthy* state (the VMs have initial configuration) does not guarantee their SSH servers are ready. _netlab_ thus performs an additional "*is the device ready*" check during the initial configuration process: it attempts to establish an SSH session with the device and run a command[^NLCC]. The SSH session is retried up to **netlab_check_retries** times (default: 20), with a delay of **netlab_check_delay** (default: 5), for a total of over 100 seconds. Each retry takes up to 10 seconds in case the TCP session is stuck in the SYN state.

[^NLCC]: Specified in the **netlab_check_command** variable; usually `show version`

The **netlab_check_retries** parameter is set higher in system defaults for virtual machines that are slow to boot (for example, **vjunos-router**), but if you're working on a slow system, you might have to increase it even further. Set the **netlab_check_retries** node variable to increase the number of retries for an individual node or set the **defaults.devices._device_.clab.group_vars.netlab_check_retries** variable to increase the number of retries for a specific device (see also [](topo-defaults) and [](defaults-user-file))

Alternatively, you could set the minimum wait time for container SSH servers (VM SSH servers are checked by Vagrant) with the **defaults.providers.clab.ssh_timeout** [system default](topo-defaults).
Alternatively, you could set the minimum wait time for container SSH servers (VM SSH servers are checked by Vagrant) with the **defaults.providers.clab.ssh_timeout** [topology default](topo-defaults).

Finally, if you have trouble starting very large topologies because of CPU usage spikes while containers or virtual machines boot, try [starting containers in batches](clab-batches).

Expand Down
1 change: 1 addition & 0 deletions netsim/providers/clab.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ kmods:
sr: [ mpls-router, mpls-iptunnel ]
vxlan: [ vxlan, udp_tunnel, ip6_udp_tunnel ]
vrf: [ vrf ]
healthy_timeout: 600 # Reasonably large default "wait for containers to become healthy" timeout
attributes:
node:
type: dict
Expand Down
2 changes: 1 addition & 1 deletion netsim/providers/clab/labops.py
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ def health_check(containers: list, timeout: int, topology: Box) -> None:
skip_header=True,
module='clab',
category=log.FatalError,
more_hints=['Increase or remove the defaults.providers.clab.health_timeout parameter'],
more_hints=['Increase or remove the defaults.providers.clab.healthy_timeout parameter'],
doc_url='labs/clab/#vrnetlab-wait')

def get_unhealthy_containers(containers: list,topology: Box) -> list:
Expand Down
Loading