Clean orphaned ifstate.veth* files on Ubuntu 14.04 (#3007)

Ubuntu 14.04 has a well-known bug that leaves orphaned `ifstate.veth*` files in `/run/network` when creating / removing a virtual ethernet interface pair with `ip link add name veth1 type veth peer name veth2` and `ip link delete veth1`. Docker containers use these `veth` pairs: when a Docker container is created, Docker creates a pair and removes it when the container is removed. If a lot of containers are created, the file system where `/run/network` resides, runs out of inodes so that no more containers can be created.

Invokers running on Ubuntu 14.04 VMs can be affected by the described situation if a lot of cold invocations are run.

Remove orphaned `ifstate.veth*` files older than 60 minutes in `/run/network` on Ubuntu 14.04 when cleaning an invoker host. The cleaning commands always have to be run as `root` because other users are not allowed to remove the files.

Author: sven-lange-last

ansible/roles/invoker/tasks/clean.yml

@@ -38,3 +38,14 @@
     path: "{{ whisk_logs_dir }}/invoker{{ groups['invokers'].index(inventory_hostname) }}"
     state: absent
   become: "{{ logs.dir.become }}"
+
+# Workaround for orphaned ifstate.veth* files on Ubuntu 14.04
+# See https://github.com/moby/moby/issues/22513
+# Remove inactive files older than 60 minutes
+- name: "Clean orphaned ifstate.veth* files on Ubuntu 14.04"
+  shell: |
+    ACTIVE_IFACES_REGEXP=$(ip -oneline link show | grep --only-matching --extended-regexp 'veth[0-9a-f]+' | tr '\n' '|' | sed -e 's/.$//' | xargs -I '{}' /bin/echo '/run/network/ifstate\.({})')
+    find /run/network -regextype posix-egrep \( -not -regex ${ACTIVE_IFACES_REGEXP} \) -and -name 'ifstate.veth*' -and -mmin +60 -delete
+  ignore_errors: True
+  become: True
+  when: ansible_distribution == 'Ubuntu' and ansible_distribution_version == '14.04'