problem
Upon any exception during VmStatsCleaner and VolumeStatsCleaner threads the scheduled task is not going to run again until the management servers are restarted.
Following is the related snippet to handle
https://github.com/shapeblue/cloudstack/blob/e3999923a7b7faf46cd584502baea9f106fef69a/server/src/main/java/com/cloud/server/StatsCollector.java#L2009-L2043
Observed the below error after which the threads are not running again
2026-07-19 12:44:43,622 ERROR [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-cb54c457]) (logid:494590e1) DB Exception on: HikariProxyPreparedStatement@311375727 wrapping com.mysql.cj.jdbc.ServerPreparedStatement[28]: DELETE FROM vm_stats WHERE vm_stats.timestamp < '2026-06-18 03:59:03' com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:175)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:555)
at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:339)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1320)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:994)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1305)
at com.cloud.utils.db.GenericDaoBase.batchExpunge(GenericDaoBase.java:1329)
at com.cloud.vm.dao.VmStatsDaoImpl.removeAllByTimestampLessThan(VmStatsDaoImpl.java:126)
at jdk.internal.reflect.GeneratedMethodAccessor566.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
versions
Observed in 4.20.2
The steps to reproduce the bug
- When MS is running, try to either lock the tables "vm_stats" and "volume_stats" or kill the mysql communication
- Observe that the table vm_stats and volume_stats piles up
What to do about it?
No response
problem
Upon any exception during VmStatsCleaner and VolumeStatsCleaner threads the scheduled task is not going to run again until the management servers are restarted.
Following is the related snippet to handle
https://github.com/shapeblue/cloudstack/blob/e3999923a7b7faf46cd584502baea9f106fef69a/server/src/main/java/com/cloud/server/StatsCollector.java#L2009-L2043
Observed the below error after which the threads are not running again
versions
Observed in 4.20.2
The steps to reproduce the bug
What to do about it?
No response