Skip to content

VM and Volume stats cleaner threads gets killed after any exception #13648

Description

@harikrishna-patnala

problem

Upon any exception during VmStatsCleaner and VolumeStatsCleaner threads the scheduled task is not going to run again until the management servers are restarted.

Following is the related snippet to handle

https://github.com/shapeblue/cloudstack/blob/e3999923a7b7faf46cd584502baea9f106fef69a/server/src/main/java/com/cloud/server/StatsCollector.java#L2009-L2043

Observed the below error after which the threads are not running again

2026-07-19 12:44:43,622 ERROR [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-cb54c457]) (logid:494590e1) DB Exception on: HikariProxyPreparedStatement@311375727 wrapping com.mysql.cj.jdbc.ServerPreparedStatement[28]: DELETE FROM vm_stats WHERE vm_stats.timestamp < '2026-06-18 03:59:03' com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
	at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:175)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
	at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:555)
	at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:339)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1320)
	at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:994)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
	at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1305)
	at com.cloud.utils.db.GenericDaoBase.batchExpunge(GenericDaoBase.java:1329)
	at com.cloud.vm.dao.VmStatsDaoImpl.removeAllByTimestampLessThan(VmStatsDaoImpl.java:126)
	at jdk.internal.reflect.GeneratedMethodAccessor566.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

versions

Observed in 4.20.2

The steps to reproduce the bug

  1. When MS is running, try to either lock the tables "vm_stats" and "volume_stats" or kill the mysql communication
  2. Observe that the table vm_stats and volume_stats piles up

What to do about it?

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    Fields

    No fields configured for Bug.

    Projects

    Status
    No status

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions