This one bit me really bad. Oracle support could not not figure it out but thanks to Roddy Rodstein’s Oracle cloud cookbook, we fixed the problem
C-states allow an idle processor to turn off unused components to save power. Some of the components that C-states turns off include the processor clock and interrupts. Under certain conditions, when C-states turns off unused CPU components the Oracle VM OCFS2 heartbeat mechanism will trigger.
To validate if C-states is enabled, as root log into the to an Oracle VM Server and type:
# xenpm get-cpuidle-states | grep total | uniq
total C-states : 2
If the "total C-states" is greater than 2, then C-states is enabled in the server’s BIOS.