Missed Heartbeat

What are Missed Heartbeat alerts?

For multi-party signing, liveness is signalled via a “heartbeat” response that is sent every 50 blocks. Occasionally, issues related to validator companion processes can cause a validator to fail to submit a heartbeat. Note that if a validator fails to submit a heartbeat, they stop accruing rewards from multi-party signing until they submit their next heartbeat (minimum of 50 blocks). To inform users about potential issues, we provide an alert when a validator misses more than x heartbeats, where x is the alert threshold set by the user (default value: 1).

How are missed heartbeat alerts triggered?

Missed heartbeat alerts are evaluated every 5 minutes (i.e. ~50 blocks). Once our alerting system identifies that a validator has missed more than x heartbeats within the last hour, an alert will be triggered.

What notifications will I receive?

You will receive a single notification per validator you are subscribed to informing you when a heartbeat issue has occurred. If the issue persists two consecutive periods, you will not receive any additional notifications, but its status will remain “Firing” in the Metrika platform. Once the issue is resolved (your validator missed at most x heartbeats within the past hour), you will receive a single notification informing you about the resolution of the heartbeat issue.

How do I resolve it?

There may be various root causes causing this alert. Here are some recommended checks on your validator’s health to identify the potential issue:

• Check if vald process is up and running,

  • if vald is down, try restarting it, that usually helps (assuming auto-restart is not enabled)

  • check logs before process crashed, see if you can identify notable issues

• Check network connectivity to validator node or an Axelar full node

• Search for anomalies in log such as error message, warnings, panics

  • if issues found, please report it on Axelar Discord, to private team channel or validator channel