We had an unexpected outage today. The actual cause is unknown and was reported to the team. The symptoms were that the node lost sync suddenly and was unable to catch up. It then subsequently start throwing MailBox closure errors. Figment reported the same issue.
An attempt was made to failover to our fully synced backup node, however, after moving the keys and restarting the node it could not find any peers, even after modifying the boot_nodes and adding 9 additional peers.
Finally, we failed back to the primary node and allowed it to sync headers. We have lowered our pools commission rate to 0.1% for the next 30 days to assist our delegators.