Stakecito Delegator Information — Redelegate to New Stakecito Validator

Validators
1

Dear delegators, unfortunately we just got informed this afternoon, that one of our providers this night restored a wiped server from a backup/snapshot, thus we got hard jailed on DYDX.

To continue supporting us we kindly ask to to redelegate to our new Stakecito validator: dYdX | Keplr Dashboard

Following is the Post mortem Issue report:

  • After the upgrade of the NextBlockDelqy to 200ms suddenly our validator did not sign blocks anymore (many other validators had the same issue). Current analysis showed that it was mainly due to peering issues (see other validator reports in Slack).

  • Validator was running with Horcrux MPC signer, 3 signers, 2 sentries (double sign protected).

  • Some validators recommended to switch to local signer, as latency could be too high between signers. Set double_sign_check_height on all nodes to „5“ to prevent double signing.

  • Switched off MPC signing and moved to local signer, Double-Sign parameter was set. Uptime did not get better.

  • Validator still could not keep up. Analysis showed, that the network throughput was limited, reported this to the provider, to analyze the issue. Server rebooted to rescue mode by provider.

  • Provider reported a firmware issue and server was down for analyzing further. Coordinated with provider to wipe the disks and to boot into rescue mode afterwards, as there is a double sign risk.

  • Moved local signer to backup server for signing, uptime recovered to 99%.

  • Provider upgraded firmware. Unfortunately, afterwards the provider recovered the wiped disks from a backup/snapshot (which leads to having the key on the server again), booted the server and despite having the „double_sign_check_height“ parameter set, we double signed and got ejected from the validator set.

Impact:

  • Got hard jailed, means not possible to unjail the validator. Only by creating a new validator or by governance proposal.

  • Hard jailing is meant for malicious behavior, no malicious behavior here.

  • No financial penalty for double signing as slash fraction doublesign is 0% in DYDX parameters.

Measures taken:

  • Created a new validator, communicate the matter and coordinate with the foundation and delegators to re-delegate to the new validator, as the double sign was no malicious behaviour.

  • Lessons learned.

New validator: https://www.mintscan.io/dydx/validators/dydxvaloper18jr2wx3qsu8g9gep6jypcxunx2jvcjquxk60wc

We would appreciate your continued support.

In case of any questions please let us know.

Thank you.

1 Like