With the recent tombstonings and jailings that happened on dYdX v4, I wanted to open a discussion around proper monitoring, security, and migration of dYdX validators.
The greater cosmos ecosystem has a flourishing community that creates and maintains tooling around monitoring their validators, and I’ll get to those, but I first want to talk about best practices for validators. There’s a huge amount to be said here, but I’m going to focus on 2 key points:
- securing your
- securing your
This IS your validator. If there’s one thing I want you to take away from this post, it’s to backup your key, and never let it exist on more than 1 node at a time. Tombstoning (permanent jailing) happens when this key is used on multiple nodes at the same time, by accident or not.
This is the latest sign information of your validator. You want to protect this. If you’re migrating your validator, it’s not a bad idea to move this along side your
priv_validator_key.json in order to be absolutely certain you don’t double sign.
Steps to migrate your validator safely:
- let your current validator continue running
- sync a 2nd node to current
- stop your current validator (note that you’ll start missing blocks here - THAT’S OKAY!)
- move the
- restart the 2nd node (confirm your validator has started signing)
- delete the
priv_validator_key.jsonfrom your old validator
— if you’re intending to decommission your old dYdX node completely, you’re done! —
- if you intend to use the old validator as a backup, restart it
Use a remote signing solution. There are 2 options here:
These are critical because they separate the
priv_validator_state.json from the node itself, allowing you to migrate without worrying about double signing. The primary difference between the two solutions is that:
- TMKMS uses a single remote signer to talk to a single node. TMKMS supports HSM to protect your key such as the YubiHSM.
- Horcrux is a many:many solution; you can have many signers point to many nodes, where the signers act like a multisig. Horcrux does NOT support hardware key protection.
There are plenty of tools to monitor your signing, but Lavender.Five’s preferred method is Tenderduty. It’s very easy to set up, and can notify you via Pagerduty, Slack, Telegram, Etc. if your node isn’t signing.