Background
The MEV Committee is a grants-funded initiative to help the community enforce a social mitigation strategy against malicious block proposers. The committee was assigned to actively monitor, analyze, and report any potential MEV activity, so that the community may respond appropriately to bad actors.
Monthly reports are shared with the community, offering insights into our on-chain findings, actions taken to address any issues, and improvements to our workflows. In case of malicious activity, these reports also detail the incident, the parties involved, and provide recommendations for any retroactive measures the community should consider. We welcome any feedback or questions on the report!
Activity
In October, we identified a few high-discrepancy instances that warranted further investigation. More importantly, we are noticing a concerning trend among a handful of validators that we believe merits notifying the community. As always, the committee has been in touch with validators, digging through configurations and setups to better understand the problems identified.
Empty Blocks and Matching Trends
The concerning trend identified is a growing rate of ‘empty blocks’ among a few validators, of which the most problematic have been the Meria validators. In this context, ‘empty blocks’ are defined as block proposals that have no order matches included in their proposed operations. In other words, these blocks do not execute any user trades. More importantly, user orders were not matched when possible matches did exist in the mempool, implying that these validators do not have an accurate view of open orders and possible execution paths.
Below, we see a 7 day moving average for the number of empty blocks proposed. While most validators hover around the same percentage, a few trend considerably higher. In the particular case of Polychain and other validators, conversations during the last months led to some improvements on their configuration, which led to an improvement in their number of empty blocks.
This led us to further investigation, which uncovered additional concerning trends. For context, there exists today two particularly active trading accounts. Over a period of 1976503 blocks, the two accounts made up the following:
- 531828 (26.91%) blocks included matches with dydx14dltc2w6y3dhf0naz8luglsvjt0vhvswm2j6d0 (“dydx14dlt”)
- 247916 (12.54%) blocks included matches with dydx15m3lvgfwe4xad7wqyskvn6qz5w5ahue60hhemn (“dydx15m3”)
These stats imply that for any given validator, we should expect to see 27% of blocks proposed include matches with dydx14dlt and 13% of blocks proposed include matches with dydx15m3. However, we find the following to be true for both Meria validators:
- Less than 5% of blocks proposed by the Meria validators include matches with dydx14dlt.
- More than 40% of blocks proposed by the Meria validators include matches with dydx15m3.
We have been in contact with the Meria team and have reviewed both their configuration files and peering networks with the hope of finding a cause for these discrepancies. The issues initially persisted following our recommended changes to their configurations. Recently, the Meria team have conducted additional changes to their mempool configurations, specifically with regards to their ttl-num-blocks parameter, which was previously still set to 0. As mentioned in our validator guidelines, this amount should be set to 20 so as to recycle stale orders, freeing up the mempool for new active orders. Recently updated to 20, we are now seeing a significant improvement to the empty blocks proposed, as seen below:
We still see, however, that the number of blocks proposed by the Meria validators including matched orders by dydx15m3 are still well above average on a daily basis, although the number of blocks with no matches by dydx14dlt has decreased. The reason for these order matching tendencies is still being investigated, but we suspect it’s a result of peering and order placing habits. If a user has a more direct peering route, and submits more orders per blocks, they’re more likely to be included in the next block.
High Discrepancy Blocks
In the past month, we also witnessed an above-average discrepancy block. Block 27194807 was proposed by Polychain with over $200k in orderbook discrepancies. Here is our assessment of the block:
A user sell order for 200 BTC at $60,024.00 with a conditional price of $63,215 was triggered at block 27194807. According to our node, the order should have been matched with open buy orders at an average price of $62,294.00 for a total value of $12,458,945.18. Instead, the block matched the order at an average price of $61,946.00 for a total value of $12,389,371.59. The $200k discrepancy reported on Observatory’s dashboard implies that other nodes averaged even better prices for the order.
This difference in discrepancies between our node and the discrepancy reported by Observatory’s dashboard does also suggest a high amount of last-minute orders. In fact, the bulk of the difference can be attributed to a single buy order submitting roughly two seconds before block proposal, which was missed by the proposer. As such, we can confidently attribute this discrepancy to latency and networking issues.
Finally, we present below trending averages for orderbook discrepancies across the validator set. We would expect all validators to perform similarly, but we can also expect slight differences across voting power given the number of blocks proposed. Averages might vary as a result, so we present charts with validators grouped by voting power.
For validators with 2.5% or more voting power, we observed notable trends over the past month. In particular, we see a huge spike by Polychain which is due to the analyzed block.
For the 1 to 2.5% voting power validators, we see a spike by Ledger by Meria, due to the discussed issues, as well as an above average discrepancy by Pareto, which seemed to be temporary as in November they’re in an average range.
Finally, in the lower percentage in VP, we do not see any concerning trends which would require further comments.
Conclusion
In October and November, the Committee continued its monitoring of block discrepancies across validators, identifying potential performance issues and some potential malicious activity. Our findings suggest that empty blocks were a result of misconfiguration at the node level, and we are seeing improvements following changes. Order matching trends indicate possible issues with peering.
