Background
The MEV Committee is a grants-funded initiative to help the community enforce a social mitigation strategy against malicious block proposers. The committee was assigned to actively monitor, analyze, and report any potential MEV activity, so that the community may respond appropriately to bad actors.
Monthly reports are shared with the community, offering insights into our on-chain findings, actions taken to address any issues, and improvements to our workflows. In case of malicious activity, these reports also detail the incident, the parties involved, and provide recommendations for any retroactive measures the community should consider. We welcome any feedback or questions on the report!
September Activity
In September, we identified a few high-discrepancy instances that warranted further investigation. After close examination, we identified a set of validators whose configurations were contributing to these discrepancies.
Additionally, a number of outlier blocks showing significant order book discrepancies were detected. Following detailed reviews, we concluded that none of these events were linked to malicious activity, but consistent with previous issues around network connectivity and validator peering.
Below, we provide an in-depth overview of some key cases to illustrate the type of discrepancy data we found and the processes we used to analyze them.
Empty Blocks
In the past month, we identified a trend involving validators proposing above-average amounts of blocks with no matches included, or ‘empty blocks’ as we call them. As a reminder, blocks proposed can include transactions (e.g. DYDX transfers), non-transfer related events (e.g. governance actions), market price updates, and orderbook activity (e.g. trade matches). Orderbook activity is needed to confirm user trades and update market orderbooks. As such, an above-average amount of empty blocks would indicate possible order matches being missed by the validator, and user orders not being executed in time.
As seen in the pictures above, the empty blocks percentage among some validators is above average when comparing them to all other validators. We contacted most of these teams and found issues with performance-related node configurations. We referred these teams back to our optimal validator configuration guideline, and found a few new parameters to adjust surrounding max peer counts and open connection limits.
We also suspect validators using remote signing configurations could be adding unnecessary latency to their proposal process, adding to the likelihood of proposing empty blocks. As a reminder, we recommend that all validators run with local signing instead.
Discrepancies
We analyzed the average order book discrepancy per validator over 7-day and 14-day rolling windows. To provide a clearer picture, we categorized validators based on their voting power.
Looking at the chart below, we will see that particular trends emerge over a set of days when comparing order book discrepancy per block amongst all validators. We would expect all validators to perform similarly, but there will be cases in which those validators who get to propose more blocks due to their voting power will have different averages in case of high volatility, so it makes sense to differentiate them.
For validators with 2.5% or more voting power, we observed notable trends over the past month. In particular, OKX Earn and Polychain stood out with higher-than-average order book discrepancies.
We analyzed a few blocks (e.g. 25479124 and 26364947), and found missing orders that were expected to be matched based on network activity. The timing of these discrepancies aligns with the trend of increased empty blocks, suggesting that resolving the empty block issue may also reduce order book discrepancies. We expect to see those discrepancies compared to other validators improve once the empty block situation is mitigated.
Similarly, validators in the mid-range voting power category exhibited order book discrepancies linked to the empty block trend.
For validators with lower voting power, the discrepancies were less frequent but more pronounced when they did occur, as proposing fewer blocks means that outliers have a greater impact on the moving average.
Conclusion
In September, the Committee continued its monitoring of block discrepancies across validators, identifying potential performance issues but no malicious activity. Validators proposing empty blocks and missing orders were of particular concern. Our findings suggest that these issues stem from network configuration problems and performance limitations.
We will continue working closely with validators to address these issues and will provide updates to ensure the integrity of the trading execution is maintained.