Lessons From The Fuzzing Trenches
Lessons From The Fuzzing Trenches
Lessons learned from building a fuzzing suite for Renzo Protocol
Any fuzzing engagement is typically comprised of two parts: achieving coverage and actually fuzzing properties that you’ve defined on the system. Over the course of a few weeks I worked on achieving the former for a fuzzing scaffolding I created for Renzo protocol and have learned valuable lessons about fuzzing in general which I’ll share here.
Note: this engagement was not paid for or endorsed in any way by Renzo, it was meant as a practical research experiment to develop techniques for fuzzing projects in the EigenLayer ecosystem.
Getting to coverage
The primary objective that I was tasked with was to scaffold the initial harness for Renzo and ensure that coverage was achieved on all the functions targeted by the fuzzer (in this case Medusa). While a seemingly simple task at first, made easier with Recon’s scaffolding tool, this proved to be significantly more involved than I had initially thought, requiring continuous iteration and thinking about what’s actually happening as the fuzzer executes calls.
EigenLayer Integration
Renzo is built on top of EigenLayer to simplify the process of allowing Stakers to stake into EigenLayer and allocate their stake to different AVSs.
EigenLayer is an Ethereum protocol that allows restaking, whereby Stakers can stake native ETH or LSTs through EigenLayer and extend cryptoeconomic security services to other protocols on top of EigenLayer (AVSs) for which they receive additional staking rewards.
Because Renzo is tightly coupled with the functionality of EigenLayer, first I needed a way to deploy the EigenLayer system so that it could be integrated into the Renzo fuzzing suite to more closely resemble the actual system behavior, rather than potentially introducing different behavior by simply mocking the EigenLayer system. Given that the EigenLayer system is relatively compact, this fortunately wasn’t as involved as it can often be.
Fortunately the EigenLayer test suite already included a full system deployment for their test setup with foundry, unfortunately for me I didn’t notice this until I had already built my own based off of the deployment script. This exercise in doing something that’s already been done wasn’t all bad however, because the EigenLayerSystem deployment contract that I created ended up providing extra utility that would later be needed.
After scaffolding a fuzzing harness for Renzo with Recon, the EigenLayer setup I had created could be easily added as a submodule into the project and integrated into Renzo’s Setup contract by inheriting from the EigenLayerSystem contract and calling the deployEigenLayerLocal function which deploys the EigenLayer system and allow easy integration into other protocols with all the core contracts exposed as public state variables:
I was now ready to start getting some coverage on the primary contract in the Renzo system, RestakeManager, or so I thought…
Simple Clamping
Right away after setting up the Recon scaffolding following the Chimera framework, I had a TargetFunctions contract that looked like this:
But after running this for any extended period of time, coverage would get stuck at certain points.
The root of the problem was obvious enough after some investigating via the coverage reports and using unit tests to execute call sequences from the Medusa corpus, fundamentally, certain code paths would always revert because the search space of the fuzzer was too large and so needed to be narrowed (carefully, so as not to miss potentially valid states).
Alternatively, the fuzzer could simply be left to run for hours on end until it discovered the code paths that wouldn’t revert, but our general view at Recon is that the fuzzer should be useable by an entire protocol team or external reviewers and so having a fuzzing setup that requires days or even weeks to build up a useable corpus, even without properties defined on it, is less conducive to being run frequently and as a result discovers less potential vulnerabilities as the code base changes.
The solution in this case was to narrow the search space via limited clamping, whereby input values that were executed by the fuzzer were reduced to a smaller possible set. For example, addresses have a total of 2^160 possibilities, but only a small subset of these (2 using the actor setup that was implemented) will actually be able to execute any actions within the Renzo system as they require holding ETH or different liquid staking tokens, with the rest causing the fuzzer to expend countless runs on calls that revert for uninteresting reasons such as insufficient balances.
The decision was therefore simple to clamp these actor addresses along with the token addresses passed into function calls so that only actors and tokens actually used in the system would be used for call execution or passed as inputs to functions in the system.
Dynamic deployments
The next realization that happened over the course of this work was that the Renzo system was actually capable of more combinations of collateral tokens (with a corresponding strategy in EigenLayer) + OperatorDelegators than the initial setup I had created in the RenzoSetup contract.
In my setup the system was deployed with the two liquid staking tokens that Renzo had indicated they would be supporting (from their code4rena contest page): stETH and wbETH, with both being set on two instances of an OperatorDelegator (essentially a wrapper around an EigenLayer Operator) contract with each having a different allocation (operatorDelegator1 - 70%, operatorDelegator2 - 30%).
Renzo however would eventually add support for more LSTs (as indicated in their documentation) and eventually there may be multiple combinations of LSTs and OperatorDelegators, each with its own differing allocation, as shown in the diagram below:
This was similar to the insight in Recon’s engagement with Centrifuge discussed here.
So with the thinking that Renzo would likely add support for more LSTs after launch, given that EigenLayer currently already supports an additional 11 on top of the ones already used by Renzo, I introduced the ability to allow the fuzzer to dynamically deploy and switch between new tokens and strategies in the system in a restakeManager_deployTokenStratOperatorDelegator function.
This would ultimately prove to be a lesson in overcomplicating something more than it needs to and proved to not actually add any new interesting system variability that wasn’t already possible by fuzzing different allocation values, and so was later reverted. However, the mental model of different OperatorDelegators with differing allocations was a key unlock to gaining a deeper understanding of the Renzo system.
Externalities
After achieving coverage over the target contract with the methods described above, the focus then turned to how to simulate potentially adverse events in the larger system that Renzo was a part of in order to determine if they could create events that would break any of the invariants defined for Renzo.
I refer to these here as externalities because similar to externalities in economics, the types of events explored could have benefits to one party that would be borne by another, namely resulting in a potential loss of funds for a user at the expense of another.
Native Slashing
Native slashing is an integral part of the security mechanism of Ethereum proof-of-stake in which validator nodes that perform their duties maliciously, trying to modify the global state root incorrectly, have a portion of the stake they locked to become a validator removed from their accounting.
This was important to test in the system because users that stake native ETH through Renzo are delegating their funds to an OperatorDelegator which stakes them into EigenLayer in a beacon chain validator on their behalf via an EigenPod. If this OperatorDelegator is therefore penalized via a slashing event for misbehavior, this would have side-effects within the Renzo system’s accounting.
With access to the entire EigenLayer system via the deployEigenLayerLocal function, this simplified the implementation of a slashNative function in an EigenLayerSystem contract which simulates a slashing event by reducing the balance of the EthPOSDeposit contract being used and accounts for the change in balance in the EigenPod by calling the EigenPodManager:
This can then be called by one of the fuzzed target functions of protocols integrating with EigenLayer, in this case restakeManager_slash_native from Renzo’s RestakeManagerTargets contract:
AVS Slashing
The core feature of EigenLayer is that it allows Stakers that delegate their stake to Operators to provide services (AVSs) built on top of EigenLayer. But given that these Operators are providing vital services to systems in a trustless manner, EigenLayer implements AVS slashing whereby an Operator can have their stake slashed if they are shown to be acting maliciously.
In many ways this is similar to native ETH slashing, however because users can delegate liquid staking tokens (LSTs) to Operators, these need to also be accounted for when an AVS slashing event occurs to ensure properly penalizing the malicious Operator.
Because the Slasher contract for the EigenLayer system hasn’t yet been finalized, the implementation used for the Renzo fuzzing repo is based on an interpretation of how it may behave via the ISlasher interface. Essentially, the slashAVS function works similar to the slashNative function described above where it burns tokens held by the Operator and modifies the accounting for the given Operator to reflect this:
Similar to slashNative, this can then be called by one of the fuzzed target functions in the integrating protocol, in this case restakeManager_slash_native:
Discounting & Rebasing: two sides of the same coin
LSTs are meant to maintain a peg to their staked ETH backing but the mechanism by which they do this can have effects on systems that integrate with them. In the case of Renzo, users can deposit into the system via LSTs for which they receive ezETH (native Renzo token) as a receipt of their stake. To determine how much ezETH a user should receive for the amount of deposited LST, Renzo uses a price oracle along with the currently totalSupply of circulating ezETH.
In an ideal world this would allow normal calculation of exchange rates, however, historically LSTs have shown to be subject to depegging events (described here). Additionally, each LST token implements a rebasing mechanism to handle the increase in value of the token as it accumulates validator staking rewards.
To more accurately simulate these effects, the following mechanisms for discounting and rebasing described below were implemented into the Renzo fuzzing suite.
Discounting
Because discounting implies a reduction in token price, this just required a decrease of the price returned by the oracle used in the Renzo system to simulate this. Given that the testing suite was setup for local testing, the mock oracle could be directly called by the fuzzer to simulate this sort of event in the following target function:
The decision was made to not introduce any clamping of the new price to allow a greater simulation of the possible edge case scenarios that might occur for a significant price swings in adverse market conditions.
Rebasing
The tokens that Renzo plans to initially support (stETH and wbETH) use different mechanisms to simulate price increases due to rebasing events with one using interest accrual via shares and the other using a modified exchange rate. After a brief diversion into exploring the creation of a shared interface that would support these mechanisms I realized that because they both do not actually effect the supply of tokens in the system, the only thing of interest was once again the exchange rate between the LST and ezETH.
This meant that rebasing events in the Renzo system could be accounted for with an increase in the price of the LST relative to ezETH, which was implemented in the following function:
Conclusion
Fuzzing has proven itself as a very useful tool for uncovering edge case scenarios that could lead to exploits that can often be difficult for a manual reviewer to reason through themselves. However, to allow the fuzzer the ability to properly explore these edge cases there’s often a significant amount of manual work required in the setup of the fuzzer and reasoning about events external to the core system that could introduce side-effects and what the best way to simulate such events using the fuzzer is.
Luckily once these functionalities have been defined in the base protocol, such as slashNative and slashAVS they can be easily integrated into others and therefore only have to be created once.
For other protocol-specific edge-case events such as LST discounting and rebasing, while their implementation may be unique to a specific protocol, their ideas are often reusable with other protocols using the same mechanism (as was the case with the rebasing implementation shown here, inspired by the implementation in this engagement with eBTC).