The Right Way To Multisig
How to implement best practices in multisig security to prevent your protocol from getting rekt.
Consider this your definitive guide to how to secure a multisig, we’ve compiled knowledge from subject matter experts in one place so protocol teams can rest easy knowing they have the best process in place to prevent an exploit before it happens.
Introduction
Multisigs are often the most powerful account that can interact with a protocol’s contracts, and as the saying goes: “with great power comes great responsibility”.
Understanding the best way to manage this responsibility though has been, until now, unclear. How many signers is safe? How do you manage multisig privileges? Do you use a timelock or not?
We’ll answer all of these questions and more in this post which we’ve compiled from security researchers that focus specifically on multisig security from the web2 and web3 side.
We’ll then take these answers and combine them into best practices that we’ve developed using Gnosis Safes, a RolesAuthority contract and a Timelock contract to help make the lives of protocol teams simpler without compromising security.
Starting with categorizing the risks of each action that a protocol team needs to take, we’ll build a framework for how to think about multisigs that we can add different security mechanisms to.
Risk Categories
Risk categories are the framework that we can use to think about the different actions that need to be taken by a protocol team. We can then apply different procedures to each action to ensure maximum safety while still allowing teams to get things done.
We can broadly classify multisig actions into three different risk categories:
Low - e.g. donating gas/ETH for protocol operations
Medium - e.g. changes to system configs, deployment of new contracts (exploitable via MEV)
High - e.g. contract upgrades, fund transfers, ownership changes
We can link these categories to a required number of signers for each action, where actions in higher risk categories need more signers. This helps lower the chance of a hostile takeover due to compromised or colluding signers.
Below we'll see how multisig thresholds corresponding to the risk category can help us avoid these hostile takeovers. Then, we'll discuss adding a timelock based on the risk category as an extra security layer.
Thresholds
In Safe multisigs, which many projects use, the threshold sets how many signers need to approve a transaction. The multisig can execute the transaction only once the threshold is met.
The signer threshold is the first step in determining the safety of the multisig because it implicitly determines how much risk is being taken on.
For example, in a ⅗ multisig 3 of the signers are required to undertake any action. From another angle, we see that only 3 signers need to collude or have their private keys compromised for full control of the multisig. In the worst case, if this is the only multisig for admin actions, these 3 signers could completely wreck the protocol.
In this scenario where we only have one multisig that has the ability to perform all operations, it would default to the highest risk category and therefore require a high threshold.
If however we have multiple multisigs, we can link the threshold to the risk category for the actions that each multisig can take. Lower risk operations subsequently need a lower threshold, while higher risk operations need a higher one.
Our recommendation is to try to achieve a 70% threshold consensus for medium and high risk operations.
Adding external tiers like a security firm can also boost security. They offer unbiased oversight and help reduce risks. A 5-out-of-7 multisig setup with one external tier achieves the 70% threshold while adding this extra layer of protection.
Signer Coordination Risks
Multisigs inherently add a time factor to the signing process, due to the fact that they require coordination among several people, often across different time zones. This often means multisigs are slow to respond to emergency events. But by definition, these events require fast action to reduce potential damages.
Special tools like the Safe Panic Modules by @go_su_to can be added to Safe multisigs. This tool helps teams move quickly in an emergency situation. With it a single multisig signer can take action, like revoking approvals for routers or trusted contracts or withdraw from an integrating protocol.
isSigner modifier allows multisig signers to take pre-approved actions that can protect a protocol in an emergency scenario.For attacks like the one on Euler where there was knowledge that something was up before the protocol team paused the contracts, this would have allowed an individual signer for integrating protocols to withdraw from Euler before the Euler team paused the contracts or funds were completely stolen.
This speeds up responses to bad events while keeping the multisig secure by preventing individual signers from taking actions privileged to only the multisig.
Signer thresholds however are just one part of our layered security. The next layer adds time more explicitly to the multisig with timelocks.
Timelocks
Timelocks delay the execution of an action taken by an address to a predetermined time in the future. In the context of protocol operations, the address initiating the action is the multisig.
This delay between scheduling an action and its execution helps the protocol team. It gives them extra time to check if the action is correct. For example, if a protocol upgrade has a sufficient timelock, the team can let an external security team check the payload and its effects. This helps them see if the upgrade will work as planned when the timelock ends.
Additionally, timelocks give users of a protocol the ability to opt out of any potential changes that they’re not in favor of before they take effect.
Timelocks also add protections for users if the multisig is compromised, allowing them time to exit before an attacker executes a malicious action.
A commonly used timelock implementation is the TimelockController from OpenZeppelin:
schedule and execute functions allow scheduling an admin operation and executing it once the delay has passed.This contract allows setting a multisig as the proposer with the PROPOSER_ROLE. The multisig can then schedule a call to an admin function in the protocol contracts. It does this by passing the needed parameters to the schedule function. Then once the delay has passed, one of the signers with the EXECUTOR_ROLE can call the execute function.
The call to execute is less sensitive since it only runs the payload already approved by multisig signers. However, assigning the EXECUTOR_ROLE to these signers helps prevent problems if multiple payloads are queued. Anyone could execute the payloads out of order with a permissionless function. This could lead to unintended side effects.
Timelocks boost security by introducing a time element. But how can we use them with multisigs? We want to keep security high without causing unnecessary delays in operations.
The Multi Multisig Approach
Luckily you can choose any combination of threshold/timelocks you want by using multiple multisigs! This allows you to balance the risk, speed and security for any operation.
You can set up multiple multisigs like this for the risk categories mentioned earlier:
Low - low threshold with short timelock
Medium - high threshold (>=70%) with a short timelock
High - high threshold (>=70%) with long timelock
Please note these are only meant to be used as examples, your project’s risk security profile will differ and should be taken into account before choosing your own parameters.
This improves the protocol team’s efficiency. It reduces waiting time for signers and limits delays on minor actions. At the same time, it ensures they have strong safeguards for more significant actions.
Managing The Multisigs
With multiple multisigs, the challenge is managing permissions. Each multisig must have the right access to perform its tasks.
From our work with various protocols, we've seen that the method first used by Maker and now adopted by the Badger DAO and Corn teams is the most effective. This method employs a RolesAuthority contract (another example is the AuthorizerAdaptor contract used by Balancer DAO governance). It separates which multisig can access specific governance functions.
The RolesAuthority is based around roles that determine the function signatures that can be called on a given contract by a calling address. This is first set using the setRoleCapability function:
The role is a number assigned to multisig addresses. It decides which function signatures they can call on the target contract.
The setUserRole function then allows us to assign one of the roles created above to a given multisig/timelock address:
One benefit of this approach is that you only deploy the RolesAuthority contract once. This centralizes multisig permissions in one place, rather than spreading them across all contracts in your system.
For example, if your system uses several ERC4626 vaults, you don’t need to set permissions for each multisig separately. Instead, you just set it once when you deploy the vault. Use the setUserRole function, and then you can forget about it.
Having one contract as a source of truth also allows you to query which multisigs have access to which functions on a given contract. You can use the doesUserHaveRole and doesRoleHaveCapability functions to see all permissions in your deployed contracts.
The RolesAuthority contract lets any address authorized by the Auth contract set role capabilities and user roles. However, it lacks fine-tuning for different addresses to act as admins for specific capabilities or user roles. Make sure to take this into consideration when using the RolesAuthority contract in your system.
Bringing It All Together
We now have three key components: Gnosis Safes, the RolesAuthority contract and the TimelockController contract. We can combine these to create a tailored governance process for every action in the protocol which we’ve visualized in the following diagram:
Starting from the bottom up we see that the core of the system is the Gnosis Safe Multisig. We add the following owner guard to the multisig:
checkTransaction function checks if the sender is an owner of the Safe being used.This guard stops any queued transaction that’s reached its signer threshold from being executed by anyone other than a multisig signer.
This protects us from exploits where attackers phish signers for their signatures. With these signatures, attackers could execute a malicious payload if there's no owner guard. With the owner guard in place an attacker must get an additional signature from a signer to call the execute function. This step is unlikely and would probably alert the signer to the malicious activity.
We then add a timelock to each multisig corresponding to its risk category. A timelock by itself however doesn’t ensure much, besides delaying the eventual execution of a payload. A timelock is most useful when a canceller role is assigned to a signer who is not one of the multisig signers. This setup allows the canceller to stop a malicious payload from being executed.
CANCELLER_ROLE can cancel any pending timelock transactions. This stops harmful payloads from being carried out.In the worst case scenario with this setup, where a canceller’s private key is compromised, the canceller can still be used to prevent new proposals. Cancellers can also prevent a misconfigured payload from being executed in the scenario where a payload was scheduled but later verified to be incorrect.
The final piece is managing multiple timelocks. We talked about this earlier with the RolesAuthority contract. This allows viewing all multisigs that have permissions in your system all in one place.
This setup makes it easier to handle your multisig security resources. However, there are other factors, not directly tied to your multisig organization, that you should also think about to ensure a safe operating environment.
Other Considerations
Trusted Payloads
When signing a payload in a Safe multisig, the payload being signed is generated off chain by one signer. This puts a lot of trust in the signer. They need to create the payload accurately, or it could negatively affect the protocol.
This utility made by @go_su_to for Balancer DAO verifies the payload data as JSON before it's uploaded via the Safe frontend. It includes extra checks in the CICD pipeline. These checks ensure the information encoded and uploaded to the Safe UI meets the expected proposal requirements. This process helps prevent errors in the payload that other signers might approve unknowingly.
Permissioned Executions
Using tools such as Kleidi you can add further fine-grained controls to your timelock.
As an example you can approve a signer for rebalancing operations. This means they can withdraw from one pre-approved contract and deposit into another pre-approved contract instantaneously. However, they can’t do anything else. You set the two contracts for rebalancing during the initial setup, which prevents withdrawals to random contracts. If the signer's private key is compromised, the attacker can only transfer funds between the two approved locations.
With this precision, a protocol team can balance efficiency and security. They can complete low-risk maintenance operations without unnecessary delays.
Governance Fuzzing
Recon offers a service we call governance fuzzing. This lets you test protocol invariants for any changes using the forked chain state. You’ll know exactly how the system will act after the change.
You can use this to verify any protocol changes. This includes changes triggered by receiving an external message through a bridge, which we did for the Corn team. Corn is an L2 that lets you use Bitcoin in DeFi.
Their system uses cross-chain messaging via LayerZero to send and swap tokens between Corn and Ethereum Mainnet. Testing interactions between two chains is challenging with forked fuzzing because you can't fork two chain environments at once, but with governance fuzzing we can simulate this.
Governance fuzzing lets us listen for bridging-related events on Corn. We can then trigger a forked fuzzing job on Ethereum Mainnet. This tests existing invariants against changes that will occur on Mainnet as a result of the cross-chain action before they happen.
target_doGovFuzzing function executes a call to the LayerZero endpoint on Mainnet with the payload from the event on Corn. The endpoint_lzCompose function then gets randomly called by the fuzzer to preview the effects of the event from Corn on Mainnet.For governance changes, we use a similar approach. This lets us listen to updates from the multisig or timelock whose results we can preview in the system before they happen. Then, we check the invariants with fuzz testing on the forked chain state. This helps us see if the upgrade will break any invariants. If it does, we can stop any unexpected behavior using the canceller role mentioned earlier.
Good Practices
Members of a protocol multisig should also be sure to enact the following good practices:
Mandatory hardware wallets for all signers.
Recommended models: Ledger Nano S Plus, Trezor Model One, Trezor Safe 3 (avoid Ledger Nano X for its battery issues).
All multisig signers should coordinate via a restricted and encrypted communication channel (signal/telegram).
All multisig signers must have 4 hardware wallets (2 for signing + 2 backups).
Implement a key rotation system (e.g. every 2 years).
Signers of high risk payloads MUST only use their hardware wallets for these multisig transactions.
Conclusion
We’ve seen how we can categorize different governance operations based on their risk. These risk categories help us better apply tools like thresholds and timelocks. Ultimately this results in unique signing schemes appropriate to an operations’ risk.
Using the categories we defined, we saw that higher risk operations need more signers and longer timelocks. In contrast, lower risk operations require fewer signers and shorter timelocks. However, this can change based on the specific operations a protocol’s governance team is handling.
Usine multiple multisigs to manage these risks we then saw how to manage the multisigs with the RolesAuthority contract. This contract lets us give certain permissions to specific multisigs for which contract functions they can use.
We then looked at some aspects of multisig security not directly related to the multisig setup, namely payload formation, designating operations and private key security:
Establishing a review process and restricting payload executors to the signers helps prevent accidental or malicious payload approval.
Kleidi helps teams assign low-risk tasks, like rebalancing, to their signers. This speeds up maintenance tasks while keeping security intact.
Recommended practices for private keys to help protect against outside threats.
Implementing these best practices can help prevent your protocol from becoming another statistic. If you feel overwhelmed though, reach out to us. We’ll help you get started!
Contributors
This post wouldn’t have been possible without the hours of work that other researchers put into these topics. Some of these researchers recently did an event with Recon where they went in-depth into the topics discussed above:
Many thanks to the following researchers for their contributions to this piece:
Souilos - Smart contract audits are essential, but they’re not enough. Operational Security (OpSec) audits protect your protocol from single points of failure, they complement smart contract audits and are just as critical. Louis (aka Souilos) helped us in this article and provides OpSec audits, having already helped different Web3 protocols (including Optimism).
Elliot Friedman - A smart contract engineer focused on security, having built numerous DeFi and governance protocols—including Kleidi Wallet, a self-custody solution for security maximalists—and the Forge proposal simulator to help DAOs create secure governance proposals.
Gosuto - Taking traditional finops experience onchain, @go__su__to has been operating multisigs, analytics and reporting on EVMs full time for the last 4+ years. He is most comfortable on the edge of the blockchain, where traditional data science tools can be combined with customised smart contracts to build the end-to-end pipeline every onchain asset holder dreams of!
Alex The Entreprenerd - Top 32 SR of all time according to Sherlock. SR at Spearbit / C4. Co-founder of Recon. Over 25 contests judged at C4. Builder for Badger, Hundreds of Millions TVL.