UPDATE: 1:30pm PST:
From Vitalik Buterin:
Today the network was attacked by a transaction spam attack that repeatedly called the EXTCODESIZE opcode (see trace sample here), thereby creating blocks that take up to ~20-60 seconds to validate due to the ~50,000 disk fetches needed to process the transaction. The result of this was a ~2-3x reduction in the rate of block creation while the attack was taking place; there was NO consensus failure (ie. network fork) and neither the network nor any client at any point fully halted. The attack has since, as of the time of this writing, mostly halted, and the network has for the time being recovered.
The short-term fix is for users, including miners, enterprise users (including exchanges) and individuals to run geth with the flags:
--cache 1024 --targetgaslimit 1500000 --gasprice 20000000000
Or Parity with the flags:
--cache-size-db 1024 --gas-floor-target 1500000 --gasprice 20000000000 --gas-cap 1500000
This (i) increases the cache size, reducing the number of disk reads that nodes need to make, and (ii) votes the gas limit down by ~3x, reducing the maximum processing time of a block by a similar factor.
In the medium term (ie. several days to a week), we are actively working on several fixes for the Go client that should both provide a more stable resolution for the present issue and mitigate the risk of similar attacks, including:
- A change to miner software that automatically temporarily cuts the gas limit target by 2x when the miner sees a block that takes longer than 5 seconds to process, allowing for adjustments similar to what was coordinated today to happen automatically (see here for a pull request; note that this is a miner strategy change and NOT a soft fork or hard fork)
- Numerical tweaks to cache settings
- Adding additional caches
- Adding an additional cache for EXTCODESIZE specifically (as it is likely that EXTCODESIZE reads are several times slower than other IO-heavy operations since the contracts that are being read are ~18 KB long)
- An on-disk cache of state values that allows them to be more quickly (ie.
We are also exploring the option of replacing the leveldb database with something more performant and optimized for our use case, though such a change would not come soon. The Parity team is working on their own performance improvements.
In the longer term, there are low-level protocol changes that can also be explored. For example it may be wise to add a feature to Metropolis to increase the gas costs of opcodes that require reads of account state (SLOAD, EXTCODESIZE, CALL, etc), and especially read operations that read external accounts; increasing the gas cost of all of these operations to at least 500 would likely be sufficient, though care would need to be taken to avoid breaking existing contracts (eg. simultaneously implementing EIP 90 would suffice).
This would put a much lower upper bound on the maximum number of bytes that a transaction may read, increasing safety against all potential attacks of this kind, and reducing the size of Merkle proofs and hence improving security for both light clients and sharding as a side effect. At present, we are focusing on the more immediate software-level changes; however, in the long term such proposals should be discussed and contract developers should be aware that changes of this sort may take place.
UPDATE: 8:45 am PST:
2/2: we're looking in to a short-term solution mapping data structs and long term by optimising or switching away from leveldb.— JΞFF (@jeffehh) September 22, 2016
It appears that we're fine implementation wise. The leveldb implementation we're using seems to be the culprit. 1/2— JΞFF (@jeffehh) September 22, 2016
9/22 8:23 am PST: ETHNews' source at Shanghai notified our team: “Most core devs across multiple protocols are in communication. It’s being fixed right now. People here immediately jumped on it and started working and brainstorming solutions. All of this will make the network stronger ultimately.”
URGENT ALL MINERS: The network is under attack. The attack is a computational DDoS, ie. miners and nodes need to spend a very long time processing some blocks. This is due to the EXTCODESIZE opcode, which has a fairly low gasprice but which requires nodes to read state information from disk; the attack transactions are calling this opcode roughly 50,000 times per block. The consequence of this is that the network is greatly slowing down, but there is NO consensus failure or memory overload. We have currently identified several routes for a more sustainable medium-term fix and have developers working on implementation.
It is highly recommended to switch to Parity mining. Use these settings:
–cache-size-db 1024 –gas-floor-target 1000000 –gasprice 500000000000
Parity mining guides:
If you continue to use geth, use the following settings:
–cache 1024 –targetgaslimit 1000000 –gasprice 50000000000
(9/22/16 7:00 am PST) Ethereum developers are alerted to DoS attack.
#ethereum is under DoS. No nodes are crashing; blocks take longer to process. We're looking in to it.— JΞFF (@jeffehh) September 22, 2016