Note: This article attempts to break down a complicated issue so that readers of every technical ability can understand. However, that makes it very long. Section headers are included to help readers navigate to the parts that meet their interest/technical level.
Twenty-nine days ago, Ethereum Foundation (EF) contributor and Oaken Innovations co-founder Hudson Jameson alerted the community that the recent surge in token offering (TO) popularity may lead to severe network congestion and high transaction costs over the coming weeks. In a reddit post, he explained his belief that a temporary fix miners had implemented late last year in response to a Denial of Service (DoS) attack is what was causing blocks to “fill up,” or reach the limit of acceptable transactions per block, requiring many users to endure long wait times and high fees to send transactions on the network. Recognizing the frustration that users unaccustomed to Ethereum’s evolutionary rate might have with these long wait times, he suggested that miners raise their gas limit and gas price settings to help alleviate these issues. He called for miners to re-institute a pre-DoS attack “adaptive gas limit” that would track a moving average of previous block use and organically grow (or shrink) to help prevent blocks from filling up.
In a discussion with ETHNews, Jameson explained that Ethereum’s “gas limit” is similar to Bitcoin’s often-discussed block size in that it is an artificial cap on chain storage and bandwidth for including transactions in each block. The gas limit determines the upper limit of how much “gas” users give “miners” – the machines that bundle Ethereum transactions into blocks to be included in the blockchain – in order to incentivize the inclusion of a transaction in the next block. As Jameson put it, “The main problem [was] that people are throwing huge amounts of Ether at these Token Offerings and that is causing a lot of transactions to get added to the network. But the network isn’t set up to support the transactions as quick as they’re being put in. So, it causes network congestion.” Once the combined gas of all transactions slated to be included in the next block reaches the gas limit, any remainders must wait for a later block for inclusion on the blockchain. If the number of pending transactions – and their associated gas – is tremendously greater than the gas limit, a backlog can develop that lasts for hours or even days.
When a backlog is established, users can offer higher and higher gas fees to “jump the line” and avoid being lumped in the pool of pending transactions. Each type of transaction has a different amount of gas required to win miners’ selection, with simple wallet-to-wallet transactions requiring much less gas than more complicated transactions, such as the execution of Executable Distributed Code Contracts (EDCCs).
Nearly a month later, Jameson’s concerns were apparently on point. The cluster of enormous TOs from Civic, Status, Bancor, and Brave manifested Jameson’s predictions sooner than expected, causing an average of close to 45 percent of the previous 2,500 blocks to be full at any given time. This high network congestion resulted in a persistent backlog of transactions, and Jameson pointed to the Status TO in particular for having caused the largest strain on the network so far, with some transactions reportedly taking nearly a whole day to complete. Typical wallet-to-wallet transfers (which only cost around a few cents back in December) sometimes reached an average cost of as much as a dollar. At least one attempt at participating in the BAT TO ran $6,600 for the single transaction!
Despite Jameson’s (and others’) calls to the mining community to help fix the problem, the gas limit had not increased. Some believed the issue was simply a lack of communication within the community, and wondered whether miners were hearing the calls or were confused as to what action to take. Others asserted a less friendly motive, accusing some miners of acting in their own self-interest and deliberately interfering with the scaling efforts. Did this signal Ethereum’s own emerging standoff between miners and developers that could manifest in a civil war like the ongoing debate in Bitcoin? Time would tell, but some asserted that this fear might have been a strong factor in Ether’s volatile price over the last couple of weeks.
Regardless of cause, the community recognized that with more highly-publicized TOs scheduled over the next month, this bottleneck would only get worse, possibly frustrating new users and investors to the point of spooking them away from the ecosystem. r/ethereum, Ethereum Gitters, and practically every other known chat channel bustled with confusion about how to solve the issue. But this morning around 8:00 a.m. EDT, the community breathed a collective sigh of relief and celebrated its cohesiveness, upvoting a reddit post that demonstrated that the gas limit was finally increasing.
Blockchain Toll Booths, Indicator Tags, Parity and Geth, and Last Year’s DoS Attack – A Layman’s Explanation of Ethereum’s Biggest Technical Scaling Problem to Date
You’re not alone if you got lost in some of the technical language above, but it may help to think of the blockchain like a freeway with a tollbooth at the on-ramps. Each transaction is a car trying to pass the tollbooth, and the miners act as the toll booth operators that collect gas (or toll fees) to allow entry to the freeway. Larger cars are more likely to congest the freeway and are therefore charged a higher fee. As traffic forms in front of the tollbooth, cars can “jump” to the front of the line by offering toll booth operators a larger fee than other cars, and many choose to do this during periods of high traffic. The gas limit acts as the toll booths’ goal for the total amount of fees collected at any given time, indirectly limiting the number of cars that can pass the turnstile.
As more cars hoping to avoid the forming traffic jam pay more, the average price of transactions increases. Ironically, this traffic jam avoidance exacerbates the jam for those who do not make it past the turnstile, because a higher average cost in a fixed total amount will result in fewer cars getting past.
The gas limit exists because early Ethereum developers understood that it would prevent attackers from forcing “infinity loops,” or a computer error in which programming code refers to itself and therefore can never complete. Miners set gas limits by entering certain “indicator tags” into their “Geth” and “Parity” clients – the two main software packages Ethereum miners use to talk to each other and the blockchain – which then “votes” with the rest of the network on where they believe that limit should be set. Miners will continue working at the current gas limit until at least a 67 percent supermajority of miners have set indicators to change the gas limit, at which point the whole network will adjust to comply. The decentralized nature of Ethereum miners makes it difficult to reach the supermajority threshold in any situation because many miners are believed to “set and forget” their machines, potentially not paying attention when a call for change is made and failing to act. The existence of “mining pools” – which allow individuals to pool their mining resources together in exchange for consistent returns – helps with this obstacle, as only the pool operator needs to signal a change on behalf of all its members.
It would be highly inconvenient and nearly impossible for every Ethereum miner (or even the mining pool operators) to monitor the network and manually adjust their indicator tags at every change in transaction volume. Fortunately, Geth and Parity clients have a built-in mechanism to help with this. As Jameson explained to ETHNews, each has a setting that will automatically set indicator tags based on the percentage of full blocks in the blockchain’s recent history, allowing the gas limit to naturally grow and shrink to account for surges in demand.
However, this “auto-scaling” mechanism feature was not executing during the recent surge in TO-related traffic. Jameson explained that this was likely a carryover from a response last year’s DoS attack. The source of that attack was one or more users creating EDCCs that managed to run very computationally complicated calculations for a very low amount of gas. They spammed the network with these low-cost attacks, causing the network to come to a near stand-still. Using our analogy, the attack represented thousands of eighteen-wheelers per second lining up at the toll booth, while tricking the booth operators to charge them the price of a motorcycle. In a flurry of responses that attempted to mitigate that attack, Ethereum Foundation (EF) members issued several recommendations and protocol updates for miners and clients. Among them, there was a call for miners to lower the gas limit. Miners successfully achieved the requisite supermajority to follow the EF’s advice, as well as disabled their auto-scaling settings to ensure their clients wouldn’t automatically undo their efforts.
But nearly ten months later, during a TO popularity surge that happened much sooner than anyone expected, those settings were apparently still in place.
The Community Discusses Solutions
Jameson’s original solution was for all miners to enter the following settings into their clients, re-allowing for the auto-scaling feature to kick in:
(For those more technically inclined, he provides a detailed explanation for these optimal amounts on his blog).
Yet his call apparently fell on deaf ears. For whatever reason, miners did not seem to enter these settings, and a consistent 4.7 million gas ceiling emerged for weeks. Did he fail to account for miner economics? His suggestion may require mining pools to abandon their short-term interests in favor of promoting the overall health and efficiency of the network. As the specter of self-interested miners deviating from those interests – similar to the current standoff threatening a possible hard-fork of Bitcoin – loomed throughout the community, repeated calls for the change were made. Jameson believed that Ethereum inventor Vitalik Buterin’s suggested Ethereum Improvement Proposal #648 (EIP 648) – which seems to allow statistical tracking of what proportion of resources are being used for different types of calculations (such as EDCC execution vs. simple wallet-to-wallet transactions) in the past, and then aims to adjust the types of transactions accepted by miners to meet that proportion – could also work. Some wondered if it was the auto-scaling mechanism itself – that miners had indeed followed directions, but for whatever reason their Geth and Parity clients were not automatically adjusting – and gave an alternate solution of raising the gas limit to an arbitrarily high number to allow for current volume fluctuations and manually adjusting in the future as needed. Others criticized this solution, leading to calls for stronger direction from the Ethereum Foundation. But the EF remained silent, seeming to signal that they believed this to be a good opportunity to incorporate new minds into Ethereum’s collective problem-solving abilities rather than rely on the foundation’s centralized leadership.
Nonetheless, a lack of clear direction consumed the discussion, which may in itself have caused its own problem. As the curator of EthGasStation (an unidentified community member that goes by the reddit handle “u/latetot”) explained to EthNews:
“To me it appear[ed] that miners [were] just unsure what to do with the settings from a technical view and [were] not actually opposed to raising the limit. Gas profit is still a very small percentage of their revenue even in [that] situation. In my view, the miners realize[d] that having a healthy network that is scaling as designed will help their profits the most over the long term and they [would] support higher gas limits once they receive[d] clear instructions from people they trust about how to raise the limit.”
But still others speculated that the real reason behind the problem was that it might not be in some mining pools’ best interests. Specifically, some have suggested smaller mining pools might be concerned that they cannot compete with bigger miners to arrive at blocks fast enough if the gas limit is raised, and therefore would be prone to “uncle blocks” – blocks that arrive to the network slightly slower than the fastest accepted block, and therefore receive a smaller mining reward – if too many transactions are allowed in a block. Several community members have even accused certain mining pools of mining “empty blocks” – blocks with an artificially small number of transactions – to trick any miners who have implemented the auto-scaling mechanism into thinking blocks are not really that full and therefore do not raise the gas limit when conditions otherwise request it. This would be akin to some toll booth operators closing their lanes during rush hour, further contributing to the traffic jam.
A Test of Strength of the Ethereum Community’s Ability to Persevere
If that concern about miners’ motives was true, it would have indicated an emerging rift between miners and developers that could present major obstacles in Ethereum’s ability to scale, both for the instant problem as well as for the Ethereum foundation’s long-term roadmap. For example, the current plan for the transition to Proof of Service (PoS) requires a hybrid period where both the current Proof of Work (PoW) and the eventual PoS systems are simultaneously implemented. As PoS is essentially the elimination of miners, this means that miners are expected to cooperate in their own elimination for the sake of the health of the network. A signal that miners will not cooperate against their own self-interest could be devastating for that effort.
Jameson discussed this possibility and its ramifications in his conversation with ETHNews, recognizing that it may be an impending obstacle in Ethereum’s goals on the horizon. However, he noted that Ethereum has several features baked into its code to push miners towards cooperation that he hopes will help Ethereum avoid a fate like Bitcoin’s. First, he noted the impending “difficulty bomb,” which would boost miners’ artificial “difficulty” placed on their mathematical processing abilities to a point where they are faced with a decision to either comply with system upgrades or endure economic losses by mining on the old protocols. Second, the move to PoS has been on Ethereum’s road-map since the very beginning, so – unlike Bitcoin’s scaling problems that were manifested by surprise years after its initial development – miners who joined the Ethereum knew full well that this is the direction the network is going in. Third, also unlike Bitcoin miners that are now using “ASIC” hardware, which can only be used on the Bitcoin blockchain, Ethereum miners are still using Graphic Processing Units that are much easier to convert to another network, avoiding miners’ “my way or the highway” mentality formed from possibly abandoning their hardware investments.
Nonetheless, Jameson remained optimistic, sticking to the belief that miners were just confused and had not updated their settings since the previous attack. And as of this morning, both his and u/latetot’s beliefs seem to be vindicated.
The Path Forward
It is not yet clear which solutions miners followed or what finally caused the gas limit to raise. In a post this morning, it was announced that large mining pools Ethpool and Ethermine had signaled a targeted gas limit of 6.7 million, well above the 4.7 million ceiling in place for nearly a month. Whether this was the direct cause, or if other miners’ actions also contributed is unclear, but in Jameson’s own words to ETHNews, “it is awesome that the community effectively reached out to miners to increase the block gas limit. It says a lot for the maturity of both Ethereum users and miners.” Indeed, it seems that the community is strong enough to avoid its own block size stand-off.
Moving forward, we will see how the solution holds up under the next wave of popular TOs. Jameson suggested that the newly introduced EOS TO may be gaining momentum in a way that will overwhelm the network like the ones last week did. If the network adjusts, it represents a major victory for Ethereum’s scaling abilities.
In the meantime, Jameson tells us that EF members are already working on more solutions to further alleviate future gas limit problems. “I can confirm that in Ethereum core developer chat channels that block gas limit and EIPs affecting ETH issuance (like https://github.com/ethereum/EIPs/issues/186) leading up to our switch to PoS and Casper have been and continue to be discussed.” For those interested, he recommends checking out tomorrow’s Core meeting agenda and watching its livestream, the link to which is located on his blog. But for now, it is clear that even without EF’s specific leadership, the community came together on its own to successfully avoid what may have been the biggest scaling obstacle in Ethereum’s history.