TON Technical Incident Report

For the last two days, the TON blockchain has realized a new milestone while experiencing a significant slowdown in transaction processing.

The most substantial periods were on 5 Dec 22:30 - 6 Dec 02:30 (Dubai Time) and 6 Dec 22:45 to the current time.

No smart contracts or user assets are at risk. All sent messages will be processed.

Source

Since the evening of December 5, the number of user transactions on the TON blockchain has increased significantly, with over 3 million new transactions seen since then.

Reason

TON's architecture and implementation allow it to quickly handle a much larger load. However, we encountered some of the main network's validator nodes running on weak hardware.

Limiting the bandwidth of validators saves money but goes against our guidelines. These validators rented hardware for low load with no provision for load growth. Since the network had been running low load for the previous months, this was not a problem. As soon as the load increased by 50-100 times in 30 minutes, these validators started to slow down the whole network with them.

Solution

TON has a system of decentralized penalties for validators who do their job poorly or incorrectly. However, at the moment, penalties are too kind.

To solve this problem comprehensively, we plan to significantly improve the penalty system and increase the size of penalties for bad validators.

However, we can only do it with time because responsible administrators who keep an eye on their validator and don't skimp on hardware should not be affected. We need some time for debugging to eliminate false positives.

At the moment, we have released a patch that will improve transaction processing on current validators. All validators should update their hardware urgently. We may still need a few more patches to eliminate all problems, so urge validators to continue following this channel for more updates. 

FAQ

Has TON gone down?

No, the blockchain is still performing, but transaction processing has slowed down significantly.

Is TON a scalable blockchain?

Yes, TON is the most scalable and fastest blockchain in the world. We have proven this in performance tests by setting the TPS world record. We have also previously tested the current testnet, where only 12 validator nodes could handle several times more load without the problems that we currently see on the main network.

Now, there are issues with the system penalizing weak validators, not conceptual performance issues.

Why can't we fine and suspend bad validators now?

In previous validation rounds, it was challenging to identify the poorly performing from the responsible due to general network slowdown. We are required to stabilize the situation first.

Is TON capable of handling many transactions sent to one account?

Yes, TON can handle many transactions sent to one account because both accounts and queues of sent and received messages between accounts are sharded.

Of course, the more balanced the load on the network, the better, but sending to a single account is not a conceptual problem.

TON has no plans to raise commissions/introduce a dynamic commission system?

No, with the target hardware at validators, there will be no problem with the current fixed low commissions.