DoubleZero Testnet Incident on September 2, 2025

On September 2nd, 2025, following a routine software update on the DoubleZero (DZ) Testnet environment, Solana validators lost the ability to communicate with non-validator nodes on the DZ Testnet. The impacted validators successfully failed over to the public internet and were able to communicate with other validators uninterrupted. This is the intended failsafe behavior for a validator connected to DoubleZero. However, non-validators (RPCs) did not failover from DZ Testnet and, due to a routing issue, Solana validators on DZ were unable to communicate with these other hosts.

The problems began at approximately 20:20 UTC and were resolved at approximately 23:40.

In this post, we explain what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again.

Background

DoubleZero is building a distributed, dedicated, high-performance global network, aimed to bring low-latency communication to everyone, everywhere. Initially, the network targets the Solana ecosystem as primary users: Solana validators and RPCs. The network itself is made of links contributed by 3rd-party contributors, on a rewards basis and terminating each link in the network, is a DoubleZero Device (DZD). DZD configurations are managed via an off-chain process known as the DoubleZero Controller. User hosts connect to the network by opening a GRE tunnel to the DZD. These tunnels can be thought of as similar to a VPN tunnel, in that it can pipe traffic through the internet from point A to point B as if the host is directly connected to the DZD. In our most common configuration, a user host continues to use their public IP address that they connect with on the DZ network. The DZDs use a routing protocol called BGP to pass routes to the user hosts, such that each host knows about each of the other connected DZ users’ hosts. As a result of LPM (longest prefix match) lookup the Linux kernel routing table will route packets to the DZ network when sending to another DZ-conected host rather than the public internet.

What happened

A routine update to the DoubleZero software stack (v0.6.2) occurred before 20:20 UTC on September 2, 2025. The update completed successfully with all QA tests passing and in itself did not impact user connectivity. Due to forthcoming validator fee calculations, our software needed the associated validator public key (pubkey) added to the DZ’s user account, which is stored on the DZ ledger. In the newly deployed version of the software stack, this account field would be set through a mechanism related to the new AccessPass which was recently introduced. However, DZ Testnet users existed before AccessPass was introduced, so this field was not set on their accounts. Immediately following the software update, one of our engineers ran a script to properly set this field using the

mechanism. At this point, the update was considered successful and complete.

Approximately 2 hours and 40 minutes later, one of network engineers noticed an anomaly in our traffic graphs:

It showed a sharp reduction in traffic across the network following the update. Immediately, 3 other engineers joined the conversation and began investigating. It was found that the validator user accounts all had 0.0.0.0, incorrectly, as their DZ IP address and that the Controller was failing to update the DZDs with configuration for these tunnels. As designed, since there was no valid IP address associated with the GRE tunnel endpoints, BGP removed routes for these hosts around the entire network. Since the change only effected validators, and all validator user accounts were updated, and all had the incorrect 0.0.0.0 IP address, they all were removed from routing tables by BGP. This meant that validator-to-validator traffic continued to flow over the public internet, however, validator to RPC traffic (where the RPC node was on DoubleZero) failed, since the RPCs remained on DZ and validators had a route to them via DoubleZero, but could not communicate over DoubleZero. Quickly, the root cause was identified: the rarely used “doublezero user update” command had a bug in it that cause the DZ IP to be set to 0.0.0.0. Engineers then rushed to update the DZ IP for each of these validators, resetting it correctly from a prior snapshot. Upon completion, the network quickly repaired itself (within 1-2 minutes) and traffic flows shot up to normal levels.

Timeline

Remediations and follow-up steps

Immediately following the discovery, the “doublezero user update” command was fixed such that it would not inadvertently set the DZ IP to 0.0.0.0 and and appropriate tests were added for it. Within a few days, a patch was made to do basic validations on all onchain structures any time they are created or updated. We also amended our deployment instructions to have a human being validate the traffic chart for a period of time after the update completes. Other remediations that will be coming soon include:

• Alert on traffic chart step changes
• There may be some alerting around BGP statistics that can be added
• Update the doublezero-controller to validate input from the blockchain before applying changes. E.g. 0.0.0.0 is an invalid address, don’t apply the change to the DZD, and raise an alert.
• Update onchain monitor to detect invalid data and emit metrics for alerting
• Audit and improve doublezero-serviceability onchain program test coverage

Conclusion

Unfortunately, users of DZ Testnet were affected for several hours from this outage. However, it is wonderful to see a real world example of IBRL’s internet failover working and minimizing the impact of the outage across validators. We are taking note of lessons learned and moving forward with improved processes and procedures as well as a ways to improve our tests to reduce the size and scope of future problems.