The incident at hand was related to issues originating from maintenance work done in Frankfurt under the last maintenance announcement. Internal routing convergency with the OSPF protocol resulted in a majority of customer routes flapping and re-routing frequently due to a software error of the router control plane we use at this time. This mainly affected legacy tunnel customers but had short impact for more destinations.
Timeline: At 0:45 am CEST: Maintenance was concluded and the colleague responsible for the maintenance falsely marked it as successful and headed off for the shift At 1:03 am CEST: After alerts and customer reports, other on call engineers were then alerted and responded At around 1:20 am CEST: The issue was identified and potential fixes were discussed At 1:35 am CEST: Our head network engineer was called in and responded At 1:40 am CEST: Network and on call engineers began to apply fixes As of around 2:00 am CEST: The general issue had been solved and confirmed
Furthermore single customer issues that mainly related to Legacy network setup structure bottlenecks were solved later on and communicated individually.
We understand the severance of such occurance and are working pro-actively to prepare ourselfs better for such scenarios in the future. Our new planned network concept will also greatly help to reduce such legacy setup bottlenecks in their entirety, announcements will follow in regards to that. We thank you for your understanding and patience, please contact support should you have further questions.
Posted Aug 24, 2025 - 09:53 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 23, 2025 - 00:00 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Aug 22, 2025 - 23:52 UTC
This incident affected: Network Edge Locations (FRA - Frankfurt Edge Network, AMS - Amsterdam Edge Network, BUC - Bucharest Edge Network, PAR - Paris Edge Network, WAW - Warsaw Edge Network, MAD - Madrid Edge Network, MRS - Marseille Edge Network, ASH - Ashburn Edge Network, MIA - Miami Edge Network, CHI - Chicago Edge Network, DAL - Dallas Edge Network, LAX - Los Angeles Edge Network, SGP - Singapore Edge Network, HKG - Hong Kong Edge Network, TYO - Tokyo Edge Network, SEA - Seattle Edge Network, LON - London Edge Network).