This article assumes a base understanding of Internet and networking concepts.
2024-07-04 – Edit: We are currently working with MikroTik
A sharp increase of DDoS attacks have been observed since the beginning of 2023. A new trend is to send high packet rate attacks though. This article introduces the findings of our teams in order to bring new insights regarding this threat.
Introduction
Distributed Denial of Services attacks (DDoS) are a longstanding issue which remains an effective way to impact the availability of an online service. Over the past decade, several actors demonstrated how easily they could raise an army of zombie devices with their botnet using a wide range of techniques, from using phishing to install malware on a desktop host up to leveraging different kind of vulnerabilities affecting IoT devices, CCTV, or residential routers.
These botnets have been largely used to send DDoS attacks using tens of thousands of compromised devices all around the world. The patterns of attack were often the same: just craft as many packets as you can in an attempt to reach the highest possible bit rate (or packet rate) to cripple your target’s networking capabilities. This is what happened with the Mirai botnet back in 2016 which has been the first one to generate more than 1 Tbps (Terabit per second). Since then, several botnets have far surpassed the original Mirai up to 3.47 Tbps in 2021. However, 1+ Tbps attacks remained quite rare… until recently.
Since the beginning of 2023, we noticed a sharp increase of DDoS attacks, both in frequency and intensity. Moreover, starting from November of the same year, a significant acceleration of the trend has been observed by our teams at OVHcloud: while DDoS reaching 1 Tbps or above were occasional, they aren’t anymore. In the past 18 months, we went from 1+ Tbps attacks being quite rare, then weekly, to almost daily (averaged out over one week). The highest bit rate we observed during that period was ~2.5 Tbps.
Interestingly, the recent announcement of the 911 S5 Botnet dismantling between May 25th and May, 30th 2024 loosely coincides with a significant decline of DDoS attacks, which started by mid of May. However, we cannot affirm with certainty these events are linked together.
While attacks frequency is seemingly back to normal, we are still observing a large amount of DDoS attacks involving packet rates greater than 100 Mpps (Millions packets per second).
What about packet rate attacks?
Usually, most DDoS attacks rely on sending a lot of garbage data to sature the bandwidth (network-layer attacks) or sending a lot of application requests to cause excessive CPU or memory usage (application-layer attacks). Of course there are other methods to leverage: among those are packet rate attacks or packets per second based attacks.
Packet rate attacks objective is to overload the packet processing engines of networking devices close to the destination, instead of starving the available bandwidth. The general idea is to cripple the infrastructures in front of the targeted service (e.g., load-balancers, anti-DDoS systems, …), thus possibly impacting a large infrastructure as collateral damage. Simply put: instead of trying to find holes in anti-DDoS systems, just take them down!
Packet rate attacks are quite effective since dealing with a lot of small packets is usually harder than dealing with bigger but less numerous packets. This is because the computing cost is generally higher. For instance, if you’re using software to process packets, each packet means one memory access at the very least (excluding possible copy, access to stored data such as connection tables, …), instead of simply iterating over more bytes. If you’re using hardware, while packet processing performances are not necessarily affected by the packet rate, the processing pipeline probably depends on other components, such as memory (again!), which could be stressed a lot by high packet rates. In those conditions, you may reach some limits due to the very high rate or just because you don’t have enough buffers to store it all, which will probably induce latency or performance losses. We can summarize this problem into a single sentence: if your job is to deal mostly with payloads, bandwidth may be the hard limit ; but if your job is to deal mostly with packet headers, packet rate is the hard limit.
That’s why in most conditions, dealing with small packets is harder than dealing with big packets. In a nutshell, a 10 Gbps DDoS attack with big packets (1480 bytes) yields ~0.85 Mpps: in comparison, 10 Gbps with the smallest packets (84 bytes on wire for Ethernet) yields a massive ~14.88 Mpps.
In the context of standard Internet MTU (1500), you can fit 17 times more packets on wire when generating only the smallest packets possible, compared to big packets. To give an idea of the computing capabilities needed in the context of DDoS mitigation, consider a 100 Gbps link will fit ~149 Mpps line-rate: this allows up to 6 nanoseconds of processing time per packet, or 18 cycles for a single compute pipeline running at 3 GHz clock speed. Saying it otherwise: even with tens of parallel pipelines, you don’t have many cycles available, especially if you need to access some memory.
As a side note, this is one of the reason why OVHcloud is building its own networking appliances for DDoS infrastructures. We use a combination of FPGA and userland software (DPDK) to build appliances with off-the-shelf hardware. Each network appliance used to mitigate DDoS attacks is designed, implemented and maintained in-house (just like the rest of our anti-DDoS systems by the way!). Thanks to this streamlined approach, we are able to finely adjust performance expectations and constraints, then ensure our appliances match accordingly.
The rise of (big) packet rate attacks
DDoS attacks relying on high packet rate are not new and network operators all over the world had to face such attacks at least once. As an example, the highest publicly-known packet rate attack was reported by Akamai in 2020 and reached 809 Mpps. However, despite this big figure, a vast majority of packet rate attacks are well below 100 Mpps. This is probably because generating a lot of small packets is harder than generating big ones (you need much more compute – similarly to processing) and is harder to hide from network monitoring and anti-abuse systems.
Packet rate attacks started to seriously get some attention at OVHcloud two years ago, after we were hit — but successfully mitigated — by a gigantic UDP flood for more than 6 hours, reaching ~700 Mpps in average for ~4 hours.
In the past 18 months, and especially in the past 6 months, we noticed a sharp increase of DDoS attacks leveraging packet rates greater than 100 Mpps. We went from mitigating few of them each week, to tens or even hundreds per week. Our infrastructures had to mitigate several 500+ Mpps attacks at the beginning of 2024, including one peaking at 620 Mpps. In April 2024, we even mitigated a record-breaking DDoS attack reaching ~840 Mpps, just above the previous record reported by Akamai.
This attack was 99% of TCP ACK, originating from approximately 5,000 source IPs. Interestingly, the 1% remainder was a DNS reflection attack, leveraging ~15,000 DNS servers to amplify the traffic, which is not really efficient when trying to achieve high packet rate attacks.
While the attack was distributed worldwide, 2/3 of total packets entered from only 4 PoPs, all located in the US with 3 of them being on the west coast. This highlights the capability of the adversary to send a huge packet rate through only few peerings, which can prove very problematic. Generally, anti-DDoS response teams — not only at OVHcloud — assumes that it’s really hard to send massive DDoS from only few geographical locations. Based on this assumption, our infrastructures are scaled horizontally, spread worldwide, and they absorb the load more easily. However, the traffic distribution of the 840 Mpps attack has strongly questioned this assumption. While we do have the local capacity to mitigate this attack, we will consider to adjust the general scaling and distribution model of our anti-DDoS infrastructures to ensure to cope with future (and probably bigger) attacks, just as we do today.
In the end, the significant rise of high packet rate attacks led us to deep dive the topic. As a worldwide cloud provider, OVHcloud is scrubbing many DDoS attacks on a daily basis, which gives us an exceptional vantage point on this topic. We wanted to understand how these attacks were generated, where they came from, and possibly determine what we could do to better protect our infrastructures and customers against this kind of firepower.
Unveiling evil core routers
During our analysis campaign, manually dissecting almost a hundred packet rate attacks ranging from 100 up to 500 Mpps, we noticed that many attacks originated from not-that-many sources, which are sending a large proportion of the total traffic. We established a list of well-known offending IPs capable of generating at least 1 Mpps each, and decided to dig further.
We analyzed the top 70 IPs issuing the highest packet rates, up to 14.8 Mpps per IP. These IPs belongs mostly to Autonomous Systems (AS) in Asia, but Europe, Middle-East, North American and South America are also represented. The ASs identified seem to belong mostly to business ISPs or Cloud Connectivity providers.
To understand what kind of devices were involved in these DDoS, we used Onyphe to determine if these IPs were known. Indeed, a great part of these IPs are known as MikroTik Routers and are exposing on the Internet — at least — the configuration webpage.
At this point, it remains possible this traffic is generated either by servers located behind a router configured with NAT, using spoofed IP, or leveraging some kind of weird TCP reflection. However, we quickly dismissed these hypotheses due to the improbability of encountering such a significant number of identified MikroTik routers, given that MikroTik does not hold a proportionately large market share. In addition, exposing an administration interface reflects poor management practices. It increases the device’s attack surface and can facilitate its compromise by an attacker. Moreover, RouterOS — MikroTik’s operating system — has suffered from several critical CVE over the past years. Even if a patch has been released, these devices may have not been patched so far.
Since the HTTP interface is open on most of the devices, it is possible to use it to recover the version of RouterOS running on said devices. Half of them are running a RouterOS version prior to 6.49.8 – released on May, 23rd. 2023 – and the other half is running a later version. For instance, devices running RouterOS 6.49.14 – released on Aprit, 4th. 2024 – have been identified.
We have been surprised to discover devices with a recent firmware being potentially compromised though. As far as we know, no vulnerability affecting RouterOS 6.49.14 and later versions have been publicly published so far. A possible explanation would be these devices may have been patched after their compromission.
We can’t say yet why these devices are involved in coordinated DDoS attacks, but one possible hypothesis could be the “Bandwidth test” feature from RouterOS. It allows the administrator to test the real throughput of a router by crafting packets and perform stress tests. Coincidentally, the documentation states the following: “Up to RouterOS version 6.44beta39, Bandwidth Test used only single CPU core and reached its limits when core was 100% loaded. Bandwidth Test [now] uses all available bandwidth (by default) and may impact network usability“. This is quite interesting since we mostly identified RouterOS v6.44 or above among the offending IPs.
99,382 devices available on the Internet
Using SNMP on devices exposing it, we have been able to determine what kind of devices were capable to issue such a high packet rate. As expected these are not residential routers, but rather core network devices.
The results highlight the MikroTik CCR series, which stands for Cloud Core Router. Indeed, SNMP returned several CCR1036-8G-2S+ and CCR1072-1G-8S+.
In order to have an overview of how many devices could be compromised and used in such massive packet rate DDoS attacks, we used Onyphe once again to search for CCR devices wide open on the Internet. 99,382 CCR devices were identified.
We can see that both device models involved in the packet rate attacks observed by our teams – CCR1036-8G-2S+ and CCR1072-1G-8S+ – represents at least 40,000 devices open on the Internet. The CCR1036-8G-2S+ is the most found device on the Internet with 30,976 occurrences, and the CR1072-1G-8S+ is the 4th most found device with 9,353 occurrences. Since we still don’t know what kind of vulnerability has been leveraged to compromise these device models, we can’t say yet whether other CCR models could be compromised as well or not. Nonetheless, exposing the administration panel on the Internet remains a big risk for the security of the device though.
Even more evil models?
Thanks to internal data sharing and discussions, we were reminded about a L7 attack which occurred during November 2023. At the time, MikroTik routers were identified, but it did not ring bells. This attack reached 1.2 millions requests per second using HTTPS and involved roughly 3,000 source IPs. Considering our recent findings, we decided to take another look at it.
In order to understand which kind of routers were involved, we recovered the 3,000 IPs involved in the attack. Past investigations shown approximately 700 IPs identified as MikroTik routers and exposing the port TCP/8291. However, we did not check at the time which kind of device were involved.
Just like we did before – we made a quick research using Onyphe. We first did it manually and quickly found the exact same results as before: Cloud Core Routers devices were involved in this attack as well. Among the IPs identified as CCR devices, over 10% were publicly exposing SMNP. Once again, we found core network devices: for instance, 16% of exposed devices are CCR1009-7G-1C-1S+ which is another similar model. This specific model is the 2nd most exposed model on the Internet according to our findings described in the previous section.
Determining the RouterOS version running on identified devices 8 months ago is probably not relevant, since we can’t say which version was running on the device at the time. However, analysis shows that 22% of the devices are running a RouterOS released between July 1st. 2023 and July 1st 2024. The most recent version is v6.49.15 (2024-05-24), while the oldest one is v5.20 (2012-08-15).
Unfortunately, we do not have enough data anymore to provide possible requests rate depending on the device model.
As said before, it’s still hard to know how these devices have been compromised. Likewise, it is difficult to determine whether these attacks are related and if the same botnet is involved in the high packet rate attacks and the L7 attacks. In any case, highlighting the presence of network core devices in L7 attacks remains interesting since it demonstrates how much of a threat these devices could represent.
Let’s do some math
In order to hint at the possible capacity of a botnet leveraging these devices, we decided to focus on packet rate attacks which are well identified.
A quick overview of advertised capabilities for identified devices shows they are capable of handling up to 28 Gbps – for the CCR1036-8G-2S+ – or 80 Gbps – for the CCR1072-1G-8S+. In terms of packet rate, they claim to handle the approximate theoretical packet line-rate with respect to their bandwidth processing capabilities. As a reminder for readers, you can fit ~1.5 Mpps at most in a 1 Gbps link. Depending on the capabilities of devices to craft packets instead of just forwarding them, which is generally done in CPU whenever necessary instead of using ASICs, the amount of packets per second possibly generated by the device could vary greatly and be far lower than advertised processing capabilities. Moreover, generating traffic using the hardware of a compromised device is far from trivial: an intruder will most likely attempt to leverage CPU capabilities only or built-in features whose use has been slightly diverted from their intended usage.
In the context of this thought exercise, we assume networking devices are capable of crafting packets at a rate equal to 10% of their maximal capacity, leading to the following assumptions:
- CCR1036-8G-2S+ should be able to generate 4 Mpps each
- CCR1072-1G-8S+ should be able to generate 12 Mpps each
These estimations seems mostly accurate when compared to what we actually observed in terms of packet rates depending on the identified model. Considering the available CPU on these devices, we think these estimations are quite conservative: for instance, in the case of CCR1036-8G-2S+ devices, generating more than 4 Mpps with 36 cores @ 1.2GHz should not be difficult.
At this point, anyone can do the math to build a naive scale model of a botnet leveraging these devices. Considering a rate of 1% (arbitrary conservative value) of exposed devices being compromised, and focusing only on the first two models we identified as compromised:
- ~ 300x CCR1036-8G-2S+ / 4 Mpps each
- ~ 90x CCR1072-1G-8S+ / 12 Mpps each
Such a botnet would theoretically be able to generate 2.28 billions packets per second (or Gpps).
In terms of requests per second capacity for L7 attacks, we do not have enough data to emit a strong enough hypothesis. We can only affirm theses devices seems well able to perform L7 attacks and high packet rate attacks. Attempting to estimate possible L7 capacity is left as an exercise for the reader.
Conclusion
The evidences highlighted in this article suggests a new trend: leveraging compromised network core devices to perform powerful attacks. Even if MikroTik devices may have been already involved in DDoS attacks, no evidence yet was indicating these botnets were relying upon network core devices.
While any high-end server could perfectly be capable of generating packet rates at this scale, they will probably be limited by the actual amount of available public bandwidth. Because of their location within network, core devices are much less affected by this assertion: they are generally connected to even bigger devices using high-capacity network links. Moreover, the mitigations implemented by network administrators to identify abnormal behaviours on the network — such as servers initiating DDoS attacks — can be bypassed in this case because routers are generally not subject to these measures.
Depending on the number of compromised devices and their actual capabilities, this could be a new era for packet rate attacks: with botnets possibly capable of issuing billions of packets per second, it could seriously challenge how anti-DDoS infrastructures are built and scaled. We will definitely take this new threat into account when thinking about how we build and scale our own anti-DDoS infrastructures to make sure we remain out of any possible impact.
To conclude, the security of network devices is both a pressing concern and an actual issue. Since January, 1st. 2024, more than 10 critical CVEs have been released affecting various network devices from multiple vendors (Ivanti, Cisco, Fortinet, Palo Alto, ..). Some of them were even exploited in the wild before the public release of the CVE. However, this is the first time we face network core devices participating in coordinated DDoS attacks. This is somewhat concerning since identified devices are designed for small & medium-sized network cores, and much more powerful equipment is available today.
Closing note: we reached out to MikroTik through several communication channels to expose them the situation. MikroTik reached out to us on 2024-07-04 and is investigating the possible causes of the issue.
We are also currently contacting different AS to report them the issue.
Head of Computer Security Incident Response Team
Christophe Bacara
Technical Lead - Network Security Engineering at OVHcloud