Fighting abuse at the edge

Anti-abuse at the network edge: From two tribes to one team.

Take a look at org charts, international standards, conferences and forums…you will observe there are two tribes; one for the ‘network’ the other for ‘applications’. It’s a distinction that’s embedded in Information Technology with the Network Layer ‘below’ all applications with a dedicated team dealing with connectivity, routers, upstreams and peering, all quite independently from the nature of the data that is flowing. Another team deals with ‘applications’; email, web services, etc., that do their job without having to consider the underlying aspects related to networking.

It’s a logical separation that has shaped the formation of jobs so that it is now becoming rare to find people with skills both in networking and applications. This separation often becomes painfully obvious when dealing with Internet abuse. Historically, anti-abuse has grown in the application world, with well established communities; forums, mailing lists, conferences and meetings, to track trends and share knowledge.

The networking community is certainly loaded with serious challenges related to the core operations of the Internet but can miss trends in the anti-abuse world directly connected to networking.

At Spamhaus we routinely observe ‘clashes’ between the staff that handle routing and the staff that administers email servers within organizations. In very simple terms, to a networking person all networks are similar and they generally feel that the purpose of their job is to ensure the best possible connectivity with every other entity on the Internet. In contrast, a mail server administrator knows that, for instance, some IP addresses only send bad traffic and so it makes sense to block them - DNSBLs are one useful tool to do this.

But one of the main trends we have observed recently is that more and more networks - full IP address ranges - are used for serious criminal activities. We have noted that tools designed for protection at the edge are not as used as they could be to prevent routing of traffic generated by dedicated criminal networks.

Let’s focus first on issues that, at Spamhaus, we believe deserve more attention and an approach to improve the overall security of networks.

How do bad guys acquire control of IP address ranges?

Basically there are four ways used to control a whole network.

1. Have the IP address range directly assigned to them by a Regional Internet Registry (AFRINIC, APNIC, ARIN, LACNIC or RIPE NCC).
2. Have an IP address range directly assigned to them by an ISP.
3. "Hijack" an abandoned IP address range by impersonating the original owner using tricks.
4. Steal portions of allocated IP address ranges, temporarily use them for their activity and then release them, leaving the innocent victim to deal with consequences.

Some of these bad guys operate their own Autonomous System (AS). In these cases, those ASes should be considered rogue and all the routes they announce over BGP should be discarded. It is also common to see chains of bad ASes in a BGP path, possibly a trick that bad guys use to play the "customer-of-a-customer" card when questioned by their upstream network provider(s).

How to detect if a BGP customer is a bad guy?

It is very unfortunate that we often see the upstream networks of rogue entities accepting and propagating routes of dedicated bad networks for very long periods - at times, months. This is typically the result of insufficient verification/vetting procedures when the customer is acquired, and later when the customer adds new routes. In many cases it appears that automated processes are set up to accept, almost blindly, whatever customers announce, possibly using easily spoofed route objects in public databases as if they were a reliable source of information (they are not). While protection on the maximum number of routes accepted is usually in place to prevent accidental leaking, this is an ineffective measure to stop criminal activity (where, for instance, rotation of rogue routes is a common practice).

We fully understand the impracticality of validating routes taken from BGP peers at exchange points or from customers which are in turn transit providers. But ISPs that connect leaves or short branches of the routing tree, owned by entities which do not have a well-known, established reputation in the routing world, should carefully examine what they accept from their BGP customers. Anti-abuse/reputation people should have a role in carrying out this delicate task, which may imply an investigation on the customer's abuse and routing history.

Applying DROP, EDROP and ASN-DROP at the edge

For many years we have included in our public DROP and EDROP lists the networks belonging to the first and second category respectively, with the hope that organizations could use these lists to deny Internet connectivity altogether to those networks. More recently we have started publishing ASN-DROP, which as suggested by the name is a list of rogue Autonomous Systems.

Many organizations are indeed using this family of lists with success, however their implementation has been traditionally hampered by the fact that connectivity issues are in the hands of networking teams, which are often afraid to create problems by blocking complete networks and are often too busy to recognize that such networks are fully controlled by cyber criminal groups. The bad activity emanating from these networks is very often affecting horizontally several applications and protocols, such as email, http, telnet/ftp/ssh, control of botnets. Edge routers are therefore the natural place where blocking should occur.

Clearly, criminals do not like being identified and blocked for good: Networks listed in DROP/ EDROP can be enormously penalized in terms of connectivity, and TCP connections from them are often blocked at the application level. Furthermore, the IPv4 depletion makes it very difficult and expensive for the cyber crime groups to find new IP address ranges. The end of IPv4 availability brought a dramatic increase of the third and fourth type - hijacked and stolen networks.

In some cases, the groups using these networks do not route them globally, but manage to establish direct peering with the service providers whose users they want to target. This decreases their risk to be noticed; those networks never appear in global routing tables. Operators of large email facilities are particularly exposed to this problem, but the issue is rather general.

Applying the Botnet Controller List (BCL) at the edge

This trend of cyber crime drags the whole networking world right in the middle of the fight to abuse. The main routing protocol, BGP, has become the most important vector and is therefore also the place where defense should occur: This is why we have been proposing BGP-based solutions for quite some time.

Of course, blocking bad networks is not enough. There are always millions of compromised bot machines that emanate nefarious activity but, more importantly, thousands of machines that control bot members. Disruption of communication between these machines and the bot members is highly desirable and effective, and this is best done at the network edge to protect from the threat of botted machines located within the network. Botnet controllers are typically located within legitimate networks that host either a server controlled by criminals or a legitimate device that has been compromised (see for instance 2017 Botnet threat report).

Our dynamic BCL list can be applied at the BGP level to disrupt botnets by blackholing Command & Control systems at the single IP address level, by diverting routing into a blackhole or a sandbox. This list is not very large, but the trend is certainly toward a size growth. Some organizations with tight security may wish to block larger sets of abusing IP addresses (such as abused IoT systems) at the network edge, but current technology often makes it difficult or impossible to use large lists (possibly including millions of records) on edge devices, particularly when timely and frequent updates are required. So, these threats have put an unprecedented pressure on routing technology, due to the need to blackhole bad traffic with a considerable granularity (the single IP address level) and in a highly dynamic way.

BGP Flowspec: the future?

Luckily, the router industry has picked up the challenge. One technology designed for such purpose is Flowspec, defined in RFC5575 and RFC7674 by engineers of Cisco Systems, Juniper Networks, Arbor Networks and NTT America. In Flowspec ACL rules, rather than being statically uploaded on the edge device, are dynamically updated in real time by an external controller device in the form of a special BGP flow. The flow can of course reach all the edge routers of an organization simultaneously. Flowspec rules can match on source or destination IP addresses, ports and other parameters, and action can range from packet drop to rate limiting, and sometimes to traffic sampling and forwarding to special collectors.

Threats on the Internet are often very active and damaging within seconds from the time they show up for the first time. Fast detection and real time delivery to users are extremely important for an effective defense. We are working on both fronts. On one side, we constantly try to speed up detection of badness to incorporate threats into products as quickly as possible. But having our data reaching customers very quickly is perhaps even a larger challenge that we are facing, because environments at the customer side are heterogeneous. The BGP interface offered by Flowspec and in the process of being standardized represents an important step toward consistent and effective delivery of data to edge devices.

While at this time we do not have Flowspec interfaces for our data, this technology appears to be extremely promising for the delivery of our BCL IP address data. Blocking botnet controllers at the edge level would allow to prevent internal machines running malware to contact their controllers and therefore avoid further - often catastrophic - damage, and to disclose their existence to the network administrators. We would be delighted to hear from organizations willing to experiment with this new technology and work with us toward a widespread deployment.

Conclusion

In summary:

• We believe that networking teams should work to acquire more knowledge and skills in the anti-abuse arena, possibly getting input from application folks that have considerable experience on this topic. This may require some adjustments in the internal organization charts of several companies, but in many cases improving cooperation and communication between networking and anti-abuse teams may be already of help.
• ISPs offering connectivity/transit to customers should carefully verify the route announcements, making sure that they are not propagating bad routes.
• We encourage organizations to use our free DROP family of tools at the network edge. This is normally rather easy to do, as updates do not need to be very frequent.
• Flowspec appears to be an extremely promising technology to supply dynamic data in real time to gateway router/appliances, and we would like to hear from anybody interested in starting to use this technology.

About Spamhaus

The Spamhaus Project is an international nonprofit organization whose mission is to track the internet's spam operations, to provide dependable realtime anti-abuse protection & threat-intelligence for Internet networks and to work with Law Enforcement Agencies to identify and pursue spammers worldwide. The number of internet users whose mailboxes are currently protected by Spamhaus DNSBLs now exceeds 2.7 Billion. Founded in 1998, Spamhaus is based in Geneva, Switzerland and London, UK and is run by a dedicated team of 30 investigators and forensics specialists located in 9 countries.