“Ex uno plures”: Finding and blocking a spanning set of IP addresses when starting from just one
Introduction
IP addresses make the Internet work. An Autonomous Systems (“AS”), better known by its number (“ASN”), is a collection of network prefixes and routing policies under a single administrative control, such as an ISP, corporation, or another organization. When an organization is allocated IP addresses from a Regional Internet Registry, an AS is used to manage and control the routing of traffic to and from the IP address blocks announced by that autonomous system. DomainTools itself, for example, has its own ASN, AS17318, to manage the routing for the IP address blocks we own.
This direct relationship between ASNs and IP addresses creates room for some interesting pivots. Given an initial IP address, such as an IP found in a log file, you may want to find the Autonomous System that originated that IP. Having found that ASN, you may then want to get the complete set of IP addresses (IPv4 and/or IPv6 CIDR blocks) originated by that ASN. This piece explains how to do that pivoting using data from the Route Views Project at the University of Oregon.
Since routing table entries may include deaggregated and redundant announcements, we explain how to condense what we find down to a minimal set, also known as a spanning set, of prefixes sufficient to cover all the listed IPs, and then show how to use them, for example, for firewall ‘ipset’ rules.
The First Step: Mapping an IPv4 Addresses to the ASN Announcing It
We’ll do this using IP-to-ASN data available from Oregon Route Views as a publicly queryable DNS zone. For example, perhaps we want to know the ASN associated with www.google.com
Begin by resolving the FQDN (fully qualified domain) to find the IP address of interest:
$ dig www.google.com +short
142.250.69.196
Then to find the ASN, make a 2nd query against the DNS “asn.routeviews.org” zone for a TXT record with the address of interest reversed chunk-by-chunk and prepended to the asn.routeviews.org zone. For example, for 142.250.69.196:
$ dig 196.69.250.142.asn.routeviews.org txt +short
"15169" "142.250.69.0" "24"
So, we now know that www.google.com uses 142.250.69.196, which is announced by AS15169 as part of the 142.250.69.0/24 CIDR block.
We can automate that process with a little shell script:
$ cat /usr/local/bin/ip2asn
#!/bin/sh
origip=`echo $1`
revip=`echo $1 | sed \ 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)/\4.\3.\2.\1/'`
listing=`host -w -t txt ${revip}.asn.routeviews.org 2>/dev/null | tail -1 `
#echo "${listing}\n"
listing2=`echo ${listing} | awk '{print $4}' | sed 's/"//g'`
echo "${listing2} ${origip}"
$ ip2asn 142.250.69.196
15169 142.250.69.196
Getting the Full Set of CIDR Blocks Originated by an ASN
Our next task is to find the full set of CIDR blocks originated by an ASN. One approach we’ve previously discussed leverages the website https://bgp.he.net/ . You could simply search for the ASN and then check the IPv4 (or IPV6) prefix tab. Of late, however, that site has become increasingly busy and has from time-to-time been unavailable due to load; other times, that site has had to limit access for public users.
Therefore, the time may be ripe to explain how you can easily extract a list of prefixes on your own system using downloaded data. We’ll leverage MRT-format data files from Oregon Route Views (for info on MRT format, see https://www.rfc-editor.org/rfc/rfc6396). Data may be available from different collector locations from Route Views (see https://routeviews.org/ for a list), but for this post we will use data only from route-views2.oregon-ix.net:
To retrieve the IPv4 zone file from route-views2.oregon-ix.net for 00:00 UTC on February 17th, 2023, we’d do:
$ wget https://routeviews.org/bgpdata/2023.02/RIBS/rib.20230217.0000.bz2We then rename it to signal that this is an IPv4 RIB (and to avoid having it conflict with the IPv6 RIB we’ll download later):
$ mv rib.20230217.0000.bz2 rib-ipv4.20230217.0000.bz2The file should then look something like:
$ ls -lh rib-ipv4.20230217.0000.bz2
-rw-r--r-- 1 joe staff 102M Feb 16 16:00 rib-ipv4.20230217.0000.bz2
To parse that and similar MRT files, we’ll use zebra-dump-parser (https://github.com/rfc1036/zebra-dump-parser). To install that file:
$ git clone https://github.com/rfc1036/zebra-dump-parser.git
# cd zebra-dump-parser
# cp zebra-dump-parser.pl /usr/local/bin/zebra-dump-parser
# chmod a+rx /usr/local/bin/zebra-dump-parserYou can then dump the data files by saying:
$ bzcat rib-ipv4.20230217.0000.bz2 | zebra-dump-parser.pl > v4-routes.txt 2> BGPERR
$ sort -u v4-routes.txt > v4.txt
$ wc -l v4.txt
978026 v4.txt
Some routes might be ones to exclude, including:
- 0.0.0.0/0 (normally the largest IPV4 prefix you should expect to see will be a /8)
- IPv4 prefixes more specific than /24 (e.g., you may want to remove any /25’s through any /32’s)
After removing those, we’re left with 975,196 IPv4 routes from that file. The top of that file looks like:
$ more v4.txt
1.0.0.0/24 13335
1.0.128.0/17 23969
1.0.128.0/18 23969
1.0.128.0/19 23969
1.0.128.0/24 23969
1.0.129.0/24 23969
[etc]
At this point we have a list of prefixes and the ASNs that are originating them. We can then search that file for an ASN of interest, such as:
$ grep " 15169" v4.txt | wc -l
841
$ grep " 15169" v4.txt
104.154.0.0/15 15169
104.154.0.0/19 15169
104.154.128.0/19 15169
104.154.160.0/19 15169
104.154.192.0/19 15169
[etc]
Using “cidrmerge” to Find the Set of Minimal Covering Prefixes
There may be deaggregated routes and/or overlapping routes in that list. After selecting a set of prefixes, we can drop the 2nd column and pump the prefixes through cidrmerge ( https://github.com/kfeldmann/cidrmerge) to get a minimal equivalent list:
$ grep " 15169" v4.txt | awk '{print $1}' | cidrmerge > 15169.txt
$ wc -l 15169.txt
80 15169.txt
Going from 841 prefixes to 80 prefixes represents a nice bit of CIDR prefix aggregation “magic”. Something really helpful when we want to take action on this data!
IPv6 MRT-Format Routing Data
Many ISPs are now multihomed to both IPv4 and IPv6, so let’s look up the IPv6 prefixes as well as the IPv4 prefixes. To be able to do that, we’ll retrieve the IPv6 zone file from route-views6.oregon-ix.net for midnight (00:00) February 17th, 2023:
$ wget https://routeviews.org/route-views6/bgpdata/2023.02/RIBS/rib.20230217.0000.bz2
$ mv rib.20230217.0000.bz2 rib-ipv6.20230217.0000.bz2
$ ls -lh rib-ipv6.20230217.0000.bz2
-rw-r--r-- 1 joe staff 22M Feb 16 16:00 rib-ipv6.20230217.0000.bz2
$ bzcat rib-ipv6.20230217.0000.bz2 | zebra-dump-parser-v6.pl > v6-routes.txt 2> BGPERR-V6
$ sort -u v6-routes.txt > v6.txt
$ wc -l v6.txt
184450 v6.txt
Again, there are some IPv6 routes might be ones you want to exclude, such as:
/0/112, /123, /124, /125, /126, /127, /128
After removing those, we’re left with 184,252 IPv6 routes from that file. The top of that file looks like:
$ more v6.txt
2001:0004:0112::/48 112
2001:0200:0900::/40 7660
2001:0200:0e00::/40 4690
2001:0200::/32 2500
2001:0200:c000::/35 23634
2001:0200:e000::/35 7660
2001:0218:2002::/48 2914
2001:0218:2200::/40 18259
2001:0218:3004::/48 20940
2001:0218:8000::/38 2914
2001:0218::/32 2914
2001:0240::/32 2497
2001:0250:0002::/48 24348
[etc]
We can then search that file for an ASN of interest, such as:
$ grep " 15169" v6.txt | wc -l
86
$ grep " 15169" v6.txt
2001:4860:4864::/48 15169
2001:4860::/32 15169
2404:6800:4001::/48 15169
2404:6800:4002::/48 15169
2404:6800:4003::/48 15169
2404:6800:4004::/48 15169
2404:6800:4005::/48 15169
2404:6800:4006::/48 15169
[etc]
Finding the Minimum Covering IPv6 Prefix Set
Just as with IPv4, there may be opportunities to condense the number of IPv6 prefixes. The cidrmerge program we used for the IPv4 prefixes isn’t able to handle IPv6 CIDRs, so we’ll use cidr-merger (https://github.com/zhanhb/cidr-merger) for our IPv6 CIDRs instead:
$ grep " 15169" v6.txt | awk '{print $1}' | cidr-merger > 15169-v6.txt
This example condenses our 86 original IPv6 CIDR prefixes down to just 13:
$ wc -l 15169-v6.txt
13 15169-v6.txt
Applying the Above: Using ‘ipset’ to Add CIDR Blocking Rules to ‘ufw’ on Debian 11 (“Bullseye”) and Debian 12 (“Bookworm”)
Once you’ve obtained a minimum set of covering prefixes, you will typically want to do something with them. Often you may find yourself wanting to look those prefixes up in DNSDB, but other times you may simply want to load them into a firewall – perhaps the popular ufw (“Uncomplicated Firewall”) that’s popular on many versions of Linux, including Debian 11 (“Bullseye”) and Debian 12 (“Bookworm”).
We cannot provide a comprehensive introduction to all types of firewalling for all operating systems in one blog post. This section is meant solely to give you a conceptual overview of using ipsets to block traffic by CIDR netblock under Debian. This section will not and cannot give you the in-depth training you may need to run a firewall safely and effectively. Please consult a local firewall mentor or firewall documentation for assistance. Proceed at your own risk, and resolve any questions you may have before trying any of the following, particularly on a production firewall!
We’ll begin by assuming that you’ve already got ufw installed and configured, but that you’re not currently using ipset rules to block CIDR prefixes. If not already present, on Debian 11 or 12 you should also install the following packages:
# apt install ipset iptables-persistent netfilter-persistent ipset-persistentWe will create two ipsets, one for IPv4 CIDRs and one for IPv6 CIDRs (obviously, if you don’t have IPv6 connectivity, you can skip the IPv6 bits below). When created with default values for maxelem and hashsize (as we’ll do for the cidr-blocklist-ipv6 ipset), cidr-blocklist ipset should be able to accommodate up to 65,536 entries. On the off chance that you’re going to be an “enthusiastic” IPv4 “firewaller,” we’ll preemptively increase the capacity of the IPv4 ipset to 262,144 entries:
# ipset create cidr-blocklist nethash maxelem 262144 hashsize 32768
# ipset create cidr-blocklist-ipv6 nethash family inet6Once the ipsets have been created, you can add IPv4 CIDRs to the IPv4 cidr-blocklist ipset with commands such as:
# ipset add cidr-blocklist CIDR-prefix-here
# ipset add cidr-blocklist another-CIDR-prefix-here
[etc]
Note: if you’re adding IPv6 entries, replace “ipset add cidr-blocklist” with “ipset add cidr-blocklist-ipv6“.
Once you’ve finished loading your CIDR block entries, confirm that the rules you’ve just loaded into ipset look right to you:
# ipset list cidr-blocklist
# ipset list cidr-blocklist-ipv6To keep an eye on the total size of our ipset rules (remembering our 262,144 IPv4 rule and 65,536 IPv6 rule limits), use:
# ipset list cidr-blocklist | wc -l
# ipset list cidr-blocklist-ipv6 | wc -lNow we’ll want to connect the ipset(s) with iptables (iptables runs “behind the scenes” for ufw):
# iptables -I INPUT -m set --match-set cidr-blocklist src -j DROP
# iptables -I FORWARD -m set --match-set cidr-blocklist src -j DROP
# ip6tables -I INPUT -m set --match-set cidr-blocklist-ipv6 src -j DROP
# ip6tables -I FORWARD -m set --match-set cidr-blocklist-ipv6 src -j DROPWhen everything’s ready, make your current rulesets persistent, otherwise they may disappear if/when you reboot:
# netfilter-persistent save
Finally, you may want to ensure that netfilter-persistent is routinely started at reboot:
# dpkg-reconfigure ipset-persistent
# dpkg-reconfigure iptables-persistent
# systemctl enable netfilter-persistent
# systemctl start netfilter-persistent
# systemctl status netfilter-persistentWith that, you should now have CIDR blocking rules active and persistent in your firewall.
Conclusion
Cyber threats to your organization can come from many different places, but all involve IP Addresses. In this blog post we demonstrated that you can use freely available data and open source tools to pivot from a starting IP Address to the ASN responsible for its routing information to the spanning set of IP address CIDR blocks under that ASN. We then showed how you can take action on those CIDR blocks by adding them as rules into a firewall.
This blocking activity is only one of a large set of detection, enrichment, and investigation processes your network defender team might need to perform. DomainTools has the data and expertise if you need to do more.
Thank you to the University of Oregon Route Views Project for making available their fantastic set of IP Address to ASN mapping data.
