Using blacklists with iptables

Last Updated on 2022-07-08 by Joop Beris

If you have any kind of system connected to the Internet, you are no doubt aware that no matter how small or unimportant it might seem, it is frequently probed, tested or subject to various attempts at abuse. These attacks come from so many malicious hosts that it is impossible to keep track by hand. So I started looking for a way to implement an automated blacklist to use with iptables.

ipset

There are good solutions to detect and block hosts that are attacking your Linux system, such as denyhosts or fail2ban and I highly recommend you implement one of them depending on your needs. However, if hosts are known to be malicious in the security community, it is much more elegant to catch them at the front door, before they connect to your services. This is where a blacklist of known malicious hosts comes in.

Of course I wasn’t the first to have considered using blacklists with iptables. Many commercial firewall solutions distribute updates frequently, adding detection signatures and blacklist information. Someone must have done something similar with iptables, I thought. And indeed, a few moments spent searching online, revealed a good deal of questions and some good answers too. One method that I particularly liked, was the use of ipset to administer large lists of IP addresses inside the kernel, eliminating the need for thousands upon thousands of iptables rules. This sounded exactly like what I wanted achieve. Now all I needed was a good blacklist.

Creating the blacklist

A bit more searching lead me to the site Cyberpunk, which had both a shell script to create an extensive blacklist and a way to feed it to ipset, so iptables could use the list. I am reproducing this script in full below, should the original ever disappear. I take no credit for writing this script.

#!/bin/bash
IP_TMP=/tmp/ip.tmp
IP_BLACKLIST=/etc/ip-blacklist.conf
IP_BLACKLIST_TMP=/tmp/ip-blacklist.tmp
IP_BLACKLIST_CUSTOM=/etc/ip-blacklist-custom.conf # optional
list="chinese nigerian russian lacnic exploited-servers" 
BLACKLISTS=(
"http://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1" # Project Honey Pot Directory of Dictionary Attacker IPs
"http://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1" # TOR Exit Nodes
"http://www.maxmind.com/en/anonymous_proxies" # MaxMind GeoIP Anonymous Proxies
"http://danger.rulez.sk/projects/bruteforceblocker/blist.php" # BruteForceBlocker IP List
"http://rules.emergingthreats.net/blockrules/rbn-ips.txt" # Emerging Threats - Russian Business Networks List
"http://www.spamhaus.org/drop/drop.lasso" # Spamhaus Don't Route Or Peer List (DROP)
"http://cinsscore.com/list/ci-badguys.txt" # C.I. Army Malicious IP List
"http://www.openbl.org/lists/base.txt" # OpenBL.org 30 day List
"http://www.autoshun.org/files/shunlist.csv" # Autoshun Shun List
"http://lists.blocklist.de/lists/all.txt" # blocklist.de attackers
)
for i in "${BLACKLISTS[@]}"
do
 curl "$i" > $IP_TMP
 grep -Po '(?:\d{1,3}\.){3}\d{1,3}(?:/\d{1,2})?' $IP_TMP >> $IP_BLACKLIST_TMP
done
for i in `echo $list`; do
 # Download
 wget --quiet http://www.wizcrafts.net/$i-iptables-blocklist.html
 # Grep out all but ip blocks
 cat $i-iptables-blocklist.html | grep -v \< | grep -v \: | grep -v \; | grep -v \# | grep -v \" | grep [0-9] > $i.txt
 # Consolidate blocks into master list
 cat $i.txt >> $IP_BLACKLIST_TMP
done

sort $IP_BLACKLIST_TMP -n | uniq > $IP_BLACKLIST
rm $IP_BLACKLIST_TMP
wc -l $IP_BLACKLIST

ipset flush blacklist
egrep -v "^#|^$" $IP_BLACKLIST | while IFS= read -r ip
do
 ipset add blacklist $ip
done

Basically what this script does, is download lists of IP netblocks and IP addresses from various sites hosting such lists, strip out everything that isn’t an IP netblock or address and then put all those lines in a single text file. This results in a file containing thousands of lines (at the moment well over 44.000). It would be impossible to manage this by hand. If you set this script to run once per day from your crontab, you’ll have a fairly up to date list of malicious hosts. Please refrain from running this script too often, since the websites where the various source lists are hosted, need to pay for this traffic. Updating too often will probably get you banned.
At the very bottom, the ipset is flushed and the new list added line by line to the blacklist.

Please note that the instructions that follow, are written for Ubuntu 14.04 but they should translate to other distributions as well, since we are only using the command line.

Plugging the blacklist into iptables

Running this script from the command line will fail at the moment. While it will create the blacklist file in /etc/ip-blacklist.conf, it will not be able to load the ipset because we have not yet created an ipset called blacklist. We can create it by hand in the following way:

$ sudo ipset create blacklist hash:net

This command creates an ipset called “blacklist” of type “hash:net”. This type of ipset is used to store different sized IP network addresses, ranging from large netblocks right down to single hosts. Running the above script now will create the blacklist and populate the ipset with the created blacklist. Next we need to add a rule to iptables so that it will use the blacklist. I recommend inserting this rule at or near the top of the INPUT chain so that it will be processed early in your ruleset. Let’s take a look at the INPUT chain, so we will know where to insert the new rule.

$ sudo iptables -L INPUT -n -v --line-numbers
1   24711 4922K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
2     101  4372 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            state INVALID
3       0     0 LOG        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 flags:0x17/0x02 #conn src/32 > 20 LOG flags 0 level 4 prefix "iptables rate limit: "
4       0     0 REJECT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 flags:0x17/0x02 #conn src/32 > 20 reject-with tcp-reset
5    7347   11M ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp spt:80

....

The command above will list all the rules currently in the INPUT chain of iptables, with line numbers, which makes it easy to see where to insert the new rule. A snippet of my INPUT chain can be seen above. The first rule accepts all traffic to the loopback interface and we’ll leave that at the top. The second rule drops all incoming packets that are in an invalid state which is good. The rule below that limits the number of incoming connections to port 80. Since that is already quite specific, let’s insert our new rule above this one.

$ sudo iptables -I INPUT 3 -m set --match-set blacklist src -j DROP

This command inserts our rule at position 3 in the INPUT chain and matches incoming traffic to the set  called “blacklist”, dropping corresponding traffic. At this point, iptables is silently dropping all traffic coming from hosts and netblocks in the blacklist. However, if we want to see what is being blocked, we need to add a logging rule above rule 3. Since I am curious to see what gets blocked, I added the following rule.

$ sudo iptables -I INPUT 3 -m set --match-set blacklist src -j LOG --log-prefix "IP Blacklisted: "

This rule is inserted in position 3, pushing the drop rule to position 4. Now, each incoming traffic matching our blacklist is first logged with the prefix “IP Blacklisted:” by rule 3 and then discarded by rule 4. A look at the logging reveals this:

Apr 22 11:53:15 beris kernel: [44542.993330] IP Blackisted: IN=eth0 OUT= MAC=00:19:99:3b:92:81:9c:c7:a6:1d:40:d6:08:00 SRC=188.240.132.151 DST=X.X.X.X LEN=60 TOS=0x00 PREC=0x00 TTL=55 ID=15222 DF PROTO=TCP SPT=57238 DPT=443 WINDOW=14600 RES=0x00 SYN URGP=0
Apr 22 11:53:16 beris kernel: [44543.993847] IP Blackisted: IN=eth0 OUT= MAC=00:19:99:3b:92:81:9c:c7:a6:1d:40:d6:08:00 SRC=188.240.132.151 DST=X.X.X.X LEN=60 TOS=0x00 PREC=0x00 TTL=55 ID=15223 DF PROTO=TCP SPT=57238 DPT=443 WINDOW=14600 RES=0x00 SYN URGP=0
Apr 22 11:53:18 beris kernel: [44545.993815] IP Blackisted: IN=eth0 OUT= MAC=00:19:99:3b:92:81:9c:c7:a6:1d:40:d6:08:00 SRC=188.240.132.151 DST=X.X.X.X LEN=60 TOS=0x00 PREC=0x00 TTL=55 ID=15224 DF PROTO=TCP SPT=57238 DPT=443 WINDOW=14600 RES=0x00 SYN URGP=0

Here we see several attempts by a host with IP address 188.240.132.151 attempting to connect to port 443 (https) of the server. This connection will never be established because iptables is on the job.

Since the rules we have written seem to be working, we need to save them so that they will be loaded again should iptables get restarted. On Ubuntu, we can use iptables-persistent for that purpose. Issuing the command below will save the rules in /etc/iptables/rules.v4.

$ sudo service iptables-persistent save

Creating the ipset at boot

So far, we’ve managed to download and compile an extensive blacklist, learned how to load it into ipset and plug that ipset into iptables as a blacklist. We’ve also set up a rule to log detected connection attempts from our blacklist. So far so good. There is one remaining problem, though. The moment we reboot our server, you’ll notice the firewall fails to initialise the ruleset saves by iptables-persisent because it is referencing an ipset that doesn’t exist. We’re going to need a way to create the ipset at boot time. I’ve done quite a bit of searching on how to do that properly but there appears to be little documentation available. In the end, I decided that creating a script to create and fill the ipset when one (or more) Ethernet interfaces come up would make sense. For that purpose, I created a script to run when initialising the networking system.

#!/bin/bash

# Script to set up ipset called blacklist
# to be populated by update-blacklist.sh

BLISTFILE="/etc/ip-blacklist.conf"
IPSET=/sbin/ipset

# Make sure no blacklist exists!
$IPSET flush blacklist && $IPSET destroy blacklist

# Recreate and populate blacklist
$IPSET create blacklist hash:net
egrep -v "^#|^$" $BLISTFILE | while IFS= read -r ip
do
        ipset add blacklist $ip
done

What this script does, is quite simple. It first makes sure there is no ipset called blacklist by emptying and destroying any ipset of that name. It then (re-)creates the ipset called blacklist and populates it, using the /etc/ip-blacklist.conf file we’ve created before. I then integrated that script in the networking initialisation process by adding it as a post-up script in /etc/network/interfaces as below.

# The primary network interface
auto eth0
iface eth0 inet static
address X.X.X.X
netmask 255.255.255.0
network X.X.X.A
broadcast X.X.X.Z
gateway X.X.X.Y
dns-nameservers X.X.X.Y 8.8.8.8 8.8.4.4
post-up /usr/local/bin/ipset.sh

As soon as the primary interface (in my case eth0) is up, the ipset is created and populated by running the /usr/local/bin/ipset.sh script. By the time iptables is initialised, the ipset is available and filled so malicious hosts are blocked nearly immediately. After doing this, our blacklist will survive a reboot, ensuring we always have its protection.

For those interested in taking look at the blacklist that is created, you may download it from here: https://www.beris.nl/downloads/blacklist/

If you have any corrections or tips pertaining to the above, I am all ears. If this post helps you in any way, I’d also like to hear about it.

Update: This article is update so that the script used to download the blacklists will work again. The wizcrafts.net page where the blacklists live, has had a change of code which caused the original script to fail. The statement to download and grep the lists into a single file now looks like this (change highlighted):

cat $i-iptables-blocklist.html | grep -v \< | grep -v \: | grep -v \; | grep -v \# | grep -v \" | grep [0-9] > $i.txt

Now you can get back to using blacklists with iptables in a proper manner.