DNSBL: Not just for spam
Security practitioner Menin_TheMiddle is using DNS to stop botnet, spammers and anonymous traffic with Nginx, Lua and DNSBL. Find out how.
DNSBL is a Domain Name System-based Blackhole List, also known as "RBL" Real-time Blackhole List. It's nothing more than a black (or white) list distributed via DNS protocol and it's widely used by MTA, mail server, antispam, etc... in order to block spammers.
The whole thing is based on the idea that you resolve a hostname related to an IP address of a client that is connecting to your SMTP server, and you get a response with an IP on the 127.0.0.1/8 subnet that tells you a lot of information about the reputation of the connecting client. Usually, the hostname is something like [user's IP in reverse-octet format].dnsbl.example.com and the response is something like 127.28.50.4 where the first octect is always 127 and it means that the user's IP Address was found in the DNS zone. The second octect could represents the number of days since last activity (28 days in our example). The third octect could represents a "threat score" for that IP and the last octect could represents the "type of visitor" (for example 4 = Comment Spammer).
It's very easy to understand how it works, let's do an example:
- A client with IP
1.2.3.4connects to my whatever server - My server performs a DNS query to
4.3.2.1.dnsbl.example.com - I get a response from
dnsbl.example.comlike127.7.50.4 - Now I know that the user has a "Bad Reputation" because is a (
4) Comment Spammer with a threat score of50/255and it was blacklisted a week ago. - Drop connection: you shall not pass.
This offers a huge amount of benefits: DNS query is fast, easily cacheable, a Zone DNS is easily upgradable, and you can add more than one DNSBL service or even create your own! (yes, keep reading... we'll see how to create one). You can easily integrate a DNS based blacklist on your daemon because basically anything can do a DNS query.
Project Honey Pot has taken this idea and has created a DNSBL not only for fighting spam but also to protect HTTP world and WebApp. It's named http:BL (https://www.projecthoneypot.org/httpbl_api.php).
For many years email recipients have benefited from the use of various DNSBLs in the fight against spam. Through efficient DNS lookups, mail servers are able to check individual connecting clients against various black lists. This provides mail servers with the ability to decide to how client requests are handled from hosts based on individual black list criteria. Hosts are able to decide to block requests, allow requests, or perform extra spam filtering scrutiny to messages from hosts based on results from black lists lookups.
Http:BL is similar, but is designed for web traffic rather than mail traffic. The data provided through the service allows website administrators to choose what traffic is allowed onto their sites. This document describes how to integrate with and take advantage of the http:BL service.
This is awesome, we can get all the advantages of having a wide honeypot network and using it in order to have a real-time Bad Reputation database via DNS!
Now, how can we integrate this into our web application/website? There's a lot of projects that you can find here: projecthoneypot.org and if you use ModSecurity and Nginx there's an interesting tutorial on the nginx blog
Http:BL is free, you just need to sign up with Project Honey Pot to receive an API key.
Using ModSecurity you have the @rbl operator that you can use in order to check users IP over a DNSBL service (something like SecRule "@rbl dnsbl.example.com" ...). Unfortunately, there're two main problems (IMHO): The first is that there isn't control on the DNS response (check my issue here: 1845) and on Nginx, there is some performance issue on implementing this with ModSecurity because the worker process is blocked on the socket read and this will ruin performance (thanks p0pr0ck5 for the hint).
So, I decided to implement it from scratch using the Nginx Lua Module (or OpenResty). Let's see how:
Nginx, Lua and Project Honey Pot
First, we need to know in which nginx variable is the client's IP address (usually is $remote_addr but, as in my case, if you are using a Load Balancer you could have it on the X-Forwarded-For header or if you're using CloudFlare you can find it on CF-Connecting-IP header, etc..). Let's assume that is in the X-Forwarded-For because our application is behind a DigitalOcean Load Balancer.
Now we need something that could make DNS query from the Nginx Lua Module. There're many projects on github but I want to use the resty.dns.resolver that is included on OpenResty and you can find here: https://github.com/openresty/lua-resty-dns. The first problem is that you can't use it in code contexts like set_by_lua*, log_by_lua*, and header_filter_by_lua* where the ngx_lua cosocket API is not available. So, we'll use it in the rewrite_by_lua* context. Following a test:
location / {
rewrite_by_lua_block {
local resolver = require "resty.dns.resolver"
local r, err = resolver:new {
nameservers = {"8.8.8.8"},
retrans = 1,
timeout = 2000,
}
local answers, err, tries = r:query("www.google.com", nil, {})
for ak,ans in ipairs(answers) do
ngx.say(" -> " .. ans.address)
end
}
...
}
This simple test shows you the r:query function that queries 8.8.8.8 in order to get the www.google.com IP address (type A). Now we need to "create" the hostname to resolve based on the client's IP. As I said before, we need to resolve something like [access-key].[client-ip-reverse-octet-format].dnsbl.httpbl.org so:
...
-- set the client IP from x-forwarded-for
local clientip = ngx.var.http_x_forwarded_for
-- set the Project Honey Pot access-key
local accesskey = "abcdefghijkl"
-- split IP address assign each octect to a variable
a,b,c,d = clientip:match("([%d]+).([%d]+).([%d]+).([%d]+)")
-- create the hostname string
local dnsblhost = accesskey..d.."."..c.."."..b.."."..a..".".."dnsbl.httpbl.org"
-- make the query
local answers, err, tries = r:query(dnsblhost, nil, {})
...
and we need to do the same thing in order to parse the DNS response:
if answers ~= nil then
for ak,ans in ipairs(answers) do
if ans.address ~= nil then
-- assign each octect to a variable
e,f,g,h = ans.address:match("([%d]+).([%d]+).([%d]+).([%d]+)")
-- if the first octect is 127 = OK!
if e == '127' then
-- if the last octect != 0 (search engine)
if h > 0 then
-- Block request
ngx.say("Your request has been blocked")
ngx.exit(ngx.HTTP_FORBIDDEN)
end
end
end
end
end
Don't worry, I'll attach the whole nginx.conf with all code and configuration you need. As you can see, in the example above, if there's a DNS response and if the first octect is equal to 127 and the last octect is greater than 0 (0 means "Search Engine" for Project Honey Pot http:BL) then send a 403 Forbidden to the user (ngx.exit(ngx.HTTP_FORBIDDEN)).
Cool! isn't it? Now, every time a user visit our website, Nginx will check the http:BL looking for a reputation of the user's IP. I know, now you're thinking something like: "but if my nginx makes a DNS query for each request, this won't affects on performances?" ehmm yes... it does, surely it degrades a little the nginx performances. But don't worry, we can easily create a cache mechanism using a little thing called ngx.shared of the lua-nginx-module (https://github.com/openresty/lua-nginx-module#ngxshareddict): "Fetching the shm-based Lua dictionary object for the shared memory zone named DICT defined by the lua_shared_dict directive. Shared memory zones are always shared by all the nginx worker processes in the current nginx server instance."
In order to create it, first we need to define a shared memory zone on the nginx config file like this:
http {
...
lua_shared_dict dnsblcache 10m;
...
server {
...
}
}
I've called it dnsblcache with a size of 10 Megabyte. Now each time we check the http:BL DNS we'll put the DNS response inside that shared memory zone and we'll check that zone for the subsequent requests instead making a new DNS query for the same IP Address. I've used only two functions: first the safe_set(clientip, ngx.var.dnsblres, 86400) that save the DNS response for $clientip and keep it valid for 86400 seconds. Second I use the get(clientip) function in order to check if the user's IP has been already checked. I've done it with something like this:
rewrite_by_lua_block {
...
-- check shared memory before making a new DNS query
local dnsblccval, dnsblccflag = ngx.shared.dnsblcache:get(clientip)
if dnsblccval ~= nil then
ngx.say("Your request has been blocked")
ngx.exit(ngx.HTTP_FORBIDDEN)
end
...
-- check Project Honey Pot http:BL
...
-- save the results in my dnsblcache shared memory zone
ngx.shared.dnsblcache:safe_set(clientip, ngx.var.dnsblres, 86400)
...
If you want, you can use my configuration file (I've published it on github gist):
As you can see, all DNSBL services are added to the dnsblserv variable (so you can add as many as you want, I've added httpbl.org and dnsbl.sorbs.net). You can include it on your nginx location block like this:
http {
lua_shared_dict dnsblcache 10m;
...
server {
...
location / {
include DNSBL.conf;
}
}
}
The more observant among you may have noticed that I collect all DNS response on $dnsblres nginx variable and then I send to client a custom response header x-dnsbl containing all DNSBL response (if any). Why putting the DNS query results in a response header? Simply because of the logging through ModSecurity! I need to integrate this blacklist to my WAF and treat it such as a rule that matches and write a log. Let's see how to do it.
DNSBL and ModSecurity
Since my Lua script send a response header when it found an IP with a Bad Reputation, I can write some rules in order to intercept that header and write a log with a bunch of useful information about the visitor.
By simulating a visit from an IP listed in the Project Honey Pot Blacklist, I've got the following result:
curl -v -H 'X-Forwarded-For: 171.25.193.77' 'http://wordpress/test123'
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to wordpress (127.0.0.1) port 80 (#0)
> GET /test123 HTTP/1.1
> Host: wordpress
> User-Agent: curl/7.54.0
> Accept: */*
> X-Forwarded-For: 171.25.193.77
>
< HTTP/1.1 403 Forbidden
< Server: openresty/1.13.6.2
< Date: Tue, 07 Aug 2018 15:31:29 GMT
< Content-Type: text/html
< Transfer-Encoding: chunked
< Connection: keep-alive
< x-dnsbl: projecthoneypot=127.76.63.5
As you can see, the x-dnsbl header says projecthoneypot=127.76.63.5 and it means that Project Honey Pot has categorized that IP as Suspicious & Comment Spammer (the last octect 1+4), with a threat score of 63/255 (third octect) and it was seen 76 days ago (second octect).
Having all these information I can write a couple of rules on ModSecurity in order to log when I block a request from the Lua script. The following 7 rules check the RESPONSE_HEADERS:X-DNSBL variable for each Project Honey Pot category and write a log that includes thread score and last seen:
#
# Name: projecthoneypot
# URL: https://www.projecthoneypot.org/httpbl_api.php
#
SecRule RESPONSE_HEADERS:X-DNSBL "@rx projecthoneypot\=127\.([0-9]+)\.([0-9]+)\.1" "id:32000201,\
phase:3,\
pass,\
log,\
capture,\
severity:'6',\
maturity:'9',\
accuracy:'9',\
tag:'dnsbl',\
tag:'projecthoneypot',\
tag:'suspicious',\
tag:'blacklist',\
logdata:'Blacklist: Project Honey Pot - [last_seen: %{tx.1} days] [threat_score: %{tx.2}] - DNS Response: %{RESPONSE_HEADERS:X-DNSBL}',\
msg:'DNSBL: Suspicious'"
SecRule RESPONSE_HEADERS:X-DNSBL "@rx projecthoneypot\=127\.([0-9]+)\.([0-9]+)\.2" "id:32000202,\
phase:3,\
pass,\
log,\
capture,\
severity:'6',\
maturity:'9',\
accuracy:'9',\
tag:'dnsbl',\
tag:'projecthoneypot',\
tag:'harvester',\
tag:'blacklist',\
logdata:'Blacklist: Project Honey Pot - [last_seen: %{tx.1} days] [threat_score: %{tx.2}] - DNS Response: %{RESPONSE_HEADERS:X-DNSBL}',\
msg:'DNSBL: Harvester'"
SecRule RESPONSE_HEADERS:X-DNSBL "@rx projecthoneypot\=127\.([0-9]+)\.([0-9]+)\.3" "id:32000203,\
phase:3,\
pass,\
log,\
capture,\
severity:'6',\
maturity:'9',\
accuracy:'9',\
tag:'dnsbl',\
tag:'projecthoneypot',\
tag:'suspicious',\
tag:'harvester',\
tag:'blacklist',\
logdata:'Blacklist: Project Honey Pot - [last_seen: %{tx.1} days] [threat_score: %{tx.2}] - DNS Response: %{RESPONSE_HEADERS:X-DNSBL}',\
msg:'DNSBL: Suspicious / Harvester'"
SecRule RESPONSE_HEADERS:X-DNSBL "@rx projecthoneypot\=127\.([0-9]+)\.([0-9]+)\.4" "id:32000204,\
phase:3,\
pass,\
log,\
capture,\
severity:'6',\
maturity:'9',\
accuracy:'9',\
tag:'dnsbl',\
tag:'projecthoneypot',\
tag:'comment-spammer',\
tag:'blacklist',\
logdata:'Blacklist: Project Honey Pot - [last_seen: %{tx.1} days] [threat_score: %{tx.2}] - DNS Response: %{RESPONSE_HEADERS:X-DNSBL}',\
msg:'DNSBL: Comment Spammer'"
SecRule RESPONSE_HEADERS:X-DNSBL "@rx projecthoneypot\=127\.([0-9]+)\.([0-9]+)\.5" "id:32000205,\
phase:3,\
pass,\
log,\
capture,\
severity:'6',\
maturity:'9',\
accuracy:'9',\
tag:'dnsbl',\
tag:'projecthoneypot',\
tag:'suspicious',\
tag:'comment-spammer',\
tag:'blacklist',\
logdata:'Blacklist: Project Honey Pot - [last_seen: %{tx.1} days] [threat_score: %{tx.2}] - DNS Response: %{RESPONSE_HEADERS:X-DNSBL}',\
msg:'DNSBL: Suspicious / Comment Spammer'"
SecRule RESPONSE_HEADERS:X-DNSBL "@rx projecthoneypot\=127\.([0-9]+)\.([0-9]+)\.6" "id:32000206,\
phase:3,\
pass,\
log,\
capture,\
severity:'6',\
maturity:'9',\
accuracy:'9',\
tag:'dnsbl',\
tag:'projecthoneypot',\
tag:'harvester',\
tag:'comment-spammer',\
tag:'blacklist',\
logdata:'Blacklist: Project Honey Pot - [last_seen: %{tx.1} days] [threat_score: %{tx.2}] - DNS Response: %{RESPONSE_HEADERS:X-DNSBL}',\
msg:'DNSBL: Harvester / Comment Spammer'"
SecRule RESPONSE_HEADERS:X-DNSBL "@rx projecthoneypot\=127\.([0-9]+)\.([0-9]+)\.7" "id:32000207,\
phase:3,\
pass,\
log,\
capture,\
severity:'6',\
maturity:'9',\
accuracy:'9',\
tag:'dnsbl',\
tag:'projecthoneypot',\
tag:'suspicious',\
tag:'harvester',\
tag:'comment-spammer',\
tag:'blacklist',\
logdata:'Blacklist: Project Honey Pot - [last_seen: %{tx.1} days] [threat_score: %{tx.2}] - DNS Response: %{RESPONSE_HEADERS:X-DNSBL}',\
msg:'DNSBL: Suspicious / Harvester / Comment Spammer'"
And that is the result on my WAF console:
Using DNSBL to block Tor Exit Nodes
The idea is to create a DNS zone on a domain with all IP Addresses listed on https://check.torproject.org/exit-addresses and use it as shown before with Project Honey Pot. Of course, we can use a bind DNS server or the DNS API of our provider (such as DigitalOcean or CloudFlare) but I want to KIS and I'll use FakeDNS (https://github.com/Crypt0s/FakeDns), a python regular-expression based DNS server!
First clone FakeDNS from github:
root@themiddle ~/$ git clone https://github.com/Crypt0s/FakeDns.git
root@themiddle ~/$ cd FakeDns
Then, create the Tor exit nodes list from torproject.org (I'll use egrep for capture just IP Addresses and then awk to create the hostname with reverse-octect IP format) and convert it to a FakeDNS configuration format (something like <type> <hostname> <ip address>):
root@themiddle ~/$ curl -s 'https://check.torproject.org/exit-addresses' | \
egrep -o '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' | \
awk 'BEGIN{FS="."}{print "A "$4"."$3"."$2"."$1".dnsbl.example.com 127.7.10.1"}'
A 117.34.236.89.dnsbl.example.com 127.7.10.1
A 82.124.27.103.dnsbl.example.com 127.7.10.1
A 21.160.248.185.dnsbl.example.com 127.7.10.1
A 27.101.220.185.dnsbl.example.com 127.7.10.1
A 4.102.220.185.dnsbl.example.com 127.7.10.1
A 202.227.250.77.dnsbl.example.com 127.7.10.1
A 102.186.64.45.dnsbl.example.com 127.7.10.1
A 246.16.106.151.dnsbl.example.com 127.7.10.1
A 15.19.182.46.dnsbl.example.com 127.7.10.1
A 11.123.200.185.dnsbl.example.com 127.7.10.1
A 57.38.36.46.dnsbl.example.com 127.7.10.1
A 235.224.42.66.dnsbl.example.com 127.7.10.1
A 253.57.249.173.dnsbl.example.com 127.7.10.1
A 108.224.123.195.dnsbl.example.com 127.7.10.1
A 225.221.137.64.dnsbl.example.com 127.7.10.1
A 45.101.220.185.dnsbl.example.com 127.7.10.1
A 146.77.2.5.dnsbl.example.com 127.7.10.1
...
Ok, now I save the output to tor.dns.conf file and run fakedns:
root@themiddle ~/FakeDns$ python fakedns.py -c ./tor.dns.conf --noforward
>> Parsed 2801 rules from ./tor.dns.conf
Now I'm able to perform DNS query to my fakedns server. I just need to delegate a third level domain to my fakedns IP Address. For example, using CloudFlare, you could just click on DNS button and create a record A and a record NS like this:
Well, it's almost done! We've a DNS server, we've created a DNS zone from the Tor exit nodes list and we've delegated a 3rd level domain to FakeDNS. Now we can add our DNSBL service to the Lua script and create a ModSecurity Rule in order to log all blocked request from Tor:
Add dnsbl.example.com to the Lua script:
...
local dnsblserv = {
["tor"]={
["accesskey"]="",
["host"]="dnsbl.example.com"
},
["projecthoneypot"]={
["accesskey"]="kmneblytpjwm",
["host"]="dnsbl.httpbl.org"
},
["sorbs"]={
["accesskey"]="",
["host"]="dnsbl.sorbs.net"
}
}
...
And add a ModSecurity Rule for logging:
SecRule RESPONSE_HEADERS:X-DNSBL "@rx tor\=127\.([0-9]+)\.([0-9]+)\.1" "id:32000302,\
phase:3,\
pass,\
log,\
capture,\
severity:'6',\
maturity:'9',\
accuracy:'9',\
tag:'dnsbl',\
tag:'secthemall',\
tag:'tor-exit-node',\
tag:'blacklist',\
logdata:'Blacklist DNS Response: %{RESPONSE_HEADERS:X-DNSBL}',\
msg:'DNSBL: Tor Exit Node'"
After few days of test it seems that it works like a charm! This is a screenshot of my WAF web console:
Cool! isn't it? 😎
Conclusion
IMHO using DNS for blacklist has many advantages, it makes me able to distribute and keep update all blacklist in all my WAF nodes distributed around the world and on different providers. Probably the type A is not the best choice for doing it. I think that a TXT record should be better for a DNSBL used on a webapp, it would be nice to store more information about the blacklisted IP address, I mean something like this:
root@themiddle ~/$ host -t TXT 4.3.2.1.dnsbl.example.com
4.3.2.1.dnsbl.example.com descriptive text "{'block':1, 'category':'bruteforce', 'requests':1234, 'path':'/wp-login.php'}"
At the moment, if you query the http:bl using TXT type (instead of A) it points to the Project Honey Pot listing page where additional information is available.
root@themiddle ~/$ host -t TXT abcdefghijkl.77.193.25.171.dnsbl.httpbl.org
abcdefghijkl.77.193.25.171.dnsbl.httpbl.org descriptive text "See: http://www.projecthoneypot.org/ipr_77.193.25.171"
The most important and awesome thing about Project Honey Pot is that users can participate by installing honeypots on their website. It means that the more people that participate in the project the better it is for everyone.
To participate in Project Honey Pot, webmasters need only install the Project Honey Pot software somewhere on their website. We handle the rest — automatically distributing addresses and receiving the mail they generate. As a result, we anticipate installing Project Honey Pot should not increase the traffic or load to your website.
You can find more information here.
Special thanks
- All OWASP CRS team
- Eric Langheinrich (Project Honey Pot)
- Bryan "Crypt0s" Halfpap (FakeDNS)
If you liked this post...
Twitter: @AndreaTheMiddle
GitHub: theMiddleBlue
LinkedIn: Andrea Menin