Fast Reverse DNS Lookups using FDNS and MongoDB

As part of my research, I wanted a way to find all the DNS records which points to a particular IP address. Not only should it be fast, it should be cheap as well. If you are a DNS researcher you would know about the Rapid7’s free FDNS dataset.

I was not able to find any online post that showed me how to get subdomains using IP address in the FDNS dataset. There were plenty of articles that said how to find all the subdomains using the domain name.

There are some cloud solutions like AWS Athena and GCP BigQuery that makes the life easier when dealing with large datasets. The life seems easy till the monthly bill of the cloud service arrives in your email inbox.

Finally I wanted to give databases a shot and selected MongoDB for its simplicity when loading JSON documents to the DB.

Loading FDNS dataset to MongoDB

I setup a mongo DB instance on my local machine. Then I tried to import the uncompressed JSON data to a database called fdns and collection named dns.

1

pigz -dc fdns.json.gz | mongoimport --db fdns --collection dns

After the data is loaded into database, one could go ahead and search for all domains pointing to an IP using the following command.

1
2

use fdns
db.dns.find({"value": "127.0.0.1"})

The problem with this is it searches all the rows to check if the IP is 127.0.0.1. If we have 100k JSON documents (rows) in the collection, it searches for all the 100k JSON documents and returns us the documents whose value matches 127.0.0.1. This takes a lot of time. If you want to do it just once, this is the most simplest method.

Creating MongoDB Indexes

If you want to reverse lookup multiple times then you will definitely be frustrated with the time it takes to complete one search. To save everyone from this frustration, MongoDB has something called Indexes. Creating index will reduce the search time to seconds.

As we search the value field (column in MongoDB terms) and try to get the subdomain, to make our search faster we need to create the index based on the value field.

1
2

use fdns
db.dns.createIndex( { value : 1 } )

This creation of index takes long time but once it is done, the search just takes seconds to complete.