Faster nmap scanning with the help of GNU parallel

When you give access to developers to create firewall rules, they generally open all sorts of ports to the internet. All they need to do / expected to do is get the product working. The rest doesn’t matter.

This was the same with the company I worked with. There were tonnes of firewall rules in the GCP projects which opened many ports to the public internet. One could think of deleting firewall rules straight away and check if something is breaking on production. But that’s a bad solution.

Security measures, in my opinion, should not be at the cost of production downtime (*unless there is a very critical bug to patch on prod).

So the next idea was to scan all the public IPs of VMs and K8s nodes and get a list of closed ports. This will help us get a list of firewall rules that are no longer required.

Got a CSV with GCP project name and IP address assigned with the project. The CSV file looks something like the below (headers stripped off):

1
2
3

projectOne,xxx.xxx.xxx.xxx
projectOne,xxy.xxx.xxx.xxx
projectTwo,yyy.yyy.xyx.xxx

After looking at the time it took to complete nmap scan on one single IP for all ports, I was sure that running scan for 1000+ IPs would take a very long time.

GNU Parallel for the win !

GNU Parallel is a command line tool that allows to execute multiple jobs in parallel. A job can be a single command or a small script that has to be run for each of the lines in the input.

For my case, I want to run nmap scan in parallel.

1
2
3
4
5
6

mkdir logs
cat gcp-project-ips.csv | cut -f1 -d, | sort -u | while read project; do mkdir logs/${project}; done

cat gcp-project-ips.csv | while IFS="," read -r project ip; do echo nmap -sT -T5 -Pn -p- -oG logs/${project}/${ip}.gnmap $ip; done > scan-all-ips.out

parallel --jobs 32 < scan-all-ips.out 

The above code starts 32 parallel nmap scans. You can tune the number of jobs as per your network bandwidth and the CPU limit. The greppable output of the nmap scan is put into logs/{project}/{ip}.gnmap files once the scan is done.

The other part of the job was to grep the closed ports from the nmap output files and report it in a parsable format.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

for dirname in logs/*; 
do 
    for filename in ${dirname}/*.gnmap; 
    do 
        project=`echo $filename | cut -d/ -f2`;
        ip=`echo $filename | cut -d/ -f3`;
        ip=${ip::-6};
        cat $filename | cut -d' ' -f2,4- | sed -n -e 's/Ignored.*//p' | awk -v ip="$ip" -v project="$project" '{$1=""; for(i=2; i<=NF; i++) { a=a" "$i; }; split(a,s,","); for(e in s) { split(s[e],v,"/"); if (v[2] == "closed") { printf project "," ip ",%s,%s\n" , v[1], v[5]}}; a="" }'; 
    done; 
done