Working with nmap-summariser & nmap-grepper

We all love nmap right? Once you get through that learning spike at the start that is. Once you are happily battering out “-sS -sV -A -Pn ..” you feel like you have made it as a hacker.  Not quite there yet? Practice kiddo.

A tool for nearly every challenge.

Finder of ports. Enumerator of versions. Confirmer of many vulnerabilities. Friend. Ally. For some of the hackers afraid of human contact I might even guess at… “lover”? But that is idle speculation.

The tl; dr version of this blog post is simply that there is a github repository here:

https://github.com/cornerpirate/nmap-summariser

Clone it and play with the python scripts if that is what you want to do. Thanks for coming!

Some of you are reading on? Ok then. I will mention both of the scripts and why they exist.

Create Some Nmap Results

Before proceeding I need to have some results to play with. Lets port scan google.com because they are the heroes we need and can take it.

Here is the command that was run:

nmap -sS -sV -Pn -oA nmap-google.com -v www.google.com
Nmap scan report for www.google.com (216.58.201.4)
Host is up (0.0080s latency).
Other addresses for www.google.com (not scanned): 2a00:1450:4009:805::2004
rDNS record for 216.58.201.4: lhr35s03-in-f4.1e100.net
Not shown: 998 filtered ports
PORT STATE SERVICE VERSION
80/tcp open http
443/tcp open ssl/https
2 services unrecognized despite returning data.

I beg the courts indulgence to explain the flags:

1. -sS – A TCP SYN Scan. A privileged command so you need to have root privileges.
2. -sV – Conduct a version scan. When nmap knows your service the “service” and “version” and”extra” parts of the output and XML files are populated with juicy details.
3. -oA – Output to all three common nmap formats.
   1. “.nmap” – as shown in the above example.
   2. “.xml” – an XML file that nmap-summariser and nmap-grepper use.
   3. “.gnmap” – greppable nmap format. Comma separated variables.
4. -v – Be a little verbose. Just a little.

The core bit here is that you must export the XML file format to use the tools I am talking about later. The optional part is using “-sV” to get more information to play with.

Nmap-Summariser

I made this to meet a need I had when writing a report. You pass it an nmap “.xml” file and it will spit out the data. Primarily I published this code so that someone had a simple baseline Python script that they could manipulate the output of easily to match whatever they needed. Funnily enough, nmap-grepper exists because it turned out I needed it!

This is not really meant to be used by people, but it had the hooks in the right places of the XML file so you could simply alter what you printed out to match your needs.

How does it work? Hackers are often curious so lets peek under the hood:

This is the key part of the script. The code above line 48 gets a reference to the XML file as an object and allows us to pick out specific XML nodes to get data from. To understand the for loop we need to know a little bit about the XML file format.

Here is a high level view of the XML structure:

<nmaprun>
 <scaninfo> Information related to how the scan was configured </scaninfo>
 <host> all data about host A </host>
 <host> all data about host B </host>
 <host> all data about host C </host>
 <runstats> Information about how long it took to run etc. </runstats>
</nmaprun>

The key part to note is that there is one or more “<host>” tags. Each IP address in your scan will be represented by a “<host>” tag. I have showed three to get the point across. Of-course our XML file from scanning google.com will have only one “<host>” tag.

Lets look at that for loop again. As I think of myself as a nice guy here it is below:

You can probably tell that this for loop is now essentially going through every “<host>” tag. Lines 49 and 50 are then getting access to a child node and then an attribute of that child node respectively. This is shown below:

I haven’t made my script amazingly robust. You might find errors at line 49 if it somehow does not have an entry at position [0]. It may also behave unexpectedly on an IPv6 network at line 50. Who knows?

For more information there is no substitute for opening your “nmap-google.com.xml” file in a text editor, and viewing the nmap-summariser.py script to see which tags/attributes it picks out:

https://github.com/cornerpirate/nmap-summariser/blob/master/nmapsummariser.py

For bonus points. Modify your script to display output in this format:

<IP>:<HOSTNAME>:<OPERATING_SYSTEM>

You will need to find a target, and re-run nmap with “-O” to force an operating system guess.

You will have to heavily modify the output lines to achieve this, and my script doesn’t even vaguely touch the OS tags so you have to find them in the XML. Happy hunting!

Nmap-Grepper

I went onsite and had a need to grep ports very quickly since there was a lot of data to sift through. So I modified “nmapsummariser.py” and ended up with “nmapgrepper.py”:

https://github.com/cornerpirate/nmap-summariser/blob/master/nmapgrepper.py

Why does this exist when nmap outputs a “.gnmap” which is greppable?  Well, because that isn’t very greppable! Is the answer. The “.gnmap” format uses one line per host which can include any number of services. It gets very messy to work with very quickly.

So what does “nmapgrepper.py” do ? It simply flips it so you get a port/service view instead of a host view. Like this:

With one host that only has two services we do not have much to work with here. Imagine this blown up to an internal network which has thousands of internal services. You can use “grep” and “cut” very quickly to interact with this file format. Want to find all “http”services ?

A simple grep will give you all services that operate on plain-text HTTP.

Want to just store the IP addresses that have “http” ? Go for a cut mate, like this:

There you go you can now  slice and dice an nmap scan to focus on particular services quickly. May no FTP service permit anonymous logins without you knowing about it.

When tackling a massive network I try to take a service approach first because a few common flaws will “get you in” most times. If that isn’t working flipping your attention to individual hosts that have some interesting services you can enumerate will often yield results.

If you want another coding challenge here then why not make “nmapgrepper.py” spit out this format instead:

<IP>:<HOSTNAME>:<OPERATING_SYSTEM>:<PORT>:<PROTOCOL>:<SERVICE>:<VERSION>:<EXTRA>

That means more opportunities for greppability. Find me all things running Microsoft Server 2008? Show me any host with “dev” or “test” in the hostname? These are things you will be able to answer if you do.

Note: you can achieve the same effect with metasploit if you have enabled the Postgres database. Your reading for that is 100% this URL:

https://www.offensive-security.com/metasploit-unleashed/using-databases/

I don’t claim nmapgrepper is better. It is just sometimes fun to roll your own to know more.

Hope that helps