Monthly Archives: December 2016

Using CVE data, with CVE-Offline

Tl; dr version – I made a thing that makes the bulk of CVE vulnerability details easy to grep for use when reporting. Get it here: I update it once a month from the NIST database.

Anyone sticking around to read the rest gets a process for checking for known weaknesses in a service. Rationales for why, as well as a bit about how to use CVE-Offline.

Rationale and Process for vulnerability identification

When writing penetration test reports you will need to interact with many online resources to do that well. The simplest thing that you should do is include a summary of any known weaknesses within an installed version of software.

The biggest database of shared vulnerability knowledge is the CVE database. An overview of that is available at the URL below:

Almost every vulnerability in common services will result in a vendor patch AND an entry with a unique CVE identifier. When you are targeting a customer you need to compile a list of known flaws. This is so you can advise them of the exact technical risk posed by using version X of Apache on the day the test was conducted (as an example, not to pick on Apache).

To do this, you can follow the process outlined below:

  1. Identify a service version – “nmap -sV -p <portnumber> <ip>” will to this in most cases.
  2. Look for known vulnerabilities in version – If you are using Nessus, or an alternative vulnerability scanner, they do a pretty decent job of maintaining their database of CVE issues. “IF $version == x.x.x THEN vulnerable to x,y,z” is the logic they use.
    1. If the service is unknown to your vulnerability scanner you will have to find your own list of CVEs if possible. Look up the vendor and product on
    2. They have an export facility of sorts so you can use the output for your product to achieve what CVE-Offline does when you have an Internet connection.
    3. Alternatively, you are going to have to find the release notes, bug tracker, or change log for your target service. They usually track security defects using CVE references. This is not universal though.
  3. Format your list for your report – now that you have a list you are going to want to present them to your customer.
    1. If your list of vulnerabilities is around 20 you can probably show a summary of each issue to the customer.
    2. If the list is ENORMOUS, then are you adding much value in creating 20 pages in your report? Probably not.
    3. In both scenarios you can present the most significant risks with fuller write ups. Go out looking for exploit code to help quantify the exploitability of the service, and present a statistical summary.
    4. For example, “The service had 200 known weaknesses within the CVE database. Of those X were in the high risk range, Y were in the the medium risk range, and Z were in the low risk range. The consultant would advise special attention is paid to CVE-….-…. and CVE-….-…. which allow remote code execution, and denial of service respectively.”

Following that process should get what you want in your reports.

A word on false positives

The above process is pretty simple. There is one massive caveat that you will need to be aware of. That process is, more often than not, based on the service banner returned by your target.

If the OS uses “backporting” to supply security updates, or the admin has mucked about, then the banner number will not necessarily reflect the target’s exploitability. In this case we can get into “false positives”. A quick Google gives us the following definition:


Definition of a false positive – Circa Christmas 2016

If the target is reporting a service banner which is inaccurate when clashed against the vendor’s official release history. Then your results will be inaccurate and you have a false positive.

If we are testing in a black-box scenario we have no access to the underlying operating system by default and cannot fully confirm the banner status in many cases.

What to do? Ensure that your report includes phrases like these:

  1. Vulnerability based on service banner only.
  2. Potential false positive result.
  3. Impact and risk based on assumption that banner was accurate as this is the worst case scenario.

The first line of your recommendations section should definitely be “Conduct an investigation to ensure that the service is vulnerable”. Be sure that you convey these sentiments on all phone calls with your customer and they should learn to trust the other results more as they do not have them!

With that caveat dealt with by the language in your report you can be sure to prevent awkward phone calls when customers call up going “we looked into it, and it was NOT vulnerable.”

Experience tells me that even if you include these caveats you have to justify yourself sometimes. Be aware of it coming by preparing some polite reasons to give.

Using CVE-Offline

It is pretty simple to obtain CVE-Offline. Just clone the git down:

git clone

You can then use your platforms “grep” of choice to find a CVE in the “cve-summary.csv” file. For example, lets look for “CVE-2016-0142” do the following:

grep "CVE-2016-0142" cve-summary.csv
CVE-2016-0142,9.3,"Video Control in Microsoft Windows Vista SP2, Windows 7 SP1, Windows 8.1, Windows RT 8.1, and Windows 10 Gold, 1511, and 1607 allows remote attackers to execute arbitrary code via a crafted web page, aka ""Microsoft Video Control Remote Code Execution Vulnerability."""

This has given you a comma separated line of output that you can now work with. The format of this line is:


You can pipe that bad boy into a .csv file, open it in excel and make pretty tables that you can paste into your report. You are welcome.

Updating CVE-Offline

I update the repository once a month from the nist export feeds. You can get the raw data from here if you want:

Otherwise you basically need to use git to update once a month. Enter your local “cve-offline” directory and do this:

git pull

Personally I have often forgotten how the heck to update a repository I use. So there you are, it is written down!

Simple HTTP/HTTPS Servers in Python

There are loads of situations where you need a quick way to spin up an HTTP server. The following are just off the top of my head but I bet you can all think of more:

  • You have established a shell and you want to upload tools to your new computer.
  • You want to exfil data back out of an environment.
  • You have found an XSS and you want to prove you can get useful data out of a victim’s session easily.
  • You want to host a JavaScript file to prove the dangers of poor Transport Layer Security or sites which host files on 3rd party hosts.

In these situations configuring Apache is overkill and you probably only want a simple service which will last a matter of minutes. I am going full Python for this one and I have reworked this post to include Python3 syntax to keep it current in 2018.

Don’t Run with Scissors

The HTTP and HTTPS listeners that I show in this post will share the contents of the folder where you run them. As with all situations “BE AWARE OF WHAT IS IN YOUR WEB ROOT“. If you are starting a listener on the Internet then it will be port scanned within 30 minutes, and you will see automated attackers spamming your logs.

Before you run any of these services take a moment to consider where you are running them. I usually create a folder in “/tmp/” for an engagement such as “/tmp/safeplace” since this will not persist for very long.

Additionally, these servers have Directory Listings enabled by default. If you want to limit what can be seen it is a good idea to create an “index.html” file in your folder. That way your assailant will be forced to use Dirbuster which gets noisy.

Simple HTTP Server (Python 2)

For most situations this is my goto move:

python -m SimpleHTTPServer <port> # Syntax
python -m SimpleHTTPServer 8080   # Example

When you run this it starts to log things to the console as shown:


By default you get the IP address of the client. Then the files and response codes.

Simple HTTP Server (Python 3)

Python 3 is the future!!! They have renamed the module to “http.server”. Time to update your tools to do this:

python3 -m http.server <port> # Syntax
python3 -m http.server 8080   # Example

Which does very similar things as shown below:


Not a lot different really.

Generating Keys and Certificates for HTTPS

Sometimes you need to use HTTPS. The two best cases are:

  1. To Gain Privacy – if your target has traffic inspection but they do not terminate SSL then you can genuinely smuggle shells through SSL. I have seen a customer network which would catch plain-text but did nothing about an HTTPS connection.
  2. To Avoid Browser Mixed Content Warnings – these warnings can hinder a decent proof of concept. If you are trying something that “should be working goddamn it!”. Consider looking at the developer tools for your web browser and look for mixed content exceptions.

Whatever your motivation the HTTPS services I show will rely on you having a “privatekey.pem” and “cert.pem” file. You can generate these with openssl using the commands below:

openssl genrsa > privatekey.pem
openssl req -new -x509 -key privatekey.pem -out cert.pem -days 365 

You will be prompted to enter information to populate the fields of your self-signed SSL certificate. I stuff those with spam for proof of concepts. If I was engaged in a phishing exercise then a legit SSL certificate is the right move. 

Simple HTTPS Server (Python 2)

I really liked “twisted” because it can do a *lot* of things. First you will need to install that using python’s pip package manager as shown:

pip install twisted

You can then start an HTTPS service using the command below:

twistd web --https=<port> --path=. -c </path/to/cert.pem> -k </path/to/privatekey.pem> #Syntax
twistd web --https=8443 --path=. -c cert.pem -k privatekey.pem                         #Example

Twisted automatically creates a logfile called “twisted.log” in the current directory it does not want to echo to stdout so you need to use:

tail -f twisted.log

The “-f” means read this file when it changes. The following shows what this means:


The content of the output includes User-Agents and more debugging related to SSL but is otherwise similar to “SimpleHTTPServer”.

Simple HTTPS Server (Python 3)

Python 3 is the future!!! Python 3 is the future!!! Python 3 is the future!!!

Again there are minor differences between python 2 and 3. This time you have to use “pip3” to install Twisted:

pip3 install twisted

If you already have “twisted” installed for python 2 this can get a bit confusing. Running the above seems to clobber the version in the path. When you run “which twistd” you will be pointed at the python3 version. Don’t trust me? Check the first line of that file:


Great. So is the syntax to run the server different? Fortunately not in my test the following worked just fine once more:

twistd web --https=<port> --path=. -c </path/to/cert.pem> -k </path/to/privatekey.pem> #Syntax
twistd web --https=8443 --path=. -c cert.pem -k privatekey.pem                         #Example

Hope that saves you some time when you want a simple HTTPS server!

Take care


Working with nmap-summariser & nmap-grepper

We all love nmap right? Once you get through that learning spike at the start that is. Once you are happily battering out “-sS -sV -A -Pn ..” you feel like you have made it as a hacker.  Not quite there yet? Practice kiddo.

A tool for nearly every challenge.

Finder of ports. Enumerator of versions. Confirmer of many vulnerabilities. Friend. Ally. For some of the hackers afraid of human contact I might even guess at… “lover”? But that is idle speculation.

The tl; dr version of this blog post is simply that there is a github repository here:

Clone it and play with the python scripts if that is what you want to do. Thanks for coming!

Some of you are reading on? Ok then. I will mention both of the scripts and why they exist.

Create Some Nmap Results

Before proceeding I need to have some results to play with. Lets port scan because they are the heroes we need and can take it.

Here is the command that was run:

nmap -sS -sV -Pn -oA -v
Nmap scan report for (
Host is up (0.0080s latency).
Other addresses for (not scanned): 2a00:1450:4009:805::2004
rDNS record for
Not shown: 998 filtered ports
80/tcp open http
443/tcp open ssl/https
2 services unrecognized despite returning data.

I beg the courts indulgence to explain the flags:

  1. -sS – A TCP SYN Scan. A privileged command so you need to have root privileges.
  2. -sV – Conduct a version scan. When nmap knows your service the “service” and “version” and”extra” parts of the output and XML files are populated with juicy details.
  3. -oA – Output to all three common nmap formats.
    1. “.nmap” – as shown in the above example.
    2. “.xml” – an XML file that nmap-summariser and nmap-grepper use.
    3. “.gnmap” – greppable nmap format. Comma separated variables.
  4. -v – Be a little verbose. Just a little.

The core bit here is that you must export the XML file format to use the tools I am talking about later. The optional part is using “-sV” to get more information to play with.


I made this to meet a need I had when writing a report. You pass it an nmap “.xml” file and it will spit out the data. Primarily I published this code so that someone had a simple baseline Python script that they could manipulate the output of easily to match whatever they needed. Funnily enough, nmap-grepper exists because it turned out I needed it!

This is not really meant to be used by people, but it had the hooks in the right places of the XML file so you could simply alter what you printed out to match your needs.

How does it work? Hackers are often curious so lets peek under the hood:


The for loop to look at

This is the key part of the script. The code above line 48 gets a reference to the XML file as an object and allows us to pick out specific XML nodes to get data from. To understand the for loop we need to know a little bit about the XML file format.

Here is a high level view of the XML structure:

 <scaninfo> Information related to how the scan was configured </scaninfo>
 <host> all data about host A </host>
 <host> all data about host B </host>
 <host> all data about host C </host>
 <runstats> Information about how long it took to run etc. </runstats>

The key part to note is that there is one or more “<host>” tags. Each IP address in your scan will be represented by a “<host>” tag. I have showed three to get the point across. Of-course our XML file from scanning will have only one “<host>” tag.

Lets look at that for loop again. As I think of myself as a nice guy here it is below:


Hey, it is that for loop again!

You can probably tell that this for loop is now essentially going through every “<host>” tag. Lines 49 and 50 are then getting access to a child node and then an attribute of that child node respectively. This is shown below:


Marrying the XML to the Python Code

I haven’t made my script amazingly robust. You might find errors at line 49 if it somehow does not have an entry at position [0]. It may also behave unexpectedly on an IPv6 network at line 50. Who knows?

For more information there is no substitute for opening your “” file in a text editor, and viewing the script to see which tags/attributes it picks out:

For bonus points. Modify your script to display output in this format:


You will need to find a target, and re-run nmap with “-O” to force an operating system guess.

You will have to heavily modify the output lines to achieve this, and my script doesn’t even vaguely touch the OS tags so you have to find them in the XML. Happy hunting!


I went onsite and had a need to grep ports very quickly since there was a lot of data to sift through. So I modified “” and ended up with “”:

Why does this exist when nmap outputs a “.gnmap” which is greppable?  Well, because that isn’t very greppable! Is the answer. The “.gnmap” format uses one line per host which can include any number of services. It gets very messy to work with very quickly.

So what does “” do ? It simply flips it so you get a port/service view instead of a host view. Like this:


Output of

With one host that only has two services we do not have much to work with here. Imagine this blown up to an internal network which has thousands of internal services. You can use “grep” and “cut” very quickly to interact with this file format. Want to find all “http”services ?


Using grep to list only http services

A simple grep will give you all services that operate on plain-text HTTP.

Want to just store the IP addresses that have “http” ? Go for a cut mate, like this:


Using cut to display only the IP

There you go you can now  slice and dice an nmap scan to focus on particular services quickly. May no FTP service permit anonymous logins without you knowing about it.

When tackling a massive network I try to take a service approach first because a few common flaws will “get you in” most times. If that isn’t working flipping your attention to individual hosts that have some interesting services you can enumerate will often yield results.

If you want another coding challenge here then why not make “” spit out this format instead:


That means more opportunities for greppability. Find me all things running Microsoft Server 2008? Show me any host with “dev” or “test” in the hostname? These are things you will be able to answer if you do.

Note: you can achieve the same effect with metasploit if you have enabled the Postgres database. Your reading for that is 100% this URL:

I don’t claim nmapgrepper is better. It is just sometimes fun to roll your own to know more.

Hope that helps