Tag Archives: python

Grep Extractor a Burp Extender

Burp Suite’s “Intruder” is one of my favourite features. It automates various parts of my job for me by repeating a baseline request with minor variations. You can then check out how a target responded. Unlike the “Reapeater” you get a nice table of results and at a glance can find things with different response codes. Basically Intruder is brilliant.

Intruder has a feature called Grep Extract which allows you to find content within HTTP Responses and then extract the values. You might want to do this if you are enumerating users by an ID and you want to extract the email addresses for example.

I looked but could not find the same functionality via the Proxy History so I made a simple Extender to add that functionality. This blog post covers:

  • Basic Usage of Grep Extract – showing how to use Grep Extract within Intruder. Why not show the inspiration?
  • Grep Extractor – showing the code and how to use it.

This extender is designed to have the code altered by you when you want to extract something. It has never been easier for you to get your hands dirty and get a new Extender that does something useful!

Basic Usage of Grep Extract

When you are inspecting the results of an intruder attack you can use the “options” tab and “Grep – Extract” down at the bottom to extract data from a response. Here is what the options look like:


Click on “Add” to bring up the screen below where you can simply highlight the part you want to extract:


In this case the response page has a Credit Card number so I highlighted that part. When you apply that the Intruder results table will update to include a new column with the extracted data:


You can export the results to a CSV file via that “Save” menu. This is all very well and good when you are using Intruder.

Grep Extractor

You have seen how Burp provides this feature within Intruder. It uses a nice GUI approach which we are not replicating at all. The following shows the source code for Grep Extractor:

#burp imports
from burp import IBurpExtender
from burp import IBurpExtenderCallbacks
from burp import IExtensionHelpers
from burp import IContextMenuFactory
from burp import IContextMenuInvocation
import re

# java imports
from javax.swing import JMenuItem
import threading

class BurpExtender(IBurpExtender, IContextMenuFactory):

   def registerExtenderCallbacks(self, callbacks):
      self.callbacks = callbacks
      self.helpers = callbacks.getHelpers()
      self.callbacks.setExtensionName("Grep Extractor")

   def createMenuItems(self, invocation):
      menu_list = []
      menu_list.append(JMenuItem("Grep Extractor", None, actionPerformed= lambda x, inv=invocation:self.startThreaded(self.grep_extract,inv)))
      return menu_list

   def startThreaded(self, func, *args):
      th = threading.Thread(target=func,args=args)

   def grep_extract(self, invocation):
      http_traffic = invocation.getSelectedMessages()
      count = 0
      for traffic in http_traffic:
         count = count + 1

         if traffic.getResponse() != None:
            # if the string is in the request or response
            req = traffic.getRequest().tostring() 
            res = traffic.getResponse().tostring()

            # start is the string immediately before the bit you want to extract
            # end is the string immediately after the bit you want to extract
            start = "" 
            end   = ""

            # example parsing response. Change res to req if data is in request.

            i = 0
            for line in res.split('\n'):
               if start in line:
                  # extract the string
                  extracted = line[line.find(start)+len(start):]
                  extracted  = extracted [0:extracted .find(end)]
                  # print exracted string, visible in Burp
                  print extracted 

Nothing too scary in there and the comments should help you out. Lets give one simple example of how to use it. Lets say the site you are targeting has the “X-Powered-By” header. Was that consistent across all responses or did it alter at any point? Perhaps some folder is redirecting to a different backend system and you didn’t notice.

Modify the start and end strings as shown below:

start = "X-Powered-By:" 
end   = "\n"

Any data between “X-Powered-By:” and the next newline character will be printed out. Save your code and then reload the Extender within Burp. At this point you can right click on one or more entries in the proxy history and send to Grep Extractor via the option shown below:


Any “print” commands issued from the Extender will goto the output for the extender. This is visible on the following menu:

Extender -> Select “Grep Extractor” -> Select “Output” tab.

The following shows output from the proxy history with our target:


It looks like the target site is consistent with it’s “X-Powered-By” headers. Well we struck out there but hopefully you can see the benefits of getting dirty and dipping your toes in the ocean of Burp Extenders. With relatively little coding knowledge you can get powerful results from Grep Extractor.

Example X-CSRF-Token

This example shows how to markup each request which did NOT include the HTTP header “X-CSRF-Token”:

   def grep_extract(self, invocation):
      http_traffic = invocation.getSelectedMessages()
      count = 0
      for traffic in http_traffic:
         count = count + 1

         if traffic.getResponse() != None:
            # if the string is in the request or response
            req = traffic.getRequest().tostring() 
            if req.find("X-CSRF-Token:")== -1:
                traffic.setComment("Request without X-CSRF-Token header")

This uses the “setComment” and “setHighlight” methods as documented at the following URL:


Instead of logging information to the stdout this will update all requests within proxy visibly with a pink background and a useful comment. This does not alter any pre-existing highlights or comments (at least when I tested it).

By reviewing the proxy history I discovered the token was consistently set for everything apart from the login form. There was no impact but it helped me get to this answer quickly.

Example Set-Cookie

This example shows how to print out every “Set-Cookie” directive in the selected responses:

   def grep_extract(self, invocation):
      http_traffic = invocation.getSelectedMessages()
      count = 0
      for traffic in http_traffic:
         count = count + 1

         if traffic.getResponse() != None:
            # if the string is in the request or response
            req = traffic.getRequest().tostring() 
            res = traffic.getResponse().tostring()

            start = "Set-Cookie:"
            end = "\n"

            for line in res.split("\n"):
               if line.find(start) !=-1:
                  line = line.strip()
                  print line

I needed to do this when conducting a re-test of an application which had certain cookies set without “httpOnly” and others without “secure” flags. By printing the full “Set-Cookie” directive I even visually caught a few anomalies where rare cases resulted in “secure; secure;”. Most likely the result of the framework and then reverse proxy ensuring the flag was set. It only affected one folder.

Vulnerable Test Site

The data shown in the proxy logs all comes from browsing the vulnerable website from Acunetix available below:


This was just to populate my Burp history with a few requests and responses.

Hope that helps,

Using Python and Scapy to hunt for VLAN IDs

A customer asked me to check for Cisco Discovery Protocol (CDP) based VLAN hopping on their LAN. It had been reported the year before and, while they hoped that it had been addressed, they wanted me to confirm that it had.

When pentesting it can often be the case that you are basically verifying the solutions to problems that some hacker from another company got to run riot with before.

Far from a burden, it gave me a chance to brush up a little bit. It got me to play with scapy a bit and so this post is basically so I can save my python code in-case I need it again, while sharing it in case it helps someone.

The easy tool for this has been Frogger for several years. It will conduct packet sniffing on an interface looking for CDP packets. Snarffle the VLAN IDs and some other information and then ask you exactly how you want to own a network.

I ran frogger and shock! No VLAN data and no CDP packets to play with. I optimistically increased the packet capture to 3 minutes and tried again. Nope…

Well I guess last years team had much more fun than me.

False Negatives

A thing to remember about Frogger is that if you are trying to run it inside a virtual machine then you might actually be getting false negatives here. Where a false negative is the absence of the vulnerability despite you doing almost the right thing to check for it.

The Readme.md for Frogger says this:

Notes for VMware. VLAN hopping generally (not just this script) can have issues within VMware if running the VM within Windows with certain Intel drivers. The Intel drivers strip off the tags before it reaches the VM. Either use an external USB ethernet card such as a DLink USB 2.0 DUB-E100 (old model 10/100 not gigabit) or boot into Backtrack nativiely from a USB stick. Intel has published a registry fix, to work on some models: http://www.intel.com/support/network/sb/CS-005897.htm – adding “MonitorMode” 1 to the registry does resolve the issue.”

I am paranoid about false negatives so I always run around with a USB Ethernet Device. I did use this for my run of frogger and so I was pretty much certain it was not a false negative.

Checking for the VLAN IDs

Earlier in the engagement I had captured around 2 hours worth of network traffic into a pcap file. This was done from my host OS and not from within my Kali VM. This would therefore have no doubt about capturing all the layer-2 juiciness. Since it was already sitting around why not use python to parse the pcap file and tell me any VLAN ID’s it contained?

You could use a wireshark filter (and I show you the expression at the end of this post). But I am doing things pythony more an more so lets get to scapy.

Enter the VLAN ID hunting script:

from scapy.all import *
from scapy.utils import *
import sys

if len(sys.argv)!=2:
   print sys.argv[0] + " <pcap_file>"

# If we get here we have a file
pcapfile = sys.argv[1]

# Loop through the file and check each packet
for pkt in pkts:
   if pkt.haslayer(Dot1Q):
      print "VLAN ID: " + str(pkt[Dot1Q].vlan)

For full disclosure the above is heavily based on a stack overflow answer here:


All I did was give it a usage and command line argument for re-usability.

I ran this against my 2 hours worth of packets and found not a single VLAN ID. So absolutely on that day, on that part of the network, there was no evidence of last years vulnerability. Well done to the customer they fixed a thing!

I beg the audiences indulgence for a sidebar

There is an episode of Red Dwarf where Dave Lister gets sick with a radioactively mutated virus. He meets characters that represent both his confidence, and his paranoia. Right about now I was listening to my inner US game show host (representing my confidence). I was thinking: job done CornerPirate. Job done.

Then my paranoia spoke up. You haven’t tried the above script against a PCAP where you knew there was a VLAN id. What if it didn’t work?

Generating a PCAP with VLAN IDs

So lets make a pcap file which has some VLAN IDs in there. Back to python and scapy:

from scapy.all import *
from scapy.utils import *
import sys

def usage():
   print "\nUsage:"
   print "\t" + sys.argv[0] + " <vlan id>"
if len(sys.argv)!=2:

if sys.argv[1].isdigit() == False:
   print "Specified VLAN ID is not a number"
# If we get here we have a vlan id to inject
vlanid = int(sys.argv[1])

# Craft a packet and send it
sendp(Ether(dst='ff:ff:ff:ff:ff:ff', src='11:22:33:44:55:66')/Dot1Q(vlan=vlanid)/IP(dst='', src='')/ICMP())

I ran the above with a few different numbers while using Wireshark to capture the packets. Visually I could now see there were definitely VLAN ID’s to be had as shown below:


Finding the VLAN ID using Wireshark Expression

The expression used was “vlan.id” which means “show packets which have a VLAN ID”. My 2 hours worth of packets resulted in zero packets for the same filter. My confidence was now building.

Rerunning my “check-vlan.py” script now showed the same results as Wireshark:


Listing out VLAN

So there you have it. I now know my script does what it intended to.



Simple HTTP/HTTPS Servers in Python

There are loads of situations where you need a quick way to spin up an HTTP server. The following are just off the top of my head but I bet you can all think of more:

  • You have established a shell and you want to upload tools to your new computer.
  • You want to exfil data back out of an environment.
  • You have found an XSS and you want to prove you can get useful data out of a victim’s session easily.
  • You want to host a JavaScript file to prove the dangers of poor Transport Layer Security or sites which host files on 3rd party hosts.

In these situations configuring Apache is overkill and you probably only want a simple service which will last a matter of minutes. I am going full Python for this one and I have reworked this post to include Python3 syntax to keep it current in 2018.

Don’t Run with Scissors

The HTTP and HTTPS listeners that I show in this post will share the contents of the folder where you run them. As with all situations “BE AWARE OF WHAT IS IN YOUR WEB ROOT“. If you are starting a listener on the Internet then it will be port scanned within 30 minutes, and you will see automated attackers spamming your logs.

Before you run any of these services take a moment to consider where you are running them. I usually create a folder in “/tmp/” for an engagement such as “/tmp/safeplace” since this will not persist for very long.

Additionally, these servers have Directory Listings enabled by default. If you want to limit what can be seen it is a good idea to create an “index.html” file in your folder. That way your assailant will be forced to use Dirbuster which gets noisy.

Simple HTTP Server (Python 2)

For most situations this is my goto move:

python -m SimpleHTTPServer <port> # Syntax
python -m SimpleHTTPServer 8080   # Example

When you run this it starts to log things to the console as shown:


By default you get the IP address of the client. Then the files and response codes.

Simple HTTP Server (Python 3)

Python 3 is the future!!! They have renamed the module to “http.server”. Time to update your tools to do this:

python3 -m http.server <port> # Syntax
python3 -m http.server 8080   # Example

Which does very similar things as shown below:


Not a lot different really.

Generating Keys and Certificates for HTTPS

Sometimes you need to use HTTPS. The two best cases are:

  1. To Gain Privacy – if your target has traffic inspection but they do not terminate SSL then you can genuinely smuggle shells through SSL. I have seen a customer network which would catch plain-text but did nothing about an HTTPS connection.
  2. To Avoid Browser Mixed Content Warnings – these warnings can hinder a decent proof of concept. If you are trying something that “should be working goddamn it!”. Consider looking at the developer tools for your web browser and look for mixed content exceptions.

Whatever your motivation the HTTPS services I show will rely on you having a “privatekey.pem” and “cert.pem” file. You can generate these with openssl using the commands below:

openssl genrsa > privatekey.pem
openssl req -new -x509 -key privatekey.pem -out cert.pem -days 365 

You will be prompted to enter information to populate the fields of your self-signed SSL certificate. I stuff those with spam for proof of concepts. If I was engaged in a phishing exercise then a legit SSL certificate is the right move. 

Simple HTTPS Server (Python 2)

I really liked “twisted” because it can do a *lot* of things. First you will need to install that using python’s pip package manager as shown:

pip install twisted

You can then start an HTTPS service using the command below:

twistd web --https=<port> --path=. -c </path/to/cert.pem> -k </path/to/privatekey.pem> #Syntax
twistd web --https=8443 --path=. -c cert.pem -k privatekey.pem                         #Example

Twisted automatically creates a logfile called “twisted.log” in the current directory it does not want to echo to stdout so you need to use:

tail -f twisted.log

The “-f” means read this file when it changes. The following shows what this means:


The content of the output includes User-Agents and more debugging related to SSL but is otherwise similar to “SimpleHTTPServer”.

Simple HTTPS Server (Python 3)

Python 3 is the future!!! Python 3 is the future!!! Python 3 is the future!!!

Again there are minor differences between python 2 and 3. This time you have to use “pip3” to install Twisted:

pip3 install twisted

If you already have “twisted” installed for python 2 this can get a bit confusing. Running the above seems to clobber the version in the path. When you run “which twistd” you will be pointed at the python3 version. Don’t trust me? Check the first line of that file:


Great. So is the syntax to run the server different? Fortunately not in my test the following worked just fine once more:

twistd web --https=<port> --path=. -c </path/to/cert.pem> -k </path/to/privatekey.pem> #Syntax
twistd web --https=8443 --path=. -c cert.pem -k privatekey.pem                         #Example

Hope that saves you some time when you want a simple HTTPS server!

Take care