What is Detectify?

Detectify releases Ugly Duckling, an open-source web scanner for ethical hackers

May 18, 2021

Detectify Crowdsource is a private ethical hacking community where members earn recurring rewards for submissions of known and undisclosed vulnerabilities in widely used web technologies.

Detectify released ugly duckling

I’m Carolin Solskär, Community Manager of Detectify Crowdsource and I interviewed Tom Hudson, Security Research Team Lead at Detectify to share more on a new tool we added to Github. Recently, the Detectify Research team released an open-source web scanner on Github called Ugly Duckling  to make it easier for the Crowdsource community to submit vulnerability reports with better precision. 

Speed is of the essence, especially in the world of cybersecurity, where an average of 50 CVEs are reported daily (RedScan in 2020). So when our Detectify Crowdsource ethical hacker community asked for a way to develop security tests sooner and with better accuracy, we started building. 

Background on Detectify Crowdsource

Detectify hosts a private ethical hacker platform called Detectify Crowdsource, where researchers and bug hunters submit vulnerability information for rewards. However, it’s not your average bug bounty program because it’s based on technology with a recurring income model. Community members must complete skills tests and a thorough background check before being invited.

What’s the advantage here?
Detectify web security solutions are powered by hacker research and run tests using actual hacker payloads, which means it’s more than version testing. Vulnerabilities are verified, and there’s less noise.

Overall, 2020 was a great year for collaboration with our ethical hacking community, with over 65,000 unique hits from the 650+ new security tests and a record number of 0-days built-in 2020. Our community’s eagerness to keep improving security transparency and knowledge tells us collaboration is vital in responsible disclosure of vulnerabilities and activating more hands on deck for security. 

Tell me more about Ugly Duckling

The research team built Ugly Duckling, a lightweight web scanning tool developed with ethical hackers in mind and structuring the reporting format for sooner implementation. 

Ugly Duckling detects “stateless” vulnerabilities. Like the Detectify web vulnerability scanner, it finds vulnerabilities with a single HTTP request and analyzes the response that comes back. The user will write modules as JSON files that describe what request is required and what analysis should follow on the response.

Tom Hudson, Security Research Team Lead at Detectify:
“With the Ugly Duckling format, the researcher gets the ability to look for specific pieces of text in the response, perform regular expressions, check the response code, or if particular headers exist.”

What is the motivation for building this? 

Each time a Proof-of-Concept report is sent back and forth for clarification, speed to implement and make it available for app scanning and earning bounties is lost.

Tom: “A typical piece of feedback from our Crowdsource researchers is that they would like to be able to develop modules themselves to speed up the module development process on our end. Therefore they can precisely communicate what needs to be done to detect a particular vulnerability.” 

We wanted to make this easy for our community, and we came up with a stripped-back format that includes just the bare essentials to make things work. 

How is this useful for non-Detectify hackers?

We firmly believe that knowledge-sharing and collaboration are essential for a safer Internet. We made Ugly Duckling  available as a stand-alone vulnerability scanner, which means anyone can use it for anything they like; bug bounty hunting, security research, penetration testing, etc. 

Tom: “The stand-out feature is that the code is MIT-licensed and straightforward, so if you wanted to use it as a jumping-off point to build a custom scanner, it might be worth a look too!”

What benefits/results have you seen? 

The reports we’ve had in the Ugly Duckling format are quicker to implement than other reports because we know we have all the information required to detect the vulnerability in question accurately. 

It’s still early days, but the feedback from researchers that have used it has been positive, and less reports are rejected.


Tom says, “We’ve had to ask fewer questions about submissions using the new format, which is often one of the more time-consuming things we need to do.”

 

What’s in the crystal ball?

There are still quite a few things that our internal scanning engines can do that Ugly Duckling cannot, including looking for DOM/Client-side issues like DOM XSS or Prototype Pollution. Once we’re sure the idea is working well as it is, and if there’s demand for it, we might be able to add those capabilities.

Tom: “New vulnerabilities and ideas appear all the time. If we start to see an increase in submissions that don’t work well in Ugly Duckling, we’ll try to adapt it and make more things possible where we can.”


Technical details (from Github)

Building

ugly-duckling is written in Go and has no external dependencies.

You can install it with go get:
▶ go get github.com/detectify/ugly-duckling

Or clone the repository and build it manually:
▶ git clone https://github.com/detectify/ugly-duckling.git
▶ cd ugly-duckling
▶ go install

Usage

ugly-duckling reads URLs on stdin, and takes a list of modules as its arguments (defaulting to ./modules/*.json if none are provided).
A standard invocation to run a single module against a single URL might look like this:
▶ echo https://example.com/ | ugly-duckling modules/test3.json

Or to run against multiple URLs contained in urls.txt:
▶ cat urls.txt | ugly-duckling modules/test.json

Options

  • c / --concurrency - set the concurrency for HTTP requests (defaults to 1)
  • v / --verbose - display debug type output (e.g. which modules have been loaded)

Module Format

Here is an example module that demonstrates all functionality in ugly-duckling:

{
	"request": {
		"method": "POST",
		"path": "/anything",
		"body": "{\"magicWord\": \"please!\"}",
		"headers": [
			"Content-Type: application/json",
			"Accept: application/json"
		]
	},
	"response": {
		"matchesRequired": 2,
		"matches": [
			{"type": "static", "pattern": "please!", "required": true},
			{"type": "regex", "pattern": "magic\\w"},
			{"type": "status", "code": 200},
			{"type": "header", "name": "Content-Type", "pattern": "application/.*"}
		],
		"mustNotMatch": [
			{"type": "regex", "pattern": "(server error|not found)"}
		]
	}
}

The request and response sections are both required. The minimum possible module has a path in the request section, and at least one thing in the matches list in the response section:

{
	"request": {
		"path": "/anything"
	},
	"response": {
		"matches": [
			{"type": "static", "pattern": "data"}
		]
	}
}

Request Section

The request section contains details about the HTTP request to be sent.
method – HTTP method to use; GET, POST, HEAD etc
path – Path and query string to append to the input URL
body – Data to be sent as the request body
headers – An array of headers to send with the request

Response Section

The response section contains details about how to check the response for a hit.
matchesRequired – How many matches must be made for a module to be considered a ‘hit’
matches – An array of objects describing the matches to be performed
mustNotMatch – An array of objects describing things which must not be matched

Matches

Match objects can have one of a few different types:
static – an exact text match is performed using the pattern parameter
regex – a regular expression match is performed using the pattern parameter; the regex engine is the default Go engine.
status – the status code of the response is compared to the integer in the code parameter
header – a regular expression match is performed against the header specified in the name parameter using the pattern specified in the pattern parameter

Any match object in the matches array can have a required parameter set to true so that a match must be met for a module to be considered a hit.

For more details, go to Detectify’s Github page.

Secure millions of end users with your next vulnerability finding. Learn more on how you can earn recurring rewards while making the Internet safer with Detectify Crowdsource.