How I found the Grafana zero-day Path Traversal exploit that gave me access to your logs

TL/DR: On December 2, open-source analytics solution Grafana released an emergency security patch for critical zero-day Path Traversal vulnerability CVE-2021-43798, after proof-of-concept code to exploit the issue was published online. The flaw, which received a 7.5 CVSS score enabling remote access to local files, is no longer exploitable on servers that have the latest Grafana update. I (Jordy Versmissen, a Detectify Crowdsource security researcher) alerted Grafana about the zero-day before it blew up on Twitter. 

How I discovered it

I decided to download the latest version of Grafana’s code base and started searching through the code to see if there was something out of the ordinary. 

During a source code audit of the open source Grafana project, I was searching for typical ways in Golang to read files. When I do a first code review I always search for typical functions which can cause bugs, for example path traversals, but also methods like ‘exec.Command’ and other commonly known examples and search they way up in the tree of method calls.

Image: Vulnerable code in Grafana

One of the functions you can use in Golang is os.Open. A call to os.Open from the getPluginAssets method in the pkg/api/plugins.go file got my attention.The value was passed to the filepath.Clean function, which is an interesting function. According to Go’s documentation, it removes inner .. elements, and .. elements when it starts with a forward slash. But when the value doesn’t start with a slash, the traversal won’t be removed. I created a quick script to test a few things out and it turned out this could be a bug in Grafana.

I started Grafana in a Docker container and browsed through the file system of the container to find a file which shouldn’t be public. I used a commonly known wordlist with all kinds of path traversal notations to fuzz for a possible path traversal and read the file and after a few hits I manually tested some payloads to get access to other system files. 

A lot of proxies or WAFs will normalize or block a typical ../ path traversal so I tried many different alternatives. I used the traversals-8-deep-exotic-encoding.txt wordlist in the swisskyrepo/PayloadsAllTheThings GitHub repository and replaced {FILE} with VERSION for every line.  

Image: The terminal where I’m running Grafana in a Docker container and the curl commands at the bottom. You can see the output of the logs in Grafana which already output the traversed path (open /usr/share/grafana/public/VERSION)

This is how I found a new zero day bug in Grafana which could lead to a registered CVE. When the moments of jubilation passed, reality hit and I realized just how serious this was and could become. I did what every ethical hacker does – notify Grafana before bad actors start to misuse it. 

What’s the worst that could happen?

If CVE-2021-43798 in Grafana is taken advantage of, it’s possible to read files on the server which are readable by Grafana. Given that a lot of Grafana instances are using the default Sqlite database for storage, the files are generally located at a predictable location on the server. Consequently, with a special crafted GET request the database file can be downloaded by an unauthenticated attacker. This database contains a lot of data, including users’ auth tokens and data sources. 

Furthermore, it’s also possible to read the content of the Grafana config file, which may contain the plain text passwords for databases, the admin user, OAuth credentials or LDAP credentials. This can give an attacker access to the database or admin access to the Grafana instance. If the attacker has admin access, he can create a new datasource in Grafana and use the datasource proxy API endpoint to use Grafana as a HTTP proxy. 

Grafana was quick on the trigger 

Attempting to patch the bug (CVE-2021-43798 Grafana) as soon as it was flagged, Grafana released 8.3.1, 8.2.7, 8.1.8, and 8.0.7 versions and confirmed that its cloud-hosted Grafana dashboards were not impacted by this vulnerability.

All Grafana installations between v8.0.0-beta1 and v8.3.0 should be upgraded as soon as possible. If you cannot upgrade, running a reverse proxy in front of Grafana that normalizes the PATH of the request will mitigate the vulnerability. For example, the normalize_path setting in envoy.

Grafana said, “The vulnerable URL path is: <grafana_host_url>/public/plugins/<“plugin-id”&gt; where <“plugin-id”> is the plugin ID for any installed plugin.”

Path Traversal – more common than you think

The filepath.Clean function is used a lot in Golang projects, and as a result, it’s likely that many more applications and libraries also use the method but don’t make sure the passed value starts with a forward slash. In other programming languages, it’s easy for a software developer to forget to normalize the path or to test the huge list of possible bypasses and encodings. 

In September 2019, a critical severity directory traversal vulnerability was found in Atlassian’s Jira Service Desk Server and Jira Service Desk Data Center that allowed attackers to access information belonging to the company’s customers.

Atlassian wasn’t the only company to make news with such a vulnerability. Citrix ShareFile Storage Zones Controller was affected by a relative path traversal vulnerability (CTX328123/CVE-2021-22941) when processing upload requests which could be exploited by unauthenticated users to gain Remote Code Execution. 

Mitigation methods for CVE-2021-43798 in Grafana

Path traversal attacks rely on two vulnerable elements – the web application code and the web server configuration. By avoiding vulnerabilities in both areas and ensuring to not store sensitive configuration files inside the web root, one can mitigate the majority of such attacks.

Most of the time the programming language or framework will handle these kinds of situations or they provide a function to normalize a file path and remove the typical path traversal cases. In this case the developer used the Golang function ‘path.Clean’ which, as the name suggests, should clean the path. It’s definitely a good choice to use this function, but the function has a bit of a strange effect. 

This bug is a good reminder to always read the documentation or add some unexpected cases to your unit tests. The learned lesson is the way Golang’s path.Clean method works and that you always have to read the documentation.

Refer to OWASP which has more information about Path Traversals. 

Timeline of the event 

Here is a detailed timeline starting from when Grafana originally learned of the issue. 

• 2021-12-02: Security researcher sends the initial report
• 2021-12-02: Confirmed for 8.0.0-beta1 through 8.3.0
• 2021-12-02: Confirmed that Grafana Cloud is not vulnerable
• 2021-12-02: Security fix determined and committed to Git
• 2021-12-02: Release timeline determined: 2021-12-07 for private customer release, 2021-12-14 for public release
• 2021-12-06: Second report about the vulnerability received
• 2021-12-07: Grafana received information that the vulnerability CVE-2021-43798 has been leaked to the public, turning it into a zero-day
• 2021-12-07: Private release with reduced 2-hour grace period

Written by:
Jordy Versmissen
Software developer and ethical hacker

Apart from looking into anomalies in Grafana, Versmissen has a day job as software developer and is part-time security researcher. His interest in security started two years ago and since then he reports vulnerabilities in open source projects and bug bounty programs. Some of his achievements include finding CVE in the Kubernetes SDK (CVE-2021-25738) and in the New Relic java agent. 

Detectify scans for CVE-2021-43798 Grafana 

Detectify has security tests available to find CVE-2021-43798, among other vulnerabilities that were once zero-days. The testing is payload-based which means that we only alert about vulnerabilities which we can verify with exploitation (in a safe environment).

Detectify automatically checks your websites, web servers and other software hosted for actively exploited web vulnerabilities beyond the OWASP Top 10. 

Curious to see what Detectify will find in your web apps? Sign up for a 2-week free trial today.

Into ethical hacking and want to join Crowdsource? Learn more on how you can earn recurring rewards while making the Internet safer with Detectify Crowdsource.