• Stealth Security
  • Posts
  • Finding Hidden Directories, Sub-Domains, and S3 Buckets using Gobuster

Finding Hidden Directories, Sub-Domains, and S3 Buckets using Gobuster

Gobuster is a fast brute-force tool to discover hidden URLs, files, and directories within websites. Learn how to work with Gobuster in this practical tutorial.

Before we get into the article, we would like to emphasize that we are not responsible for any damage you do trying to attack systems. Its illegal. You should have written permission before you even try to scan a system or a network.

There’s much more about web servers and websites than what appears on the surface.

The first step in attacking a website is to find the list of URLs and sub-domains. Web developers often expose sensitive files, URL paths, or even sub-domains while building or maintaining a site.

This is a great attack vector for malicious actors.

For example, if you have an e-commerce website, you might have a sub-domain called “admin”. This might not be linked anywhere on the site but since the keyword “admin” is common, the URL is very easy to find. This is why you must often scan your websites to check for unprotected assets.

The usual approach is to rely on passive enumeration sites like crt.sh to find sub-domains. But these passive approaches are very limited and can often miss critical attack vectors.

Gobuster is a tool that helps us to perform active scanning on web sites and applications. Crackers use it to find attack vectors and we can use it to defend ourselves.

In this article, we’ll learn to install and work with Gobuster. We will also look at the options provided by Gobuster in detail. Finally, we will learn how to defend against these types of brute-force attacks.

What is Gobuster?

Written in the Go language, Gobuster is an aggressive scanner that helps us to find hidden Directories, URLs, Sub-Domains, and S3 Buckets seamlessly.

This is where people ask: What about Ffuf?

Ffuf is a wonderful web fuzzer, but Gobuster is a faster and more flexible alternative. Gobuster also has support for extensions with which we can amplify its capabilities. Gobuster also can scale using multiple threads and perform parallel scans to speed up results.

How to install Gobuster?

Let’s see how to install Gobuster. If you are using Kali or Parrot OS, Gobuster will be pre-installed.

If you are using Ubuntu or Debian-based OS, you can use apt to install Gobuster.

$ apt install gobuster

To install Gobuster on Mac, you can use Homebrew.

$ brew install gobuster

To install Gobuster on Windows and other versions of Linux, you can find the installation instructions here.

Once you have finished installing, you can check your installation using the help command.

$ gobuster -h 

Gobuster help command

What are Wordlists?

If you are new to wordlists, a wordlist is a list of commonly used terms. This can be a password wordlist, username wordlist, subdomain wordlist, and so on. You can find a lot of useful wordlists here.

I would recommend downloading Seclists. Seclists is a collection of multiple types of lists used during security assessments. This includes usernames, passwords, URLs, etc. If you are using Kali Linux, you can find seclists under /usr/share/wordlists.

To try Gobuster in real-time, you can either use your own website or use a practice web app like the Damn Vulnerable Web app (DVWA). DVWA is an intentionally misconfigured vulnerable web application that is used by pen testers for practicing web application attacks.

Working with Gobuster

Now that we have installed Gobuster and the required wordlists, let’s start busting with Gobuster.

Note: I have DWVA running at 10.10.171.247 at port 80, so I ll be using that for the examples. Please replace that with your website URL or IP address. I ll also be using Kali linux as the attacking machine.

If you look at the help command, we can see that Gobuster has a few modes.

  1. dir — Directory enumeration mode.

  2. dns — Subdomain enumeration mode.

  3. fuzz — Fuzzing mode.

  4. s3 — S3 enumeration mode.

  5. vhost — Vhost enumeration mode.

In this article, we will look at three modes: dir, dns, and s3 modes.

Each mode serves a unique purpose and helps us to brute force and find what we are looking for. Let's look at the three modes in detail.

Directory mode (dir)

The Directory mode of Gobuster helps us to look for hidden files and URL paths. This can include images, script files, and almost any file that is exposed to the internet.

Here is the command to run the dns mode.

$ gobuster dir -u <url> -w <wordlist>

We can also use the help mode to find the additional flags that Gobuster provides with the dir mode.

$ gobuster dir -h

Gobuster dir mode help

Now let’s try the dir mode. Here is the command to look for URLs with the common wordlist.

$ gobuster dir -u 10.10.171.247:80 -w /usr/share/wordlists/dirb/common.txt

And here is the result. We can see that there are some exposed files in the DVWA website.

dir enumeration results

If we want to look just for specific file extensions, we can use the -x flag. Here is a sample command to filter images.

$ gobuster dir -u 10.10.171.247:80 -w /usr/share/wordlists/dirb/common.txt -x jpg,png,jpeg

DNS mode (dns)

The DNS mode is used to find hidden subdomains in a target domain. For example, if you have a domain named mydomain.com, sub-domains like admin.mydomain.com, support.mydomain.com, etc. can be found using Gobuster.

Let’s start by looking at the help command for dns mode.

$ gobuster dns -h

Gobuster dns help

To execute a dns enumeration, we can use the following command.

$ gobuster dns -d mydomain.com -w /usr/share/wordlists/dirb/common.txt

Since we can't enumerate IP addresses for sub-domains, we have to run this scan only on websites we own or the ones we have permission to scan.

S3 mode (s3)

AWS S3 is used by a large number of companies to deliver content. This can include both public content like website images and private/sensitive files.

Let’s look at the help command to see the flags that Gobuster provides for S3 mode.

$gobuster s3 -h

Gobuster S3 mode help

The S3 mode was recently added to Gobuster and is a great tool to discover public S3 buckets. Since S3 buckets have unique names, they can be enumerated by using a specific wordlist.

For example, if we have a company named Acme, we can use a wordlist with acme-admin, acme-user, acme-images, etc. This wordlist can then be fed into Gobuster to find if there are public buckets matching the bucket names in the wordlist.

Here is the command to execute an S3 enumeration using Gobuster.

$gobuster s3 -w bucket_list.txt

How to defend against Gobuster?

Gobuster is a remarkable tool that you can use to find hidden directories, URLs, Sub- Domains, and S3 Buckets.

But this enables hackers to use it and attack your web application assets. So how do we defend against Gobuster?

You can use the following steps to prevent and stop brute-force attacks on your web application.

  1. Audit yourself: Use Gobuster on your own applications and perform an audit. This will help you find the information that will be visible to the attackers.

  2. Apply security policies: To prevent resources like S3 from being exposed on the internet, use AWS bucket policies to prevent unauthorized access.

  3. Use bot protection solutions: Bot protection services like Cloudflare will stop any brute-force attacks making it incredibly difficult to attack your web application.

Conclusion

Gobuster is a fast brute-force tool to discover hidden URLs, files, and directories within websites. This will help us to remove/secure hidden files and sensitive data.

Gobuster also helps in securing sub-domains and virtual hosts from being exposed to the internet. Overall, Gobsuter is a fantastic tool to help you reduce your application’s attack surface.

To learn how to build a career in Cybersecurity, check out The Hacker’s Handbook. To practice hacking real systems and get help from other hackers, join The Hacker’s Hub.

Reply

or to participate.