SecurityTrails Blog

Please note: The audio version doesn't include code or commands. Those parts of the post can be seen in the text version.

I spent a very long time automating my recon for bug bounties. I collaborated with a couple of friends for about 12 months to build out an automation beast. We had a custom framework, and constant recon scanning with good distribution, at times we scaled up to 100+ servers. We stored data on millions of targets and had Slack notifications for vulnerability detection. It was the third iteration of our automation and we thought it was great. I mean, it was pretty great, and it definitely helped us earn some cash on a few popular bounty programs.

I learned a lot while I was coding this but ultimately, looking back I wish I had done things differently. Namely, I wish I had decoupled each component of the automation into small chainable tools instead of building everything as one giant framework. In other words, I wish I had followed the Unix philosophy. What's the Unix philosophy? Wikipedia puts it best: The Unix philosophy emphasizes building simple, short, clear, modular, and extensible code that can be easily maintained and repurposed by developers other than its creators. In this blog I'm going to show you how you can use three basic, free, open source tools to implement continuous monitoring for new subdomains in just five minutes. The three tools are:

Anew by tomnomnom. Haktrails by me, you could use any subdomain enumeration tool here, like Subfinder or OWASP-Amass. Notify by Projectdiscovery.

Setting up your environment

Let's see what the necessary steps are to build a basic monitoring and alerting system for detecting new subdomains.

Picking a VPS

This is the kind of automation that you probably want to run 24, 7, 365. The easiest way to do this is to set up a VPS. You can use whatever Linux distribution but I tend to use Ubuntu because that's where I feel at home. You can set up an Ubuntu VPS on any popular hosting provider. Linode and Digital Ocean have them starting at $5 per month. If you're a new AWS customer you could also use a free tier EC2 instance which will be free for the first 12 months.

Installing Golang and Tmux

Once you've set up your VPS, you'll need to install Golang. If you're using Ubuntu, this should be as simple as. You probably also want Tmux so that you can leave the automation running without stopping every time your SSH session ends.

Installing the tools

Now that you have Golang installed, you can use the built-in Go package manager to install the tools that we'll be using. These commands work at the time that this blog post was written, but installation instructions for these tools may change so it's always best to check the installation instructions on the repositories directly.

Adding GOBIN to your $PATH

In order to run these new tools by typing the name of the tool: haktrails Instead of the full path. We need to modify our bash to include. We can do this by adding the following line to our file. Then restart your terminal, or run.

Setting up your config files

To use haktrails, you'll need to set up your config file containing your Securitytrails AP-I. You can find instructions to do that here. To use notify, you'll need to set up your config file with your Discord, Slack, Telegram webhook(s). You can find instructions here.

Gathering existing subdomains

For the purpose of this blog, let's monitor securitytrails.com. The first step is to gather the existing subdomains for that domain and pop them into a file. You can achieve this by running the following command:

Setting up constant monitoring

There are a bunch of ways to do this. My personal preference would be to use hakcron, but you can easily achieve the same thing with some simple bash, so let's do that. To use the code above. Copy the code above into a file called monitor.sh. The bash script runs the command we created above, then does nothing for 3600 seconds. It repeats this process indefinitely. If you want to l...

I spent a very long time automating my recon for bug bounties. I collaborated with a couple of friends for about 12 months to build out an automation beast. We had a custom framework, and constant recon scanning with good distribution, at times we scaled up to 100+ servers. We stored data on millions of targets and had Slack notifications for vulnerability detection. It was the third iteration of our automation and we thought it was great. I mean, it was pretty great, and it definitely helped us earn some cash on a few popular bounty programs.

I learned a lot while I was coding this but ultimately, looking back I wish I had done things differently. Namely, I wish I had decoupled each component of the automation into small chainable tools instead of building everything as one giant framework. In other words, I wish I had followed the Unix philosophy. What's the Unix philosophy? Wikipedia puts it best: The Unix philosophy emphasizes building simple, short, clear, modular, and extensible code that can be easily maintained and repurposed by developers other than its creators. In this blog I'm going to show you how you can use three basic, free, open source tools to implement continuous monitoring for new subdomains in just five minutes. The three tools are:

Anew by tomnomnom. Haktrails by me, you could use any subdomain enumeration tool here, like Subfinder or OWASP-Amass. Notify by ProjectDiscovery.

Setting up your environment

Let's see what the necessary steps are to build a basic monitoring and alerting system for detecting new subdomains.

Picking a VPS

This is the kind of automation that you probably want to run 24, 7, 365. The easiest way to do this is to set up a VPS. You can use whatever Linux distribution but I tend to use Ubuntu because that's where I feel at home. You can set up an Ubuntu VPS on any popular hosting provider. Linode and Digital Ocean have them starting at $5 per month. If you're a new AWS customer you could also use a free tier EC2 instance which will be free for the first 12 months.

Installing Golang and Tmux

Once you've set up your VPS, you'll need to install Golang. If you're using Ubuntu, this should be as simple as:

sudo apt update && sudo apt install golang. You probably also want Tmux so that you can leave the automation running without stopping every time your SSH session ends. apt install tmux.

Installing the tools

Now that you have Golang installed, you can use the built-in Go package manager to install the tools that we'll be using. These commands work at the time that this blog was written, but installation instructions for these tools may change so it's always best to check the installation instructions on the repositories directly.

GO111MODULE=on go get -v github.com/projectdiscovery/notify/cmd/notify.

go get -u github.com/hakluke/haktrails.

go get -u github.com/tomnomnom/anew.

Adding GOBIN to your $PATH

In order to run these new tools by typing the name of the tool: haktrails Instead of the full path:

/go/bin/haktrails. We need to modify our bash $PATH to include ~/go/bin/. We can do this by adding the following line to our ~/.bashrc file :

export PATH="$PATH:~/go/bin/".

Then restart your terminal, or run:

source ~/.bashrc.

Setting up your config files

To use haktrails, you'll need to set up your config file containing your Securitytrails API key. You can find instructions to do that here. To use notify, you'll need to set up your config file with your Discord, Slack, Telegram webhook(s). You can find instructions here.

Gathering existing subdomains

For the purpose of this blog, let's monitor securitytrails.com. The first step is to gather the existing subdomains for that domain and pop them into a file. You can achieve this by running the following command: echo securitytrails.com , haktrails subdomains , tee subdomains.txt

Setting up constant monitoring

There are a bunch of ways to do this. My personal preference would be to use hakcron, but you can easily achieve the same thing with s...

Cloud computing is beneficial. Many organizations already know this and are reaping the benefits cloud adoption has brought them: reduced IT costs, scalability, collaboration efficiency and, above all else, flexibility in accessing storage and software to meet their needs. Users can now more easily engage services and solutions that will make their everyday jobs easier.

That flexibility, however, is a double-edged sword. And you might be wondering how it can possibly be a bad thing, especially when it makes everyone work more efficiently. The thing is, many of those solutions and software your employees gain access to aren't under the governance of your IT department. While IT usually enforces policies that dictate the software, hardware and other resources used within the organization, with oversight regarding how they're used, the introduction of cloud computing means users can much more easily access resources they need with limited visibility from IT. So yes, using these resources can improve productivity, but their introduction without the governance and approval of an organization's IT department can lead to numerous security risks, data loss, non-compliance and exponential growth of the attack surface.

What is shadow IT?

When something is in a shadow, you don't know what it is. You don't know if it's malicious or completely harmless. How can you even begin to handle and manage something for which you have no overview? Well, your organization's IT has its own shadow. It's appropriately named "shadow IT". Shadow IT is the use of systems, devices, software, apps and services without approval from an organization's IT department. Most users who employ unauthorized solutions don't do it with any ill intention but to be more productive at their job. Shadow IT can include:

Hardware and physical devices - smartphones, tablets, IoT, flash drives, external drives. SaaS, PaaS, IaaS, and other cloud services - productivity apps, messaging apps, cloud storage. Data repositories such as spreadsheets with internal data. APIs. VPNs. Commercial off-the-shelf software.

Shadow IT is not all bad, though. While some may view it as a potentially dangerous nuisance that needs to be addressed and prevented, others see it as an innovation-driver and the natural manifestation of an ever-changing business environment's constant need to catch up. No matter what your stance on shadow IT, it's here and it's staying.

Why is shadow IT so prevalent?

If there are employees in an organization, there is shadow IT. Statistics show that 80% of end users use unapproved software and services. There was a time, under the shadow IT umbrella, when we saw a lot of software that impatient employees had downloaded and used, but it wasn't as prevalent as all of today's packaged software nor as easy to engage. Definitely not as easy as a click of a button, which the adoption of the cloud has introduced. Shadow IT now includes personal technology and devices employees bring in, which is propagated with remote work and BYOB policies. Rapid growth of the business landscape has increased the need for additional applications that can make employees' daily tasks easier and more efficient. With numerous businesses, productivity, storage, automation and other applications available in the cloud, they drive innovation, productivity and efficiency.

It's easy to see why they're so enticing and why employees don't hesitate to download and employ these apps.Some corporate solutions might not only be incompatible with users' devices, they might also be slower, outdated and less effective. Combine this with the often long and tiring process of seeking approval from an IT department, and we can begin to see why shadow IT is continuously growing. One thing is certain: shadow IT is inevitable, so it wouldn't be fruitful to "fight it", but to understand its risks and the appropriate way to manage it.

Risks associated with shadow IT

There are numerous cybersecurity risks associated w...