The trouble with modern Linux security monitoring and an easy to use tool that aims to fix it

Lets face it - the state of Linux security monitoring sucks. Linux is often treated as a second class citizen in terms of feature set when compared to its windows counter parts.

When you look at the major EDR vendors - they pretty much all have a lot of the same problems:

• No support for that weird Linux distribution that is rarely deployed in an enterprise, however for whatever reason is your companies most business critical application
• Limited support for non-standard CPU architectures if any (IE: ARM and monitoring all those pesky Raspberry Pis laying around)
• New kernel versions or major LTS Distribution versions come out and support for said versions lags behind
• Almost entirely hash reputation and process monitoring focus on detection
• Limited Out-of-the-Box Rulesets
• Minimal forensic data captured outside of process execution data and common network 5-tuple information

So lets talk about one of those rare tools that can help jump start your organizations Linux security monitoring program with something that actually works, and written by someone in the field rather a paid fluff piece. That tool is Sandfly Security.

Transparency: I have not been paid for this blog post and have no financial ties to Sandfly. All opinions of this should be considered my own, and not an opinion of my current or any former employers.

Note: Every organizations use cases are different and you should always identify your organizations own use cases before starting to look for a product!

Linux Distributions, Kernel Versions and CPU Architectures

This is a sore spot for me. Historically, Next-Gen AV (NGAV)/End Point Protection (EPP)/Endpoint Detection and Response (EDR) vendors have always treated Linux as a second class citizen. There are so many Linux distributions and not all of them are very common to see in an enterprise, be it that they are just an older no longer maintained version, a home grown distribution, the dev team wanting to use Arch, or whatever it may be.

This makes it very difficult to develop for when you take a traditional EDR agent approach.

The way a lot of organizations handle this, is through ingesting auditd logs into a SIEM. However this approach has its own problems:

• It isn't a realistic expectation for an organization to even have a SIEM
• A company that doesn't do cybersecurity shouldn't have to have an in house detection engineering team just to have Linux visibility
  • (The Sigma project is helping with this problem, but I've found the sort of organization that knows about the Sigma project, is typically the sort of organization with a detection engineering team)
• There is no "prevention" capability built in
• auditd is heavy and has a big impact on system performance

This is where I Sandfly really shines. It's an "agentless" Linux security monitoring tool that works on every single Linux distro I've come across, regardless of CPU architecture or distribution. Additionally - with the ability to adjust the binary priority there is a very low risk of knocking over a server with a Sandfly scan (bonus points for also providing an option for monitoring in environments where availability is king.)

Let's talk about how it works.

Setup

Setup for Sandfly is simple:

# replace X.X.X with version https://github.com/sandflysecurity/sandfly-setup/releases
wget https://github.com/sandflysecurity/sandfly-setup/releases/download/vX.X.X/sandfly-setup-X.X.X.tgz

tar -xzvf sandfly-setup-X.X.X.tgz

cd ~/sandfly-setup/setup/

sudo ./auto_install_allinone.sh

Login to the console with the password given to change it / add your license file, enter the CIDR range(s) you want to scan, and provide SSH credentials with root access and you're off to the races.

If you use Digital Ocean - it has an easy to deploy image in the market place that can be deployed in just a few button clicks - https://marketplace.digitalocean.com/apps/sandflysecurity

That's it.

High Level Architecture

Sandfly bills itself as an "Agentless" Intrusion Detection system. The way it works:

• It performs various "scans" known as Sandflies on a scheduled basis that you configure
• It'll perform an initial "profiling" scan to identify the Linux Distribution/Kernel Version/Architecture against the provided IP Addresses
• Once identified, it'll SSH to the IP Addresses being scanned, and drop a pre-compiled Golang binary written entirely in the Golang stdlib on the system (How bitchin' is this? Much more resistant to supply chain attacks IE: log4j)
• The Golang binary will execute the sandfly(s) in question and return the results to the Sandfly server
• Sandfly removes itself from the system

It has a few different type of Sandflies that are able to be configured:

• Recon Sandflies - Used for information gathering purposes -
  • IE: I want to get all new process / network data from a system to send to the SIEM regardless if it's malicious for historical/compliance purposes
• Detection Sandflies - Looking for malicious activity
• Incident Response Sandflies - Used if something is detected to aid investigation if the initial data returned upon detection is not comprehensive enough.
  • Not typically ran unless an investigation started.

I thought you said it was agentless...

The way that this works in my personal opinion should be considered agentless. There are no added dependencies needed to the Linux server itself. There is no installation needed or need to change a standard image that is deployed. You do not need to worry about operations teams not installing it/forgetting to install it. You can adjust the priority at which the binary executes which is great in environments where availability is king. And lastly, it's not consistently running on the box as it runs and removes itself until it is scheduled to run again.

Back to the initial problem - Monitoring weird Linux distributions / kernel versions / Linux in general

Because it is built entirely from the golang standard library, it is able to compile the sandflies in question for the Linux distribution, kernel, and CPU architecture on the intended host. Because of this, it doesn't matter if it's a cluster of Raspberry Pis, a brand new major LTS release of a distribution, or the dev team running Arch Linux. It will likely be supported. (Hell it supports MIPS.)

I personally haven't come across a Linux distribution it doesn't support yet.

High Quality Out of the Box Detection Sets

The state of Linux detections for the major EDR vendors is severely lacking. As mentioned before - Linux is often treated as an afterthought and because of this, most Linux EDR tooling is limited in what it can detect on. In most cases it is entirely limited to process monitoring, malicious hash detection (simple to evade), and "anomaly detection" (Anyone working in a SOC knows how low fidelity anomaly alerting is. So many applications are unfortunately ductaped together.)

Now it should be noted - process monitoring will give you great visibility on a system, however it isn't the only facet that should be considered in detection, especially when there are so many great detection opportunities on linux.

Speaking of detection, this is where I really feel Sandfly really shines. It has probably the most OOTB detection set for Linux that I've come across, and it focuses on behaviorial detections (also known as tactics, techniques, and procedures or TTP's. Which are typically the strongest to focus on.)

By default - Sandfly has OOTB detectors on behaviors related to:

• Files
• Processes
• Users
• Directories
• Logs
• Policies

However it's custom detection creation is incredibly powerful and actually built for Linux itself. It allows you to build detectors on as you can write detectors on all of these attributes yourself as well without having to ingest a ton of Linux log data.

On top of that, you can write detectors of content contained within bash scripts themselves which is incredibly powerful for catching a large quantity of Linux malware.

Pulling Forensic Data

Sounding like a broken record, again Linux visibility is incredibly weak across EDR's and is commonly limited in the amount of data that can be pulled. Not only that, while a lot of the EDR "flight recorder" style visualizations are helpful for determining maliciousness, it's extremely common to also need to pull other forensic data requiring other tooling that can't be pulled with an EDR itself. (This actually isn't just a problem on the Linux front - but that's another story.)

Sandfly also has the concept of "Incident Response" scans, or scans that would pull an incredible amount of forensic data that you may not typically want to constantly pull (such as all files created over the past hour).

I cannot overstate how much I love how these scans work. One common problem I come across, even in organizations lucky enough to have a security operations center or security team is a lack of Linux forensic and investigative know how.

For example - it's not uncommon for an EDR tool to detect malicious behavior on a Linux host, but provide limited visibility on what was actually done - such as what directories were accessed/modified/created? What were the last logged in users? Were there failed login attempts before/after? What processes executed outside of the direct process chain from the alert? What SSH keys are on the box (a great way to track intrusion campaigns)?

These are all things that Sandfly makes incredibly easy to do, through it's concept of running Incident Response scans:

These can be automated via the API, so if you're using SOAR (Security Orchestration Automation and Response) tooling you can collect this sort of forensic data in an automated manner upon detection.

My favorite part as someone that has worked in a few different SOC roles: all of the data being collected by Sandfly can be sent to your SIEM. Meaning if I get a detection as an analyst investigating an alert, I don't have to swap to my EDR dashboard and have to search through multiple different consoles. All of my Incident Response data goes to the SIEM which makes it SUPER easy to query against other log sets (such as your firewall logs) and create visualisations against if you're using a SIEM such as ElasticSearch or Graylog2.

Summary

If you are looking to get visibility into your Linux environment quickly, and you want something that is extremely easy to deploy I'd highly recommend taking a look at Sandfly.

It has an extremely generous free tier - and is awesome for monitoring your homelab, especially if you're like me and have a million Raspberry Pis running.

Find a technical error in a blog post?

Contact me on mastodon @signalblur