Lets face it - the state of Linux security monitoring sucks. Linux is often treated as a second class citizen in terms of feature set when compared to its windows counter parts.
When you look at the major EDR vendors - they pretty much all have a lot of the same problems:
So lets talk about one of those rare tools that can help jump start your organizations Linux security monitoring program with something that actually works, and written by someone in the field rather a paid fluff piece. That tool is Sandfly Security.
Transparency: I have not been paid for this blog post and have no financial ties to Sandfly. All opinions of this should be considered my own, and not an opinion of my current or any former employers.
Note: Every organizations use cases are different and you should always identify your organizations own use cases before starting to look for a product!
This is a sore spot for me. Historically, Next-Gen AV (NGAV)/End Point Protection (EPP)/Endpoint Detection and Response (EDR) vendors have always treated Linux as a second class citizen. There are so many Linux distributions and not all of them are very common to see in an enterprise, be it that they are just an older no longer maintained version, a home grown distribution, the dev team wanting to use Arch, or whatever it may be.
This makes it very difficult to develop for when you take a traditional EDR agent approach.
The way a lot of organizations handle this, is through ingesting auditd logs into a SIEM. However this approach has its own problems:
This is where I Sandfly really shines. It's an "agentless" Linux security monitoring tool that works on every single Linux distro I've come across, regardless of CPU architecture or distribution. Additionally - with the ability to adjust the binary priority there is a very low risk of knocking over a server with a Sandfly scan (bonus points for also providing an option for monitoring in environments where availability is king.)
Let's talk about how it works.
Setup for Sandfly is simple:
# replace X.X.X with version https://github.com/sandflysecurity/sandfly-setup/releases
wget https://github.com/sandflysecurity/sandfly-setup/releases/download/vX.X.X/sandfly-setup-X.X.X.tgz
tar -xzvf sandfly-setup-X.X.X.tgz
cd ~/sandfly-setup/setup/
sudo ./auto_install_allinone.sh
Login to the console with the password given to change it / add your license file, enter the CIDR range(s) you want to scan, and provide SSH credentials with root access and you're off to the races.
If you use Digital Ocean - it has an easy to deploy image in the market place that can be deployed in just a few button clicks - https://marketplace.digitalocean.com/apps/sandflysecurity
That's it.
Sandfly bills itself as an "Agentless" Intrusion Detection system. The way it works:
Sandflies
on a scheduled basis that you configureIt has a few different type of Sandflies that are able to be configured:
Recon Sandflies
- Used for information gathering purposes - Detection Sandflies
- Looking for malicious activityIncident Response Sandflies
- Used if something is detected to aid investigation if the initial data returned upon detection is not comprehensive enough. The way that this works in my personal opinion should be considered agentless. There are no added dependencies needed to the Linux server itself. There is no installation needed or need to change a standard image that is deployed. You do not need to worry about operations teams not installing it/forgetting to install it. You can adjust the priority at which the binary executes which is great in environments where availability is king. And lastly, it's not consistently running on the box as it runs and removes itself until it is scheduled to run again.
Because it is built entirely from the golang standard library, it is able to compile the sandflies in question for the Linux distribution, kernel, and CPU architecture on the intended host. Because of this, it doesn't matter if it's a cluster of Raspberry Pis, a brand new major LTS release of a distribution, or the dev team running Arch Linux. It will likely be supported. (Hell it supports MIPS.)
I personally haven't come across a Linux distribution it doesn't support yet.
The state of Linux detections for the major EDR vendors is severely lacking. As mentioned before - Linux is often treated as an afterthought and because of this, most Linux EDR tooling is limited in what it can detect on. In most cases it is entirely limited to process monitoring, malicious hash detection (simple to evade), and "anomaly detection" (Anyone working in a SOC knows how low fidelity anomaly alerting is. So many applications are unfortunately ductaped together.)
Now it should be noted - process monitoring will give you great visibility on a system, however it isn't the only facet that should be considered in detection, especially when there are so many great detection opportunities on linux.
Speaking of detection, this is where I really feel Sandfly really shines. It has probably the most OOTB detection set for Linux that I've come across, and it focuses on behaviorial detections (also known as tactics, techniques, and procedures or TTP's. Which are typically the strongest to focus on.)
By default - Sandfly has OOTB detectors on behaviors related to:
However it's custom detection creation is incredibly powerful and actually built for Linux itself. It allows you to build detectors on as you can write detectors on all of these attributes yourself as well without having to ingest a ton of Linux log data.
On top of that, you can write detectors of content contained within bash scripts themselves which is incredibly powerful for catching a large quantity of Linux malware.
Sounding like a broken record, again Linux visibility is incredibly weak across EDR's and is commonly limited in the amount of data that can be pulled. Not only that, while a lot of the EDR "flight recorder" style visualizations are helpful for determining maliciousness, it's extremely common to also need to pull other forensic data requiring other tooling that can't be pulled with an EDR itself. (This actually isn't just a problem on the Linux front - but that's another story.)
Sandfly also has the concept of "Incident Response" scans, or scans that would pull an incredible amount of forensic data that you may not typically want to constantly pull (such as all files created over the past hour).
I cannot overstate how much I love how these scans work. One common problem I come across, even in organizations lucky enough to have a security operations center or security team is a lack of Linux forensic and investigative know how.
For example - it's not uncommon for an EDR tool to detect malicious behavior on a Linux host, but provide limited visibility on what was actually done - such as what directories were accessed/modified/created? What were the last logged in users? Were there failed login attempts before/after? What processes executed outside of the direct process chain from the alert? What SSH keys are on the box (a great way to track intrusion campaigns)?
These are all things that Sandfly makes incredibly easy to do, through it's concept of running Incident Response
scans:
These can be automated via the API, so if you're using SOAR (Security Orchestration Automation and Response) tooling you can collect this sort of forensic data in an automated manner upon detection.
My favorite part as someone that has worked in a few different SOC roles: all of the data being collected by Sandfly can be sent to your SIEM. Meaning if I get a detection as an analyst investigating an alert, I don't have to swap to my EDR dashboard and have to search through multiple different consoles. All of my Incident Response data goes to the SIEM which makes it SUPER easy to query against other log sets (such as your firewall logs) and create visualisations against if you're using a SIEM such as ElasticSearch or Graylog2.
If you are looking to get visibility into your Linux environment quickly, and you want something that is extremely easy to deploy I'd highly recommend taking a look at Sandfly.
It has an extremely generous free tier - and is awesome for monitoring your homelab, especially if you're like me and have a million Raspberry Pis running.
Find a technical error in a blog post?
Contact me on mastodon @signalblur
Signalblur Cyber Threat Intelligence is a cybersecurity company that delivers exceptional services and empowers organizations with accessible, cutting-edge solutions. Our founder brings years of experience from a Fortune 50 company and building Cyber Security Operations Centers.
For any inquiries, please email us at contact@signalblur.io