OSSEC For Website Security – Part I

OSSSEC is my preferred host-based intrusion detection system (HIDS). I have to admit I am a bit partial to it because my good friend Daniel Cid built it and sold it to Trend Micro / Third Brigade back in 2008. I have what many don’t have, that’s the ability to pester Daniel until he tells me and guides through all my issues. In the process I have learned a number of things and made some very interesting observations about the product, here is where I will be sharing them.

Being that my focus is on website security my employment and utilization of the product will be as such. I won’t talk much to the configuration and monitoring of large scale enterprises, but will likely get into large n-tier implementations of web enterprises. This could include the utilization of load balancers, web servers and database servers, and possibly some storage devices. Pretty straight forward stuff.

Unfortunately, one of the observations I have made is that the implementation of OSSEC is piss poor. They are not monitoring anything of value. The other observation is simplicity. What I have learned about my dear friend is that he doesn’t believe in complexity, where possible he tries to simplify, and while I know if he had the chance he would rewrite OSSEC, simplicity is still at its core. I mention this because it’s very important to note when you work with the application.

So let’s start there.

The one caveat I need to make is that I don’t run off the version of OSSSEC provided here: http://www.ossec.net/?page_id=19. I actually run off Daniel’s branch here: https://bitbucket.org/dcid/ossec-hids/get/tip.tar.gz. This includes almost all the changes provided in the latest version of 2.7 and is pulled into the latest versions on the main site.

Default Installation

Note that this assumes you have already installed OSSEC.

Don’t get fancy with it, keep all the default configuration on install. It will automatically drop the install here:

/var/ossec/

If you’re one of those that has to fiddle then just be sure to make note of where you put things.

When you are starting off you want to be familiar with three specific directories:

/var/ossec/bin/ << Contains your OSSEC modules

and

/var/ossec/logs/ << As the name implies, it's not just the logs for the alerts but for the OSSEC install. This is important as you'll be using it to troubleshoot when something goes wrong.

and

/var/ossec/etc/ << This will contain your configuration information.

When you make changes to the configuration file you’re going to have to restart OSSEC, easiest way to do that is going to be:

# /var/ossec/bin/ossec-control

Your options are:

Usage: /var/ossec/bin/ossec-control {start|stop|restart|status|enable|disable}

To apply it’ll be:

# /var/ossec/bin/ossec-control restart

You will have to be a root user to get this going.

The Configuration File

This is where folks go wrong, they forget to pull in all the appropriate log files. When you configure it on your webserver it pulls in the bare minimum by default, you need to add all the log files on the server to get the best value.

Here is an idea of what it pulls by default on a server running Apache:

<location>/var/log/messages</location>
<location>/var/log/secure</location>
<location>/var/log/maillog</location>
<location>/var/log/httpd/access_log</location>
<location>/var/log/httpd/error_log</location>

Here is the problem with that, what if you are configured with vhosts and you have 20 sites on the server? You need to add each one of those access and error logs into OSSEC manually so that you can see everything that is going on. No worries there is a quick way to do that using the utility shell. here is an example:

# /var/ossec/bin/util.sh addfile /var/log/httpd/somesite.access_log

In addition you want to scan your server and see what else it might be logging, here is an easy way:

# find / -name *log -type f | grep -v “/var/ossec/”

Or you can run:

# lsof | grep log | grep -v ".so" | egrep -v "ossec|proc|dev"

All I’m doing is opening a list of all the open files, but then parsing out the noise like module files and the /proc and /dev directories, as well as OSSEC. You want to make sure not to readd OSSEC directories and files, it’ll just give you headaches.

Now use go through the list and add each of the log files using the utility shell in /bin. All that it is doing is adding this line to the ossec.conf file.

<localfile>
<log_format>apache</log_format>
<location>/var/log/httpd/access_log</location>
</localfile>

If you want to add them manually then just add it manually.

Remember when you’re done, you’ll want to restart OSSEC so that your settings reload:

# /var/ossec/bin/ossec-control restart

If you’re going to make more changes then wait until you’re done with the configuration file, then restart.

Default Configurations for Web Sites

Ok, so you added all the logs to OSSEC. AWESOME!! You’re not done, there are a few more things you want to do to make sure you have it ready to work for you.

Log aggregation and correlation is but one benefit of OSSEC. Another is its ability to do integrity checks in your environment. This means it checks to make sure that there aren’t any unauthorized changes to the environment (i.e., modified files, new files, etc…). So you want to make sure that you’re monitoring key directories and configure it to parse out unnecessary noise.

The file you will be working on is here:

# /var/ossec/etc/ossec.conf

The section you’re particularly interested in is the syscheck section. Default configuration looks like this:

<!-- Frequency that syscheck is executed - default to every 22 hours -->
<frequency>79200</frequency>

<!-- Directories to check (perform all possible verifications) -->
<directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
<directories check_all="yes">/bin,/sbin</directories>

You want to make some of these changes by default:

<!-- Frequency that syscheck is executed - set to every 4 hours -->
<frequency>14400</frequency>

<!-- Directories to check (perform all possible verifications) -->
<directories realtime="yes" check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
<directories realtime="yes" check_all="yes">/bin,/sbin</directories>
<directories realtime="yes" report_changes="yes" restrict=".htaccess|.php|.html|.js">[path to the root of your site]</directories>

<alert_new_files>yes</alert_new_files>
<scan_on_start>no</scan_on_start>
<auto_ignore>no</auto_ignore>

The frequency is in seconds, you can reduce that to whatever value you want. Do realize that depending on how you’re configured running syscheck could kill system resources. What you should notice is the addition of this line item:

<directories realtime="yes" report_changes="yes" restrict=".htaccess|.php|.html|.js">[path to the root of your site]</directories>

I’ve set it to notify of any changes and to monitor that directory in real time so that I get notifications. I also use restrict to isolate what to notify on. I don’t want the additional noise of images being added, or text files, things like that. I don’t care about those files because to run them .htaccess would have to be modified my Linux servers to run and recognize as PHP files.

There is one more change you’ll have to make. Seeing when new files are added is very important to me, which is why I added this line:

<alert_new_files>yes</alert_new_files>

But you actually have to take one more step to get it to work. By default OSSEC will not alert new files, that is intentional because of the amount of noise it can generate. By default it is configured in the following file/var/ossec/rules/ossec_rules.xml. You do not want to edit this file, editing will allow it to be overwritten when you update.

In that file you’ll find this rule:

<rule id=”554″ level=”0″>
<category>ossec</category>
<decoded_as>syscheck_new_entry</decoded_as>
<description>File added to the system.</description>
<group>syscheck,</group>
</rule>

We’ll get into why it’s not alerting another time, for now realize that its set not to alert by default with it’s severity level set to 0. What you will want to do is add the following into the following file which is ignored when you update: /var/ossec/rules/local_rules.xml. You’ll want to open that file and add the following:

<rule id=”554″ level=”7″ overwrite=”yes”>
<category>ossec</category>
<decoded_as>syscheck_new_entry</decoded_as>
<description>File added to the system.</description>
<group>syscheck</group>
</rule>

Because of the amount of noise that this could add you could get a bit crazy with it and set it to only monitor when files are added to specific directories, for instance you could do this:

<rule id=”554″ level=”7″ overwrite=”yes”>
<category>ossec</category>
<decoded_as>syscheck_new_entry</decoded_as>
<match>/var/www/html/[site directory]</match>
<description>File added to the [site] directory.</description>
<group>syscheck</group>
</rule>

With this configuration I will now see only when files are add to my sites directory. This is was the addition here:

<match>/var/www/html/[site directory]</match>

But Why Configure OSSEC?

I recently came across a page on a site in which they mentioned that logs provided very little value to website owners. I found myself dumbfounded by the statement, it was evident that the author of that post had very little insight into what logs were and how to leverage them. But it got me thinking, this is likely a very common perception amongst website owners, it actually motivated me to write this post.

You want to configure a HIDS like OSSEC because you want to know what’s going on with your website/webserver. OSSEC isn’t just aggregating the logs into one location, but it’s also correlating and analyzing. It has a set of rules, of which you are able to modify, designed to help you read and understand what the logs are saying. This becomes exceptionally important when you’re doing forensic work.

Think about the questions you ask yourself after a compromise:

  • What did the attacker do?
  • How did the attacker gain access?
  • What was the vector?
  • Do they still have control?
  • ….

Logs make this possible.

They paint the picture required to understand what’s going on. More importantly, actively monitoring logs allows you respond accordingly. Imagine you wake up and you’re experiencing a DDOS or DOS, or someone is beating up your server with malicious or vulnerability scans. OSSEC allows you to see this and respond accordingly.Another very common thing is brute force attempts against your SSHD / FTP ports. You want to see this, you want to react accordingly, unfortunately not enough people are doing that and they find themselves on the short end of the stick when they are compromised.

The point of this is to express the importance of actively monitoring your websites / web servers. With a few minor configurations, out of the box, the application is ready to be integrated into your security posture. There are a few other things you might want to do and I’ll get into that in other posts. If you find yourself in need of something like this you might want to consider Sucuri’s Realtime Enterprise Log Management (RELM) solution, it’s based on the OSSEC product but customized to fit what we see on a daily basis, in terms of attacks, and our clients.

Any questions let me know. Cheers.

  • http://www.mockel.se Falkowich

    Thanks for the great writeup!
    There were a lot of interesting stuff in here.

  • Rules

    Great job tony! It’s works perfectly. We’re waiting for the second part.

    Thank You!

    • perezbox

      What were you looking for specifically?

      Thanks

  • http://www.effortlesshr.com/blog/ Aaron @ Effortless HR Blog

    Perfect Tony.. thanks for the nice tutorial. Any plans on writing part 2?

    • perezbox

      Every time I sit to try and work on it, I get distracted with a few things.. hopefully soon.

      Was there a specific part you were looking for?

      Tony

      • http://www.effortlesshr.com/blog/ Aaron @ Effortless HR Blog

        Just more of your best practices and suggestions for setting up OSSEC or integrating it with systems such as Suricata for a more rounded approach to IDS.

  • Danny

    Very Helpful after suffering from a recent botnet attack! thanks a million

    • perezbox

      That’s awesome. Was it targeting your admin login? What was the botnet doing?

      Thanks

      • Danny

        I got a call early one morning saying all posts had disapeared as well as no one was able to login. we had a lovely post on the front page of our site saying hacked by IP man. Turns out all the posts and pages had been moved to trash but not deleted as well as all users removed and admin login email was diverted to another email address. That same day i removed the admin login and created new users. i then put a plugin on limiting login attempts and i was getting +/- 100 emails saying that an account was locked out due to failed login attempt from random IP’s. those IP’s were also trying variations of user names. Not fun but i consider myself very lucky