Author: regit

Stamus Networks is proud to announce the availability of the first technology preview of Amsterdam.

Amsterdam is a ready-to-use Suricata IDS/NSM based system running on Docker. It features the same components as SELKS our live and installable Suricata distibution. So by running Amsterdam you get:

  • Suricata: latest version of the IDS/NSM engine
  • Elasticsearch: the powerful search engine
  • Logstash: the data pipeline application injecting Suricata events in the database
  • Kibana: Version 4 of the famous dashboard interface
  • Scirius: Stamus Networks’ Suricata ruleset management interface

Each component runs in a separate container and Amsterdam is using the official images for ELK stack to provide you an always up-to-date experience. Amsterdam is available under GPLv3 license.

Using Amsterdam is really simple. Once installed, you need to setup a directory that will contain data and configuration. For example to create and use a directory named ams-wlan0 and sniff the wlan0 interface on host, one can run once:

amsterdam -d ams-wlan0 -i wlan0 setup

You can then start Amsterdam by running:

amsterdam -d ams-wlan0 start

Scirius running in Amsterdam

Once all containers are built and started, you can point your browser to http://localhost:8000 to get access to the management interface.

It is possible to run multiple instances of Amsterdam on the same system by using different data directories. Each data directory contains the configuration files of the components so you can easily tune your installation.

More information and source code available on Github.

Stamus Networks team is proud to announce the availability of Scirius 1.1. This new release brings a ton of new features:

  • Easier enabling/disabling of rules and categories
  • Delete events from Elasticsearch corresponding to all alerts of a given rule
  • Compatibility with Kibana 4
  • Add a graphic presenting Logstash insertion speed
  • Graphics with Suricata performance indicators
  • Greatly improved source addition
  • Preliminary support for Sourcefire ruleset
  • Validity checking of ruleset, imported sources and rules

One of the main new feature is the validity checking system. Scirius is now able to test whether a rule, a source or a ruleset will be valid for Suricata. This is not a simulated test as Scirius is using the configuration testing capability of Suricata to get the information.

One of the updated page is the the rule details page. It does now contain the validity of this rule respectively to the existing rulesets:
Validity check

In our example, the rule is invalid for first ruleset because the source containing the lua script is not active in the ruleset. In the second ruleset, the rules is not valid because the Suricata on the system is not supporting the not-yet-official(as of the moment of this writing) TLS extension for lua scripting. As you can see, the validity checking system is much more than a simple parsing and gives you a real view on the validity of rules on your own system. This system will work with any Suricata but it will give accurate information if the system is hosting the latest Suricata. This is due to the fact that Scirius is parsing the JSON console output of Suricata which has been added recently to the Suricata git master. Our team has packaged the latest Suricata and made it available in Stamus Networks repository to the users to give them the best of this new feature.

This testing system has been used to revamp the Source addition. Adding a source to an existing ruleset was complicated and involving multiple steps. Now, user can select to which rulesets a new source should be added when creating the source. So there is now basically one step:
Source addition form
The validity checking system will warn the user about a possible problem during the source creation:
Source addition page
As some rulesets may contain invalid signatures, user has the capability to choose whether or not he/she wants to ignore the detected problems.

The second main change in Scirius is its ability to graph some performance indicators. It is using the logstash metrics capability to get statistics on the insertion speed. This is a good indicator of the load of your logging system.

Elasticsearch and Logstash information

The second set of graphics is building performance indicators for Suricata. The Suricata page of Scirius is now able to graph:

  • Capture stats
  • Memory usage
  • Problem indicators

If capture stats is showing the usual drop and accept, the two others tabs are more interesting as they are showing the memory consumption of Suricata subsystem succh as TCP, Flow, DNS and HTTP protocols:
Suricata memory usage
The third tab is showing some Suricata indicators that can help to diagnose problem. All these indicators are extracted from Suricata statistics output in JSON format that will be part of Suricata 2.1.

Happy SELKS users can simply run apt-get update && apt-get dist-upgrade to get new version. Others can get it from Github.

Suricata EVE JSON format is becoming the de-facto standard for this IDS. All type of events are now exported to this format. The JSON format allows a nice handling of data in external tool like Elasticsearch or even DOM. The output is readable by human but as an event/record can contain a lot of data it can be difficult to do a by-eye analysis when looking at a file. The following screenshot give you an idea of the possible output:

Tailing EVE

Using standard unix tools like grep on the EVE JSON file is not the perfect idea. For example if you want to extract a field to get some statistics you may want to try using grep, cut or awk but you may find it painful. And it is worthed to mention here that JSON fields are not ordered.

Here to the rescue comes the jq utility. jq is a tool dedicated to the transformation/parsing of a JSON entry. It is Debian packaged, so a simple apt-get install jq is enough for the install.

Some jq examples

The most basic usage is to colorize the entry. To do that, just do something like

$ tail -n100 eve.json| jq '.'

The output is done the pretty way:
JQ displaying an event
To get a one line per event output, just add the -c flag to the command:
One line

To extract a single field from the JSON events, one can do:

$ jq '.src_ip' eve.json
"58.218.211.155"
"58.218.211.155"
"58.218.211.155"

The point to remember is that the point in .src_ip is a place holder for the current entry.

By default when a field is not present null is displayed in the output. To fix that, it is possible to filter the event to only get the one we are interested in. This is done via the select keyword. For instance to select the SSH events and extract the information about the client part one can do:

$ tail eve.json | jq -c 'select(.event_type == "ssh")|.ssh.client'
{"proto_version":"2.0","software_version":"PUTTY"}
{"proto_version":"2.0","software_version":"PUTTY"}

Far more things can be done with jq. Good starting points are the jq manual and wiki.

Stamus Networks is proud to announce the availability of Scirius 1.0. This is the first stable release of our web interface for Suricata ruleset management. It is providing an efficient way to manage and update the ruleset.

Scirius is displaying some graphics that will help you to get an idea of the activity of your Suricata probe and easily select rules that may be noisy and need to be deactivated:
Screenshot from 2015-04-20 22:07:08

But the main focus is on ruleset handling. You can for example follow the change of a signature source:

Screenshot from 2015-04-20 22:05:36

Scirius is not meant to replace a good dashboard interface, so it is providing a link to Kibana dashboard:
Screenshot from 2015-04-20 22:38:48

Scirius also allows you to search inside its database to find the elements you are looking for:

Screenshot from 2015-04-20 22:06:06

Scirius is able to handle multiple sources. So you can mixed local rules and rules download from outside sources such as Emerging Threats or SSLBL from abuse.ch:
Screenshot from 2015-04-20 22:05:52

Scirius is fetching activity information from Elasticsearch and it is now even able to display some interesting information about the state of your cluster
Screenshot from 2015-04-20 22:06:51

Scirius 1.0 is part of SELKS our live and installable Suricata NSM/IDS distribution. Happy SELKS can upgrade to scirius 1.0 via a simple apt-get update && apt-get upgrade. Other users can simply grab the release from Github.

The development will now focus on getting Scirius ready to handle IPS. So the changes will mostly be about rules transformation and the main features of Scirius should stay alike.

Stamus Networks is proud to announce the availability of the third release candidate of Scirius 1.0. This new release features minor bugfixes and an improvement.

Scirius 1.0-rc3 brings a new set of graphs that improve the visualization of the probe activity. The following video demonstrate the usage of this feature:



To sum up, the suricata page has now one zoomable sunburst graph who split the rules by classification:
sunburst-crop

There is also an alternate display via hierarchic circles:
circles-crop

The selection between the different type of graphs is done via the local settings:
Screenshot from 2015-04-03 09:07:31

You can download Scirius 1.0-rc3 from Github.

0

Some words about PRscript

PRSCript is a script that run a series of builds and tests on a given branch. It was reserved to some developers so they can check the quality of their work before submission. The test builds are run on Suricata buildbot which is composed of some different dedicated hardware system. buildbot is an open-source framework for automating software build, test, and release processes. In the case of Suricata instance, it is set up to run various builds, unit tests as well as functional tests (such as pevma’s regression script).

The fact that this script was reserved to some users was a limitation as many contributors are not registered as Suricata buildbot users. As well, the fact that the code has to be public was not convenient as you could have to expose code before it is ready (with shameful TODO inside). Another point is that you were not able to customize your build. For instance, if you were introducing a new library as dependency it was not possible to test it before a global modification of the buildbot.

PRscript with docker support

To get over these limitations, Victor Julien and I have discussed on using Docker to allow developers to simply run a Suricata dedicated buildbot. As you may/should already know Docker is an open platform for distributed applications for developers and sysadmins. It allows you to quickly install install, manage and run containers. In our case, the idea was to start a pre-configured buildbot container using your local git as reference code. This way you can simply start test builds on your private code without even needing.

So, I have worked on this Docker based buildbot installation dedicated to Suricata and it has been merged in Suricata mainstream by Victor Julien.

It is now possible to use the prscript locally via Docker. Installation had been made simple so you should just have a few commands to run before being ready.

The buildbot will run various builds (gcc and clang, different builds options) and run suricata against some pcaps to check against possible crash.

Screenshot from 2015-04-07 16:22:19

Installation

Prerequisites

You need to have docker and python-docker installed on your system. Optionally you can install pynotify on your system to get desktop notification capability. On recent Debian based distribution you can use:

sudo apt-get install docker python-docker python-notify

Create the container

This operation has only to be done once. From the root of
Suricata sources, run:

sudo qa/prscript.py -C

It will take some times as the download is several hundred Mo. The result will be a docker container named ‘suri-buildbot’.

Using the buildbot

Start the buildbot

When you need to use the buildbot, you can start it from the command line:

sudo qa/prscript.py -s

You can check it is running via:

sudo docker ps

And you can connect to the buildbot web interface via http://localhost:8010

Start a build

Once the buildbot is active, you can start a build:

qa/prscript.py -d -l YOUR_BRANCH

This will start a build of the local branch YOUR_BRANCH without requiring any connectivity.

To get warned of the result of the builds via a desktop notification:

qa/prscript.py -d -l YOUR_BRANCH -n

Stop the buildbot

When you don’t need the buildbot anymore, you can stop it from the command line

 sudo qa/prscript.py -S

For further details, check Suricata docker QA page on OISF redmine.

Advanced usage

Build customisation

Buildbot will make suricata read all the pcap files available in qa/docker/pcaps/. So you can use this directory to add your own test pcaps.

Buildbot configuration is stored inside your suricata sources. It is the file qa/docker/buildbot.cfg. So, you can change the Buildbot configuration by editing this file. Then stop and start the docker container to get the new version used. This can be for example used when you need to add a flag to the configure command to activate a new feature.

What is great about this docker way of doing things is that it solves easily some complex points. For instance, if the buildbot configuration were coming from the Docker image then it will not be possible to easily edit it. Furthermore developer will loose any changes made in case of image upgrade. Also, the configure flags used by the buildbot will always be related to the current state of the code. So there will be no issue with running builds even if you are working on some older code as your buildbot configuration will be synchronized first.

Connect via ssh

The docker instance can be accessed via SSH using the admin account (password being ‘admin’ too). To get the port to use for ssh run the following command to get the port to use:

$ sudo docker port suri-buildbot
22/tcp -> 0.0.0.0:49156
8010/tcp -> 0.0.0.0:8010

and then connect:

ssh admin@localhost -p 49156

This can be used to install new dependencies inside the container. For instance if you are introducing a new library in Suricata, you may have to install the library in the docker instance.

Customizing the Docker image

On Docker side, the build recipes is available from GitHub. Feel free to modify it or propose updates and fixes.

Stamus Networks is proud to announce the availability of the second release candidate of Scirius 1.0. This new release features bugfixes and improvements.

On the bugfix side, the main one is a fix in the display of graph that could fail on fresh install due to a problem in the Elasticsearch request
Screenshot from 2015-02-12 18:10:59

The system status has been improved to feature a warning phase on disk and memory usage.

The only new feature is the System Settings menu:

Screenshot from 2015-02-12 18:11:15

For now, it allows the administrator to setup two things:

  • HTTP proxy parameters: if activated and setup it will allow scirius to fetch rules updates using the specified proxy
  • Elasticsearch usage: some people are using scirius without Elasticsearch so displaying empty graph is not interesting for them. By unchecking elasticsearch, the graphs based on elasticsearch information are not displayed anymore.

Screenshot from 2015-02-12 18:05:54

You can download Scirius 1.0-rc2 from Github. SELKS users can upgrade to this release by doing apt-get update && apt-get dist-upgrade.

Stamus Networks is proud to announce the availability of version 1.0-rc1 of Scirius, our web interface for Suricata ruleset management. This new release is first 1.0 release candidate. You can download it from Github download page.

It features a lot of bug fixes and improvements over the previous (beta) release. Among the new features, Scirius is now displaying a system status in the left sidebar.

system-status-scirius

It displays :

  • Status of the Elasticsearch cluster (in SELKS and if setup).
  • Status of Suricata.
  • Memory usage: alerting if swap is used.
  • Disk status: alerting if disk is filled in.

An other important improvement is the support of flowbit, scirius now disables all rules sharing a flowbit if one is disabled. This helps preventing entering is some weird state where an incomplete set of rules could trigger a lot of events.

Last but not least, the copyright has been updated with a new year inside. Happy new year 2015 from Stamus Networks team.

SELKS user can upgrade to Scirius 1.0-rc1 via apt-get update && apt-get dist-upgrade.

0

Introduction

Elasticsearch and Kibana are wonderful tools but as all tools you need to know their limits. This article will try to explain how you must be careful when reading data and explain how to improve this situation by using an existing Elastisearch feature.

The Problem

All did start with the analysis of an SSH bruteforce attack coming from Vietnam. This attack was interesting because of the announced SSH client “PuTTY-Local: Mar 19 2005 07:19:17” which really looks like a correct PuTTY software version when most attack don’t spoof their software version and reveal what they are using.

The Kibana dashboard was showing all information needed to get a good idea of attacks:

Screenshot from 2014-12-03 16:36:47

But when looking at less used and most used passwords, there was something really strange:

Screenshot from 2014-12-02 08:59:41

For example, webmaster is seen in the two panels with different values which is not logical.

By adding a filter on this value, the result was a bit surprising:

Screenshot from 2014-12-02 09:08:59

When looking at the detail of events, it was obvious this last result was correct. This SSH bruteforce has tried 10 different logins and has always used the same dictionary of 23 passwords.

To a solution

So the panels with top passwords and less seen passwords are displaying incorrect data in some circumstances. They have been setup in Kibana using the terms type.

This corresponds in Elasticsearch to a facets query. Here’s is the content of the query with the filter removed for readability:

{
 "facets": {
    "terms": {
      "terms": {
        "field": "password.raw",
        "size": 10,
        "order": "count",
        "exclude": []
      },
  }
}

So we have a simple request and it is not returning the correct data. The explanation of this problem can be found in Elasticsearch Issue #1305.

Adrien Grand is explaining that a algorithm returning possibly inaccurate values has been chosen to avoid a too memory intensive and network intensive search. The per-default algorithm is mainly wrong when they are more different values than searched values.

We can confirm that behavior in our case by asking for 30 values (on the 23 different passwords we have):

Screenshot from 2014-12-02 09:30:30

The result is correct this time.

If we continue reading Adrien Grand comment on the issue, we see that a shard_size parameter has been introduced to be able improve the algorithm accuracy.

So we can use this parameters to improve the accuracy of the queries. Patching this in Kibana is trivial:

diff --git a/src/vendor/elasticjs/elastic.js b/src/vendor/elasticjs/elastic.js
index ba9c8ee..8daa72a 100644
--- a/src/vendor/elasticjs/elastic.js
+++ b/src/vendor/elasticjs/elastic.js
@@ -3085,6 +3085,7 @@
         }
 
         facet[name].terms.size = facetSize;
+        facet[name].terms.shard_size = 10 * facetSize;
         return this;
       },

Here we just choose a far larger shard_size than the number of elements asked in the query. We could also have used the special value 0 (or Integer.MAX_VALUE) for shard_size to get perfect result. But in our test setup, Elasticsearch is failing to honor the request with this parameter. And furthermore, the result was already correct:

Screenshot from 2014-12-02 10:10:10

This patch has been proposed to Elasticsearch as PR 2106.

That was a small patch but this fixed our dashboard as the value in the terms panels are now correct:

Screenshot from 2014-12-03 16:44:57

Stamus Networks is proud to announce the availability of version 1.0-beta1 of Scirius, our web interface for Suricata ruleset management. This new release is a huge step toward 1.0 release as it contains a lot of new features and improvements. You can download it from Github download page.

The most visible update is the new design of the interface. It has been been completely changed thanks to Bootstrap CSS framework.

Screenshot from 2014-11-12 10:45:55

But the first change for user is that authentication and user management is now by default. Scirius is now multi user and features three level of permissions from read-only to superuser.

Another new feature is the display of graphics in some page. They are using Elasticsearch data. For example, the next screenshot is showing detail of a rule. A graph has been added to show the activity for that specific rules:

Screenshot from 2014-11-12 10:46:46

The interface is now more responsive as asynchronous requests are used to interact with Elasticsearch. This guarantee a responsive interface even if your Elasticsearch is slow.

SELKS user can upgrade to Scirius 1.0-beta1 via apt-get update && apt-get dist-upgrade. Please note that the default user/password on SELKS is selks-user/selks-user. Do not forget to change it after first login.