Home > netflow > Netflow collection and visualization with Elastiflow

Netflow collection and visualization with Elastiflow

For the past few years I was collecting netflow data on a single vm from several core routers on the network I manage. I used nfdump to collect the flow data and nfsen to visualize it. This vm worked fine as it didn’t do anything to the flows other than store the raw flow data and nfsen had bare bones RRD graphs and search functionality to visualize data within certain time spans.

The hardware my vm was running on was decommissioned and instead of migrating that vm to new hardware I started looking for a solution with more functionality. There are several solutions out there from Solarwinds, Plixer, ManageEngine, etc. etc. Considering my budget for this was zero, I looked for open source alternatives. Enter Elastiflow which is built using the ELK stack (Elasticsearch, Logstash and Kibana).

  • Logstash is the actual flow collector that runs the custom Elastiflow pipeline to process netflow, sflow or ipfix flow data into a standard format that can be visualized using a common dashboard.
  • Elasticsearch is a distributed search and analytics engine where flow data will be stored
  • Kibana is the web based front end to your data that will help you search and visualize it as well as manage Elasticsearch.

Elastiflow is developed and maintained by Rob Cowart and all files needed and install instructions are on his GitHub page: https://github.com/robcowart/elastiflow . It not only collects netflow but also sflow and ipfix flows as well.

This blog will be a quick overview of what you can do with Elastiflow. In my next blog, I will give a comprehensive guide to installing the ELK stack and Elastiflow. Suffice to say, for small networks or labs with low flows per second, a single host or vm will be enough to get your feet wet. If you have a larger network where the FPS is over 1500 or so, you will definitely want to scale out the ELK install (mostly Logstash) to be able to handle the processing of the incoming flows.

Let’s take a look at what we can see in Elastiflow. Here is the Top-N dashboard showing top talkers on the network.

You can drill down on any of the client or servers by hovering over one and clicking the spyglass icon. Any filters you set stay with you as you go to any of the other tabs as well such as Threats, Flows, AS Traffic etc.

Let’s keep this filter and see what AS’s this host is receiving traffic from.

This host was only downloading as there is no traffic showing in the outbound direction. We could also add one of the AS’s to the filter list and see what conversations our host was having with just hosts from a certain AS as well.

There are also other interesting visualizations such as this Sankey graph visualizing different flows between client and server.

Curious to see where all your traffic is coming from or going to? The GeoIP dashboard can help.

These are just a few of the ways you can visualize your flow data. Hopefully you have an idea of how Elastiflow can help you visualize and search your flow data making it easier to spot high utilization and analyze just who the participants are and what applications are in use. Next blog will deep dive into the installation details of getting this up and running.

Categories: netflow Tags: , , ,
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: