Installing Elastiflow flow monitoring solution

February 28, 2020 Leave a comment

Elastiflow is built upon the ELK stack so lets get that installed first. Now, we can install all of this on a single host (virtual or physical) for lab use, but for production use in high FPS environments, you will really want to scale the ELK stack horizontally to be able to process and search all your flow data. I am using RHEL for this guide so yum will be the package manager but it will work just as well on Ubuntu or other flavors with your package manager of choice. Elastiflow files and documentation can be found here:

I am going to show you how to install Logstash on its own server and Elasticsearch/Kibana together on another. This will allow horizontal scaling of Logstash. You can just as easily separate Kibana on its own server as well if you plan on building Elasticsearch as a multi node cluster. If this is just going in a lab or you have low flows per second (<1000 or so) then you can install all components on a single server. Below is a diagram of this deployment:

Installing pre-reqs:

The following commands we will run on all our servers so we have java and the ELK repo added to our package manager. We first need to install java.

sudo yum -y install java-openjdk-devel java-openjdk

Next add the ELK repo and key.

sudo rpm –import

cat <<EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo
name=Elasticsearch repository for 7.x packages

Installing Elasticsearch

We will start by installing Elasticsearch and setting it to start on boot.

sudo yum install -y elasticsearch
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service

Lets modify some configs and start Elasticsearch. We will first modify the elasticsearch.yml file:

sudo nano /etc/elasticsearch/elasticsearch.yml

Modify the below variables as shown. elastiflow ${HOSTNAME}
bootstrap.memory_lock: true (whatever the IP is of your server, set to “localhost” if running a single server for ELK stack) (whatever the IP of your server is, not needed if running a single server for ELK stack)

Add the following additional variables as well to the bottom of the file:

indices.query.bool.max_clause_count: 8192
search.max_buckets: 100000

Since we enabled bootstrap.memory_lock so that Elasticsearch memory is never swapped to disk, we need to modify the service file as well since that is where systemd requires system limits to be specified:

sudo nano /usr/lib/systemd/system/elasticsearch.service


Lets also set some java memory options. By default it is set to a min/max of 1G which is fine for a lab but if you are going into production, you want Elasticsearch to have a large heap to be able to store its internal data structures. Good rule of thumb is 50% of total system RAM but no more than 32GB which is the threshold JVM uses for compressed object pointers. Also set the min/max values the same so JVM doesn’t resize the heap which is a costly operation.

My Elasticsearch server has 8 GB of RAM so I will set JVM to use 4 GB of it.

sudo nano /etc/elasticsearch/jvm.options


We are now ready to start Elasticsearch:

sudo systemctl daemon-reload
sudo systemctl start elasticsearch.service

And we can verify Elasticsearch is running by sending a request to it locally from the command line:

[ed48@localhost ~]$ curl http://localhost:9200/_cluster/health?pretty
“cluster_name” : “elastiflow”,
“status” : “green”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 0,
“active_shards” : 0,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 0,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 100.0

Installing Kibana

We will be installing Kibana, which is the web front end, on the same server as Elasticsearch but you can install on a seperate server as well and just point it to Elasticsearch.

sudo yum install -y kibana
sudo systemctl daemon-reload
sudo systemctl enable kibana.service

Next we will edit the kibana.yml config file to tell Kibana what IP to listen to web requests on as well as tell Kibana where the Elasticsearch server is.

sudo nano /etc/kibana/kibana.yml “”
elasticsearch.hosts: ["http://localhost:9200"]

Now we can start Kibana and after a minute or so you should be able to point your browser to the IP of your Kibana install on port 5601 and you should see the home page for Kibana.

sudo systemctl start kibana.service

Installing Logstash

Now lets move over to the other server at to install Logstash. Remember, this server will be collecting flows as well as running that data through a custom pipeline to normalize all the data, add/merge fields etc so that it can be easily viewed in Kibana. Logstash is also the server that will most likely run out of resources first which is why we are installing it on its own server so that we can scale it horizontally. The sweet spot for this is around 4 CPU’s as adding more does not scale linearly. I am running 8 CPU’s with 8 GB of RAM (with 6 GB dedicated to JVM) and I currently get around 3000 – 3500 FPS without dropping packets due to the UDP input buffers being full. The UDP buffer in Linux is another reason why adding more CPU’s results in diminishing returns as Linux only assigns a single CPU core to handle UDP traffic. The rest of the CPU’s can be used to run Logstash workers to process the data which helps but there is a finite limit of how fast that single core can pull data off the network card.

If you haven’t yet, make sure Java is installed and the ELK key and repo are added to the package manager.

sudo yum -y install java-openjdk-devel java-openjdk

sudo rpm –import

cat <<EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo
name=Elasticsearch repository for 7.x packages

Now we can install and enable Logstash to run on boot.

sudo yum install -y logstash

sudo systemctl enable logstash.service

Finally we will install or update the required plugins for Logstash.

sudo /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow
sudo /usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow
sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-udp
sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-tcp
sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-dns
sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-geoip
sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-translate

Installing Elastiflow

Ok, now lets download all the files we will need for Elastiflow onto the Logstash server and get it configured. First lets create a temp directory to download the Elastiflow files. We will be installing release 4.0.0-beta1 You can see all official releases here:

sudo mkdir temp
cd temp
sudo wget

You should now have a file called v4.0.0-beta1.tar.gz in the temp directory. Go ahead and unzip it into the same directory.

sudo tar xvf v4.0.0-beta1.tar.gz

You should now have a directory named “elastiflow-4.0.0-beta1”

Before we go any further, I recommend you read over the install instructions for the release you are installing ( just to familiarize yourself with the process.

Now that we have most of the files we need, lets first copy the global variable file for Elastiflow to the logstash.service.d directory under systemd.

sudo cp -a elastiflow-4.0.0-beta1/logstash.service.d/. /etc/systemd/system/logstash.service.d/

Now lets copy the Elastiflow config files to the logstash directory.

sudo cp -a elastiflow-4.0.0-beta1/logstash/elastiflow/. /etc/logstash/elastiflow/

Now we need add the Elastiflow pipeline to the pipelines.yml file. This tells Logstash where all the config files are to run the custom Elastiflow processing pipeline. Make sure to comment out the default pipeline as it is not needed.

– elastiflow
path.config: “/etc/logstash/elastiflow/conf.d/*.conf”

Here is what the file should look like. The formatting matters in this file so include the spaces as shown below.

Now lets set some of the global variables in the elastiflow.conf file.

sudo nano /etc/systemd/system/logstash.service.d/elastiflow.conf

First we will turn on DNS name resolution processing by setting the first variable below to true and enter the IP of your DNS resolver for the second.

Next we will set ELASTIFLOW_ES_HOST to the IP of our Elasticsearch instance. If we were running the ELK stack on a single host we would accept the default of (localhost).

That is all that needs to be modified so save the file and we are ready to create the Logstash startup script which will take the variables we just edited and use them in the actual config files for the Elastiflow pipeline. If you make more changes to elastiflow.conf later, make sure to rerun this script and restart Logstash.

sudo /usr/share/logstash/bin/system-install

One last thing is to modify the jvm.options config file to give logstash enough RAM to operate efficiently. Since we are doing DNS lookups 4GB is a good value to use. Remember to keep this value around 50% of your system RAM.

sudo nano /etc/logstash/jvm.options

And now we can start logstash

sudo systemctl daemon-reload
sudo systemctl start logstash.service

Lastly, we need to import the index pattern and template Elasticsearch will use to index the flow data and the dashboards Kibana will use to display data to you. This is all located in a single file within the elastiflow-master directory and can be imported through the Kibana Gui. In Kibana go to the Management -> Saved objects page and import the elastiflow.kibana.7.5.x.ndjson file located at elastiflow-master/kibana/

You should now be able to start sending netflow/sflow/ipfix data on port 2055 to the IP of your Logstash server and in a few minutes you should see data when your click on the Dashbords icon on the left side of the page.

If you don’t see any data after a while, you can check the logs under /var/log/logstash/logstash-plain.log and see if there are any issues. If you are receiving v9 netflow, it may take a few minutes (depending on your template update rate) for Logstash to recieve a template to be able to decode the flows.

Categories: netflow Tags: , , ,

Netflow collection and visualization with Elastiflow

February 24, 2020 Leave a comment

For the past few years I was collecting netflow data on a single vm from several core routers on the network I manage. I used nfdump to collect the flow data and nfsen to visualize it. This vm worked fine as it didn’t do anything to the flows other than store the raw flow data and nfsen had bare bones RRD graphs and search functionality to visualize data within certain time spans.

The hardware my vm was running on was decommissioned and instead of migrating that vm to new hardware I started looking for a solution with more functionality. There are several solutions out there from Solarwinds, Plixer, ManageEngine, etc. etc. Considering my budget for this was zero, I looked for open source alternatives. Enter Elastiflow which is built using the ELK stack (Elasticsearch, Logstash and Kibana).

  • Logstash is the actual flow collector that runs the custom Elastiflow pipeline to process netflow, sflow or ipfix flow data into a standard format that can be visualized using a common dashboard.
  • Elasticsearch is a distributed search and analytics engine where flow data will be stored
  • Kibana is the web based front end to your data that will help you search and visualize it as well as manage Elasticsearch.

Elastiflow is developed and maintained by Rob Cowart and all files needed and install instructions are on his GitHub page: . It not only collects netflow but also sflow and ipfix flows as well.

This blog will be a quick overview of what you can do with Elastiflow. In my next blog, I will give a comprehensive guide to installing the ELK stack and Elastiflow. Suffice to say, for small networks or labs with low flows per second, a single host or vm will be enough to get your feet wet. If you have a larger network where the FPS is over 1500 or so, you will definitely want to scale out the ELK install (mostly Logstash) to be able to handle the processing of the incoming flows.

Let’s take a look at what we can see in Elastiflow. Here is the Top-N dashboard showing top talkers on the network.

You can drill down on any of the client or servers by hovering over one and clicking the spyglass icon. Any filters you set stay with you as you go to any of the other tabs as well such as Threats, Flows, AS Traffic etc.

Let’s keep this filter and see what AS’s this host is receiving traffic from.

This host was only downloading as there is no traffic showing in the outbound direction. We could also add one of the AS’s to the filter list and see what conversations our host was having with just hosts from a certain AS as well.

There are also other interesting visualizations such as this Sankey graph visualizing different flows between client and server.

Curious to see where all your traffic is coming from or going to? The GeoIP dashboard can help.

These are just a few of the ways you can visualize your flow data. Hopefully you have an idea of how Elastiflow can help you visualize and search your flow data making it easier to spot high utilization and analyze just who the participants are and what applications are in use. Next blog will deep dive into the installation details of getting this up and running.

Categories: netflow Tags: , , ,

IPv4 and IPv6 Netflow on Cat 6500 and Nexus 7k

January 29, 2015 2 comments

Netflow can be a handy tool to use on the network when you need to see information such as total number of flows/sec, bps per source/destination pair, and top talkers. It can also be used by service providers for usage based billing purposes or for help in mitigating DDoS attacks etc. etc. Here I will show you how to configure netflow on Cisco Nexus 7k and Catalyst 6500 based switches for both IPv4 and IPv6.

There are two main versions of netflow that I will talk about and what their differences are. The are V5 and V9 netflow. V4 has been around for a while and only supports IPv4. It is also a fixed format in that the info you can get from a V4 netflow record is all that you will ever get. There is no expandability built into V4 to allow it to support future data types.

Netflow V9 on the other hand supports both IPv4 and IPv6. It is also template based. This means you can customize the data being sent to your collectors. You can pick and chose which data fields you want to see. V9 is also extendable in that new data types can be added to the template as required in the future.

Lets get to the nuts and bolts of actually getting some netflow data off of our switches and into a collector so that we can search on it and visualize it if we so choose. For my examples below I will be using V9 as I like to collect both IPv4 and IPv6 statistics as I am running a dual stacked network.

Cat 6500 Config
Here is the basic config to get netflow v9 running and exporting flow data to your collector of choice:

mls netflow interface
mls flow ip interface-full
mls flow ipv6 interface-full
mls aging long 300
mls aging normal 120
mls nde sender

Here is what is going on in the first few lines of config. We are enabling per VLAN flow data to be collected, setting both the IPv4 and IPv6 flow masks to interface-full. Setting the mask to this exports all netflow data captured. There are several flow masks you can configure, each one sending less and less data. Setting the MLS aging timers is a good idea to get a more accurate view of flows as netflow data is not sent for a flow until it has timed out. For long lived flows, this is 32 mins by default. Setting the long timer to 300 means that long lived flows will be timed out in the netflow cache every 5 minutes thus allowing the collection of data for that flow to happen much sooner. I have set the normal timers to 120 just to age out inactive flows a little quicker than the default of 300 (5 minutes). Finally we enable the export of netflow data.

ip flow-export source Vlan100
ip flow-export version 9
ip flow-export destination 9995

Here we are specifying netflow updates should be sourced from vlan 100, they are to be version 9 records, due to the fact I am exporting both IPv4 and IPv6 flows and my netflow collector is at and listenign on port 9995.

Now we configure our interface to collect netflow data.

interface Vlan100 ip flow ingress

That is pretty much it. If you have tons of flows you could also set up sampled netflow. Where 1 out of every x packets are sampled for the netflow chache. This will improve CPU utilization on the switch and will be less flow data to have to store on the netflow collector as well.

Nexus 7k Config
Configuring netflow on a Nexus 7k is a bit different in that Nexus supports what is called Flexible Netflow. Here we configure netflow in a more modular approach. We configure Flow Records which state what data we want collected, Flow Exporters that specify where to send the flow data and what version of netflow to use and Flow Monitors which are placed on the interfaces where we wish to capture flow data. You can optionally specify a Flow Sampler which will do what its name implies. Capture sampled netflow data instead of looking at every packet.

Here I am configuring two separate flow records, one for IPv4 and one for IPv6 as you cannot have both IPv4 and IPv6 protocols in the same flow record. I am matching on specific keys and then collecting data based off of those keys.

flow record ipv4-netflow
match ipv4 source address
match ipv4 destination address
match ip protocol
match ip tos
match transport source-port
match transport destination-port
collect routing source as
collect routing destination as
collect routing next-hop address ipv4
collect transport tcp flags
collect counter bytes
collect counter packets
collect timestamp sys-uptime first
collect timestamp sys-uptime last

flow record ipv6-netflow
match ip protocol
match transport source-port
match transport destination-port
match ipv6 source address
match ipv6 destination address
collect routing source as
collect routing destination as
collect routing next-hop address ipv6
collect transport tcp flags
collect counter bytes
collect counter packets
collect timestamp sys-uptime first
collect timestamp sys-uptime last

Now lets create the flow exporter and flow monitors. Again you must create a seperate flow monitor for IPv4 and IPv6.

flow exporter flow-exporter
source loopback0
transport udp 9995 !This is the default value
version 9

flow monitor ipv4-netflow-mon
record ipv4-netflow
exporter flow-exporter

flow monitor ipv6-netflow-mon
record ipv6-netflow
exporter flow-exporter

Finally we configure the flow monitors on the interfaces we wish to obtain netflow data from:

interface Vlan300
ip flow monitor ipv4-netflow-mon input
ipv6 flow monitor ipv6-netflow-mon input

That’s it! You can verify you are collecting flow data by issuing the following commands. On a 6500 the command is:

show mls netflow ip – Shows the flow cache table.
show mls netflow ipv6 – Shows the flow cache table for IPv6.
show mls nde – Shows netflow export info.

For the Nexus the verification commands are:

show hardware flow ip – Shows IPv4 flow info
show hardware flow ipv6 – Shows IPv6 flow info
show flow exporter – Shows flow export info

Categories: IPv6, netflow, nexus

Find the port a host is connected to in a Fabricpath fabric

November 5, 2014 14 comments

Finding a host in a CE (Classic Ethernet) switched datacenter is a simple matter of showing the MAC address table on a switch and following the port that MAC is seen on until you end up at an access layer switch the host you are looking for is connected to. The command “show mac-address address aaaa.bbbb.cccc results in a nice easy to read output as such:

# sh mac address-table address 90e2.ba5b.3f90
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports/SWID.SSID.LID
* 800      90e2.ba5b.3f90    dynamic   30         F    F  Po1118


As you can see, we get the VLAN and what port the MAC shows up on. Now, when we do the same thing in a Fabricpath topology, we get a little different results. I am running this command from one of my spine switches in the Fabricpath topology. (I’m also using a different MAC as I’m on a different switch running Fabricpath):

# sh mac address-table address 000c.29af.42a7
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports/SWID.SSID.LID
  100      000c.29af.42a7    dynamic   0          F    F  1002.0.0

Now, instead of the port the MAC address is seen on we get three numbers separated by decimal points. Lets go over what these are. The numbers correspond to the SWID – Switch ID, SSID – Sub Switch ID and LID – Local ID. The SWID is a unique value assigned per switch what ISIS uses to route traffic within the fabric. The SSID is used when VPC+ is configured and is locally significant to each switch. More specifically, it is used to specify exactly what VPC+ port-channel to forward traffic on to reach its destination. Finally, there is the LID also know as the Port ID. This identifies the physical port that traffic was sourced from or is destined to and is also locally significant to each switch in the fabric.

The entry we are most concerned with is the SWID as this will be the destination switch traffic is being routed to and thus will be the switch our host we are trying to find is connected to. In order to find that switch, lets find out what port (or ports) the switch is accessible by. To do this we show the Fabricpath route for the SWID in question. I am running this command from one of my spine switches in the Fabricpath topology.

#sh fabricpath route switchid 1002
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

1/1002/0, number of next-hops: 2
        via Eth1/1, [115/10], 97 day/s 21:40:34, isis_fabricpath-default
        via Eth2/1, [115/10], 54 day/s 10:45:17, isis_fabricpath-default

Looks like SWID 1002 is accessable via Eth1/1 and Eth1/2. Usually, a switch ID will have only one next hop per spine switch but int this case, the switch ID 1002 is actually a vpc Fabricpath switch ID associated with a vpc pair of switches thus it is reachable from either switch in the pair. No matter as the next step is the same regardless. We need to figure out what switch that is and its management IP so we can continue on our way. I will do this by showing the CDP neighbor info of one of the ports listed.

# sh cdp neighbor int e1/1 detail
Device ID:5f-n6001-a(abcdef12345)
System Name: 5f-n6001-a

Interface address(es):
    IPv4 Address:
Platform: N6K-C6001-64P, Capabilities: Router Switch IGMP Filtering Supports-STP-Dispute
MTU: 1500
Physical Location: snmplocation
Mgmt address(es):
    IPv4 Address:

I cut some of the output but you see we have the management IP’s of the device so lets get on there and check the MAC address table to see if our host is connected to a port that is on this switch.

5f-n6001-a# sh mac address-table address 000c.29af.42a7
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link
   VLAN     MAC Address      Type      age     Secure NTFY   Ports/SWID.SSID.LID
* 100      000c.29af.42a7    dynamic   0          F    F   Po200

Looks like our host is connected to this switch in the fabric on port channel 200. We can now show what ports are bundled in the port channel to get the physical ports our host is connected to.

5f-n6001-a# sh port-channel summary int po200
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        S - Switched    R - Routed
        U - Up (port-channel)
        M - Not in use. Min-links not met
Group Port-       Type     Protocol  Member Ports
200   Po200(SU)   Eth      LACP      Eth1/25(P)


So in summary, we looked up the MAC of the host we were searching for which returned a SWID that is the source/destination of traffic to/from that MAC address. Looked up the Fabricpath route to that SWID which gave us the port(s) that the switch ID is reachable via. Then we obtained the management address of the switch on the other side of those ports and checked to see if the MAC we were looking for was directly connected to that switch. If so, then we get a port that the MAC is seen on. If not, then we would have gotten another SWID.SSID.LID and gone from there.

Categories: fabricpath, nexus

My journey into the land of IPv6: Addressing

There has been much talk about needing to move to IPv6 for a while now. With the last IPv4 allocations being handed out, reality is now sinking in that moving forward, IPv6 will be the way we start addressing our networks. Yes, IPv4 will undoubtedly be around for a while as carriers use CGN to work around the limited address space they have, but moving forward, IPv6 will see more use.

I have had the good fortune to be able to turn up and test IPv6 on a new core network that sits alongside the current network. In this network I am running MPLS L3 VPNs so the actual core is IPv4 only due to LDP not having support for IPv6 currently but that has no impact on running IPv6 within each VRF. Running IPv6 within a L3 VPN will be for another blog post though. In the next few posts, I will talk about how we can easily get IPv6 up and running. Assuming the infrastructure supports it of course!  First stop, addressing.

Once you have recieved your prefix from ARIN (most likely a /48) what in the world are we supposed to do with it?  The most visible difference between IPv6 and IPv4 is the larger addresses: IPv4 addresses are 32 bits long and written in ‘dotted decimal’ format  i.e. IPv6 uses 128 bit addresses, written in 8 groups of 16 bit blocks (words)  written in hex and separated by colons – i.e. 2620:0000:691:1001:0000:0000.000a:0001.

When writing addresses, within each block of 4 hex digits the leading zeroes may be omitted. In addition, once per address (and no more) a contiguous set of one or more blocks which are all 0000 may be replaced by a double colon. So the above address can be rewritten as: 2620:0:691:1001::a:1.

Netmasks are written in the same mask length format as IPv4, with /length after the address. For example, a /126 subnet has 126 bits of network address, leaving 2 bits for host addresses . The bottom address being the network anycast address, although this is not required. There is no top address for a directed broadcast).

When subnetting, always remember to subnet on nibble (every 4 bits ) boundaries so as not to have a subnet end in weird places,  i.e. having a subnet end on 2620:0:691:7:: and the host portion begin on 2620:0:691:8::

Also, unless you have special circumstances, always use /64’s as end host networks.  Not using /64 as your network will break things such as Neighbor discovery, privacy extensions etc. Click has a good post on why using /64 networks is the way to go.

There are several different addressing methods for IPv6 as well:

Stateless Autoconfiguration (SLAAC)

In this scenario, a host will address itself based off of a network prefix learned from an advertising router via periodic router advertisements (RA’s). The host will append its MAC address (with FF:FE inserted in the middle) and this will form the 128 bit host address. This could be a security concern as the hosts MAC address is easily obtained from this address so most operating systems replace the MAC with a randomized unique identifier.

Lets talk about the RA for a second. If you only configure SLAAC on your router you aren’t manually specifying a DHCP pool with default gateway etc as with IPv4 for end hosts.  So how does your PC know how to send traffic off net? The RA specifies several different parameters to end hosts including:

  • IP prefix (or multiple prefixes)
  • Flags (such as used DHCP to obtain DNS info)
  • Default gateway


In this scenario, no config flags are set in the RA messages.  What about DNS?  This is either handled manually on each end host or you can do SLAAC with DHCPv6.

SLAAC with Stateless DHCPv6

In this scenario, the Other-Config-Flag is set in the RA to tell the host to use SLAAC to get an address and use DHCPv6 to obtain other parameters such as DNS.  The DHCPv6 server is stateless in that it does not keep track of what IPv6 address each end host has.

Stateful DHCPv6

In this scenario the Managed-Config-Flag is set in the RA and tells the end host to use DHCPv6 only to obtain an address, DNS etc.

Static Addressing

This option is really only useful for server farms etc where you don’t want to have these addresses randomly assigned.  This can be accomplished using the no-autoconfig flag enabled.  This sets the A bit to zero in RA’s so that autoconfig is not used by the end host to assign itself an address.

In my next post I’ll go over the configurations to make it work within a SLAAC and Stateless DHCPv6 environment.

Categories: IPv6

Some QoS Notes on Cisco Nexus 7k

February 4, 2014 3 comments

Now that I have moved a significant portion of my enterprise network to our new core based on Nexus 7k switches, I need to start thinking about how to implement QoS as I am in a healthcare environment and some traffic MUST make it to its destination regardless of congestion. Currently, I don’t see any congestion on our network but traffic is always increasing and a good QoS policy will keep the business running.

Here are some important things to remember:
– QoS is on by default and cannot be turned off.
– There are default class-maps and policy-maps configured and applied to all interfaces by default.
– default policy-map cannot be modified.
– Ports trust CoS by default and use it for inbound and outbound queue selection.
– The 7k assigns EGRESS traffic to queues based on CoS even if the egress interface does not carry CoS in the frame (i.e. it’s a L3 interface).
– Interface based QoS takes precedence over VLAN based
– VLAN based QoS policy is configured in VLAN Database, no SVI is required.

There are four default rules to remember for CoS/DSCP and queue assignment. I copied these verbatim from the BRKDCT-3346 slide deck presented by Lukas Krattiger at CLUS 2013:

For Bridged Traffic:
If CoS and DSCP is present:
– CoS is used for ingress queue selection
– CoS is preserved
– DSCP is unmodified
– CoS is used for egress queue selection

If only DSCP is present:
– No CoS is treated as CoS 0 on ingress
– CoS 0 is used for iungress and egress queue selection
– DSCP is unmodified

For Routed Traffic
If CoS and DSCP is present:
– Cos is used for ingress queue selection
– DSCP is preserved and rewrites CoS (top 3 bits)
– Re-written CoS is used for egress queue selection

If only DSCP is present:
– No CoS gets treated as CoS 0 in ingress
– DSCP is preserved and rewrites CoS (top 3 bits)
– Re-written CoS is used for egress queue selection

Default Policy
Nexus 7k’s do have a default QoS policy in place that is applied to all interfaces in all VDC’s which assigns COS 5-7 on the ingress side to queue 1 and everything else to the default queue. On the egress side COS 5-7 is assigned to the egress priority queue and everything else to the default queue as shown below:

sh class-map type queuing Type queuing class-maps ======================== class-map type queuing match-any 8q2t-in-q1 Description: Classifier for ingress queue 1 of type 8q2t match cos 5-7 class-map type queuing match-any 8q2t-in-q-default Description: Classifier for ingress default queue of type 8q2t match cos 0-4 class-map type queuing match-any 1p7q4t-out-pq1 Description: Classifier for egress priority queue of type 1p7q4t match cos 5-7 class-map type queuing match-any 1p7q4t-out-q-default Description: Classifier for egress default queue of type 1p7q4t match cos 0-4

The above queue structure is from a M2 10gig module which has 8 ingress queues with 2 tail drop thresholds per queue and on the egress side there is 1 priorty queue and 7 queues with 4 tail drop thresholds each. Each module is different though. For example, a M148GS-11L module which has 48 1gig ports has a 2q4t ingress queue structure and a 1p3q4t egress queue structure. The class maps for it would resemble the one above except with fewer queues to map to.

So back to the defaults, there is also a default policy map applied to all ports using the above COS to queue mappings:

sh policy-map type queuing Type queuing policy-maps ======================== policy-map type queuing default-in-policy class type queuing in-q1 queue-limit percent 50 bandwidth percent 80 class type queuing in-q-default queue-limit percent 50 bandwidth percent 20 policy-map type queuing default-out-policy class type queuing out-pq1 priority level 1 queue-limit percent 16 class type queuing out-q2 queue-limit percent 1 class type queuing out-q3 queue-limit percent 1 class type queuing out-q-default queue-limit percent 82 bandwidth remaining percent 25

You can see on ingress, 50% of the queues are allocated to queue 1 and the other half to the default inbound queue. On the outbound side, pq1 is defined as the priority queue given a percentage of the queue space as are queue 3,4 and the default queue. Notice the names don’t match up with the actual class-map names. These are default class-map names that match on any queue type, for example:

class-map type queuing match-any in-q1 Description: This is a default class-map name. Prefix the module queue type to this class name to see the actual cos2 map (e.g. To print in-q1, use 2q4t-in-q1 or 8q2t-in-q1)

Where do I configure these things?
– Class maps for queue selection are done in the admin/default VDC and apply to all interfaces across all VDC’s.
– Policy maps are configured per VDC and are assigned on a per-port or per-vlan basis.

If you configure a custom class map, be sure to assign queue-limits within your policy map for your queues. If you do not, no queue space will be allocated for that queue and you will start tail dropping traffic assigned to that queue immediately.

I will post more on this topic as I flesh out my QoS policy a little more and we can get into creating custom class-maps and policy-maps.

Categories: nexus, qos

Debugging on Cisco Nexus

August 13, 2013 Leave a comment

I had an issue where I had a need to do some PIM debugging recently on the Nexus platform in an MPLS environment and there are some nice features that make it pretty handy to use.

First off I did what I usually do and forgot to specify the VRF I wanted to actually debug PIM in so received no log messages. I looked for a bit to see where I could specify the VRF but there wasn’t any option under the debug command:

n7k-dis2# debug ip pim ? assert PIM Assert packet events bidir PIM Bidir DF packet events data-register PIM data register packet events ha PIM system HA events hello PIM Hello packet events internal PIM system internal events join-prune PIM Join-Prune packet events mvpn MVPN related events null-register PIM Null Register packet events packet PIM header debugs policy PIM policy information rp PIM RP related events vpc VPC related events vrf PIM VRF creation/deletion events

That VRF option is not the one to specify the VRF, it debugs just what the description says it does.

Then I found you can specify debug-filters! Here is where you can specify which VRF to actually apply the debugging command to, along with alot of other filtering options:

n7k-dis2#debug-filter ip pim ? group Debug information for a particular Group interface Debug information for a particular Interface vrf Debug information for a particular VRF n7k-dis2# debug-filter ip pim vrf user

As you can see, you can filter your debugs to a specific multicast group and/or interface as well. Now that we have applied the debug filter to the user VRF, now we can turn on the specific PIM debugging we want to see:

n7k-dis2# debug ip pim join-prune 2013 Aug 13 13:04:18.022567 pim: [7034] (user-base) No (, route exists, not to us 2013 Aug 13 13:04:18.022615 pim: [7034] (user-base) No (, route exists, not to us 2013 Aug 13 13:04:18.022627 pim: [7034] (user-base) ----- 2013 Aug 13 13:04:18.084726 pim: [7034] (user-base) Received Join-Prune from on mti5, length: 34, MTU: 1376, ht: 420 2013 Aug 13 13:04:18.084758 pim: [7034] (user-base) Upstream address:, group count: 1, holdtime: 420 2013 Aug 13 13:04:18.084770 pim: [7034] (user-base) Group:, join-list count: 1, prune-list count: 0 .....

Now this produced a lot of output so you could have applied another debug-filter to filter on a specific address, packet direction, interface, etc. It is pretty easy to extract exactly what you want to see in the debug output.

You can also send the output of a debug to a file like so (I named the file “test”):

n7k-dis2# debug logfile test size ? 4096-4194304 Enter the logfile size in bytes

Categories: cisco, nexus

Simplified QoS on 3850 with MQC

August 9, 2013 21 comments

Configuring QoS has been much improved in Cisco’s new 3850 line of switches thanks to its implementation of MQC (Modular QoS Cli) configuration instead of the old “mls qos” commands from the 3750 and 3560 lines of switches.  Here I will show just how easy it is to generate a simple QoS config to handle voice and video traffic.

First we define what traffic we would like to work with.  In this case it is voice traffic marked as dscp ef and video traffic marked as dscp af41.

class-map match-any VIDEO match ip dscp af41 class-map match-any VOICE match ip dscp ef

Easy enough. Now lets configure the policy maps to actually do something with this traffic. I will configure two separate policy maps. one for user facing ports and one for the uplinks to my distribution layer.

I would like to prioritize both voice and video traffic over everything else but I want voice to be prioritized over everything. The 3850 actually has 2 priority queues that enable me to do this, a level 1 and level 2 priority queue.

policy-map UPLINK class VOICE priority level 1 percent 10 police cir percent 10 conform-action transmit exceed-action drop class VIDEO priority level 2 percent 20 police cir percent 20 conform-action transmit exceed-action drop class class-default bandwidth remaining percent 100 policy-map USER class VOICE priority level 1 percent 1 police cir percent 1 conform-action transmit exceed-action drop class VIDEO priority level 2 percent 5 police cir percent 5 conform-action transmit exceed-action drop class class-default bandwidth remaining percent 100

Above I have separate policy maps based on if it’s a user or uplink port. For user ports, voice traffic is put in priority queue 1 and guaranteed 1% or 10Mbps of throughput. I also capped it at 10Mbps with the police command to prevent this queue from hogging bandwidth. Same process is followed for voice. It is placed in priority queue 2 and it is guaranteed 50Mbps throughput. On the uplinks, the only change is the percentages of guaranteed bandwidth has been increased.All other traffic gets 100% of the remaining bandwidth.

All that is left is to apply these policy maps to some interfaces.

interface GigabitEthernet1/0/1 switchport access vlan 302 switchport mode access switchport voice vlan 402 trust device cisco-phone spanning-tree portfast service-policy output USER interface GigabitEthernet1/1/1 description --- UPLINK --- switchport trunk allowed vlan 302,402 switchport mode trunk service-policy output UPLINK ip dhcp snooping trust

Done! We can even get some meaningful stats now as well. I haven’t pushed any traffic through this switch yet but you get the idea.

3850#sh policy-map int gig 1/1/1 GigabitEthernet1/1/1 Service-policy output: UPLINK queue stats for all priority classes: Queueing priority level 1 (total drops) 0 (bytes output) 0 queue stats for all priority classes: Queueing priority level 2 (total drops) 0 (bytes output) 0 Class-map: VOICE (match-any) Match: ip dscp ef (46) Priority: 10% (100000 kbps), burst bytes 2500000, Priority Level: 1 police: cir 10 % cir 100000000 bps, bc 3125000 bytes conformed 0 bytes; actions: transmit exceeded 0 bytes; actions: drop conformed 0000 bps, exceed 0000 bps Class-map: VIDEO (match-any) Match: ip dscp af41 (34) Priority: 20% (200000 kbps), burst bytes 5000000, Priority Level: 2 police: cir 20 % cir 200000000 bps, bc 6250000 bytes conformed 0 bytes; actions: transmit exceeded 0 bytes; actions: drop conformed 0000 bps, exceed 0000 bps Class-map: class-default (match-any) Match: any Queueing (total drops) 0 (bytes output) 0 bandwidth remaining 100%

As you can see pretty painless QoS implementation. Of course you can get much more complicated if needed with nested policy-maps, table maps etc. Here is the IOS-XE QoS documentation if you want to delve further into QoS IOS XE QoS Config Guide”

Categories: 3850, cisco, qos

Cisco 3850 Fails to Boot

August 8, 2013 3 comments

While working on some QoS configuration, I hit a strange issue with some Cisco 3850 switches. I saved my config and shut down the switch only to have it boot into rommon the next day when powered back on

@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@done. Memory Test Pass! Base ethernet MAC Address: 20:37:06:cf:d4:00 Interface GE 0 link down***ERROR: PHY link is down Initializing Flash... flashfs[7]: 0 files, 1 directories flashfs[7]: 0 orphaned files, 0 orphaned directories flashfs[7]: Total bytes: 6784000 flashfs[7]: Bytes used: 1024 flashfs[7]: Bytes available: 6782976 flashfs[7]: flashfs fsck took 1 seconds....done Initializing Flash. : no such device : no such device Error loading "" Interrupt within 5 seconds to abort boot process. Boot process failed... The system is unable to boot automatically. The BOOT environment variable needs to be set to a bootable image. switch:

Hmmm, not what I expected. I looked in the directory to try and find the .bin file to boot the switch byt was presented with some .pkg files instead.

switch: dir flash: Directory of flash:/ 7745 drwx 4096 . 2 drwx 4096 .. 7746 -rwx 2097152 nvram_config 7747 -rwx 74410468 cat3k_caa-base.SPA.03.02.00SE.pkg 7748 -rwx 2773680 cat3k_caa-drivers.SPA.03.02.00.SE.pkg 7749 -rwx 32478044 cat3k_caa-infra.SPA.03.02.00SE.pkg 7750 -rwx 30393116 cat3k_caa-iosd-universalk9.SPA.150-1.EX.pkg 7751 -rwx 18313952 cat3k_caa-platform.SPA.03.02.00.SE.pkg 7752 -rwx 63402700 cat3k_caa-wcm.SPA. 7753 -rwx 1218 packages.conf 7754 -rwx 156 express_setup.debug 7755 -rw- 684 vlan.dat

Hmm, no .bin file here. After some digging, you actually boot with the packages.conf file which then invokes the necessary .pkg files and off you go.

switch: boot flash:packages.conf Getting rest of image Reading full image into memory....done Reading full base package into memory...: done = 74410468 .....

This behavior is actually a documented bug in the 3.2.0SE code. The bug ID is CSCue76684. The workaround for this is to manually specify the packages.conf file in the boot statement in the config.

boot system switch all flash:packages.conf

This bug is fixed in 3.2.1SE. Hopefully this saves someone some time when deploying these switches.

Categories: 3850, bug, cisco

Multicast, MPLS and MDT’s, oh my!

Here, as promised, is my first post of real substance!  I’ve been building a new core network for the health system I work at and one of the requirements was to enable multicast across the wired as well as wireless network.  Seems fairly straightforward at first, enable PIM on all interfaces that could potentially carry multicast traffic, set up your RP’s (auto-rp, bsr, or manually) and off you go.

Well, the new core network is MPLS based, and has a set of firewalls filtering traffic between VRF’s so there is a bit more to getting multicast to work which I will walk through here.  I also have a mix of Cisco Nexus 7k’s in my core and 6500’s in my distribution layers and there is a caveat that needs to be addressed which I will go over as well.

The network is a pair of Nexus 7k’s as the core of the MPLS network with 6509’s hung off of them acting as building distribution PE’s.  Routing between VRF’s is handled by an “Interchange” router that is northbound of the core switches and an ASA in transparent mode, sits in between the core and interchange routers handing firewall duties for each VRF:

network overview
Read more…

Categories: cisco Tags: ,