Archive for the ‘nexus’ Category

IPv4 and IPv6 Netflow on Cat 6500 and Nexus 7k

January 29, 2015 2 comments

Netflow can be a handy tool to use on the network when you need to see information such as total number of flows/sec, bps per source/destination pair, and top talkers. It can also be used by service providers for usage based billing purposes or for help in mitigating DDoS attacks etc. etc. Here I will show you how to configure netflow on Cisco Nexus 7k and Catalyst 6500 based switches for both IPv4 and IPv6.

There are two main versions of netflow that I will talk about and what their differences are. The are V4 and V9 netflow. V4 has been around for a while and only supports IPv4. It is also a fixed format in that the info you can get from a V4 netflow record is all that you will ever get. There is no expandability built into V4 to allow it to support future data types.

Netflow V9 on the other hand supports both IPv4 and IPv6. It is also template based. This means you can customize the data being sent to your collectors. You can pick and chose which data fields you want to see. V9 is also extendable in that new data types can be added to the template as required in the future.

Lets get to the nuts and bolts of actually getting some netflow data off of our switches and into a collector so that we can search on it and visualize it if we so choose. For my examples below I will be using V9 as I like to collect both IPv4 and IPv6 statistics as I am running a dual stacked network.

Cat 6500 Config
Here is the basic config to get netflow v9 running and exporting flow data to your collector of choice:

mls netflow interface mls flow ip interface-full mls flow ipv6 interface-full mls aging long 300 mls aging normal 120 mls nde sender

Here is what is going on in the first few lines of config. We are enabling per VLAN flow data to be collected, setting both the IPv4 and IPv6 flow masks to interface-full. Setting the mask to this exports all netflow data captured. There are several flow masks you can configure, each one sending less and less data. Setting the MLS aging timers is a good idea to get a more accurate view of flows as netflow data is not sent for a flow until it has timed out. For long lived flows, this is 32 mins by default. Setting the long timer to 300 means that long lived flows will be timed out in the netflow cache every 5 minutes thus allowing the collection of data for that flow to happen much sooner. I have set the normal timers to 120 just to age out inactive flows a little quicker than the default of 300 (5 minutes). Finally we enable the export of netflow data.

ip flow-export source Vlan100 ip flow-export version 9 ip flow-export destination 9995

Here we are specifying netflow updates should be sourced from vlan 100, they are to be version 9 records, due to the fact I am exporting both IPv4 and IPv6 flows and my netflow collector is at and listenign on port 9995.

Now we configure our interface to collect netflow data.

interface Vlan100 ip flow ingress

That is pretty much it. If you have tons of flows you could also set up sampled netflow. Where 1 out of every x packets are sampled for the netflow chache. This will improve CPU utilization on the switch and will be less flow data to have to store on the netflow collector as well.
Nexus 7k Config
Configuring netflow on a Nexus 7k is a bit different in that Nexus supports what is called Flexible Netflow. Here we configure netflow in a more modular approach. We configure Flow Records which state what data we want collected, Flow Exporters that specify where to send the flow data and what version of netflow to use and Flow Monitors which are placed on the interfaces where we wish to capture flow data. You can optionally specify a Flow Sampler which will do what its name implies. Capture sampled netflow data instead of looking at every packet.

Here I am configuring two separate flow records, one for IPv4 and one for IPv6 as you cannot have both IPv4 and IPv6 protocols in the same flow record. I am matching on specific keys and then collecting data based off of those keys.

flow record ipv4-netflow match ipv4 source address match ipv4 destination address match ip protocol match ip tos match transport source-port match transport destination-port collect routing source as collect routing destination as collect routing next-hop address ipv4 collect transport tcp flags collect counter bytes collect counter packets collect timestamp sys-uptime first collect timestamp sys-uptime last flow record ipv6-netflow match ip protocol match transport source-port match transport destination-port match ipv6 source address match ipv6 destination address collect routing source as collect routing destination as collect routing next-hop address ipv6 collect transport tcp flags collect counter bytes collect counter packets collect timestamp sys-uptime first collect timestamp sys-uptime last

Now lets create the flow exporter and flow monitors. Again you must create a seperate flow monitor for IPv4 and IPv6.

flow exporter flow-exporter destination source loopback0 transport udp 9995 !This is the default value version 9 flow monitor ipv4-netflow-mon record ipv4-netflow exporter flow-exporter flow monitor ipv6-netflow-mon record ipv6-netflow exporter flow-exporter

Finally we configure the flow monitors on the interfaces we wish to obtain netflow data from:

interface Vlan300 ip flow monitor ipv4-netflow-mon input ipv6 flow monitor ipv6-netflow-mon input

That’s it! You can verify you are collecting flow data by issuing the following commands. On a 6500 the command is:

show mls netflow ip – Shows the flow cache table.
show mls netflow ipv6 – Shows the flow cache table for IPv6.
show mls nde – Shows netflow export info.

For the Nexus the verification commands are:

show hardware flow ip – Shows IPv4 flow info
show hardware flow ipv6 – Shows IPv6 flow info
show flow exporter – Shows flow export info

Categories: IPv6, netflow, nexus

Find the port a host is connected to in a Fabricpath fabric

November 5, 2014 10 comments

Finding a host in a CE (Classic Ethernet) switched datacenter is a simple matter of showing the MAC address table on a switch and following the port that MAC is seen on until you end up at an access layer switch the host you are looking for is connected to. The command “show mac-address address aaaa.bbbb.cccc results in a nice easy to read output as such:

# sh mac address-table address 90e2.ba5b.3f90
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports/SWID.SSID.LID
* 800      90e2.ba5b.3f90    dynamic   30         F    F  Po1118


As you can see, we get the VLAN and what port the MAC shows up on. Now, when we do the same thing in a Fabricpath topology, we get a little different results. I am running this command from one of my spine switches in the Fabricpath topology. (I’m also using a different MAC as I’m on a different switch running Fabricpath):

# sh mac address-table address 000c.29af.42a7
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False
   VLAN     MAC Address      Type      age     Secure NTFY Ports/SWID.SSID.LID
  100      000c.29af.42a7    dynamic   0          F    F  1002.0.0

Now, instead of the port the MAC address is seen on we get three numbers separated by decimal points. Lets go over what these are. The numbers correspond to the SWID – Switch ID, SSID – Sub Switch ID and LID – Local ID. The SWID is a unique value assigned per switch what ISIS uses to route traffic within the fabric. The SSID is used when VPC+ is configured and is locally significant to each switch. More specifically, it is used to specify exactly what VPC+ port-channel to forward traffic on to reach its destination. Finally, there is the LID also know as the Port ID. This identifies the physical port that traffic was sourced from or is destined to and is also locally significant to each switch in the fabric.

The entry we are most concerned with is the SWID as this will be the destination switch traffic is being routed to and thus will be the switch our host we are trying to find is connected to. In order to find that switch, lets find out what port (or ports) the switch is accessible by. To do this we show the Fabricpath route for the SWID in question. I am running this command from one of my spine switches in the Fabricpath topology.

#sh fabricpath route switchid 1002
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

FabricPath Unicast Route Table for Topology-Default

1/1002/0, number of next-hops: 2
        via Eth1/1, [115/10], 97 day/s 21:40:34, isis_fabricpath-default
        via Eth2/1, [115/10], 54 day/s 10:45:17, isis_fabricpath-default

Looks like SWID 1002 is accessable via Eth1/1 and Eth1/2. Usually, a switch ID will have only one next hop per spine switch but int this case, the switch ID 1002 is actually a vpc Fabricpath switch ID associated with a vpc pair of switches thus it is reachable from either switch in the pair. No matter as the next step is the same regardless. We need to figure out what switch that is and its management IP so we can continue on our way. I will do this by showing the CDP neighbor info of one of the ports listed.

# sh cdp neighbor int e1/1 detail
Device ID:5f-n6001-a(abcdef12345)
System Name: 5f-n6001-a

Interface address(es):
    IPv4 Address:
Platform: N6K-C6001-64P, Capabilities: Router Switch IGMP Filtering Supports-STP-Dispute
MTU: 1500
Physical Location: snmplocation
Mgmt address(es):
    IPv4 Address:

I cut some of the output but you see we have the management IP’s of the device so lets get on there and check the MAC address table to see if our host is connected to a port that is on this switch.

5f-n6001-a# sh mac address-table address 000c.29af.42a7
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link
   VLAN     MAC Address      Type      age     Secure NTFY   Ports/SWID.SSID.LID
* 100      000c.29af.42a7    dynamic   0          F    F   Po200

Looks like our host is connected to this switch in the fabric on port channel 200. We can now show what ports are bundled in the port channel to get the physical ports our host is connected to.

5f-n6001-a# sh port-channel summary int po200
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        S - Switched    R - Routed
        U - Up (port-channel)
        M - Not in use. Min-links not met
Group Port-       Type     Protocol  Member Ports
200   Po200(SU)   Eth      LACP      Eth1/25(P)


So in summary, we looked up the MAC of the host we were searching for which returned a SWID that is the source/destination of traffic to/from that MAC address. Looked up the Fabricpath route to that SWID which gave us the port(s) that the switch ID is reachable via. Then we obtained the management address of the switch on the other side of those ports and checked to see if the MAC we were looking for was directly connected to that switch. If so, then we get a port that the MAC is seen on. If not, then we would have gotten another SWID.SSID.LID and gone from there.

Categories: fabricpath, nexus

Some QoS Notes on Cisco Nexus 7k

February 4, 2014 3 comments

Now that I have moved a significant portion of my enterprise network to our new core based on Nexus 7k switches, I need to start thinking about how to implement QoS as I am in a healthcare environment and some traffic MUST make it to its destination regardless of congestion. Currently, I don’t see any congestion on our network but traffic is always increasing and a good QoS policy will keep the business running.

Here are some important things to remember:
– QoS is on by default and cannot be turned off.
– There are default class-maps and policy-maps configured and applied to all interfaces by default.
– default policy-map cannot be modified.
– Ports trust CoS by default and use it for inbound and outbound queue selection.
– The 7k assigns EGRESS traffic to queues based on CoS even if the egress interface does not carry CoS in the frame (i.e. it’s a L3 interface).
– Interface based QoS takes precedence over VLAN based
– VLAN based QoS policy is configured in VLAN Database, no SVI is required.

There are four default rules to remember for CoS/DSCP and queue assignment. I copied these verbatim from the BRKDCT-3346 slide deck presented by Lukas Krattiger at CLUS 2013:

For Bridged Traffic:
If CoS and DSCP is present:
– CoS is used for ingress queue selection
– CoS is preserved
– DSCP is unmodified
– CoS is used for egress queue selection

If only DSCP is present:
– No CoS is treated as CoS 0 on ingress
– CoS 0 is used for iungress and egress queue selection
– DSCP is unmodified

For Routed Traffic
If CoS and DSCP is present:
– Cos is used for ingress queue selection
– DSCP is preserved and rewrites CoS (top 3 bits)
– Re-written CoS is used for egress queue selection

If only DSCP is present:
– No CoS gets treated as CoS 0 in ingress
– DSCP is preserved and rewrites CoS (top 3 bits)
– Re-written CoS is used for egress queue selection

Default Policy
Nexus 7k’s do have a default QoS policy in place that is applied to all interfaces in all VDC’s which assigns COS 5-7 on the ingress side to queue 1 and everything else to the default queue. On the egress side COS 5-7 is assigned to the egress priority queue and everything else to the default queue as shown below:

sh class-map type queuing Type queuing class-maps ======================== class-map type queuing match-any 8q2t-in-q1 Description: Classifier for ingress queue 1 of type 8q2t match cos 5-7 class-map type queuing match-any 8q2t-in-q-default Description: Classifier for ingress default queue of type 8q2t match cos 0-4 class-map type queuing match-any 1p7q4t-out-pq1 Description: Classifier for egress priority queue of type 1p7q4t match cos 5-7 class-map type queuing match-any 1p7q4t-out-q-default Description: Classifier for egress default queue of type 1p7q4t match cos 0-4

The above queue structure is from a M2 10gig module which has 8 ingress queues with 2 tail drop thresholds per queue and on the egress side there is 1 priorty queue and 7 queues with 4 tail drop thresholds each. Each module is different though. For example, a M148GS-11L module which has 48 1gig ports has a 2q4t ingress queue structure and a 1p3q4t egress queue structure. The class maps for it would resemble the one above except with fewer queues to map to.

So back to the defaults, there is also a default policy map applied to all ports using the above COS to queue mappings:

sh policy-map type queuing Type queuing policy-maps ======================== policy-map type queuing default-in-policy class type queuing in-q1 queue-limit percent 50 bandwidth percent 80 class type queuing in-q-default queue-limit percent 50 bandwidth percent 20 policy-map type queuing default-out-policy class type queuing out-pq1 priority level 1 queue-limit percent 16 class type queuing out-q2 queue-limit percent 1 class type queuing out-q3 queue-limit percent 1 class type queuing out-q-default queue-limit percent 82 bandwidth remaining percent 25

You can see on ingress, 50% of the queues are allocated to queue 1 and the other half to the default inbound queue. On the outbound side, pq1 is defined as the priority queue given a percentage of the queue space as are queue 3,4 and the default queue. Notice the names don’t match up with the actual class-map names. These are default class-map names that match on any queue type, for example:

class-map type queuing match-any in-q1 Description: This is a default class-map name. Prefix the module queue type to this class name to see the actual cos2 map (e.g. To print in-q1, use 2q4t-in-q1 or 8q2t-in-q1)

Where do I configure these things?
– Class maps for queue selection are done in the admin/default VDC and apply to all interfaces across all VDC’s.
– Policy maps are configured per VDC and are assigned on a per-port or per-vlan basis.

If you configure a custom class map, be sure to assign queue-limits within your policy map for your queues. If you do not, no queue space will be allocated for that queue and you will start tail dropping traffic assigned to that queue immediately.

I will post more on this topic as I flesh out my QoS policy a little more and we can get into creating custom class-maps and policy-maps.

Categories: nexus, qos

Debugging on Cisco Nexus

August 13, 2013 Leave a comment

I had an issue where I had a need to do some PIM debugging recently on the Nexus platform in an MPLS environment and there are some nice features that make it pretty handy to use.

First off I did what I usually do and forgot to specify the VRF I wanted to actually debug PIM in so received no log messages. I looked for a bit to see where I could specify the VRF but there wasn’t any option under the debug command:

n7k-dis2# debug ip pim ? assert PIM Assert packet events bidir PIM Bidir DF packet events data-register PIM data register packet events ha PIM system HA events hello PIM Hello packet events internal PIM system internal events join-prune PIM Join-Prune packet events mvpn MVPN related events null-register PIM Null Register packet events packet PIM header debugs policy PIM policy information rp PIM RP related events vpc VPC related events vrf PIM VRF creation/deletion events

That VRF option is not the one to specify the VRF, it debugs just what the description says it does.

Then I found you can specify debug-filters! Here is where you can specify which VRF to actually apply the debugging command to, along with alot of other filtering options:

n7k-dis2#debug-filter ip pim ? group Debug information for a particular Group interface Debug information for a particular Interface vrf Debug information for a particular VRF n7k-dis2# debug-filter ip pim vrf user

As you can see, you can filter your debugs to a specific multicast group and/or interface as well. Now that we have applied the debug filter to the user VRF, now we can turn on the specific PIM debugging we want to see:

n7k-dis2# debug ip pim join-prune 2013 Aug 13 13:04:18.022567 pim: [7034] (user-base) No (, route exists, not to us 2013 Aug 13 13:04:18.022615 pim: [7034] (user-base) No (, route exists, not to us 2013 Aug 13 13:04:18.022627 pim: [7034] (user-base) ----- 2013 Aug 13 13:04:18.084726 pim: [7034] (user-base) Received Join-Prune from on mti5, length: 34, MTU: 1376, ht: 420 2013 Aug 13 13:04:18.084758 pim: [7034] (user-base) Upstream address:, group count: 1, holdtime: 420 2013 Aug 13 13:04:18.084770 pim: [7034] (user-base) Group:, join-list count: 1, prune-list count: 0 .....

Now this produced a lot of output so you could have applied another debug-filter to filter on a specific address, packet direction, interface, etc. It is pretty easy to extract exactly what you want to see in the debug output.

You can also send the output of a debug to a file like so (I named the file “test”):

n7k-dis2# debug logfile test size ? 4096-4194304 Enter the logfile size in bytes

Categories: cisco, nexus