My brief experience with Quad Sup VSS (VS-SUP720-10G)

This is a REALLY short post, about an amazing topic that with the assistance of Google, could easily eat up a couple days worth of reading. I just wanted to talk a little bit about the behavior of your 6500, running VS-SUP720-10G with Quad Sup VSS. The quick and dirty on VSS, for those who don't know, is it essentially stacks two independent 6500 chassis into a single logical unit. So the question that came up, when each of the two chassis have redundant supervisors... how the hell does that work?

Configuration wise, not much is different from VSS with only two sups in the mix. Aside from the fact you now have four 10G interfaces (two from each sup) in your EtherChannel for the VSL. Now the "weird" behavior comes in with what roles each sup takes, and what happens with failover. Essentially your sups break down into one of two distinctions. ICA or ICS, in-chassis active/in-chassis standby. Also note that on VS-SUP720, your ICS goes into RPR (cold standby) and you only get SSO between the two ICAs from each chassis.

Give that a moment to sink in lol. So that means if your active supervisor fails... the hot standby supervisor FROM THE OTHER CHASSIS, goes active. Alright, a little odd maybe. So if chassis A's sup fails, chassis B's sup will go active SSO. What happens to chassis A then? Or it's RPR sup for that matter? Well, bad news bears, that whole chassis reloads. It may sound bad, but if you build this right, you should have MCE (multichassis EtherChannel) built to everything off that 6500. So, theoretically the failure would be almost entirely unnoticed.

QoS Task

If you're looking for a task that will annoy you, this is it! Alright, currently you have a 10Mb connection to your upstream provider, you've been charged with implementing some basic QOS to prioritize voice traffic (marked DSCP EF). You're told voice should be given priority of 10%, and the QOS technique you're to use should allow for future growth with additional class based queues. No sweat right?

While you're out on vacation, one of your co-workers brings in a second WAN connection, and connects it to the same subnet as your primary WAN (10Mbps). This connection is only 512Kbps, better yet your boss wants the same QOS policy above to adapt in the event of a failure so that voice gets 50%. Like any good Engineer, you tell him it's impossible and this is a terrible solution. Now you're job's on the line, smooth move. 

Your mission, should you choose to keep your imaginary job:

- Create a QOS policy giving 10% to DSCP EF when the primary WAN link is up, and 50% when it's down.

- QOS policy needs to automatically adjust for the change in bandwidth in the event the primary WAN link goes down. Your solution should also include adjusting for the bandwidth change when the primary comes back online.

- Do not add any new IPs

- Do not create any additional interfaces

- Bonus avoid using the bandwidth statement

Topology 

Solution

The video got cut short, but you'll see the solution =]

MPLS Tools and Tricks - Part 1, show mpls forwarding-table (aka your best friend)

First off, I know it's been a while since I've posted anything. Between failing the CCIE (again), moving and changing jobs... I've been busy. Back off. That being said, I'm getting the itch again, and it's time to start studying, I don't care if it takes 20 tries over 3 different blueprints I'm going to get my number in R&S. So to get things rolling, I thought I'd start with my favorite topic - MPLS.

Let's get some lingo straight first, that way you impress all your friends with new acronyms.

LSR - Label Switch Router. This guy's life is label switching, LSRs make up your MPLS backbone.
LER - Label Edge Router. Your provider edge (PE), and as the name implies the edge of your MPLS network.
LSP - Label Switch Path. The MPLS path for a particular source and destination.

Now that we know who all the players are, lets take a look at our topology

You'll notice we're dealing with more "P" routers than usual, but having a few extra LSRs should help visual the MPLS process. First off, before setting up any VRFs, or doing any l2 vpns let's take a look at probably the single most important command for troubleshooting MPLS.  That's right kids, I'm talking about show mpls forwarding-table. As a sample we'll look at the LFIB (Label Forwarding Information Base) on P1.

P1#show mpls forwarding-table

Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop

Label  Label or VC   or Tunnel Id      Switched      interface

16     Pop Label     11.11.11.11/32     0             Et1/0      172.16.111.11

17     Pop Label     172.16.112.0/24   0             Et1/1      172.16.12.2

         Pop Label     172.16.112.0/24   0             Et1/0      172.16.111.11

18     Pop Label     3.3.3.3/32             0             Et1/2      172.16.13.3

19     Pop Label     2.2.2.2/32             0             Et1/1      172.16.12.2

20     Pop Label     172.16.223.0/24   0             Et1/2      172.16.13.3

21     Pop Label     172.16.34.0/24     0             Et1/2      172.16.13.3

         Pop Label     172.16.34.0/24     0             Et1/3      172.16.14.4

22     Pop Label     172.16.24.0/24     0             Et1/1      172.16.12.2

         Pop Label     172.16.24.0/24     0             Et1/3      172.16.14.4

23     Pop Label     172.16.23.0/24     0             Et1/1      172.16.12.2

         Pop Label     172.16.23.0/24     0             Et1/2      172.16.13.3

24     26                 22.22.22.22/32     0             Et1/2      172.16.13.3

         26                 22.22.22.22/32     0             Et1/3      172.16.14.4

25     Pop Label     4.4.4.4/32              0             Et1/3      172.16.14.4

26     Pop Label     172.16.224.0/24    0             Et1/3      172.16.14.4

Alright... what are looking at right?? I'll get to that, first let's examine why this command is so darn important, simple. By issuing it on our MPLS enabled routers, we can establish our LSP between two given endpoints. After all, MPLS is multiprotocol label switching, so when we're troubleshooting it we should always give the labels the majority of our attention. Now, what are we looking at? The first column is our local labels, generated by the router we're on. The second column, when all is working well, will have either an out going label received (in this case) via LDP or it will show "Pop Label". Without digging too deep, Pop Label comes from penultimate hop popping, really fun to say by the way. This is the process whereby the second to the LSR in an LSP will pop the label off an mpls packet, this is an efficiency mechanism. 

Now back to show mpls forwarding-table. Let's say there is a VPN built between PE1 and PE2, routing tables on the CEs look correct, import export statements look good while you're troubleshooting. However, CE1 and not ping CE2. So spoiler alert, I played proctor and jacked up P3 with some control plane policing, more on that later. Further more, since the LFIB is built off the FIB I adjusted the ospf cost on P4's link to PE2 so that P3 was the preferred path. VPNs are built between the PEs, and for a VPN to work there has to be a fully functional LSP between their peering address (loopbacks in our example). So starting on PE1, we'll verify we have labels to get to 22.22.22.22 (PE2's loopback).

PE1#show mpls forwarding-table 22.22.22.22

Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop

Label  Label or VC   or Tunnel Id      Switched      interface

26     24            22.22.22.22/32    0             Et1/0      172.16.111.1

         26            22.22.22.22/32    0             Et1/1      172.16.112.2

So far so good. We have egress labels of 24 and 26 going to P1 and P2 respectfully. Next hop, I'm going to look at P1, but you could go to either.

P1#show mpls forwarding-table 22.22.22.22

Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop

Label  Label or VC   or Tunnel Id      Switched      interface

24     No Label      22.22.22.22/32    1523          Et1/2      172.16.13.3

Trouble in paradise, P1 shows that it's forwarding the packet out with no label towards P3, and has no other path. Let's see what P2 shows.

P2#show mpls forwarding-table 22.22.22.22

Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop

Label  Label or VC   or Tunnel Id      Switched      interface

26     No Label      22.22.22.22/32    126           Et1/3      172.16.23.3

Same thing, at this point we can jump onto P3 and start diagnosing. First up, look at the labels.

P3#show mpls forwarding-table 22.22.22.22

Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop

Label  Label or VC   or Tunnel Id      Switched      interface

26     No Label      22.22.22.22/32    1218          Et1/0      172.16.223.22

Verify it has any labels

P3#show mpls forwarding-table

Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop

Label  Label or VC   or Tunnel Id      Switched      interface

16     No Label      11.11.11.11/32    0             Et1/2      172.16.13.1

         No Label      11.11.11.11/32    0             Et1/3      172.16.23.2

17     No Label      2.2.2.2/32            0             Et1/3      172.16.23.2

18     No Label      1.1.1.1/32            0             Et1/2      172.16.13.1

19     No Label      172.16.24.0/24    0             Et1/3      172.16.23.2

         No Label      172.16.24.0/24    0             Et1/1      172.16.34.4

20     No Label      172.16.14.0/24    0             Et1/2      172.16.13.1

         No Label      172.16.14.0/24    0             Et1/1      172.16.34.4

21     No Label      172.16.112.0/24  0             Et1/3      172.16.23.2

22     No Label      172.16.111.0/24  0             Et1/2      172.16.13.1

23     No Label      172.16.12.0/24    0             Et1/2      172.16.13.1

         No Label      172.16.12.0/24    0             Et1/3      172.16.23.2

24     No Label      4.4.4.4/32            0             Et1/1      172.16.34.4

25     No Label      172.16.224.0/24  0             Et1/0      172.16.223.22

26     No Label      22.22.22.22/32    0             Et1/0      172.16.223.22

Well that's not good! Why doesn't P3 have any labels?? I know I've told you the problem already, but humor me. So what next? We verify the router has any mpls enabled interfaces, then if we have any neighbors.

P3#show mpls interfaces

Interface              IP            Tunnel   BGP Static Operational

Ethernet1/0          Yes (ldp)     No       No  No     Yes

Ethernet1/1          Yes (ldp)     No       No  No     Yes

Ethernet1/2          Yes (ldp)     No       No  No     Yes

Ethernet1/3          Yes (ldp)     No       No  No     Yes

P3#show mpls ldp neighbor

P3#

So from this we see that all the expected interfaces are in fact MPLS enabled and using LDP. However we have no LDP. Since LDP is our current environment's mechanism for exchanging labels, this would explain why we're having issues. This point requires you to know a little more about LDP, LDP sends hello messages out 224.0.0.2 on udp port 646. However, the LDP session is actually built between the LDP router-ids (loopback interfaces for us, this is also the LDP/TDP default behavior) on TCP port 646. That being said, you need to check a couple things on R3

Can we ping our neighbors? Can we ping their LDP RIDs (loopbacks here) from our RID? We'll pick on R1 for these tests

P3#ping 172.16.13.1

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.13.1, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 16/20/28 ms

P3#ping 1.1.1.1 source lo0

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:

Packet sent with a source address of 3.3.3.3

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 4/21/60 ms

Alright that looks good, so now back our TCP and UDP ports... do we have any access-lists configured?

P3#sh ip access

Extended IP access list 100

    10 permit ospf any any (949 matches)

    20 permit tcp any any eq bgp

    30 permit tcp any eq bgp any

    40 permit udp any any eq 646 (314 matches)

Extended IP access list 101

    10 permit icmp any any (10 matches)

Extended IP access list 103

    10 permit ip any any (2229 matches)

We do! Great... now where are they applied??

P3#show run | inc 101

 match access-group 101

access-list 101 permit icmp any any

"match access-group" sounds like a class-map, let's take a look.

!

class-map match-all ICMP

 match access-group 101

class-map match-all ALL

 match access-group 103

class-map match-any ROUTING

 match access-group 100

!

end

P3#show run policy-map

Building configuration...

Current configuration : 141 bytes

!

policy-map CoPP

 class ROUTING

 class ICMP

   police rate 100 pps

     exceed-action drop

 class ALL

   drop

 class class-default

!

end

And finally, where is it applied?

P3#show policy-map interface  <-- Nothing here

P3#show policy-map control-plane <-- Nailed it!

 Control Plane

  Service-policy input: CoPP

    Class-map: ROUTING (match-any)

     ... out ommitted

So now we can modify CoPP to allow for tcp communication on 646. After doing that LDP will come back up, and CE1 will be able to ping CE2. Mystery solved.

Running Labs Again!

I'm back at it! I'd be lying if I didn't say I was considering quitting. CCIE is tough man, and failing it is soul crushing (especially failing it four times). However, the record's 19 attempts... so I figure if anything I can at least try to top that! I have a post talking about 'following the labels' that I'll be publishing immediately after this one... I'm not SUPER happy with it, but I hope it helps someone out there.

So here's an off the wall topic... window strategy on the lab. That's right, it's probably the most nontechnical thing you have to worry about after you sit down at your workstation. However, it's extremely important. Aside from practice, and strong knowledge you need two things to pass the lab - Speed and Accuracy. So it's actually pretty important that you lay your windows out in such a way you can quickly switch between devices and keep an eye out for console prompts. The general idea of how I lay my desktop out is identically to method I learned at an IE bootcamp. So I thought I'd share it out, I have all device tiled from the top left with notepad starting at the bottom of my last router. Why you ask? So I know, at a glance, exactly where my switches start.

It seems pretty basic I know, but it makes a HUGE difference in speed.