Just thought I'd share this video out from my YouTube channel here also. So the two people who mistakenly read my blog this week can see how this is done.
Get your nerd hats on! We're freaking pushing labels over our DMVPN network, like a boss. As you might have gathered thus far, I'm a little excited. The only downer here is your label switch path has to be hub-to-spoke, so no mas' spoke-to-spoke tunnels. If you want labels between your spokes, per Cisco documentation, traffic flow absolutely has to be spoke-hub-spoke. Calm down, dry those tears sunshine... because this is still awesome. I can hear you all now "But Jon! One of the best things about DMVPN is building dynamic tunnels between spokes!" shut up Debbie downer. We do loose dynamic tunnels, but we gain having full blown PEs connected only via DMVPN.
Ok, enough build up. How does this work? Surprisingly easy, if you've configured MPLS before... this isn't going to be super exciting. First things first, here's our topology:
All spokes are connected back to the hub via Serial links in a 192.168.zy.x/30 space (where z=lower router number, and y=higher router number). For example the link between R1-Hub and R2-Spoke is 192.168.12.0/30. Then we have Loopback0 configured on each router in the 192.168.x.x/32 space, this is our tunnel source. All traffic supporting DMVPN backhaul is routed via OSPF. Finally, for routing within the DMVPN cloud we're using good old reliable EIGRP. Here's our base DMVPN configurations.
R1-Hub !interface Tunnel100 ip address 10.10.100.1 255.255.255.0 no ip redirects no ip split-horizon eigrp 100 ip nhrp map multicast dynamic ip nhrp network-id 100 mpls ip tunnel source Loopback0 tunnel mode gre multipoint ! ! interface Loopback100 description BGP peering over DMVPN ip address 10.10.1.1 255.255.255.255 !
router eigrp 100 network 10.0.0.0
R2/R3/R4 interface Tunnel100 ip address 10.10.100.x 255.255.255.0 no ip redirects ip nhrp map multicast 192.168.1.1 ip nhrp map 10.10.100.1 192.168.1.1 ip nhrp network-id 100 ip nhrp nhs 10.10.100.1 mpls ip tunnel source Loopback0 tunnel mode gre multipoint ! interface Loopback100 description BGP peering over DMVPN ip address 10.10.2.2 255.255.255.255 ! router eigrp 100 network 10.0.0.0
Pretty simple so far right? Alright, lets get some labels in here.
All Routers mpls ip mpls ldp router-id lo100 ! int tun100 mpls ip !
I know what you're thinking "Jon, here's $5... because you just blew my mind." Well thank you, and I do accept donations. So let's check the output on R1
*Aug 21 01:21:12.009: %LDP-5-NBRCHG: LDP Neighbor 10.10.2.2:0 (1) is UP *Aug 21 01:21:13.005: %LDP-5-NBRCHG: LDP Neighbor 10.10.3.3:0 (2) is UP *Aug 21 01:21:14.106: %LDP-5-NBRCHG: LDP Neighbor 10.10.4.4:0 (3) is UP
Sweet sweet success, but do we have labels? Best place to check is on one of the spokes, I'll look at R4 (he seems lonely).
R4-MPLS#show mpls forwarding-table | ex No Label Local Outgoing Prefix Bytes Label Outgoing Next Hop Label Label or Tunnel Id Switched interface 16 Pop Label 10.10.1.1/32 0 Tu100 10.10.100.1 17 16 10.10.2.2/32 0 Tu100 10.10.100.1 18 17 10.10.3.3/32 0 Tu100 10.10.100.1
Awesome! Don't ignore the next hop, remember that's the secret sauce here. Since we excluded "no ip next-hop-self eigrp 100" from our Hub config, we're forcing all traffic between spokes to route through the hub. As I demonstrate in the video, if we allow the dynamic tunnels this all breaks. So we have a functioning LSP it would seem between spokes so lets get some VRF running and go ping crazy! You don'thave to configure BGP on the Hub, but I am and I'll configure spokes as route reflector clients to minimize spoke configuration.
Now that we have BGP up and running, we'll configure a basic VRF, assign a loopback to said VRF and redistribute connected with our ipv4 address-family (for the VRF).
R2 int lo1001 ip vrf f MPLS ip address 172.16.2.1 255.255.255.0 ! R3 int lo1001 ip vrf f MPLS ip address 172.16.3.1 255.255.255.0 ! R4 int lo1001 ip vrf f MPLS ip address 172.16.4.1 255.255.255.0 !
Last but not least, let's test from R3.
R3-MPLS#show ip bgp vpnv4 vrf MPLS | b Route Route Distinguisher: 65000:1 (default for vrf MPLS) *>i 172.16.2.0/24 10.10.2.2 0 100 0 ? *> 172.16.3.0/24 0.0.0.0 0 32768 ? *>i 172.16.4.0/24 10.10.4.4 0 100 0 ? ! R3-MPLS#show ip route vrf MPLS bgp | b Gateway Gateway of last resort is not set
172.16.0.0/16 is variably subnetted, 4 subnets, 2 masks B 172.16.2.0/24 [200/0] via 10.10.2.2, 00:31:16 B 172.16.4.0/24 [200/0] via 10.10.4.4, 00:31:11 ! R3-MPLS#ping vrf MPLS 172.16.2.1 source lo1001 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 172.16.2.1, timeout is 2 seconds: Packet sent with a source address of 172.16.3.1 !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 19/19/20 ms ! R3-MPLS#traceroute vrf MPLS 172.16.2.1 Type escape sequence to abort. Tracing the route to 172.16.2.1 VRF info: (vrf in name/id, vrf out name/id) 1 10.10.100.1 [MPLS: Labels 16/24 Exp 0] 20 msec 20 msec 20 msec 2 172.16.2.1 20 msec 19 msec 20 msec
Well that's it every body! MPLSoDMVPN! See attached video if you want to hear me talk really fast about doing everything you just read.
Oh man... BGP Dynamic Neighbors. This is a freaking cool way of setting up BGP on a device like a Hub router where you're expecting numerous BGP neighbors. Before dynamic neighbors, I remember configuring my hub router with peer-groups and having an insane amount of syntax since we had ~30-40 spokes. No more my friends, in this post we'll not only look at how to configure iBGP for DMVPN routing, but also using dynamic neighbors to dramatically reduce the amount of configuration on the hub.
Before I jump into the config, you might wonder "why use BGP for DMVPN routing?" Simple answer my friend, it's awesome. So, since most moderately sized organizations have BGP running anyway (think about your MPLS, unless you're super cool and have a full on VPLS... you're peering with your MPLS provider, and more than likely using BGP) using BGP for DMVPN allows a relatively seamless integration of the DMVPN cloud into your organization. I used to preach about "consistent BGP information", because that model sincerely does allow you to build more stable and scalable networks. Also, per Cisco, distance vector routing protocols just play nicer with DMVPN's hub and spoke model. SO enough with the sales pitch, let's get into it. Here's our topology:
We'll configure the spokes first, since there's nothing too exciting happening there. *This post assumes you already have DMVPN up and running, see DMVPN Part 1 for that*
That's it... rinse and repeat on Spokes 2 and 3 (just change your network statement). Now here's the magic, configuring the Hub. Dynamic neighbors aside, there's one key feature we're really concerned with on the Hub... route-reflector-client. Specifically telling the hub that all DMVPN peers are RR clients. Why? Well young padawans, what's the rule about iBGP? BGP expects that internal peerings are configured in a full mesh, and to prevent routing loops, it will not advertised iBGP learned prefixes to other iBGP peers... think of this like BGP's split horizon. Well, that's not going to work for us at all, so we're effectively going to turn it off by telling the hub our DMVPN peers are RR clients. Also note the bgp listen syntax... we'll talk about that bit next.
That's it! Calling "bgp listen range x.x.x.x/x peer-group abcd" is the entire configuration of dynamic neighbors. Now the default behavior allows for 100 dynamic peers, but this can be increased to 5000 with "bgp listen limit 5000". Check out the bgp summary table on the hub, so informative:
HUB#show ip bgp summary | b ^Neighbor Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd *172.16.10.10 4 65000 30 30 11 0 0 00:22:24 1 *172.16.10.20 4 65000 29 34 11 0 0 00:22:22 1 *172.16.10.30 4 65000 29 34 11 0 0 00:22:24 1 * Dynamically created based on a listen range command Dynamically created neighbors: 3, Subnet ranges: 1
BGP peergroup DMVPN listen range group members: 172.16.10.0/24
Total dynamically created neighbors: 3/(5000 max), Subnet ranges: 1
So you can see "Total dynamically created neighbors: 3/(5000 max)" I did bump up the maximum allowed dynamic neighbors to 5000. Also note the "*" next to our neighbors indicating that they were learned dynamically. Alright, the last thing we should look at is a couple pings showing spoke to spoke communication, our routing tables (since iBGP does not update next hop information), and our dmvpn neighbor table after said pings. We'll test between Spoke 2 and Spoke 3.
SPOKE2#ping 172.17.30.30 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 172.17.30.30, timeout is 2 seconds: !!!!! ! SPOKE2#show ip route bgp | b ^Gateway Gateway of last resort is not set
172.17.0.0/32 is subnetted, 4 subnets B 172.17.1.1 [200/0] via 172.16.10.1, 00:28:58 B 172.17.10.10 [200/0] via 172.16.10.10, 00:28:58 B 172.17.30.30 [200/0] via 172.16.10.30, 00:28:58 !
SPOKE2#show dmvpn | b ^ # Ent # Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb ----- --------------- --------------- ----- -------- ----- 1 1.1.1.1 172.16.10.1 UP 00:38:05 S 1 1.1.1.10 172.16.10.10 UP 00:28:34 D 1 1.1.1.30 172.16.10.30 UP 00:00:05 D
Well that's all there is too it! See the linked video for a walk through of this post, and a quick blurb on design considerations.
So now that I'll off my on-call rotation, I'm getting back to studying. I needed something to fire me, so I started watching DMVPN videos from Cisco Live! 2014. One thing that really sparked my interest was spoke-to-spoke multicast communication. I'll lead by saying this... not a production solution I'd recommend. However, for purposes of studying and better understanding DMVPN I think it's very cool.
First up, how do we get it working? Well let's take a look at some sample config from a Hub and two spoke routers before spoke-to-spoke multicast is running.
Hub ... interface tunnel 100 ip address 172.10.100.1 255.255.255.0 ip nhrp map multicast dynamic ip nhrp network-id 100 tunnel source 10.10.1.1 tunnel mode gre multipoint
Spoke1 interface tunnel 100 ip address 172.10.100.2 255.255.255.0 ip nhrp map multicast ip nhrp map 172.10.100.1 10.10.1.1 ip nhrp nhs 172.10.100.1 ip nhrp network-id 100 tunnel source 10.10.1.2 tunnel mode gre multipoint
Spoke2 interface tunnel 100 ip address 172.10.100.3 255.255.255.0 ip nhrp map multicast ip nhrp map 172.10.100.1 10.10.1.1 ip nhrp nhs 172.10.100.1 ip nhrp network-id 100 tunnel source 10.10.1.3 tunnel mode gre multipoint
Alright so that's nothing new, super basic DMVPN. Now with OSPF, if we wanted or were required to use the broadcast network type we go ahead and treat this connection that same as though it were hub and spoke frame relay. The spokes would all have their interfaces set to ospf priority 0 making the hub router the DR with no BDR. So assuming we've done that let's take a look at our OSPF neighbor table on the hub and spoke 1.
Hub#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface 172.16.2.2 0 FULL/DROTHER 00:00:39 172.16.10.2 Tunnel100 172.16.3.3 0 FULL/DROTHER 00:00:37 172.16.10.3 Tunnel100
SPOKE1#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface 172.16.1.1 1 FULL/DR 00:00:37 172.16.10.1 Tunnel100
So this works just fine... but what happens if we make the following changes on spoke 1 and 2?
SPOKE1#conf t SPOKE1(config)#int tun100 SPOKE1(config-if)#ip nhrp map multicast 10.10.1.3 SPOKE1(config-if)#end
SPOKE2#conf t SPOKE2(config)#int tun100 SPOKE2(config-if)#ip nhrp map multicast 10.10.1.2 SPOKE2(config-if)#end
Something magical happens... our spokes (without any neighbor statements) become OSPF neighbors. This gives us some really interesting properties within our DMVPN cloud. For one, we have this strange static mapping, always on, of broadcast and multicast traffic between spokes... but unicast traffic is still dynamic. Again, this isn't a best practice but rather an exercise attempting to better understand nhrp within dmvpn. Quick output of shows commands from spoke 1 (just trust me, they're about the same on spoke 2), then some closing thoughts.
SPOKE1#show ip ospf nei
Neighbor ID Pri State Dead Time Address Interface 172.16.1.1 1 FULL/DR 00:00:38 172.16.10.1 Tunnel100 172.16.3.3 0 2WAY/DROTHER 00:00:38 172.16.10.3 Tunnel100 SPOKE1#show ip nhrp brief Target Via NBMA Mode Intfc Claimed 172.16.10.1/32 172.16.10.1 10.10.1.1 static Tu100 < > 172.16.10.3/32 172.16.10.3 10.10.1.3 dynamic Tu100 < > SPOKE1#show dmvpn Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete N - NATed, L - Local, X - No Socket # Ent --> Number of NHRP entries with same NBMA peer NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting UpDn Time --> Up or Down Time for a Tunnel =====================================
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb ----- --------------- --------------- ----- -------- ----- 1 10.10.1.1 172.16.10.1 UP 01:55:53 S 1 10.10.1.3 172.16.10.3 UP 00:44:32 D
The best way to think of NHRP (paraphrasing Mike Sullenberger) is a beefy ARP. If you think about the purpose and function of arp compared to nhrp they're incredibly similar. Where arp attempts to map ip address to mac-address for forwarding on an ethernet network, nhrp attempts to map gre ip to non-broadcast ip. Of course that's only it's primary function, nhrp can also carry cool info like nhrp groups used to assign per-tunnel QoS.
First off, fear not. I've already recorded videos from DMVPN part 2, I didn't give up lol. I just need to get some time to toss everything into text format. I just thought I'd do a really short post about DMVPN on the new blueprint. I haven't tested yet, however from Cisco 360 workbooks, I'm starting to get a good idea of what kind of curve balls to expect on the lab. It's gets me SUPER excited to see how deep just the workbooks have been getting on the subject matter, Cisco is treating it as a frame-relay replacement more than just a subtle new topic.
So far I seen scenarios where you can't allow broadcast/multicast in the DMVPN cloud, so OSPF behavior is similar to FR networks without broadcast. I've also seen some per-tunnel QoS, where you have to leverage NHRP to apply policy-maps per-dynamic tunnel via NHRP groups (IOS help nhrp, you'll be fine. Not "ip nhrp", just nhrp ?). Cisco also loves to trip you up with phrasing like "Map broadcast traffic between spoke routers R1 and R2", but they'll make no mention of the hub. This sounds like you're digging into additional nhrp maps, but that's not the case. Since so long as you're mapping multicast to the hub, nhrp will dynamically map broadcast/multicast traffic between spokes, just as it does with unicast traffic.
So I know I'm just rambling in this post, but this new content should get you excited, not scared. Honestly, one of the best study sessions I've had on the subject matter was sitting down with a follow CCIEer, building a DMVPN network... and looking for ways to break it. Then doubling back and seeing how manipulate behavior, and of course lots of DOC CD and IOS help along the way.
Study hard! I'll have my next DMVPN post out soon.
So I wanted to do a couple posts on DMVPN, one of the new topics to CCIE v5. I'll run through a basic setup in this post, with a video at the end. In the next few posts I'll show some of the more advanced things we can do with DMVPN.
First off, for mostDMVPN configurations... it's not that bad. In a single hub environment, you're either the hub or you're a spoke. Hub routers are responsible for keeping track of NHRP (next hop resolution protocol) registrations from spokes, and for informing spokes of that information when they need to build dynamic tunnels between one another. So take the topology below:
The first picture is just for reference, it doesn't matter how many provider routers are between these devices (or if there are any at all). So in a DMVPN deployment spoke routers are statically configured with the public and tunnel IP of the hub. Here's the good news, if you're goal is "Let's just get it working!" then the hub configuration is really really simple. Check it out:
Hub(config)# interface tunnel 0 Hub(config-if)# tunnel source 150.100.1.2 Hub(config-if)# tunnel mode gre multipoint Hub(config-if)# ip nhrp map multicast dynamic Hub(config-if)# ip nhrp network-id 100 Hub(config-if)# ip address 172.16.10.1 255.255.255.0
That's it, obviously if you've ever seen a production headend router you cando more... but as for "what's the bare minimum to make it work" that's it! Also note that the nhrp network-id is locally significant and complete arbitrary, however it is required (think OSPF process id). Now the spoke configuration is a little more involved, because remember spokes have to be statically mapped to the headend/hub router.
SPK1(config)# interface tunnel 0 SPK1(config-if)# tunnel source 150.100.10.2 SPK1(config-if)# tunnel mode gre multipoint SPK1(config-if)# ip nhrp map multicast 150.100.1.2 SPK1(config-if)# ip nhrp map 172.16.10.1 150.100.1.2 SPK1(config-if)# ip nhrp nhs 172.16.10.1 SPK1(config-if)# ip nhrp network-id 100 SPK1(config-if)# ip address 172.16.10.10 255.255.255.0
So what's going on here? On the hub router we're saying "Any broadcasts/multicasts going out this interface - the next hops are going to be learned dynamically". Then on the spoke(s) we're saying "Any broadcasts/multicasts need to go to 150.100.1.2 so we can learn about the next hop. Also, any traffic going 172.16.10.1, that needs to be forwarded to 150.100.1.2". Ok, so that about covers our NHRP map statements, so the last thing we should talk about on the spoke configuration is "ip nhrp nhs". NHS=Next-hop server, this line of code is instructing the spoke to register with the IP address that follows (the hub's tunnel IP). Without specifying who your NHS is, the spoke wouldn't know that it has to tell the hub about it's NBMA(the tunnel source) and tunnel IPs.
Alright! So that's configuring DMVPN! We can look on the Hub after all spokes are configured and issue a 'show dmvpn' to see all nhrp registered devices like so:
HUB#show dmvpn Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete N - NATed, L - Local, X - No Socket # Ent --> Number of NHRP entries with same NBMA peer NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting UpDn Time --> Up or Down Time for a Tunnel =============================================================== Interface: Tunnel0, IPv4 NHRP Details Type:Hub, NHRP Peers:3, # Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb ----- --------------- --------------- ----- -------- ----- 1 150.100.10.2 172.16.10.10 UP 00:01:24 D 1 150.200.20.2 172.16.10.20 UP 00:00:46 D 1 150.200.30.2 172.16.10.30 UP 00:00:23 D
Very good, so by the attribute code "D" we can see that the hub has learned all these peers dynamically (as expected). What does this output look like on Spoke1?
SPK1#sh dmvpn Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete N - NATed, L - Local, X - No Socket # Ent --> Number of NHRP entries with same NBMA peer NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting UpDn Time --> Up or Down Time for a Tunnel =============================================================== Interface: Tunnel0, IPv4 NHRP Details Type:Spoke, NHRP Peers:1, # Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb ----- --------------- --------------- ----- -------- ----- 1 150.100.1.2 172.16.10.1 UP 00:03:04 S
As expected, we currently only have (1) peer, the hub. Check out that attribute code "S" too, since we mapped him out statically. Lastly, what happens if we ping spoke2's tunnel ip from spoke1, how does that impact the show dmvpn output?
... Interface: Tunnel0, IPv4 NHRP Details Type:Spoke, NHRP Peers:2, # Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb ----- --------------- --------------- ----- -------- ----- 1 150.100.1.2 172.16.10.1 UP 00:04:46 S 1 150.200.20.2 172.16.10.20 UP 00:00:02 D
True to it's name, DMVPN builds a dynamic tunnel between spoke1 and spoke2. Pop quiz hot shots, where did it get the NBMA address from? You guessed it, or at least I hope you did, the Hub! Alright, this is getting to be a lot of typing lol, I'll demonstrate the above configuration as well as getting EIGRP and OSPF running over DMVPN in a video.
So I'm finally getting back into lab mode, and getting excited out R/S technologies again. After realizing that the requirements for Cisco's CSR1000v (Cloud Services Router) has been dramatically decreased I wanted to get one into virtualbox and throw it in the mix with GNS3. The results thus far have been phenomenal. Hence what this post is about, something I've wanted to get working in a home lab for some time now is VPLS. For those of you not familiar with VPLS, I'm speaking of virtual private lan services. With a 7200 in GNS3 we've been able to create point-to-point pseudowires for some time now. A pseudowire completely hides the MPLS provider network from clients and gives the illusion that both CEs are directly connected by a very long cable. This works because we're encapsulating frames received on the PE and associating them with a virtual circuit that gets label switched between two PE routers. Very cool stuff...
However, what if you had a situation where the client wanted this layer 2 style of WAN... but they had multiple sites. Even worse they wanted a full mesh! Enter VPLS, giving us the ability to almost carve out a small switch inside the MPLS cloud. This is something that GNS3 has been lacking for sometime, because no GNS3 emulated router can host a multipoint pseudowire. Until now! Well... technically it's not GNS3 doing the heavy lifting, it's the CSR1000v. So I made a video about this setup, of course, before trying this in your own lab a couple key notes about the CSR1000v:
- IOS XE 3.10+ is your best bet because of the lowered requirements
- IOS XE 3.10+ requires 2.5GB of RAM, but it can run on a single vCPU (this is vastly improved from the 3.9 release)
- In order to test out the more advance features, after booting up your CSR1000v go into global configuration and enter "license boot level premium", other wise you can't enable MPLS. After setting the boot level to premium and accepting the EULA for eval licensing, wr mem and reload.
- Lastly, unless you want to do all your configs within virtualbox via CSR's virtual console, install CSR with a Serial Console (from the GRUB menu on first boot) and connect via a named pipe. I'm using SOCAT in OS X to redirect the named pipe to /dev/ttys001. Your mileage may vary.