Cisco IOS & SNMP: A backdoor into devices you can't access.

We've all been there. You desparately need to get into a router or a switch, but every username and password you try doesn't seem to work. Or even worse, you can ping the device, see it in CDP but SSH isn't replying and telnet is disabled (presumably because the device doesn't have an RSA key). If you were close to the device, sure, you could walk over to it and try to console in. What if that's not an option? Reload and do password recovery?? Well, maybe not. In today's super quick post I want to give you an alternative if SNMP is enabled and you know the rw community string.

Ingredients:

1. Router you can't log into.
1. SNMP read-write community string
2. SNMP tools installed on your machine. (e.g. net-snmp for Windows)
3. TFTP Server running on your machine.
4. Deep breaths. Seriously, calm down. You're locked out of a router, it's not the end of the world or anything Eddie. Yeah, I'm calling you out Eddie.

Don't Panic

Now that you're definitely not panicking, let's open a command prompt on our machine. Since this is a lab, I'm actually using an Ubuntu docker image in GNS3, however the actual process and syntax doesn't change at all if you're running net-snmp in Windows. Where I'm in bash, you'll be in CMD issuing the exact same commands. My TFTP server is also running on this Ubuntu docker image, but again, this works absolutely fine with SolarWind's tftp server or tftpd64. Finally my device is a router named R1 running at 192.168.122.211, and my Ubuntu machine is 192.168.122.93. Let's first confirm the state of things.

root@LX1:~# ssh admin@192.168.122.211     
Password:
Password:
Password:
admin@192.168.122.211's password:
Connection to 192.168.122.211 closed by remote host.
Connection to 192.168.122.211 closed.
root@LX1:~#  


Yup. Good and locked out. So the first thing I'll need to do is create a file in my TFTP directory that contains the changes I wish to make. For this post I'm just going to change the username admin's password to cisco123. However, get creative with this. You could just as easily modify vty lines to permit telnet temporarily, or disable AAA if it's pointing to TACACS/RADIUS for authentication. So I'll create a small file called 'configme.txt' and put said file in my TFTP root directory. Important note here, you don't need to put the full running config in this file, only the changes you wish to merge with existing config on the router. That is to say, this doesn't overwrite and replace the entire running-config, say it with me Eddie "merge". Here's what my file looks like.

root@LX1:~# cat /var/TFTP-ROOT/configme.txt
username admin privi 15 secret cisco123
!
end
root@LX1:~# 

Putting the word "end" on a line all by itself let's the router know it's reached the end of the config file. You don't have to do this, but if you don't the router generates an error saying the config file ended unexpectedly. Alright, now that we have all that setup, and our tftp server is running, let's send our router some commands via SNMP and tell it to download 'configme.txt' and merge it into our running-config. Lucky for me, I just so happen to know this device has 'write' as a read-write community string.

root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.2.111 i 1

Created directory: /var/lib/snmp/mib_indexes
iso.3.6.1.4.1.9.9.96.1.1.1.1.2.111 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.3.111 i 1

iso.3.6.1.4.1.9.9.96.1.1.1.1.3.111 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.4.111 i 4

iso.3.6.1.4.1.9.9.96.1.1.1.1.4.111 = INTEGER: 4
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.5.111 a 192.168.122.93

iso.3.6.1.4.1.9.9.96.1.1.1.1.5.111 = IpAddress: 192.168.122.93
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.6.111 s configme.txt

iso.3.6.1.4.1.9.9.96.1.1.1.1.6.111 = STRING: "configme.txt"
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.14.111 i 1

iso.3.6.1.4.1.9.9.96.1.1.1.1.14.111 = INTEGER: 1

Now let's quickly confirm everything works as expected.

root@LX1:~# ssh admin@192.168.122.211 
Password:


R1#show ip int br | ex unass
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.122.211 YES DHCP   up                    up     

R1#show logging | inc tftp
Jul 14 15:40:49.808: %SYS-5-CONFIG_I: Configured from tftp://192.168.122.93/configme.txt by console
R1#exit
Connection to 192.168.122.211 closed.

What just happened?

All in all we sent (6) SNMP strings to the router, I'll really quickly break those out here.

1. "1.3.6.1.4.1.9.9.96.1.1.1.1.2.[random_number] i 1" sets the transfer protocol to TFTP.
2. "1.3.6.1.4.1.9.9.96.1.1.1.1.3.[random_number] i 1" sets the source file as a network file
3. "1.3.6.1.4.1.9.9.96.1.1.1.1.4.[random_number] i 4" sets the destination as running-config
4. "1.3.6.1.4.1.9.9.96.1.1.1.1.5.[random_number] a 192.168.122.93" sets the TFTP Server Address
5. "1.3.6.1.4.1.9.9.96.1.1.1.1.6.[random_number] s configme.txt" sets the file name
6. "1.3.6.1.4.1.9.9.96.1.1.1.1.14.[random_number] i 1" initiates the transfer.

If you look above, my [random_number] was 111, now if I wanted to issues these commands again, maybe to change the file name or TFTP server address, I'd need to pick a new random number. Just re-running those MIBs with the same random number as before will error out as seen here:

root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.5.111 a 192.168.122.93
Error in packet.
Reason: inconsistentValue (The set value is illegal or unsupported in some way)
Failed object: iso.3.6.1.4.1.9.9.96.1.1.1.1.5.111

However, if I change .111 to .112 we're back in business.

root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.2.112 i 1

iso.3.6.1.4.1.9.9.96.1.1.1.1.2.112 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.3.112 i 1

iso.3.6.1.4.1.9.9.96.1.1.1.1.3.112 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.4.112 i 4

iso.3.6.1.4.1.9.9.96.1.1.1.1.4.112 = INTEGER: 4
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.5.112 a 192.168.122.93

iso.3.6.1.4.1.9.9.96.1.1.1.1.5.112 = IpAddress: 192.168.122.93
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.6.112 s configme.txt

iso.3.6.1.4.1.9.9.96.1.1.1.1.6.112 = STRING: "configme.txt"
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.14.112 i 1

iso.3.6.1.4.1.9.9.96.1.1.1.1.14.112 = INTEGER: 1
root@LX1:~#


Best of luck getting back into your devices, that's it for this post.

FlexVPN: Spoke-2-Spoke PSK

So in my last post I covered configuring FlexVPN with dynamic spoke to spoke communication using certificates for authentication. What if you wanted to use pre-shared-keys though? We can do that, it's not nearly as secure as certificate based authentication, but we can do that.

We'll be using the exact same topology as the last post, and seeing as I went through the config in a good bit of detail in the last post, I'll keep this one short and sweet trying to only spend time on the differences in config.

Ingredients:

• (3) Cisco Routers. I'll be using CSR1000v running 16.03.01.
• Patience. 

Estimate lab time: ~15min

FlexVPN Topology

FlexVPN Configuration

In the interest of exploring some more interesting features in IKEv2 with respect to pre-shared-keys, I thought we'd up the ante a bit and do both asymmetric authentication between Hub and Spokes, as well as symmetric authentication between spokes. That means we'll need an ikev2 keyring.  Without further to do, let's look at the Hub configuration. I'll call out the biggest differences in red from my last post.

Hub Router:

aaa new-model
aaa authorization network flexauthz local
aaa session-id common
!
ip local pool flexpool 10.254.0.10 10.254.0.254
!
ip access-list standard flexroutes
 permit 10.0.0.0 0.255.255.255
!
crypto ikev2 authorization policy flexauthzpol
 pool flexpool
 route set access-list flexroutes
!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20
!
crypto ikev2 policy flexpol
 proposal flexprop
!
crypto ikev2 keyring flexkeys
 peer spokes
  address 0.0.0.0 0.0.0.0
  pre-shared-key local HubsLikeCerts
  pre-shared-key remote SoDoSpokes
 !
crypto ikev2 profile flexprof
 match identity remote address 0.0.0.0
 authentication remote pre-share
 authentication local pre-share
 keyring local flexkeys
 aaa authorization group psk list flexauthz flexauthzpol
 virtual-template 1
!
crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel
!
crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprof
!
interface Virtual-Template1 type tunnel
 ip unnumbered Loopback1
 ip nhrp network-id 1337
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect

So not too shabby at all right? No PKI to worry about, all we have to do here is define a keyring and call that keyring in our IKEv2 profile. On the spokes we'll see a litte more configuration within the keyring to meet the requirements I defined above (asymm for hub<->spokes and symm spoke<->spoke). Now let's look at the spoke config, this configuration is identical on both S2 and S3 routers, except the ACL changes. S2 has permit 10.0.2.0 0.0.0.255 and S3 has permit 10.0.3.0 0.0.0.255.

Spoke Routers:

aaa new-model
aaa authorization network flexauthz local
aaa session-id common
!
ip access-list standard flexroutes
 permit 10.0.x.0 0.0.0.255
 # ACL is the only part of the config that differs spoke to spoke.
!
crypto ikev2 authorization policy flexauthzpol
 route set access-list flexroutes
!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20
!
crypto ikev2 policy flexpol
 proposal flexprop
!
crypto ikev2 keyring flexkeys
 peer hub
  address 169.254.0.5
  pre-shared-key local SoDoSpokes
  pre-shared-key remote HubsLikeCerts
 !
 peer spokes
  address 0.0.0.0 0.0.0.0
  pre-shared-key Spokes<3Certs
 !
crypto ikev2 profile flexprof
 match identity remote address 0.0.0.0
 authentication remote pre-share
 authentication local pre-share
 keyring local flexkeys
 aaa authorization group psk list flexauthz flexauthzpol
 virtual-template 1
!
crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel
!
crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprof
!
interface Tunnel0
 ip address negotiated
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel destination 169.254.0.5
 tunnel protection ipsec profile flexprotect
!
interface Virtual-Template1 type tunnel
 ip unnumbered Tunnel0
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect

So as you can see, using pre-shared-keys isn't too different from a configuration stand point than certificate based authentication. However, in the real world, what you're giving up is the ability to revoke a single router's creditials. Following a certificate auth model, if a device was lost or compromised, we could revoke it's certificate and once all devices got an up to date CRL, they'd reject any authentication attempt from that device. With pre-shared-keys we'd have to touch each one of our devices to update PSKs to achieve the same goal. No bueno. That's enough lecturing, let's do some quick verfication from S2's router to make sure it can both talk to the Hub and build a dynamic tunnel with S3.

Verification on S2

S2#show ip int br | ex unass
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       169.254.2.5     YES NVRAM  up                    up     
GigabitEthernet2       10.0.2.1        YES NVRAM  up                    up     
Tunnel0                10.254.0.10     YES manual up                    up     
Virtual-Template1      10.254.0.10     YES unset  up                    down   
!
!
S2#show ip route static | beg ^Gate
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0
!
S2#show crypto ikev2 sa
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/1224 sec
!
S2#ping 10.0.3.1 source 10.0.2.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.3.1, timeout is 2 seconds:
Packet sent with a source address of 10.0.2.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/6/7 ms
S2#
*Jul 11 19:25:26.917: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to down
S2#
*Jul 11 19:25:27.160: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to up
S2#show ip route static | beg ^Gateway
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 6 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0
S   %    10.0.3.0/24 is directly connected, Virtual-Access1
S2#show crypto ikev2 sa
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
3         169.254.2.5/500       169.254.3.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/19 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/1262 sec

And boom goes the dynamite. That's all for this one, happy labbing.

FlexVPN: Spoke-2-Spoke Tunnels

Let's talk about FlexVPN, a prime contender as a DMVPN replacement and sometimes referred to as DMVPN phase 4. In this post, I'm going to explore the nuts and bolts of getting FlexVPN up and running between (3) routers and, for added flare, I'm going to also configure dynamic spoke-2-spoke tunnels. The dynamic spoke-2-spoke portion of the config will look very similar to those of you familiar with DMVPN, with some key differences. Before diving in, let's take a look at our diagram. I'd like to also point out in this lab, the Hub is also our certificate authority (because pre-shared keys are for suckers).

Ingredients

• (3) Cisco Routers. I'll be using CSR1000v running 16.03.01.
• Patience.

Estimate lab time: ~30min

 

Configuring IOS CA & Provision Certificates

 

Since we'll be using RSA for authentication, as opposed to pre-shared keys, we're going to need a trusted certificate authority and for each router to have it's own certificate issued from said CA. Now, in real life this can be either an online or offline CA running on IOS, Linux, or Windows. However, to keep things simple, in this lab I'm just going to configure the Hub router as the CA, then have all routers request a certificate from the CA.

IOS CA Configuration:

ip domain name hop16.com
!
! Replace 12:00:00 10 JUL 2019 with current time and date
do clock set 12:00:00 10 JUL 2019
!
!
ntp master 3
!
crypto pki server flexvpnca
 database level complete
 no database archive
 issuer-name CN=Hub.hop16.com,OU=NOC,L=FOO,C=US
 hash sha256
 database url flash:
 grant auto
 no shutdown
%Some server settings cannot be changed after CA certificate generation.
% Please enter a passphrase to protect the private key
% or type Return to exit
Password: 
Re-enter password: 
% Generating 1024 bit RSA keys, keys will be non-exportable...
!

All Routers (yes, even the Hub/IOS CA from above)

ip domain name hop16.com
!
crypto key generate rsa modulus 2048 label FLEXKEYS
!
crypto pki trustpoint flexvpn-tp
 enrollment url http://169.254.0.5:80
 hash sha256
 subject-name CN={ChangeForEachSite}.hop16.com,OU=NOC,L=FOO,C=US
 fqdn {ChangeForEachSite}.hop16.com
 # e.g. subject-name CN=S2.hop16.com,OU=NOC,L=FOO,C=US for S2
 # e.g. subject-name CN=S3.hop16.com,OU=NOC,L=FOO,C=US for S3
 revocation-check crl
 rsakeypair FLEXKEYS
!
crypto pki authenticate flexvpn-tp
% Do you accept this certificate? [yes/no]: yes
!
crypto pki enroll flexvpn-tp
 # This command will prompt you to create a password for revocation,
 # and some additional options about including router serial number,
 # and IP address in the certification subject name. Both are
 # optional, I opted to include the router's IP address but
 #exclude the serial number.

After a couple seconds, you'll see a prompt on each router confirming they've received their cert from the CA "%PKI-6-CERTRET: Certificate received from Certificate Authority". At this point, assuming you haven't hit any errors, we should be able to verify we have a valid certificate from our CA with 'show crypto pki certificates flexvpn-tp'.


S2#show crypto pki certificates 
Certificate
  Status: Available
  Certificate Serial Number (hex): 03
  Certificate Usage: General Purpose
  Issuer:
    cn=Hub.hop16.com
    ou=NOC
    l=FOO
    c=US
  Subject:
    Name: S3.hop16.com
    IP Address: 169.254.3.5
    ipaddress=169.254.3.5+hostname=S3.hop16.com
    cn=S3.hop16.com
    ou=NOC
    l=FOO
    c=US
  Validity Date:
    start date: 14:26:30 UTC Jul 10 2019
    end   date: 14:26:30 UTC Jul 9 2020
  Associated Trustpoints: flexvpn-tp
  Storage: nvram:Hubhop16com#3.cer

CA Certificate
  Status: Available
  Certificate Serial Number (hex): 01
  Certificate Usage: Signature
  Issuer:
    cn=Hub.hop16.com
    ou=NOC
    l=FOO
    c=US
  Subject:
    cn=Hub.hop16.com
    ou=NOC
    l=FOO
    c=US
  Validity Date:
    start date: 14:20:00 UTC Jul 10 2019
    end   date: 14:20:00 UTC Jul 9 2022
  Associated Trustpoints: flexvpn-tp

Configuring FlexVPN - Hub and Spoke

The biggest downside to FlexVPN in my humble opinion is that it's ssoooo config intensive. The benefit is that once you get the configs ironed out, it's incredibly easy to template. Most everything (including the vast majority of the Hub configs) can be copy/paste to other routers. That said, there are a couple bits of config unique to the Hub, and I'll call those out as we go.

Hub Router:

aaa new-model
aaa authorization network flexauthz local
aaa session-id common
!
! Defining a local pool is only needed on the Hub router, and only
! required if you wish to have the spokes get their tunnel IP
! dynamically which is preferred.
!
ip local pool flexpool 10.254.0.2 10.254.0.254
!
ip access-list standard flexroutes
 permit 10.0.0.0 0.255.255.255
 #
 # Note here, normally your route ACL only has local routes, but we
 # need to be more general on the Hub so S2 knows how to reach S3. 
 #
!
crypto ikev2 authorization policy flexauthzpol
 # Using an address pool here so our spokes can negotiate a tunnel
 # IP from the Hub.
 pool flexpool
 route set access-list flexroutes
!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20
!
crypto ikev2 policy flexpol
 proposal flexprop
!
crypto ikev2 profile flexprofile
 match identity remote fqdn domain hop16.com
 identity local fqdn Hub.hop16.com
 authentication remote rsa-sig
 authentication local rsa-sig
 pki trustpoint flexvpn-tp
 aaa authorization group cert list flexauthz flexauthzpol
 virtual-template 1
#
# Calling out a virtual-template within our IKEv2 profile is only
# needed if the router has a DVTI interface, which in our initial
# configurations will only be the Hub. Also, if you forget to
# specify this here, your Hub won't work and you'll start
# pulling your hair out troubleshooting IKEv2/IPsec messages 
# (which will all report successful authentications). Maddening.
#
crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel
!
crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprofile
!
interface Loopback1
 ip address 10.254.0.1 255.255.255.0
!
interface Virtual-Template1 type tunnel
 # Again, DVTI interface is only required on the Hub
 # while we're doing a Hub and spoke topology.
 ip unnumbered Loopback1
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect
!

Easy peasy right? It's a lot of config... like all of the config lol. We could trim this down a little if we were doing PSK with an IKEv2 keyring (follow up blog post for that). However, I like doing certificate based authentication so here we are.

Moving on to the spoke router configs, let's look to S2 as an example. I'll call out the bits in red that will need to change before copy/pasting this config to S3's router.

Spoke Router (S2):

ip access-list standard flexroutes
 permit 10.0.2.0 0.0.0.255
!
aaa new-model
aaa authorization network flexauthz local
aaa session-id common
!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20
!
crypto ikev2 policy flexpol
 proposal flexprop
!
crypto ikev2 profile flexprofile
 match identity remote fqdn domain hop16.com
 identity local fqdn S2.hop16.com
 authentication remote rsa-sig
 authentication local rsa-sig
 pki trustpoint flexvpn-tp
 aaa authorization group cert list flexauthz flexauthzpol
!
crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel
crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprofile
!
interface Tunnel0
 ip address negotiated
 tunnel source GigabitEthernet1
 tunnel destination 169.254.0.5
 tunnel protection ipsec profile flexprotect


See how easy that is to template out though? Absolutely nuts, and we could potentially make this even more generic with only injecting routes to our local interfaces (route set interface instead of route set access-list) and running EIGRP or BGP over the tunnels. Moving on with this config, router S3 has an identical config, except the IKEv2 local identity is S3.hop16.com and my access-list flexroutes is permitting 10.0.3.0 0.0.0.255. Now that our Hub and Spoke FlexVPN is up and running, let's see what it takes to convert that to a mesh style VPN topology like DMVPN.

Configuring FlexVPN - Mesh (DMVPN Phase 4)

Finally we're getting at the subject of this post, spoke to spoke dynamic tunnels. The mechanism for getting spoke to spoke communication is the very same used in DMVPN deployments, NHRP. With some key changes.

• No NHRP Next-Hop Server (NHS)
• Instead mapping NBMA to Tunnel IP is derived from IKEv2
• Spokes use SVTI (Tunnel0 interface) for communication to the Hub
• Spokes use DVTI (Virtual-Template) for dynamic tunnels between one another. 

Now, in order to make the required changes to enable NHRP we have to shutdown our SVTI (Tunnel0) interfaces on the spokes. Otherwise the Hub's virtual-template will be locked and IOS will prevent us from making any changes. After shutting down Tunnel0 on both routers, we can add the following to our Hub.

Hub Router:

interface Virtual-Template1 type tunnel
 ip nhrp network-id 1337
 ip nhrp redirect 

Believe it or not, that's actually we need to add on the Hub side. We're enabling NHRP redirects, just like DMVPN Phase 3, so the Hub can signal the Spokes to build dynamic tunnels. Also just like DMVPN, the first packet or two will transit the Hub while the Spokes are receiving NHRP information via the redirect message and building their dynamic tunnels. Alright, let's look at the required config on our Spokes.

Spoke Routers:

crypto ikev2 profile flexprofile
 virtual-template 1
!
interface Virtual-Template1 type tunnel
 no ip address
 ip unnumbered Tunnel0
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect
!
interface Tunnel0
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect

Much like most things FlexVPN, while easily templated and generic, it's configuration intensive. That said, this is a super interesting configuration. Where in DMVPN all the traffic happens on our multipoint Tunnel interface, in FlexVPN the tunnel interface remains a static P2P tunnel destined to the Hub. Then we're pushing our NHRP shortcut to a seperate virtual-template (DVTI) that is will build P2P virtual-access interfaces to other spokes. If we had multiple dynamic tunnels to other spokes each would get their own Virtual-Access interface. Alright, let's bring the tunnels on the spokes back up, and do some verification.

S2#show ip route static | begin ^Gate
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0
S        10.254.0.1/32 is directly connected, Tunnel0
S2#show crypto ikev2 sa             
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
      Life/Active Time: 86400/26 sec

Now I'm going to enable NHRP debugs, and try to ping 10.0.3.1 which is connected to S3.


S2#debug nhrp
NHRP protocol debugging is on
S2#ping 10.0.3.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.3.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/8/11 ms
S2#
 [...]
*Jul 11 11:53:55.363: NHRP: Sending NHRP Resolution Request for dest: 10.0.3.1 to nexthop: 10.0.3.1 using our src: 10.254.0.9 vrf:global(0x0)
*Jul 11 11:53:55.363: NHRP: Attempting to send packet through interface Tunnel0 via DEST  dst 10.0.3.1
*Jul 11 11:53:55.364: NHRP: Send Resolution Request via Tunnel0 vrf global(0x0), packet size: 72
*Jul 11 11:53:55.364:       src: 10.254.0.9, dst: 10.0.3.1
 [...]
*Jul 11 11:53:55.364: NHRP: 96 bytes out Tunnel0
*Jul 11 11:53:55.372: NHRP: Receive Resolution Request via Tunnel0 vrf global(0x0), packet size: 92
*Jul 11 11:53:55.372: NHRP: Route lookup for destination 10.254.0.9 in vrf global(0x0) yielded interface Tunnel0, prefixlen 32
*Jul 11 11:53:55.373: NHRP: Request was to us. Process the NHRP Resolution Request.
*Jul 11 11:53:55.373: NHRP: nhrp_rtlookup for 10.254.0.9 in vrf global(0x0) yielded interface Tunnel0, prefixlen 32
*Jul 11 11:53:55.373: NHRP: Request was to us, responding with ouraddress
*Jul 11 11:53:55.373: NHRP: Checking for delayed event 10.254.0.8/10.254.0.9 on list (Tunnel0 vrf: global(0x0))
*Jul 11 11:53:55.373: NHRP: No delayed event node found.
*Jul 11 11:53:55.374: NHRP: Enqueued Delaying resolution request nbma src:169.254.2.5 nbma dst:169.254.3.5 reason:IPSEC-IFC: need to wait for IPsec SAs.
*Jul 11 11:53:55.374: NHRP: Interface: Tunnel0 configured with FlexVPN. Deferringcache creation for nhop 10.254.0.8
*Jul 11 11:53:55.374: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to down
 [...]
*Jul 11 11:53:55.387:  Instructing NHRP to create Virtual-Access from Virtual template 1 for interface Virtual-Access1
*Jul 11 11:53:55.387: NHRP: NHRP Redirect Feature PI-code Initialized
*Jul 11 11:53:55.387: NHRP: Redirect Feature Initialized - Attempting Platform Init
*Jul 11 11:53:55.393: NHRP: Updating delayed event with destination 169.254.3.5 on interfaceTunnel0 with the new interface Virtual-Access1
*Jul 11 11:53:55.595: NHRP:
*Jul 11 11:53:55.595:  Fetched address from underlying IKEv2 for interfaceVirtual-Access1. Pre-NATed = 169.254.2.5, Post-NATed = UNKNOWN
*Jul 11 11:53:55.595: NHRP: Processing delayed event on interface Tunnel0 with NBMA 169.254.3.5
 [...]
*Jul 11 11:53:55.598: NHRP: No need to delay processing of resolution event nbma src:169.254.2.5 nbma dst:169.254.3.5
*Jul 11 11:53:55.598: NHRP: Attempting to send packet through interface Virtual-Access1 via DEST  dst 10.254.0.8
*Jul 11 11:53:55.598: NHRP: Send Resolution Reply via Virtual-Access1 vrf global(0x0), packet size: 120
*Jul 11 11:53:55.598:       src: 10.254.0.9, dst: 10.254.0.8
*Jul 11 11:53:55.598: NHRP: 144 bytes out Virtual-Access1
*Jul 11 11:53:55.604: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to up
 [...]
!
!
S2#show ip route static | begin ^Gate
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 7 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0
S   %    10.0.3.0/24 is directly connected, Virtual-Access1
S        10.254.0.1/32 is directly connected, Tunnel0
S   %    10.254.0.8/32 is directly connected, Virtual-Access1
S2#show crypto ikev2 sa
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
2         169.254.2.5/500       169.254.3.5/500       none/none            READY
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
      Life/Active Time: 86400/399 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
      Life/Active Time: 86400/443 sec

 IPv6 Crypto IKEv2  SA


I did my best to trim down the NHRP debug messages, but there's a lot of interesting things happening here. Now looking at our static routes, we have the original summary to 10.0.0.0/8 via Tunnel0 to the Hub, but we also have a specific route to 10.0.3.0/24  and a route to 10.254.0.8/32 via Virtual-Access1 directly to router S3 with the % indicating next hop override (due to NHRP). Well that's about it for now, I'll kick out a quick follow up post for those interested in using pre-shared-keys for authentication.

CCIE Security: Troubleshooting (Ticket #1) - Solution

Alright it's been a couple of days since the original post, so after much fanfare and exactly 0 people attempting to solve, let's break this one down.

SPOILER ALERT

Issue #1

Since BGP is relying on OSPF for connectivity between peering interfaces (Loopback1), this seems like a natural place to start. The first thing we'll notice on R1 and R2, is when running debug ip ospf packet, we're only seeing traffic leave each respective router. We're not actually seeing either router receive ospf traffic.

R1#debug ip ospf adj  

OSPF adjacency debugging is on
R1#
*Jul  3 16:14:56.959: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 16:15:06.512: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 16:15:15.996: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0

R2#debug ip ospf packet
OSPF packet debugging is on
R2#
*Jul  3 16:16:34.653: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 16:16:44.567: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 16:16:53.574: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0

So let's go over to the ASA and look at our logs to see if anything looks amiss.

ASAv1(config)# logging buffered 7
ASAv1(config)# logging enable

ASAv1(config)# end

ASAv1# show logging

ASAv1# show logging             
[...]
%ASA-3-106010: Deny inbound protocol 89 src INSIDE:10.0.0.2 dst OUTSIDE:224.0.0.5

[...]

Alright, that explains part of what's going on. The ASA is dropping OSPF traffic from INSIDE host R2. This is because, even though our INSIDE interface has security-level 100, multicast traffic is blocked by default. The easiest solution here would be to create a new ACL with 'permit ip any any' and apply it ingress on our INSIDE interface. This shouldn't break any of our rules as technically (multicast aside) INSIDE->OUTSIDE traffic was already allowing everything. Alternatively, would could permit ospf from 10.0.0.2 to [224.0.0.5, 224.0.0.6, 10.0.0.1] and ICMP between Loopbacks. 

ASAv1(config)# access-list INSIDE_in permit ip any any
ASAv1(config)# access-group INSIDE_in in interface INSIDE

ASAv1(config)# end

ASAv1# show access-list INSIDE_in
access-list INSIDE_in; 1 elements; name hash: 0x52aada44
access-list INSIDE_in line 1 extended permit ip any any (hitcnt=1) 0x4c917233
ASAv1# show conn
2 in use, 3 most used

OSPF OUTSIDE 10.0.0.1 INSIDE  224.0.0.5, idle 0:00:07, bytes 22780, flags 
OSPF OUTSIDE 224.0.0.5 INSIDE  10.0.0.2, idle 0:00:01, bytes 1088, flags  

That looks better, but looking at debug ip ospf packet, we're not actually seeing much of a change from the routers' perspective.

R1#debug ip ospf packet
OSPF packet debugging is on
R1#
*Jul  3 17:04:40.021: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 17:04:49.492: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 17:04:59.313: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0

R2#debug ip ospf packet
OSPF packet debugging is on
R2#
*Jul  3 17:04:25.424: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 17:04:34.534: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 17:04:43.792: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#

Issue #2

Which brings us to issue #2, and this one is kind of dirty. Unless you have a keene eye looking through the config, or using capture with trace you might not catch this one, before I give it away, let's look at a capture with trace output (especially since we can't use packet tracer on a transparent firewall). *Note 'clear conn' is actually pretty important, otherwise our trace will only have fast-path information, and we won't see the full packet flow.*

ASAv1# capture in interface INSIDE trace trace-count 1 match ospf any any
ASAv1# show cap
capture in type raw-data trace trace-count 1 interface INSIDE [Capturing - 0 bytes]
  match ospf any any  

!

ASAv1# clear conn
1 connection(s) deleted.

!

ASAv1# show cap in trace detail

2 packets captured

   1: 22:10:39.930249 0cb1.82a8.a800 0100.5e00.0005 0x0800 Length: 102
      10.0.0.2 > 224.0.0.5:  ip-proto-89, length 68 [tos 0xc0]  [ttl 1] (id 20491)
Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ACCESS-LIST
Subtype: log 
Result: ALLOW
Config:
access-group INSIDE_IN in interface INSIDE
access-list INSIDE_IN extended permit ip any any
Additional Information:

Phase: 4
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map class-default
 match any
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: QOS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 20, packet dispatched to next module

Result:
input-interface: INSIDE
input-status: up
input-line-status: up
Action: allow


   2: 22:10:49.253740 0cb1.82a8.a800 0100.5e00.0005 0x0800 Length: 102
      10.0.0.2 > 224.0.0.5:  ip-proto-89, length 68 [tos 0xc0]  [ttl 1] (id 20492)
2 packets shown

Now, that probably looks good at first glance... but take a closer look. A big giagantic hint? Look at the TTL of that OSPF hello, then walk through each phase in the trace.
*Pause for dramatic effect*
Yup. Phase 4, the global policy is set to decrement-ttl. Which means that OSPF packet, while allowed, is going to get dropped due to the ttl being 0. Now, in a perfect world, 'show asp drop frame ttl-exceeded' would show this too. However, at least on my ASAv, it does not. So let's change the global policy.

ASAv1#  conf t
ASAv1(config)# policy-map global_policy
ASAv1(config-pmap)#  class class-default
ASAv1(config-pmap-c)# no   set connection decrement-ttl
ASAv1(config-pmap-c)# end
ASAv1# clear conn
2 connection(s) deleted.

Now let's check the output on R1 and R2.

R1#
*Jul  5 22:23:12.245: %OSPF-5-ADJCHG: Process 1, Nbr 10.2.2.2 on GigabitEthernet1 from LOADING to FULL, Loading Done
R1#show ip route ospf | beg ^Gateway
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O        10.2.2.1/32 [110/2] via 10.0.0.2, 00:01:51, GigabitEthernet1

R2#
*Jul  5 22:23:12.217: %OSPF-5-ADJCHG: Process 1, Nbr 10.1.1.2 on GigabitEthernet1 from LOADING to FULL, Loading Done
R2#show ip route ospf | beg ^Gateway
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O        10.1.1.1/32 [110/2] via 10.0.0.1, 00:02:17, GigabitEthernet1


Much better, OSPF is up but now we have a brand new error related to BGP.

R1#
*Jul  5 22:25:12.344: %TCP-6-BADAUTH: Invalid MD5 digest from 10.2.2.1(31914) to 10.1.1.1(179) tableid - 0

R2#
*Jul  5 22:25:29.900: %TCP-6-BADAUTH: Invalid MD5 digest from 10.1.1.1(43305) to 10.2.2.1(179) tableid - 0

CCIE Security: Troubleshooting (Ticket #1)

So in getting ready for the CCIE Security lab this year, I've been spending some time trying to come up with my own troubleshooting scenarios. The process for this, if you're curious, is normally born from practicing config and noting issues that come up when I misconfigure something. Then I'll try to layer it with other similar or related issues. Alright, so let's write this one up, standard sort of CCIE-ish rules apply.

• Changes must be specific to issue you're troubleshooting (i.e. Make your changes as specific as possible)
• Do not remove any security related configuration. Only adjust or add configuration to correct issues.

Scenario:

 

R1 and R2 are connected via ASAv1, running in transparent mode. The two routers have (2) loopback interfaces, Loopback1 and Loopback2 addressed as 10.x.x.1/32 and 10.x.x.2/32 where X is the router number. Loopback1 should be learned via OSPF, and is used for BGP peering whereas Loopback2 is advertised by BGP. Neither OSPF or BGP peerings are successfully forming, troubleshoot and resolve so that both routers establish OSPF and BGP peerings. Ping both of R1's loopback interfaces from both of R2's loopback interfaces to confirm you've resolved the issue. Additionally, match the output below to both routers.

Super Complex Network Diagram

Download Intitial Configs

R1 Output:

R1#show ip route | inc ^B|^O                        
O        10.2.2.1/32 [110/2] via 10.0.0.2, 00:09:48, GigabitEthernet1
B        10.2.2.2/32 [20/0] via 10.2.2.1, 00:08:54
!
R1#show bgp ipv4 unicast neighbors 10.2.2.1 | inc md5
Option Flags: nagle, path mtu capable, md5, Retrans timeout 

R2 Output:

R2#show ip route | inc ^B|^O                        
O        10.1.1.1/32 [110/2] via 10.0.0.1, 00:11:47, GigabitEthernet1
B        10.1.1.2/32 [20/0] via 10.1.1.1, 00:10:52
!
R2#show bgp ipv4 unicast neighbors 10.1.1.1 | inc md5
Option Flags: nagle, path mtu capable, md5 


Alright interwebs, have at and let me know what you think. Find the solution(s) here.

This Blog is getting an overhaul

I'm changing the name, and theme of my blog and website. NetworkKnerd is no more, and I'm (slowly) moving everything to Hop16. Mostly because I was tired of NetworkKnerd, but also because my focus on Routing and Switching has been less and less over the years as I've transitioned into a more network security focused role professionally. I'll keep this post short and sweet, just to serve as a heads up so when re-directs go into effect in the next couple of weeks there's some reference as to what's going on.