Cisco ISE and Azure AD - Part 1

COVID times have been strange for all of us I'm sure. One of the strangest things to emerge from my customer base during these times was a desire to authenticate users in Azure Active Directory with ISE. Some for VPN authentication, and even a couple of requests for 802.1X authentication. So that desire lead me down two different paths, and I'll cover the first (slightly more involved and less obvious) path in this post. That path can briefly be summed up as "Azure AD as an authentication source, over the Internet".

Ingredients:

1. Azure Active Directory Domain Services
2. Cisco ISE, duh.
3. Cisco AnyConnect Network Access Manager (NAM).

Starting State

In the interest of not pretending this is an Azure blog, or that I'm an Azure expert, I'm not going to cover some of the basics in this setup. The assumption will be you have access Azure Active Directory, have configured your desired custom domain name (not required, but definitely looks nicer) and have a basic understanding about how subscriptions work and how to create resources. I'll cover some of this, but I'm trying really hard to keep this focused on ISE + Azure AD, not get to deep into the nuances of Azure AD itself.

1. Azure Active Directory Domain Services (AADDS)

Alright, so many of you know setting up an Azure directory creates an Active Directory for you store users and groups. However to actually interact with it, we'll need AADDS. This resource spins up two domain controllers, allowing for AD joins, LDAP binds, etc. To create this resource, search for 'Azure AD Domain Services'.

The process for creating this resource is much like spinning up any resource in Azure, and is largely next/next/finish. For later labs, I would be careful to create a VNET that doesn't overlap with any on-prem subnets. More on that in Part 2 though.

After creation you'll be able to view a fairly basic domain services configuration. There's some good info, like the private IPs of the Domain Controllers that were spun up as part of enabling this resource, but what we're interested in is enabling Secure LDAP.

This will require a server certificate to be uploaded in PFX format, the requirements for which Microsoft outlines here. To create this certificate I'll use openSSL to generate a wildcard csr for the domain of hop16.com, and sign it with a CSR1Kv I have configured as a certificate authority.

1. Generate CSR with OpenSSL

~$ openssl genrsa -out ldaps.key 2048

~$ openssl req -new -key ldaps.key -out ldaps.csr

[output] 

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:North Carolina
Locality Name (eg, city) []:Raleigh
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Hop16
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []:*.hop16.com
Email Address []:admin@hop16.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

[/output]

~$ cat ldaps.csr  

[output] 

-----BEGIN CERTIFICATE REQUEST-----
MIIC0TCCAbkCAQAwgYsxCzAJBgNVBAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBDYXJv
[...]

-----END CERTIFICATE REQUEST----- 

[/output]

 

 2. Configure CSR as Certificate Authority

conf t
!
crypto key generate rsa gen mod 4096 label hop16-CA
!
crypto pki server hop16-CA
!
crypto pki server hop16-CA
database level names
no database archive
grant auto
hash sha256
eku server-auth client-auth
!
# Note eku config here, our templates for both parts need #server and client auth.
!
no shutdown
!
end
!
(commands run in privilege exec, not config from here)

crypto pki server hop16-CA request pkcs10 terminal pem 

[output] 

PKCS10 request in base64 or pem

% Enter Base64 encoded or PEM formatted PKCS10 enrollment request.
% End with a blank line or "quit" on a line by itself.
-----BEGIN CERTIFICATE REQUEST-----
MIIC0TCCAbkCAQAwgYsxCzAJBgNVBAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBDYXJv
[...]
-----END CERTIFICATE REQUEST-----
quit
% Granted certificate:
-----BEGIN CERTIFICATE-----
MIIEezCCAmOgAwIBAgIBAjANBgkqhkiG9w0BAQsFADATMREwDwYDVQQDEwhob3Ax
[...]
-----END CERTIFICATE----- 

[/output]

3. Create PFX file with openSSL

To do this we'll also need the CA's certificate, which we can retrieve in PEM format from the CSR with 'show crypto pki cert pem hop16-CA'. Then we can copy that also with our issued certificate back to our computer running openSSL (WSL for me).

~$ openssl pkcs12 -export -out ldaps.pfx -inkey ldaps.key -in ldaps.crt \
-certfile ca.crt
Enter Export Password:
Verifying - Enter Export Password:

4. Enable Secure LDAP

Back in our Azure portal we'll navigate back to Azure AD Domain Services | Secure LDAP and toggle Secure LDAP to 'Enable', toggle Allow secure LDAP access over the internet to 'Enable', upload our freshly created PFX and decryption password then click save. Fair warning, if your template doesn't have to correct EKUs, the CA certificate is missing from the chain or any other similar Certificate issues this process will fail. Speaking 100% from experience here lol.

Also note the message about needing to modify the network security group associated with your AADDS subnet.

Once this process completes successfully, you can pull the public IP for Secure LDAP from Azure AD Domain Services -> Properties.

We'll either want a DNS record (e.g. ldaps[.]yourdomain[.]com or aadds[.]yourdomain[.]com) or we can had a host record in ISE if we're in a pinch. Since we're using a wildcard certificate you can use whatever DNS name makes the most sense to you. For this post I'm going with ldaps.hop16.com. We also need to make sure LDAPS is allowed inbound on our NSG (as highlighted in the previous screen shot).

2. ISE Configuration

Most of the ISE configuration is pretty straight forward, we're going to make sure the CA certificate from our CSR1000v is a trusted CA, configure an external ID source that points to Azure Secure LDAP, then make a quick tweak to the subject name attribute. So let's jump in!

1. Navigate to Administration -> System | Certificates -> Trusted Certificates and upload CSR CA certificate.

2. After clicking submit, navigate to Administration -> Identity Management | External Identity Sources -> LDAP and click 'Add' to add an LDAP Identity Source. Most of the settings are fairly intuitive, but the part you'll really want to pay special attention to is setting the schema to Active Directory, but changing the 'subnet name attribute' to sAMAccountName (default is userPrincipalName). Doing this will change the schema from Active Directory to Custom, but leaves all the defaults from Active Directory.

3. Now we'll configure our Secure LDAP connection, specifiying a bind account as well. I've previously added a user in my Azure AD called 'ldapbind' that has basic user permissions.

4. Fill in your appropriate Directory Organization settings, I'm opting to strip anything before "\" in user authentications in a (mostly hopeless) attempt to keep ISE from trying to authenticate users as "azuread\un".

5. Now add groups you plan to use in Authorization Policies later.

6. Add some attributes for ISE to retrieve, most importantly sAMAccountName and memberOf.

7. Finally, we can click submit.

8. I already have my c3750-X added as a Network Device, using shared secret 'supersecretsharedpasswd'. Also I have a policy-set setup to catch all Wired dot1X and Wired MAB authentications.

9. Inside that policy-set, I'm going to modify the authentication policy to point only to LDAP. You could also just add your LDAP source to an existing identity source sequence.

10. I'm also going to setup a basic rule in my authorization policy to allow network access to members of the AAD Domain Users group.

3. Super Quick Switch Config.

Here's a quick and dirty dot1X config for a 3750-X, with my test PC connected to Gig1/0/48.

aaa new-model
aaa group server radius ISE27_RADIUS
 server-private 192.168.45.10 auth-port 1812 acct-port 1813 key supersecretsharedpasswd
!
aaa authentication login default local
aaa authentication dot1x default group ISE27_RADIUS
aaa authorization exec default local
aaa authorization network default group ISE27_RADIUS
aaa accounting dot1x default start-stop group ISE27_RADIUS
aaa accounting network default start-stop group ISE27_RADIUS
aaa server radius dynamic-author
 client 192.168.45.10 server-key supersecretsharedpasswd
!
dot1x system-auth-control
!
radius-server vsa send accounting
radius-server vsa send authentication
radius-server attribute 6 on-for-login-auth
radius-server attribute 8 include-in-access-req
radius-server attribute 25 access-request include
!
interface GigabitEthernet1/0/48
 switchport mode access
 authentication host-mode multi-auth
 authentication order mab dot1x
 authentication priority dot1x mab
 authentication port-control auto
 authentication periodic
 authentication violation restrict
 dot1x pae authenticator
 spanning-tree portfast

4. AnyConnect NAM Configuration

Now we reach the downside of using Secure LDAP (or LDAP in general for that matter). We only have two supported authentication methods, EAP-TLS or EAP-GTC. EAP-TLS would require client certificates, but would be supported by Window's native supplicant. EAP-GTC supports password authentication, but we need NAM to use it. So, we'll setup a profile for NAM to use PEAP-EAP-GTC, luckily this isn't particularly difficult. I have Network Access Manager Profile Editor installed locally, however you could configure a policy and push it out to other clients. I'm editing the default wired connection, and saving this as configuration.xml in 'C:\ProgramData\Cisco\Cisco AnyConnect Secure Mobility Client\Network Access Manager\newConfigFiles'. Revelant screen shots below:

PEAP-EAP-GTC

Use SSO Credentials

After saving this profile as configuration.xml in newConfigFiles as mentioned above, we have our moment of truth!

AnyConnect looking happy

ISE looking happy

And as a final verification, we'll check what the switch sees.

switch# show authentication sessions int gig1/0/48 details
            Interface:  GigabitEthernet1/0/48
          MAC Address:  000a.f721.edff
         IPv6 Address:  Unknown
         IPv4 Address:  192.168.45.100
            User-Name:  jon
               Status:  Authorized
               Domain:  DATA
       Oper host mode:  multi-auth
     Oper control dir:  both
      Session timeout:  3600s (local), Remaining: 1873s
       Timeout action:  Reauthenticate
    Common Session ID:  000000000000005529C90DB6
      Acct Session ID:  0x00000022
               Handle:  0x4B000034
       Current Policy:  POLICY_Gi1/0/48

Local Policies:
        Service Template: DEFAULT_LINKSEC_POLICY_SHOULD_SECURE (priority 150)
      Security Policy:  Should Secure
      Security Status:  Link Unsecure

Server Policies:

Method status list:
       Method           State
       dot1x            Authc Success

Well, that about does it for this post! Next time we'll look at actually connecting to Azure AD as though it were a traditional Active Directory environment.

Cisco IOS & SNMP: A backdoor into devices you can't access.

We've all been there. You desparately need to get into a router or a switch, but every username and password you try doesn't seem to work. Or even worse, you can ping the device, see it in CDP but SSH isn't replying and telnet is disabled (presumably because the device doesn't have an RSA key). If you were close to the device, sure, you could walk over to it and try to console in. What if that's not an option? Reload and do password recovery?? Well, maybe not. In today's super quick post I want to give you an alternative if SNMP is enabled and you know the rw community string.

Ingredients:

1. Router you can't log into.
1. SNMP read-write community string
2. SNMP tools installed on your machine. (e.g. net-snmp for Windows)
3. TFTP Server running on your machine.
4. Deep breaths. Seriously, calm down. You're locked out of a router, it's not the end of the world or anything Eddie. Yeah, I'm calling you out Eddie.

Don't Panic

Now that you're definitely not panicking, let's open a command prompt on our machine. Since this is a lab, I'm actually using an Ubuntu docker image in GNS3, however the actual process and syntax doesn't change at all if you're running net-snmp in Windows. Where I'm in bash, you'll be in CMD issuing the exact same commands. My TFTP server is also running on this Ubuntu docker image, but again, this works absolutely fine with SolarWind's tftp server or tftpd64. Finally my device is a router named R1 running at 192.168.122.211, and my Ubuntu machine is 192.168.122.93. Let's first confirm the state of things.

root@LX1:~# ssh admin@192.168.122.211     
Password:
Password:
Password:
admin@192.168.122.211's password:
Connection to 192.168.122.211 closed by remote host.
Connection to 192.168.122.211 closed.
root@LX1:~#  


Yup. Good and locked out. So the first thing I'll need to do is create a file in my TFTP directory that contains the changes I wish to make. For this post I'm just going to change the username admin's password to cisco123. However, get creative with this. You could just as easily modify vty lines to permit telnet temporarily, or disable AAA if it's pointing to TACACS/RADIUS for authentication. So I'll create a small file called 'configme.txt' and put said file in my TFTP root directory. Important note here, you don't need to put the full running config in this file, only the changes you wish to merge with existing config on the router. That is to say, this doesn't overwrite and replace the entire running-config, say it with me Eddie "merge". Here's what my file looks like.

root@LX1:~# cat /var/TFTP-ROOT/configme.txt
username admin privi 15 secret cisco123
!
end
root@LX1:~# 

Putting the word "end" on a line all by itself let's the router know it's reached the end of the config file. You don't have to do this, but if you don't the router generates an error saying the config file ended unexpectedly. Alright, now that we have all that setup, and our tftp server is running, let's send our router some commands via SNMP and tell it to download 'configme.txt' and merge it into our running-config. Lucky for me, I just so happen to know this device has 'write' as a read-write community string.

root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.2.111 i 1

Created directory: /var/lib/snmp/mib_indexes
iso.3.6.1.4.1.9.9.96.1.1.1.1.2.111 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.3.111 i 1

iso.3.6.1.4.1.9.9.96.1.1.1.1.3.111 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.4.111 i 4

iso.3.6.1.4.1.9.9.96.1.1.1.1.4.111 = INTEGER: 4
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.5.111 a 192.168.122.93

iso.3.6.1.4.1.9.9.96.1.1.1.1.5.111 = IpAddress: 192.168.122.93
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.6.111 s configme.txt

iso.3.6.1.4.1.9.9.96.1.1.1.1.6.111 = STRING: "configme.txt"
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.14.111 i 1

iso.3.6.1.4.1.9.9.96.1.1.1.1.14.111 = INTEGER: 1

Now let's quickly confirm everything works as expected.

root@LX1:~# ssh admin@192.168.122.211 
Password:


R1#show ip int br | ex unass
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       192.168.122.211 YES DHCP   up                    up     

R1#show logging | inc tftp
Jul 14 15:40:49.808: %SYS-5-CONFIG_I: Configured from tftp://192.168.122.93/configme.txt by console
R1#exit
Connection to 192.168.122.211 closed.

What just happened?

All in all we sent (6) SNMP strings to the router, I'll really quickly break those out here.

1. "1.3.6.1.4.1.9.9.96.1.1.1.1.2.[random_number] i 1" sets the transfer protocol to TFTP.
2. "1.3.6.1.4.1.9.9.96.1.1.1.1.3.[random_number] i 1" sets the source file as a network file
3. "1.3.6.1.4.1.9.9.96.1.1.1.1.4.[random_number] i 4" sets the destination as running-config
4. "1.3.6.1.4.1.9.9.96.1.1.1.1.5.[random_number] a 192.168.122.93" sets the TFTP Server Address
5. "1.3.6.1.4.1.9.9.96.1.1.1.1.6.[random_number] s configme.txt" sets the file name
6. "1.3.6.1.4.1.9.9.96.1.1.1.1.14.[random_number] i 1" initiates the transfer.

If you look above, my [random_number] was 111, now if I wanted to issues these commands again, maybe to change the file name or TFTP server address, I'd need to pick a new random number. Just re-running those MIBs with the same random number as before will error out as seen here:

root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.5.111 a 192.168.122.93
Error in packet.
Reason: inconsistentValue (The set value is illegal or unsupported in some way)
Failed object: iso.3.6.1.4.1.9.9.96.1.1.1.1.5.111

However, if I change .111 to .112 we're back in business.

root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.2.112 i 1

iso.3.6.1.4.1.9.9.96.1.1.1.1.2.112 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.3.112 i 1

iso.3.6.1.4.1.9.9.96.1.1.1.1.3.112 = INTEGER: 1
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.4.112 i 4

iso.3.6.1.4.1.9.9.96.1.1.1.1.4.112 = INTEGER: 4
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.5.112 a 192.168.122.93

iso.3.6.1.4.1.9.9.96.1.1.1.1.5.112 = IpAddress: 192.168.122.93
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.6.112 s configme.txt

iso.3.6.1.4.1.9.9.96.1.1.1.1.6.112 = STRING: "configme.txt"
root@LX1:~# snmpset -c write -v 2c 192.168.122.211 1.3.6.1.4.1.9.9.96.1.1.1.1.14.112 i 1

iso.3.6.1.4.1.9.9.96.1.1.1.1.14.112 = INTEGER: 1
root@LX1:~#


Best of luck getting back into your devices, that's it for this post.

FlexVPN: Spoke-2-Spoke PSK

So in my last post I covered configuring FlexVPN with dynamic spoke to spoke communication using certificates for authentication. What if you wanted to use pre-shared-keys though? We can do that, it's not nearly as secure as certificate based authentication, but we can do that.

We'll be using the exact same topology as the last post, and seeing as I went through the config in a good bit of detail in the last post, I'll keep this one short and sweet trying to only spend time on the differences in config.

Ingredients:

• (3) Cisco Routers. I'll be using CSR1000v running 16.03.01.
• Patience. 

Estimate lab time: ~15min

FlexVPN Topology

FlexVPN Configuration

In the interest of exploring some more interesting features in IKEv2 with respect to pre-shared-keys, I thought we'd up the ante a bit and do both asymmetric authentication between Hub and Spokes, as well as symmetric authentication between spokes. That means we'll need an ikev2 keyring.  Without further to do, let's look at the Hub configuration. I'll call out the biggest differences in red from my last post.

Hub Router:

aaa new-model
aaa authorization network flexauthz local
aaa session-id common
!
ip local pool flexpool 10.254.0.10 10.254.0.254
!
ip access-list standard flexroutes
 permit 10.0.0.0 0.255.255.255
!
crypto ikev2 authorization policy flexauthzpol
 pool flexpool
 route set access-list flexroutes
!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20
!
crypto ikev2 policy flexpol
 proposal flexprop
!
crypto ikev2 keyring flexkeys
 peer spokes
  address 0.0.0.0 0.0.0.0
  pre-shared-key local HubsLikeCerts
  pre-shared-key remote SoDoSpokes
 !
crypto ikev2 profile flexprof
 match identity remote address 0.0.0.0
 authentication remote pre-share
 authentication local pre-share
 keyring local flexkeys
 aaa authorization group psk list flexauthz flexauthzpol
 virtual-template 1
!
crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel
!
crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprof
!
interface Virtual-Template1 type tunnel
 ip unnumbered Loopback1
 ip nhrp network-id 1337
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect

So not too shabby at all right? No PKI to worry about, all we have to do here is define a keyring and call that keyring in our IKEv2 profile. On the spokes we'll see a litte more configuration within the keyring to meet the requirements I defined above (asymm for hub<->spokes and symm spoke<->spoke). Now let's look at the spoke config, this configuration is identical on both S2 and S3 routers, except the ACL changes. S2 has permit 10.0.2.0 0.0.0.255 and S3 has permit 10.0.3.0 0.0.0.255.

Spoke Routers:

aaa new-model
aaa authorization network flexauthz local
aaa session-id common
!
ip access-list standard flexroutes
 permit 10.0.x.0 0.0.0.255
 # ACL is the only part of the config that differs spoke to spoke.
!
crypto ikev2 authorization policy flexauthzpol
 route set access-list flexroutes
!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20
!
crypto ikev2 policy flexpol
 proposal flexprop
!
crypto ikev2 keyring flexkeys
 peer hub
  address 169.254.0.5
  pre-shared-key local SoDoSpokes
  pre-shared-key remote HubsLikeCerts
 !
 peer spokes
  address 0.0.0.0 0.0.0.0
  pre-shared-key Spokes<3Certs
 !
crypto ikev2 profile flexprof
 match identity remote address 0.0.0.0
 authentication remote pre-share
 authentication local pre-share
 keyring local flexkeys
 aaa authorization group psk list flexauthz flexauthzpol
 virtual-template 1
!
crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel
!
crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprof
!
interface Tunnel0
 ip address negotiated
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel destination 169.254.0.5
 tunnel protection ipsec profile flexprotect
!
interface Virtual-Template1 type tunnel
 ip unnumbered Tunnel0
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect

So as you can see, using pre-shared-keys isn't too different from a configuration stand point than certificate based authentication. However, in the real world, what you're giving up is the ability to revoke a single router's creditials. Following a certificate auth model, if a device was lost or compromised, we could revoke it's certificate and once all devices got an up to date CRL, they'd reject any authentication attempt from that device. With pre-shared-keys we'd have to touch each one of our devices to update PSKs to achieve the same goal. No bueno. That's enough lecturing, let's do some quick verfication from S2's router to make sure it can both talk to the Hub and build a dynamic tunnel with S3.

Verification on S2

S2#show ip int br | ex unass
Interface              IP-Address      OK? Method Status                Protocol
GigabitEthernet1       169.254.2.5     YES NVRAM  up                    up     
GigabitEthernet2       10.0.2.1        YES NVRAM  up                    up     
Tunnel0                10.254.0.10     YES manual up                    up     
Virtual-Template1      10.254.0.10     YES unset  up                    down   
!
!
S2#show ip route static | beg ^Gate
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0
!
S2#show crypto ikev2 sa
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/1224 sec
!
S2#ping 10.0.3.1 source 10.0.2.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.3.1, timeout is 2 seconds:
Packet sent with a source address of 10.0.2.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/6/7 ms
S2#
*Jul 11 19:25:26.917: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to down
S2#
*Jul 11 19:25:27.160: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to up
S2#show ip route static | beg ^Gateway
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 6 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0
S   %    10.0.3.0/24 is directly connected, Virtual-Access1
S2#show crypto ikev2 sa
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
3         169.254.2.5/500       169.254.3.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/19 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/1262 sec

And boom goes the dynamite. That's all for this one, happy labbing.

FlexVPN: Spoke-2-Spoke Tunnels

Let's talk about FlexVPN, a prime contender as a DMVPN replacement and sometimes referred to as DMVPN phase 4. In this post, I'm going to explore the nuts and bolts of getting FlexVPN up and running between (3) routers and, for added flare, I'm going to also configure dynamic spoke-2-spoke tunnels. The dynamic spoke-2-spoke portion of the config will look very similar to those of you familiar with DMVPN, with some key differences. Before diving in, let's take a look at our diagram. I'd like to also point out in this lab, the Hub is also our certificate authority (because pre-shared keys are for suckers).

Ingredients

• (3) Cisco Routers. I'll be using CSR1000v running 16.03.01.
• Patience.

Estimate lab time: ~30min

 

Configuring IOS CA & Provision Certificates

 

Since we'll be using RSA for authentication, as opposed to pre-shared keys, we're going to need a trusted certificate authority and for each router to have it's own certificate issued from said CA. Now, in real life this can be either an online or offline CA running on IOS, Linux, or Windows. However, to keep things simple, in this lab I'm just going to configure the Hub router as the CA, then have all routers request a certificate from the CA.

IOS CA Configuration:

ip domain name hop16.com
!
! Replace 12:00:00 10 JUL 2019 with current time and date
do clock set 12:00:00 10 JUL 2019
!
!
ntp master 3
!
crypto pki server flexvpnca
 database level complete
 no database archive
 issuer-name CN=Hub.hop16.com,OU=NOC,L=FOO,C=US
 hash sha256
 database url flash:
 grant auto
 no shutdown
%Some server settings cannot be changed after CA certificate generation.
% Please enter a passphrase to protect the private key
% or type Return to exit
Password: 
Re-enter password: 
% Generating 1024 bit RSA keys, keys will be non-exportable...
!

All Routers (yes, even the Hub/IOS CA from above)

ip domain name hop16.com
!
crypto key generate rsa modulus 2048 label FLEXKEYS
!
crypto pki trustpoint flexvpn-tp
 enrollment url http://169.254.0.5:80
 hash sha256
 subject-name CN={ChangeForEachSite}.hop16.com,OU=NOC,L=FOO,C=US
 fqdn {ChangeForEachSite}.hop16.com
 # e.g. subject-name CN=S2.hop16.com,OU=NOC,L=FOO,C=US for S2
 # e.g. subject-name CN=S3.hop16.com,OU=NOC,L=FOO,C=US for S3
 revocation-check crl
 rsakeypair FLEXKEYS
!
crypto pki authenticate flexvpn-tp
% Do you accept this certificate? [yes/no]: yes
!
crypto pki enroll flexvpn-tp
 # This command will prompt you to create a password for revocation,
 # and some additional options about including router serial number,
 # and IP address in the certification subject name. Both are
 # optional, I opted to include the router's IP address but
 #exclude the serial number.

After a couple seconds, you'll see a prompt on each router confirming they've received their cert from the CA "%PKI-6-CERTRET: Certificate received from Certificate Authority". At this point, assuming you haven't hit any errors, we should be able to verify we have a valid certificate from our CA with 'show crypto pki certificates flexvpn-tp'.


S2#show crypto pki certificates 
Certificate
  Status: Available
  Certificate Serial Number (hex): 03
  Certificate Usage: General Purpose
  Issuer:
    cn=Hub.hop16.com
    ou=NOC
    l=FOO
    c=US
  Subject:
    Name: S3.hop16.com
    IP Address: 169.254.3.5
    ipaddress=169.254.3.5+hostname=S3.hop16.com
    cn=S3.hop16.com
    ou=NOC
    l=FOO
    c=US
  Validity Date:
    start date: 14:26:30 UTC Jul 10 2019
    end   date: 14:26:30 UTC Jul 9 2020
  Associated Trustpoints: flexvpn-tp
  Storage: nvram:Hubhop16com#3.cer

CA Certificate
  Status: Available
  Certificate Serial Number (hex): 01
  Certificate Usage: Signature
  Issuer:
    cn=Hub.hop16.com
    ou=NOC
    l=FOO
    c=US
  Subject:
    cn=Hub.hop16.com
    ou=NOC
    l=FOO
    c=US
  Validity Date:
    start date: 14:20:00 UTC Jul 10 2019
    end   date: 14:20:00 UTC Jul 9 2022
  Associated Trustpoints: flexvpn-tp

Configuring FlexVPN - Hub and Spoke

The biggest downside to FlexVPN in my humble opinion is that it's ssoooo config intensive. The benefit is that once you get the configs ironed out, it's incredibly easy to template. Most everything (including the vast majority of the Hub configs) can be copy/paste to other routers. That said, there are a couple bits of config unique to the Hub, and I'll call those out as we go.

Hub Router:

aaa new-model
aaa authorization network flexauthz local
aaa session-id common
!
! Defining a local pool is only needed on the Hub router, and only
! required if you wish to have the spokes get their tunnel IP
! dynamically which is preferred.
!
ip local pool flexpool 10.254.0.2 10.254.0.254
!
ip access-list standard flexroutes
 permit 10.0.0.0 0.255.255.255
 #
 # Note here, normally your route ACL only has local routes, but we
 # need to be more general on the Hub so S2 knows how to reach S3. 
 #
!
crypto ikev2 authorization policy flexauthzpol
 # Using an address pool here so our spokes can negotiate a tunnel
 # IP from the Hub.
 pool flexpool
 route set access-list flexroutes
!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20
!
crypto ikev2 policy flexpol
 proposal flexprop
!
crypto ikev2 profile flexprofile
 match identity remote fqdn domain hop16.com
 identity local fqdn Hub.hop16.com
 authentication remote rsa-sig
 authentication local rsa-sig
 pki trustpoint flexvpn-tp
 aaa authorization group cert list flexauthz flexauthzpol
 virtual-template 1
#
# Calling out a virtual-template within our IKEv2 profile is only
# needed if the router has a DVTI interface, which in our initial
# configurations will only be the Hub. Also, if you forget to
# specify this here, your Hub won't work and you'll start
# pulling your hair out troubleshooting IKEv2/IPsec messages 
# (which will all report successful authentications). Maddening.
#
crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel
!
crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprofile
!
interface Loopback1
 ip address 10.254.0.1 255.255.255.0
!
interface Virtual-Template1 type tunnel
 # Again, DVTI interface is only required on the Hub
 # while we're doing a Hub and spoke topology.
 ip unnumbered Loopback1
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect
!

Easy peasy right? It's a lot of config... like all of the config lol. We could trim this down a little if we were doing PSK with an IKEv2 keyring (follow up blog post for that). However, I like doing certificate based authentication so here we are.

Moving on to the spoke router configs, let's look to S2 as an example. I'll call out the bits in red that will need to change before copy/pasting this config to S3's router.

Spoke Router (S2):

ip access-list standard flexroutes
 permit 10.0.2.0 0.0.0.255
!
aaa new-model
aaa authorization network flexauthz local
aaa session-id common
!
crypto ikev2 proposal flexprop
 encryption aes-gcm-256
 prf sha256
 group 20
!
crypto ikev2 policy flexpol
 proposal flexprop
!
crypto ikev2 profile flexprofile
 match identity remote fqdn domain hop16.com
 identity local fqdn S2.hop16.com
 authentication remote rsa-sig
 authentication local rsa-sig
 pki trustpoint flexvpn-tp
 aaa authorization group cert list flexauthz flexauthzpol
!
crypto ipsec transform-set flextset esp-gcm 256
 mode tunnel
crypto ipsec profile flexprotect
 set transform-set flextset
 set ikev2-profile flexprofile
!
interface Tunnel0
 ip address negotiated
 tunnel source GigabitEthernet1
 tunnel destination 169.254.0.5
 tunnel protection ipsec profile flexprotect


See how easy that is to template out though? Absolutely nuts, and we could potentially make this even more generic with only injecting routes to our local interfaces (route set interface instead of route set access-list) and running EIGRP or BGP over the tunnels. Moving on with this config, router S3 has an identical config, except the IKEv2 local identity is S3.hop16.com and my access-list flexroutes is permitting 10.0.3.0 0.0.0.255. Now that our Hub and Spoke FlexVPN is up and running, let's see what it takes to convert that to a mesh style VPN topology like DMVPN.

Configuring FlexVPN - Mesh (DMVPN Phase 4)

Finally we're getting at the subject of this post, spoke to spoke dynamic tunnels. The mechanism for getting spoke to spoke communication is the very same used in DMVPN deployments, NHRP. With some key changes.

• No NHRP Next-Hop Server (NHS)
• Instead mapping NBMA to Tunnel IP is derived from IKEv2
• Spokes use SVTI (Tunnel0 interface) for communication to the Hub
• Spokes use DVTI (Virtual-Template) for dynamic tunnels between one another. 

Now, in order to make the required changes to enable NHRP we have to shutdown our SVTI (Tunnel0) interfaces on the spokes. Otherwise the Hub's virtual-template will be locked and IOS will prevent us from making any changes. After shutting down Tunnel0 on both routers, we can add the following to our Hub.

Hub Router:

interface Virtual-Template1 type tunnel
 ip nhrp network-id 1337
 ip nhrp redirect 

Believe it or not, that's actually we need to add on the Hub side. We're enabling NHRP redirects, just like DMVPN Phase 3, so the Hub can signal the Spokes to build dynamic tunnels. Also just like DMVPN, the first packet or two will transit the Hub while the Spokes are receiving NHRP information via the redirect message and building their dynamic tunnels. Alright, let's look at the required config on our Spokes.

Spoke Routers:

crypto ikev2 profile flexprofile
 virtual-template 1
!
interface Virtual-Template1 type tunnel
 no ip address
 ip unnumbered Tunnel0
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect
 tunnel source GigabitEthernet1
 tunnel protection ipsec profile flexprotect
!
interface Tunnel0
 ip nhrp network-id 1337
 ip nhrp shortcut virtual-template 1
 ip nhrp redirect

Much like most things FlexVPN, while easily templated and generic, it's configuration intensive. That said, this is a super interesting configuration. Where in DMVPN all the traffic happens on our multipoint Tunnel interface, in FlexVPN the tunnel interface remains a static P2P tunnel destined to the Hub. Then we're pushing our NHRP shortcut to a seperate virtual-template (DVTI) that is will build P2P virtual-access interfaces to other spokes. If we had multiple dynamic tunnels to other spokes each would get their own Virtual-Access interface. Alright, let's bring the tunnels on the spokes back up, and do some verification.

S2#show ip route static | begin ^Gate
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0
S        10.254.0.1/32 is directly connected, Tunnel0
S2#show crypto ikev2 sa             
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY 
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
      Life/Active Time: 86400/26 sec

Now I'm going to enable NHRP debugs, and try to ping 10.0.3.1 which is connected to S3.


S2#debug nhrp
NHRP protocol debugging is on
S2#ping 10.0.3.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.0.3.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/8/11 ms
S2#
 [...]
*Jul 11 11:53:55.363: NHRP: Sending NHRP Resolution Request for dest: 10.0.3.1 to nexthop: 10.0.3.1 using our src: 10.254.0.9 vrf:global(0x0)
*Jul 11 11:53:55.363: NHRP: Attempting to send packet through interface Tunnel0 via DEST  dst 10.0.3.1
*Jul 11 11:53:55.364: NHRP: Send Resolution Request via Tunnel0 vrf global(0x0), packet size: 72
*Jul 11 11:53:55.364:       src: 10.254.0.9, dst: 10.0.3.1
 [...]
*Jul 11 11:53:55.364: NHRP: 96 bytes out Tunnel0
*Jul 11 11:53:55.372: NHRP: Receive Resolution Request via Tunnel0 vrf global(0x0), packet size: 92
*Jul 11 11:53:55.372: NHRP: Route lookup for destination 10.254.0.9 in vrf global(0x0) yielded interface Tunnel0, prefixlen 32
*Jul 11 11:53:55.373: NHRP: Request was to us. Process the NHRP Resolution Request.
*Jul 11 11:53:55.373: NHRP: nhrp_rtlookup for 10.254.0.9 in vrf global(0x0) yielded interface Tunnel0, prefixlen 32
*Jul 11 11:53:55.373: NHRP: Request was to us, responding with ouraddress
*Jul 11 11:53:55.373: NHRP: Checking for delayed event 10.254.0.8/10.254.0.9 on list (Tunnel0 vrf: global(0x0))
*Jul 11 11:53:55.373: NHRP: No delayed event node found.
*Jul 11 11:53:55.374: NHRP: Enqueued Delaying resolution request nbma src:169.254.2.5 nbma dst:169.254.3.5 reason:IPSEC-IFC: need to wait for IPsec SAs.
*Jul 11 11:53:55.374: NHRP: Interface: Tunnel0 configured with FlexVPN. Deferringcache creation for nhop 10.254.0.8
*Jul 11 11:53:55.374: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to down
 [...]
*Jul 11 11:53:55.387:  Instructing NHRP to create Virtual-Access from Virtual template 1 for interface Virtual-Access1
*Jul 11 11:53:55.387: NHRP: NHRP Redirect Feature PI-code Initialized
*Jul 11 11:53:55.387: NHRP: Redirect Feature Initialized - Attempting Platform Init
*Jul 11 11:53:55.393: NHRP: Updating delayed event with destination 169.254.3.5 on interfaceTunnel0 with the new interface Virtual-Access1
*Jul 11 11:53:55.595: NHRP:
*Jul 11 11:53:55.595:  Fetched address from underlying IKEv2 for interfaceVirtual-Access1. Pre-NATed = 169.254.2.5, Post-NATed = UNKNOWN
*Jul 11 11:53:55.595: NHRP: Processing delayed event on interface Tunnel0 with NBMA 169.254.3.5
 [...]
*Jul 11 11:53:55.598: NHRP: No need to delay processing of resolution event nbma src:169.254.2.5 nbma dst:169.254.3.5
*Jul 11 11:53:55.598: NHRP: Attempting to send packet through interface Virtual-Access1 via DEST  dst 10.254.0.8
*Jul 11 11:53:55.598: NHRP: Send Resolution Reply via Virtual-Access1 vrf global(0x0), packet size: 120
*Jul 11 11:53:55.598:       src: 10.254.0.9, dst: 10.254.0.8
*Jul 11 11:53:55.598: NHRP: 144 bytes out Virtual-Access1
*Jul 11 11:53:55.604: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to up
 [...]
!
!
S2#show ip route static | begin ^Gate
Gateway of last resort is 169.254.2.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 169.254.2.6
      10.0.0.0/8 is variably subnetted, 7 subnets, 3 masks
S        10.0.0.0/8 is directly connected, Tunnel0
S   %    10.0.3.0/24 is directly connected, Virtual-Access1
S        10.254.0.1/32 is directly connected, Tunnel0
S   %    10.254.0.8/32 is directly connected, Virtual-Access1
S2#show crypto ikev2 sa
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
2         169.254.2.5/500       169.254.3.5/500       none/none            READY
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
      Life/Active Time: 86400/399 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         169.254.2.5/500       169.254.0.5/500       none/none            READY
      Encr: AES-GCM, keysize: 256, PRF: SHA256, Hash: None, DH Grp:20, Auth sign: RSA, Auth verify: RSA
      Life/Active Time: 86400/443 sec

 IPv6 Crypto IKEv2  SA


I did my best to trim down the NHRP debug messages, but there's a lot of interesting things happening here. Now looking at our static routes, we have the original summary to 10.0.0.0/8 via Tunnel0 to the Hub, but we also have a specific route to 10.0.3.0/24  and a route to 10.254.0.8/32 via Virtual-Access1 directly to router S3 with the % indicating next hop override (due to NHRP). Well that's about it for now, I'll kick out a quick follow up post for those interested in using pre-shared-keys for authentication.

CCIE Security: Troubleshooting (Ticket #1) - Solution

Alright it's been a couple of days since the original post, so after much fanfare and exactly 0 people attempting to solve, let's break this one down.

SPOILER ALERT

Issue #1

Since BGP is relying on OSPF for connectivity between peering interfaces (Loopback1), this seems like a natural place to start. The first thing we'll notice on R1 and R2, is when running debug ip ospf packet, we're only seeing traffic leave each respective router. We're not actually seeing either router receive ospf traffic.

R1#debug ip ospf adj  

OSPF adjacency debugging is on
R1#
*Jul  3 16:14:56.959: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 16:15:06.512: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 16:15:15.996: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0

R2#debug ip ospf packet
OSPF packet debugging is on
R2#
*Jul  3 16:16:34.653: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 16:16:44.567: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 16:16:53.574: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0

So let's go over to the ASA and look at our logs to see if anything looks amiss.

ASAv1(config)# logging buffered 7
ASAv1(config)# logging enable

ASAv1(config)# end

ASAv1# show logging

ASAv1# show logging             
[...]
%ASA-3-106010: Deny inbound protocol 89 src INSIDE:10.0.0.2 dst OUTSIDE:224.0.0.5

[...]

Alright, that explains part of what's going on. The ASA is dropping OSPF traffic from INSIDE host R2. This is because, even though our INSIDE interface has security-level 100, multicast traffic is blocked by default. The easiest solution here would be to create a new ACL with 'permit ip any any' and apply it ingress on our INSIDE interface. This shouldn't break any of our rules as technically (multicast aside) INSIDE->OUTSIDE traffic was already allowing everything. Alternatively, would could permit ospf from 10.0.0.2 to [224.0.0.5, 224.0.0.6, 10.0.0.1] and ICMP between Loopbacks. 

ASAv1(config)# access-list INSIDE_in permit ip any any
ASAv1(config)# access-group INSIDE_in in interface INSIDE

ASAv1(config)# end

ASAv1# show access-list INSIDE_in
access-list INSIDE_in; 1 elements; name hash: 0x52aada44
access-list INSIDE_in line 1 extended permit ip any any (hitcnt=1) 0x4c917233
ASAv1# show conn
2 in use, 3 most used

OSPF OUTSIDE 10.0.0.1 INSIDE  224.0.0.5, idle 0:00:07, bytes 22780, flags 
OSPF OUTSIDE 224.0.0.5 INSIDE  10.0.0.2, idle 0:00:01, bytes 1088, flags  

That looks better, but looking at debug ip ospf packet, we're not actually seeing much of a change from the routers' perspective.

R1#debug ip ospf packet
OSPF packet debugging is on
R1#
*Jul  3 17:04:40.021: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 17:04:49.492: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0
R1#
*Jul  3 17:04:59.313: OSPF-1 PAK  : Gi1: OUT: 10.0.0.1->224.0.0.5: ver:2 type:1 len:44 rid:10.1.1.2 area:0.0.0.0 chksum:D79A auth:0

R2#debug ip ospf packet
OSPF packet debugging is on
R2#
*Jul  3 17:04:25.424: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 17:04:34.534: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#
*Jul  3 17:04:43.792: OSPF-1 PAK  : Gi1: OUT: 10.0.0.2->224.0.0.5: ver:2 type:1 len:44 rid:10.2.2.2 area:0.0.0.0 chksum:D698 auth:0
R2#

Issue #2

Which brings us to issue #2, and this one is kind of dirty. Unless you have a keene eye looking through the config, or using capture with trace you might not catch this one, before I give it away, let's look at a capture with trace output (especially since we can't use packet tracer on a transparent firewall). *Note 'clear conn' is actually pretty important, otherwise our trace will only have fast-path information, and we won't see the full packet flow.*

ASAv1# capture in interface INSIDE trace trace-count 1 match ospf any any
ASAv1# show cap
capture in type raw-data trace trace-count 1 interface INSIDE [Capturing - 0 bytes]
  match ospf any any  

!

ASAv1# clear conn
1 connection(s) deleted.

!

ASAv1# show cap in trace detail

2 packets captured

   1: 22:10:39.930249 0cb1.82a8.a800 0100.5e00.0005 0x0800 Length: 102
      10.0.0.2 > 224.0.0.5:  ip-proto-89, length 68 [tos 0xc0]  [ttl 1] (id 20491)
Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
MAC Access list

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 3
Type: ACCESS-LIST
Subtype: log 
Result: ALLOW
Config:
access-group INSIDE_IN in interface INSIDE
access-list INSIDE_IN extended permit ip any any
Additional Information:

Phase: 4
Type: CONN-SETTINGS
Subtype:
Result: ALLOW
Config:
class-map class-default
 match any
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: QOS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 20, packet dispatched to next module

Result:
input-interface: INSIDE
input-status: up
input-line-status: up
Action: allow


   2: 22:10:49.253740 0cb1.82a8.a800 0100.5e00.0005 0x0800 Length: 102
      10.0.0.2 > 224.0.0.5:  ip-proto-89, length 68 [tos 0xc0]  [ttl 1] (id 20492)
2 packets shown

Now, that probably looks good at first glance... but take a closer look. A big giagantic hint? Look at the TTL of that OSPF hello, then walk through each phase in the trace.
*Pause for dramatic effect*
Yup. Phase 4, the global policy is set to decrement-ttl. Which means that OSPF packet, while allowed, is going to get dropped due to the ttl being 0. Now, in a perfect world, 'show asp drop frame ttl-exceeded' would show this too. However, at least on my ASAv, it does not. So let's change the global policy.

ASAv1#  conf t
ASAv1(config)# policy-map global_policy
ASAv1(config-pmap)#  class class-default
ASAv1(config-pmap-c)# no   set connection decrement-ttl
ASAv1(config-pmap-c)# end
ASAv1# clear conn
2 connection(s) deleted.

Now let's check the output on R1 and R2.

R1#
*Jul  5 22:23:12.245: %OSPF-5-ADJCHG: Process 1, Nbr 10.2.2.2 on GigabitEthernet1 from LOADING to FULL, Loading Done
R1#show ip route ospf | beg ^Gateway
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O        10.2.2.1/32 [110/2] via 10.0.0.2, 00:01:51, GigabitEthernet1

R2#
*Jul  5 22:23:12.217: %OSPF-5-ADJCHG: Process 1, Nbr 10.1.1.2 on GigabitEthernet1 from LOADING to FULL, Loading Done
R2#show ip route ospf | beg ^Gateway
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O        10.1.1.1/32 [110/2] via 10.0.0.1, 00:02:17, GigabitEthernet1


Much better, OSPF is up but now we have a brand new error related to BGP.

R1#
*Jul  5 22:25:12.344: %TCP-6-BADAUTH: Invalid MD5 digest from 10.2.2.1(31914) to 10.1.1.1(179) tableid - 0

R2#
*Jul  5 22:25:29.900: %TCP-6-BADAUTH: Invalid MD5 digest from 10.1.1.1(43305) to 10.2.2.1(179) tableid - 0

CCIE Security: Troubleshooting (Ticket #1)

So in getting ready for the CCIE Security lab this year, I've been spending some time trying to come up with my own troubleshooting scenarios. The process for this, if you're curious, is normally born from practicing config and noting issues that come up when I misconfigure something. Then I'll try to layer it with other similar or related issues. Alright, so let's write this one up, standard sort of CCIE-ish rules apply.

• Changes must be specific to issue you're troubleshooting (i.e. Make your changes as specific as possible)
• Do not remove any security related configuration. Only adjust or add configuration to correct issues.

Scenario:

 

R1 and R2 are connected via ASAv1, running in transparent mode. The two routers have (2) loopback interfaces, Loopback1 and Loopback2 addressed as 10.x.x.1/32 and 10.x.x.2/32 where X is the router number. Loopback1 should be learned via OSPF, and is used for BGP peering whereas Loopback2 is advertised by BGP. Neither OSPF or BGP peerings are successfully forming, troubleshoot and resolve so that both routers establish OSPF and BGP peerings. Ping both of R1's loopback interfaces from both of R2's loopback interfaces to confirm you've resolved the issue. Additionally, match the output below to both routers.

Super Complex Network Diagram

Download Intitial Configs

R1 Output:

R1#show ip route | inc ^B|^O                        
O        10.2.2.1/32 [110/2] via 10.0.0.2, 00:09:48, GigabitEthernet1
B        10.2.2.2/32 [20/0] via 10.2.2.1, 00:08:54
!
R1#show bgp ipv4 unicast neighbors 10.2.2.1 | inc md5
Option Flags: nagle, path mtu capable, md5, Retrans timeout 

R2 Output:

R2#show ip route | inc ^B|^O                        
O        10.1.1.1/32 [110/2] via 10.0.0.1, 00:11:47, GigabitEthernet1
B        10.1.1.2/32 [20/0] via 10.1.1.1, 00:10:52
!
R2#show bgp ipv4 unicast neighbors 10.1.1.1 | inc md5
Option Flags: nagle, path mtu capable, md5 


Alright interwebs, have at and let me know what you think. Find the solution(s) here.

This Blog is getting an overhaul

I'm changing the name, and theme of my blog and website. NetworkKnerd is no more, and I'm (slowly) moving everything to Hop16. Mostly because I was tired of NetworkKnerd, but also because my focus on Routing and Switching has been less and less over the years as I've transitioned into a more network security focused role professionally. I'll keep this post short and sweet, just to serve as a heads up so when re-directs go into effect in the next couple of weeks there's some reference as to what's going on.