Tuesday, November 20, 2018

Firepower Threat Defense AVC and SmartCLI


Congratulations, you've purchased one of Cisco's shiny new Next-Generation Firewalls. It's going inspect more packets, push bits at blazing speed, and finally lockdown your disgraceful network. You've built your access control policy and added a ton of rules allowing traffic outside to your inside zone. Maybe you even took this opportunity to build out that DMZ you've always been talking about. Good for you, I'm happy for you. But, uh, did you restrict which ports were open inside to outside? Did you use AVC to restrict which applications were allowed on the ports you opened or did you just open port 80 and 443 like a child?!


Well, that's why we're here, Derek, to talk about both this underused feature and also about some of the new features in Firepower Device Manager (standalone manager for non-FMC deployments). Time for us all to start firewalling better, all of us... Derek. To start things off, let's look aoutt our starting environment. Built-in GNS3, I'll be using FTDv for KVM running 6.2.3 and a layer3 switch running whocare.png. Honestly, the L3 switch is just so I have another OSPF speaker on net, use whatever you like.

Super Complex Topology

All I've done on the FTD is assign a management IP on the same subnet as FDM-Client and FDM-Server. GigabitEthernet 0/1, which will be my inside interface, connected to SW1 on a separate routed link.

> show network
===============[ System Information ]===============
Hostname                  : FTDv
DNS Servers               : 208.67.222.222
                            208.67.220.220
Management port           : 8305
IPv4 Default route
  Gateway                 : 172.31.45.1

======================[ br1 ]=======================
State                     : Enabled
Channels                  : Management & Events
Mode                      : Non-Autonegotiation 
MDI/MDIX                  : Auto/MDIX 
MTU                       : 1500
MAC Address               : 0C:ED:AF:17:A6:01
----------------------[ IPv4 ]----------------------
Configuration             : Manual
Address                   : 172.31.45.45
Netmask                   : 255.255.255.0
Broadcast                 : 172.31.45.255
----------------------[ IPv6 ]----------------------
Configuration             : Disabled

> show managers 
Managed locally.


So from here, before diving into AVC, I'm just going to get basic addressing knocked out then get connectivity to the outside setup. Along the way, I'll show off using SmartCLI to configure OSPF.

FDM - Basic Setup and Addressing

Of note, I did have some initial issues getting into FDM on this GNS3 appliance. When trying to access the WebUI I was receiving 503 Service Unavailable. To resolve this, I went to the CLI and issued 'configure manager delete' followed by 'configure manager local'.

When you first log into FDM, you're prompted to complete Device Setup which configures the outside interface, a default policy rule, DNS servers for your management interface, and also gives you the option to change your hostname. The next two pages you can configure time settings, and add Smart Licensing. This is a lab, so I'm just going to use a 90-day trial. Once complete you'll arrive at a page that looks similar to below.

FDM Quick Setup complete




Next, we're going to configure our inside interface with an IP address 172.31.99.1/29. To do that I'll click on "3 Enabled" under Interfaces. Then select edit for interface GigabitEthernet0/1. From here I'll need to both change the IP address as well as delete the DHCP configurations to successfully modify this interface. After clicking OK, I'll then deploy my changes (top right, the icon with an 🠟, second from the left).

Change inside IP and delete associated DHCP scope

SmartCLI - Configuring OSPF in FDM

I know, I know... that was really boring. Thanks for hanging in there, Derek, I promise we're going to start getting into the meat and potatoes of this post now. Also, sorry for making you hungry. First up, to finish establishing connectivity, let's configure OSPF on our Firepower device. Super basic, we're going to drop Gig0/1 (inside) into area 0.0.0.0 and have it originate a default route. The L3 switch behind the firewall also has a basic OSPF configuration looking for a neighbor on this subnet. From the Device tab in FDM, in the bottom right you'll see 'Advanced Configuration', and that's where we're headed.

FDM Advanced Config - SmartCLI and FlexConfig

At present, SmartCLI serves a single purpose, which is configuring OSPF and OSPF alone (what no ODR?! #ObscureRoutingReference). From the OSPF template, the initial view leaves much to be desired. So we're going to click on 'Show Disabled' and then change enable 'setup ospf advanced'. From here you'll have a ton of OSPF options to configure, however, I'm going to disable most of them by clicking on the + to the left of each option I want to disable. Then I'll fill out all my relevant information.


SmartCLI - Advanced options hidden
SmartCLI - Advanced OSPF enabled


Additionally, there's a SmartCLI template called "Interface" that allows you to configure interface-specific OSPF options such as authentication, cost, and network type. After deploying, we can go to the CLI to confirm everything is working as expected.

> show ospf neighbor 


Neighbor ID     Pri   State           Dead Time   Address         Interface
192.168.122.22    1   FULL/BDR        0:00:34    172.31.99.2     inside
> show route ospf | begin ^Gateway

Gateway of last resort is 192.168.122.1 to network 0.0.0.0

O        172.31.45.0 255.255.255.0 [110/20] via 172.31.99.2, 00:41:15, inside


Application Visibility & Control (AVC)

Now onto the main event, AVC. I'll demo this out using FDM, but the concept is exactly the same in Cisco's full-blown Firepower Management Center. This is a feature I don't see implemented near often enough, and when I do see it deployed typically not in a way that maximizes its potential (looking at you again Derek). Let's start with a very basic objective and we'll work outwards from there. You've deployed this firewall, and you only want to allow the following egress traffic from your users.
  • Web browsing over HTTP and HTTPS (foreboding-music.mp3)
  • ICMP echo
  • DNS to OpenDNS
Everything else should be dropped. Additionally, you want to block URLs for online gaming sites. So you build out a couple ACEs in your policy, click deploy and call it a day. Tell me I'm wrong Derek. Let's first take a quick look at what our policy looks like, before talking about how AVC can help us.

Derek's super secure bulletproof ACP

Then we'll go to our client machine and make sure everything is working as expected.

DNS and ICMP work
HTTPS works

URL Filtering works

So that's it, right? Right?...Right? What's guaranteeing that the ports we've opened are going to used by the applications we intend? Of course, there are obvious things like SSLVPN and proxies to worry about - but then also less obvious concerns. There's an open-source, easily deployed, VPN client now that allows tunneling over ICMP and DNS. You read that right.

**Full disclosure, this was actually kind of a pain in the ass. While specifically blacklisting applications works perfectly well, that's not what I wanted to show off in this post. I wanted to show how we can not just open ports in our ACEs, but also specifically dictate which applications can use those ports. With respect to DNS and ICMP, that worked fine, but trying to write a rule that basically said 'allow tcp/80 and tcp/443 only for web browsing traffic' ended up being a huge undertaking that set me back on this post, and forced me to drop the FDM entirely in favor of a full-blown FMC. Cisco... this shouldn't be as hard as it was, fix it.**

There are a couple different ways to approach this problem, and normally what people do is opt to create a separate ACE that just blocks undesirable application types. That works fine, but I'd argue that implicitly dropping traffic is much more effective than explicitly blocking in this case. With our DNS and ICMP rules, this is easy from the FDM. Web traffic, on the other hand, requires a little more horsepower and granularity, so I'm going to forklift my configs into a Firepower Management Center. So let's tackle the easy tasks first. Within our access control entries for DNS and ICMP, I'm just going to drop into the applications tab and specify those applications respectfully. Additionally, I'll add a catchall rule that blocks w/reset so we don't have to wait for dropped traffic to timeout.

ICMP/DNS Matching Application

Now web browsing traffic, this is where things get a little tricky. Before we dive into how to build our HTTP/HTTPS rule to only allow Web Browsing traffic, let's look and see why you might want to do this. Eyes up Derek, no sleeping in class today. Remember our URL filter blocking access to online games? What happens to it if we tunnel all our traffic over an SSL VPN? On my test machine, I have both SoftEther and Window's native SSTP client setup, so let's get on our dodgy VPN and subvert security policies!

Connected to SSLVPN
Game On

This may seem like a silly example, but start thinking about how easily this could be exploited. Simply tunneling your traffic over port 443 suddenly bypasses DLP policies. Just opening the port to allow web browsing becomes a means of establishing a covert channel. This doesn't have to be over a VPN either, by default TOR will connect to relays over tcp/443 if it's default port (tcp/9001) is closed. So how to address this issue? The easy task is allowing the majority of expected applications for web browsing allowed, then we'll come back to the hard part. So to start, let's get some client-side applications added in (common browsers like Chrome and FireFox) as well as some well-known web-based applications (i.e. Google, Bing, Cisco, Facebook). The web-based applications become important because with HTTPS traffic Firepower can't determine the client-side application in use. To make things easier on ourselves, I'm going to allow any Web Application with a risk of Medium or Less. Now, logically, one would think you could select Risk "Very Low", "Low", and "Medium" and Web Application as a single filter. At the time of this writing, that does not seem to work. So instead we need three separate filters pairing Web Application with each Risk. 

Common Web Browsing

With this deployed, let's go back to our test machine. I'll try to connect to VPN first, then browse to a couple websites.

No Luck connecting to SSLVPN


Cisco page loads

AVC vs. Unclassified Web Sites

So if you've caught yourself thinking "Now we're done." You wouldn't be wrong, but user's experiences on the Internet would be mixed at best. The issue you'll often run into is, "what happens when a user browses to a site that isn't well-known?", that is, a site that Firepower doesn't have a defined Web Application for. Well to answer that, let's first see what happens when we browse to a little known, seldom visited site like networkknerd.com.

Congratulations! You've played yourself.



So now we're in a pickle. We could try creating web applications for every website Firepower doesn't have a canned definition for, but that seems like an administrative nightmare. So instead, I'm going to write a different sort of rule to get around this problem. Essentially, we need to write an application definition that can apply to browsing any HTTPS website (not already defined). To do this, I ran wireshark while trying to browse to my website, so I could poke around the TLS client/server messages. Doing so, I found a field in the client hello that I could match on.

We Got a Match

When we're writing our rules, we'll have the option to match on ASCII string or Hex. While I could just match on http/1.1, I decided to match the entire field with Hex 68 74 74 70 2f 31 2e 31. So, let's move over to our FMC, under Policies -> Application Detectors. Click on 'Create Custom Detector'. I'll leave Detector Type as 'Basic', and I'll Add to add an Application Protocol. From the Application Editor I'll create a protocol like this:

Custom Application Protocol

Click 'OK' to return to the Create A Custom Application Detector dialog. Set the Application Protocol to be "HTTPoverSSL", add a description then click "OK". Now we need to tell Firepower what pattern we'll be detecting to classify this traffic. Since I'm going with a field in the client hello field, I'll set the direction to be client, protocol tcp, and add my hex value from above. To test, we can actually upload a pcap file and click 'test' (script with gear icon to the left of the trash can). Got all that?! Derek?! Don't worry, accompanying screen shots below.

Custom Application Detector

Traffic Pattern to Match
Successful Test

Well that was easy.

Not really, but now we have a custom application detector to add to our ACP that should work as a catch all for all HTTPS browsing (in addition to the previous rules implemented based on application type/risk level). After adding that to my INSIDE-to-WEB rule, browsing works flawlessly and SSLVPN is still implicitly blocked. So, a couple more screen shots then I'm call this post from hell done.

Adding HTTPoverSSL to ACE

Web Browsing Working

SSTP Still Dead

Alright. Well... I need a drink. 


Monday, April 23, 2018

We need to talk about GETVPN

We really have to talk about GETVPN. Despite its drawbacks, I can't seem to get it out of my head now and I'm constantly running through scenarios where using it might make sense.


If you're not too familiar with GETVPN, let me offer this high-level summary of the technology. GETVPN is a technology that allows all endpoints to share a common phase2 Security Association (SA) by building their phase1 tunnels to a shared key server. This keyserver tells all VPN endpoints (referred to as Group Members) about what traffic is to be encrypted, what ciphers to use, etc. The benefit being, if you have two Group Members that have never exchanged information with one another, you don't have to build a tunnel between them as they will already have a shared/common SA. Anyone who's ever used ping to bring a tunnel up knows this pain. The first ping always drops right? Not in GETVPN. GETVPN is recommended for securing communication in full mesh networks (like MPLS or DMVPN). So today I thought I'd run through what I think is a fairly practical use of GETVPN, securing DMVPN traffic.

Goals

  • Use front-door VRF on all DMVPN devices
  • Protect traffic using GETVPN
  • Avoid pre-shared-keys for phase 1 tunnels, use certificate authentication instead

Let's take a quick look at our topology



One annoying caveat to deploying GETVPN is that the Keyserver cannot also be a group member. This means, as in my topology, you have a router who's not actively going to passing user/data plane traffic in the mix. The good news here, you could probably virtualize this function in the real world leveraging a CSR1000v to be your KS. I'll also be using my KS as a Certificate Authority (you don't have to do this, pre-shared-keys work just fine with GETVPN, I just like using certs). At my hub location, I also have an ASA firewall. This will be using NAT to forward both ports 80 (for cert requests to my CA) as well as udp port 848 (GETVPN's default port for ISAKMP).

Relevant Config from ASAv

interface GigabitEthernet0/0
 nameif OUTSIDE
 security-level 0
 ip address 208.0.0.201 255.255.255.0 
!
interface GigabitEthernet0/1
 nameif INSIDE
 security-level 100
 ip address 10.12.12.254 255.255.255.0 
 summary-address eigrp 47884 0.0.0.0 0.0.0.0 20
!
router eigrp 47884
 network 10.0.0.0 255.0.0.0
!
object network KS
 host 10.12.12.10
!
object network KS
 nat (INSIDE,OUTSIDE) static 208.0.0.254
!
nat (INSIDE,OUTSIDE) after-auto source dynamic any interface
!
access-list OUTSIDE_access_in extended permit udp any object KS eq 848 
access-list OUTSIDE_access_in extended permit udp any object KS eq isakmp 
access-list OUTSIDE_access_in extended permit udp any object KS eq 4500 
access-list OUTSIDE_access_in extended permit tcp any object KS eq www 
!
access-group OUTSIDE_access_in in interface OUTSIDE

We shouldn't have to go back to the ASA to make any additional changes from here on out. Next I'm going to setup my front-door VRF (FVRF) and get all base routing setup for my two hubs and two spokes.


Hub1 and Hub2

vrf definition FVRF
 !
 address-family ipv4
 exit-address-family
!
interface GigabitEthernet0/0
 vrf forwarding FVRF
 ip address 208.0.0.1 255.255.255.0
!
interface GigabitEthernet0/1
 ip address 10.12.12.x 255.255.255.0
!
ip route vrf FVRF 0.0.0.0 0.0.0.0 208.0.0.254

Spoke1 and Spoke2

vrf definition FVRF
 !
 address-family ipv4
 exit-address-family
!
interface GigabitEthernet1
 vrf forwarding FVRF
 ip address 208.0.0.1 255.255.255.0
!
interface GigabitEthernet2
 ip address 10.x0.x0.x 255.255.255.0
!
ip route vrf FVRF 0.0.0.0 0.0.0.0 208.0.0.254

Configuring the KS as a Certificate Authority


Again, this part is optional if you would prefer to use pre-shared-keys. You could also use a Microsoft CA server, something I've done in previous posts (e.g. FlexVPN). With that said, let's quickly run through a basic PKI server setup on our future KS.

clock set [input time/date]
!
ip domain name networkknerd.com
crypto key generate rsa general-keys modulus 2048!
!
crypto pki server KSCA
 no database archive
 issuer-name CN=KSCA.networkknerd.com
 hash sha256
 database url flash:
 no shutdown

You could also configure your IOS CA to automatically grant certificates requests, but I'm fine with manually granting requests. 

Requesting certificates from IOS CA


This is pretty basic, and I'll provide the config output from Hub1 but it's copy/paste and find and replace for the configs on the other DMVPN routers. Obviously changing only the fqdn and subject-name values to match.

crypto pki trustpoint KSCA
 enrollment retry count 3
 enrollment url http://208.0.0.254:80
 fqdn Hub1.networkknerd.com
 subject-name CN=Hub1.networkknerd.com
 vrf FVRF
!
crypto pki authenticate KSCA
% Do you accept this certificate? [yes/no]: yes
!
crypto pki enroll KSCA
% The subject name in the certificate will include: CN=Hub1.networkknerd.com
% The subject name in the certificate will include: Hub1.networkknerd.com
% Include the router serial number in the subject name? [yes/no]: no
% Include an IP address in the subject name? [no]: 
Request certificate from CA? [yes/no]: yes
% Certificate request sent to Certificate Authority
% The 'show crypto pki certificate verbose KSCA' command will show the fingerprint.
!
[Rise and Repeate on Hub2, Spoke1, and Spoke2]

You'll also want an RSA signature certificate on the KS, which in our case is also our CA. So back on the KS/CA router issue the following:

crypto pki trustpoint KSCA_sig
 enrollment retry count 3
 enrollment url http://10.12.12.10:80
 fqdn Hub1.networkknerd.com
 subject-name CN=KSCA.networkknerd.com
!
crypto pki authenticate KSCA_sig
% Do you accept this certificate? [yes/no]: yes
!
crypto pki enroll KSCA_sig
% The subject name in the certificate will include: CN=KSCA.networkknerd.com
% The subject name in the certificate will include:
KSCA.networkknerd.com
% Include the router serial number in the subject name? [yes/no]: no
% Include an IP address in the subject name? [no]: 
Request certificate from CA? [yes/no]: yes
% Certificate request sent to Certificate Authority
% The 'show crypto pki certificate verbose KSCA' command will show the fingerprint.



Then from the IOS CA server

crypto pki server KSCA grant all

Configuring the Keyserver


The key server has the unique role of being the only router in our topology other routers (Group Members) will actually build phase 1 tunnels. This can be a little unsettling to look at if you've only ever worked with more traditional IPsec deployments, but essentially 'show crypto isakmp sa' on your routers will only ever should an SA with the Keyserver. That said the KS is also where most of our GETVPN configurations actually live. First we'll setup an isakmp policy (NOTE GDOI/GKM does support IKEv2, I'm just using IKEv1 in this lab because I have enough moving peices as is). 

ALL ROUTERS (GMs and KS)

crypto isakmp policy 10
 encr aes 256
 hash sha256
 lifetime 28800

You'll notice I didn't specify "authentication pre-share", since I'm using certificates I don't need this.

KS Configurations

ip access-list extended GKM_PROTECT
 permit gre any any
!
ip access-list extended KS

 permit ip host 10.12.12.10 any
!
crypto ipsec transform-set ESP-AES256-SHA2 esp-aes 256 esp-sha256-hmac
!
crypto ipsec profile GKM_PROF
 set transform-set ESP-AES256-SHA2 
!
crypto gkm group GDOI
 identity number 47884
 server local
  rekey algorithm aes 128
  rekey sig-hash algorithm sha256
  rekey address ipv4 KS
  rekey authentication mypubkey rsa KSCA
  rekey transport unicast
  sa ipsec 1
   profile GKM_PROF
   match address ipv4 GKM_PROTECT
   replay counter window-size 128
   no tag
  address ipv4 10.12.12.10

First I defined a couple access-lists, GKM_PROTECT is my interesting traffic ACL. Telling GMs to encrypt gre traffic (since this is protecting DMVPN traffic). If you weren't deploying DMVPN, your ACL here may contain something like "permit ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255" to protect any IP traffic betwen 10.0.0.0/8 subnets. Then second ACL "KS" is used to specify my rekey address, basically telling GETVPN I want to use 10.12.12.10 to rekey GMs. This can be skipped when multicast is used for rekey transport, but in this lab (since I'm mimicking an Internet deployment style) I'm using unicast for transport.

Next, I'm setting up my phase 2 transform-set, and an IPsec profile calling that transform-set. This information is actually managed by the KS and pushed out to GMs later on. After that I have a fairly basic GETVPN server configuration, however I am changing the default rekey sig-hash to sha256 and rekey encryption algorithm to aes-128.

Group Member Configuration

 This configuration is copy/paste, so I'll only provide the output from one GM, but it needs to be applied to all GMs.

crypto gkm group GDOI
 identity number 47884
 server address ipv4 208.0.0.254
!
crypto map GETVPN 1 gdoi 
 set group GDOI
!
interface GigabitEthernet0/0
 crypto map GETVPN

Notice how much simpler this configuration compared to the KS. Here we only have to define the server's address, match the identity number (or address if chose to use that instead) and plug that GKM configuration into a crypto map that's assigned to our WAN interface. If all is successful you should see some similar output to this:

Apr 23 16:06:22.673: %CRYPTO-6-GDOI_ON_OFF: GDOI is ON
Apr 23 16:06:22.679: %CRYPTO-5-GM_REGSTER: Start registration to KS 208.0.0.254 for group GDOI using address 208.0.0.1 fvrf FVRF ivrf FVRF
Apr 23 16:06:22.810: %GDOI-5-SA_TEK_UPDATED: SA TEK was updated
Apr 23 16:06:22.821: %GDOI-5-SA_KEK_UPDATED: SA KEK was updated 0xFDA735C08E607248BF18DCE835ACEE53
Apr 23 16:06:22.824: %GDOI-5-GM_REGS_COMPL: Registration to KS 208.0.0.254 complete for group GDOI using address 208.0.0.1 fvrf FVRF ivrf FVRF
Apr 23 16:06:22.841: %GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 208.0.0.254 for group GDOI & gm identity 208.0.0.1 fvrf FVRF ivrf FVRF

Now we can run through and do some quick validation on the KS and GM. After that we'll do our DMVPN configuration and call it a day!

GM Validations:

show crypto gkm gm 
Group Member Information For Group GDOI:
    IPSec SA Direction       : Both
    ACL Received From KS     : gdoi_group_GDOI_temp_acl

    Group member             : 208.0.0.1       vrf: FVRF
       Local addr/port       : 208.0.0.1/848
       Remote addr/port      : 208.0.0.254/848
       fvrf/ivrf             : FVRF/FVRF
       Version               : 1.0.12
       Registration status   : Registered
       Registered with       : 208.0.0.254
       Re-registers in       : 3247 sec
       Succeeded registration: 1
       Attempted registration: 1
       Last rekey from       : 0.0.0.0
       Last rekey seq num    : 5
       Unicast rekey received: 0
       Rekey ACKs sent       : 0
       Rekey Received        : never
       DP Error Monitoring   : OFF
       IPSEC init reg executed    : 0
       IPSEC init reg postponed   : 0
       Active TEK Number     : 2

       SA Track (OID/status) : disabled
show crypto gkm gm acl 
Group Name: GDOI
 ACL Downloaded From KS 208.0.0.254:
   access-list   permit gre any any
 ACL Configured Locally: 
 ACL of default bypass policy for group-key management traffic:

   GigabitEthernet0/0: deny udp host 208.0.0.1 eq 848 any eq 848 vrf FVRF

KS Validations:

show crypto gkm ks 
Total group members registered to this box: 4

Key Server Information For Group GDOI:
    Group Name               : GDOI
    Re-auth on new CRL       : Disabled
    Group Identity           : 47884
    Group Type               : GDOI (ISAKMP)
    Group Members            : 4
    Rekey Acknowledgement Cfg: Cisco
    IPSec SA Direction       : Both
    IP D3P Window            : Disabled
    Split Resiliency Factor  : 0
    CKM status               : Disabled
    ACL Configured: 
        access-list GKM_PROTECT
!
show crypto gkm ks members 

Group Member Information : 

Number of rekeys sent for group GDOI : 5
Number of retransmits during the last rekey for group GDOI : 0
Duration of the last rekey for group GDOI : 10011 msec

Group Member ID    : 208.0.0.1   GM Version: 1.0.12
 Group ID          : 47884
 Group Name        : GDOI
 Group Type        : GDOI (ISAKMP)
 GM State          : Registered
 Key Server ID     : 10.12.12.10
 Rekeys sent       : 5
 Rekeys retries    : 0
 Rekey Acks Rcvd   : 5
 Rekey Acks missed : 1

 Sent seq num : 2       3       4       5
Rcvd seq num :  2       3       4       5

Group Member ID    : 208.0.0.2   GM Version: 1.0.12
 Group ID          : 47884
          
Group Member Information : 

 Group Name        : GDOI
 Group Type        : GDOI (ISAKMP)
 GM State          : Registered
 Key Server ID     : 10.12.12.10
 Rekeys sent       : 5
 Rekeys retries    : 0
 Rekey Acks Rcvd   : 5
 Rekey Acks missed : 0

 Sent seq num : 2       3       4       5
Rcvd seq num :  2       3       4       5

Group Member ID    : 208.0.0.10  GM Version: 1.0.16
 Group ID          : 47884
 Group Name        : GDOI
 Group Type        : GDOI (ISAKMP)
 GM State          : Registered
 Key Server ID     : 10.12.12.10
 Rekeys sent       : 5
 Rekeys retries    : 0
 Rekey Acks Rcvd   : 5
 Rekey Acks missed : 0

          
Group Member Information : 

 Sent seq num : 2       3       4       5
Rcvd seq num :  2       3       4       5

Group Member ID    : 208.0.0.20  GM Version: 1.0.16
 Group ID          : 47884
 Group Name        : GDOI
 Group Type        : GDOI (ISAKMP)
 GM State          : Registered
 Key Server ID     : 10.12.12.10
 Rekeys sent       : 5
 Rekeys retries    : 0
 Rekey Acks Rcvd   : 5
 Rekey Acks missed : 0

 Sent seq num : 2       3       4       5
Rcvd seq num :  2       3       4       5


DMVPN Configuration

Spoiler alert, there's not much exciting happening here :-). Fairly standard Dual-Hub single cloud DMVPN deployment, with the exception I'm not going to configure any tunnel protection (since we have a crypto map tied to the phyiscal WAN interface protecting traffic via GETVPN). So I'll share the configurations from Hub1 and Spoke1 below to give you an idea of what the config looks like.

Hub1

interface Tunnel0
 ip address 10.255.255.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp map multicast dynamic
 ip nhrp network-id 1337
 ip nhrp shortcut
 ip nhrp redirect
 ip tcp adjust-mss 1360
 tunnel source GigabitEthernet0/0
 tunnel mode gre multipoint
 tunnel vrf FVRF
!
router eigrp 47884
 network 10.0.0.0

Spoke1

interface Tunnel0
 ip address 10.255.255.10 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp map multicast 208.0.0.1
 ip nhrp map multicast 208.0.0.2
 ip nhrp map 10.255.255.1 208.0.0.1
 ip nhrp map 10.255.255.2 208.0.0.2
 ip nhrp network-id 1337
 ip nhrp nhs 10.255.255.1
 ip nhrp nhs 10.255.255.2 priority 10
 ip nhrp redirect
 ip tcp adjust-mss 1360
 tunnel source GigabitEthernet1
 tunnel mode gre multipoint
 tunnel vrf FVRF
!
router eigrp 47884
 network 10.0.0.0

Annnndddddd that's it. Really basic, straightforward DMVPN configuration using a front-door VRF and dual hubs. So let's do some quick validations.

Hub1

show dmvpn | b IPv4 NHRP Details
Interface: Tunnel0, IPv4 NHRP Details 
Type:Hub, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 208.0.0.10        10.255.255.10    UP 01:53:19     D
     1 208.0.0.20        10.255.255.20    UP 01:52:37     D
!
show ip eigrp neighbors 
EIGRP-IPv4 Neighbors for AS(47884)
H   Address                 Interface              Hold Uptime  
                                                   (sec)         (ms)       Cnt Num
4   10.255.255.20           Tu0                      11 00:06:04  
3   10.255.255.10           Tu0                      13 00:06:15  
2   10.12.12.10             Gi0/1                    12 05:44:28    
1   10.12.12.2              Gi0/1                    13 05:46:00  
0   10.12.12.254            Gi0/1                    12 05:46:09  

show ip eigrp neighbors 
EIGRP-IPv4 Neighbors for AS(47884)
H   Address                 Interface              Hold Uptime   SRTT   RTO  Q  Seq
                                                   (sec)         (ms)       Cnt Num
4   10.255.255.20           Tu0                      11 00:06:04   14  1398  0  6
3   10.255.255.10           Tu0                      13 00:06:15   16  1398  0  7
2   10.12.12.10             Gi0/1                    12 05:44:28    4   100  0  7
1   10.12.12.2              Gi0/1                    13 05:46:00   79   474  0  23
0   10.12.12.254            Gi0/1                    12 05:46:09    8   100  0  11  


Spoke1

show dmvpn | b IPv4 NHRP Details
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1 208.0.0.1          10.255.255.1    UP 01:56:37     S
     1 208.0.0.2          10.255.255.2    UP 01:56:32     S
!
show ip route | b Gateway
Gateway of last resort is 10.255.255.2 to network 0.0.0.0

D*    0.0.0.0/0 [90/26880512] via 10.255.255.2, 00:09:36, Tunnel0
                [90/26880512] via 10.255.255.1, 00:09:36, Tunnel0
      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
C        10.10.10.0/24 is directly connected, GigabitEthernet2
L        10.10.10.1/32 is directly connected, GigabitEthernet2
D        10.12.12.0/24 [90/26880256] via 10.255.255.2, 00:09:36, Tunnel0
                       [90/26880256] via 10.255.255.1, 00:09:36, Tunnel0
C        10.255.255.0/24 is directly connected, Tunnel0
L        10.255.255.10/32 is directly connected, Tunnel0
!
ping 10.20.20.1 source gig2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.20.20.1, timeout is 2 seconds:
Packet sent with a source address of 10.10.10.1 
!!!!!
!
show dmvpn | b IPv4 NHRP Details
Interface: Tunnel0, IPv4 NHRP Details 
Type:Spoke, NHRP Peers:3, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     2 208.0.0.20        10.255.255.20    UP 00:00:23   DT1
                         10.255.255.20    UP 00:00:23   DT1
     1 208.0.0.1          10.255.255.1    UP 02:00:16     S
     1 208.0.0.2          10.255.255.2    UP 02:00:11     S
!
show ip route | b Gateway
Gateway of last resort is 10.255.255.2 to network 0.0.0.0

D*    0.0.0.0/0 [90/26880512] via 10.255.255.2, 00:13:31, Tunnel0
                [90/26880512] via 10.255.255.1, 00:13:31, Tunnel0
      10.0.0.0/8 is variably subnetted, 7 subnets, 2 masks
C        10.10.10.0/24 is directly connected, GigabitEthernet2
L        10.10.10.1/32 is directly connected, GigabitEthernet2
D        10.12.12.0/24 [90/26880256] via 10.255.255.2, 00:13:31, Tunnel0
                       [90/26880256] via 10.255.255.1, 00:13:31, Tunnel0
H        10.20.20.0/24 [250/255] via 10.255.255.20, 00:00:56, Tunnel0
C        10.255.255.0/24 is directly connected, Tunnel0
L        10.255.255.10/32 is directly connected, Tunnel0
H        10.255.255.20/32 is directly connected, 00:00:56, Tunnel0

Notice that there was no ping lost during the dynamic tunnel creation between Spoke 1 and Spoke 2, since there was no latency caused by IPsec negotiation. Exactly the key reason WHY GETVPN is so cool, and why we had to spend a few minutes talking about it today. I'm uploading all the configs below if you want to test this out on your own... and I might even do a short video recapping all the above.