Archive for the ‘Troubleshooting’ Category

Consider the following network.

Things to notice:

Subnets 22.22.22.0/24 and 33.33.33.0/24 are being routed to the outside of the ASA.

There is a static NAT statement in the ASA to translate the real IP 192.168.5.22 to 22.22.22.22

There is a static NAT statement in the ASA to translate the real IP 192.168.5.33 to 33.33.33.33

 

So how do you get this to work properly?

In ASA pre-8.3 code the ASA would ARP for the static NATs it would have regardless if it’s connected or not.

In ASA 8.3-8.4(4), THIS IS IMPOSSIBLE

In ASA 8.4(5)+ Cisco realized their major mistake and implemented the command:

arp permit-nonconnected

 

When else can I use this?

Another scenario to use this is when you have a router with multiple IPs on its interface that is connected to an ASA with a single IP. The ASA won’t accept any packets for the other subnets that the router thinks is connected. By applying this command it will accept packets for the other subnets.

 

What’s the risk?

Well, by enabling this feature it could facilitate denial of service (DoS) attack against the ASA; a user on any interface could send out many ARP replies and overload the ASA ARP table with false entries.

 

 

Sometimes when you are using a laptop with multiple displays and then unplug from those displays there will be windows that are outside the viewing area of your laptop. Sometimes these windows can be pesky and not want to come back to the screen. If you are using Windows 7, try this:

Select the program from the task bar
Alt + Space + M
Tap arrow key
Reposition window with mouse

 

 

Suppose you are trying to troubleshoot a site to site VPN tunnel that is designed like this:

Upon doing `show ipsec sa peer` on the blue ASA you see the following:

interface: OUTSIDE
Crypto map tag: MAP-OUTSIDE, seq num: 200, local addr: 11.11.11.11

local ident (addr/mask/prot/port): (192.168.11.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (172.16.22.0/255.255.255.0/0/0)
current_peer: 22.22.22.22

#pkts encaps: 61, #pkts encrypt: 61, #pkts digest: 61
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 61, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#send errors: 0, #recv errors: 0

The problem above shows that Phase 1 of the tunnel is successfully establishing but phase 2 has problems. Specifically the firewall is encrypting packets but not decrypting them.

If an ASA or router is getting encaps but not decaps, this means it is encrypting the data and sending it but has not received anything to decrypt in return.

• Verify the other end has a route outside for the interesting traffic.
• Check that both VPN ACL’s are not mismatched.
• Double check NAT’s to make sure the traffic is not NAT’ing correctly.
• Is what you are trying to ping even responding back? Often what you’re sending traffic to is not able to accept or is not responding to this traffic. I prefer to put a packet capture on the remote end firewall to see if the traffic is coming back into that firewall.

TCPDump is an extremely handy tool for verifying if packets are getting to the linux box or not. Here are the commands I use most often:

 

To specify which interface to listen on:

tcpdump -i eth1

To specify which host the traffic is coming from:

tcpdump host 10.64.45.53

To specify a port:

tcpdump port 8080

 

And of course you can add all of that together in one line using the “and” keyword:

tcpdump -i eth1 host 10.64.45.53 and port 514

 

 

 

Goal: Get into an 1811 that has the No Service Password-Recovery feature enabled.

Limitation: Because this feature is enabled it simply means when you do password recovery the entire config will be wiped.

Problem: Cisco’s documentation on this is WRONG.

Solution: Using SecureCRT and consoled into the router the break sequence is CTRL+BREAK (if you are on a lenovo laptop then hold CTRL then push Fn+BREAK). You have to send it at the right time. Look at the output below to see when to hit the break sequence.

 

 

ÿ
System Bootstrap, Version 12.3(8r)YH13, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2008 by cisco Systems, Inc.
C1800 platform with 131072 Kbytes of main memory with parity disabled

Readonly ROMMON initialized
PASSWORD RECOVERY FUNCTIONALITY IS DISABLED
program load complete, entry point: 0×80012000, size: 0xc0c0

Initializing ATA monitor library…….
program load complete, entry point: 0×80012000, size: 0xc0c0

Initializing ATA monitor library…….

program load complete, entry point: 0×80012000, size: 0x1974ba0
Self decompressing the image : ######################################################################
######################################################################
######################################################################
################################ [OK]

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software – Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS Software, C181X Software (C181X-ADVIPSERVICESK9-M), Version 15.0(1)M3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Sun 18-Jul-10 00:57 by prod_rel_team
Image text-base: 0×80012118, data-base: 0x82E13000
[ 5 Second window begins here to send BREAK ]

PASSWORD RECOVERY IS DISABLED.
Do you want to reset the router to factory default
configuration and proceed [y/n] ? y
Reset router configuration to factory default.

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
[email protected]

Installed image archive
Cisco 1811 (MPC8500) processor (revision 0×400) with 118784K/12288K bytes of memory.
Processor board ID FTX141780Z7, with hardware revision 0000

10 FastEthernet interfaces
1 Serial interface
1 terminal line
1 Virtual Private Network (VPN) Module
62720K bytes of ATA CompactFlash (Read/Write)
[OK][OK]
SETUP: new interface FastEthernet0 placed in “shutdown” state
SETUP: new interface FastEthernet1 placed in “shutdown” state

Press RET5 01:35:07.427: %LINK-5-CHANGED: Interface FastEthernet0, changed state to administratively down
*Jan  5 01:35:07.431: %SYS-5-RESTART: System restarted –
Cisco IOS Software, C181X Software (C181X-ADVIPSERVICESK9-M), Version 15.0(1)M3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Sun 18-Jul-10 00:57 by prod_rel_team
*Jan  5 01:35:07.431: %SNMP-5-COLDSTART: SNMP agent on host Router is undergoing a cold start
*Jan  5 01:35:07.459: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Jan  5 01:35:07.459: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Jan  5 01:35:07.943: %LINK-5-CHANGED: Interface FastEthernet1, changed state to administratively down
*Jan  5 01:35:08.823: %LINK-3-UPDOWN: Interface FastEthernet2, changed state to up
*Jan  5 01:35:08.823: %LINK-3-UPDOWN: Interface FastEthernet3, changed state to up
*Jan  5 01:35:08.823: %LINK-3-UPDOWN: Interface FastEthernet4, changed state to up
*Jan  5 01:35:08.827: %LINK-3-UPDOWN: Interface FastEthernet5, changed state to up
*Jan  5 01:35:08.827: %LINK-3-UPDOWN: Interface FastEthernet6, changed state to up
*Jan  5 01:35:08.827: %LINK-3-UPDOWN: Interface FastEthernet7, changed state to up
*Jan  5 01:35:08.831: %LINK-3-UPDOWN: Interface FastEthernet8, changed state to up
*Jan  5 01:35:08.831: %LINK-3-UPDOWN: Interface FastEthernet9, changed state to up
*Jan  5 01:35:09.823: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet2, changed state to down
*Jan  5 01:35:09.823: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet3, changed state to down
*Jan  5 01:35:09.823: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet4, changed state to down
*Jan  5 01:35:09.827: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet5, changed state to down
*Jan  5 01:35:09.827: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet6, changed state to down
*Jan  5 01:35:09.827: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet7, changed state to down
*Jan  5 01:35:09.831: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet8, changed state to down
*Jan  5 01:35:09.831: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9, changed state to down
Router>

 

Today I was given a problem that our Cisco Ironport was not accepting email from outside people sending mail to inside people (backstory: this occurred right after we moved our Ironport to a different location). I looked in Ironport and spotted a lot of messages failing to be delivered. Specifically the error in Ironport was:
Message 1006902 aborted: Receiving aborted

I spent a long time doing packet captures to try to troubleshoot and determined the remote end was sending a reset which made me think this is the sender’s problem and not mine. However I was wrong.

The next thing I checked was the MX Record at MXToolbox.com (a great site for looking up DNS records and stuff). Specifically the SMTP test showed this:

Specifically I didn’t like seeing this warning:
SMTP Reverse DNS Mismatch — Warning – Reverse DNS does not match SMTP Banner
and
SMTP TLS — Warning – Does not support TLS

But what does that mean? I specifically wanted to know what two strings are being compared that resulted in a mismatch. Well in the case above the two strings it was comparing were *********************** and mail3.example.com. For some reason this took me a long time to realize the ********************* was the banner… You can see it in the image above after 220.

Looking around on the internet it turns out that our Cisco ASA we have in front of the Ironport has “inspect esmtp” turned on (which is on by default). Upon turning that inspect off the issue immediately cleared up and the results were this:

Mail was then flowing into the Ironport properly and being delivered as expected. Looking back at the problem if I would have looked at the logs in the ASA I would have seen these syslogs:

ASA-4-108004: ESMTP Classification: Dropped connection for ESMTP Request from outside:75.75.75.75/35314 to DMZ:10.0.25.101/25; matched Class 4: header line length gt 998
%ASA-4-507003: tcp flow from outside:75.75.75.75/35314 to PUBLIC_DMZ:10.0.25.101/25 terminated by inspection engine, reason - inspector disconnected, dropped packet.

After a lengthy phone call with Cisco TAC I learned an interesting link between a few commands on an ASA for analyzing tunnels.

Suppose we are REALLY having trouble getting a tunnel up. You are sure the traffic is hitting the firewall that should be encrypted but the tunnel is just not even attempting phase one. These show commands may help identify a problem.

Suppose our tunnel allows traffic from the 10.100.0.0/16 inside subnet to the 10.10.15.0/24 remote subnet.

ASA# PACKET-TRACER INPUT INSIDE ICMP 10.100.10.100 8 0 10.10.15.15. DETAILED

…
Phase: 12
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0xd9354580, priority=70, domain=encrypt, deny=false
hits=2198, user_data=0x922fac, cs_id=0xd8c45e10, reverse, flags=0×0, protocol=0
src ip=10.100.0.0, mask=255.255.0.0, port=0
dst ip=10.10.15.0, mask=255.255.255.0, port=0, dscp=0×0

 

Take note of the “user_data” value above. Grab that, and capitalize the hex letters to use this command:

ASA# SHOW ASP TABLE VPN-CONTEXT DETAIL | begin 922FAC

VPN CTX  = 0x00922FAC

Peer IP  = 10.10.15.0
Pointer  = 0xD91404E8
State    = UP
Flags    = ENCR+ESP
SA       = 0x1664DD33
SPI      = 0xE5C56C30
Group    = 47
Pkts     = 362631
Bad Pkts = 0
Bad SPI  = 0
Spoof    = 0
Bad Crypto = 0
Rekey Pkt  = 44
Rekey Call = 44
VPN Filter = <none>

Above is the Context, SA and SPI of the tunnel we are dealing with. You can see the flags above are ENCR which means this is the encaps or outbound packets. Also verify that there are “Pkts” increasing.

You can then verify that SPI is the same that is used in the IPSEC SA (if you have one up) by using this command:

ASA# SHOW CRYPTO IPSEC SA PEER 66.162.66.162

access-list ACL-PPP-VPN extended permit ip 10.100.0.0 255.255.0.0 10.10.15.0 255.255.255.0
local ident (addr/mask/prot/port): (10.100.0.0/255.255.0.0/0/0)
remote ident (addr/mask/prot/port): (10.10.15.0/255.255.255.0/0/0)
current_peer: 66.162.66.162

#pkts encaps: 402798, #pkts encrypt: 403786, #pkts digest: 403786
#pkts decaps: 306215, #pkts decrypt: 306215, #pkts verify: 306215
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 402798, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 988, #pre-frag failures: 0, #fragments created: 1976
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 2693
#send errors: 0, #recv errors: 0

local crypto endpt.: 202.2.202.2, remote crypto endpt.: 66.162.66.162

path mtu 1500, ipsec overhead 74, media mtu 1500
current outbound spi: E5C56C30
current inbound spi : A40D0530

inbound esp sas:
spi: 0xA40D0530 (2752316720)
transform: esp-aes esp-sha-hmac no compression
in use settings ={L2L, Tunnel, }
slot: 0, conn_id: 180224, crypto-map: mymap
sa timing: remaining key lifetime (kB/sec): (4372199/2855)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xE5C56C30 (3854920752)
transform: esp-aes esp-sha-hmac no compression
in use settings ={L2L, Tunnel, }
slot: 0, conn_id: 180224, crypto-map: mymap
sa timing: remaining key lifetime (kB/sec): (4299344/2855)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0×00000000 0×00000001

 

So that’s just another tool that can be used at troubleshooting  a VPN tunnel on an ASA. There are a lot of opportunities to learn here. If your packet tracer doesn’t pick up on any encryption then you’ve got a problem. If your asp table isn’t forming an SPI or isn’t getting Pkts then you’ve got a problem to examine. If your IPSEC SA has a different SPI than your asp table then you’ve got a problem to examine.

 

MM_WAIT_MSG4 is the stage where the firewall that initiated the tunnel is sending its pre-shared key hash to the receiver. This is NOT the stage that actually checks to see if the pre-shared keys match, it only exchanges the hashes for them.

The Initiator will stay at MSG4 until it gets a PSK back from its peer. If the receiver is missing a tunnel group or PSK the initiator will stay at MM_WAIT_MSG4

If the tunnel has to pass through a firewall to reach both end points you might have problems with the way traffic is passed through that middle firewall.

The tunnel can sometimes get stuck here due to incompatible vendor types. Try to get both devices on a modern version of software to try to eliminate any vendor mismatch issues.

This can sometimes fail due to mismatched ASA versions (rare).

Recently I had this problem pop up and I wanted to deconstruct the debug message leading up to the problem

! — Tunnel initiated - OK lets go!
Jun 10 2010 17:36:05: %ASA-7-715077: Pitcher: received a key acquire message, spi 0×0
Jun 10 2010 17:36:05: %ASA-5-713041: IP = 233.22.22.233, IKE Initiator: New Phase 1, Intf Inside_Network, IKE Peer 233.22.22.233  local Proxy Address 192.168.245.120, remote Proxy Address 10.3.10.122,  Crypto map (MAP-VPN)
Jun 10 2010 17:36:05: %ASA-7-715046: IP = 233.22.22.233, constructing ISAKMP SA payload
Jun 10 2010 17:36:05: %ASA-7-715046: IP = 233.22.22.233, constructing NAT-Traversal VID ver 02 payload
Jun 10 2010 17:36:05: %ASA-7-715046: IP = 233.22.22.233, constructing NAT-Traversal VID ver 03 payload
Jun 10 2010 17:36:05: %ASA-7-715046: IP = 233.22.22.233, constructing Fragmentation VID + extended capabilities payload
Jun 10 2010 17:36:05: %ASA-7-713236: IP = 233.22.22.233, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 148
Jun 10 2010 17:36:05: %ASA-7-713236: IP = 233.22.22.233, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108
Jun 10 2010 17:36:05: %ASA-7-715047: IP = 233.22.22.233, processing SA payload
Jun 10 2010 17:36:05: %ASA-7-713906: IP = 233.22.22.233, Oakley proposal is acceptable
! — At this point the firewall’s have agreed on the same ISAKMP proposals
Jun 10 2010 17:36:05: %ASA-7-715047: IP = 233.22.22.233, processing VID payload
Jun 10 2010 17:36:05: %ASA-7-715049: IP = 233.22.22.233, Received Fragmentation VID
Jun 10 2010 17:36:05: %ASA-7-715064: IP = 233.22.22.233, IKE Peer included IKE fragmentation capability flags:  Main Mode:        True  Aggressive Mode:True
Jun 10 2010 17:36:05: %ASA-7-715046: IP = 233.22.22.233, constructing ke payload
Jun 10 2010 17:36:05: %ASA-7-715046: IP = 233.22.22.233, constructing nonce payload
Jun 10 2010 17:36:05: %ASA-7-715046: IP = 233.22.22.233, constructing Cisco Unity VID payload
Jun 10 2010 17:36:05: %ASA-7-715046: IP = 233.22.22.233, constructing xauth V6 VID payload
Jun 10 2010 17:36:05: %ASA-7-715048: IP = 233.22.22.233, Send IOS VID
Jun 10 2010 17:36:05: %ASA-7-715038: IP = 233.22.22.233, Constructing ASA spoofing IOS Vendor ID payload (version: 1.0.0, capabilities: 20000001)
Jun 10 2010 17:36:05: %ASA-7-715046: IP = 233.22.22.233, constructing VID payload
Jun 10 2010 17:36:05: %ASA-7-715048: IP = 233.22.22.233, Send Altiga/Cisco VPN3000/Cisco ASA GW VID
Jun 10 2010 17:36:05: %ASA-7-713236: IP = 233.22.22.233, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + KE (4) + NONCE (10) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 256

! — All the syslogs above happened in the same second. All of a sudden things slowed down. 5 seconds passed which made me concerned already.

Jun 10 2010 17:36:10: %ASA-7-715077: Pitcher: received a key acquire message, spi 0×0
! — Wait, what? A new key acquire message, this isn’t right…
Jun 10 2010 17:36:10: %ASA-6-713219: IP = 233.22.22.233, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
Jun 10 2010 17:36:13: %ASA-7-713236: IP = 233.22.22.233, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + KE (4) + NONCE (10) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 256
! — Resending the payload? That’s not a good sign…
Jun 10 2010 17:36:13: %ASA-7-713236: IP = 233.22.22.233, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length :68
Jun 10 2010 17:36:13: %ASA-7-713236: IP = 233.22.22.233, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + NOTIFY (11) + NONE (0) total length :68
Jun 10 2010 17:36:13: %ASA-5-713904: IP = 233.22.22.233, Received an un-encrypted INVALID_COOKIE notify message, dropping
! — un-encrypted cookie? Why did the other side send us an un-encrypted packet? Could it be because the other side doesn’t have any information for this peer??
Jun 10 2010 17:36:13: %ASA-4-713903: IP = 233.22.22.233, Information Exchange processing failed
Jun 10 17:36:13 [IKEv1]: IP = 233.22.22.233, Information Exchange processing failed

The ultimate reason why this tunnel didn’t form is because the other side had the wrong Peer IP defined for the tunnel-group/pre-shared key. So when this side sent its isakmp proposals the other side only checked if it had a crypto map statement for this peer, it didn’t check if it had a tunnel-group defined. When they tried to exchange the pre-shared keys the remote end realized it doesn’t have any pre-shared key for this peer and just stopped responding. When this side tried to send another packet the remote end sent some cookie over which of course was not encrypted because it doesn’t have the right peer IP to do the encryption with. The debugs above are from an ASA 8.0.4 code connecting to a Cisco VPN concentrator running unknown code.

I have found this page to be handy from time to time:

http://www.cisco.com/en/US/products/ps6120/products_tech_note09186a00807e0aca.shtml

ISAKMP (IKE Phase 1) Negotiations States

The MM_WAIT_MSG state can be an excellent clue into why a tunnel is not forming. If your firewall is hanging at a specific state review this graph below to find where along the path the VPN is failing.

Graph source: tunnelsup.com

 

These are the possible ISAKMP negotiation states on an ASA firewall. ISAKMP stands for: The Internet Security Association and Key Management Protocol

ASA ISAKMP STATES

• MM_WAIT_MSG2 Initiator
  Initial DH public key sent to responder. Awaiting initial contact reply from other side.
  Initiator sends encr/hash/dh ike policy details to create initial contact. Initiator will wait at MM_WAIT_MSG2 until it hears back from its peer. If stuck here it usually means the other end is not responding. This could be due to no route to the far end or the far end does not have ISAKMP enabled on the outside or the far end is down.

• MM_WAIT_MSG3 Receiver
  Receiver is sending back its IKE policy to the initiator.
  Initiator sends encr/hash/dh ike policy details to create initial contact. Initiator will wait at MM_WAIT_MSG2 until it hears back from its peer. Hang ups here may also be due to mismatch device vendors, a router with a firewall in the way, or even ASA version mismatches.

• MM_WAIT_MSG4 Initiator
  Initiator is sending the Pre-Shared-Key hash to its peer.
  Initiator sends a hash of its PSK. Initiator will stay at MSG4 until it gets a PSK back from its peer. If the receiver is missing a tunnel group or PSK the initiator will stay at MM_WAIT_MSG4
  

• MM_WAIT_MSG5 Receiver
  Receiver is sending its PSK hash to its peer.
  Receiver does not yet check if PSK hashes match. If receiver has a tunnel-group and PSK configured for this peer it will send the PSK hash to the peer. If PSKs don’t match, receiver will stay at MM_WAIT_MSG5. I have also seen the tunnel stop here when NAT-T was on when it needed to be turned off.
  

• MM_WAIT_MSG6 Initiator
  Initiator checks if PSK hashes match.
  If PSK keys match, Initiator becomes MM_ACTIVE and lets receiver know of match. If PSK doesn’t match, initiator stays at MM_WAIT_MSG6. I have also seen the tunnel stop here when NAT-T was on when it needed to be turned off.
  However, if the state goes to MSG6 then the ISAKMP gets reset that means phase 1 finished but phase 2 failed. Check that IPSEC settings match in phase 2 to get the tunnel to stay at MM_ACTIVE.

• AM_ACTIVE / MM_ACTIVE
  The ISAKMP negotiations are complete. Phase 1 has successfully completed.

PIX ISAKMP STATES

• MM_NO_STATE

ISAKMP SA has been created but nothing else has happened yet.

• MM_SA_SETUP

The peers have agreed on parameters for the ISAKMP SA.

• MM_KEY_EXCH

The peers have exchanged Diffie-Hellman public keys and have generated a shared secret. The I SAKMP SA remains unauthenticated.

• MM_KEY_AUTH

The ISAKMP SA has been authenticated. If the router initiated this exchange, this state trans itions immediately to QM_IDLE and a Quick mode exchange begins.

• AG_NO_STATE

The ISAKMP SA has been created but nothing else has happened yet.

• AG_INIT_EXCH

The peers have done the first exchange in Aggressive mode but the SA is not authenticated.

• AG_AUTH

The ISAKMP SA has been authenticated. If the router initiated this exchange, this state transitions immediately to QM_IDLE and a Quick mode exchange begins.

• QM_IDLE

The ISAKMP negotiations are complete. Phase 1 successfully completed. It remains authenticated with its peer and may be used for subsequent Quick mode exchanges.

 

But what is the difference between MM and AM?

Main mode vs Aggressive mode. Here is a image taken from Cisco’s website to show the difference.

As you can see the Main mode is the same as the flowchart at the top of the page. Aggressive mode only uses 4 steps to establish the tunnel.

 

Errors With ISAKMP Or Phase 1 Establishing

A very common problem is phase 1 not establishing correctly.

• Verify ISAKMP parameters match exactly.
• Verify pre-shared-keys match exactly.
• Check that each side has a route to the peer address that you are trying to form a tunnel with.
• Verify ISAKMP is enabled on the outside interfaces.
• Is ESP traffic permitted in through the outside interface?
• Is UDP port 500 open on the outside ACL?
• Some situations require that UDP port 4500 is open for the outside.