Traceroute Through the ASA

The Cisco ASA has some interesting characteristics when dealing with traceroute.  With most traffic, including ICMP echo, outbound traffic can be inspected to allow the incoming traffic associated with the same flow.  Inspecting “ICMP” or even “ICMP Error” does not result in traceroute functioning through the ASA.

The first thing that we need to look at is how traceroute works.  Traceroute uses two different types of ICMP packets.  Windows systems use an ICMP with incrementing TTL’s to illicit an ICMP TTL exceeded message from each hop along the way.  Linux and Cisco use a UDP port with pseudorandom destination port.  With the UDP method, an incrementing TTL is still used to illicit a message from each hop along the way.  However, the message that is produced at the final destination is often an “ICMP unreachable port-unreachable” message.

Note to reader: All ASA content can be accessed by clicking here (or choosing ASA from the menu at the top of the page).

To understand how traceroute works, it is important to understand the function of the TTL field in the IP packet header.  This field is an 8-bit value that works with routers to keep packets from looping forever as a result of network configuration issues.  The way this is accomplished is that each router along the way decreases the value by 1.  Normal traffic may start out with a TTL of 64, 128, 255 or any other non-zero value.  As the traffic traverses a router, this TTL in the IP header is reduced by one.

When a router decreases the value to zero, it drops the packet.  When this happens the device will respond with an “ICMP TTL exceeded” if it is in response to an ICMP packet.  If a TCP or UDP packet makes it to the final destination without that port listening, the host will respond with an “ICMP Unreachable/Port-Unreachable”.  It is worth noting that any device can usually be configured to not respond at all.

Traceroute takes advantage of this behavior and generates a series of packets.  The first packet(s) will have a TTL of one and be dropped by the first router.  The next packet(s) will have a TTL of two and be dropped by the second router.   This is used to build a map of the network.  If there is a device configured not to respond traceroute will indicate the presence of a device, but its IP address will not be identified.

With our default ASA configuration, let’s see if traceroute will work.

Windows PC

C:\Documents and Settings\paul>tracert -d www.google.com

Tracing route to google.navigation.opendns.com [208.69.36.231]
over a maximum of 30 hops:

  1     *        *        *     Request timed out.
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12     *        *        *     Request timed out.
 13     *        *        *     Request timed out.
 14    16 ms    24 ms     7 ms  208.69.36.231

Trace complete.

As we can see, there is no intermediary information.  We know that we are not receiving the TTL-exceeded messages from the routers in the network.  The ASA requires special configuration to permit the traffic.  The first challenge is to permit these TTL-exceeded and port unreachable messages back into the network.  This can be done by using an ACL bound to the outside interface.

ASA Config

//create an ACL that permits the incoming ICMP
access-list outside_access_in remark ICMP type 11 for Windows Traceroute
access-list outside_access_in extended permit icmp any any time-exceeded
access-list outside_access_in remark ICMP type 3 for Cisco and Linux
access-list outside_access_in extended permit icmp any any unreachable

//bind the ACL to the outside interface
access-group outside_access_in in interface outside

Now let’s test traceroute again.

Windows PC

C:\Documents and Settings\paul>tracert -d www.google.com

Tracing route to google.navigation.opendns.com [208.69.36.230] over a maximum of 30 hops:   1     3 ms     3 ms     4 ms  

//note that this first hop is not my ASA
71.30.192.1   2     3 ms     6 ms     3 ms  
151.213.31.168   3     4 ms     3 ms     4 ms  
75.90.222.23   4     4 ms     4 ms    15 ms  
75.90.222.25   5     6 ms     6 ms     5 ms  
151.213.254.36   6     7 ms    10 ms     7 ms  
12.118.104.17   7     7 ms     7 ms     5 ms  
12.122.132.98   8    36 ms     8 ms    28 ms  
12.123.7.250 9    10 ms     6 ms     6 ms  
12.122.132.17 10    26 ms    10 ms    14 ms  
192.205.33.194 11    27 ms    18 ms    12 ms  
154.54.3.242 12    26 ms    14 ms    11 ms  
38.112.35.122 13     8 ms     9 ms     6 ms  
38.104.102.62 14     7 ms     7 ms     6 ms  
208.69.36.230 Trace complete.

Now that looks much better.  However, I can see that my ASA is not listed in the path.  That is very strange.  Upon investigation, I determine that the ASA itself does not decrease the TTL as it passes traffic.  Firewalls often play by slightly different rules than a router and this is one of those exceptions.  However, we can change this behavior using the set connection option in the modular policy framework (MPF).

ASA Config

ciscoasa(config)# policy-map global_policy
ciscoasa(config-pmap)# class class-default
ciscoasa(config-pmap-c)# set connection decrement-ttl

Now, let’s test traceroute again.

Windows PC

C:\Documents and Settings\paul>tracert -d www.google.com

Tracing route to google.navigation.opendns.com [208.69.36.231]
over a maximum of 30 hops:

  1     1 ms     *        1 ms  75.117.163.238 <— My ASA
  2     3 ms     3 ms     4 ms  71.30.192.1
  3     8 ms     3 ms     3 ms  151.213.31.170
  4    47 ms    31 ms     4 ms  75.90.222.23
  5    94 ms     4 ms     8 ms  75.90.222.25
  6     5 ms     4 ms     6 ms  151.213.254.36
  7    18 ms     5 ms     6 ms  12.118.104.17
  8     9 ms     6 ms     5 ms  12.122.133.98
  9     7 ms     7 ms     8 ms  12.123.7.110
 10     7 ms     6 ms    32 ms  12.122.133.13
 11    11 ms    11 ms    22 ms  192.205.33.194
 12    59 ms    48 ms    36 ms  154.54.3.234
 13    15 ms    12 ms    13 ms  38.112.35.122
 14    13 ms     5 ms    17 ms  38.104.102.62
 15     8 ms     6 ms     5 ms  208.69.36.231

Trace complete.

We can now see an extra hop (75.117.163.238 is an address on my ASA), but there are missing statistics (see the *). This is a result of the fact that the ASA is not responding to all of the traceroute packets.  This is due to the rate-limiting of ICMP on the ASA.  We can adjust this as well.

ASA Config

ciscoasa(config)# icmp unreachable rate-limit 10 burst-size 5

Now let’s test this one more time.

Windows PC

C:\Documents and Settings\paul>tracert -d www.google.com

Tracing route to google.navigation.opendns.com [208.69.36.230]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  75.117.163.238 <— My ASA
  2     6 ms     5 ms     4 ms  75.117.160.1
  3     5 ms     5 ms     4 ms  151.213.31.168
  4     5 ms     4 ms     6 ms  75.90.222.23
  5     9 ms     7 ms     7 ms  75.90.222.25
  6     6 ms     5 ms     8 ms  151.213.254.36
  7    10 ms     8 ms    14 ms  12.118.104.17
  8    19 ms     9 ms     8 ms  12.122.133.98
  9    12 ms     7 ms    30 ms  12.123.7.110
 10     8 ms    10 ms    29 ms  12.122.133.9
 11    38 ms    47 ms    14 ms  192.205.33.194
 12    12 ms    48 ms    32 ms  154.54.3.246
 13    15 ms    42 ms    14 ms  38.20.40.174
 14     8 ms     7 ms    26 ms  38.104.102.62
 15     9 ms     8 ms    11 ms  208.69.36.230

Trace complete.

Now we can see solid statistics on the first hop.  Now our ASA is working correctly with traceroute traffic.  I want to show one more example of a way to break traceroute.

Let’s set the IP Audit Attack policy on the outside interface.

ASA Config

ciscoasa(config)#ip audit name myattack attack action alarm drop
ciscoasa(config)#ip audit interface outside myattack

Now we can run our test again.

Windows PC

C:\Documents and Settings\paul>tracert -d www.google.com

Tracing route to google.navigation.opendns.com [208.69.36.231] over a maximum of 30 hops:
1     *        *        *     Request timed out.   2    19 ms    19 ms     4 ms  75.117.160.1 ^C

We can see the issue has resurfaced.  If we have logging enabled, we can see the IDS engine is detecting this as a “land” attack.

ciscoasa(config)# show log | inc IDS
%ASA-4-400008: IDS:1102 IP land attack from 75.117.163.238 to 75.117.163.238 on

Let’s disable just this one signature.

ASA Config

ciscoasa(config)# ip audit signature 1102 disable

Now we can retest this once again.

Windows PC

C:\Documents and Settings\paul>tracert -d www.google.com

Tracing route to google.navigation.opendns.com [208.69.36.230]
over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  75.117.163.238
  2    11 ms     8 ms     4 ms  75.117.160.1
  3     9 ms    13 ms    11 ms  151.213.31.168
  4     7 ms     8 ms    10 ms  75.90.222.23 
 ^C

Now you can see that we have a successful traceroute configuration on our ASA.  So if I were to receive a lab question that says to make sure traceroute will work through my ASA, that question can mean several things.  If I get such a question on the lab and it is vague, I will verify with the proctor exactly what a successful traceroute means.  I will also keep in mind that if I enable ids on the ASA that I should probably re-check the traceroute to verify it still works.

Firepower Administrators should also check out – Traceroute through Firepower Threat Defense.

No related content found.

About Paul Stewart, CCIE 26009 (Security)

Paul is a Network and Security Engineer, Trainer and Blogger who enjoys understanding how things really work. With over 15 years of experience in the technology industry, Paul has helped many organizations build, maintain and secure their networks and systems.
This entry was posted in How-To and tagged . Bookmark the permalink.

28 Responses to Traceroute Through the ASA

  1. Ed says:

    Hi Paul, I want to enable traceroutes through from my desktop to a secure network. Path is as follows:
    Desktop -> router -> FWSM v3.2(4) (transparent mode, multi context) -> ASA 5520 v8.4(2) (routed mode) -> secure network. Any help is appreciated.

    • Ed, I have been experimenting with “icmp inspection error”. That is an interesting and underdocumented command (to say the least). I was planning on writing a post on it (and may still do that at some point), when I found a great blog post by Joe Astorino. One challenge I have found to this command (at least in my initial testing) is that if there is a configuration that matches a PAT entry for the intermediary hosts, the ASA drops the outgoing icmp errors if error inspection is enabled. When it does, it produces the error below. I’ll try to keep this thread posted if I post a inbound equivalent of this article

      Error:
      %ASA-3-305006: regular translation creation failed for icmp src inside:192.168.3.2 dst outside:192.0.2.1 (type 3, code 3)

  2. Depending on the desktop operating system, you would most likely be initiating an ICMP (Windows) or UDP (Linux) based traceroute. So those packets would need to be delivered toward the destination. This means that there might need to be a “static” or equivalent “nat” (in 8.4) statement in the new syntax. Exceptions would also be required in the applicable ACL’s. Additionally, the ICMP messages (TTL Exceeded and Unreachables) would need to flow back in the other direction.

    For the ASA itself to show up in the trace, the TTL would need to be decremented as described in this article. I doubt that would work on the transparent FWSM since it is basically a bump on the wire.

    Some additional magic can be performed on the ICMP messages as they flow back toward the desktop. This is invoked by enabling “ip inspect error” on the applicable policy-map. This will cause the real IP addresses to be exposed to the station that initiated the trace.

    I wish I had an 8.4 inbound example readily available, but I don’t today. I’ll try to get one out there in the future.

  3. Umar mushtaq says:

    Thanks for the post..!!

  4. Umar mushtaq says:

    Keep up the good work..

  5. deneb829 says:

    Paul,
    This finally got traceroute working through my ASA. I had been working on this on and off for awhile from a variety of different web and book sources. From your instructions, I found I was missing one line:
    access-list outside_access_in extended permit icmp any any time-exceeded
    This fixed it! Also liked the part about adding the ASA to the trace.
    Thanks!
    Mike

  6. Hi, this is a hit ! any other blogs don’t fix this.
    Thank you Paul, from Rosario, Argentina
    Ernesto Vilarrasa
    CCNA, CCNA Security, CCAI

  7. Ben says:

    “When a router decreases the value to zero, it drops the packet. When
    this happens the device will respond with an “ICMP TTL exceeded” if it
    is in response to an ICMP packet. If it is in response to a TCP or UDP
    packet, it will respond with an “ICMP Unreachable Port-Unreachable”. ”

    Hi Paul,

    correct me if I am wrong but doesn’t a UNIX based UDP traceroute packet illicit an ICMP time-exceeded response when the TTL is decremented to 0. My understanding is that all hops along a UNIX traceroute path will respond with an ICMP time-exceeded packet except for the destination who accepts the packet as it is addressed to themselves but replies with a Port Unreachable packet because the destination ports (something like UDP 33467) is not open on the host/device.

    So my understanding is that for UNIX/IOS traceroute ICMP time-exceeded and unreachable packets are required to be allowed to return but all except on (the final) of these packets in response to the traceroute will be ICMP time-exceeded with the last being ICMP unreachable.

    Ben

    • Paul Stewart says:

      Your understanding seems to be the same as mine. That aligns with what I’ve seen in the field and, to the best of my recollection, what I saw when I wrote this article.

      • Aris says:

        Paul, dont you say the opposite in the article? That is each hop (router) responds with “Port-Unreachable”.
        I also think that the response is “time-exceeded ” except the final destination which replies with “Port-Unreachable”

  8. Medford says:

    I know I’m late to the party, but I wanted to thank you for a great article – excellent instructions and explanations. Much appreciated!

  9. Ralf Mann says:

    Nice! Thanks!

  10. thug113 says:

    Nice article ..
    I tried to solve this issue and ((google it)) for about 3 months and finally you gave us the perfect solution …
    Thanks for this great article ..
    with regards;

  11. Pingback: ASA problems

  12. Dino says:

    Thanks, definitely helpful and succinct.

  13. Bjon says:

    Great article. Luckily it was the first one I read. Issue resolved.
    Thanks Paul

  14. Kevin says:

    Thanks for the artikel, windows tracert is working now! But somehow traceroute on Apple’s macbook still does not work. So it must use a different method, any idea on how to let macbooks trace as well?

    • Apple uses UDP ports with a base destination port of 33434. It will increase with each additional hop. If this isn’t working with OSX, my assumption is that you are permitting ICMP egress, but not UDP. Obviously, weigh the risks against the benefits to your deployment.

  15. Pingback: Allowing traceroutes though the ASA firewall | Random Tech Notes

  16. Naser says:

    HI Paul
    In this case after letting the ASA permit the tracert then how secure is the ASA, or it has noting to do with security

    • There is some security impact. I would qualify my position by saying any connectivity has some associated risk. The safest computer is the one that is buried in concrete somewhere, but we know that to be unrealistic. I find that we often have to balance security and functionality. So the risks from an ASA perspective fall into two categories–1) the ability to protect hosts and 2) the ability to protect the ASA itself. I typically wouldn’t be overly concerned about the security ramifications of permitting traceroute through a firewall. Hope that helps.

      • Naser says:

        Hi Paul
        Thank you for replying ,but my idea is to enable the access list when you want to test something and then disable it when you finish .
        In my case I have some access lists just for specific things but all the time are disabled .
        thanks again.

      • There’s certainly nothing wrong with your approach. I have often used ACEs in an ACL set to disabled for similar use cases. The approach is conservative, but there are use cases that require least privilege.

  17. Roberto says:

    Hi Paul, thanks for this great post.

    However by decreasing the TTL, I had to use ebgp multihop 2 for the direct connected BGP neighbors at a layer 3 perspective as the BGP neighhborship is set with a TTL of 1 by default.

    So I have a ASA 5520 in transparent mode in between the ISP and internal routers and a /29 block for the interconnection, I use one IP address for the ISP router, another for the internal router which has the BGP session configured, and another one for the BVI interface used at the ASA.

    If I do not decrease the TTL at the ASA, the BGP session is formed successfully, however if I decrease the ttl and clear the BGP session, the BGP session is not coming up again until I configure it with ebgp multihop for increasing the TTL to 2 or more.

    Is there any other way to overcome the ebgp-multihop feature in order to form the BGP session with the TTL being decreased at the ASA?

    Thanks,

    Roberto

  18. Craig Tompkins says:

    Will this allow trace on all interfaces or just Outside? In addition to my outside interface I have one that I’ll call “ABC” that I’d like to trace across. Do I just add “access-group outside_access_in in interface abc” as well or is there more too it?
    Thanks for any help!

  19. MANI says:

    thank you !! I was struggling to solve the same issue from my scenario . This is helpful for me . Thanks again .

  20. Willy Chuang says:

    Hi Paul
    I have been following with your step,but…still fail,
    i already haven’t any idea…so, can you give me a hand , please…!!
    my e-mail is [email protected] , can i send config & topology for you ?
    let me know,Thank you.

Comments are closed.