TOT ADSL problems with Yahoo, Hotmail, MSN, sending mail

Symptoms: user cannot access Yahoo, Hotmail, cannot send large mail; MSN Messenger is unable to login.

Were affected: TOT ADSL customers connected via Pacific Internet.

First case reported on 8/09/2005. Our company submitted the findings to the Pacific Internet administration on 15/09/2005 about 1 am; the problem was fixed late morning of 15/09/2005.

Diagnostics

Let's first make a traceroute from the affected computer to some well-known host not too far away, Loxinfo DNS server for example:

  1     3 ms     2 ms     2 ms  192.168.123.254
  2    16 ms    16 ms    15 ms  tot-102-130.pacific.net.th [221.128.102.130]
  3    33 ms    32 ms    32 ms  tot-105-17.pacific.net.th [221.128.105.17]
  4    39 ms    36 ms    32 ms  172.23.126.38 (RFC1918)
  5    34 ms    34 ms    35 ms  sw6509-cat079.pacific.net.th [203.121.130.79]
  6    51 ms    50 ms    47 ms  221.128.127.216
  7    37 ms    37 ms    36 ms  202.129.63.97
  8    36 ms    35 ms    43 ms  202.129.63.82
  9    35 ms    35 ms    39 ms  210.1.46.225
 10    35 ms    34 ms    36 ms  cor21-G-dom.csloxinfo.net [210.1.45.17]
 11    34 ms    36 ms    39 ms  cor36-G-cor21.csloxinfo.net [210.1.45.35]
 12    41 ms    38 ms    39 ms  ns.loxinfo.co.th [203.146.0.20]

Seeing the router with private network RFC1918 adress in the path (#4) should hint any person calling himself a network engineer about possible path MTU discovery problems.

Ping to 203.146.0.20 works; but if we try to ping using larger packet size (-l 1300) and the "don't fragment" switch (-f):

C:\>ping -l 1300 -f 203.146.0.20

Pinging 203.146.0.20 with 1300 bytes of data:

Request timed out.
Request timed out.
Request timed out.
Request timed out.

Here we can see what our packets, being 1300 bytes in size, cannot reach the target, or the reply (of the same size) cannot reach us. Let's lower the size (you might need to try it several times until you find a size which works):

C:\>ping -l 1000 -f 203.146.0.20

Pinging 203.146.0.20 with 1000 bytes of data:

Reply from 203.146.0.20: bytes=1000 time=119ms TTL=55
Reply from 203.146.0.20: bytes=1000 time=121ms TTL=55
Reply from 203.146.0.20: bytes=1000 time=140ms TTL=55
Reply from 203.146.0.20: bytes=1000 time=304ms TTL=55

This tells us what somewhere on the way we have a router with lower MTU (maximum transmission unit) than usual. We can actualy try to ping the routers closer and closer to us to determine what exact router is causing problem. But why we don't get the usual ICMP "Fragmentation needed " message from the problem router? Because it's most probably blocked somewhere on the way back! Now I will explain my suspicions about RFC1918 address router. It's a good practice to block on the firewall the packets coming from the RFC1918 addresses (to prevent attacks on the private addresses subnets). But if they are blocked, and there is a router with the RFC1918 address in the path, you will never get any ICMP replies from it, including "fragmentation needed". And if you cannot get these packets, the Path MTU Discovery mechanism breaks; your system cannot determine the correct MTU size anymore.

Please note what in our case browsing most of the sites works, it means what Path MTU Discovery works from their side and they can send us correctly sized packets. If we try to send a small size email , like the simple "test" string, it will work because packet sent to SMTP server is still smaller whan the MTU of the problematic router. But the email with HTML body or with the attachment will fail, because your system will try to send it in several full-sized (according to your system MTU) packets.

MSN Messenger fail because it needs to send a large packet just to log in; Yahoo Messenger, on the other hand, works with smaller packets and still able to log in.

Why Yahoo and Hotmail fail while other websites work? Remember what with each HTTP request browser sends cookies to the web server. With Hotmail and Yahoo (and some other sites of course) the request size with cookies exceeds again our problematic MTU threshold and again our system don't receive any notification about it. Sessions just time out.

Workarounds

For Windows systems, read "How to Troubleshoot Black Hole Router Issues" from the Microsoft Knowledge Base. We recommend Method 1 "Enabling PMTU Black Hole Detection"; however, all workarounds will cause some performance degradation.

Related Pages

Path MTU Discovery and Filtering ICMP. More detailed explanation of the same effect.