We had a power outage in this area a week and a half ago. Normally, this is not an issue. However, this was wide-spread enough that Comcast decided to add custom routes in the network, and hasn't figured out how to remove them.
What it means for me
I have to have Internet access for my job. I need it to be accessible from both sides (public to the private network, and vice-versa). However, I cannot ping the public IP address that they provide from the public side. I can't connect to it from there, either. Since I'm an idiot with some networking experience, I start looking at trace routes. From the inside :
traceroute to virtualserver (XXX.XXX.XXX.XXX), 64 hops max, 52 byte packets
1 router (192.168.1.1) 7.623 ms 1.502 ms 1.832 ms
2 XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX) 10.849 ms 11.573 ms 11.102 ms
3 te-0-5-0-16-sur04.saltlakecity.ut.utah.comcast.net (68.87.220.177) 11.262 ms 13.049 ms 11.200 ms
4 te-0-5-0-8-sur03.saltlakecity.ut.utah.comcast.net (69.139.231.57) 11.564 ms
te-0-10-0-4-ar02.sandy.ut.utah.comcast.net (69.139.231.97) 15.901 ms *
5 te-2-3-ar01.saltlakecity.ut.utah.comcast.net (68.86.180.65) 16.767 ms 14.778 ms 15.931 ms
6 pos-0-9-0-0-cr01.denver.co.ibone.comcast.net (68.86.90.233) 27.885 ms 28.708 ms 24.411 ms
7 te3-5.ccr01.den03.atlas.cogentco.com (154.54.10.33) 36.089 ms 246.011 ms 185.173 ms
8 te0-1-0-3.ccr22.den01.atlas.cogentco.com (154.54.83.33) 24.435 ms
te0-6-0-3.ccr22.den01.atlas.cogentco.com (154.54.45.185) 27.180 ms
te0-1-0-3.ccr22.den01.atlas.cogentco.com (154.54.83.33) 26.129 ms
9 be2131.mpd22.mci01.atlas.cogentco.com (154.54.26.130) 81.675 ms 82.868 ms 87.086 ms
10 be2157.ccr22.ord01.atlas.cogentco.com (154.54.6.118) 89.675 ms 88.852 ms
be2159.mpd22.ord01.atlas.cogentco.com (154.54.24.82) 92.166 ms
11 be2140.ccr22.bos01.atlas.cogentco.com (154.54.43.186) 124.252 ms 122.275 ms
be2138.ccr22.bos01.atlas.cogentco.com (154.54.43.202) 124.109 ms
12 endurance-international-group.demarc.cogentco.com (38.97.106.34) 122.540 ms 122.649 ms 123.540 ms
13 * * *
14 vpslinkrouter (XXX.XXX.XXX.XXX) 122.334 ms 124.073 ms 127.955 ms
15 virtualserver (XXX.XXX.XXX.XXX) 121.408 ms 121.116 ms *
From the outside looking back to that IP address :
traceroute to XXX.XXX.XXX.XXX (XXX.XXX.XXX.XXX), 30 hops max, 40 byte packets
1 vpslinkrouter (XXX.XXX.XXX.XXX) 0.063 ms 0.082 ms 0.073 ms
2 72.22.78.1 (72.22.78.1) 0.297 ms 0.392 ms 6.757 ms
3 237.252.148.207.static.yourhostingaccount.com (207.148.252.237) 0.232 ms 0.259 ms 0.316 ms
4 te0-7-0-4.ccr22.bos01.atlas.cogentco.com (38.97.106.33) 0.654 ms 1.991 ms 1.971 ms
5 be2096.ccr22.jfk02.atlas.cogentco.com (154.54.30.41) 6.401 ms be2097.mpd22.jfk02.atlas.cogentco.com (154.54.30.117) 6.502 ms 6.519 ms
6 be2061.ccr21.jfk05.atlas.cogentco.com (154.54.3.70) 6.563 ms 6.502 ms 6.400 ms
7 be2073.ccr21.jfk10.atlas.cogentco.com (154.54.0.230) 8.090 ms be2074.ccr21.jfk10.atlas.cogentco.com (154.54.86.226) 7.971 ms be2073.ccr21.jfk10.atlas.cogentco.com (154.54.0.230) 6.892 ms
8 comcast.jfk10.atlas.cogentco.com (154.54.13.162) 57.804 ms 57.789 ms 57.833 ms
9 he-2-1-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.83.101) 61.152 ms he-2-4-0-0-cr01.newyork.ny.ibone.comcast.net (68.86.83.105) 58.932 ms 58.892 ms
10 comcast-isc.pao1.isc.org (68.86.88.222) 77.619 ms 77.642 ms 77.613 ms
11 he-1-13-0-0-cr01.denver.co.ibone.comcast.net (68.86.85.246) 100.511 ms 100.459 ms 100.443 ms
12 pos-0-11-0-0-ar01.sfsutro.ca.sfba.comcast.net (68.86.90.226) 109.087 ms 112.585 ms 111.537 ms
13 te-0-1-0-5-sur03.saltlakecity.ut.utah.comcast.net (162.151.9.214) 111.017 ms 109.492 ms 111.080 ms
14 te-6-1-acr04.saltlakecity.ut.utah.comcast.net (68.87.170.222) 110.072 ms 108.898 ms 110.121 ms
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
Outbound traffic was routed from SLC, UT, USA to Denver, CO, USA. In Denver, things take a different path than the inbound traffic. In fact, inbound traffic is getting routed through New York City, NY, USA. Even with THAT, it should be easy to pin point the problem.
Why I hate "Big Box" ISP's
So, I jump on the phone with them. 2 hours and 25 minutes (yes, I tried for this long), 8 transfers (3 times at the same group, too), 9 "that's beyond what I can do for you", and two online chat sessions later, my phone battery dies. Not one idiot works for Comcast that knew that when I talked about an IP address in Denver that was their router, I wasn't meaning my home network router. When this contract is over (if I can find a good ISP), kiss Comcast good bye. If someone wants to pitch in to buy me a fiber connection, I'd be one happy camper! But, in the mean time, I have to deal with big box idiots that associate "router" with "in house wireless" rather than a nice Foundry device sitting at their Denver office.