Comcast is Fucking With Your Port 53 Traffic

Ok, *my* port 53 traffic. And maybe yours. And it’s probably Earthlink, not Comcast.

Posted in Uncategorized by comcastisfuckingwithyourport53traffic on June 9, 2009

Update:
Holy shit, it’s stopped.

I re-ran the same experiments a few times over the course of the day to try things that people suggested (for example, using the IP address instead of the name “testserv.mydomain.com” — which made no difference, incidentally), and my results were the same every time.

Just now, I tried again, and the traffic makes it from ‘homemachine’ to ‘testserv’ completely unadulterated.

Now all you frenzied slashdotters can tell me I imagined the whole thing. Let’s see if it lasts…

[original post left for posterity]

We all know about the annoying DNS tricks folks like Comcast play, right?  The ones where you mistype a site name and you end up on an annoying advertising page because their DNS server resolves it to their own server?  Well, it’s worse than you think.

See, some people are actually savvy enough to simply not use Comcast’s DNS servers so they don’t get this obnoxious behavior.  But Comcast has put a stop to that, at least for some people.  (Full disclosure: I don’t know if it’s Comcast or Earthlink that’s responsible for this behavior, but Comcast is who I pay for internet access, so I’m blaming them for now, even though it’s obvious Earthlink is involved).  What they actually do is to take any UDP traffic bound for port 53 on any server and redirect it to their own server.  This means that you simply can’t use a different DNS server, even if you have one.

Here’s a little experiment I did to prove it.  There are three computers involved: “testserv.mydomain.com” is a Slicehost machine I have for various experiments (note: actual domain is not mydomain.com, of course).  “homemachine” is my laptop, sitting at home and connecting via Fucking Comcast.  “okok” is a machine which is elsewhere (doesn’t matter where other than to say it’s not connected via Goddamn Comcast).  All of these machines are running Ubuntu linux (Hardy, in case you care).

Step 1: Start a fake DNS server on testserv:

On the server, run the following command:

testserv$ sudo nc -l -p 53 -u | od -hc

This listens for UDP packets on port 53, which is where DNS requests will show up, and it prints them to stdout after piping them through ‘od’ so the binary garbage doesn’t hose up my terminal.

Step 2:  Test it from a known good computer (“okok”)

On the “good” machine (“good” in this case means it is not using comcast), run this command:

okok$ nslookup comcast.sucks.com testserv.mydomain.com

On the server, I see the following output from the nc process:

0000000 69f4 0001 0100 0000 0000 0000 6307 6d6f
 364   i 001     001              \a   c   o   m
0000020 6163 7473 7305 6375 736b 6303 6d6f 0000
 c   a   s   t 005   s   u   c   k   s 003   c   o   m   
0000040 0001 f401 0169 0000 0001 0000 0000 0700
 001   001 364   i 001     001              \a
0000060 6f63 636d 7361 0574 7573 6b63 0373 6f63
 c   o   m   c   a   s   t 005   s   u   c   k   s 003   c   o
0000100 006d 0100 0100 69f4 0001 0100 0000 0000
 m     001   001 364   i 001     001       
0000120 0000 6307 6d6f 6163 7473 7305 6375 736b
     \a   c   o   m   c   a   s   t 005   s   u   c   k   s

Now, granted, the “DNS server” we have set up is just netcat, and it will never respond, so the client tries a few times (the above hex dump shows three requests), and then eventually gives up:

;; connection timed out; no servers could be reached
okok$

This is what we’d expect.  We’ve proven that an nslookup from ‘okok’ to ‘testserv’ makes it through to our fake server, which doesn’t respond.  So far so good.

Step 3: Try it from the home machine (via comcast):

Same test, but from the machine behind comcast:

homemachine$ nslookup comcast.sucks.com testserv.mydomain.com
Server:        testserv.mydomain.com
Address:    [censored]#53

Non-authoritative answer:
Name:    comcast.sucks.com
Address: 207.69.131.9
Name:    comcast.sucks.com
Address: 207.69.131.10

homemachine$

Well look at that! The request succeeded! Even though we explicitly specified a server which doesn’t actually respond!  And we can see from netcat that the server never even received a request!  Oh joy!  The astute reader will notice that the addresses returned are those of the Earthlink host-not-found advertising page.

Step 4: Verdict

Comcast can suck it.

Note that this setup actually causes me a lot of problems, because Comcast seems to have a hell of a time keeping their DNS servers actually functioning.  Tonight I spent about 30 minutes connected to the internet and able to contact any server I knew the address of with no trouble, but DNS requests all failed because their DNS server was down.  This happens frequently.  Back in the day when I could, I’d simply manually point my machines at a known good DNS server and continue on, but now these fuckers actually hijack my DNS requests and send them to their broken server.  Thanks a lot, assholes.  You just earned yourself a support call, which is a complete waste of my time and your money.  The support folks just have me reboot my modem until the problem goes away, which it eventually does regardless of whether I call or not.  And nobody there is particularly receptive to or capable of understanding the troubleshooting I’ve done.

This is why your customers hate you, you fuckers.

Step 5: Solutions

I haven’t actually set this up yet, because it’s going to be a pain in the ass and frankly, I shoudln’t fucking have to, but here’s what you’re going to force me to do, Comcast.  I’m going to set up a bind proxy which tunnels requests to an outside host (e.g. testserv) via SSH so the packet-sniffing dicks can’t get their grubby mitts on them.  Fuck you for making this necessary.  You suck.

Addenda

Note 1: Someone either here or on slashdot commented that there might be a problem looking up “testserv.mydomain.com” from the home machine, which is why the request ends up not going to my server. This theory doesn’t hold water, because nslookup will complain in that case, plus I actually performed a similar experiment using TCP port 53 and netcat on both ends. The name was resolved fine, TCP port 53 makes it through. Switching to UDP and netcat on both ends results in the traffic going to Earthlink’s servers.

Note 2: It’s true, I have no actual proof this is being done by Comcast. Let’s just say that someone controlling the path between me and my server really wants me to use a different DNS server than the one I’m trying to use. And the server they want me to use returns Earthlink error pages.

Note 3: Someone named “ComcastBonnie” over on twitter, who seems to be some kind of official Comcast twitsperson, is vociferously denying this. For a while today, she was just tweeting “engineering confirms we do not hijack any DNS traffic in our network and certainly not to 3rd party resolvers” over and over. I have yet to see any plausible alternative explanations for the results of my very simple experiments. Am I missing something? Tell me.

44 Responses

Subscribe to comments with RSS.

  1. Marin said, on June 9, 2009 at 6:47 am

    NFW

  2. davidwr said, on June 9, 2009 at 6:28 pm

    >Goddamn Comcast

    Now now there’s no need to be redundant.

  3. Matt said, on June 9, 2009 at 6:29 pm

    why do I not hear any verification of this on slashdot?

  4. a said, on June 9, 2009 at 6:33 pm

    I’d check to make sure your router isn’t changing to it’s DNS server during the NAT translation

    • comcastisfuckingwithyourport53traffic said, on June 9, 2009 at 6:58 pm

      This happens whether my router is in the picture or not

  5. deadtoyou said, on June 9, 2009 at 6:35 pm

    RFC the motherfuckers to the ground!

  6. Quentin Hartman said, on June 9, 2009 at 6:43 pm

    Looks like this is a regional/service level thing and/or someone else is the culprit. I’m on Comcast business at my office here in Oregon and my DNS requests are moving along unmolested.

  7. user said, on June 9, 2009 at 6:50 pm

    openvpn … screw blocking outgoing SMTP, outgoing DNS, etc… ya, it may be a pain,
    initially, but getting back that control is well worth it…

  8. Just Keep The Internet Working said, on June 9, 2009 at 6:55 pm

    “I’m going to set up a bind proxy which tunnels requests to an outside host (e.g. testserv) via SSH so the packet-sniffing dicks can’t get their grubby mitts on them. Fuck you for making this necessary. You suck.”

    This is the wave of the future, I think. From a customer’s perspective, all I want my ISP to do is concern itself with quickly and reliably moving data where it’s supposed to go. When this model transformed into something where the ISP can play with the content of the communication I don’t know, but it’s none of their business and it’s blatantly anti-consumer. It’s really time to figure out a way to take what we’ve learned from HTTPS and both encrypt all communications and make computers loudly complain if the machine at the other end isn’t what you expected.

    • comcastisfuckingwithyourport53traffic said, on June 9, 2009 at 7:00 pm

      Agreed. I actually think a paid service that offers a no-nonsense VPN for home users (with support) will actually make sense if the last mile bastards don’t get their acts together.

  9. Brian said, on June 9, 2009 at 6:59 pm

    http://www.opendns.com/ works fine for me. When I got my account, I could not get past the “agree to TOS” screen. They said my account was messed up, and not even the Gods of Tier 2 could fix it. So I just switched to opendns and life has been great since. Last night twitter dns had an issue and up came the friendly opendns page, so at least of last night everything was still working.

    I am happy with opendns, and I am happy that I never had to *formally* agree to comcast’s tos.

  10. Eric Goff said, on June 9, 2009 at 7:03 pm

    This article is flat out wrong.

    Try :

    nslookup http://www.google.com flibitteyglibbit.com

    And presto, it works. Perhaps testserv.mydomain.com simply could not be resolved or reached for some reason ?

    • comcastisfuckingwithyourport53traffic said, on June 9, 2009 at 7:14 pm

      That only works if you’re having your DNS hijacked. This is what it’s supposed to look like:

      okok$ nslookup http://www.google.com flibbiteyglibbit.com
      nslookup: couldn’t get address for ‘flibbiteyglibbit.com': not found
      okok$

      (btw, neither I nor Eric actually typed “http://” in the command; that’s wordpress auto-linkifying)

  11. fist said, on June 9, 2009 at 7:07 pm

    I can’t reproduce this…from what you’re saying, I should be able to dig any IP address & get a response as if its’ a DNS server?

    doesn’t seem to work…
    dig @1.5.4.1 yahoo.com
    dig @208.99.1.2 yahoo.com
    dig @yahoo.com yahoo.com

    • comcastisfuckingwithyourport53traffic said, on June 9, 2009 at 7:22 pm

      Yep, that’s exactly what I’m saying:

      homemachine$ dig @208.99.1.2 yahoo.com


      ;; ANSWER SECTION:
      yahoo.com. 12978 IN A 209.191.93.53
      yahoo.com. 12978 IN A 69.147.114.224
      yahoo.com. 12978 IN A 209.131.36.159

  12. Cogwheel said, on June 9, 2009 at 7:08 pm

    “Comcast can suck it”

    I’ve been saying this for years! I finally get to make it so… AT&T’s u-verse just came to my neighborhood. SUCK IT, COMCAST!!!

  13. comcastisfuckingwithyourport53traffic said, on June 9, 2009 at 7:11 pm

    As a datapoint for those trying to prove/disprove my claim, I’m in the Seattle area purchasing internet access from Comcast, and am apparently getting the “Earthlink” package. I have no idea what it even means to buy Earthlink service from Comcast, other than the fact that sometimes when I call about trouble, Comcast tells me I should call Earthlink and Earthlink tells me I should call Comcast. Best I can tell, you get a special discount for allowing two separate companies a crack at fucking up your internet access.

  14. Frank said, on June 9, 2009 at 7:17 pm

    Not saying that you’re wrong. I’m just saying that this doesn’t happen on my Comcast connection. In fact it goes to OpenDNS the way it’s supposed to. I’d like to see if other people are having the same problem.

    • comcastisfuckingwithyourport53traffic said, on June 9, 2009 at 7:19 pm

      I don’t disagree. I am not surprised to hear it’s not universal.

  15. rattus said, on June 9, 2009 at 7:18 pm

    The broken DNS thing is simple. They run the enterprise Cisco DNS crap. They employ more than few people simply to kick it when it falls over. This happens on an hourly basis.

    Why not just replace it with a workable solution? I don’t know either.

  16. PerComp said, on June 9, 2009 at 7:21 pm

    I feel the same way about comcast – but damn your comments made me laugh! You are just stating what we all feel.

  17. Harry said, on June 9, 2009 at 7:21 pm

    I can’t replicate when you’re experiencing. DNS traffic goes thorugh from my comcast connection just fine.

  18. James said, on June 9, 2009 at 7:27 pm

    Along with everyone else: NACK
    DNS goes where I tell it from Comcast in Maryland.

    • comcastisfuckingwithyourport53traffic said, on June 9, 2009 at 7:56 pm

      It’s quite possible that this problem isn’t actually very widespread. But it looks to me like Eric Goff (below) is actually confirming it and misinterpreting the results.

  19. Bret said, on June 9, 2009 at 7:31 pm

    I can’t replicate this and I have comcast. I can’t help but notice that you’re using a mydomain.com address. I hope you realize that testserv.mydomain.com is a real server that may or may not have been running DNS at the time…

    • comcastisfuckingwithyourport53traffic said, on June 9, 2009 at 8:43 pm

      Eh, you will notice in the article that I specifically say I’m not really using “mydomain.com”.

  20. Eric Goff said, on June 9, 2009 at 7:37 pm

    nslookup http://www.google.com flibitteyglibbit.com
    Server: build.labnet.local
    Address: 192.168.2.1

    Name: http://www.l.google.com
    Addresses: 74.125.45.105, 74.125.45.99, 74.125.45.104, 74.125.45.147, 74.125.45.106
    74.125.45.103

    No error messages where officer

    • comcastisfuckingwithyourport53traffic said, on June 9, 2009 at 7:41 pm

      Eric, your test does not show what you think it does. nslookup attempted to talk to flibitteyglibbit.com, which does not exist, and it succeeded. This is because the DNS request did not actually go to flibbitteyglibbit.com.

      what happens if you try to look up a nonexistent address, e.g. “nslookup flibitteyglibbit.com flibitteyglibbit.com”?

  21. Tyler said, on June 9, 2009 at 7:40 pm

    I have run an adblocking DNS server. Sometimes public internet connections such as at coffee shops or hotels prevent one from using another DNS server.

    My solution (works):
    Run set DNS server to 127.0.0.1
    Use Fpipe: Fpipe -l 53 -u -r 123 70.75.160.232 (using my dns server)

    fpipe downloadable at

    http://www.foundstone.com/us/resources/proddesc/fpipe.htm

    Send feedback to adblocking@calgarycomputerfix.com

  22. [...] 2009/06/09 — koitsu This is in response to the Slashdot article and the official blog post claiming that Comcast is transparently tinkering with TCP port 53 [...]

    • comcastisfuckingwithyourport53traffic said, on June 9, 2009 at 8:43 pm

      Nice post title :)

  23. Sam_dal said, on June 9, 2009 at 8:43 pm

    Don’t know how long you have been with Comcast but long time ago they had sent out a notice (to Dallas residents) that all residential users’ DNS requests with go through their DNS servers no matter what. I did not want it and their service so terminated their service.

  24. Kelly Hallman said, on June 9, 2009 at 9:03 pm

    I am in California and I am not able to reproduce this.

    $ nslookup flibbiteyglibbit.com flibbiteyglibbit.com
    nslookup: couldn’t get address for ‘flibbiteyglibbit.com': not found

    I set up your example and was able to query the external non-Comcast port 53 listener without problems. So the title of your post/blog is not necessarily correct. It should be “Comcast is fucking with MY port 53 traffic”.

  25. Nathan Campbell said, on June 9, 2009 at 9:20 pm

    This may not be applicable for every comcast connection. I manage the internet for a co-op and use a computer running pf-sense as the gateway/router. The gateway is set to use open-dns as the dns server, and I can see all the reroute stats on the open-dns page. Now this could be because I have a business comcast connection and not a residential connection. I do know that the 250GB/month rule doesn’t apply to business accounts since we push more than 450GB a month since we have 62 college students utilizing the connection. Maybe this dns reroute only applies to residential customers.

  26. James said, on June 9, 2009 at 9:26 pm

    Just an FYI. It is possible to get Earthlink internet service through Comcast. My mother has this. She is billed by Comcast, but her connection is routed completely through Earthlinks’ network.

    So while you may have Comcast internet service, its probably Earthlink doing the “mucking” with traffic.

    Earthlink also redirects her HTTP page not found requests to an ad site. I live about 15 miles away, but have pure Comcast. Mine do not.

  27. Nobody said, on June 9, 2009 at 9:48 pm

    Interesting. The nc method is a good idea.
    As workaround i would use an DNS server which is not on port 53.
    Here i found a list of uncensored DNS servers which can also be reached
    via port 110:

    http://server.privacyfoundation.de/

    You can test it with

    dig @85.25.251.254 -p 110 | grep SERVER

    dig @85.25.251.254 -p 110 +short yahoo.com

    And i found software for getting blacklists from DNS servers by comparing an uncensored and a censored one:

    http://apophis.ch/de/node/120

    http://sourceforge.net/projects/censorshiptools/

  28. Anonymous Coward said, on June 9, 2009 at 10:39 pm

    I encountered this same issue when doing some work during early 2008 using a Sprint EVDO card on the Sprint network.

    I even tried using an alternate UDP port for DNS and yet I still received responses which clearly were not actually from my destination server. (tcpdump verified, the packets never reached the destination.)

    Took me quite some time to figure out what was going on.

  29. Jason said, on June 10, 2009 at 1:44 am

    Sounds like you subscribe to Earthlink’s Internet service, which use Earthlink DNS servers, not Comcast service with Comcast DNS servers.

    Jason
    Comcast National Engineering & Technical Operations

    • comcastisfuckingwithyourport53traffic said, on June 10, 2009 at 6:18 am

      That’s quite possible, but it doesn’t change the fact that Comcast is the one selling me the service. Maybe you need to have a talk with those folks who are sullying your good name.

  30. C. Robert Allen said, on June 10, 2009 at 2:46 am

    I can confirm DNS is redirected from Comcast IP 76.115.5.109 in Salem, Oregon using NetCat to open a port on a server I have sitting in Seattle, Washington and verifying the port responds to a dig client sitting in Kent, Washington. The Seattle and Kent machines are not on Comcast’s network.

    If I allow Comcast’s server to resolve my request (both existant and NXDOMAIN) the result is correct. If I try to force use of a specific resolver, my request never makes it to the resolver and times out.

  31. samrolken said, on June 10, 2009 at 4:42 am

    I don’t get any of this port 53 redirection or anything, but I can back up what you say that Comcast’s dns servers for their customers are crap. I spent a long time with weird internet “outages” that were really just DNS servers being broken and host names not resolving.

    I wonder how much comcast pays people to support customers whose internet is broken and can’t just fix their dns servers? it’s so embarassing. For this and other face-palmingly sad reasons, I’m obviously no longer a comcast customer.

  32. deep grewal said, on June 10, 2009 at 7:12 pm

    This was a hilarious article. You hate Comcast like rappers from the late 80s hated the police. The language you used, in my opinion, is awesome and not overly done.

    • comcastisfuckingwithyourport53traffic said, on June 10, 2009 at 8:04 pm

      Your sincere and one hundred percent correct compliment is appreciated.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: