Ok, *my* port 53 traffic. And maybe yours. And it’s probably Earthlink, not Comcast.

Update:
Holy shit, it’s stopped.

I re-ran the same experiments a few times over the course of the day to try things that people suggested (for example, using the IP address instead of the name “testserv.mydomain.com” — which made no difference, incidentally), and my results were the same every time.

Just now, I tried again, and the traffic makes it from ‘homemachine’ to ‘testserv’ completely unadulterated.

Now all you frenzied slashdotters can tell me I imagined the whole thing. Let’s see if it lasts…

[original post left for posterity]

We all know about the annoying DNS tricks folks like Comcast play, right?  The ones where you mistype a site name and you end up on an annoying advertising page because their DNS server resolves it to their own server?  Well, it’s worse than you think.

See, some people are actually savvy enough to simply not use Comcast’s DNS servers so they don’t get this obnoxious behavior.  But Comcast has put a stop to that, at least for some people.  (Full disclosure: I don’t know if it’s Comcast or Earthlink that’s responsible for this behavior, but Comcast is who I pay for internet access, so I’m blaming them for now, even though it’s obvious Earthlink is involved).  What they actually do is to take any UDP traffic bound for port 53 on any server and redirect it to their own server.  This means that you simply can’t use a different DNS server, even if you have one.

Here’s a little experiment I did to prove it.  There are three computers involved: “testserv.mydomain.com” is a Slicehost machine I have for various experiments (note: actual domain is not mydomain.com, of course).  “homemachine” is my laptop, sitting at home and connecting via Fucking Comcast.  “okok” is a machine which is elsewhere (doesn’t matter where other than to say it’s not connected via Goddamn Comcast).  All of these machines are running Ubuntu linux (Hardy, in case you care).

Step 1: Start a fake DNS server on testserv:

On the server, run the following command:

testserv$ sudo nc -l -p 53 -u | od -hc

This listens for UDP packets on port 53, which is where DNS requests will show up, and it prints them to stdout after piping them through ‘od’ so the binary garbage doesn’t hose up my terminal.

Step 2:  Test it from a known good computer (“okok”)

On the “good” machine (“good” in this case means it is not using comcast), run this command:

okok$ nslookup comcast.sucks.com testserv.mydomain.com

On the server, I see the following output from the nc process:

0000000 69f4 0001 0100 0000 0000 0000 6307 6d6f
 364   i 001     001              \a   c   o   m
0000020 6163 7473 7305 6375 736b 6303 6d6f 0000
 c   a   s   t 005   s   u   c   k   s 003   c   o   m   
0000040 0001 f401 0169 0000 0001 0000 0000 0700
 001   001 364   i 001     001              \a
0000060 6f63 636d 7361 0574 7573 6b63 0373 6f63
 c   o   m   c   a   s   t 005   s   u   c   k   s 003   c   o
0000100 006d 0100 0100 69f4 0001 0100 0000 0000
 m     001   001 364   i 001     001       
0000120 0000 6307 6d6f 6163 7473 7305 6375 736b
     \a   c   o   m   c   a   s   t 005   s   u   c   k   s

Now, granted, the “DNS server” we have set up is just netcat, and it will never respond, so the client tries a few times (the above hex dump shows three requests), and then eventually gives up:

;; connection timed out; no servers could be reached
okok$

This is what we’d expect.  We’ve proven that an nslookup from ‘okok’ to ‘testserv’ makes it through to our fake server, which doesn’t respond.  So far so good.

Step 3: Try it from the home machine (via comcast):

Same test, but from the machine behind comcast:

homemachine$ nslookup comcast.sucks.com testserv.mydomain.com
Server:        testserv.mydomain.com
Address:    [censored]#53

Non-authoritative answer:
Name:    comcast.sucks.com
Address: 207.69.131.9
Name:    comcast.sucks.com
Address: 207.69.131.10

homemachine$

Well look at that! The request succeeded! Even though we explicitly specified a server which doesn’t actually respond!  And we can see from netcat that the server never even received a request!  Oh joy!  The astute reader will notice that the addresses returned are those of the Earthlink host-not-found advertising page.

Step 4: Verdict

Comcast can suck it.

Note that this setup actually causes me a lot of problems, because Comcast seems to have a hell of a time keeping their DNS servers actually functioning.  Tonight I spent about 30 minutes connected to the internet and able to contact any server I knew the address of with no trouble, but DNS requests all failed because their DNS server was down.  This happens frequently.  Back in the day when I could, I’d simply manually point my machines at a known good DNS server and continue on, but now these fuckers actually hijack my DNS requests and send them to their broken server.  Thanks a lot, assholes.  You just earned yourself a support call, which is a complete waste of my time and your money.  The support folks just have me reboot my modem until the problem goes away, which it eventually does regardless of whether I call or not.  And nobody there is particularly receptive to or capable of understanding the troubleshooting I’ve done.

This is why your customers hate you, you fuckers.

Step 5: Solutions

I haven’t actually set this up yet, because it’s going to be a pain in the ass and frankly, I shoudln’t fucking have to, but here’s what you’re going to force me to do, Comcast.  I’m going to set up a bind proxy which tunnels requests to an outside host (e.g. testserv) via SSH so the packet-sniffing dicks can’t get their grubby mitts on them.  Fuck you for making this necessary.  You suck.

Addenda

Note 1: Someone either here or on slashdot commented that there might be a problem looking up “testserv.mydomain.com” from the home machine, which is why the request ends up not going to my server. This theory doesn’t hold water, because nslookup will complain in that case, plus I actually performed a similar experiment using TCP port 53 and netcat on both ends. The name was resolved fine, TCP port 53 makes it through. Switching to UDP and netcat on both ends results in the traffic going to Earthlink’s servers.

Note 2: It’s true, I have no actual proof this is being done by Comcast. Let’s just say that someone controlling the path between me and my server really wants me to use a different DNS server than the one I’m trying to use. And the server they want me to use returns Earthlink error pages.

Note 3: Someone named “ComcastBonnie” over on twitter, who seems to be some kind of official Comcast twitsperson, is vociferously denying this. For a while today, she was just tweeting “engineering confirms we do not hijack any DNS traffic in our network and certainly not to 3rd party resolvers” over and over. I have yet to see any plausible alternative explanations for the results of my very simple experiments. Am I missing something? Tell me.