HomeSite Downtime – What Happened?
Just putting up a quick post to explain the site outage over the weekend, which saw a series of unfortunate events combine to make this the longest downtime since the infamous datacenter fire of 2008.
And it’s all because of a stupid reverse DNS entry.
First up, sorry for the inconvenience of the last few days. But as frustrating it must have been for you, believe me, it was much more frustrating from where I was standing, being able to do very little while the site remained down.
What is Reverse DNS?
Before we get to what happened, a little backgrounder on what a reverse DNS entry is. I think most people already know what a normal DNS entry does, in that it translates the domain name (eg. example.com) to an IP address (eg. 1.2.3.4). A reverse DNS entry does the opposite, by telling people what domain name the IP address belongs to.
Reverse DNS entries are not really as useful as normal DNS entries, without which would make it impossible to use domain names (not just for web browsing, but also for emails). Reverse DNS entries are mostly to help humans find out quickly a domain name that’s linked to the IP address, and also for email servers for security purposes – most email servers will reject emails sent from IP addresses that do not have a reverse DNS record, although it doesn’t really matter what the record actually says.
Unlike normal DNS records, reverse DNS entries are not managed by the person who holds the domain name, but by ISPs and web hosts that owns the IP address. For web hosts, when they assigns an IP address to a server that you rent, they may change the reverse DNS entry to match the name of your server (for example, server1.digital-digest.com), which while not essential, looks nice at the very least.
So what happened?
Well, an IP address that no longer belonged to us, but once did, still had the reverse DNS record of the digital-digest.com domain name (while the reverse DNS record may have been assigned automatically to us when the server was procured, when the server was subsequently cancelled, the reverse DNS record apparently remained, for years afterwards). This IP address was being used by its new servers for a phishing scam. A company that investigates this sort of thing did a reverse DNS lookup and found that digital-digest.com was the entry. Using information from the WHOIS entry of digital-digest.com, this web security company subsequently sent emails to us and to our domain name registrar (and possibly others) to inform of the possible abuse going on. This is despite little or not effort apparently being made to check if the IP address did still belong to Digital Digest, which it did not.
Our domain name registrar then decided to suspend the domain name immediately, even though as the IP address was being used in the phishing links, suspending the domain name did nothing to actually prevent the phishing link from continuing to work (this is assuming that the domain name still had something to do with the IP address, which it no longer did).
A rough analogy would be the post office cancelling your mail deliveries because your old phone number, which is still listed in the most recent issue of the White Pages (as you had moved after it was published), was being used in a scam!
And while I was informed of the suspension shortly after it occurred, due to time differences (I was asleep at the time), I wasn’t to know until hours later. Unfortunately (and this is my fault entirely), I chose a domain registrar that did not have 24×7 tech support, and so the issue could not be resolved until Monday despite emails and unanswered phone calls.
Without an active domain name, nothing would work even though the site itself was still running on the server
Adding to my bad fortune, Monday was a public holiday in the US, but luckily, somebody had turned up to work, read my email explaining that the IP address had nothing to do with this domain name anymore, and re-activated the domain name. The only piece of luck in this whole incident.
As nobody really came out of this incident with any credit, myself included, I shall forgo assigning blame. Suffice to say that lessons have been learnt and that I will be transferring my domain names to a different registrar, one that has 24×7 support.
The registrar I’m transferring to, Namecheap, was also one of the many involved in the anti-SOPA protests of last year. Protesting GoDaddy’s then support of SOPA last year, Namecheap hosted a promotion that gave away cheap domain transfers for one day, with some of the proceeds going to support Internet freedom groups. And coincidentally, today is Internet Freedom Day, celebrating the defeat of SOPA a year ago, and the same promotion is running again, with at least $0.50 (up to $1.50, depending on the number of transferred domains) from every cheap $3.99 domain transfer (normally closer to $10) going to the Electronic Frontier Foundation. Which makes the decision to switch registrars that much simpler for me 🙂