HostGator reported on Wednesday, April 16, 2014, at 11:00 AM CST USA, that their Provo datacenter had a network outage. The outage turned into a solid downtime of 13 to 15 hours for some hostees, including myself. In fact, my dedicated server is still having hiccups – short downtime spells.
This is what I had to look at for nearly 15 hours:
There have still been hiccups reported as late as 5PM CST USA on Thursday, April 17, 2014. Just to expand upon the seriousness of the outage, this downtime includes the following web hosting companies: Hostgator, Justhost, Hostmonster, and Bluehost.
Joshua Martin, Director of Customer Service at HostGator stated that the reason for this outage was:
We experienced a degradation of network service in one of our data centers due to a firmware bug in one of our vendor’s hardware solutions. This was an undocumented bug and we worked with our partner to diagnose the issue and deployed a firmware update to the systems to remediate the problem. Only websites that were being served by this hardware were affected.
Mr. Martin also revealed that it took so long to fix the problem because:
We started to address the problem immediately when we began to see performance issues. The root cause of the problem was complicated to diagnose because it was an undocumented bug in software of a vendor’s hardware solution. Full service for some customers was restored immediately, but some servers were not visible on our network. We apologize for any downtime that you experienced. The servers continued to operate during this entire period, which means, that at no point in time was your data at risk. The problem was access to the servers because of the firmware issue.
This network outage wasn’t the first occurrence. In fact, since Endurance International Group acquired HostGator, in late 2012, there have been several outages. It’s important to note that while HostGator and many other hosts are under EIG (Endurance), that they are actually independently operated.
One longtime HostGator member, guruperl, even recalled the HostGator outage dates since 2013:
No excuse. If you updated the firmware during the day you were stupid. If you patched a bug into the network load balancer, firewall or edge routers, you should have only done it to one or two of them to see how they responded. This outage tells me you don’t have an edge network that can withstand the outage of one device, it tells me you don’t have a core network that can be routed to ‘hot” standby, and it tells me that you rely on the “vendor” to solve the problem vs. having talent in house to do so.
More responses from Facebook:
Much of the responses were somewhat the same, including those on Twitter.
On the jesting side, I found the following meme being passed around. (Let me know the origin of this source so I can credit them.):
While there is no doubt about an outage, and no doubt that some hard work was done, this has left a bad taste in a lot of HostGator hostees’ mouths. Much of the problems seems to stem from yet another procedure problem. Most of the responses from dedicated server customers like myself have revolved around the fact that server data was moved from the Texas data center to the one in Utah without the customer’s permission.
Are you using HostGator? Did you experience downtime on April 16th or 17th of 2014? And if so, for how long? What were your thoughts during the downtime? Are you going to remain with HostGator?