The website has been down for abut 14 hours and I am very sorry for that. It is the first time that this has happened to me in 7 years of owning and running the website. The problem is exclusively with the company, Endurance International Group brands (EIG) who host the site. They own many website hosting companies including the following: Bluehost, HostMonster, HostGator, and JustHost. PoC is hosted by Hostgator, one of these companies.
Every website has to be hosted by a business of some sort and not by the person who owns the site. This is the normal arrangement. The reason why a person who owns a website cannot host it himself is because it is complicated and requires resources so nearly all website owners pay a company to host the site for them. There are many such companies.
Some research has indicated that EIG has suffered similar problems in the past (August and December 2013) which is disturbing to me. The current problem originates in what they call a datacentre in a town called Provo, Utah. Hostgator is based in Houston Texas.
A datacentre is a place were many computers called servers are kept in controlled conditions and it is these servers which hold the websites. The datacentre is owned by Ace Data Centers. I do not know whether they are another company that is owned by EIG but that may be the case.
So the cause of the downtime is technical problems, probably mechanical problems in a datacentre in the middle of America. The technical problems prevent this website being seen on the Internet. Update: it was caused by a bug (this means a programming defect)
Our Networks Operation team uncovered a bug in firmware utilized in our vendor’s hardware that caused the network issues, which impacted some of our customers
In addition, I suspect that many hundreds of thousands perhaps millions of other websites were also down because EIG hosts over 10,000,000 websites or more.
This is a massive problem and a catastrophe for Hostgator and their parent company EIG. I am extremely disappointed in them. PoC is held on a dedicated server which is an individual computer dedicated to holding the website. It is expensive. A lot of other websites are held on shared servers. Despite that the problem still existed.
The company promises 99.8% uptime meaning 0.2% downtime. On my calculation that means if the website is down for more than 17 and a half hours in one year they are in breach of their promise. At the time of writing this the site has been down to 12 hours approximately. If this goes on, it is conceivable that they will be sued by hundreds of thousands of website owners for breach of contract. That could be another catastrophe.
Update: the downtime lasted about 14 hours on my calculation.
It was hopefully a good choice, is what I meant, and we are all supporting you. This must be so worrisome. Only time will tell in this situation, right? You must be tremendously busy doing the necessary research from all angles. Maybe we can all do a bit. At least it gives us more insight into what you have to deal with on a daily basis. 🙂 Cherish your strength and determination, Michael.
Yes, Caroline, it was worrisome because I wasn’t sure whether data was going to be lost or whether personal details were going to be accessed. It as also worrisome because I did not want the site to be inaccessible for a long time because you lose revenue through advertising and people might begin to believe that the website has been closed down. This website will never be closed down while I am alive and healthy.
That, my ailurophilic friend, is exactly what I hoped you would say, emphatically. <3 …hmph.
I’m guessing that EIG will not allow a breach like that to occur again, if it can be helped. and it can. (No doubt, we are all thinking the same thing…a bug in the software that should not have been there.) I can hardly imagine how that felt, Michael. Many of the popular websites were down, internationally, and I think we all know what it feels like to question your recent choice in hosting. My heart goes out to you. -cal
Thank you Caroline. Yes, hundreds of thousands of websites were down. I had a pleasant surprise yesterday evening when I received an e-mail from the hosting company written by the CEO requesting that I telephone the company for a credit. I telephoned and received a credit for $219 which I think is quite good. I have the impression that many bigger sites who sell things online will have received much larger sums in compensation. Clearly, the company is frightened of people leaving and moving their websites which is distinctly possible because 14 hours or more downtime is totally unacceptable.
I’m currently with Poundhost and though expensive (as I need a lot of space & resources for multiple sites on my server) they’ve been reliable and their customer service has been excellent. The only downtime I had was during migration a while back as the DNS nameservers were being updated (these get rippled through the web).
Thanks Sarah. I’ll check it out. I hate to move the site again as it is a nightmare.
I was wondering if “P.O.C” was being censored for some reason or the other !
Rudolph, I would have been proud if that was the case! Sadly not as far as I can tell. It was a simple software/hardware problem at a datacentre which holds the website. As I understand it, perhaps hundreds of thousands of other sites were similarly affected.
What a nightmare for you- and for the folks wanting to get to the site.
Thank you for your notice, and letting us know what happened. You are a DEDICATED provider to all of us, Michael.
On my reckoning the site can be down for no more than about 4 hours the remainder of the year because if it is I have the right to sue them for compensation!
There must be a lot of big sites who make money through sales which have suffered loss and I wonder how those are being dealt with.
Michael we pay £29.99 +VAT for a dedicated server with 1and1 dot co dot uk. They haven’t so far let us down. They are a huge company so talking to them can be annoying – first you always get an automated response. But I guess that’s normal.
I won’t tell you what I pay for a dedicated server but it is much more than that. And I am being let down and so are hundreds of thousands of other people by the sound of it. I would move the website if it was easier to do, but moving a website the size of this one can be a nightmare because often you get formatting problems and if there are 6000+ pages (on the main site) you get an unmanageable number of formatting problems. The one good thing about Hostgator is they do have good customer service until, as was the case in this instance, they are flooded out with phone calls in which case they never answer the phone but they did put out a notice on their website which was constantly updated informing us what was going on, so there was some communication.