On Tuesday, June 7, 2011 Grasshopper.com experienced a severe outage that lasted for more than 2 days. For Grasshopper.com it was not only a major technical disaster but also a PR nightmare! One thing is for sure, there will be many lessons learned from this “disaster” – on both the technical and PR sides.
At the very start of the problem, no communication went out to Grasshopper.com customers notifying them of the problem. This was most likely due to the fact that all internal systems at Grasshopper.com were down(including their main website and phone lines), along with access to all of their customers email accounts. So now they have a major outage that is affecting their entire customer base, with no way to notify that customer base. That’s bad to begin with.
Now add to this nightmare, the fact that customers of Grasshopper.com had to suffer through the embarassment of having calls go to dead phone lines or much worse, a message that stated that their phone line had been “temporarily disconnected”, making it seem like those customers hadn’t paid their bills. This can cause a small business unmeasureable damage to their reputation, not to mention the lost business.
Twenty four hours into the outage, Grasshopper was still not giving any indications of what had happened or when their systems would be back up and running – leaving customers with no idea of when they could expect to have working phones/faxes again. For a full explanation of what happened you can read about the Grasshopper Outage details here and the Co-Founders Response.
Unfortunately for a company whose sole purpose is to provide reliable phone service, it seems that Grasshopper.com failed on several levels in this situation. Anyone that works in IT will tell you that hardware is going to fail… it comes with the territory… and even in some unfortunate cases RAID or redundant systems will fail. It happens!
That’s why you have backups and fail over systems… but it’s also why you TEST those systems! Setting up backups and fail over systems and then forgetting about them is one of the biggest wastes of time and money any company can undertake. Having a fail-over system and never testing it is the same as not having one at all! Because, as Grasshopper.com recently learned the hard way, when it comes down to “brass-tacks”, and needing that fail-over site… it may just fail on you too!
The Grasshopper.com co-founders have responded that they have learned many lessons from this outage and have listened to the many customers voicing their frustrations. They either have, or are in the process of putting measures in place, to insure that an outage like this doesn’t happen again, and that if in the future there are any other issues with their service that they are better equipped to communicate with their customers.
We hope that this is truly the case, and at the very least they take some of the profits from Grasshopper.com and reinvest it into that company and it’s infrastructure and testing, rather than in other Grasshopper group startup companies such as Chargify – an online billing system – and the recently failed Spreadable(a referral marketing tool)which launched last October(2010). Grasshopper.com’s 100,000 customer base at a minimum, deserves the dependable, reliable phone system they thought they were purchasing.
Take away’s from this whole situation for any business using a Virtual PBX or contemplating using a VirtualPBX:
- Investigate what backup systems your Virtual PBX provider has in place and ask if they test them!
Example: VirtualPBX Does Reliability Right
- If you can not live with an outage, have a backup toll free number with another provider
- If you can not live with an outage(regardless of duration), then consider if the cost savings of a Virtual PBX is right for you. Remember however, even landlines experience outages!