My web hosting company Dotster and I are getting a divorce – firing each other is another way to think about it. I’m not sure they know it, but they should. My bags are packed, the car is gassed up, and I’m just figuring out where to go next. And in this saga there’s a lot to say about how good companies go astray, and how good companies stay good.
Trust – whether it’s a colleague, friend, client, or vendor like Dotster – can be parsed into three elements: reliability, faith in motive, and competence. Lose any of those three and you lose trust: lose all three and you’re a walking disaster if you work with other humans.
And as written elsewhere here , running a company requires the ability to do the 3 R’s well: reliability, responsiveness, and recovery. Dotster of late has struggled with all three.
I picked my web hosting company out of a recommendation from a former colleague from the days of my time as a principal in a surviving Internet development company. While I’m not a technologist, I know enough topically to know when I’m being gamed, and I’ve spent enough time in the industry to know reasonable standards of service and competence. I’d been with Yahoo small business web hosting earlier, and while they were OK, customer focused service were not three words I would use to describe them.
Dotster appealed to me on several fronts: decent reputation (then), smaller, scrappy firm, and they were based in Vancouver, Washington, right across from the Portland metro area where I grew up. I figured we Pacific Northwesterners needed to band together and if you couldn’t trust one of your own, whom could you trust?
Until a few months ago Dotster was fine. My little static HTML site for my executive and team coaching practice at BackWest.com chugged along. Not much activity, but it was solid.
Three months ago I started blogging, using a WordPress install option on the Dotster hosting side at this address. Things were fine, I got a chance to share thoughts on how people, teams, and organizations perform and can improve their performance, and my volume started to climb to several thousands of page views a week.
Around 40 days ago my blog site was down and when I followed up the response was “Yeah, it looks like the (shared) server is down. We’d been having problems with it and were wondering if it was going to fail.” Blog hosting is not flying jets, but even so it’s not the phrase I wanted to hear – if something was going to fail, I would hope that the web hosting side would catch it as it failed, rather than wait for a customer to call in to report it.
A fire started in an electrical room Friday , July 3rd before the 4 th of Saturday holiday at a place in Seattle called Fisher Plaza. It toasted any number of companies who had web hosting operated from that facility, including some hosted by Dotster. Though the fire had started at 11 PM on Thursday night before, no info was on the Dotster support page when I called in at 8 AM on Friday, July 3 rd , which was a holiday for them and they were operating with holiday hours rather than the usual 6 AM start. The customer service rep did know about the outage, and said they’d be posting something. They posted a notice shortly after 8 AM, and another mid-afternoon with an estimated time for sites back up in the 5-11 hour range.
I was a little surprised in the age of Twitter, and when Dotster has e-mail address and phone numbers, no one called to say “Hey! Your site’s down. We don’t know when it will be up but we’re on it.”
Dotster’s choice of response reminded me that I had received an automated robot call from Costco when the peanut butter salmonella scare hit and it wasn’t clear if the peanut flavored Clif Bars (I’d bought some) at Costco had any issues. Costco ran through cashier sales records to find out who had bought any potentially contaminated items, cross referenced those sales records with the membership ID numbers to locate phone numbers, and fed the data to an automated robocaller to get the word out.
On the Friday afternoon I called Dotster and the their word out was they hoped to be back up in 5-11 hours. I could kiss any traffic good bye for that day. Early on Independence Day morning my site was back up.
The bumpy service from Dotster was the inspiration for a post here , spelling out the qualities that make good customer service – well – good.
Twitter was alive, as you might imagine, with tweets about the fire and anyone who was impacted, such as Dotster. @Dotster_Inc. saw a tweet I posted, saw my post on customer service, and wrote, “Checked out your blog post. We’re going to explain more what happened now that the weekend is over and service is restored.”
On Wednesday Dotster posted to their support page the following as their update:
July 8, 2009
We would like to take this opportunity to provide greater detail with respect to the fire at our Seattle data center on Friday, July 3, 2009 and how it affected Dotster and our customers.
• The fire only affected one of our data centers, but we gave it our full attention. In response to the situation, we brought in additional customer support staff to field calls and emails and mobilized our entire IT operations team to work to restore service as quickly as possible. We also posted regular updates on our support pages and updated phone messaging as we learned more information.
• The fire damaged the power distribution equipment in the building where our data center resides in Seattle, WA just before midnight on July 2, 2009. This interrupted the building’s power distribution to the data center that Dotster uses and interrupted normal redundant power systems so that within a few hours the power in UPS systems was depleted and our servers went offline. Other large companies who use this data center were affected as well and experienced similar outages.
• As soon as we were notified of the fire, Dotster sent IT staff to the data center to prepare to bring servers back online upon restoration of the building’s power distribution system. The data center engineers quickly determined that vital systems in the electrical room were damaged and could not be immediately repaired after the Fire Marshall cleared the facility for re-entry. As a result, extra generators were brought in to ensure resources were available to power and cool the building. Much of the outage time resulted from waiting for the building power to be restored and waiting for the building to cool enough to restart servers without heat issues.
• As soon as power was restored, our employees went to work bringing systems back online. Once they went to work, some systems were back online almost immediately, and nearly all services were restored within four hours.
This was an unfortunate situation, and we sincerely apologize for any inconvenience it may have caused. The outage lasted much longer than we would have liked, but thank you for bearing with us during these extraordinary circumstances.
Posted by Customer Care – 07/8/2009@08:58
Nice information and would have been helpful if was out on a timely basis (Sunday or Monday after the outage). Since most customers are not looking into support pages unless there is a current problem, posting there instead of some form of outbound communication (Twitter, e-mail, phone call) would have seemed more appropriate. Wednesday morning seemed like a push. And if you search the Dotster tweets, you’ll find that other folks had concerns as well. John Penn tweeted “When things go wrong, like a failure at a data center, companies are tested. Dotster failed me. No response to web, mail, phone or twitter.” With ears burning red, the low profile communication was underwhelming.
This past Thursday my blog site started to fail, again. First the WordPress stats pack wouldn’t come up, and then the whole blog and static html sites went down. I called customer support early in the afternoon after all the remote items from my end said things showed green. The customer service guy said it looked like there was some sort of serious trouble, and the server engineers would have to fix it. He indicated he wasn’t sure what the problem was or when it would be fixed.
Three hours later, with the sites still down, I called back, and got another customer service rep. More helpful, he said that the server folks would need to catch it, and because of their work schedule (early in, usually early out) it was not likely to happen until the next day. Did you know, he asked, about Dotster’s 24-hour turnaround on service requests? Meaning, I asked, that it’s possible it would be down until the following late morning? Yes was the answer I didn’t’ want to hear, and the one he gave me.
I pinged the Dotster tweeter and they provided me with an e-mail of someone I could ping directly. I e-mailed him and 90 minutes later received a note back:
I apologize for the delay; we have restored the service to the hosting package. It seems you are going over the maximum connection limits our shared hosting servers give for a database. You are allotted 20 simultaneous connections to your database at any given time. Since WordPress and other blog software primarily run off a database it is recommended that you get either a Virtual Private Server or a dedicated server to house the website. We have restored the blog and it is working http://backwest.com/wordpress.backwest.com/ however it is more than likely that this will occur again. It is not a bandwidth, or disk space or even size of the database its self simply your site being too popular. We do offer VPS packages at http://www.dotster.com/products/hosting/vps/ they are more expensive than a shared hosting package but it will get the job done and you wouldn’t see down time due to connection issues.
With the site mostly back up (blog was up, static html would not be fully back up until mid-morning on Thursday), I pinged the contact to see if I could catch him to talk about the VPS solution he recommended. I’m still waiting to hear back from him.
I did talk to a customer service rep the next morning about the issue, and possible solutions. Since I’m barely scratching the surface in any of the usual areas (bandwidth, visitors, size of database, etc.) their take was the going to VPS was overkill and to be avoided.
Next steps – find a new hosting service and transition over. There is an old saying: “Fool me once, shame on you. Fool me twice, shame on me.”
I have been a fan of Costco – another Pacific Northwest based company – forever. You might not think of big box companies as customer serviced focused but you’d be wrong to apply that to Costco, and they have a lot they can teach people about customer focused service.
I know by sight roughly 50% of their cashiers from my every six weeks Costco run over the last 10 years, which is a high level of retention you’re unlikely to find at other places such as Walmart or Target. People, it seems don’t turn over at Costco, and that usually suggests there’s something that keeps them there. Costco’s sales per square foot – the holy grail of retailing – make other competitors like Walmart green with envy.
And unlike my experiences at Walmart, staff at Costco are helpful. When my then 3-year-old son managed to duck behind an aisle and get separated from me, the front desk at Costco had the place on lockdown in 20 seconds, and my son, shaken and crying, back to me in three minutes. And last week a Costco employee without asking walked me to the new location for an item that had just been moved, all the way to the opposite end of the store.
It’s been well observed that we live in a world where our expectations – think food, politicians, or vendors – get set at the high-water mark. Nordstrom’s (I know, another company from the Pacific Northwest) set standards for customer service that informed places and people miles from the retailing environment for the last 20 years. When people have a choice, they chose to spend their money on companies that provide good value with good service. We live in an era where service expectations are like Costco, not Walmart.
So while a place like Costco seems to practice the 3 R’s of good customer service, and earn the trust of a single customer like me, Dotster seems to struggle with those customer service R’s – even one – and losing considerable trust in the process.
I’m sure that Dotster will be fine without my business. But I still remember my friend Sue Spezza, from growing up in Oregon, talking about how her then big company Wang went out of business. “Wang.” she said, “went out of business one customer at a time.”