About the author.

Welcome to The blog of whall

Come on in and stay a while… laugh a little. Maybe even think. Read more...

Hi, This is Wayne. This is my site, my stuff, my blog, blahblahblah. The site itself is powered by WordPress and the Scary Little theme. I thought it was cool, and I still do.

December
27
2006
3:38 pm
Categories:
Tags:
Post Meta :

As you may have seen in my previous post about my sites being down at 1&1, I was waiting for a response to my complaint letter. In summary, my internet hosting was inaccessible (ie, “down”) for 4 days while I made numerous attempts with their support department to understand and ideally resolve the outage. I was handed “The Support Runaround” baton approximately three million times and in the end, the resolution was what I had suggested to them the first day (me being a Unix admin myself).

My complaint letter is below in the extended entry, as is 1&1‘s reply. I must say, they surprised me. With some of the other customer experiences I’ve seen on some forums, blog posts, and some dissatisfied rants, I was expecting AT BEST a “we’re sorry this happened, oh, that’s terrible, here’s a free month of hosting and we’l try not to do it again. kthanxbye!” My actual expectation was more along the lines of being ignored or maybe being dropped for being a vocal complainer.

Instead, I received a response that was actually very nice. In fact, it had the following qualities:

Specific Quality of Response Letter What it showed me
Written quite well someone there is competent.
Lengthy and detailed enough they took my complaint seriously
Addressed my explicit concerns they heard me
Went into full disclosure of the breakdown on their end they are honest about their mistakes.
Assured me that my experience is irregular gave me confidence about staying
Expressed understanding and shared frustration at what I went through they have empathy; increasing confidence
Offered compensation via service credit they put some teeth into the apology
Were gracious with generous offer they’re humble and polite enough

So in general, I’m happy again. I’m glad 1&1 is going to make it easy to stay a customer.

My initial complaint letter to 1&1

Customer # ——-
Case # 6077—– and 6079—-
 
You can read the actual 1&1 case notes if you wish for whatever your support techs put down as why I called in several times, or you can read my explanation here.  I’m looking for some sort of decent explanation as to why this episode happened to me, and if this is normal operating procedure for your company.

I know it’s lengthy, but it accurately portrays my frustration and hopefully some action will be taken as a result.  I’ve put my request for action at the bottom.

My webspace was down four days. 

This is the biggie.  Had I not been down 4 days I probably wouldn’t care about much else.

Several of my sites stopped functioning sometime Friday morning, the 15th.  They just started working again this morning, the 19th.  Although the majority of the 120+ domains I have hosted at 1&1 are simply parked at sedo (through your partnership with them), the outage affected more than just a few of my websites: whall.org, jennefer.net, ncgraustin.com, austinastrology.org, astrologicalsocietyofaustin.com, fergopedia.com, 4mon.com and others. 

I can understand outages here and there – I work as IT Director for a large ISP and believe me, I know outages.  But I cannot understand what would cause four (4) days of downtime, coupled with absolutely no explanation or help/support from your support department.

What ended up being the issue was that I was being migrated from one package to another.  And it failed somehow.

No notification / business hours

There was absolutely no notification by 1&1 to me that this was going to happen. 

Evidently, my downtime was due to abnormal or failed migration from the legacy Premier Business Package to the Developer package.  I surely can appreciate the added drive space and the added transfer (who wouldn’t?) but there was no advanced warning of the migration.

Plus, this migration started during US business hours.  While I understand your Admins likely have a million things to do, and catering to some customers’ schedules might not be possible, notification would be key in times like this.

Had I been notified, I could have easily gotten up a spare server on my own, parked my webspace there and redirected DNS to my servers, with redirects and what-not, and not had any downtime and you could have taken all the time you wanted to migrate me to the new server hotness.  But I had no messages on my account or any emails to that effect.

No back-out plan?

So I mentioned that there was no notification of the migration about to occur.  I understand things get missed sometimes, and process isn’t always followed.

Ok, so why wasn’t I notified once the migration was failing?

Also, once my migration was absolutely a failure (I would think that if a site wasn’t migrated within 8-12 hours, we should consider it a failure), I should have been moved back to the old server so I could recover, get some uptime and then do other preparations for a future migration attempt.  I even called in and begged to be moved back to my old server, or to be able to manually create some redirect files on the new server.  Anything.  I even said “hey, you can delete this one directory, which is 29GB, because I can recreate it”.  Still, no response and no back-out plan.

Timing couldn’t have been worse

Quite literally, just a week ago I convinced several colleagues about how good 1&1 was and how happy I’d been for the last 5 years.  Service was great, price is outstanding, uptime always real good, and the servers are decently fast enough. 

So we transfer a bunch of domains from other providers to 1&1 (not to mention procuring many other new domains), only to have the server just totally go absent during this lengthy, unannounced migration.

So I look quite the fool after touting 1&1’s qualities.  I’ve been a customer for a very long time and always comment about how great 1&1 is. I link to you on my sites.  I rave about you in user groups.  People ask me where I host and I gladly tell them.

And then this incident happens and it has reshaped my opinion of your company.  Is this what you want?

Your support has been *no* help

I could dredge a re-enactment of my calls into your support but I’ll sum them up here: they were no help at all.

The first time I called, I was told “oh, just wait 30 minutes and it should get taken care of.”  Second time I call, 2 hours later, I’m told “just wait 24-48 hours” but absolutely no explanation whatsoever of why I would wait.

Then on a subsequent call, the tech support person is babbling about DNS propagation and clearly not understanding *anything* about how DNS works.  They misuse the terms TTL, zone files, and everything about DNS.  I’ve managed DNS for 200,000 domains for the last 7 years and know DNS very well, and this person is BS’ing me.  They try to tell me that since I’ve been moved (and that was a surprise – this was the first time someone mentioned I was being moved), I would have a new IP address and it hadn’t propagated yet.  I tried to tell them that the new IP was already propagated due to the semi-low-TTL and that I could already tell I was on a new IP address.

Then I was informed of the package migration, but again, told to wait 24-48 hours for the migration to take place.  It’s Saturday evening by the time I’m told of this package migration.

Every time I call in, the support person cannot tell me a) what’s going on, b) how long it’s going to take or c) give me to someone who knows what they’re talking about.  I was hung up on twice.  I was told on Saturday evening that my case was escalated at the highest priority and it was in the admin’s hands, and they can’t do anything about it.

Then on Sunday when things are STILL not up, I call in to find out status and again are met with blank stares.  Then this next support rep tells me they’re escalating at the HIGHEST priority they have, and when I say “well, yesterday I was told I was escalated”, the person tells me that “well, yesterday you were escalated at STANDARD priority.  Today I’m going to escalate you to HIGH priority”.  I’m not filled with absolute confidence.

Shortly after being escalated to this new super-duper HIGH priority, I get an email form Anthony Paris saying “I received a case about your domains being down.  I was able to access all of your domains without a problem”.

This was CLEARLY a lie as all of the sedo-parked domains were up (why wouldn’t they be? they’re on servers that actually exist), but all the ones hosted on 1&1 servers were down.  When I responded immediately with example domains that were certainly down, I got no response for another 24 hours.

Monday came and went with no response, no follow-up, no working system and no help.  Calls into support again gave me “we don’t know anything, we can’t do anything, and we can’t get you to anyone who can help you, but we *really* appreciate your call and thank you so much for calling 1&1”.

I happened to try my sites this morning and they were working again, although several sites have miscellaneous php and name resolution errors – it seems that not all the same packages/modules/includes are on the new server(s) but I’ll tackle that separately.

No escalation path

Things might not have been so bad if my case was treated with priority once I let them know what the problem was.  That’s why it’s essential to be able to get knowledgeable customers escalated to knowledgeable tech support staff quickly. 

The 1st level support folks had absolutely *no* idea what was really going on.  They were just repeating the same basic script “we’re working on it; wait 24-48 hours; thank you *so* much for calling 1&1.”

I could have told them that the migration was failing.  Had the right person known, they could have fixed it soon.  In fact, I told them as much.

Just check out the case notes

Webspace is stuck in status 20104 ‘space is moving’. Tried to fix it with ‘movespace.pl -F’ (re-sets all status flags) but doesnt work. Mail to Raf…

What do you want to bet, this “webspace stuck status 20104” problem occurred Friday morning about 20 minutes into the migration, and SAT UNATTENDED for four days?  What do you want to bet had someone actually looked at it after a proper escalation, that I would have had my sites back up in a couple of hours instead of a few days later?

Treating your valuable customers better

As I’ve mentioned, I’ve been a customer for over 5 years.  I’ve always had your top package.  I keep adding more domains and business your way.  I’ve always paid my bill on time and never disputed anything.  I don’t call into support.  I’m a super happy customer who gets you more happy customers.  Many companies would consider me the ideal customer.

Although still not proper, I can see you treating a $4/mo customer this way who just started last month.  But not someone like me.

Why, then, would I be treated this way?

Rectification

I would like some understanding at your company for what I’ve gone through.  I’d also like to know that this is abnormal and shouldn’t happen.  I want to know that my confidence isn’t poorly placed in your company.

I want to be able to keep referring customers your way.

Can you help me do that?

I await your response before I post my grievances online. I want to give you guys a chance to make me feel better.

Wayne Hall
http://whall.org/blog

 


        

1&1‘s Response Letter

 

Dear Wayne Hall, (Customer ID: ———-)

Thank you for contacting us.

I sincerely apologize for the frustrations that you’ve seen thus far.

The kind of downtime that you had suffered is not something that we consider as acceptable. I have read through your email, as well as your case history. I was looking for a reason to explain such downtime. I wanted to know why it happened, why it lasted so long, and why no relevant information was passed along to our support team to give any kind of explanation. This is what I have found, along with the best explanation that I can give.

First, I want to assure you that this sort of ordeal is extremely irregular. This is not something that we often see, or at all for that matter. I want to sincerely apologize for the downtime suffered, coupled with the frustrations of not knowing what was going on. Again, this is something that is not common.

When you had switched packages, there should have been information given to you that you might see some minor downtime. Nothing on par with 4-5 days, but minor downtime. The package, during it’s server switch, did see the downtime expected. However, there was a problem. It never came back up. There was a problem on the server regarding FTP/SSH/HTTP. The biggest problem that I saw here was not that the server was experiencing an issue. As you are aware, these types of things occur within any hosting company. The problem was that our agents didn’t recognize the abnormality of the issue. This should have been passed to our third level administrators for immediate resolve.

Our agents were making notes in the case that your functionality should be back soon, and that it was just a matter of time. This was clearly not the case. This was something that our administrators needed to fix on the server level. With enough time passed as it was, the case was passed to our second level support here in the US. However, the problem was not clearly stated in the case notes. You received a response from one of our second level agents saying that the issue had since been resolved. The problem that was drawn out in the notes was inaccurate.
According to the notes that were passed up, the agent was correct.
There was not an issue with what had been explained to him. As more time passed, the issue was eventually explained to us here. Once we had the issue in our hands, we could then resolve it ASAP.

Our third level administrators received the issue. Within the mix of their normal workload, they had it fixed on the server level as soon as they could get to it.

From what I have read, this issue wasn’t even that large of a deal to fix. Something needed to be reset on the server level, and that was it.
The third level administrators had clearly done their jobs correctly.
The second level of support had done exactly what was expected of them, given the information they had received. The first level of support had misread the issue.

When I said that this was something that doesn’t normally happen, that was the truth. It is very rare that any of our support teams will misread an issue or misdiagnose the problem. However, this was not the case here. What the first level of support should have done was simple.
Understand the problem and pass it to the people who can fix it. They felt as though they had the correct answer to the problem, which was to give it time because it should fix itself. 99 out of 100 times it would have fixed itself. But they should have realized that something unusual was happening.

I felt it was only right to give you a fair explanation of what had happened. This doesn’t happen often, so when it does, we take it pretty seriously. Believe me when I say that we will be further training our support agents to now recognize when this happens, so that it doesn’t happen again to another customer.

I know that my apologies and explanations don’t take back the downtime suffered. We want you to have faith in our company and we certainly want to keep you as a customer. As some kind of compensation, I have added (withheld).  You will see this credit on your next invoice/statement.

Your case certainly was passed up the right avenues and reached the right place in the end. It just took far too long for it to get there.

We apologize for what you’ve been through. A customer who has been with us for three years does mean something to us. I am sorry for what you’ve been through, and it won’t happen again. Thanks.

If you have any further questions please do not hesitate to contact us.


Sincerely,
(name omitted to protect identity)
Technical Support
1&1 Internet

 

Nice job, 1 & 1

And lo, the people did comment thus:

3 Comments

  1. phill says:

    You park over 100 domains??? You know http://www.whall.org and whall.org don’t count as 2, right? 🙂

  2. whall says:

    Hey, whaddya know, http://www.phillsimpsonsucks.com is already taken. I guess someone already knows it and wants to advertise.

    made ya look.

Want to comment?

Hey, we all want to share our voice. And I particularly love comments, especially if you took the time to read my blog entry. I'll take the time to read your comment, I swear! But due to spammers, robots, and the fact that I want my blog to be PG rated, I need to approve the comments. This should be same day, but please don't get mad if it takes me a while to approve the comment.







Comment:


PLEASE help keep this blog family-friendly by refraining from profanity and vulgarity.


CommentLuv badge


Admin
tsk tsk

Ajax CommentLuv Enabled 336ad6ab990e8080f1c0ad1f892428a0