WTF happened?

One of the satisfying things about 123host has been the organic growth.  There is no advertising, no purchasing customers.  Everyone has been referred here by word of mouth or through a blog course or something like that.  123host has been doubling in size every year.

It is great, but it has drawbacks.  A server that was perfect a while back may not be powerful enough now which means upgrading to keep things running smoothly.  If it is just upgrading the same server with more RAM or hard disks or something it is no big deal.  But every once in a while there needs to be a leap in technology.

123host started of on shared hosting.  All the websites were on the same server with maybe thousands of other websites, this is the typical $4 a month hosting situation.   If another customer causes a problem it affects everyone.  Everything is managed for you, your resources and control are pretty limited and they don’t care much about you as you don’t make them much money.  Think of it like living in a share house.

The next step is a VPS (Virtual Private Server) where 123host has its own space on a server on which there are a bunch of other VPSs.  Now we have moved from a share house to an apartment building.  At this point you need to know a bit about managing a server.  Now other customers don’t affect you at all.  It is more expensive but with that comes more resources and more control.  You start to think you know what you are doing (hah!).

Late last year 123host started suffering growing pains on the VPS. The contract wasn’t up until March so it seemed like a good idea to find and start planning a move to a Dedicated Server.  This is the next step where you have your own machine…ahhh, we finally have our own house, to ourselves.

I have always been a supporter of Australian businesses and in particular local businesses.  With the lousy $US exchange rate it became economical to move my server to Australia and I did.  A lot of work went into migrating from the VPS to the new server, but it turned out to be something of a headache.  The machine, despite looking like it would do the job,kept buckling under the strain.

One measure of server performance is the load on the server which is averaged over 1, 5 and 15 minutes.  A reasonable load on a quad core machine working well would be 4.0 and the server was running consistantly at around 8.0 with spikes to 30+  No wonder it was slow. By comparison, the new server is running at 0.73 with exactly the same websites on it.

load

The server tech people were helpful, they spent a lot of time looking at it but couldn’t get to the bottom of the problem.  Fortunately I was on a month to month account rather than going for a 12 month discount (did I know what was coming?).

It is one thing to offer great customer service, it is something completely different to offering a great product.  My product wasn’t being great, so to keep the great service going and get a great product happening again, last week I decided I had to act, did some research and found a new server.  Surprise to me, I was able to haggle a pretty good discount off the published price, that made it viable.

That was all background, now for the grubby details: I started the migration to the new server on Thursday, slowly at first and it pretty much went perfectly.  By Saturday mid morning , despite the old server still running the show, I had all the sites moved and being displayed from the new server through a form of sorcery.  Things were looking great!

I decided it was time to throw the switch so that the new server was now boss, and that is where things started to unravel.  Over the next few hours people were seeing errors ranging from ‘account suspended’ to ‘this site has moved’ to nothing at all.  I was getting phone calls, emails, facebook posts and messages…no pressure 😛

Diversion to anecdote: Years ago, in the very early days of the internet I wanted to learn about this thing.  I took a part time job at a local ISP which got me lots of experience and free internet…dial up – remember that?  I clearly remember one day when a server died.  The phones were going off.  We were trying to placate customers.  It was bedlam.  But the thing I noticed was the system administrators who were responsible for fixing it were cool as.  They were sitting calmly discussing the problem, getting a coffee, very relaxed.  And it dawned on me that you get more done like that than freaking out and having a meltdown.  I have never forgotten that experience.

So yesterday I freaked out and had a meltdown.

As most of you know, this web stuff can be pretty complicated.  Now multiply that by eleventy and server management is harder than that.  Effectively, yesterday the server fell off the internet.  There was never any problem with your site or content or anything like that.  It was a network issue, not a hardware problem at all.

I was getting confused, going in circles, not able to spot the problem and while the freakout and meltdown had subsided, I was not thinking clearly.  Then I found what I think was the problem, but despite making a change, it still wasn’t working yet.

So I called the cavalry, I engaged an emergency server helper…in India, though I didn’t know it at the time.  I do have to say, this guy was great.  At one point he had logged in to the server via my computer and I could see him working. I was in awe of how fast he moved around and the things he was doing.

And then I could see activity on the server as visitors started hitting websites.  It was starting to happen!!!  In fact, I think it was my fix that did it, but server dude did some other work that needed to be done.

And now I am hardly going to touch anything.  I seem to have a good product.  This post – yeah, I better start learning to use WordPress – is part of my commitment to great service which includes being honest and transparent.

I am now quite sure how to do it yet, but as a makeup to you, I am going to push all hosting renewal dates back by 4 weeks effectively giving you some free time.

And one day I will try to improve the look of this blog, which I plan to add to when I have something that might be of interest to you or someone on the internet.