Sunday 16th Feb

All sites have been moved to the new server and a few minor hiccups ironed out. It all seems good. I am still monitoring for re-infection and there has been none so far. This is very good.

The migration was less stressful than I expected. Maybe it was just less so compared to the last couple of weeks.

I have had some people send me emails they don’t understand…this is good too. Keep it up.

Also let me know anything weird or not working via support.123host.au

Today I was surprised to find I was chilled enough to take up a freebie offer at Dreamworld where we went on all the rides.

Steve

Server update Saturday 15th

All accounts have been moved to the new server.

There have been some minor issues when the transfer tool hasn’t migrated DNS settings properly but they are easy to fix and it all sorts itself out relatively quickly.

So far so good.

If you received a notification about SSL expiring, that was because the new server came with those notifications (which are annoying) enabled. I have switched them off

I remind you that if you you have any issues at all head to support.123host.au and open a ticket.

Steve

Email to all 14/2

Following up from the email the other day (which you can read at https://blog.123host.net.au) I have been unable to repel the attackers.  It seems that they have installed a back door into the system (their own lock and key) and can come and go as they wish.  If there is one good thing in all this, they are not being destructive.  Their method is to insert files into sites (not modify existing files) and extract bank details from people who click links in phishing emails.  I have had a process running that advises me of some system changes and I have been jumping on anything quick smart and deleting it.  But it is a stressful, unsustainable, tail chasing exercise and I am over it.

After a lot of deliberation and following a wave of intrusions this week, I have decided to abandon this server.   I have no choice…the bastards are smarter than me and are relentless. The process of moving all accounts to a new clean server and checking them for malware is under way.  I don’t anticipate much disruption (except to my weekend) but it is a semi- automated process so errors do happen.

What I would like you to do is keep an eye on your website and let me know if anything changes.  Keep an eye on your emails and make sure they keep working. 

I know some of you have had your sites affected.  If you have, please contact me, I would like to make it up to you.

To all of you I deeply and sincerely apologise.  I have been humbled by this event.  It has affected me in many ways and has certainly made me realise that there are some bastards out there who just don’t care…my faith in human nature has been rattled a bit. 

However I intend to keep offering the awesome level of support that has seen 123host grow by word of mouth only.  I thank you for your ongoing support.

Steve

Server update 12/2

The 123host server compromise from a while back stopped.  We put in extra security measures and audited the server.  All felt good.

However it seems that feelings of confidence were misplaced.  The checking has resumed and is overwhelming me.  It is having a range of knock-on effects as sites are marked deceptive and the server’s trust level wavers.

I have no choice but to rebuild the existing server in order to eliminate whatever back door software has been hidden.

Over the next few days all accounts will be moved one at a time to a new machine.  During this time your site may be disrupted.

I can’t even from the words of apology that are needed.  You have trusted me to look after your site and I have failed.  I especially apologise to newer customer who came to 123host based on someone’s recommendation -FWIW 123host has never advertised and has grown by word of mouth.


I can assure you that in consultation with the data centre we are investigating all options.  As you can imagine, this is going to have a huge impact, not only on me and my business.  I am quite behind answering emails and I will get to them all asap.

I don’t want to overdo it with email updates so if you want to see the latest I will keep updating here.


Carbon capture

Late October 2019 and I know the contract on my current server expires in early December.

The current server has been in the UK paid for in $US. The company (BigWetFish.hosting) has been awesome but between the crappy exchange rate and the tyranny of distance it is time to move the server back to Australia.

This is not a trivial task.

I spent a few weeks researching what I wanted and then finding a new hosting service. I negotiated with them and decided they were the mob to go with. I bit the bullet in early November and placed an order for the new server giving me a month to prepare it and move.

Then they told me that the server is not in stock. WTF!?!? FWIW, 2 weeks later it is still listed on their site. I am not going to dignify them with a name.

Moving on…breathing…trying to not get stressed…

I found another mob and I think I am already glad the last one fell through. Though timing wise I may have to take out one more month on the old server, but the new host is flexible enough to make that up to me after 3 months…I like this guy. He is like me…can work with his customers instead of some set of rules decided by a CEO.

So…tomorrow is December and I have moved sites up to the letter H.

woosh!
Stock image that doesn’t look like me or anything at 123host but still gives the impression that there is something cool happening.

The new server is so much faster I sometimes think that there is something wrong…I have gotten so used to waiting for a page to load (I have lousy internet anyways) that when they load so fast I wonder if it actually did load…no kidding.

For the tech-heads it is running E5-2670v2 with 64GB RAM and 4 x 1.92TB SSD RAID 10. It is quick! It is the combination of solid state hard drives, fast processors and being in Australia that have made the difference.

This migration hasn’t been without a few hiccups but they are generally quickly fixed.

Soon my anxiety level will drop back to extreme…

Oh and the cryptic title? I have named servers sequentially going through the periodic table. Old server was boron this one is carbon.

You haven’t been hacked

There is a phishing campaign going on at the moment with emails like this going to loads of people

Hello!

My nickname in darknet is dickhead53.
I hacked this mailbox more than six months ago,
through it I infected your operating system with a virus (trojan) created by me and have been monitoring you for a long time.

If you don’t belive me please check ‘from address’ in your header, you will see that I sent you an email from your mailbox.

Even if you changed the password after that – it does not matter, my virus intercepted all the caching data on your computer
and automatically saved access for me.

I have access to all your accounts, social networks, email, browsing history.
Accordingly, I have the data of all your contacts, files from your computer, photos and videos.

blah blah blah

Ignore it.

They may even include your email address and a real password!  This makes the emails and the claims look authentic, but don’t be fooled.

There are numerous sites that have been compromised and had customer data stolen. It happened to Facebook again recently. Generally the baddies get email addresses and passwords plus other info. Most often, the site they stole if from has reset everyone’s password, assuming they know someone has gotten in, so it won’t work there. Knowing that most people are in password overwhelm and use the same password on multiple sites, they start trying them out at gmail, facebook, hotmail, ect, etc, sometimes it works.

They also try a different approach like the emails we have all been receiving. The fact that they have a password is “proof” that they know what you are up to, but it is a fishing expedition, they are hoping to trick some vulnerable and gullible people.

You can check if your email is on any of the published lists of stolen data at https://haveibeenpwned.com/

Don’t panic if you are. There are some important things you should do

  1. Ignore the emails
  2. If you are using that email/password combination anywhere change it immediately
  3. Continue to ignore the emails
  4. Tell vulnerable people who you know (e.g. my mum)
  5. Don’t stop ignoring the emails

If an email that looks official doesn’t include my name, that would be the first warning sign for me. I am always happy to check out emails that you receive that you aren’t sure about.  Forward them to support@

Hope that relieves some anxiety about this stuff.

WordPress management

While doing some research for a recent email to all customers about a severe WordPress bug, I came across a solution that allows me to manage the administration side of your WordPress site without having to log in to it.

The tasks I look after on your behalf include

    • Updating WordPress
    • Update plugins
    • Update themes  (on request. I am reluctant to update themes in case it over-writes your customisation)
    • Ensure https compliance
    • Monthly report to you
    • other things…

123host is pretty much all-inclusive.  I have always disliked the way some hosting businesses start with a cheap base price and then charge extra for every little thing.

However, subscribing to this service is quite expensive for me, and I pay based on the number of sites connected, so if you would like to have me manage the trickier part of your WordPress dashboard, the cost is $66 inc GST per year* per site.  This isn’t much more than about $1 per week for peace of mind!

* yeah, the fine print - these prices were correct when I wrote this, it might have changed.

Hiccup with _some_ outgoing mail [UPDATE II]

UPDATE:  The issue is now resolved.  When the email reputation went bad the mail server was moved to a different IP address that had a good reputation.  The spam hadn’t been stopped so after a day or so that also had it’s reputation battered.  BUT…in the meantime the original IP address had time to repent as it wasn’t sending any email at all, much less spam.  Suspecting this would happen the email server was put back to the original IP address and it did come good.  Email is back to normal.  Apologies for any hassles.

There is a system of rating email server reputation based on the quality of the mail they send.  Too much spam and you get a bad reputation.  This is all evaluated automatically by some servers somewhere…there are lots of them that do it.

What is happening is that due to a WordPress site infected with spam sending script, servers have decided that the 123host mail server deserves to sit in the naughty corner.

Some ISPs use the naughty list to say “we won’t accept mail from you until your behaviour improves”. That’s where we are at right now. Note that it is only some ISPs.  If you are getting bounced emails, maybe the people you are sending to have their email hosted there.

I am pretty confident the spam source has been stopped and things will return to normal soon.  I am working on this with the data centre right now.

I will update this when I know more.

Scheduled database downtime April 23rd

I am scheduling some system maintenance that will only affect sites that use databases, this includes WordPress sites.  If your site doesn’t use a database this notification doesn’t concern you.

Sunday April 23rd at 0300hrs Australian Eastern Standard Time the mySQL database service will be shut down while all databases are moved to a different part of the server to allow for the continued growth of 123host.  The data centre advises that the scheduled window for this work is 45 minutes.

Unfortunately because the data held in databases changes so much and so often, it isn’t possible to move the data while keeping the service running and not cause some data loss.

People will still be able to access your site, however if it is using a database it will display an error during the period that the database service is stopped.

That new server

Moving servers seems to happen too often and hopefully the latest will be the last for a while.

When the server was set up at the data centre, a couple of odd things happened with the way some software was installed.  The simplest (though more expensive) solution was to accept an upgrade offer.

I guess it is a testimony to good planning and good tech people that the whole thing was remarkably painless and for about 98% completely transparent.  What a relief…it is always stressful for me.  I end up dreading turning on my computer in the morning half expecting to find dozens of support tickets…but it didn’t happen.

The only issue was 1 customer who didn’t follow up on a warning email I sent about a change of IP address and 5 people who use Outlook who needed to change a setting to be able to send emails.  That’s it!  Bliss.  Relief.

Here’s what is running now:

Dell hardware with 8 x Intel Xeon E3-1270 v5 @ 3.60GHz processors
64GB RAM
4 x 2TB Hard Drives
Hardware RAID 10 – this makes a real time copy of the hard drives in case one fails
10TB Bandwidth
CentOS 7 operating system
PHP7 with PHP 5.* also available
A good strong firewall

The server is running at about 25% of its capacity.  This is a good thing.  Having made the mistake of letting a server get too crowded in the early days, I won’t do that again and look forward to 123host explanding to another server.

While talking about servers, a timely reminder about backups.  123host user accounts are backed up every night to BackBlaze B2 which is ridiculousely cheap at $US0.005c per Gb per month.  If you have a lot of files to store, it is worth checking out, but it doesn’t have a particularly friendly user interface.

Each nightly backup is kept for 14 days.  There is also a weekly copy kept for 4 weeks and a monthly copy kept for 2 months.

This means I can restore files from any day up to 14 days ago.  Or any of the last 4 Saturdays.  Or the 1st of each of the last 2 months.  If that can’t find the file you are looking for it must be long gone.

If you are a 123host customer, you know that one of the aims of the business is to provide the sort of customer service we wish we received elsewhere.  I have had a mix of excellent and appalling customer service recently.  We all know which we prefer.  Here’s some stats that I love…and they are pretty consistant going back years.

February 2017 support ticket response time average

March 2017 support ticket response times (so far)

And for those who understand such things, here is a snapshot of the server right now – Load Averages: 1.08 1.33 1.35 – I haven’t seen it go above 3.5 yet, I’m pretty happy with that.

Everything about this server is better.  It is faster, it is more powerful and you (my customers) are having fewer issues, and there were never many!