The Day When 80% Of My Clients Were Hacked

Surprised bearded man holding a laptop
5 Apr 2019

Website hacks are common. But what happens when 100% of all websites on one hosting server are hacked? Here’s what happened for my clients.

Building and hosting websites is an industry based on trust. You need to trust that the domain name registrar is decent and has fast systems. You have to trust that your web copywriter and web designer know what they are doing. You have to trust that your web host is running up-to-date and secure servers. You have to trust that you will be notified if something goes wrong and that all parties step up to fix the problem.

In the past few weeks, I have seen what happens when things go severely pear-shaped, and trust is irrevocably damaged.

Many of my clients took my recommendation of being hosted with a local boutique web hosting company when they had their websites built. The hosting company was a local Brisbane company, and for the better part of a decade their service was exemplary, and the people in the business had become more than simply suppliers, but they had become friends.

Then part of the way through last year things started going downhill. Promises were made but never kept. Help tickets would go unanswered or unacknowledged for weeks at a time, and phone calls went unanswered or not returned.

Every task required multiple follow-ups, checks and corrections – nothing was 100% the first time around.

Problems cascaded to the point where I couldn’t complete simple maintenance tasks for my clients without extensive workarounds and extra expense.

Problems compounded

I won’t bore you with all of the tiny details (suffice to say there is a very long and detailed paper trail starting in September last year), but things had gotten so bad that I had a face-to-face meeting in February with the CEO to discuss my concerns.

It is just what you do when someone is a friend and going through a rough patch. You give them the benefit of the doubt over many months and see if you can fix things.

They apologised and promised to do better. But things didn’t get better … they got much, much worse.

There was a problem I had identified late last year with one server they had. It was running an out-of-date version of SQL that was contributing to the issues I was having. The CEO acknowledged this was a security problem and that they wanted to move clients to a more secure server.

The move was what I had been asking for since November when I first told them about the problem. I had my own sites on that server, along with a stack of my client sites.

Nothing happened for a couple of weeks after the meeting, and then without any alert, the company decided to move my heartcomms site to a more secure server without notice. It was the right action but handled very poorly.

In the process, they broke the site because it was not moved correctly, and it was down for more than 24 hours.

I had enough of the shenanigans and made an emergency move in the middle of the night of my personal sites to another host. Of course, this was two days before a major website launch I had been working on for months, so I was running on no sleep and maximum stress levels.

It cost me the better part of $1000 with lost hosting, lost time, additional domain name renewal costs on the move, tech support and new hosting, but it was worth every cent!

The days the hackers came

The day the hackers came

I had flagged with all my clients on the problem server that when their sites were up for hosting renewal that we should revisit their hosting plans given all the dramas we had been having with the company.

Unfortunately, within a week of my emergency midnight dash to a new host, the old server started to be pounded by hack attempts.

About 1 pm on the 18th of March, I started getting alerts every few minutes that all client sites on that server were constantly up and down, (I have an uptime monitor on all my client sites through Uptime Robot). This went on all afternoon and night.

The next morning, the instability continued, and I notified the hosting company of the problem as I had heard nothing from them.

By then it was too late. 100% of my client websites on that hosting had been hacked.

Many times, when a website has been hacked there is nothing visible from the front end.  You look at the site, and it looks totally fine.

But given the continuing instability of all those sites, I was on high alert and spotted an unexpected additional file in their website folders, and could not log into the sites from the front end.

When I finally accessed the first hacked site, I had found that not only had the hackers gained entrance and changed all the usernames to admin and reset all the passwords; they had deleted all the back-ups through the back-up plugin I used.

The loss of the backups were because I had not locked down access to that plugin through a password. It was a simple gut-wrenching error, but it was one I had made, so I chose to accept responsibility for fixing the issue.

Once I had checked all the client sites where I had website maintenance plans in place to see who had been hacked, I then started to check previous client sites that looked after their own updates. Where I still had access to their sites, I attempted to log in and found all of the sites on that same server had also been hacked.

100% of all my clients on that problem server had been hacked (and there were a stack of them!)

I contacted the host who said that restoring from their backup was a paid service which would cost $570+GST per site restore.

However, for some clients they offered a free site restore service. There was no logic as to who was offered billing and who received a free restoration – it all depended on who asked and which support tech they were allocated.

Backup Lessons:

  • If you use a plugin like UpdraftPlus or Back-up Buddy, make sure that you have a password added not just to the downloads/uploads, but to actually access any of the settings in the plugin itself.  Updraft Plus has a Lock Add-on hidden away in the advanced tools tab.
  • Store more backups than you think you need. I keep one month of daily database backups and 4 x weekly backups of all plugins/themes stored off-site in external cloud storage. (This works well as long as the hackers can’t get to your files.)
  • Check your hosting agreement for backups. 99% of all hosts offer free restores within seven days. Most good hosts offer 30 day free one-click restores. If your host does not explicitly offer this minimum level of service, then change hosts.
Website-security

But didn’t you have security?

Wordfence: All my clients know that website security is super important to me and something we factor into every site build. All of my clients were running paid versions of Wordfence, the gold standard of security plugins for WordPress.

None of the sites had any Wordfence alerts that they were being attacked and all high-sensitivity scans on the sites that we knew had been hacked, failed to reveal any problems. Wordfence didn’t see or stop the hackers.

Updates: I had recently updated all current and past client sites themes and plugins as there had been a security alert on the theme I use (Divi), so all sites were running current software.

Tough Usernames and Passwords: No site had admin or the name of the website as a username, and hard passwords were enforced for each site.

In other words, we had all the usual security essentials in place.

Common threads

When trying to work out if there were any commonalities between the hacked sites, not all sites were running the same themes or plugins or even levels of PHP. The only common features were they all were running the latest version of WordPress, and they shared the same server, the one I had complained about a few months before.

The bottom line is that all the extra security we were running was like having strong locks and burglar alarms. However, the hack came in through a hidden back door, and no amount of front end protection stopped it.

It could have been the server, or there could be an as yet unknown vulnerability with the latest version of WordPress. Whatever the cause, the hackers got in … and the host denied that they had any responsibility or any part in causing or fixing the problem.

My client’s responses

I took the hosts response to my current and past clients and talked them through their alternatives.

Some clients chose to have their sites totally removed, and they went back to a single “coming soon” page.

In some cases where the client had planned a site rebuild for this year, we are simply rebuilding their sites on new hosting and have a “coming soon” page up on the old hosting until we go live on the new hosts.

Sites that were restored from the host backups were hacked again within hours. After this happened to a few clients, my clients decided they could no longer trust that host, and decided to get their sites cleaned and move to different hosting.

Most of my other clients who are on different servers with the same company are also choosing to move ASAP. I am doing those moves in a triage process: hacked sites first, then the next least secure server followed by the stated better quality server.

Website-security repair

What happened next

I sourced a hack repair specialist and started sending him my client’s sites to clean and then migrate to new hosting. (You know it is bad when they offer a bulk rate!)

Unfortunately, the old hosting ran a Plesk hosting web control panel, which made migrating to cPanel web control panel hosting (the industry default) ten times more difficult.

My poor clients have had to deal with getting access to their domain name registration details (not one had been given access to their domain name registration before their requests as part of the move), taking out new hosting, finding email passwords and having to update their computers/phones or wherever they accessed their emails to reflect the new hosting details.

I have spent days with each client, talking them through the process and supporting them while they made the tech changes that were needed.

I have been an emotional punching bag and support as they dealt with the fear that a hacked site brings up.

All clients are out of pocket as part of the hacks.

Most had months to run on their hosting contracts with the old company but have simply walked away and written off that cost as a bad debt.

With some clients, the instability of their sites being constantly up and down for an extended period in March has flowed into lost revenue. One client’s visits shown in Google Analytics are down by a quarter on the same period last year and has resulted in fewer clients booking their services creating a drop in profits for the month.

Lessons from emails

The biggest headaches came from migrating clients who had their emails through their web hosting. Many people have their emails set up that way as it is included in their hosting package, but it adds extra complexity if ever they have to move hosting.

Many clients didn’t write down their email passwords, as the hosting had remote-accessed into their computers and set up their emails systems for them. This has meant changing passwords on migration and then reconfiguring all the computers/laptops/phones.

Where they were pack-rats and never deleted emails (ever), then extra steps had to be taken to find creative ways to bring their massive mail files over.

Look at options such as G Suite or Office 365 as safer, more secure ways to host your emails and to minimise disruption on changing hosts.

Get your data together

Most of my clients struggled to gather all of the information needed to move hosts. The time to gather all the information about your website is before there is a problem.

Today, if you do nothing else with this story, put together in a single document:

  • The URL of your domain name registrar and how you can log into your account. You need this so when you change hosting that the records can be changed to point to the new host.
  • How to log into the control panel of your hosting account.
  • Your email passwords.
  • How to log into your website.
Website security response

My response to the hacks

As I had recommended the host in the first place, and as I hadn’t locked down my backups as tight as I should, I did not bill my clients for any of my time I spent on investigating, organising repairs, calming nerves, helping my clients through next steps and helping to reconfigure their mail (approx. three day’s work each site).

The only cost I passed on was the cost of my external hack specialist who did the site cleans and migrations to their new hosting for them.

To help reduce my own site risks and build my skills, I hired a high-level website security expert (MaAnna from BlogAid) to audit my sites and teach me additional ways to secure websites.

For my past and current clients where I still have access to their websites (both hacked and non-hacked), I now have:

  • Locked down all backups and increased backup retention.
  • Hard-coded in additional security into htaccess and wp-config files.
  • Audited every plugin/theme to ensure they were all current. (WordPress has a nasty habit of adding themes to your site each year that you need to manually delete.)
  • Removed all non-used themes and plugins to reduce potential points of entry.
  • Hardened passwords and usernames across the board.
  • Changed passwords on databases and changed “salt keys”.

Other things you can do to increase security

These actions are way far and beyond what the majority of web designers do when they build your site, but even these steps are still not enough.

All of these actions work well at stopping hackers coming in through the front door. However, they won’t stop a hacker coming in through a compromised server or vulnerable plugin or theme.

We are seeing thousands of more attacks each month on even the smallest website, so website  security now needs to be stepped up across the board for every small business owner.

Choosing quality hosting is vital

Gone are the days when boutique hosting was enough – they are just not keeping pace with the risks. I now recommend VentraIP, WP Engine or Kinsta as website hosts.

All three are large organisations with a superb reputation across the internet community for security, responsiveness to issues and value for money.  VentraIP and WP Engine have Australian-based servers. All have comprehensive back-up and restore options and 24/7 support.

On a huge plus, all three of my recommended hosts are significantly cheaper and have better support than the previous host. With the past host, I had attempted (and failed) to negotiate a 14 day response-time for help tickets.

Add Cloudflare Pro

I have recently added Cloudflare Pro to my personal sites. It costs $20US per site per month and helps mitigate DDOS attacks and has a solid firewall and other security baked in that stops bad behaviour before it even gets to your site (think of it like a dirty great big wall surrounding your property a thousand metres away from your home).

It has a few other features that are endearing it to me right now: hotlink protection to stop people stealing your bandwidth on your site and scraping your content; email obfuscation to stop your email address being scraped by bots from your website and then sent to email spammers, as well as extra website performance features.

Cloudflare Pro does take a bit of configuration to get right, and I am still tweaking settings on my e-commerce site, but there is a definite decrease in bad behaviour hitting my site.

Website security plan

Plan for the unthinkable

Given the scale of attacks, the likelihood that a business’s website will be hacked is dramatically increasing. It is now a question of when – not if.

By having clean backups within your control, decent quality hosting, constantly up to date themes and plugins, and extra layers of security baked in at the design phase, you reduce the risks.

If your website is mission-critical or if you are covered by the Mandatory Reporting of Data Breaches legislation, then you also need to consider adding in Cyber Insurance to help cover the costs if your site and systems are attacked.

Every business needs to be across the rules for mandatory reporting of data breaches and know if they apply to your business, as even a tiny hack can trigger your legal obligations.

However, these all cost money. These costs are now simply the cost of doing business and need to be factored into all business budgets. The thing to remember is that these extra layers and costs are much cheaper than paying for a hack repair or site rebuild.

If you are getting a new website built or refreshed, ask what security your web designer bakes in from the start. Unless they have a comprehensive plan backed by evidence of what they have done if a problem occurs, then keep looking.

The security we now build in to all our sites

To give all my web design clients, current and future, the best possible security that we can, these are all the security features we build into every site:

  • No “one-click WordPress installs” – ever! We manually set up each database and username with complex names and passwords of a minimum of 18 characters.
  • Industry best practice Salt keys in your wp-config files.
  • Hard coding to turn off XML-RPC to close security holes and prevent DDoS/ Brute Force attacks.
  • One month of back-ups stored off-site. All backups protected by encryption and a security passcode.
  • Complex usernames and passwords are set to log into each site.
  • We move all clients to dual factor authentication to log into their sites once they are past their initial learning curves with site management.
  • Removal of all themes and plugins not in active use. “Hello Dolly” will NEVER be seen in any of our sites.
  • .htaccess and the wp-config files hard-coded to protect them from hackers, and to stop bots from browsing your directories.
  • Robot.txt file set up to stop Google bots from indexing files that are potential security risks.
  • We check your hosting to clear up old abandoned sites that can create a back-door for hackers.
  • We clean your database of any leftover tables from plugins you may have tried or abandoned in the past.
  • We add a paid version of Wordfence to automatically lock out incorrect usernames, to reduce bad bot behaviour and stop access to the login form from countries outside your own country.
  • We add Google reCAPTCHA to all contact forms to reduce contact form spam.
  • We add in a blog comment form spam shield to stop spammy comments.
  • We add in a plugin to hide any email addresses on your site from bot harvesting.
  • We turn off all pingbacks and trackbacks and automatically close comments after a certain time to reduce spam.
  • We ensure all sites have an SSL installed (the padlock at the top) and force https redirects from the .htaccess files to help ensure each page is securely served to your visitors.
  • We run the latest stable version of PHP on all sites.
  • We recommend quality hosting and Cloudflare Pro.

A website hack is emotionally confronting, financially challenging and takes time to recover from. Hopefully, all my clients who were affected by the hack have felt that I supported them all of the way through, and that the clients who weren’t hacked but are pre-emptively moving to different hosting end up with better, more stable and secure results for their businesses.

About the Author

Ingrid Moyle

Ingrid Moyle is a self-confessed multipotentialite. While she is now retired from the corporate rat race, she still shamelessly dallies in her latest topics of fascination. When not hardwired to her computer, she quests for the perfect decaf coffee while chasing virtual reality creatures across the backstreets of Brisbane.

Related Posts