When I installed Google Analytics I noticed a massive jump in traffic after a few months of launching my new website. At first I got excited thinking that hundreds of people were visiting every hour, unfortunately after some research it turned out to be spam referrals. In this article I will be explaining how to clean up your Google Analytics data, so you can take full control of the traffic coming through your website.
What Are Spam Bots And Why Spam Bots Are Used?
The reason we get spam referrals are because of spam bots. They are designed to hit your website a number of times through a link to make it look like you are getting lots of traffic. When you check out where you’re getting this traffic from you’ll see the link from the website the spam bots are using. Out of curiosity everyone wants to know who’s linking to their website giving you all this new traffic, so you click on the link and it gets you on their website.
Types Of Referral Spam
The spam bots can use two methods to create referral spam, through ghost referral and crawler referrer spam. A ghost referral just hits your Google Analytics directly and cannot be blocked by .htaccess this is because it never accesses your website. Crawler referrer spam on the other hand does access your website and ignores your robots.txt file, which tells crawlers the files not to visit.
There are two main reasons to remove these bots.
- To reduce server load on your website.
- To get the correct data from Google Analytics.
Main Spam Referral Domains
There are many spam domains out there these are some popular ones to look out for.
- darodar.com
- ilovevitaly.co
- semalt.com
- buttons-for-website.com
Removing The Well Behaved Bots And Spiders
The first step to take is to exclude the good bots and spiders from your data. You don’t want to block these good bots and spiders from your website because it will make you less visible on the web, we just want to remove them from your data. Google Analytics have made this process very convenient by adding in a check box that removes them for you.
- Go to the admin tab
- Select the view from the drop down you want to filter
- Click on view settings button
- Scroll down to bot filtering and check the box
- Save
The Best Way To Remove The Ghost Referrals
I have tested out loads of different ways to block ghost referrals and I’ve found setting a valid hostname works the best. Since I’ve added this one rule I haven’t had any ghost referrals. All ghost referrals use an invalid host name, so if you set a valid host name filter they are all excluded.
Setting a valid host name has lots of advantages.
- You can block the spam before it hits google analytics
- You only need one filter
- It blocks all of the ghost referrals
Setting Up The Valid Hostname Filter
To set up a valid hostname filter you need to find out all of your hostnames that you put your tracking number on. In my case I only have my tracking number on jonnyjordan.com, but you would need to add www.jonnyjordan.com or blog.jonnyjordan.com for example if I was using them too. Also if your website gets translated for visitors from lots of different countries too you might need to add translateservice.com to track the translated version.
The bots try and be sneaky and add in hostnames like google.com or amazon.co.uk etc you want to block these. The only hostname there should be are your hostnames/domain names. To find these out follow the steps below .
- Go to the reporting tab
- On the side bar click on Audience
- Expand technology
- Select Network
- Scroll down to the top of the results and sort by hostname
- Scroll down the host names and pick out all of your hostnames/domain names you use. Next you want to add them together separated by a line for example:
jonnyjordan.com | www.jonnyjordan.com | blog.jonnyjordan.com
This line is called REGEX (Regular Expression). - Go to the admin tab
- Select the view from the drop down you want to filter
- Click on filters
- Select new filter
- Enter a filter name like valid host names
- Select the custom filter button
- Check included *important
- Add the filter field as hostname
- For filter pattern paste your REGEX from earlier with your hostnames
- Verify the filter
- You should see the fake ghost referrals on the left and the right should be blank
- Save
Your Google Analytics should now be Ghost referral free, just the crawlers left to remove. This filter will work removing all of your ghost referrals in Google Analytics from now on, but it will not remove the history of them. To remove the old referral data you can read up on removing spam referral history within Google Analytics.
How To Remove The Crawler Referral Spam
After you have implemented the valid hostname filter this leaves the legitimate referrals and the crawler referral spam. Since the crawlers actually visited your website they have a valid hostname, but fortunately there aren’t as many of them. You can block these crawlers using two methods, by creating campaign source filter or by using using htaccess. I would recommend blocking these crawlers through htaccess because it stops them using server resources.
Blocking The Crawlers With Htaccess
The quickest way I found to block the crawlers was to bring up the all of your referrals and write down list of the spammy referrals. When you have a list of them go on to incredibill.me/htaccess-block-referer copy and paste your list and add it to your htaccess file. Below is an example one I created for my website.
# BLOCK SITE REFERRERS
RewriteEngine on
RewriteCond %{HTTP_REFERER} darodar\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ilovevitaly\.co [NC,OR]
RewriteCond %{HTTP_REFERER} semalt\.com [NC,OR]
RewriteCond %{HTTP_REFERER} buttons\-for\-website\.com [NC]
RewriteRule .* – [F]
Congratulations if you have completed all of the above your Google Analytics data will be free from referral spam and you can now make use of it. From time to time you will get the odd new crawler, but just add it to your htaccess when you see it. I’ve been free from referral spam now for three months now without even doing anything else!
