Cloudflare, in many ways, has a positive impact on the load times and performance of a website, and therefore it has demonstrable SEO benefits. It’s often found on articles and in lists of services that can help improve your SEO.
There’s one little thing that I discovered that is a problem with Cloudflare. That problem is called “Broken Link Checkers”.
What are broken link checkers? They are usually a plugin or an app that checks all of the links on your site to see if any of them are broken. For WordPress, the most popular example is appropriately named “Broken Link Checker“. It masquerades as a version of Chrome with a specific user agent, checks each of the websites that you’ve linked to in your blog posts, one by one, and notifies you if any of them are unreachable.
Ok – what is the problem here?
Cloudflare is very efficient at detecting bad visitors and bot traffic, but since these link checkers are essentially bots, Cloudflare blocks those important checks. This tells other webmasters that your link is broken, and there’s a good chance they’ll remove it. This is not good.
Checking links to see if they are alive and removing the “dead” ones is good housekeeping, and we can’t blame the webmaster for this. At the time of writing, this plugin has over 700,000+ active installs and counting.
I’ve personally removed links to articles that showed up as “Forbidden” or “Server Not Found” in the past, but this is something I have to be aware of now that Cloudflare is becoming increasingly popular. At least now, you can make sure that you’re not one of these webmasters who is losing valuable links to Cloudflare.
Here’s how to fix this:
When you pick through the files of the Broken Link Checker plugin, you can find their checker under the following directory:
/modules/checkers/http.php
At the time of writing, around line 178, you’ll see the following user agent is used by this plugin to check for broken links:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36
Great, now we have something to work with.
Log in to Cloudflare. Click Firewall, Firewall Rules, then “Create a Firewall Rule”. You want your new rule to look something like this:
What we just did here is tell Cloudflare to allow all users with user agents that match the one used in the Broken Link Checker plugin. This way, your links will never show the response code “Forbidden” when this plugin is checking to see if your link is still alive.
Screaming Frog is a popular SEO spider tool, and they too have a broken link checker. If you wanted, you could whitelist their user agent as well, which is:
Screaming Frog SEO Spider
There are likely other user agents used in various plugins for other content management systems – if you have any, feel free to let me know in the comments down below and I will expand this list.
Thanks James for this critical find, if broken links are not detected that would be cause of concern. FW
You’re welcome! It’s an interesting one, right?