Here are some quick steps for how to whitelist or blacklist the Ahrefs bot or crawler if you are using Cleantalk.org on your site for spam prevention.
The issue that I had was that I wanted ahrefs to crawl my site (outdoorgearreview.com) using their Site Audit function and it returned with an error saying:
The HTTP server returned error 403: “Forbidden”. This request was likely filtered by server configuration.
There are a couple of ways around this:
- Deactivate the cleantalk.org plugin while the crawl is running
- Add the following code to your robots.txt file:
User-agent: AhrefsSiteAudit
Allow: /
- Or filter the ahrefs bot in cleantalk… see steps below
Step 1:
Login To Cleantalk.org and click “Personal Lists”.
Step 2:
Scroll down and click “Add New Filter”
Step 3:
- Select “Ip Network” – Note: Do not select IP Address
- Enter in IP network range for Ahref Crawler:
54.36.148.0/24,
54.36.149.0/24,
54.36.150.0/24,
195.154.122.0/24,
195.154.123.0/24,
195.154.126.0/24,
195.154.127.0/24
- Make a note so that you know what you are either blacklisting or whitelisting
- Select the site you want to activate this for. Else select all sites.
- Choose whether you want to blacklist or whitelist
That’s it. I hope this helps. Comment below if you have questions.
