Discover Shield’s New Secret Weapon Against WordPress Bots: Anti-Bot Detection Engine

No matter how you look at WordPress security, or from which direction you approach it, it nearly always comes down to bots.

Automation is the key to hacking 1000s of WordPress websites at scale. And the simple way to automate this, is with “bots”.

But, if we can be better at detecting bots, then we can completely revolutionise our WordPress security.

The challenge we face, then, is how can we accurately detect bots hitting our WordPress sites?

ShieldPRO 11.0, due out soon, delivers our unique response to this challenge with your all-new AntiBot Detection Engine.

What Are Bots And Why Are They Such A Big Deal?

First, let’s get really clear on what we mean by “bots”.

The best way to start thinking about bots is that there are 2 types of visitors to your WordPress site:

1. Humans, and
2. Everything else (bots)

It’s a simplistic way to split up your traffic, but no matter which way you categorise your WordPress visitors, they fall into 1 of these 2 groups.

For this discussion, let’s put aside human traffic and examine everything else. This could include:

• Search engine crawlers, such as Google, Bing, Baidu etc.
• Uptime Monitoring services
• Integrated services such as iControlWP and MainWP.
• Comment SPAM bots
• Login Bots
• Probing bots and other malicious “bots”

All of these can be referred to as “bots” because they’re automatic – they’re robotic. So calling something a “bot” doesn’t mean they’re bad, they’re just non-human and automated.

The challenge we face, then, is not only in detecting which traffic is human and which is automatic, but which automatic traffic is “good bot”, and “bad bot” traffic.

If we can isolate a “bad bot” from the rest of the pack, then we can treat it differently – i.e. thwart its activity by blocking it from accessing your WordPress site.

Once it’s been blocked, the security threat to your WordPress site from that particular bad bot is neutralised.

This isn’t a trivial task, but with ShieldPRO 11.0, we’ve taken a massive step towards more accurate bad bot detection.

The question then is, can we detect these bad bots?

Yes.

How Does ShieldPRO Detect Malicious WordPress Bots

Earlier we split our traffic up into 2 main categories. Now we’re going to split it further, as follows:

• Humans
• Good Bots
• Bad Bots

Our goal is to isolate “bad bots” from our traffic wherever possible – in this discussion, we’re not interested in dealing with the others.

The steps below outline how Shield goes about identifying bad bots.

Step 1: Eliminate The “Good Bots”

We never want to block the good guys.

So the first step is to identify the “good bots”, so we can eliminate them from the picture.

ShieldPRO 11.0 has hugely improved and expanded its knowledge and processing of the most common “good bots”.

These include bots like:

• Search engine crawlers: Google, Microsoft/Bing, Yahoo!, DuckDuckGo
• Uptime Monitoring: Pingdom, Statuscake, Nodeping
• Performance: GTMetrix
• WP Management: iControlWP
• Social Media: Facebook, Pinterest
• Others: Stripe, PayPal IPN, and more…

And the best part? This list is completely flexible. We can add, remove and modify this list of services at any time without updating the Shield plugin itself. Shield will adapt and update itself to use latest list – you don’t have to do anything!

So Shield will detect a “good bot”, and verify it where necessary. You can learn more about this in the “Fake Web Crawler” section here.

Now that we’ve eliminated the good bots from the picture, the final 2 players remain: bad bots and humans.

Step 2: Bot Signals And They’re Importance In Malicious Bot Detection

This approach isn’t new to ShieldPRO, but we’ve levelled it up for v11.0.

We use the concept of “bot signals” as a way of building a clearer picture about a visitor.

You may think that if a visitor tries to login with an invalid username, then it’s a bot and should be blocked immediately!

That’s just not practical. It’s easy for any normal, WordPress user to mistype their username, or just put in the wrong password. So we can’t block them just yet.

If a visitor performs a number of missteps, or offenses, against our WordPress site, or repeats the same offense a number of times, then we can be increasingly sure it’s a bad bot.

ShieldPRO tracks all of these bot-signals, and has done so for a long time. They include:

• Fake web crawler idenfication (as discussed above)
• Triggering a 404 error (used by bots to probe a site for plugins/items)
• Failed WordPress login
• Invalid username login
• Completely invalid User Agents
• Link Cheese
• XML-RPC access

Each offenses by a visitor can’t be used in isolation to detect a bad bot, but they can be used together, in combination.

The more signals triggered, the more confidently we can say that a visitor is malicious, and then block their IP from accessing your WordPress site.

But these aren’t the only bot signals we can use.

How ShieldPRO 11.0 Expands Upon The Bot Signals System

The goal of Shield’s new AntiBot Detection Engine is to reliably categorise a visitor as a “bad bot”, or not.

Shield now creates a score for every single WordPress visitor. This score let’s Shield make a decision at any time, as to whether to allow a visitor’s request, or deny it.

To build this score, we draw upon the huge coverage that Shield has on your WordPress site, from failed logins, to comment SPAM, to firewall triggers, and more.

In our first release of the new AntiBot Detection Engine, there are over 25 different bot signals that Shield uses to identify a visitor threat.

Alongside the new…

Shield’s Not-Bot WordPress Visitor Test

A brand new addition to Shield is the visitor “Not Bot” test.

Shield adds a small snippet of Javascript (jQuery not required) to the footer of every page on a WordPress site. This Javascript fires a simple AJAX POST request to the WordPress site, silently in the background. Shield records this request to use as a new bot signal.

So what does firing this AJAX request mean?

Most simple bots don’t process Javascript well. This is either because they technically can’t do it (the vast majority of scripts/bots), or they’re directed not to (GoogleBot, for example, doesn’t fire AJAX POST requests).

If Shield records this AJAX request for a visitor, it means that in all likelihood they’re a real human being, not a bot. Shield combines this not-bot signal with other signals (or the lack of them) to determine whether a visitor is legitimate, or malicious.

Every IP Address is assigned a score based on the signals they trigger.

You can then review the full breakdown of the scores and see what signals have contributed.

This not-bot signal becomes incredibly useful when combating brute force login requests, comment SPAM, and contact form SPAM.

No More CAPTCHAs or “I’m a human” Checkboxes

With ShieldPRO‘s AntiBot Detection Engine we can now completely remove the need for annoying WordPress login form CAPTCHAs, and of course Shield’s own GASP “I’m a human” checkbox.

Since we’re passively testing a visitor’s status (no direct interaction required from the visitor), when they come to leave a comment, or submit a contact form, ShieldPRO can build their Bot Score and allow or deny the request.

Because we don’t need CAPTCHAs and checkboxes, integrating with custom forms, or 3rd party Contact Forms/Login Forms becomes much, much easier.

We’ve been asked countless times to integrate with Contact Form 7 and Gravity Forms, but the technical requirements to do so until now was always a little daunting.

With our AntiBot System, integration is far easier. And with the release of ShieldPRO 11.0, we’ve already integrated with 16 Contact Form providers, namely:

1. Contact Form 7
2. Elementor PRO
3. Fluent Forms
4. Formidable Forms
5. Forminator
6. Gravity Forms
7. Kali Forms
8. Ninja Forms
9. WPForms
10. Caldera Forms (beta)
11. Groundhogg
12. HappyForms
13. Super Forms
14. Support Candy
15. WeForms
16. wpForo

This covers millions of WordPress sites and we’re quite sure you’re likely using one of them.

Of course, if your form provider isn’t listed, please reach out to us and we’ll see what we can do to include it.

Future Enhancements To Shield’s Antibot Detection Engine

Our new Antibot Detection Engine in Shield 11.0 (due out soon) is just the start.

With Shield building scores for IP addresses on 1000s of WordPress sites, it makes sense to collate these scores centrally, and allow access to this data by everyone.

We plan to build this out later in the year.

Eventually we’ll be able to provide an API lookup for IP address scores and the more contributors we have offering IP reputation info, the more accurate the system becomes, for everyone.

As with every new system there can be some confusion – please do leave us any comments or questions below and we’ll get right back to you, or update the article as required.