Rescuing a Failed Bot Protection Project: Akamai Configuration and Ongoing Support

The client is a digital media company with a team of 20+ people. Their proprietary B2B database platform, a .NET-based web application, has been developed over 15+ years. The CRM grows as the company's audience shares their business details through web forms to access the content on its document management platform.

It happened on a Sunday at the beginning of a month. Bots went through the catalogue pages of the platform non-stop. The client's platform started going down. That month the site was down almost all the time.

The client's business depends on the leads coming through the site, so when the site went down, the business itself was at risk.

The client had been on Cloudflare before. It blocked the bots at the front door, but the bots found a way around it and reached the server directly. Akamai promised a more sophisticated solution, so the client moved to Akamai.

Akamai Security was installed and initially configured by the BV TECH Group servicing the market as an Akamai partner.

The client expected a boxed solution for the bot problem. With these tools everyone promises out-of-the-box protection. In fact, somebody has to figure out the different tools, write the rules and make sure the client's application keeps working behind those rules.

Both the application and a boxed solution like Akamai have to be properly configured and tuned.

When the bots came and the site started going down, BV TECH was managing the Akamai configuration. The client was not happy. What BV TECH did about the bots didn't help, and some of it made things worse

BV Tech knew the Akamai product, but was not able to understand the client's application. The advice was of the generic kind.

For example, Origin Health Detection, enabled on BVTech's advice, produced no noticeable result. Blocking individual IP addresses didn't work either. The last thing BV TECH did was a TLS fingerprint rule. After that real users couldn't get in any more than the bots could.

Our client needed a development partner, not just an Akamai consultancy and implementation services.

Belitsoft took the Akamai configuration into its own hands and started making the changes itself.

First, Belitsoft audited the client's system and took the time to understand how the application worked. For example, we knew which URLs exist and which don't, how many pages a real person opens in a minute, how often a real person submits a form, and where the client's audience comes from. The Akamai rules were created based on that understanding.

Then the engineer went through the settings, to note what worked and what didn't work.

Geo blocking. Blocking whole subnets of one region did work. However, the client had real users in that region too, so we switched it off. Countries with the largest share of malicious traffic were blocked instead, reducing all traffic by 31%.

Rate limiting. This was the biggest win. For some reason BV TECH had resisted it, and Belitsoft insisted on implementing it and further fine-tuning. In one month the rate limiting rules blocked 10.4M requests. Half of them came from a "fail to ban" rule for bots that keep hitting page URLs that don't exist. 4.6M came from bots that open the same pages far more often than a real person would, 0.3M from bots that submit forms too often, and the rest from rules on the home page and a few other pages.

Rules specific to the application. On top of that, Belitsoft added rules that come from knowing exactly what the application does. A block on query string parameters the application never uses removed 1.1M bot requests a month. A blacklist of paths that don't exist, with a whitelist of the ones that do, removed another 0.7M.

Known bots. Known-bot blocking was strengthened to 1.0M requests a month, and the performance drops from crawlers like SEOKicks completely disappeared.

WAF. WAF rules that had been switched off were turned on.

Content Protector. For bots that go through the site the way a real person would, Akamai suggested its Content Protector module. The client bought it, and it blocks another 0.9M requests a month.

Akamai did the bulk of the work. Thanks to the Akamai settings, the load on the server from bots was reduced. However, the work went beyond simply setting up Akamai. The team was working also on the server and the application side.

On the server side, Belitsoft's team upped the server capacity. Scaling the database and web server vertically removed the overload and improved the situation.

On the application side, the team built a bot trap. The catalogue pages the bots go through most were optimized too.

The main result is that the network of the client's websites works again. Now, as the client put it at a recent meeting, "at the moment, fingers crossed, it's not disrupting business, as before it was".

Before the month-long incident with bots attack, the server failed to serve only 3% of requests, which most users never noticed. After a month of bot scrapers going through the site, this figure went up to the point where pages didn't open at all. After Belitsoft's work it went down to 2% and less, which is in fact better than before the attack.

The bots did not go away. They still come. In one of the recent months 31.8M requests came to Akamai, however 20.7M of them, two out of three, were blocked and did not reach the server. It means less load on the servers and no errors and downtimes caused by bots.

This was a one-time rescue project, and it's finished, but Belitsoft remains on standby to provide support.

Bots get smarter year by year, and the settings that work today may be not so effective over time.

That's why tools like Akamai keep adding new configurations and tools to block them, which naturally means additional configuration overhead.

However, so far the client hasn't needed to use this option.